_INetGetSource Bypass Memory... Is it possible?

[tr1px]

Hey everyone...

I have a problem. I am downloading some source code from a web page but the source code is to large and I get an error that AutoIt can allocate enough memory. I am trying to find a way to bypass the memory and write the source to a textfile live so that the memory does not get filled. Below is my code much help is appreciated.

#include <INet.au3>
#include <File.au3>

$logFile = FileOpen("log.txt", 1)

$url = "http://www.somesite.com/page.php?page_id=../../../var/log/pronto_extranet"

$page = _INetGetSource($url)
sleep(5000)

$pageArray = StringSplit($page, @CRLF)
For $i = 1 To $pageArray[0]
    FileWriteLine($logFile, $i & @CRLF)
Next

MsgBox(0, "Download...", "Completed!")
Exit

I would like somehow the $page variable to be written right to the text file without loading completely to memory whic is the array and then to text. Please HELP!!!

[PsaltyDS]

Hey everyone...

I have a problem. I am downloading some source code from a web page but the source code is to large and I get an error that AutoIt can allocate enough memory. I am trying to find a way to bypass the memory and write the source to a textfile live so that the memory does not get filled. Below is my code much help is appreciated.

#include <INet.au3>
#include <File.au3>

$logFile = FileOpen("log.txt", 1)

$url = "http://www.somesite.com/page.php?page_id=../../../var/log/pronto_extranet"

$page = _INetGetSource($url)
sleep(5000)

$pageArray = StringSplit($page, @CRLF)
For $i = 1 To $pageArray[0]
    FileWriteLine($logFile, $i & @CRLF)
Next

MsgBox(0, "Download...", "Completed!")
Exit

I would like somehow the $page variable to be written right to the text file without loading completely to memory whic is the array and then to text. Please HELP!!!

Quit doing the string manipulation and line numbering on the fly. Save all the data to a temp file and take care of that later:

#include <INet.au3>

$logFile = FileOpen("LogTemp.txt", 1)
$url = "http://www.somesite.com/page.php?page_id=../../../var/log/pronto_extranet"
$page = _INetGetSource($url)
FileWrite($logFile, $page)

Now you can just loop FileReadLine($LogTemp) and add the lines to the other file.

[tr1px]

This does not solve anything. It is still loading the URL into memory first before writing it into the text file.

....../var/log/pronto_extranet is a huge text file which is to big to be loaded into memory first. The only way would be to download the file but is not possible since I need the source code.

If anyone else responds please read the TITLE which clearly states bypassing memory.

Edited September 17, 2008 by tr1px

[PsaltyDS]

This does not solve anything. It is still loading the URL into memory first before writing it into the text file.

....../var/log/pronto_extranet is a huge text file which is to big to be loaded into memory first. The only way would be to download the file but is not possible since I need the source code.

If anyone else responds please read the TITLE which clearly states bypassing memory.

Did you try it? I read your title and understand you seem to have memory resource issues, but the StringSplit() and other string manipulation is resulting in multiple copies of the text in multiple variable structures in memory. What I proposed reduces the number of copies of the data in memory to one, which may be enough to fit it in your available resources.

[trancexx]

I posted a possible solution for you in that other thread (writing occures every 1024 bytes).

How huge is that 'huge text file' anyway?

[tr1px]

its over 500mb and yes I did try your solution which was my first one PsaltyDS.

FileWrite($logFile, _INetGetSource($url))

That is the same as your code. In case you still wonder I did try your exact code also. Buffers seem like a good idea trancexx I will try that.