tr1px Posted September 17, 2008 Share Posted September 17, 2008 Hey everyone... I have a problem. I am downloading some source code from a web page but the source code is to large and I get an error that AutoIt can allocate enough memory. I am trying to find a way to bypass the memory and write the source to a textfile live so that the memory does not get filled. Below is my code much help is appreciated. #include <INet.au3> #include <File.au3> $logFile = FileOpen("log.txt", 1) $url = "http://www.somesite.com/page.php?page_id=../../../var/log/pronto_extranet" $page = _INetGetSource($url) sleep(5000) $pageArray = StringSplit($page, @CRLF) For $i = 1 To $pageArray[0] FileWriteLine($logFile, $i & @CRLF) Next MsgBox(0, "Download...", "Completed!") Exit I would like somehow the $page variable to be written right to the text file without loading completely to memory whic is the array and then to text. Please HELP!!! Link to comment Share on other sites More sharing options...
PsaltyDS Posted September 17, 2008 Share Posted September 17, 2008 Hey everyone... I have a problem. I am downloading some source code from a web page but the source code is to large and I get an error that AutoIt can allocate enough memory. I am trying to find a way to bypass the memory and write the source to a textfile live so that the memory does not get filled. Below is my code much help is appreciated. #include <INet.au3> #include <File.au3> $logFile = FileOpen("log.txt", 1) $url = "http://www.somesite.com/page.php?page_id=../../../var/log/pronto_extranet" $page = _INetGetSource($url) sleep(5000) $pageArray = StringSplit($page, @CRLF) For $i = 1 To $pageArray[0] FileWriteLine($logFile, $i & @CRLF) Next MsgBox(0, "Download...", "Completed!") Exit I would like somehow the $page variable to be written right to the text file without loading completely to memory whic is the array and then to text. Please HELP!!! Quit doing the string manipulation and line numbering on the fly. Save all the data to a temp file and take care of that later: #include <INet.au3> $logFile = FileOpen("LogTemp.txt", 1) $url = "http://www.somesite.com/page.php?page_id=../../../var/log/pronto_extranet" $page = _INetGetSource($url) FileWrite($logFile, $page) Now you can just loop FileReadLine($LogTemp) and add the lines to the other file. Valuater's AutoIt 1-2-3, Class... Is now in Session!For those who want somebody to write the script for them: RentACoder"Any technology distinguishable from magic is insufficiently advanced." -- Geek's corollary to Clarke's law Link to comment Share on other sites More sharing options...
tr1px Posted September 17, 2008 Author Share Posted September 17, 2008 (edited) This does not solve anything. It is still loading the URL into memory first before writing it into the text file. ....../var/log/pronto_extranet is a huge text file which is to big to be loaded into memory first. The only way would be to download the file but is not possible since I need the source code. If anyone else responds please read the TITLE which clearly states bypassing memory. Edited September 17, 2008 by tr1px Link to comment Share on other sites More sharing options...
PsaltyDS Posted September 17, 2008 Share Posted September 17, 2008 This does not solve anything. It is still loading the URL into memory first before writing it into the text file......./var/log/pronto_extranet is a huge text file which is to big to be loaded into memory first. The only way would be to download the file but is not possible since I need the source code.If anyone else responds please read the TITLE which clearly states bypassing memory.Did you try it? I read your title and understand you seem to have memory resource issues, but the StringSplit() and other string manipulation is resulting in multiple copies of the text in multiple variable structures in memory. What I proposed reduces the number of copies of the data in memory to one, which may be enough to fit it in your available resources. Valuater's AutoIt 1-2-3, Class... Is now in Session!For those who want somebody to write the script for them: RentACoder"Any technology distinguishable from magic is insufficiently advanced." -- Geek's corollary to Clarke's law Link to comment Share on other sites More sharing options...
trancexx Posted September 17, 2008 Share Posted September 17, 2008 I posted a possible solution for you in that other thread (writing occures every 1024 bytes). How huge is that 'huge text file' anyway? ♡♡♡ . eMyvnE Link to comment Share on other sites More sharing options...
tr1px Posted September 18, 2008 Author Share Posted September 18, 2008 its over 500mb and yes I did try your solution which was my first one PsaltyDS. FileWrite($logFile, _INetGetSource($url)) That is the same as your code. In case you still wonder I did try your exact code also. Buffers seem like a good idea trancexx I will try that. Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now