Sign in to follow this  
Followers 0
tr1px

_INetGetSource Bypass Memory... Is it possible?

6 posts in this topic

Hey everyone...

I have a problem. I am downloading some source code from a web page but the source code is to large and I get an error that AutoIt can allocate enough memory. I am trying to find a way to bypass the memory and write the source to a textfile live so that the memory does not get filled. Below is my code much help is appreciated.

#include <INet.au3>
#include <File.au3>

$logFile = FileOpen("log.txt", 1)

$url = "http://www.somesite.com/page.php?page_id=../../../var/log/pronto_extranet"

$page = _INetGetSource($url)
sleep(5000)

$pageArray = StringSplit($page, @CRLF)
For $i = 1 To $pageArray[0]
    FileWriteLine($logFile, $i & @CRLF)
Next

MsgBox(0, "Download...", "Completed!")
Exit

I would like somehow the $page variable to be written right to the text file without loading completely to memory whic is the array and then to text. Please HELP!!!

Share this post


Link to post
Share on other sites



Hey everyone...

I have a problem. I am downloading some source code from a web page but the source code is to large and I get an error that AutoIt can allocate enough memory. I am trying to find a way to bypass the memory and write the source to a textfile live so that the memory does not get filled. Below is my code much help is appreciated.

#include <INet.au3>
#include <File.au3>

$logFile = FileOpen("log.txt", 1)

$url = "http://www.somesite.com/page.php?page_id=../../../var/log/pronto_extranet"

$page = _INetGetSource($url)
sleep(5000)

$pageArray = StringSplit($page, @CRLF)
For $i = 1 To $pageArray[0]
    FileWriteLine($logFile, $i & @CRLF)
Next

MsgBox(0, "Download...", "Completed!")
Exit

I would like somehow the $page variable to be written right to the text file without loading completely to memory whic is the array and then to text. Please HELP!!!

Quit doing the string manipulation and line numbering on the fly. Save all the data to a temp file and take care of that later:
#include <INet.au3>

$logFile = FileOpen("LogTemp.txt", 1)
$url = "http://www.somesite.com/page.php?page_id=../../../var/log/pronto_extranet"
$page = _INetGetSource($url)
FileWrite($logFile, $page)

Now you can just loop FileReadLine($LogTemp) and add the lines to the other file.

:)


Valuater's AutoIt 1-2-3, Class... Is now in Session!For those who want somebody to write the script for them: RentACoder"Any technology distinguishable from magic is insufficiently advanced." -- Geek's corollary to Clarke's law

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

This does not solve anything. It is still loading the URL into memory first before writing it into the text file.

....../var/log/pronto_extranet is a huge text file which is to big to be loaded into memory first. The only way would be to download the file but is not possible since I need the source code.

If anyone else responds please read the TITLE which clearly states bypassing memory.

Edited by tr1px

Share this post


Link to post
Share on other sites

This does not solve anything. It is still loading the URL into memory first before writing it into the text file.

....../var/log/pronto_extranet is a huge text file which is to big to be loaded into memory first. The only way would be to download the file but is not possible since I need the source code.

If anyone else responds please read the TITLE which clearly states bypassing memory.

Did you try it? I read your title and understand you seem to have memory resource issues, but the StringSplit() and other string manipulation is resulting in multiple copies of the text in multiple variable structures in memory. What I proposed reduces the number of copies of the data in memory to one, which may be enough to fit it in your available resources.

:)


Valuater's AutoIt 1-2-3, Class... Is now in Session!For those who want somebody to write the script for them: RentACoder"Any technology distinguishable from magic is insufficiently advanced." -- Geek's corollary to Clarke's law

Share this post


Link to post
Share on other sites

I posted a possible solution for you in that other thread (writing occures every 1024 bytes).

How huge is that 'huge text file' anyway?


♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites

its over 500mb and yes I did try your solution which was my first one PsaltyDS.

FileWrite($logFile, _INetGetSource($url))

That is the same as your code. In case you still wonder I did try your exact code also. Buffers seem like a good idea trancexx I will try that.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0