Why does my script use so much memory

[neilontherock]

Hi,

I have a script written that connects to a website using _IECreate() then use _IENavigate to loop through multiple pages. On each page I use _IELinkGetCollection to obtain all the links on each page. For each valid link I then use InetGet to download a temp file.

As my script runs it starts off at about 28k but as it runs memory keeps growing to about 1g.

I do make a call to _IEQuit after I process each search. About 30 web pages are processed per search.

Does anyone know why this is happening and what/if anything I can do to free memory?

Edited August 27, 2010 by neilontherock

[Theri]

Because you aren't ever clearing up the memory used when it loads all those webpages.

Ascend4nt has a thread:

http://www.autoitscript.com/forum/index.php?showtopic=115352

with a udf full of functions to handle processes. You can use that to make a call to your autoit script to free its working load set.

[wakillon]

Hi,

I have a script written that connects to a website using _IECreate() then use _IENavigate to loop through multiple pages. On each page I use _IELinkGetCollection to obtain all the links on each page. For each valid link I then use InetGet to download a temp file.

As my script runs it starts off at about 28k but as it runs memory keeps growing to about 1g.

I do make a call to _IEQuit after I process each search. About 30 web pages are processed per search.

Does anyone know why this is happening and what/if anything I can do to free memory?

Waouh 1 Go !

Show your script for find where is the problem

Or try

_ReduceMemory(ProcessExists(@ScriptName))


Func ReduceMemory($i_PID = -1)
    If $i_PID <> -1 Then
        Local $ai_Handle = DllCall("kernel32.dll", 'int', 'OpenProcess', 'int', 0x1f0fff, 'int', False, 'int', $i_PID)
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', $ai_Handle[0])
        DllCall('kernel32.dll', 'int', 'CloseHandle', 'int', $ai_Handle[0])
    Else
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', -1)
    EndIf
    
    Return $ai_Return[0]
EndFunc;==> ReduceMemory()

Edited August 27, 2010 by wakillon

[neilontherock]

Waouh 1 Go !

Show your script for find where is the problem

Or try

_ReduceMemory(ProcessExists(@ScriptName))


Func ReduceMemory($i_PID = -1)
    If $i_PID <> -1 Then
        Local $ai_Handle = DllCall("kernel32.dll", 'int', 'OpenProcess', 'int', 0x1f0fff, 'int', False, 'int', $i_PID)
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', $ai_Handle[0])
        DllCall('kernel32.dll', 'int', 'CloseHandle', 'int', $ai_Handle[0])
    Else
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', -1)
    EndIf
    
    Return $ai_Return[0]
EndFunc;==> ReduceMemory()

Thanks, I tried your code and it doesn't help. I even wrote code to clean up the "Temporary Internet Files\Content.IE5\" folder as the write runs and that doesn't help either. I know that is working because I'm able to see the files getting deleted as the script executes.

The script starts of with about 30k and within 30 minutes it's shows 1.5 g. This is running on windows 2000 server.

When the memory reaches about 1.3 g, the INetGet function starts failing and 0 bytes is returned.

Doesn't make sense.

[neilontherock]

Hi - yes this does work but it does not reduce Virtual Memory or perhaps the amount of VM that Windows task manager is showing is wrong because my script is deleting the temp files being created by the INetGet function calls.

Thanks for sharing, now if I could only free the Virtual Memory.

Edited August 30, 2010 by neilontherock

[Bert]

Again, you are not posting your code so we can debug it. We are not mind readers nor can we see what is on your screen.

[neilontherock]

Hi - yes this does work but it does not reduce Virtual Memory or perhaps the amount of VM that Windows task manager is showing is wrong because my script is deleting the temp files being created by the INetGet function calls.

Thanks for sharing, now if I could only free the Virtual Memory.

Hi, I have found the problem. What is happening is the following folders are eating up Virtual Memory as the script runs and is never deleted. This has nothing to do with Autoit and just the way Internet Explorer cauches data.

So I created to constants in my code based on the running user;

$WindowTempIEPath1 = "C:\Documents and Settings\<user>\Local Settings\Temp\Temporary Internet Files\Content.IE5\"

$WindowTempIEPath2 = "C:\Documents and Settings\<user>\Local Settings\Temporary Internet Files\Content.IE5\"

...and then called the function with each constant after the script processed a web page. Works like a charm!!!

RecursiveFileSearch (WindowTempIEPath1 )

RecursiveFileSearch (WindowTempIEPath2 )

Func RecursiveFileSearch ($startDir, $depth = 0)

If $depth = 0 Then Global $RFSstring = ""

$search = FileFindFirstFile($startDir & "\*.*")

If @error Then

Return

EndIf

;Search through all files and folders in directory

While 1

$next = FileFindNextFile($search)

If @error Then ExitLoop

;If folder, recurse

If StringInStr(FileGetAttrib($startDir & "\" & $next), "D") Then

RecursiveFileSearch ($startDir & "\" & $next, $depth + 1)

Else

$RFSstring &= $startDir & "\" & $next & "*"

Local $result = FileDelete($startDir & "\" & $next )

EndIf

WEnd

FileClose($search)

If $depth = 0 Then Return StringSplit(StringTrimRight($RFSstring, 1), "*")

EndFunc