Embedded IE memory keeps growing

[FrozenTeeth]

I wrote a program to check the links of a page then log them and launch the links. it will loop thru the links and remove the ones I want. In the code example I'm searching the main yahoo site and removing all the http://www.yahoo.com links then removing any duplicates (_arrayunique) then logging and launching any leftover links. Everything is working perfect except the memory keeps climbing and climbing until the computer runs out of memory. I've tried some cleanup and deleting the array before it starts all over but to no avail. I even found the reduce memory progam (_ReduceMemory) from searching here and it does release some memory but never keeps it to an average low. The final program will be watching a corporate intranet. I used www.yahoo.com as an example.

Thanks for all the help in advance.

I am running this on a Windows 7 64-bit machine with 8GB of memory but it will run finally on a 32-bit server 2003 with only 2GB of memory. (If that matters)

expandcollapse popup

#include <GuiComboBox.au3>
#include <GuiComboBoxEx.au3>
#include <array.au3>
#include <GuiTab.au3>
#include <IE.au3>
#include <GuiEdit.au3>
#include <Process.au3>
#include <File.au3>
#include <String.au3>
#include <ButtonConstants.au3>
#include <GUIConstantsEx.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>
Opt('MustDeclareVars', 1)
;
; ######   ##     ## ####
;##    ##  ##     ##  ##
;##        ##     ##  ##
;##   #### ##     ##  ##
;##    ##  ##     ##  ##
;##    ##  ##     ##  ##
; ######    #######  ####
;
;
Opt("GUIOnEventMode", 1)
;
;dim local global var's
;
Dim $formLinkChecker, $oIE, $GUIActiveX, $stoplinkrun, $arraycount, $oLinks, $oLink
Dim $iNumLinks, $sHTML, $firstarray, $btnStop, $lblStatusUpdate, $lblStatusText

If FileExists(@ScriptDir & "\linksfound.txt") Then FileDelete(@ScriptDir & "\linksfound.txt")
;
;
;
$formLinkChecker = GUICreate("Link Checker", 801, 571, 192, 124)
$btnStop = GUICtrlCreateButton("Stop", 632, 520, 81, 25, $WS_GROUP)
GUICtrlSetOnEvent($btnStop, "btnStopClick")
$lblStatusUpdate = GUICtrlCreateLabel("..........................................................", 49, 522, 573, 17)
GUICtrlSetOnEvent($lblStatusUpdate, "lblStatusUpdateClick")
$lblStatusText = GUICtrlCreateLabel("Status:", 9, 522, 37, 17)
GUICtrlSetOnEvent($lblStatusText, "lblStatusTextClick")
GUISetOnEvent($GUI_EVENT_MINIMIZE, "formLinkCheckerMinimize", $formLinkChecker)
GUISetOnEvent($GUI_EVENT_MAXIMIZE, "formLinkCheckerMaximize", $formLinkChecker)
GUISetOnEvent($GUI_EVENT_RESTORE, "formLinkCheckerRestore", $formLinkChecker)
GUISetOnEvent($GUI_EVENT_CLOSE, "formLinkCheckerClose", $formLinkChecker)
GUISetState(@SW_SHOW)
$oIE = _IECreateEmbedded()
$GUIActiveX = GUICtrlCreateObj($oIE, 5, 5, 790, 510)
$stoplinkrun = True

Func formLinkCheckerMinimize()
EndFunc   ;==>formLinkCheckerMinimize

Func formLinkCheckerMaximize()
EndFunc   ;==>formLinkCheckerMaximize

Func formLinkCheckerRestore()
EndFunc   ;==>formLinkCheckerRestore

Func btnStopClick()
    $stoplinkrun = False
EndFunc   ;==>btnStopClick

Func lblStatusUpdateClick()
EndFunc   ;==>lblStatusUpdateClick

Func lblStatusTextClick()
EndFunc   ;==>lblStatusTextClick

Func formLinkCheckerClose()
    Exit
EndFunc   ;==>formLinkCheckerClose

_IENavigate($oIE, "http://www.yahoo.com")
While 1
    If $stoplinkrun = True Then
        _IENavigate($oIE, "http://www.yahoo.com")
        StartLinkProcess()
    EndIf
WEnd

Func StartLinkProcess()
    Local $firstarray[1], $arraycountNew, $secondarray[1], $thirdarray[1]
    $arraycount = 0
    $oLinks = _IELinkGetCollection($oIE)
    $iNumLinks = @extended
    For $oLink In $oLinks
        $arraycount += 1
        If IsArray($firstarray) Then
            _ArrayAdd($firstarray, $oLink.href)
        EndIf
    Next
    $firstarray[0] = $arraycount
    ;clear some vars not used any more
    $sHTML = ""
    $iNumLinks = 0
    $arraycount = 0
    $oLinks = ""
    $oLink = ""
    ;_ArrayDisplay($firstarray, "1st")
    ;
    ;remove known unwanted links
    ;
    $arraycountNew = 0
    For $u = 1 To $firstarray[0]
        If StringLeft($firstarray[$u], 20) <> "http://www.yahoo.com" Then
            $arraycountNew += 1
            _ArrayAdd($secondarray, $firstarray[$u])
        EndIf
        ;remove non http links
        If StringLeft($firstarray[$u], 5) <> "http:" Then
            $arraycountNew += 1
            _ArrayAdd($secondarray, $firstarray[$u])
        EndIf
    Next

    $secondarray[0] = $arraycountNew
    $arraycountNew = 0
    ;_ArrayDisplay($secondarray, "2nd")
    For $x = 0 To $firstarray[0]
        _ArrayDelete($firstarray, $x)
    Next
    ;
    ;if duplicates create unique array
    ;
    $thirdarray = _ArrayUnique($secondarray)
    ;_ArrayDisplay($thirdarray, "3rd")
    For $x = 0 To $secondarray[0]
        _ArrayDelete($secondarray, $x)
    Next
    ;
    ; launch the link
    ;
    For $u = 1 To $thirdarray[0]
        RecordTheLinks($thirdarray[$u])
        GUICtrlSetData($lblStatusUpdate, $thirdarray[$u])
        ;
        ;check if stop button pushed then exit loop
        ;
        If $stoplinkrun = False Then
            $u = $thirdarray[0]
        EndIf
    Next
    ;
    ;cleanup vars
    ;
    $u = 0
    For $x = 1 To $thirdarray[0]
        _ArrayDelete($thirdarray, $x)
    Next
    _ReduceMemory()
    ;
    ;wait 1 minute and try again
    ;
    Sleep(60000)
EndFunc   ;==>StartLinkProcess
;
;
;
Func _ReduceMemory($i_PID = -1); thx to wOuters
    If $i_PID <> -1 Then
        Local $ai_Handle = DllCall("kernel32.dll", 'int', 'OpenProcess', 'int', 0x1f0fff, 'int', False, 'int', $i_PID)
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', $ai_Handle[0])
        DllCall('kernel32.dll', 'int', 'CloseHandle', 'int', $ai_Handle[0])
    Else
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', -1)
    EndIf
    Return $ai_Return[0]
EndFunc   ;==>_ReduceMemory

Func RecordTheLinks($link2record)
    _FileWriteLog(@ScriptDir & "\linksfound.txt", $link2record)
    _IENavigate($oIE, $link2record)
EndFunc   ;==>RecordTheLinks

Edited March 29, 2010 by FrozenTeeth

[FrozenTeeth]

Anyone come up with anything? I haven't found anything yet.

[JohnOne]

Sounds like you have whats called a Memory Leak in your code.

Sorry but detecting such a thing is way beyond me.

Perhaps some sort of logging your function calls next to an extra memory check log, would be a start in debugging it.

Edited March 29, 2010 by JohnOne

[FrozenTeeth]

Well I have looked into this some more and I can't find anyway to tell how much memory my program is using while it's running. I searched the forums and most memory hits are about reading and writing memory itself. I looked thru the help file and couldn't find anything that tell me the allocated memory of a process.

any help is appreciated. thanks

[jchd]

There is indeed a memory leak occuring with this code but I don't believe it's due to your code by itself in any way.

Here's the version I tried:

expandcollapse popup

#include <GuiComboBox.au3>
#include <GuiComboBoxEx.au3>
#include <array.au3>
#include <GuiTab.au3>
#include <IE.au3>
#include <GuiEdit.au3>
#include <Process.au3>
#include <File.au3>
#include <String.au3>
#include <ButtonConstants.au3>
#include <GUIConstantsEx.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>
Opt('MustDeclareVars', 1)

;~ #RequireAdmin        ; a (failed) attempt at curing the memory leak

;
; ###### ##     ## ####
;## ## ##   ## ##
;##     ##  ## ##
;## #### ##     ## ##
;## ## ##   ## ##
;## ## ##   ## ##
; ######    ####### ####
;
;
Opt("GUIOnEventMode", 1)
;
;dim local global var's
;
Global $formLinkChecker, $oIE, $GUIActiveX, $stoplinkrun            ; , $arraycount, $oLinks, $oLink
Global $btnStop, $lblStatusUpdate, $lblStatusText                   ; $sHTML, $iNumLinks, $firstarray,

If FileExists(@ScriptDir & "\linksfound.txt") Then FileDelete(@ScriptDir & "\linksfound.txt")
;
;
;
$formLinkChecker = GUICreate("Link Checker", 801, 571, 192, 124)
$btnStop = GUICtrlCreateButton("Stop", 632, 520, 81, 25, $WS_GROUP)
GUICtrlSetOnEvent($btnStop, "btnStopClick")
$lblStatusUpdate = GUICtrlCreateLabel("..........................................................", 49, 522, 573, 17)
GUICtrlSetOnEvent($lblStatusUpdate, "lblStatusUpdateClick")
$lblStatusText = GUICtrlCreateLabel("Status:", 9, 522, 37, 17)
GUICtrlSetOnEvent($lblStatusText, "lblStatusTextClick")
GUISetOnEvent($GUI_EVENT_MINIMIZE, "formLinkCheckerMinimize", $formLinkChecker)
GUISetOnEvent($GUI_EVENT_MAXIMIZE, "formLinkCheckerMaximize", $formLinkChecker)
GUISetOnEvent($GUI_EVENT_RESTORE, "formLinkCheckerRestore", $formLinkChecker)
GUISetOnEvent($GUI_EVENT_CLOSE, "formLinkCheckerClose", $formLinkChecker)
GUISetState(@SW_SHOW)
$oIE = _IECreateEmbedded()
$GUIActiveX = GUICtrlCreateObj($oIE, 5, 5, 790, 510)
$stoplinkrun = True

Func formLinkCheckerMinimize()
EndFunc ;==>formLinkCheckerMinimize

Func formLinkCheckerMaximize()
EndFunc ;==>formLinkCheckerMaximize

Func formLinkCheckerRestore()
EndFunc ;==>formLinkCheckerRestore

Func btnStopClick()
    $stoplinkrun = False
EndFunc ;==>btnStopClick

Func lblStatusUpdateClick()
EndFunc ;==>lblStatusUpdateClick

Func lblStatusTextClick()
EndFunc ;==>lblStatusTextClick

Func formLinkCheckerClose()
    Exit
EndFunc ;==>formLinkCheckerClose

;~ _IENavigate($oIE, "http://www.yahoo.com")
While 1
    If $stoplinkrun = True Then
    _IENavigate($oIE, "http://www.yahoo.com")
    StartLinkProcess()
    EndIf
WEnd

Func StartLinkProcess()
    Local $oLinks = _IELinkGetCollection($oIE)
    If @error Then
        ConsoleWrite("Link collection error " & @error & @LF)
;~      Sleep(60000)
        Sleep(1000)
        Return
    EndIf
    Local $u, $linkarray[@extended + 1], $oLink
    For $oLink In $oLinks
        $u += 1
        $linkarray[$u] = $oLink.href
    Next
    $linkarray[0] = $u
    $oLinks = 0
    ;_ArrayDisplay($linkarray, "1st")
    ;
    ;remove known unwanted links
    ;
    For $u = 1 To $linkarray[0]
    If Not StringRegExp($linkarray[$u], "\Ahttp://(?!www\.yahoo\.com)", 0) Then $linkarray[$u] = ''
;~  If StringLeft($linkarray[$u], 5) <> "http:" Or StringLeft($linkarray[$u], 20) = "http://www.yahoo.com" Then $linkarray[$u] = ''
    Next
    ;_ArrayDisplay($linkarray, "2nd")

    ;
    ;if duplicates create unique array
    ;
    $linkarray[0] = ''      ; _ArrayUnique will recreate the count at 0
    $linkarray = _ArrayUnique($linkarray)
;~  _ArrayDisplay($linkarray, "3rd")
    ;
    ; launch the link
    ;
    For $u = 1 To $linkarray[0]
        If $linkarray[$u] <> '' Then
            RecordTheLinks($linkarray[$u])
            GUICtrlSetData($lblStatusUpdate, $linkarray[$u])
        EndIf
    ;
    ;check if stop button pushed then exit loop
    ;
    If $stoplinkrun = False Then ExitLoop
    Next
    ;
    ;wait 1 minute and try again
    ;
;~  Sleep(60000)
    Sleep(1000)
EndFunc ;==>StartLinkProcess


Func RecordTheLinks($link2record)
    _FileWriteLog(@ScriptDir & "\linksfound.txt", $link2record)
;~  _IENavigate($oIE, $link2record)
EndFunc ;==>RecordTheLinks

I've taken some liberty to simplify your code without changing its behavior. The memory leakage is linked to heavy use of _IENavigate. Simply commenting out the navigation to the sites filtered slows down memory growth considerably. Also I suspect that only newly visited pages cause leakage. I don't know enough about _IE* to be sure but it might have something to do with the warning about Vista+ in the help file about _IENavigate. Maybe a cache thing not releasing correctly? Memory leak occurs as well on two XP SP3 x86 PCs where I tried it here. I used IE8, patches are current.

BTW, allow me a bit of comment on your code. You don't have to allocate 3 distinct arrays to do what you want. Even if you re-insert code you ommited for posting, the simplified version is sufficient. Also beware of Dim, use either Global or Local so that your intent is clearer. Beware of variables declared as Dim at the global level and then redeclared inside functions. You don't have to clear local variables (inside functions), they will be cleared at function exit. I corrected the second test for filtering and replaced two successive tests by a single regexp. _ArrayAdd & _ArrayDelete can become expensive and were useless in your case.

I hope some _IE guru will visit your topic soon and shed light on this.

[FrozenTeeth]

Thanks for recode. Yes the _arraydelete was bogus I new that could be removed.

I'm so used to using Dim from my BASIC days

The only real code I omitted was the distinct web pages I'm looking for.

I might try to launch the IE in a separate exe maybe this will show the end-user the webpage then quit.

Thanks again for all your help. I'll continue the battle.

[jchd]

Don't take my remarks too badly, I intended nothing weird.

Since the _IE gurus are busy elsewhere, you could devote some time trying to temporarily leave out IE embedding in your GUI and use a classical _IEAttach. It would allow you to force deletion of the _IEobject (assign 0 to it, as you know) at every run, while deletion is postponed until closing the embedding GUI in the embedded approach.

It's possible that this could avoid the leak completely. With computers and women alike, less pretty often means more reliable. Go figure!

[FrozenTeeth]

No offense was taken at all. That's how we learn! You have been very helpful.

[Tvern]

I was able to reproduce this on Win7 Ultimate.

I'm now running a test using

$oIE.navigate($url)
_IELoadWait($oIE,0,5000)

Instead of _IENavigate to see if the problem remains.

Edit:

It seems to have some positive effect, but I think it's mostly because of the timeout in _IELoadWait(). After running for a while, my memory usage stayed at about 40MB. It didn't seem to matter much if I used $oIE.navigate($url) or _IENavigate($oIE,$Url,0), combined with _IELoadWait($oIE,0,5000).

I did notice some of the pages had video streams. I could imagine those filling the memory pretty fast if you let them.

Edited April 1, 2010 by Tvern

[FrozenTeeth]

I used yahoo as an example. You could change it to any page. The internal web site I'm checking has over 1000 pages and they don't contain video mainly text and pictures.

[jchd]

Anyway, the browser should release loaded items, video or whatever.

But here's the Cavalry coming ;-)

[DaleHohm]

Assuming you run it outside the GUI and it doesn't happen, it could be an issue with the GUI object container in AutoI. Suggest you investigate an alternate method by searching for IECreate2 in the forum to attach a full browser instance instead of using the GUI container.

Dale

[FrozenTeeth]

I have fixed the issue for now. The memory leaks don't happen as long as I close all IE windows when the program starts (The program kills them for me). Then it launches the IE from a different exe this way it has it's own process and does the IECreate and IEQuit just fine on it's own. The end user will just have to click a button to launch the next window to check the link.

THanks for all the help!

[jchd]

Your finding may help others. Nice to know you're on the right track now.

Cheers.