Sign in to follow this  
Followers 0
FrozenTeeth

Embedded IE memory keeps growing

14 posts in this topic

#1 ·  Posted (edited)

I wrote a program to check the links of a page then log them and launch the links. it will loop thru the links and remove the ones I want. In the code example I'm searching the main yahoo site and removing all the http://www.yahoo.com links then removing any duplicates (_arrayunique) then logging and launching any leftover links. Everything is working perfect except the memory keeps climbing and climbing until the computer runs out of memory. I've tried some cleanup and deleting the array before it starts all over but to no avail. I even found the reduce memory progam (_ReduceMemory) from searching here and it does release some memory but never keeps it to an average low. The final program will be watching a corporate intranet. I used www.yahoo.com as an example.

Thanks for all the help in advance.

I am running this on a Windows 7 64-bit machine with 8GB of memory but it will run finally on a 32-bit server 2003 with only 2GB of memory. (If that matters)

#include <GuiComboBox.au3>
#include <GuiComboBoxEx.au3>
#include <array.au3>
#include <GuiTab.au3>
#include <IE.au3>
#include <GuiEdit.au3>
#include <Process.au3>
#include <File.au3>
#include <String.au3>
#include <ButtonConstants.au3>
#include <GUIConstantsEx.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>
Opt('MustDeclareVars', 1)
;
; ######   ##     ## ####
;##    ##  ##     ##  ##
;##        ##     ##  ##
;##   #### ##     ##  ##
;##    ##  ##     ##  ##
;##    ##  ##     ##  ##
; ######    #######  ####
;
;
Opt("GUIOnEventMode", 1)
;
;dim local global var's
;
Dim $formLinkChecker, $oIE, $GUIActiveX, $stoplinkrun, $arraycount, $oLinks, $oLink
Dim $iNumLinks, $sHTML, $firstarray, $btnStop, $lblStatusUpdate, $lblStatusText

If FileExists(@ScriptDir & "\linksfound.txt") Then FileDelete(@ScriptDir & "\linksfound.txt")
;
;
;
$formLinkChecker = GUICreate("Link Checker", 801, 571, 192, 124)
$btnStop = GUICtrlCreateButton("Stop", 632, 520, 81, 25, $WS_GROUP)
GUICtrlSetOnEvent($btnStop, "btnStopClick")
$lblStatusUpdate = GUICtrlCreateLabel("..........................................................", 49, 522, 573, 17)
GUICtrlSetOnEvent($lblStatusUpdate, "lblStatusUpdateClick")
$lblStatusText = GUICtrlCreateLabel("Status:", 9, 522, 37, 17)
GUICtrlSetOnEvent($lblStatusText, "lblStatusTextClick")
GUISetOnEvent($GUI_EVENT_MINIMIZE, "formLinkCheckerMinimize", $formLinkChecker)
GUISetOnEvent($GUI_EVENT_MAXIMIZE, "formLinkCheckerMaximize", $formLinkChecker)
GUISetOnEvent($GUI_EVENT_RESTORE, "formLinkCheckerRestore", $formLinkChecker)
GUISetOnEvent($GUI_EVENT_CLOSE, "formLinkCheckerClose", $formLinkChecker)
GUISetState(@SW_SHOW)
$oIE = _IECreateEmbedded()
$GUIActiveX = GUICtrlCreateObj($oIE, 5, 5, 790, 510)
$stoplinkrun = True

Func formLinkCheckerMinimize()
EndFunc   ;==>formLinkCheckerMinimize

Func formLinkCheckerMaximize()
EndFunc   ;==>formLinkCheckerMaximize

Func formLinkCheckerRestore()
EndFunc   ;==>formLinkCheckerRestore

Func btnStopClick()
    $stoplinkrun = False
EndFunc   ;==>btnStopClick

Func lblStatusUpdateClick()
EndFunc   ;==>lblStatusUpdateClick

Func lblStatusTextClick()
EndFunc   ;==>lblStatusTextClick

Func formLinkCheckerClose()
    Exit
EndFunc   ;==>formLinkCheckerClose

_IENavigate($oIE, "http://www.yahoo.com")
While 1
    If $stoplinkrun = True Then
        _IENavigate($oIE, "http://www.yahoo.com")
        StartLinkProcess()
    EndIf
WEnd

Func StartLinkProcess()
    Local $firstarray[1], $arraycountNew, $secondarray[1], $thirdarray[1]
    $arraycount = 0
    $oLinks = _IELinkGetCollection($oIE)
    $iNumLinks = @extended
    For $oLink In $oLinks
        $arraycount += 1
        If IsArray($firstarray) Then
            _ArrayAdd($firstarray, $oLink.href)
        EndIf
    Next
    $firstarray[0] = $arraycount
    ;clear some vars not used any more
    $sHTML = ""
    $iNumLinks = 0
    $arraycount = 0
    $oLinks = ""
    $oLink = ""
    ;_ArrayDisplay($firstarray, "1st")
    ;
    ;remove known unwanted links
    ;
    $arraycountNew = 0
    For $u = 1 To $firstarray[0]
        If StringLeft($firstarray[$u], 20) <> "http://www.yahoo.com" Then
            $arraycountNew += 1
            _ArrayAdd($secondarray, $firstarray[$u])
        EndIf
        ;remove non http links
        If StringLeft($firstarray[$u], 5) <> "http:" Then
            $arraycountNew += 1
            _ArrayAdd($secondarray, $firstarray[$u])
        EndIf
    Next

    $secondarray[0] = $arraycountNew
    $arraycountNew = 0
    ;_ArrayDisplay($secondarray, "2nd")
    For $x = 0 To $firstarray[0]
        _ArrayDelete($firstarray, $x)
    Next
    ;
    ;if duplicates create unique array
    ;
    $thirdarray = _ArrayUnique($secondarray)
    ;_ArrayDisplay($thirdarray, "3rd")
    For $x = 0 To $secondarray[0]
        _ArrayDelete($secondarray, $x)
    Next
    ;
    ; launch the link
    ;
    For $u = 1 To $thirdarray[0]
        RecordTheLinks($thirdarray[$u])
        GUICtrlSetData($lblStatusUpdate, $thirdarray[$u])
        ;
        ;check if stop button pushed then exit loop
        ;
        If $stoplinkrun = False Then
            $u = $thirdarray[0]
        EndIf
    Next
    ;
    ;cleanup vars
    ;
    $u = 0
    For $x = 1 To $thirdarray[0]
        _ArrayDelete($thirdarray, $x)
    Next
    _ReduceMemory()
    ;
    ;wait 1 minute and try again
    ;
    Sleep(60000)
EndFunc   ;==>StartLinkProcess
;
;
;
Func _ReduceMemory($i_PID = -1); thx to wOuters
    If $i_PID <> -1 Then
        Local $ai_Handle = DllCall("kernel32.dll", 'int', 'OpenProcess', 'int', 0x1f0fff, 'int', False, 'int', $i_PID)
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', $ai_Handle[0])
        DllCall('kernel32.dll', 'int', 'CloseHandle', 'int', $ai_Handle[0])
    Else
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', -1)
    EndIf
    Return $ai_Return[0]
EndFunc   ;==>_ReduceMemory

Func RecordTheLinks($link2record)
    _FileWriteLog(@ScriptDir & "\linksfound.txt", $link2record)
    _IENavigate($oIE, $link2record)
EndFunc   ;==>RecordTheLinks
Edited by FrozenTeeth

Share this post


Link to post
Share on other sites



Anyone come up with anything? I haven't found anything yet.

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

Sounds like you have whats called a Memory Leak in your code.

Sorry but detecting such a thing is way beyond me.

Perhaps some sort of logging your function calls next to an extra memory check log, would be a start in debugging it.

Edited by JohnOne

AutoIt Absolute Beginners    Require a serial    Pause Script    Video Tutorials by Morthawt   ipify 

Monkey's are, like, natures humans.

Share this post


Link to post
Share on other sites

Well I have looked into this some more and I can't find anyway to tell how much memory my program is using while it's running. I searched the forums and most memory hits are about reading and writing memory itself. I looked thru the help file and couldn't find anything that tell me the allocated memory of a process.

any help is appreciated. thanks

Share this post


Link to post
Share on other sites

There is indeed a memory leak occuring with this code but I don't believe it's due to your code by itself in any way.

Here's the version I tried:

#include <GuiComboBox.au3>
#include <GuiComboBoxEx.au3>
#include <array.au3>
#include <GuiTab.au3>
#include <IE.au3>
#include <GuiEdit.au3>
#include <Process.au3>
#include <File.au3>
#include <String.au3>
#include <ButtonConstants.au3>
#include <GUIConstantsEx.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>
Opt('MustDeclareVars', 1)

;~ #RequireAdmin        ; a (failed) attempt at curing the memory leak

;
; ###### ##     ## ####
;## ## ##   ## ##
;##     ##  ## ##
;## #### ##     ## ##
;## ## ##   ## ##
;## ## ##   ## ##
; ######    ####### ####
;
;
Opt("GUIOnEventMode", 1)
;
;dim local global var's
;
Global $formLinkChecker, $oIE, $GUIActiveX, $stoplinkrun            ; , $arraycount, $oLinks, $oLink
Global $btnStop, $lblStatusUpdate, $lblStatusText                   ; $sHTML, $iNumLinks, $firstarray,

If FileExists(@ScriptDir & "\linksfound.txt") Then FileDelete(@ScriptDir & "\linksfound.txt")
;
;
;
$formLinkChecker = GUICreate("Link Checker", 801, 571, 192, 124)
$btnStop = GUICtrlCreateButton("Stop", 632, 520, 81, 25, $WS_GROUP)
GUICtrlSetOnEvent($btnStop, "btnStopClick")
$lblStatusUpdate = GUICtrlCreateLabel("..........................................................", 49, 522, 573, 17)
GUICtrlSetOnEvent($lblStatusUpdate, "lblStatusUpdateClick")
$lblStatusText = GUICtrlCreateLabel("Status:", 9, 522, 37, 17)
GUICtrlSetOnEvent($lblStatusText, "lblStatusTextClick")
GUISetOnEvent($GUI_EVENT_MINIMIZE, "formLinkCheckerMinimize", $formLinkChecker)
GUISetOnEvent($GUI_EVENT_MAXIMIZE, "formLinkCheckerMaximize", $formLinkChecker)
GUISetOnEvent($GUI_EVENT_RESTORE, "formLinkCheckerRestore", $formLinkChecker)
GUISetOnEvent($GUI_EVENT_CLOSE, "formLinkCheckerClose", $formLinkChecker)
GUISetState(@SW_SHOW)
$oIE = _IECreateEmbedded()
$GUIActiveX = GUICtrlCreateObj($oIE, 5, 5, 790, 510)
$stoplinkrun = True

Func formLinkCheckerMinimize()
EndFunc ;==>formLinkCheckerMinimize

Func formLinkCheckerMaximize()
EndFunc ;==>formLinkCheckerMaximize

Func formLinkCheckerRestore()
EndFunc ;==>formLinkCheckerRestore

Func btnStopClick()
    $stoplinkrun = False
EndFunc ;==>btnStopClick

Func lblStatusUpdateClick()
EndFunc ;==>lblStatusUpdateClick

Func lblStatusTextClick()
EndFunc ;==>lblStatusTextClick

Func formLinkCheckerClose()
    Exit
EndFunc ;==>formLinkCheckerClose

;~ _IENavigate($oIE, "http://www.yahoo.com")
While 1
    If $stoplinkrun = True Then
    _IENavigate($oIE, "http://www.yahoo.com")
    StartLinkProcess()
    EndIf
WEnd

Func StartLinkProcess()
    Local $oLinks = _IELinkGetCollection($oIE)
    If @error Then
        ConsoleWrite("Link collection error " & @error & @LF)
;~      Sleep(60000)
        Sleep(1000)
        Return
    EndIf
    Local $u, $linkarray[@extended + 1], $oLink
    For $oLink In $oLinks
        $u += 1
        $linkarray[$u] = $oLink.href
    Next
    $linkarray[0] = $u
    $oLinks = 0
    ;_ArrayDisplay($linkarray, "1st")
    ;
    ;remove known unwanted links
    ;
    For $u = 1 To $linkarray[0]
    If Not StringRegExp($linkarray[$u], "\Ahttp://(?!www\.yahoo\.com)", 0) Then $linkarray[$u] = ''
;~  If StringLeft($linkarray[$u], 5) <> "http:" Or StringLeft($linkarray[$u], 20) = "http://www.yahoo.com" Then $linkarray[$u] = ''
    Next
    ;_ArrayDisplay($linkarray, "2nd")

    ;
    ;if duplicates create unique array
    ;
    $linkarray[0] = ''      ; _ArrayUnique will recreate the count at 0
    $linkarray = _ArrayUnique($linkarray)
;~  _ArrayDisplay($linkarray, "3rd")
    ;
    ; launch the link
    ;
    For $u = 1 To $linkarray[0]
        If $linkarray[$u] <> '' Then
            RecordTheLinks($linkarray[$u])
            GUICtrlSetData($lblStatusUpdate, $linkarray[$u])
        EndIf
    ;
    ;check if stop button pushed then exit loop
    ;
    If $stoplinkrun = False Then ExitLoop
    Next
    ;
    ;wait 1 minute and try again
    ;
;~  Sleep(60000)
    Sleep(1000)
EndFunc ;==>StartLinkProcess


Func RecordTheLinks($link2record)
    _FileWriteLog(@ScriptDir & "\linksfound.txt", $link2record)
;~  _IENavigate($oIE, $link2record)
EndFunc ;==>RecordTheLinks

I've taken some liberty to simplify your code without changing its behavior. The memory leakage is linked to heavy use of _IENavigate. Simply commenting out the navigation to the sites filtered slows down memory growth considerably. Also I suspect that only newly visited pages cause leakage. I don't know enough about _IE* to be sure but it might have something to do with the warning about Vista+ in the help file about _IENavigate. Maybe a cache thing not releasing correctly? Memory leak occurs as well on two XP SP3 x86 PCs where I tried it here. I used IE8, patches are current.

BTW, allow me a bit of comment on your code. You don't have to allocate 3 distinct arrays to do what you want. Even if you re-insert code you ommited for posting, the simplified version is sufficient. Also beware of Dim, use either Global or Local so that your intent is clearer. Beware of variables declared as Dim at the global level and then redeclared inside functions. You don't have to clear local variables (inside functions), they will be cleared at function exit. I corrected the second test for filtering and replaced two successive tests by a single regexp. _ArrayAdd & _ArrayDelete can become expensive and were useless in your case.

I hope some _IE guru will visit your topic soon and shed light on this.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Thanks for recode. Yes the _arraydelete was bogus I new that could be removed.

I'm so used to using Dim from my BASIC days :(

The only real code I omitted was the distinct web pages I'm looking for.

I might try to launch the IE in a separate exe maybe this will show the end-user the webpage then quit.

Thanks again for all your help. I'll continue the battle.

Share this post


Link to post
Share on other sites

Don't take my remarks too badly, I intended nothing weird.

Since the _IE gurus are busy elsewhere, you could devote some time trying to temporarily leave out IE embedding in your GUI and use a classical _IEAttach. It would allow you to force deletion of the _IEobject (assign 0 to it, as you know) at every run, while deletion is postponed until closing the embedding GUI in the embedded approach.

It's possible that this could avoid the leak completely. With computers and women alike, less pretty often means more reliable. Go figure!


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

No offense was taken at all. That's how we learn! You have been very helpful.

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

I was able to reproduce this on Win7 Ultimate.

I'm now running a test using

$oIE.navigate($url)
_IELoadWait($oIE,0,5000)

Instead of _IENavigate to see if the problem remains.

Edit:

It seems to have some positive effect, but I think it's mostly because of the timeout in _IELoadWait(). After running for a while, my memory usage stayed at about 40MB. It didn't seem to matter much if I used $oIE.navigate($url) or _IENavigate($oIE,$Url,0), combined with _IELoadWait($oIE,0,5000).

I did notice some of the pages had video streams. I could imagine those filling the memory pretty fast if you let them.

Edited by Tvern

Share this post


Link to post
Share on other sites

I used yahoo as an example. You could change it to any page. The internal web site I'm checking has over 1000 pages and they don't contain video mainly text and pictures.

Share this post


Link to post
Share on other sites

Anyway, the browser should release loaded items, video or whatever.

But here's the Cavalry coming ;-)


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Assuming you run it outside the GUI and it doesn't happen, it could be an issue with the GUI object container in AutoI. Suggest you investigate an alternate method by searching for IECreate2 in the forum to attach a full browser instance instead of using the GUI container.

Dale


Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

Share this post


Link to post
Share on other sites

I have fixed the issue for now. The memory leaks don't happen as long as I close all IE windows when the program starts (The program kills them for me). Then it launches the IE from a different exe this way it has it's own process and does the IECreate and IEQuit just fine on it's own. The end user will just have to click a button to launch the next window to check the link.

THanks for all the help!

Share this post


Link to post
Share on other sites

Your finding may help others. Nice to know you're on the right track now.

Cheers.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0