Recently Browsing 0 members
No registered users viewing this page.
I think I have memory leak issue...
Every time I use _IENavigate() my Memory Ram goes up and never down.. I could reach 1gb just using this function. Which is weird..
#include <IE.au3> #include <GUIConstantsEx.au3> #RequireAdmin Global $oIE = _IECreateEmbedded() Global $Form1 = GUICreate("Form1", 920, 466, 502, 342) GUICtrlCreateObj($oIE, 8, 72, 897, 370) ;IE OBJ Local $Button1 = GUICtrlCreateButton("FETCH", 808, 8, 91, 57) GUISetState(@SW_SHOW) While 1 Local $nMsg = GUIGetMsg() Switch $nMsg Case $GUI_EVENT_CLOSE Exit Case $Button1 Fetch() EndSwitch WEnd Func Fetch() For $i = 0 To 20 _IENavigate($oIE, 'https://www.google.com/search?source=hp&ei=hzprXM_zN-zs_QbExry4CQ&q=sad&btnK=Google+Search&oq=&gs_l=psy-ab.3..35i39j0i67l9.2993.3926..4083...1.0..0.137.749.0j6......0....1..gws-wiz.....6.eVCDaXHQXYA') Next EndFunc
How do I know there is a memory leak ?, is it "Private Bytes" or "Working Set" too ?
I have some clues but no definitive knowing. I have no experience looking at these.
Also, If I can use AutoIt to self-check within the code would be most helpful.
I'm basically processing text files where I'm reading them from a disk file into an input array, going through the input array and skipping what I don't want and writing it out to an output array and then writing the output array to a disk file. I encountered an "Error allocating memory" message. I have stripped down the program to illustrate how the memory usage grows in a looping routine (__ReadFiles). I have tried a _ReduceMemory routine (from https://www.autoitscript.com/forum/topic/134831-is-there-a-way-to-free-up-memory-being-used/ ) and it helps somewhat, however the peak working set continues to grow regardless. I believe I'm releasing resources (resetting arrays to 0) and am trying to figure out if the leak is from FileFindNextFile or something else I'm doing. To illustrate the program creates a subdirectory with 50000 simple text files and then runs showing the growing memory usage, Ctrl+q will pause program. Rerunning the program uses the already created files so it does not have to be re-created. Any help appreciated.
; use this to debug in console window <--- LOOK #AutoIt3Wrapper_run_debug_mode=Y ;#AutoIt3Wrapper_Run_Debug=off ;#AutoIt3Wrapper_Run_Debug=on #include <Array.au3> #include <File.au3> #include <MsgBoxConstants.au3> #include <GUIConstantsEx.au3> #include <EditConstants.au3> HotKeySet ( "^q", "EndProgram" ) ;v2e - added HotKeySet ( "^q", "EndProgram" ); CTRL+q so can pause/exit script Global $hEdit ;for scrolling text box Global $textarray ;1 element $textarray = 0 ;initial count Global $newtextarray ;1 element $newtextarray = 0 ;initial count Global $dir Global $k ;loop count Global $aMemory ;v1g for use in __ShowMemoryUsage("Location: In FileFindNextFile While 1 loop.") Global $debug = 0 ;0=off, 1=on , 2=deeper, - use debug > 1 to see arrays in Funcs $version = "v1g" ;v1g - we have an Autoit "Error allocating memory" somewhere - printing out WorkingSetSize and PeakWorkingSetSize from ProcessGetStats to try and find it __scrolling_text_box_init("--- Read files ---", 1050, 450) ;__scrolling_text_box_init($title, $width, $height) ;set up box - to display text use: __scrolltext($text) WinMove("--- Read files ---", "", 2000, 300) ;move so can see it - on single screen use 10, 30 on dual screen use 2000, 300 __scrolltext("Starting: " & @CRLF) ;illustrate error using current script directory by creating unique directotry and create file in it and put some text into it ----------- $dir = @ScriptDir ;for iterative testing see if test directory already exists - i.e. only need to build this once $dir = @ScriptDir & "\testdir_xcrgua" ;highly unlikely areadly exists unless this program was already run $str = "" ;null it For $j = 1 to 100 ;put 100 lines in each file $str = $str & $j & @CRLF Next If DirGetSize($dir) = -1 Then DirCreate($dir) ;create directory ;then create and fill it For $i = 1 to 50000 ;brute force (and I know it's slow, however straightforward and only 1 time FileWrite($dir & "\" & $i & ".txt", $str) If $i/1000 = Int($i/1000) Then __scrolltext("Writing test file " & $i & "/50000" & @CRLF) Next EndIf MsgBox(262144, "DEBUG", $dir & " <-- should have 50000 files each with 100 numbered lines 1- 100.") MsgBox(262144, "DEBUG", "Paused.") __scrolltext("Reading files in dir = " & $dir & @CRLF) ;__ShowMemoryUsage("Location: Before file count.") ;v1g $fc = __GetFileCount($dir) ;returns file count in $dir ;__ShowMemoryUsage("Location: After file count.") ;v1g __scrolltext("Number of files in dir = " & $fc & @CRLF) MsgBox(262144, "DEBUG", "Paused.") ;in this we get memory allocation erroroccurs between 25K and 50K on my machine __ReadFiles($dir) MsgBox(262144, "DEBUG", "Paused out of loop.") Exit ;----------------------------------- Functions ----------------------------------- Func __ShowMemoryUsage($title) $aMemory = ProcessGetStats() ; Retrieve memory details about the current process. ; If $aMemory is an array then display the following details about the process. If IsArray($aMemory) Then __scrolltext($title & " " & "WorkingSetSize: " & $aMemory & " PeakWorkingSetSize: " & $aMemory & @CRLF) Else __scrolltext("An error occurred in ProcessGetStats() in Func __ShowMemoryUsage($title).") EndIf EndFunc ;__ShowMemoryUsage($title) Func _ReduceMemory($i_PID = -1) ;from https://www.autoitscript.com/forum/topic/134831-is-there-a-way-to-free-up-memory-being-used/ If $i_PID <> -1 Then Local $ai_Handle = DllCall("kernel32.dll", 'int', 'OpenProcess', 'int', 0x1F0FFF, 'int', False, 'int', $i_PID) Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', $ai_Handle) DllCall('kernel32.dll', 'int', 'CloseHandle', 'int', $ai_Handle) Else Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', -1) EndIf Return $ai_Return EndFunc ;==>_ReduceMemory Func __GetFileCount($dir) ;returns number of files in $dir - brute force but only about 50K files/directory ;MsgBox(0, "DEBUG", "$dir = '" & $dir & "'") #AutoIt3Wrapper_Run_Debug=off $fcount = 0 ;init count $search = FileFindFirstFile($dir & "\*.*") If $search = -1 Then ;no files found Return($fcount) ;MsgBox(0, "Error", "No files/directories matched the search pattern") ;Exit EndIf ;okay check number of files While 1 $file = FileFindNextFile($search) If @error Then ExitLoop ;exit loop when there are no more files Else $fcount = $fcount + 1 EndIf WEnd FileClose($search) ;be polite Return($fcount) #AutoIt3Wrapper_Run_Debug=on EndFunc Func __ReadFiles($dir) ;just reading to show memory alloc error Local $x Local $skipline #AutoIt3Wrapper_Run_Debug=off $tfc = __GetFileCount($dir) ;get total file count so we can show progress #AutoIt3Wrapper_Run_Debug=on $cfc = 0 ;init currect file count $search = FileFindFirstFile($dir & "\*.*") ;get search handle If $search = -1 Then ;no files found __scrolltext("Working on file: " & $cfc & "/" & $tfc & @CRLF) Return ;MsgBox(0, "Error", "No files/directories matched the search pattern") ;Exit EndIf ;okay go through the files one by one While 1 ;_ReduceMemory() ;this makes little difference - surprising $file = FileFindNextFile($search) If @error = 1 Then ExitLoop ;exit loop when there are no more to do If @extended = 1 Then $adir = 1 ;we have a directory so note it Else $adir = 0 ;a file EndIf ;MsgBox(4096, "File:", $directory&"\"&$file) $cfc = $cfc + 1 __scrolltext("Working on file: " & $cfc & "/" & $tfc & " " & $file & @CRLF) ;show the filename __ShowMemoryUsage("Location: In FileFindNextFile While 1 loop.") ;v1g ;clear out arrays so we have no artifacts from procesing the prior loop ;from here it looks like resetting it is better https://www.autoitscript.com/forum/topic/110933-solved-how-to-delete-whole-array/ $testarray = 0 $newtextarray = 0 Dim $textarray ;1 element $textarray = 0 ;initial count Dim $newtextarray ;1 element $newtextarray = 0 ;initial count ;bring file into memory for faster processing If $adir = 0 Then ;process file $x = _FileReadToArray($dir & "\" & $file, $textarray) ;bring file into memory for faster processing  has $textarray count If $x <> 1 Then MsgBox($MB_TOPMOST + $MB_ICONERROR, "ERROR", "Unable to read file into array. Error = " & $x & " @error = " & @error & " @extended = " & @extended) Exit EndIf If $debug > 0 Then _ArrayDisplay($textarray, "INPUT $textarray") ReDim $newtextarray[$textarray + 1] ;create output array ($newtextarray) of the same size as input array ($textarray) and later ReDim the output array to the correct size $newtextarray = 0 ;set the count of actual entries as $newtextarray is empty but has same size as $textarray If $debug > 0 Then _ArrayDisplay($newtextarray, "Starting $newtextarray") ;now the approach is to write out only lines we want then to file For $i = 1 to $textarray ;process all elements (rows, lines) in input array ;If $debug > 0 Then __scrolltext("$textarray[" & $i & "] = '" & $textarray[$i] & "'" & @CRLF) ;show the line $skipline = 0 ;init value to not skip line ;----------- start stripping out stuff ----------- ;this rountine is not the cause of the memory leak so removed ;------------- If $skipline = 0 Then ;write it out $newtextarray = $newtextarray + 1 ;increase count and point to index where we're going to store it in $newtextarray $newtextarray[$newtextarray] = $textarray[$i] Else ;skip the line EndIf Next $fn = $dir & "\processed-1\" & StringLeft($file, StringLen($file) - 4) & "-1.txt" __scrolltext("Processed file written to: " &$fn & @CRLF) ;show the line ;RECALL written out like an array so 1st entry which corresponds to index  has count of rows = lines in the file Else ;its a directory - skip it __scrolltext("NOT processed directory (i.e. skipped) : " & $dir & "\" & $file & @CRLF) ;show the line EndIf WEnd EndFunc ;__CleanFiles($dir) Func __scrolling_text_box_init($title, $width, $height) ;from http://www.autoitscript.com/forum/topic/110948-add-text-to-edit-box-and-scroll-it-down/page__p__971158__hl___guictrledit_scroll__fromsearch__1#entry971158 $hGUI = GUICreate($title, $width, $height) $hEdit = GUICtrlCreateEdit("", 10, 10, $width-20, $height-20, BitOr($GUI_SS_DEFAULT_EDIT, $ES_READONLY)) ;$limit = 9223372036854775807 ;2^63 $limit = 1000000000 ;works $x = GUICtrlSetLimit($hEdit, $limit) ;~ If $x = 0 Then ;~ MsgBox(0, "ERROR", "Limit of " & $limit & " is too large.") ;~ Else ;~ MsgBox(0, "Okay", "Limit of " & $limit & " is okay.") ;~ EndIf ;$hButton = GUICtrlCreateButton("Add", 10, 250, 80, 30) GUISetState() EndFunc Func __scrolltext($text) #AutoIt3Wrapper_Run_Debug=off ;prefix everything with date and time e.g. 2014-02-01 HH:MM:SS $dt = @YEAR & "-" & @MON & "-" & @MDAY & " " & @HOUR & ":" & @MIN & ":" & @SEC & " " $text = $dt & $text GUICtrlSetData($hEdit, $text, 1) #AutoIt3Wrapper_Run_Debug=on EndFunc Func EndProgram() $x = MsgBox($MB_TOPMOST + $MB_YESNO + $MB_ICONWARNING, "Paused", "Do you want to continue?") If $x = $IDNO Then Exit ;exit program Else ;just a pause - so continue EndIf EndFunc
04 - Clean files - v1g - test for memory leak v1c.au3
Hi. I'm a teacher and I do a lot of tutorials and other presentations on my computer. I've developed a tool using AutoIt and Adobe AIR to display all the shortcuts I use while I'm presenting. According to the forum rules this would mean that I've developed a keylogger, so I can't show any of the code, but I'm still hoping someone will help me solve an issue I'm having - a memory leak (or at least I think that's it).
I can see the application is taking up more and more memory, but it never goes super crazy. I think it was at 25 MB at one point and that was it. However I see that the longer the application is running less responsive it is. It doesn't capture all the events, or it simply lags.
I'm using AssocArrays and _MouseOnEvent UDFs, _WinAPI_SetTimer, _WinAPI_SetWindowsHookEx, _Singleton and TCP. I've done some research before posting this and I know there are some issues in special cases, but all solutions were "code specific". Since I can't post any of the code I couldn't respond in those threads. Other than that it really doesn't seem to be the problem with any of the UDFs, so my question is:
Is this a memory leak? If so how can I find it and remove it? What to do to avoid it in the future.
I understand that declaring variables over and over (something in the timer) may be the cause of this, so according to what I've read on the forum I've changed the variables to Global and moved them outside the functions. That way they are only declared once, and then only values are being reassigned. That unfortuantly didn't help. Is there anything else I could do or look for?
BTW - I've used Adobe AIR to create a nice UI. If someone want's to create something similar UEZ was kind enough to share his code of creating such GUI with nice antialiased labels.
All my tests done on WinXP SP3, AutoItX v184.108.40.206
My AutoItX is not up-to-date, however the changelog does not record any change regarding this problem.
The issue is known :
* each call to o_AutoItX3Obj.WinList can consumme up to 4ko (in my tests),
which are not released even when exiting a sub or setting the o_AutoItX3Obj object to Nothing
-> this is a problem, for example, for ever-runnning scripts that do repetitive call to WinList
* a work-around : enumerate windows instead of using WinList
Option Explicit MAIN() Sub MAIN() Dim s_win_id, arr Dim o_AutoItX3Obj Set o_AutoItX3Obj = CreateObject("AutoItX3.Control") ' s_win_id : the advanced filter parameter that would have been used in the WinList function ' s_win_id : must not contain the INSTANCE parameter (it seems it just would be ignored) ' WinTitleMatchMode : the title match mode that would have been used with the WinList function ' WinSearchChildren : the child windows mode that would have been used with the WinList function ' *** EXAMPLE 1 *** ' s_win_id = "TITLE:a", with WinTitleMatchMode substring (=2), No WinSearchChildren (=0) ' there should be some of theses s_win_id = "TITLE:a" o_AutoItX3Obj.AutoItSetOption "WinTitleMatchMode", 2 o_AutoItX3Obj.AutoItSetOption "WinSearchChildren", 0 ' call the pseudo WinList function (ENUM_WIN) arr = ENUM_WIN(o_AutoItX3Obj, s_win_id) ' look for results SHOW_RESULTS o_AutoItX3Obj, s_win_id, arr ' *** EXAMPLE 2 *** ' s_win_id = "TITLE:a; REGEXPCLASS:a", with WinTitleMatchMode substring (=2), No WinSearchChildren (=0) ' there should be at least the 'Program Manager' [Class:Progman], which is the Windows desktop s_win_id = "TITLE:a; REGEXPCLASS:a" o_AutoItX3Obj.AutoItSetOption "WinTitleMatchMode", 2 o_AutoItX3Obj.AutoItSetOption "WinSearchChildren", 0 ' call the pseudo WinList function (ENUM_WIN) arr = ENUM_WIN(o_AutoItX3Obj, s_win_id) ' look for results SHOW_RESULTS o_AutoItX3Obj, s_win_id, arr End Sub Sub SHOW_RESULTS(ByVal o_AutoItX3Obj, ByVal s_win_id, ByVal arr) Dim s_res, i s_res = "Number of windows matching [" & s_win_id & "] : " & arr(0, 0) & vbCr s_res = s_res & "WinTitleMatchMode : " & o_AutoItX3Obj.AutoItSetOption("WinTitleMatchMode", 0) & ", " s_res = s_res & "WinSearchChildren : " & o_AutoItX3Obj.AutoItSetOption("WinSearchChildren", 0) & vbCr s_res = s_res & "(message boxes can cut long text)" & vbCr & vbCr s_res = s_res & "Num Handle" & vbTab & "Title" & vbCr & vbCr For i = 1 To UBound(arr, 2) s_res = s_res & i & ". " & arr(1, i) & vbTab & "[" & arr(0, i) & "]" & vbCr Next MsgBox s_res, vbInformation, WScript.ScriptName End Sub ' Re-usable code begins here (could be optimized if hundreds of matching windows are expected) ' ENUM_WIN returns an 2-Dim array, same formatting as by WinList function Function ENUM_WIN(ByVal o_AutoItX3Obj, ByVal s_win_id) Dim s_win_handle, s_win_title, arr, i ReDim arr(1, 0) arr(0, 0) = 0 ' will hold the number of found windows, like the WinList function arr(1, 0) = 0 ' will not be used i = 1 ' because Instances are 1-based Do s_win_handle = o_AutoItX3Obj.WinGetHandle("[" & s_win_id & "; INSTANCE:" & i & "]") If s_win_handle = "" Then Exit Do ' enumeration finished s_win_title = o_AutoItX3Obj.WinGetTitle("[HANDLE:" & s_win_handle & "]") ReDim Preserve arr(1, i) arr(0, i) = s_win_title ' title can be an empty string arr(1, i) = s_win_handle ' handle should always be a valid hex string number (32/64 bits ?) i = i + 1 Loop arr(0, 0) = i - 1 ' record the number of found windows ENUM_WIN = arr End Function
* note 1 :
enumerating windows could have timing issues if some windows matching the criteria
are appearing/disappearing while doing the enumeration... (I didn't test that)
* note 2 :
if it impossible to avoid memory leak in COM components returning arrays,
then WinList could return a simple string with titles/handles separated by Chr(0),
(assuming there are no Chr(0) inside window titles ?)
then the user would use the Split function to get an 1-Dim array : [title1, handle1, title2, handle2, title3, handle3] etc...
This string-type return could be an optional parameter given to the WinList function :
arr = o_AutoItX3Obj.Winlist("title" [, "text" [, return type]])