1 post in this topic
[Solved] Memory Leak
I'm basically processing text files where I'm reading them from a disk file into an input array, going through the input array and skipping what I don't want and writing it out to an output array and then writing the output array to a disk file. I encountered an "Error allocating memory" message. I have stripped down the program to illustrate how the memory usage grows in a looping routine (__ReadFiles). I have tried a _ReduceMemory routine (from https://www.autoitscript.com/forum/topic/134831-is-there-a-way-to-free-up-memory-being-used/ ) and it helps somewhat, however the peak working set continues to grow regardless. I believe I'm releasing resources (resetting arrays to 0) and am trying to figure out if the leak is from FileFindNextFile or something else I'm doing. To illustrate the program creates a subdirectory with 50000 simple text files and then runs showing the growing memory usage, Ctrl+q will pause program. Rerunning the program uses the already created files so it does not have to be re-created. Any help appreciated.
; use this to debug in console window <--- LOOK #AutoIt3Wrapper_run_debug_mode=Y ;#AutoIt3Wrapper_Run_Debug=off ;#AutoIt3Wrapper_Run_Debug=on #include <Array.au3> #include <File.au3> #include <MsgBoxConstants.au3> #include <GUIConstantsEx.au3> #include <EditConstants.au3> HotKeySet ( "^q", "EndProgram" ) ;v2e - added HotKeySet ( "^q", "EndProgram" ); CTRL+q so can pause/exit script Global $hEdit ;for scrolling text box Global $textarray ;1 element $textarray = 0 ;initial count Global $newtextarray ;1 element $newtextarray = 0 ;initial count Global $dir Global $k ;loop count Global $aMemory ;v1g for use in __ShowMemoryUsage("Location: In FileFindNextFile While 1 loop.") Global $debug = 0 ;0=off, 1=on , 2=deeper, - use debug > 1 to see arrays in Funcs $version = "v1g" ;v1g - we have an Autoit "Error allocating memory" somewhere - printing out WorkingSetSize and PeakWorkingSetSize from ProcessGetStats to try and find it __scrolling_text_box_init("--- Read files ---", 1050, 450) ;__scrolling_text_box_init($title, $width, $height) ;set up box - to display text use: __scrolltext($text) WinMove("--- Read files ---", "", 2000, 300) ;move so can see it - on single screen use 10, 30 on dual screen use 2000, 300 __scrolltext("Starting: " & @CRLF) ;illustrate error using current script directory by creating unique directotry and create file in it and put some text into it ----------- $dir = @ScriptDir ;for iterative testing see if test directory already exists - i.e. only need to build this once $dir = @ScriptDir & "\testdir_xcrgua" ;highly unlikely areadly exists unless this program was already run $str = "" ;null it For $j = 1 to 100 ;put 100 lines in each file $str = $str & $j & @CRLF Next If DirGetSize($dir) = -1 Then DirCreate($dir) ;create directory ;then create and fill it For $i = 1 to 50000 ;brute force (and I know it's slow, however straightforward and only 1 time FileWrite($dir & "\" & $i & ".txt", $str) If $i/1000 = Int($i/1000) Then __scrolltext("Writing test file " & $i & "/50000" & @CRLF) Next EndIf MsgBox(262144, "DEBUG", $dir & " <-- should have 50000 files each with 100 numbered lines 1- 100.") MsgBox(262144, "DEBUG", "Paused.") __scrolltext("Reading files in dir = " & $dir & @CRLF) ;__ShowMemoryUsage("Location: Before file count.") ;v1g $fc = __GetFileCount($dir) ;returns file count in $dir ;__ShowMemoryUsage("Location: After file count.") ;v1g __scrolltext("Number of files in dir = " & $fc & @CRLF) MsgBox(262144, "DEBUG", "Paused.") ;in this we get memory allocation erroroccurs between 25K and 50K on my machine __ReadFiles($dir) MsgBox(262144, "DEBUG", "Paused out of loop.") Exit ;----------------------------------- Functions ----------------------------------- Func __ShowMemoryUsage($title) $aMemory = ProcessGetStats() ; Retrieve memory details about the current process. ; If $aMemory is an array then display the following details about the process. If IsArray($aMemory) Then __scrolltext($title & " " & "WorkingSetSize: " & $aMemory & " PeakWorkingSetSize: " & $aMemory & @CRLF) Else __scrolltext("An error occurred in ProcessGetStats() in Func __ShowMemoryUsage($title).") EndIf EndFunc ;__ShowMemoryUsage($title) Func _ReduceMemory($i_PID = -1) ;from https://www.autoitscript.com/forum/topic/134831-is-there-a-way-to-free-up-memory-being-used/ If $i_PID <> -1 Then Local $ai_Handle = DllCall("kernel32.dll", 'int', 'OpenProcess', 'int', 0x1F0FFF, 'int', False, 'int', $i_PID) Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', $ai_Handle) DllCall('kernel32.dll', 'int', 'CloseHandle', 'int', $ai_Handle) Else Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', -1) EndIf Return $ai_Return EndFunc ;==>_ReduceMemory Func __GetFileCount($dir) ;returns number of files in $dir - brute force but only about 50K files/directory ;MsgBox(0, "DEBUG", "$dir = '" & $dir & "'") #AutoIt3Wrapper_Run_Debug=off $fcount = 0 ;init count $search = FileFindFirstFile($dir & "\*.*") If $search = -1 Then ;no files found Return($fcount) ;MsgBox(0, "Error", "No files/directories matched the search pattern") ;Exit EndIf ;okay check number of files While 1 $file = FileFindNextFile($search) If @error Then ExitLoop ;exit loop when there are no more files Else $fcount = $fcount + 1 EndIf WEnd FileClose($search) ;be polite Return($fcount) #AutoIt3Wrapper_Run_Debug=on EndFunc Func __ReadFiles($dir) ;just reading to show memory alloc error Local $x Local $skipline #AutoIt3Wrapper_Run_Debug=off $tfc = __GetFileCount($dir) ;get total file count so we can show progress #AutoIt3Wrapper_Run_Debug=on $cfc = 0 ;init currect file count $search = FileFindFirstFile($dir & "\*.*") ;get search handle If $search = -1 Then ;no files found __scrolltext("Working on file: " & $cfc & "/" & $tfc & @CRLF) Return ;MsgBox(0, "Error", "No files/directories matched the search pattern") ;Exit EndIf ;okay go through the files one by one While 1 ;_ReduceMemory() ;this makes little difference - surprising $file = FileFindNextFile($search) If @error = 1 Then ExitLoop ;exit loop when there are no more to do If @extended = 1 Then $adir = 1 ;we have a directory so note it Else $adir = 0 ;a file EndIf ;MsgBox(4096, "File:", $directory&"\"&$file) $cfc = $cfc + 1 __scrolltext("Working on file: " & $cfc & "/" & $tfc & " " & $file & @CRLF) ;show the filename __ShowMemoryUsage("Location: In FileFindNextFile While 1 loop.") ;v1g ;clear out arrays so we have no artifacts from procesing the prior loop ;from here it looks like resetting it is better https://www.autoitscript.com/forum/topic/110933-solved-how-to-delete-whole-array/ $testarray = 0 $newtextarray = 0 Dim $textarray ;1 element $textarray = 0 ;initial count Dim $newtextarray ;1 element $newtextarray = 0 ;initial count ;bring file into memory for faster processing If $adir = 0 Then ;process file $x = _FileReadToArray($dir & "\" & $file, $textarray) ;bring file into memory for faster processing  has $textarray count If $x <> 1 Then MsgBox($MB_TOPMOST + $MB_ICONERROR, "ERROR", "Unable to read file into array. Error = " & $x & " @error = " & @error & " @extended = " & @extended) Exit EndIf If $debug > 0 Then _ArrayDisplay($textarray, "INPUT $textarray") ReDim $newtextarray[$textarray + 1] ;create output array ($newtextarray) of the same size as input array ($textarray) and later ReDim the output array to the correct size $newtextarray = 0 ;set the count of actual entries as $newtextarray is empty but has same size as $textarray If $debug > 0 Then _ArrayDisplay($newtextarray, "Starting $newtextarray") ;now the approach is to write out only lines we want then to file For $i = 1 to $textarray ;process all elements (rows, lines) in input array ;If $debug > 0 Then __scrolltext("$textarray[" & $i & "] = '" & $textarray[$i] & "'" & @CRLF) ;show the line $skipline = 0 ;init value to not skip line ;----------- start stripping out stuff ----------- ;this rountine is not the cause of the memory leak so removed ;------------- If $skipline = 0 Then ;write it out $newtextarray = $newtextarray + 1 ;increase count and point to index where we're going to store it in $newtextarray $newtextarray[$newtextarray] = $textarray[$i] Else ;skip the line EndIf Next $fn = $dir & "\processed-1\" & StringLeft($file, StringLen($file) - 4) & "-1.txt" __scrolltext("Processed file written to: " &$fn & @CRLF) ;show the line ;RECALL written out like an array so 1st entry which corresponds to index  has count of rows = lines in the file Else ;its a directory - skip it __scrolltext("NOT processed directory (i.e. skipped) : " & $dir & "\" & $file & @CRLF) ;show the line EndIf WEnd EndFunc ;__CleanFiles($dir) Func __scrolling_text_box_init($title, $width, $height) ;from http://www.autoitscript.com/forum/topic/110948-add-text-to-edit-box-and-scroll-it-down/page__p__971158__hl___guictrledit_scroll__fromsearch__1#entry971158 $hGUI = GUICreate($title, $width, $height) $hEdit = GUICtrlCreateEdit("", 10, 10, $width-20, $height-20, BitOr($GUI_SS_DEFAULT_EDIT, $ES_READONLY)) ;$limit = 9223372036854775807 ;2^63 $limit = 1000000000 ;works $x = GUICtrlSetLimit($hEdit, $limit) ;~ If $x = 0 Then ;~ MsgBox(0, "ERROR", "Limit of " & $limit & " is too large.") ;~ Else ;~ MsgBox(0, "Okay", "Limit of " & $limit & " is okay.") ;~ EndIf ;$hButton = GUICtrlCreateButton("Add", 10, 250, 80, 30) GUISetState() EndFunc Func __scrolltext($text) #AutoIt3Wrapper_Run_Debug=off ;prefix everything with date and time e.g. 2014-02-01 HH:MM:SS $dt = @YEAR & "-" & @MON & "-" & @MDAY & " " & @HOUR & ":" & @MIN & ":" & @SEC & " " $text = $dt & $text GUICtrlSetData($hEdit, $text, 1) #AutoIt3Wrapper_Run_Debug=on EndFunc Func EndProgram() $x = MsgBox($MB_TOPMOST + $MB_YESNO + $MB_ICONWARNING, "Paused", "Do you want to continue?") If $x = $IDNO Then Exit ;exit program Else ;just a pause - so continue EndIf EndFunc
04 - Clean files - v1g - test for memory leak v1c.au3
Find and eliminate memory leaks
Hi. I'm a teacher and I do a lot of tutorials and other presentations on my computer. I've developed a tool using AutoIt and Adobe AIR to display all the shortcuts I use while I'm presenting. According to the forum rules this would mean that I've developed a keylogger, so I can't show any of the code, but I'm still hoping someone will help me solve an issue I'm having - a memory leak (or at least I think that's it).
I can see the application is taking up more and more memory, but it never goes super crazy. I think it was at 25 MB at one point and that was it. However I see that the longer the application is running less responsive it is. It doesn't capture all the events, or it simply lags.
I'm using AssocArrays and _MouseOnEvent UDFs, _WinAPI_SetTimer, _WinAPI_SetWindowsHookEx, _Singleton and TCP. I've done some research before posting this and I know there are some issues in special cases, but all solutions were "code specific". Since I can't post any of the code I couldn't respond in those threads. Other than that it really doesn't seem to be the problem with any of the UDFs, so my question is:
Is this a memory leak? If so how can I find it and remove it? What to do to avoid it in the future.
I understand that declaring variables over and over (something in the timer) may be the cause of this, so according to what I've read on the forum I've changed the variables to Global and moved them outside the functions. That way they are only declared once, and then only values are being reassigned. That unfortuantly didn't help. Is there anything else I could do or look for?
BTW - I've used Adobe AIR to create a nice UI. If someone want's to create something similar UEZ was kind enough to share his code of creating such GUI with nice antialiased labels.
Since Authenticity was awesome enough to post how to get the Alt-Tab-able Windows, I decided to wrap it into a simple UDF that can be used just as WinList. The potential use is great. One person has already used the code to create a replacement Alt-Tab interface.
#include "_WinGetAltTabWinList.au3" #include <Array.au3> Local $aWindows=_WinGetAltTabWinList("","",True) _ArrayDisplay($aWindows)
Fixed extended-style check to skip over certain windows that shouldn't be in the list, cleaned up and shortened code a bit. Download the ZIP at my Site
Ascend4nt's AutoIT Code License agreement:
While I provide this source code freely, if you do use the code in your projects, all I ask is that:
If you provide source, keep the header as I have put it, OR, if you expand it, then at least acknowledge me as the original author, and any other authors I credit If the program is released, acknowledge me in your credits (it doesn't have to state which functions came from me, though again if the source is provided - see #1) The source on it's own (as opposed to part of a project) can not be posted unless a link to the page(s) where the code were retrieved from is provided and a message stating that the latest updates will be available on the page(s) linked to. Pieces of the code can however be discussed on the threads where Ascend4nt has posted the code without worrying about further linking.