ahha

[Solved] Memory Leak

4 posts in this topic

#1 ·  Posted (edited)

I'm basically processing text files where I'm reading them from a disk file into an input array, going through the input array and skipping what I don't want and writing it out to an output array and then writing the output array to a disk file.  I encountered an "Error allocating memory" message.  I have stripped down the program to illustrate how the memory usage grows in a looping routine (__ReadFiles).  I have tried a _ReduceMemory routine (from https://www.autoitscript.com/forum/topic/134831-is-there-a-way-to-free-up-memory-being-used/ ) and it helps somewhat, however the peak working set continues to grow regardless.  I believe I'm releasing resources (resetting arrays to 0) and am trying to figure out if the leak is from FileFindNextFile or something else I'm doing.   To illustrate the program creates a subdirectory with 50000 simple text files and then runs showing the growing memory usage, Ctrl+q will pause program.  Rerunning the program uses the already created files so it does not have to be re-created.  Any help appreciated.

; use this to debug in console window <--- LOOK
#AutoIt3Wrapper_run_debug_mode=Y
;#AutoIt3Wrapper_Run_Debug=off
;#AutoIt3Wrapper_Run_Debug=on


#include <Array.au3>
#include <File.au3>
#include <MsgBoxConstants.au3>
#include <GUIConstantsEx.au3>
#include <EditConstants.au3>

HotKeySet ( "^q", "EndProgram" )    ;v2e - added HotKeySet ( "^q", "EndProgram" ); CTRL+q so can pause/exit script

Global $hEdit   ;for scrolling text box

Global $textarray[1]    ;1 element
$textarray[0] = 0       ;initial count
Global $newtextarray[1] ;1 element
$newtextarray[0] = 0    ;initial count

Global $dir
Global $k       ;loop count

Global $aMemory ;v1g    for use in __ShowMemoryUsage("Location: In FileFindNextFile While 1 loop.")

Global $debug = 0       ;0=off, 1=on , 2=deeper,  - use debug > 1 to see arrays in Funcs


$version = "v1g"
;v1g - we have an Autoit "Error allocating memory" somewhere - printing out WorkingSetSize and  PeakWorkingSetSize from ProcessGetStats to try and find it


__scrolling_text_box_init("--- Read files ---", 1050, 450)  ;__scrolling_text_box_init($title, $width, $height)     ;set up box - to display text use: __scrolltext($text)
WinMove("--- Read files ---", "", 2000, 300)    ;move so can see it - on single screen use  10, 30 on dual screen use 2000, 300
__scrolltext("Starting:  " & @CRLF)


;illustrate error using current script directory by creating unique directotry and create file in it and put some text into it -----------
$dir = @ScriptDir

;for iterative testing see if test directory already exists - i.e. only need to build this once
$dir = @ScriptDir & "\testdir_xcrgua"   ;highly unlikely areadly exists unless this program was already run
$str = ""   ;null it
For $j = 1 to 100   ;put 100 lines in each file
    $str = $str & $j & @CRLF
Next

If DirGetSize($dir) = -1 Then
    DirCreate($dir) ;create directory   ;then create and fill it
    For $i = 1 to 50000     ;brute force (and I know it's slow, however straightforward and only 1 time
        FileWrite($dir & "\" & $i & ".txt", $str)
        If $i/1000 = Int($i/1000) Then __scrolltext("Writing test file " & $i & "/50000" & @CRLF)
    Next
EndIf
MsgBox(262144, "DEBUG", $dir & "  <-- should have 50000 files each with 100 numbered lines 1- 100.")

MsgBox(262144, "DEBUG", "Paused.")


__scrolltext("Reading files in dir = " & $dir & @CRLF)
;__ShowMemoryUsage("Location: Before file count.")  ;v1g
$fc = __GetFileCount($dir)  ;returns file count in $dir
;__ShowMemoryUsage("Location: After file count.")   ;v1g
__scrolltext("Number of files in dir = " & $fc & @CRLF)
MsgBox(262144, "DEBUG", "Paused.")

;in this we get memory allocation erroroccurs between 25K and 50K on my machine
__ReadFiles($dir)


MsgBox(262144, "DEBUG", "Paused out of loop.")

Exit


;----------------------------------- Functions -----------------------------------

Func    __ShowMemoryUsage($title)

        $aMemory = ProcessGetStats()    ; Retrieve memory details about the current process.

        ; If $aMemory is an array then display the following details about the process.
        If IsArray($aMemory) Then
            __scrolltext($title & "   " & "WorkingSetSize: " & $aMemory[0] & "      PeakWorkingSetSize: " & $aMemory[1] & @CRLF)
        Else
            __scrolltext("An error occurred in ProcessGetStats() in Func __ShowMemoryUsage($title).")
        EndIf

EndFunc     ;__ShowMemoryUsage($title)


Func _ReduceMemory($i_PID = -1) ;from https://www.autoitscript.com/forum/topic/134831-is-there-a-way-to-free-up-memory-being-used/
    If $i_PID <> -1 Then
        Local $ai_Handle = DllCall("kernel32.dll", 'int', 'OpenProcess', 'int', 0x1F0FFF, 'int', False, 'int', $i_PID)
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', $ai_Handle[0])
        DllCall('kernel32.dll', 'int', 'CloseHandle', 'int', $ai_Handle[0])
    Else
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', -1)
    EndIf
    Return $ai_Return[0]
EndFunc   ;==>_ReduceMemory



Func    __GetFileCount($dir)    ;returns number of files in $dir - brute force but only about 50K files/directory

    ;MsgBox(0, "DEBUG", "$dir = '" & $dir & "'")
#AutoIt3Wrapper_Run_Debug=off

    $fcount = 0     ;init count
    $search = FileFindFirstFile($dir & "\*.*")

    If $search = -1 Then    ;no files found
        Return($fcount)
        ;MsgBox(0, "Error", "No files/directories matched the search pattern")
        ;Exit
    EndIf

    ;okay check number of files
    While 1
        $file = FileFindNextFile($search)
        If @error Then
            ExitLoop        ;exit loop when there are no more files
        Else
            $fcount = $fcount + 1
        EndIf
    WEnd

    FileClose($search)  ;be polite

    Return($fcount)

#AutoIt3Wrapper_Run_Debug=on

EndFunc


Func    __ReadFiles($dir)   ;just reading to show memory alloc error

    Local $x
    Local $skipline

#AutoIt3Wrapper_Run_Debug=off
    $tfc = __GetFileCount($dir) ;get total file count so we can show progress
#AutoIt3Wrapper_Run_Debug=on

    $cfc = 0    ;init currect file count
    $search = FileFindFirstFile($dir & "\*.*")  ;get search handle

    If $search = -1 Then    ;no files found
        __scrolltext("Working on file: " & $cfc & "/" & $tfc & @CRLF)
        Return
        ;MsgBox(0, "Error", "No files/directories matched the search pattern")
        ;Exit
    EndIf

    ;okay go through the files one by one
    While 1

        ;_ReduceMemory()    ;this makes little difference - surprising

        $file = FileFindNextFile($search)
        If @error = 1 Then ExitLoop     ;exit loop when there are no more to do
        If @extended = 1 Then
            $adir = 1   ;we have a directory so note it
        Else
            $adir = 0   ;a file
        EndIf

        ;MsgBox(4096, "File:", $directory&"\"&$file)
        $cfc = $cfc + 1
        __scrolltext("Working on file: " & $cfc & "/" & $tfc & "     " & $file & @CRLF) ;show the filename

        __ShowMemoryUsage("Location: In FileFindNextFile While 1 loop.")    ;v1g

        ;clear out arrays so we have no artifacts from procesing the prior loop
        ;from here it looks like resetting it is better https://www.autoitscript.com/forum/topic/110933-solved-how-to-delete-whole-array/
        $testarray = 0
        $newtextarray = 0

        Dim $textarray[1]       ;1 element
        $textarray[0] = 0       ;initial count
        Dim $newtextarray[1]    ;1 element
        $newtextarray[0] = 0    ;initial count

        ;bring file into memory for faster processing
        If $adir = 0 Then   ;process file
            $x = _FileReadToArray($dir & "\" & $file, $textarray)       ;bring file into memory for faster processing [0] has $textarray count
            If $x <> 1 Then
                MsgBox($MB_TOPMOST + $MB_ICONERROR, "ERROR", "Unable to read file into array.  Error = " & $x & "    @error = " & @error & "    @extended = " & @extended)
                Exit
            EndIf
            If $debug > 0 Then _ArrayDisplay($textarray, "INPUT $textarray")

            ReDim $newtextarray[$textarray[0] + 1]  ;create output array ($newtextarray) of the same size as input array ($textarray) and later ReDim the output array to the correct size
            $newtextarray[0] = 0        ;set the count of actual entries as $newtextarray is empty but has same size as $textarray
            If $debug > 0 Then _ArrayDisplay($newtextarray, "Starting $newtextarray")

            ;now the approach is to write out only lines we want then to file
            For $i = 1 to $textarray[0] ;process all elements (rows, lines) in input array
                ;If $debug > 0 Then __scrolltext("$textarray[" & $i & "] = '" & $textarray[$i] & "'" & @CRLF)   ;show the line
                $skipline = 0   ;init value to not skip line
                ;----------- start stripping out stuff -----------
                ;this rountine is not the cause of the memory leak so removed
                ;-------------
                If $skipline = 0 Then   ;write it out
                    $newtextarray[0] = $newtextarray[0] + 1     ;increase count and point to index where we're going to store it in $newtextarray
                    $newtextarray[$newtextarray[0]] = $textarray[$i]
                Else
                    ;skip the line
                EndIf
            Next

            $fn = $dir & "\processed-1\" & StringLeft($file, StringLen($file) - 4) & "-1.txt"

            __scrolltext("Processed file written to: " &$fn & @CRLF)    ;show the line
            ;RECALL written out like an array so 1st entry which corresponds to index [0] has count of rows = lines in the file
        Else    ;its a directory - skip it
            __scrolltext("NOT processed directory (i.e. skipped) : " & $dir & "\" & $file & @CRLF)  ;show the line
        EndIf

    WEnd

EndFunc ;__CleanFiles($dir)



Func __scrolling_text_box_init($title, $width, $height)
    ;from http://www.autoitscript.com/forum/topic/110948-add-text-to-edit-box-and-scroll-it-down/page__p__971158__hl___guictrledit_scroll__fromsearch__1#entry971158

    $hGUI = GUICreate($title, $width, $height)
    $hEdit = GUICtrlCreateEdit("", 10, 10, $width-20, $height-20, BitOr($GUI_SS_DEFAULT_EDIT, $ES_READONLY))
    ;$limit = 9223372036854775807   ;2^63
    $limit = 1000000000 ;works
    $x = GUICtrlSetLimit($hEdit, $limit)
    ;~ If $x = 0 Then
    ;~  MsgBox(0, "ERROR", "Limit of " &  $limit & " is too large.")
    ;~ Else
    ;~  MsgBox(0, "Okay", "Limit of " &  $limit & " is okay.")
    ;~ EndIf

    ;$hButton = GUICtrlCreateButton("Add", 10, 250, 80, 30)

    GUISetState()

EndFunc


Func __scrolltext($text)
#AutoIt3Wrapper_Run_Debug=off
    ;prefix everything with date and time e.g. 2014-02-01 HH:MM:SS
    $dt = @YEAR & "-" & @MON & "-" & @MDAY & " " & @HOUR & ":" & @MIN & ":" & @SEC & "  "
    $text = $dt & $text
    GUICtrlSetData($hEdit, $text, 1)
#AutoIt3Wrapper_Run_Debug=on
EndFunc


Func EndProgram()
    $x = MsgBox($MB_TOPMOST + $MB_YESNO + $MB_ICONWARNING, "Paused", "Do you want to continue?")
    If $x = $IDNO Then
        Exit    ;exit program
    Else
        ;just a pause - so continue
    EndIf
EndFunc

 

04 - Clean files - v1g - test for memory leak v1c.au3

Edited by ahha
Added as a file.

Share this post


Link to post
Share on other sites

Hi ahha

This is the line in your __scrolltext() function that is using all your memory.

GUICtrlSetData($hEdit, $text,1)

It's appending to the existing contents of the control each time it is executed. If you change it to this your memory problems will disappear.

GUICtrlSetData($hEdit, $text)

 

1 person likes this

"Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to build bigger and better idiots. So far, the universe is winning."- Rick Cook

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

Bowmore,

Thanks - I totally missed that - duh - I feel stupid.:'(  I do have a question however.  Why when using scrolling text does the PeakWorkingSetSize increase but not the WorkingSetSize.  I would have thought the WorkingSetSize would increase if the text is being stored/displayed. I've forgotten how to indicate solved at the topic top.

Edited by ahha

Share this post


Link to post
Share on other sites
On 2/1/2016 at 7:44 PM, ahha said:

Bowmore,

Thanks - I totally missed that - duh - I feel stupid.:'(  I do have a question however.  Why when using scrolling text does the PeakWorkingSetSize increase but not the WorkingSetSize.  I would have thought the WorkingSetSize would increase if the text is being stored/displayed. I've forgotten how to indicate solved at the topic top.

Edit your original post and change the title to [Solved] Memory Leak


False Positive Reporter - Mass email all anti virus vendors with an attachment of your program for fast and easy whitelisting.

PortableApps.com App Creation Wizard  - A simple GUI-based Wizard for creating PortableApps.

SoundBoard - Play any song or sound you want at the press of a hotkey.

My GitHub Page: https://github.com/BetaLeaf

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now

  • Similar Content

    • Ascer
      By Ascer
      1. Description.
      Udf working with MSDN System.Collections.ArrayList. Allow you to make fast operations on huge arrays, speed is even x10 better than basic _ArrayAdd.  Not prefered for small arrays < 600 items. 2. Requirements
      .NET Framework 1.1 - 4.5 (on this version Microsoft destroy old rules) System Windows 3. Possibilities.
      ;=============================================================================================================== ; UDF Name: List.au3 ; ; Date: 2018-02-17, 10:52 ; Description: Simple udf to create System Collections as ArrayList and make multiple actions on them. ; ; Function(s): _ListCreate -> Creates a new list ; _ListCapacity -> Gets a list size in bytes ; _ListCount -> Gets items count in list ; _ListIsFixedSize -> Get bool if list if fixed size ; _ListIsReadOnly -> Get bool if list is read only ; _ListIsSynchronized -> Get bool if list is synchronized ; _ListGetItem -> Get item on index ; _ListSetItem -> Set item on index ; ; _ListAdd -> Add item at end of list ; _ListClear -> Remove all list items ; _ListClone -> Duplicate list in new var ; _ListContains -> Get bool if item is in list ; _ListGetHashCode -> Get hash code for list ; _ListGetRange -> Get list with items between indexs ; _ListIndexOf -> Get index of item ; _ListInsert -> Insert a new item on index ; _ListInsertRange -> Insert list into list on index ; _ListLastIndexOf -> Get index last of item ; _ListRemove -> Remove first found item ; _ListRemoveAt -> Remove item in index ; _ListRemoveRange -> Remove items between indexs ; _ListReverse -> Reverse all items in list ; _ListSetRange -> Set new value for items in range ; _ListSort -> Sort items in list (speed of reading) ; _ListToString -> Get list object name ; _ListTrimToSize -> Remove unused space in list ; ; Author(s): Ascer ;=============================================================================================================== 4. Downloads
      List.au3 5. Examples
      SpeedTest _ArrayAdd vs ListAdd SpeedTest ArraySearch vs ListIndexOf Basic usage - crating guild with members  
    • ronmage
      By ronmage
      So I have a loop that keeps reading data from an array and searching it for the same value. If the value is no there it does work then adds the value to the array to prevent it from doing the same work.
      If _ArraySearch($ID,$filearray[$i]) = -1 Then Work.... _ArrayAdd($ID,$filearray[$i]) EndIf This is in a for loop hence $i
      So what is happening is the code works great for several hours. After a period of time _ArraySearch($ID,$filearray[$i]) will result in -1 even if $ID = $filearray. So it ready as if there is no data in the array. Anyone have this problem? 
       
      Also I am just running in using F5 not compiling it and running it if that makes a difference.
       
    • FMS
      By FMS
      Hello,
      I'm trying to read a binary file to an array but couln't get it to work.
      Also I coul not find any help in the forum around this subject whish was helpfull.
      Is there any way it could be done?
      I tried a lot of ways but maybe somebody know's the right way?
      #AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 -w 7 #include <File.au3> #include <Array.au3> #include <AutoItConstants.au3> Local $in=FileOpen("TEST_labels.idx1-ubyte",16) ; 16+0=Read binary Local $data = FileRead($in) Local $FileArray = BinaryToString($data,4) ;~ $FileArray = StringSplit($BinarydData, @CRLF, 1+2) ;~ Local $FileArray = StringRegExp($BinarydData, "[^\r\n]+", 3) FileClose($in) _ArrayDisplay($FileArray,"$FileArray","",32) MsgBox($MB_SYSTEMMODAL, "", "$FileArray = " & $FileArray )  
      TEST_labels.idx1-ubyte
    • ur
      By ur
      I am reading a CSV file which is tab seperated as below.
      Local $aArray = FileReadToArray($file) And now, I am splitting this main array record wise so that Array contains internally another arrow to represent each row.
      For $i = 0 to (UBound($aArray) - 1) ;MsgBox(0,"",$aArray[$i]) $aArray[$i] = StringSplit(StringStripCR($aArray[$i]), Chr(9),2);Chr(9) for tab ;_ArrayDisplay($aArray[$i]) Next Afther that, _ArrayDIsplay is able to see the individual internal arrays.
      _ArrayDisplay($aArray[1]) But If I try to access the individual element of it as below.It is not showing any result.
      MsgBox(0,"",$aArray[1][1]) Any suggestion, below is the sample csv file.
      New Text Document.csv
    • Jibsbrown
      By Jibsbrown
      Need some help understanding why the ConsoleWrite works inside 2nd For loop but not out side. Between Audit Wiki, Help file , Forum searching (lots of code reading), and YouTube ( shout out to TutsTeach), I have not been able to find the reason why. 
      $sIniPath = "installLog.ini" ; - Get section name $iniSctionNames = IniReadSectionNames($sIniPath) ; - Get Keys and Vaules For $a = 1 to UBound($iniSctionNames) - 1 $keys = IniReadSection($sIniPath , $iniSctionNames[$a]) For $b = 1 to UBound($keys) - 1 $oldSysInfo = IniRead($sIniPath , $iniSctionNames[1], $keys[$b][0], "") $PntIPInfo = IniRead($sIniPath , $iniSctionNames[2], $keys[$b][0], "") $NewPCInfor = IniRead($sIniPath , $iniSctionNames[3], $keys[$b][0], "") ;ConsoleWrite($oldSysInfo & @LF) Next ;ConsoleWrite($oldSysInfo & @LF) Next ConsoleWrite($oldSysInfo) My intention is to use the variables later for Listboxes. Any explanation, forum post links or whatever would help. Sorry also very very new to Autoit.
      Also here's the ini file.
      [OldSysInfo] 4=192.168.0.4|DESKTOP-RDIU2SN|R90M05Q8 5=192.168.0.5|SD0123456789101|R9WGP9P 6=192.168.0.6|SD0123456789102|R9WGP9PT 3=192.168.0.3|DESKTOP-3RS4LKL|R9WGP9P 23=192.168.0.23|SD0123456789102|MXL1234P5I [PrinterIp] 50=192.168.0.50 48=192.168.0.48 47=192.168.0.47 [NewSysInfo] newPC = SD0123456789adfs|192.168.0.185|2UA1234FTR Thank you for your time.