Jump to content

Recommended Posts

ahha

I'm basically processing text files where I'm reading them from a disk file into an input array, going through the input array and skipping what I don't want and writing it out to an output array and then writing the output array to a disk file.  I encountered an "Error allocating memory" message.  I have stripped down the program to illustrate how the memory usage grows in a looping routine (__ReadFiles).  I have tried a _ReduceMemory routine (from https://www.autoitscript.com/forum/topic/134831-is-there-a-way-to-free-up-memory-being-used/ ) and it helps somewhat, however the peak working set continues to grow regardless.  I believe I'm releasing resources (resetting arrays to 0) and am trying to figure out if the leak is from FileFindNextFile or something else I'm doing.   To illustrate the program creates a subdirectory with 50000 simple text files and then runs showing the growing memory usage, Ctrl+q will pause program.  Rerunning the program uses the already created files so it does not have to be re-created.  Any help appreciated.

; use this to debug in console window <--- LOOK
#AutoIt3Wrapper_run_debug_mode=Y
;#AutoIt3Wrapper_Run_Debug=off
;#AutoIt3Wrapper_Run_Debug=on


#include <Array.au3>
#include <File.au3>
#include <MsgBoxConstants.au3>
#include <GUIConstantsEx.au3>
#include <EditConstants.au3>

HotKeySet ( "^q", "EndProgram" )    ;v2e - added HotKeySet ( "^q", "EndProgram" ); CTRL+q so can pause/exit script

Global $hEdit   ;for scrolling text box

Global $textarray[1]    ;1 element
$textarray[0] = 0       ;initial count
Global $newtextarray[1] ;1 element
$newtextarray[0] = 0    ;initial count

Global $dir
Global $k       ;loop count

Global $aMemory ;v1g    for use in __ShowMemoryUsage("Location: In FileFindNextFile While 1 loop.")

Global $debug = 0       ;0=off, 1=on , 2=deeper,  - use debug > 1 to see arrays in Funcs


$version = "v1g"
;v1g - we have an Autoit "Error allocating memory" somewhere - printing out WorkingSetSize and  PeakWorkingSetSize from ProcessGetStats to try and find it


__scrolling_text_box_init("--- Read files ---", 1050, 450)  ;__scrolling_text_box_init($title, $width, $height)     ;set up box - to display text use: __scrolltext($text)
WinMove("--- Read files ---", "", 2000, 300)    ;move so can see it - on single screen use  10, 30 on dual screen use 2000, 300
__scrolltext("Starting:  " & @CRLF)


;illustrate error using current script directory by creating unique directotry and create file in it and put some text into it -----------
$dir = @ScriptDir

;for iterative testing see if test directory already exists - i.e. only need to build this once
$dir = @ScriptDir & "\testdir_xcrgua"   ;highly unlikely areadly exists unless this program was already run
$str = ""   ;null it
For $j = 1 to 100   ;put 100 lines in each file
    $str = $str & $j & @CRLF
Next

If DirGetSize($dir) = -1 Then
    DirCreate($dir) ;create directory   ;then create and fill it
    For $i = 1 to 50000     ;brute force (and I know it's slow, however straightforward and only 1 time
        FileWrite($dir & "\" & $i & ".txt", $str)
        If $i/1000 = Int($i/1000) Then __scrolltext("Writing test file " & $i & "/50000" & @CRLF)
    Next
EndIf
MsgBox(262144, "DEBUG", $dir & "  <-- should have 50000 files each with 100 numbered lines 1- 100.")

MsgBox(262144, "DEBUG", "Paused.")


__scrolltext("Reading files in dir = " & $dir & @CRLF)
;__ShowMemoryUsage("Location: Before file count.")  ;v1g
$fc = __GetFileCount($dir)  ;returns file count in $dir
;__ShowMemoryUsage("Location: After file count.")   ;v1g
__scrolltext("Number of files in dir = " & $fc & @CRLF)
MsgBox(262144, "DEBUG", "Paused.")

;in this we get memory allocation erroroccurs between 25K and 50K on my machine
__ReadFiles($dir)


MsgBox(262144, "DEBUG", "Paused out of loop.")

Exit


;----------------------------------- Functions -----------------------------------

Func    __ShowMemoryUsage($title)

        $aMemory = ProcessGetStats()    ; Retrieve memory details about the current process.

        ; If $aMemory is an array then display the following details about the process.
        If IsArray($aMemory) Then
            __scrolltext($title & "   " & "WorkingSetSize: " & $aMemory[0] & "      PeakWorkingSetSize: " & $aMemory[1] & @CRLF)
        Else
            __scrolltext("An error occurred in ProcessGetStats() in Func __ShowMemoryUsage($title).")
        EndIf

EndFunc     ;__ShowMemoryUsage($title)


Func _ReduceMemory($i_PID = -1) ;from https://www.autoitscript.com/forum/topic/134831-is-there-a-way-to-free-up-memory-being-used/
    If $i_PID <> -1 Then
        Local $ai_Handle = DllCall("kernel32.dll", 'int', 'OpenProcess', 'int', 0x1F0FFF, 'int', False, 'int', $i_PID)
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', $ai_Handle[0])
        DllCall('kernel32.dll', 'int', 'CloseHandle', 'int', $ai_Handle[0])
    Else
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', -1)
    EndIf
    Return $ai_Return[0]
EndFunc   ;==>_ReduceMemory



Func    __GetFileCount($dir)    ;returns number of files in $dir - brute force but only about 50K files/directory

    ;MsgBox(0, "DEBUG", "$dir = '" & $dir & "'")
#AutoIt3Wrapper_Run_Debug=off

    $fcount = 0     ;init count
    $search = FileFindFirstFile($dir & "\*.*")

    If $search = -1 Then    ;no files found
        Return($fcount)
        ;MsgBox(0, "Error", "No files/directories matched the search pattern")
        ;Exit
    EndIf

    ;okay check number of files
    While 1
        $file = FileFindNextFile($search)
        If @error Then
            ExitLoop        ;exit loop when there are no more files
        Else
            $fcount = $fcount + 1
        EndIf
    WEnd

    FileClose($search)  ;be polite

    Return($fcount)

#AutoIt3Wrapper_Run_Debug=on

EndFunc


Func    __ReadFiles($dir)   ;just reading to show memory alloc error

    Local $x
    Local $skipline

#AutoIt3Wrapper_Run_Debug=off
    $tfc = __GetFileCount($dir) ;get total file count so we can show progress
#AutoIt3Wrapper_Run_Debug=on

    $cfc = 0    ;init currect file count
    $search = FileFindFirstFile($dir & "\*.*")  ;get search handle

    If $search = -1 Then    ;no files found
        __scrolltext("Working on file: " & $cfc & "/" & $tfc & @CRLF)
        Return
        ;MsgBox(0, "Error", "No files/directories matched the search pattern")
        ;Exit
    EndIf

    ;okay go through the files one by one
    While 1

        ;_ReduceMemory()    ;this makes little difference - surprising

        $file = FileFindNextFile($search)
        If @error = 1 Then ExitLoop     ;exit loop when there are no more to do
        If @extended = 1 Then
            $adir = 1   ;we have a directory so note it
        Else
            $adir = 0   ;a file
        EndIf

        ;MsgBox(4096, "File:", $directory&"\"&$file)
        $cfc = $cfc + 1
        __scrolltext("Working on file: " & $cfc & "/" & $tfc & "     " & $file & @CRLF) ;show the filename

        __ShowMemoryUsage("Location: In FileFindNextFile While 1 loop.")    ;v1g

        ;clear out arrays so we have no artifacts from procesing the prior loop
        ;from here it looks like resetting it is better https://www.autoitscript.com/forum/topic/110933-solved-how-to-delete-whole-array/
        $testarray = 0
        $newtextarray = 0

        Dim $textarray[1]       ;1 element
        $textarray[0] = 0       ;initial count
        Dim $newtextarray[1]    ;1 element
        $newtextarray[0] = 0    ;initial count

        ;bring file into memory for faster processing
        If $adir = 0 Then   ;process file
            $x = _FileReadToArray($dir & "\" & $file, $textarray)       ;bring file into memory for faster processing [0] has $textarray count
            If $x <> 1 Then
                MsgBox($MB_TOPMOST + $MB_ICONERROR, "ERROR", "Unable to read file into array.  Error = " & $x & "    @error = " & @error & "    @extended = " & @extended)
                Exit
            EndIf
            If $debug > 0 Then _ArrayDisplay($textarray, "INPUT $textarray")

            ReDim $newtextarray[$textarray[0] + 1]  ;create output array ($newtextarray) of the same size as input array ($textarray) and later ReDim the output array to the correct size
            $newtextarray[0] = 0        ;set the count of actual entries as $newtextarray is empty but has same size as $textarray
            If $debug > 0 Then _ArrayDisplay($newtextarray, "Starting $newtextarray")

            ;now the approach is to write out only lines we want then to file
            For $i = 1 to $textarray[0] ;process all elements (rows, lines) in input array
                ;If $debug > 0 Then __scrolltext("$textarray[" & $i & "] = '" & $textarray[$i] & "'" & @CRLF)   ;show the line
                $skipline = 0   ;init value to not skip line
                ;----------- start stripping out stuff -----------
                ;this rountine is not the cause of the memory leak so removed
                ;-------------
                If $skipline = 0 Then   ;write it out
                    $newtextarray[0] = $newtextarray[0] + 1     ;increase count and point to index where we're going to store it in $newtextarray
                    $newtextarray[$newtextarray[0]] = $textarray[$i]
                Else
                    ;skip the line
                EndIf
            Next

            $fn = $dir & "\processed-1\" & StringLeft($file, StringLen($file) - 4) & "-1.txt"

            __scrolltext("Processed file written to: " &$fn & @CRLF)    ;show the line
            ;RECALL written out like an array so 1st entry which corresponds to index [0] has count of rows = lines in the file
        Else    ;its a directory - skip it
            __scrolltext("NOT processed directory (i.e. skipped) : " & $dir & "\" & $file & @CRLF)  ;show the line
        EndIf

    WEnd

EndFunc ;__CleanFiles($dir)



Func __scrolling_text_box_init($title, $width, $height)
    ;from http://www.autoitscript.com/forum/topic/110948-add-text-to-edit-box-and-scroll-it-down/page__p__971158__hl___guictrledit_scroll__fromsearch__1#entry971158

    $hGUI = GUICreate($title, $width, $height)
    $hEdit = GUICtrlCreateEdit("", 10, 10, $width-20, $height-20, BitOr($GUI_SS_DEFAULT_EDIT, $ES_READONLY))
    ;$limit = 9223372036854775807   ;2^63
    $limit = 1000000000 ;works
    $x = GUICtrlSetLimit($hEdit, $limit)
    ;~ If $x = 0 Then
    ;~  MsgBox(0, "ERROR", "Limit of " &  $limit & " is too large.")
    ;~ Else
    ;~  MsgBox(0, "Okay", "Limit of " &  $limit & " is okay.")
    ;~ EndIf

    ;$hButton = GUICtrlCreateButton("Add", 10, 250, 80, 30)

    GUISetState()

EndFunc


Func __scrolltext($text)
#AutoIt3Wrapper_Run_Debug=off
    ;prefix everything with date and time e.g. 2014-02-01 HH:MM:SS
    $dt = @YEAR & "-" & @MON & "-" & @MDAY & " " & @HOUR & ":" & @MIN & ":" & @SEC & "  "
    $text = $dt & $text
    GUICtrlSetData($hEdit, $text, 1)
#AutoIt3Wrapper_Run_Debug=on
EndFunc


Func EndProgram()
    $x = MsgBox($MB_TOPMOST + $MB_YESNO + $MB_ICONWARNING, "Paused", "Do you want to continue?")
    If $x = $IDNO Then
        Exit    ;exit program
    Else
        ;just a pause - so continue
    EndIf
EndFunc

 

04 - Clean files - v1g - test for memory leak v1c.au3

Edited by ahha
Added as a file.

Share this post


Link to post
Share on other sites
Bowmore

Hi ahha

This is the line in your __scrolltext() function that is using all your memory.

GUICtrlSetData($hEdit, $text,1)

It's appending to the existing contents of the control each time it is executed. If you change it to this your memory problems will disappear.

GUICtrlSetData($hEdit, $text)

 

  • Like 1

"Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to build bigger and better idiots. So far, the universe is winning."- Rick Cook

Share this post


Link to post
Share on other sites
ahha

Bowmore,

Thanks - I totally missed that - duh - I feel stupid.:'(  I do have a question however.  Why when using scrolling text does the PeakWorkingSetSize increase but not the WorkingSetSize.  I would have thought the WorkingSetSize would increase if the text is being stored/displayed. I've forgotten how to indicate solved at the topic top.

Edited by ahha

Share this post


Link to post
Share on other sites
BetaLeaf
On 2/1/2016 at 7:44 PM, ahha said:

Bowmore,

Thanks - I totally missed that - duh - I feel stupid.:'(  I do have a question however.  Why when using scrolling text does the PeakWorkingSetSize increase but not the WorkingSetSize.  I would have thought the WorkingSetSize would increase if the text is being stored/displayed. I've forgotten how to indicate solved at the topic top.

Edit your original post and change the title to [Solved] Memory Leak


False Positive Reporter - Mass email all anti virus vendors with an attachment of your program for fast and easy whitelisting.

PortableApps.com App Creation Wizard  - A simple GUI-based Wizard for creating PortableApps.

SoundBoard - Play any song or sound you want at the press of a hotkey.

My GitHub Page: https://github.com/BetaLeaf

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • TrashBoat
      By TrashBoat
      So Im trying to make a simple 2d game and make some sort of collision detection so why not to make a 2 dimensional array but i have no clue how  to write it in multiple lines
      Global $map[5,5] = [0,0,0,0,0 _ [0,0,0,0,0 _ [0,0,0,0,0 _ [0,0,0,0,0 _ [0,0,0,0,0] something like this but it doesn't work
    • Zein
      By Zein
      #include "..\Include\Array.au3" #include "..\Include\File.au3" #include "..\Include\AutoItConstants.au3" Local $aRetArray Local $sFilePath = "n.csv" _FileReadToArray($sFilePath, $aRetArray, ",") ; _FileReadToArray($sFilePath, $aRetArray, $FRTA_COUNT, ",") _ArrayDisplay($aRetArray, "Original", Default, 8) The above code shows two versions of _FileReadToArray and both don't work as expected.
      The first one doesn't use the comma as a delimiter. (so I get a single column array)  I tried adding "Default" between $aRetArray and "," then it told me it had an incorrect number of parameters. 

      I looked again at the documentation:
       
      #include <File.au3> _FileReadToArray ( $sFilePath, ByRef $vReturn [, $iFlags = $FRTA_COUNT [, $sDelimiter = ""]] )
      And I with or without the flags params I should be getting a 2D array due to my file being a csv. 
      I then tried a regular flag, $FRTA_COUNT, and it tells me that I'm using a variable $FRTA_COUNT while it's not declared. Tried putting in 1 instead and it told me again, incorrect number of params. 

       
    • ternal
      By ternal
      Hi,
      Recently I have had the need to do a sort and then do a second sort while the item of the first sort stays the same ( double sorting , first on column x then while column x is the same sort column y).
      I did not put much efffort into error checking but so far I did not need it.
      For my applications so far it works perfectly however if someone is willing I want to test this extensivly.
      If anyone has big lists of random stuff to sort could you try this out please?
      #include <Array.au3> ; #FUNCTION# ==================================================================================================================== ; Name ..........: _ArraySort_Double ; Description ...: ; Syntax ........: _ArraySort_Double (Byref $array[, $first_index = Default[, $second_index = Default[, $ascending = Default]]]) ; Parameters ....: $array - 2d array to sort. ; $first_index - [optional] first column to sort. Default is 0. ; $second_index - [optional] second column to sort. Default is 1. ; $ascending - [optional] ascending/descending. Default is 1. ; Return values .: 1 if no errors occured , -1 if errors occured ; Author ........: Ternal ; Remarks .......: Needs excessive testing. ; Related .......: _arraysort() ; =============================================================================================================================== Func _ArraySort_Double (byref $array, $first_index = Default, $second_index = Default, $ascending = Default) Local $temp_value Local $counter = 1 If UBound($array, $UBOUND_DIMENSIONS) <> 2 Then MsgBox(0, "error", "error") return -1 EndIf If $first_index = Default Then $first_index = 0 If $second_index = Default Then $second_index = 1 If $ascending = Default Then $ascending = 1 _ArraySort($array, $ascending, 0, 0, $first_index); you can alter settings of primary sort here If @error Then MsgBox(0, "error", @error) return -1 EndIf $temp_value = $array[0][$first_index] For $x = 1 to UBound($array, 1) - 1 If Mod( $x, 10000) = 0 Then ConsoleWrite("at " & $x & " of a total : " & UBound($array, 1) & @CRLF) If $array[$x][$first_index] = $temp_value Then $counter+= 1 If $x = UBound($array, 1) - 1 Then; do last line here(if last line is not a new item) _ArraySort($array, $ascending, $x - $counter, $x, $second_index);you can alter settings of secondary sort here(don't forget to place line 34 the exact same) If @error Then MsgBox(0, "error", @error) return -1 EndIf EndIf Else If $counter > 0 Then ;at least 2 of the same _ArraySort($array, $ascending, $x - $counter, $x - 1, $second_index);you can alter settings of secondary sort here(don't forget to place line 29 the exact same) If @error Then MsgBox(0, "error", @error) return -1 EndIf $counter = 1 EndIf EndIf $temp_value = $array[$x][$first_index] Next Return 1 EndFunc Kind regards, Ternal
    • TrashBoat
      By TrashBoat
      So I've made this script that detects how long i have held down my left mouse button for and stores the information in an array and then sorts its using _ArraySort but the output is half sorted half broken.
      Here's my script:
      HotKeySet("{F1}","_exit") #include <Misc.au3> #include <Timers.au3> #include <Array.au3> Local $dll = DllOpen("user32.dll") $on = False Global $array[0] While(1) If _IsPressed(01,$dll) Then $timer = _Timer_Init() While _IsPressed(01,$dll) Sleep(1) WEnd $time = _Timer_Diff($timer) _ArrayAdd($array,"Time: " & Floor($time) & " ms") ;~ ConsoleWrite("Time: " & Floor($time) & " ms" & @CRLF) EndIf Sleep(50) WEnd Func _exit() _ArraySort($array) _ArrayDisplay($array) Exit EndFunc And the output:

      See how its not sorted?  What is the problem here?
    • MrCheese
      By MrCheese
      Hi guys,
      See attached for an array example.
      to simplify what i want to achieve,  I want to split this array into 9 different csv files.
      the first file would contain the list of "key" and the corresponding "ID1", the second would have "key" and the "ID2", the third would have "key" and "ID3"
      However, I want to remove all the rows that don't have an ID recorded in the respective ID2, ID3 4...5...6 etc, so the file only contains row items with a key and the ID.
      Would be the best way to loop through the rows and delete the row if the array field is blank - would I then need to repeat that row ID to check that the row that its replaced is also empty (ie the one after the one I just deleted)? I see this getting messy.
      or _arraySort, and delete everything below the last filled row? <-- this might be best?
      Or should I use the excel UDF, apply a filter (not selecting the blanks), then create/export to the array->csv?
       
      Super keen to hear your thoughts.
      thanks!
       
       
       
       
      IDArray.csv
×