Jump to content

Recommended Posts

ahha

I'm basically processing text files where I'm reading them from a disk file into an input array, going through the input array and skipping what I don't want and writing it out to an output array and then writing the output array to a disk file.  I encountered an "Error allocating memory" message.  I have stripped down the program to illustrate how the memory usage grows in a looping routine (__ReadFiles).  I have tried a _ReduceMemory routine (from https://www.autoitscript.com/forum/topic/134831-is-there-a-way-to-free-up-memory-being-used/ ) and it helps somewhat, however the peak working set continues to grow regardless.  I believe I'm releasing resources (resetting arrays to 0) and am trying to figure out if the leak is from FileFindNextFile or something else I'm doing.   To illustrate the program creates a subdirectory with 50000 simple text files and then runs showing the growing memory usage, Ctrl+q will pause program.  Rerunning the program uses the already created files so it does not have to be re-created.  Any help appreciated.

; use this to debug in console window <--- LOOK
#AutoIt3Wrapper_run_debug_mode=Y
;#AutoIt3Wrapper_Run_Debug=off
;#AutoIt3Wrapper_Run_Debug=on


#include <Array.au3>
#include <File.au3>
#include <MsgBoxConstants.au3>
#include <GUIConstantsEx.au3>
#include <EditConstants.au3>

HotKeySet ( "^q", "EndProgram" )    ;v2e - added HotKeySet ( "^q", "EndProgram" ); CTRL+q so can pause/exit script

Global $hEdit   ;for scrolling text box

Global $textarray[1]    ;1 element
$textarray[0] = 0       ;initial count
Global $newtextarray[1] ;1 element
$newtextarray[0] = 0    ;initial count

Global $dir
Global $k       ;loop count

Global $aMemory ;v1g    for use in __ShowMemoryUsage("Location: In FileFindNextFile While 1 loop.")

Global $debug = 0       ;0=off, 1=on , 2=deeper,  - use debug > 1 to see arrays in Funcs


$version = "v1g"
;v1g - we have an Autoit "Error allocating memory" somewhere - printing out WorkingSetSize and  PeakWorkingSetSize from ProcessGetStats to try and find it


__scrolling_text_box_init("--- Read files ---", 1050, 450)  ;__scrolling_text_box_init($title, $width, $height)     ;set up box - to display text use: __scrolltext($text)
WinMove("--- Read files ---", "", 2000, 300)    ;move so can see it - on single screen use  10, 30 on dual screen use 2000, 300
__scrolltext("Starting:  " & @CRLF)


;illustrate error using current script directory by creating unique directotry and create file in it and put some text into it -----------
$dir = @ScriptDir

;for iterative testing see if test directory already exists - i.e. only need to build this once
$dir = @ScriptDir & "\testdir_xcrgua"   ;highly unlikely areadly exists unless this program was already run
$str = ""   ;null it
For $j = 1 to 100   ;put 100 lines in each file
    $str = $str & $j & @CRLF
Next

If DirGetSize($dir) = -1 Then
    DirCreate($dir) ;create directory   ;then create and fill it
    For $i = 1 to 50000     ;brute force (and I know it's slow, however straightforward and only 1 time
        FileWrite($dir & "\" & $i & ".txt", $str)
        If $i/1000 = Int($i/1000) Then __scrolltext("Writing test file " & $i & "/50000" & @CRLF)
    Next
EndIf
MsgBox(262144, "DEBUG", $dir & "  <-- should have 50000 files each with 100 numbered lines 1- 100.")

MsgBox(262144, "DEBUG", "Paused.")


__scrolltext("Reading files in dir = " & $dir & @CRLF)
;__ShowMemoryUsage("Location: Before file count.")  ;v1g
$fc = __GetFileCount($dir)  ;returns file count in $dir
;__ShowMemoryUsage("Location: After file count.")   ;v1g
__scrolltext("Number of files in dir = " & $fc & @CRLF)
MsgBox(262144, "DEBUG", "Paused.")

;in this we get memory allocation erroroccurs between 25K and 50K on my machine
__ReadFiles($dir)


MsgBox(262144, "DEBUG", "Paused out of loop.")

Exit


;----------------------------------- Functions -----------------------------------

Func    __ShowMemoryUsage($title)

        $aMemory = ProcessGetStats()    ; Retrieve memory details about the current process.

        ; If $aMemory is an array then display the following details about the process.
        If IsArray($aMemory) Then
            __scrolltext($title & "   " & "WorkingSetSize: " & $aMemory[0] & "      PeakWorkingSetSize: " & $aMemory[1] & @CRLF)
        Else
            __scrolltext("An error occurred in ProcessGetStats() in Func __ShowMemoryUsage($title).")
        EndIf

EndFunc     ;__ShowMemoryUsage($title)


Func _ReduceMemory($i_PID = -1) ;from https://www.autoitscript.com/forum/topic/134831-is-there-a-way-to-free-up-memory-being-used/
    If $i_PID <> -1 Then
        Local $ai_Handle = DllCall("kernel32.dll", 'int', 'OpenProcess', 'int', 0x1F0FFF, 'int', False, 'int', $i_PID)
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', $ai_Handle[0])
        DllCall('kernel32.dll', 'int', 'CloseHandle', 'int', $ai_Handle[0])
    Else
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', -1)
    EndIf
    Return $ai_Return[0]
EndFunc   ;==>_ReduceMemory



Func    __GetFileCount($dir)    ;returns number of files in $dir - brute force but only about 50K files/directory

    ;MsgBox(0, "DEBUG", "$dir = '" & $dir & "'")
#AutoIt3Wrapper_Run_Debug=off

    $fcount = 0     ;init count
    $search = FileFindFirstFile($dir & "\*.*")

    If $search = -1 Then    ;no files found
        Return($fcount)
        ;MsgBox(0, "Error", "No files/directories matched the search pattern")
        ;Exit
    EndIf

    ;okay check number of files
    While 1
        $file = FileFindNextFile($search)
        If @error Then
            ExitLoop        ;exit loop when there are no more files
        Else
            $fcount = $fcount + 1
        EndIf
    WEnd

    FileClose($search)  ;be polite

    Return($fcount)

#AutoIt3Wrapper_Run_Debug=on

EndFunc


Func    __ReadFiles($dir)   ;just reading to show memory alloc error

    Local $x
    Local $skipline

#AutoIt3Wrapper_Run_Debug=off
    $tfc = __GetFileCount($dir) ;get total file count so we can show progress
#AutoIt3Wrapper_Run_Debug=on

    $cfc = 0    ;init currect file count
    $search = FileFindFirstFile($dir & "\*.*")  ;get search handle

    If $search = -1 Then    ;no files found
        __scrolltext("Working on file: " & $cfc & "/" & $tfc & @CRLF)
        Return
        ;MsgBox(0, "Error", "No files/directories matched the search pattern")
        ;Exit
    EndIf

    ;okay go through the files one by one
    While 1

        ;_ReduceMemory()    ;this makes little difference - surprising

        $file = FileFindNextFile($search)
        If @error = 1 Then ExitLoop     ;exit loop when there are no more to do
        If @extended = 1 Then
            $adir = 1   ;we have a directory so note it
        Else
            $adir = 0   ;a file
        EndIf

        ;MsgBox(4096, "File:", $directory&"\"&$file)
        $cfc = $cfc + 1
        __scrolltext("Working on file: " & $cfc & "/" & $tfc & "     " & $file & @CRLF) ;show the filename

        __ShowMemoryUsage("Location: In FileFindNextFile While 1 loop.")    ;v1g

        ;clear out arrays so we have no artifacts from procesing the prior loop
        ;from here it looks like resetting it is better https://www.autoitscript.com/forum/topic/110933-solved-how-to-delete-whole-array/
        $testarray = 0
        $newtextarray = 0

        Dim $textarray[1]       ;1 element
        $textarray[0] = 0       ;initial count
        Dim $newtextarray[1]    ;1 element
        $newtextarray[0] = 0    ;initial count

        ;bring file into memory for faster processing
        If $adir = 0 Then   ;process file
            $x = _FileReadToArray($dir & "\" & $file, $textarray)       ;bring file into memory for faster processing [0] has $textarray count
            If $x <> 1 Then
                MsgBox($MB_TOPMOST + $MB_ICONERROR, "ERROR", "Unable to read file into array.  Error = " & $x & "    @error = " & @error & "    @extended = " & @extended)
                Exit
            EndIf
            If $debug > 0 Then _ArrayDisplay($textarray, "INPUT $textarray")

            ReDim $newtextarray[$textarray[0] + 1]  ;create output array ($newtextarray) of the same size as input array ($textarray) and later ReDim the output array to the correct size
            $newtextarray[0] = 0        ;set the count of actual entries as $newtextarray is empty but has same size as $textarray
            If $debug > 0 Then _ArrayDisplay($newtextarray, "Starting $newtextarray")

            ;now the approach is to write out only lines we want then to file
            For $i = 1 to $textarray[0] ;process all elements (rows, lines) in input array
                ;If $debug > 0 Then __scrolltext("$textarray[" & $i & "] = '" & $textarray[$i] & "'" & @CRLF)   ;show the line
                $skipline = 0   ;init value to not skip line
                ;----------- start stripping out stuff -----------
                ;this rountine is not the cause of the memory leak so removed
                ;-------------
                If $skipline = 0 Then   ;write it out
                    $newtextarray[0] = $newtextarray[0] + 1     ;increase count and point to index where we're going to store it in $newtextarray
                    $newtextarray[$newtextarray[0]] = $textarray[$i]
                Else
                    ;skip the line
                EndIf
            Next

            $fn = $dir & "\processed-1\" & StringLeft($file, StringLen($file) - 4) & "-1.txt"

            __scrolltext("Processed file written to: " &$fn & @CRLF)    ;show the line
            ;RECALL written out like an array so 1st entry which corresponds to index [0] has count of rows = lines in the file
        Else    ;its a directory - skip it
            __scrolltext("NOT processed directory (i.e. skipped) : " & $dir & "\" & $file & @CRLF)  ;show the line
        EndIf

    WEnd

EndFunc ;__CleanFiles($dir)



Func __scrolling_text_box_init($title, $width, $height)
    ;from http://www.autoitscript.com/forum/topic/110948-add-text-to-edit-box-and-scroll-it-down/page__p__971158__hl___guictrledit_scroll__fromsearch__1#entry971158

    $hGUI = GUICreate($title, $width, $height)
    $hEdit = GUICtrlCreateEdit("", 10, 10, $width-20, $height-20, BitOr($GUI_SS_DEFAULT_EDIT, $ES_READONLY))
    ;$limit = 9223372036854775807   ;2^63
    $limit = 1000000000 ;works
    $x = GUICtrlSetLimit($hEdit, $limit)
    ;~ If $x = 0 Then
    ;~  MsgBox(0, "ERROR", "Limit of " &  $limit & " is too large.")
    ;~ Else
    ;~  MsgBox(0, "Okay", "Limit of " &  $limit & " is okay.")
    ;~ EndIf

    ;$hButton = GUICtrlCreateButton("Add", 10, 250, 80, 30)

    GUISetState()

EndFunc


Func __scrolltext($text)
#AutoIt3Wrapper_Run_Debug=off
    ;prefix everything with date and time e.g. 2014-02-01 HH:MM:SS
    $dt = @YEAR & "-" & @MON & "-" & @MDAY & " " & @HOUR & ":" & @MIN & ":" & @SEC & "  "
    $text = $dt & $text
    GUICtrlSetData($hEdit, $text, 1)
#AutoIt3Wrapper_Run_Debug=on
EndFunc


Func EndProgram()
    $x = MsgBox($MB_TOPMOST + $MB_YESNO + $MB_ICONWARNING, "Paused", "Do you want to continue?")
    If $x = $IDNO Then
        Exit    ;exit program
    Else
        ;just a pause - so continue
    EndIf
EndFunc

 

04 - Clean files - v1g - test for memory leak v1c.au3

Edited by ahha
Added as a file.

Share this post


Link to post
Share on other sites
Bowmore

Hi ahha

This is the line in your __scrolltext() function that is using all your memory.

GUICtrlSetData($hEdit, $text,1)

It's appending to the existing contents of the control each time it is executed. If you change it to this your memory problems will disappear.

GUICtrlSetData($hEdit, $text)

 

  • Like 1

"Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to build bigger and better idiots. So far, the universe is winning."- Rick Cook

Share this post


Link to post
Share on other sites
ahha

Bowmore,

Thanks - I totally missed that - duh - I feel stupid.:'(  I do have a question however.  Why when using scrolling text does the PeakWorkingSetSize increase but not the WorkingSetSize.  I would have thought the WorkingSetSize would increase if the text is being stored/displayed. I've forgotten how to indicate solved at the topic top.

Edited by ahha

Share this post


Link to post
Share on other sites
BetaLeaf
On 2/1/2016 at 7:44 PM, ahha said:

Bowmore,

Thanks - I totally missed that - duh - I feel stupid.:'(  I do have a question however.  Why when using scrolling text does the PeakWorkingSetSize increase but not the WorkingSetSize.  I would have thought the WorkingSetSize would increase if the text is being stored/displayed. I've forgotten how to indicate solved at the topic top.

Edit your original post and change the title to [Solved] Memory Leak


False Positive Reporter - Mass email all anti virus vendors with an attachment of your program for fast and easy whitelisting.

My GitHub Page: https://github.com/BetaLeaf

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • Skeletor
      By Skeletor
      Hi Virtual People,
      My array works perfectly fine. However, what is the best practice if the line in the array doesn't have the correct amount of columns and if I can add a placeholder?

       
      For $count = 1 To _FileCountLines($FileRead1) Step 1 $string = FileReadLine($FileRead1, $count) $input = StringSplit($string, ",", 1) $value1 = $input[1] $value2 = $input[2] $value3 = $input[3] _Excel_RangeWrite($oWorkbook, $oWorkbook.Activesheet, $value2, "A1") _Excel_RangeWrite($oWorkbook, $oWorkbook.Activesheet, $value1, "B1") _Excel_RangeWrite($oWorkbook, $oWorkbook.Activesheet, $value3, "C1") Next  
    • MrCheese
      By MrCheese
      hi all,
      reviewing the forum, this thread is applicable: 
       
       
      I wanted to know if there is now a better way to do this?
      In essence, I load a tab delimited txt file into an array (works well). I used tab, as some fields in the original csv contains commas.
      However, I needed autoit to manipulate this array, and output it as a csv.
      IF my array contains items with a comma, without double quotes around the field, then how best do I get a csv out of this?
      My current workaround is to filewritefromarray tab delimited, then open it in excel and save as a csv. I will need to check this to see how the address fields behave that contain a comma.
       
      Any thoughts would be appreciated.
       
    • Skeletor
      By Skeletor
      Hi All,

      I would like to know how you would take a FileLineRead and insert it into an array which then inserts it into Excel?
      One thing to know is the files content is broken up, so I only use half of the content within $FileRead1.
      So its imperative that the $value1, $value2, etc variables be used. 
      Code below:
      $FileRead1 = FileReadLine("C:\temp\sample.txt",1) For $count = 1 To _FileCountLines($FileRead1) Step 1 $string = FileReadLine($FileRead1, $count) $input = StringSplit($string, ",", 1) $value1 = $input[1] $value2 = $input[2] $value3 = $input[3] $value4 = $input[4] _Excel_RangeWrite($oWorkbook, $oWorkbook.Activesheet, $value1, "A1") _Excel_RangeWrite($oWorkbook, $oWorkbook.Activesheet, $value2, "B1") _Excel_RangeWrite($oWorkbook, $oWorkbook.Activesheet, $value3, "C1") _Excel_RangeWrite($oWorkbook, $oWorkbook.Activesheet, $value4, "D1") Next  
    • AnonymousX
      By AnonymousX
      Hello,
      I'm trying to write a script that moves copies excel cells into an array. I'll than manipulate the values and send array into another program. 
      I don't want range to be specific to a workbook, or sheet, or set of cells.
      I want user to be able to highlight desired cells and to copy either normally ("Ctrl+C") or by a hotkey ("Alt+C"). 
      Could someone help me with this?
      Thank you,
      I've tried to write the framework: (edited)
      #include <MsgBoxConstants.au3> #include <Array.au3> #include <Excel.au3> HotKeySet("!v", "Pastedata") While True Sleep(1000) WEnd func Makearray() local $bArray ;User has cells already copied ;Convert clipboard into an array ;I don;t know how excel stores data to clipboard so don;t know how to bring it into array _Arraydisplay($bArray) MsgBox(0,0,$bArray) return $bArray endfunc func Pastedata() Local $aArray MsgBox(0,0,"wait",1) ;make array based on assumption user has already copied a range to clipboard $aArray = Makearray() ;paste code ;don;t worry about this I got the rest endfunc  
    • Dzenan03
      By Dzenan03
      I want to make a while loop, that creates variables based on a array. For thist I created the array $iDsO with the number and the name of folders in an other folder. Every folder has a different name an I want to create variables(arrays) for each folder that show me all the files in that folder. For example: I have the Folder \Folder1. In it there are the Folders \1, \2, \3. In 1, 2 and 3 there are some files(.png). The array for Folder1 is $iDsO and now I want to crate the arrays $iDsO1, $iDsO2 and $iDsO3 with the files in them can I make something like this:
      While $iDs > 0 ;$iDs is the number of files in Folder1>> $iDsO[0] $iDs#here should come the Foldername for example '1'# = _FileListtoArray(@ProgramFilesDir&"\Folder1\"&$iDsO[$iDs]) $iDs = $iDs - 1 Wend So that in the End I have three variabels ($iDs1, $iDs2 and $iDs3)
       
      Is this posible or if not what could I do instead ( I don´t know the number of folders in Folder1 in the begining).
×