Jump to content

Recommended Posts

I'm basically processing text files where I'm reading them from a disk file into an input array, going through the input array and skipping what I don't want and writing it out to an output array and then writing the output array to a disk file.  I encountered an "Error allocating memory" message.  I have stripped down the program to illustrate how the memory usage grows in a looping routine (__ReadFiles).  I have tried a _ReduceMemory routine (from https://www.autoitscript.com/forum/topic/134831-is-there-a-way-to-free-up-memory-being-used/ ) and it helps somewhat, however the peak working set continues to grow regardless.  I believe I'm releasing resources (resetting arrays to 0) and am trying to figure out if the leak is from FileFindNextFile or something else I'm doing.   To illustrate the program creates a subdirectory with 50000 simple text files and then runs showing the growing memory usage, Ctrl+q will pause program.  Rerunning the program uses the already created files so it does not have to be re-created.  Any help appreciated.

; use this to debug in console window <--- LOOK
#AutoIt3Wrapper_run_debug_mode=Y
;#AutoIt3Wrapper_Run_Debug=off
;#AutoIt3Wrapper_Run_Debug=on


#include <Array.au3>
#include <File.au3>
#include <MsgBoxConstants.au3>
#include <GUIConstantsEx.au3>
#include <EditConstants.au3>

HotKeySet ( "^q", "EndProgram" )    ;v2e - added HotKeySet ( "^q", "EndProgram" ); CTRL+q so can pause/exit script

Global $hEdit   ;for scrolling text box

Global $textarray[1]    ;1 element
$textarray[0] = 0       ;initial count
Global $newtextarray[1] ;1 element
$newtextarray[0] = 0    ;initial count

Global $dir
Global $k       ;loop count

Global $aMemory ;v1g    for use in __ShowMemoryUsage("Location: In FileFindNextFile While 1 loop.")

Global $debug = 0       ;0=off, 1=on , 2=deeper,  - use debug > 1 to see arrays in Funcs


$version = "v1g"
;v1g - we have an Autoit "Error allocating memory" somewhere - printing out WorkingSetSize and  PeakWorkingSetSize from ProcessGetStats to try and find it


__scrolling_text_box_init("--- Read files ---", 1050, 450)  ;__scrolling_text_box_init($title, $width, $height)     ;set up box - to display text use: __scrolltext($text)
WinMove("--- Read files ---", "", 2000, 300)    ;move so can see it - on single screen use  10, 30 on dual screen use 2000, 300
__scrolltext("Starting:  " & @CRLF)


;illustrate error using current script directory by creating unique directotry and create file in it and put some text into it -----------
$dir = @ScriptDir

;for iterative testing see if test directory already exists - i.e. only need to build this once
$dir = @ScriptDir & "\testdir_xcrgua"   ;highly unlikely areadly exists unless this program was already run
$str = ""   ;null it
For $j = 1 to 100   ;put 100 lines in each file
    $str = $str & $j & @CRLF
Next

If DirGetSize($dir) = -1 Then
    DirCreate($dir) ;create directory   ;then create and fill it
    For $i = 1 to 50000     ;brute force (and I know it's slow, however straightforward and only 1 time
        FileWrite($dir & "\" & $i & ".txt", $str)
        If $i/1000 = Int($i/1000) Then __scrolltext("Writing test file " & $i & "/50000" & @CRLF)
    Next
EndIf
MsgBox(262144, "DEBUG", $dir & "  <-- should have 50000 files each with 100 numbered lines 1- 100.")

MsgBox(262144, "DEBUG", "Paused.")


__scrolltext("Reading files in dir = " & $dir & @CRLF)
;__ShowMemoryUsage("Location: Before file count.")  ;v1g
$fc = __GetFileCount($dir)  ;returns file count in $dir
;__ShowMemoryUsage("Location: After file count.")   ;v1g
__scrolltext("Number of files in dir = " & $fc & @CRLF)
MsgBox(262144, "DEBUG", "Paused.")

;in this we get memory allocation erroroccurs between 25K and 50K on my machine
__ReadFiles($dir)


MsgBox(262144, "DEBUG", "Paused out of loop.")

Exit


;----------------------------------- Functions -----------------------------------

Func    __ShowMemoryUsage($title)

        $aMemory = ProcessGetStats()    ; Retrieve memory details about the current process.

        ; If $aMemory is an array then display the following details about the process.
        If IsArray($aMemory) Then
            __scrolltext($title & "   " & "WorkingSetSize: " & $aMemory[0] & "      PeakWorkingSetSize: " & $aMemory[1] & @CRLF)
        Else
            __scrolltext("An error occurred in ProcessGetStats() in Func __ShowMemoryUsage($title).")
        EndIf

EndFunc     ;__ShowMemoryUsage($title)


Func _ReduceMemory($i_PID = -1) ;from https://www.autoitscript.com/forum/topic/134831-is-there-a-way-to-free-up-memory-being-used/
    If $i_PID <> -1 Then
        Local $ai_Handle = DllCall("kernel32.dll", 'int', 'OpenProcess', 'int', 0x1F0FFF, 'int', False, 'int', $i_PID)
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', $ai_Handle[0])
        DllCall('kernel32.dll', 'int', 'CloseHandle', 'int', $ai_Handle[0])
    Else
        Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', -1)
    EndIf
    Return $ai_Return[0]
EndFunc   ;==>_ReduceMemory



Func    __GetFileCount($dir)    ;returns number of files in $dir - brute force but only about 50K files/directory

    ;MsgBox(0, "DEBUG", "$dir = '" & $dir & "'")
#AutoIt3Wrapper_Run_Debug=off

    $fcount = 0     ;init count
    $search = FileFindFirstFile($dir & "\*.*")

    If $search = -1 Then    ;no files found
        Return($fcount)
        ;MsgBox(0, "Error", "No files/directories matched the search pattern")
        ;Exit
    EndIf

    ;okay check number of files
    While 1
        $file = FileFindNextFile($search)
        If @error Then
            ExitLoop        ;exit loop when there are no more files
        Else
            $fcount = $fcount + 1
        EndIf
    WEnd

    FileClose($search)  ;be polite

    Return($fcount)

#AutoIt3Wrapper_Run_Debug=on

EndFunc


Func    __ReadFiles($dir)   ;just reading to show memory alloc error

    Local $x
    Local $skipline

#AutoIt3Wrapper_Run_Debug=off
    $tfc = __GetFileCount($dir) ;get total file count so we can show progress
#AutoIt3Wrapper_Run_Debug=on

    $cfc = 0    ;init currect file count
    $search = FileFindFirstFile($dir & "\*.*")  ;get search handle

    If $search = -1 Then    ;no files found
        __scrolltext("Working on file: " & $cfc & "/" & $tfc & @CRLF)
        Return
        ;MsgBox(0, "Error", "No files/directories matched the search pattern")
        ;Exit
    EndIf

    ;okay go through the files one by one
    While 1

        ;_ReduceMemory()    ;this makes little difference - surprising

        $file = FileFindNextFile($search)
        If @error = 1 Then ExitLoop     ;exit loop when there are no more to do
        If @extended = 1 Then
            $adir = 1   ;we have a directory so note it
        Else
            $adir = 0   ;a file
        EndIf

        ;MsgBox(4096, "File:", $directory&"\"&$file)
        $cfc = $cfc + 1
        __scrolltext("Working on file: " & $cfc & "/" & $tfc & "     " & $file & @CRLF) ;show the filename

        __ShowMemoryUsage("Location: In FileFindNextFile While 1 loop.")    ;v1g

        ;clear out arrays so we have no artifacts from procesing the prior loop
        ;from here it looks like resetting it is better https://www.autoitscript.com/forum/topic/110933-solved-how-to-delete-whole-array/
        $testarray = 0
        $newtextarray = 0

        Dim $textarray[1]       ;1 element
        $textarray[0] = 0       ;initial count
        Dim $newtextarray[1]    ;1 element
        $newtextarray[0] = 0    ;initial count

        ;bring file into memory for faster processing
        If $adir = 0 Then   ;process file
            $x = _FileReadToArray($dir & "\" & $file, $textarray)       ;bring file into memory for faster processing [0] has $textarray count
            If $x <> 1 Then
                MsgBox($MB_TOPMOST + $MB_ICONERROR, "ERROR", "Unable to read file into array.  Error = " & $x & "    @error = " & @error & "    @extended = " & @extended)
                Exit
            EndIf
            If $debug > 0 Then _ArrayDisplay($textarray, "INPUT $textarray")

            ReDim $newtextarray[$textarray[0] + 1]  ;create output array ($newtextarray) of the same size as input array ($textarray) and later ReDim the output array to the correct size
            $newtextarray[0] = 0        ;set the count of actual entries as $newtextarray is empty but has same size as $textarray
            If $debug > 0 Then _ArrayDisplay($newtextarray, "Starting $newtextarray")

            ;now the approach is to write out only lines we want then to file
            For $i = 1 to $textarray[0] ;process all elements (rows, lines) in input array
                ;If $debug > 0 Then __scrolltext("$textarray[" & $i & "] = '" & $textarray[$i] & "'" & @CRLF)   ;show the line
                $skipline = 0   ;init value to not skip line
                ;----------- start stripping out stuff -----------
                ;this rountine is not the cause of the memory leak so removed
                ;-------------
                If $skipline = 0 Then   ;write it out
                    $newtextarray[0] = $newtextarray[0] + 1     ;increase count and point to index where we're going to store it in $newtextarray
                    $newtextarray[$newtextarray[0]] = $textarray[$i]
                Else
                    ;skip the line
                EndIf
            Next

            $fn = $dir & "\processed-1\" & StringLeft($file, StringLen($file) - 4) & "-1.txt"

            __scrolltext("Processed file written to: " &$fn & @CRLF)    ;show the line
            ;RECALL written out like an array so 1st entry which corresponds to index [0] has count of rows = lines in the file
        Else    ;its a directory - skip it
            __scrolltext("NOT processed directory (i.e. skipped) : " & $dir & "\" & $file & @CRLF)  ;show the line
        EndIf

    WEnd

EndFunc ;__CleanFiles($dir)



Func __scrolling_text_box_init($title, $width, $height)
    ;from http://www.autoitscript.com/forum/topic/110948-add-text-to-edit-box-and-scroll-it-down/page__p__971158__hl___guictrledit_scroll__fromsearch__1#entry971158

    $hGUI = GUICreate($title, $width, $height)
    $hEdit = GUICtrlCreateEdit("", 10, 10, $width-20, $height-20, BitOr($GUI_SS_DEFAULT_EDIT, $ES_READONLY))
    ;$limit = 9223372036854775807   ;2^63
    $limit = 1000000000 ;works
    $x = GUICtrlSetLimit($hEdit, $limit)
    ;~ If $x = 0 Then
    ;~  MsgBox(0, "ERROR", "Limit of " &  $limit & " is too large.")
    ;~ Else
    ;~  MsgBox(0, "Okay", "Limit of " &  $limit & " is okay.")
    ;~ EndIf

    ;$hButton = GUICtrlCreateButton("Add", 10, 250, 80, 30)

    GUISetState()

EndFunc


Func __scrolltext($text)
#AutoIt3Wrapper_Run_Debug=off
    ;prefix everything with date and time e.g. 2014-02-01 HH:MM:SS
    $dt = @YEAR & "-" & @MON & "-" & @MDAY & " " & @HOUR & ":" & @MIN & ":" & @SEC & "  "
    $text = $dt & $text
    GUICtrlSetData($hEdit, $text, 1)
#AutoIt3Wrapper_Run_Debug=on
EndFunc


Func EndProgram()
    $x = MsgBox($MB_TOPMOST + $MB_YESNO + $MB_ICONWARNING, "Paused", "Do you want to continue?")
    If $x = $IDNO Then
        Exit    ;exit program
    Else
        ;just a pause - so continue
    EndIf
EndFunc

 

04 - Clean files - v1g - test for memory leak v1c.au3

Edited by ahha
Added as a file.
Link to post
Share on other sites

Hi ahha

This is the line in your __scrolltext() function that is using all your memory.

GUICtrlSetData($hEdit, $text,1)

It's appending to the existing contents of the control each time it is executed. If you change it to this your memory problems will disappear.

GUICtrlSetData($hEdit, $text)

 

"Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to build bigger and better idiots. So far, the universe is winning."- Rick Cook

Link to post
Share on other sites

Bowmore,

Thanks - I totally missed that - duh - I feel stupid.:'(  I do have a question however.  Why when using scrolling text does the PeakWorkingSetSize increase but not the WorkingSetSize.  I would have thought the WorkingSetSize would increase if the text is being stored/displayed. I've forgotten how to indicate solved at the topic top.

Edited by ahha
Link to post
Share on other sites
On 2/1/2016 at 7:44 PM, ahha said:

Bowmore,

Thanks - I totally missed that - duh - I feel stupid.:'(  I do have a question however.  Why when using scrolling text does the PeakWorkingSetSize increase but not the WorkingSetSize.  I would have thought the WorkingSetSize would increase if the text is being stored/displayed. I've forgotten how to indicate solved at the topic top.

Edit your original post and change the title to [Solved] Memory Leak

 

 

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By TheAlienDoctor
      Hi, I was looking into creating a script that would detect if a file exists, then move it (and in some cases rename it, depending on the file) as well as write to a log file.
      The issue is, there is a lot of files that need to be moved, sometimes some files will exist and others won't depending on the use-case. However if a file does exist, it will always be going into the same directory with the same name.
      Currently I have an array nested inside of the array, with each array inside that array having both the old and new directory, and then a For loop to actually run through and do the file transferring. The issue I am having is how to call the Array inside of the array, because how do I specify  which the old directory is and which the new is?
      Global $FileTransfer[2000] = [Global $Dir1[2] = ["original dir 1", "new dir 1"], Global $Dir2[2] = ["original dir 2", "new dir 2"]] For $FileTransfer = [0] To [1] Step +1 If FileExists({original dir}) Then FileMove({original dir}, new dir, 1) FileOpen("log.latest.txt", 1) FileWrite("log.latest.txt", "{original dir} found, moved it to new dir." & @CRLF) FileClose("log.latest.txt") Else FileOpen("log_latest.txt", 1) FileWrite("log_latest.txt", "{original dir} not found, ignoreing it." & @CRLF) FileClose("log_latest.txt") EndIf Next I have put what I want the old and new directory to be for each array in {}, so hopefully its easier to tell which part is working and whats not.

      I am still reasonably new to AutoIT, any help is appreciated. Thankyou
    • By arunkw
      I have a spreadsheet - daily routine which has two columns: activity and time as shown here
      | Activity             | Time     | |----------------------|----------| | Sleep               |  6:00 am | | Toilet              |  6:15 am | | Get ready for gym  |  6:30 am | | Exercise            |  7:50 am | | ... more things      |  9:00 pm | | ... still more       | 10:45 pm | | Sleep               |  6:00 am |   I wanted to find out, say in C1 which activity is current for me using now() I.e., if it’s 6:45am on my watch, it should show me Exercise  in C1 Thanks to Adam D. PE, this formula works like magic to get the result =VLOOKUP(MOD(NOW(),1),{B2:B,A2:A},2,1)   Now, I want to reproduce same result in autoit, how to do that? To have easy solution say, I copy-paste spreadsheet data in array directly in code, right? Use for loop and run the above vlookup function and show the answer using tooltip. How to achieve this? please help.  
    • By goku200
      I have an Autoit script that lists files from a folder into an array list. Is there a way to separate the filenames by an underscore and include the id, version, name and date into separate columns in Excel.
      Example of filename:
      12345_v1.0_TEST Name [12345]_01.01.2022.html
      12345 would be in one column
      v1.0 would be in another column
      TEST Name [12345] would be in another column
      01.01.2022 would be in another column
      .html would be in another column
      Note: filenames always change each day.
      Here is my code that lists the files into column C and then writes the column Headers into Column D, E, F, G. Just need some help with separating them into columns by the _ delimiter
       
    • By mLipok
      Usually when I collect data from DataBase I need to give EndUser a possibility to select rows which should be taken in the processing loop.
      I was searching on the forum and I'm not able to find any UDF or even example of how to select data from array.
      I have my own solutions but I think they are not worth posting on the forum as it is very old code and I am looking for a better solution.

      Could anybody point me to some examples/solutions ?

      Thank you in advance.
      @mLipok
    • By EmilyLove
      I have a string containing the full path of an executable and an array of executables without their paths. I am trying to compare the string to the list in the array and if a match is found, remove it from the array. The entry get removed from the array successfully, and after checking its return result, uses it to update the ubound if it succeeded, but it doesn't want to update to the new value. Any ideas what I am doing wrong? It acts like it is read-only.
      #include <Array.au3> #include <File.au3> Local $sApp_Exe = "F:\App\Nextcloud\nextcloud.exe" Local $aWaitForEXEX = [3, "Nextcloud.exe", "nextcloudcmd.exe", "QtWebEngineProcess.exe"] For $h = 1 To $aWaitForEXEX[0] If StringInStr($sApp_Exe, $aWaitForEXEX[$h]) <> 0 Then $iRet = _ArrayDelete($aWaitForEXEX, $h) If $iRet <> -1 Then $aWaitForEXEX[0] = $iRet ;this line doesn't work. $aWaitForEXEX[0] doesn't update and shortly gives Error: Array variable has incorrect number of subscripts or subscript dimension range exceeded.: _ArrayDisplay($aWaitForEXEX) EndIf Next  
×
×
  • Create New...