Huge memory leak [SOLVED]

[queensoft]

Thought I'd share this with you, it's been bothering me for months.

A simple missing FileClose was causing huge RAM leaks, up to 7-8 Gb for a very simple program.

The entire program is bigger, but the main points are:

; search for .RAR files
; between 100-300 files
$f = _RecFileListToArray("e:\Download\", "*.*", 1, 1, 1)    ; function by SolidSnake <MetalGX91 at GMail dot com>

; search destination folders (by file name)
; between 100-3000 files
$dest = _RecFileListToArray("g:\", "info.txt", 1, 1, 1)

; process all .rar files
For $i=0 to UBound($f)-1

    ; process all destination folders
    For $j = 0 to UBound($dest)-1
        $f01 = FileOpen($dest[$j])
        $s01 = FileRead($f01)
        ; FileClose($f01) ; if this is missing, RAM usage explodes, as the program is running
        ; now I understand why: first loop runs from 0 to 100-300 (.rar files)
            ; for each run, the second loop runs from 0 up until folder is found, from 0 to 100-3000
        ; that could mean thousands of runs each time (first loop * second loop) and they all pile up
    Next

Next

Hope it helps, always close open files!

Edited September 3, 2021 by queensoft

[JockoDundee]

What’s the program do exactly, this part at least?  Is it finding .rar files and then looking in multiple info.txt files and seeing if the file name is listed in the text file, or?

[Zedna]

Just hint:

If you use just one FileRead() then FileOpen()+FileClose() are not neccessary, AutoIt does it automatically internaly.

[pseakins]

11 hours ago, queensoft said:

Hope it helps, always close open files

I had to prove this for myself, I ran some other tests and then the code below.  I found yes, the problem is in reusing the file handle variable without first closing the file. Everything cleans up ok when your program ends, it is only an issue while it's running. If you run my test below and watch memory allocation with Windows Task Manager you will see free memory slowly reducing. Apparently, AutoIt does not automatically deallocate whatever memory was allocated during the previous file open. With proper coding this wouldn't be a problem anyway.

It's interesting that a file can be moved (renamed) while it is still open in the code.

You should be OK with a loop of 25000 but maybe you could reduce it for your first few runs.

FileDelete(@ScriptDir & "\LeakTest*.tst")
FileCopy(@ScriptFullPath, @ScriptDir & "\LeakTest0.tst")
For $i = 1 To 25000
  ConsoleWrite($i & " ")
  If Mod($i, 25) = 0 Then ConsoleWrite(@CRLF)
  If Not FileMove(@ScriptDir & "\LeakTest" & $i - 1 & ".tst", @ScriptDir & "\LeakTest" & $i & ".tst") Then Exit
  $h1 = FileOpen(@ScriptDir & "\LeakTest" & $i & ".tst")
  $s1 = FileRead($h1)
Next
FileDelete(@ScriptDir & "\LeakTest*.tst")

 

[queensoft]

15 hours ago, Zedna said:

Just hint:

If you use just one FileRead() then FileOpen()+FileClose() are not neccessary, AutoIt does it automatically internaly.

Yes, I know, I use both options, but now I just forgot to add FileClose.

Edited September 4, 2021 by queensoft

[queensoft]

15 hours ago, JockoDundee said:

What’s the program do exactly, this part at least?  Is it finding .rar files and then looking in multiple info.txt files and seeing if the file name is listed in the text file, or?

Yes, exactly. Then runs .BAT file, to UnRAR each archive to corresponding folder.

I thought that was the problem (running external programs or maybe RAR files big 1-2 Gb), but then noticed UnRAR.exe is only using something like 15 Mb of RAM.

Yes, I know I can store all info.txt files in an array (path + file content), instead of reading the files every time, I'll probably add that in the future.

[JockoDundee]

3 hours ago, queensoft said:

Yes, I know I can store all info.txt files in an array 

3 hours ago, queensoft said:

Yes, I know, I use both options,

Ok, I get it - you are not actually in need of any help - you are just cautioning us to the dangers of opening thousands of files concurrently.

We have been warned

[queensoft]

Well, yeah, it says [SOLVED] in the title 🤦‍♂️!

[JockoDundee]

This was one hella huge memory leak, I doubt we’ll see another like it anytime soon.