Searching for MemLeak

[KaFu]

HiHo Forum ,

this is directly related to

To keep the main UI process responsive I've relocated the extraction to a child process. When the child has prepared the thumbnail it sends it to the main UI script via this excellent Exchange Variables UDF Beta by eukalyptus.

Everything works fine now ... except... that there seems to be a Memory Leak in the UDF ... I'm currently trying to hunt down this leak and assembled some test scripts.

Compile Example_GDI_Send.au3, then start Example_GDI_Receive.au3 and see the debugging info in the console.

Would be great if someone with deeper knowledge on this would give me a helping hand . Oh, and one more question, can someone with a restricted (non-admin) account run the test script and see if it works at all, or IsAdmin required (don't have a non-Admin account on hand to test )?

Best Regards

[JohnOne]

There are a steady amount of gdi objects being created and non destroyed.

You can see them counting up in task manager.

in the example_Send.exe process.

Edit:

Although the memory leak seems to be from autoit.exe running from scite, It may be because the exe is a chile of autoit.exe

Edited January 6, 2011 by JohnOne

[rover]

Hi KaFu

I see steady GDI/Obj count and handles

you forgot to dispose of the bitmap in your test code

_GDIPlus_BitmapDispose($hBitmap)

cheers

Edit: In Windows XP

Edited January 6, 2011 by rover

[KaFu]

Ah, I see . But that's only in the test-code luckily ...

Edit:

Nope, too fast... I don't see it ... imho the only sensible position for _GDIPlus_BitmapDispose($hBitmap) is in Example_GDI_Receive.au3... but the GDI objects build up in Example_GDI_Send.au3 ? (still only an issue in the test code, the prod code does not pile GDI objects up (?)).

Lucky as I am at least I found the memory leak itself. In the UDF there was this line

Assign($aData[2], $Variant)

assigning a dllstructure to another variable... which seems to leave the original dllstructure orphaned! Replacing it with

$aData[2] = $Variant

solved the mem leak . Is this an AutoIt bug to be reported ? $Variant was declared as Local within the function.

Here's an update of my test-code:

And here's a general tip for all those searching mem leaks. Add this code (courtesy of UEZ) to the top of your leaking script:

Global $iEnmu = 0
Local Const $Process_All_Access = 0x1F0FFF
$ph = DllCall("kernel32.dll", "hwnd", "OpenProcess", "dword", $Process_All_Access, "int", False, "dword", @AutoItPID)
Global $ProcessHandle = $ph[0]
Func _ProcessGetMem() ;get physical Memory of the process -> http://msdn.microsoft.com/en-us/library/ms683219%28VS.85%29.aspx
    ; by UEZ
    ; http://www.autoitscript.com/forum/topic/121389-memory-consumption-of-an-executable/page__view__findpost__p__843018
    Local $structPROCESS_MEMORY_COUNTERS, $structPROCESS_MEMORY_COUNTERS_EX, $nSize, $aRet
    If @OSBuild < 7600 Then
        $structPROCESS_MEMORY_COUNTERS = DllStructCreate("dword cb; dword PageFaultCount; dword_ptr PeakWorkingSetSize; ulong_ptr WorkingSetSize; " & _
        "dword_ptr QuotaPeakPagedPoolUsage; dword_ptr QuotaPagedPoolUsage; dword_ptr QuotaPeakNonPagedPoolUsage; " & _
        "dword_ptr QuotaNonPagePoolUsage; dword_ptr PagefileUsage; dword_ptr PeakPagefileUsage") ;http://msdn.microsoft.com/en-us/library/ms684877%28VS.85%29.aspx
        $nSize = DllStructGetSize($structPROCESS_MEMORY_COUNTERS)
        $aRet = DllCall("psapi.dll", "int", "GetProcessMemoryInfo", "hwnd", $ProcessHandle, "ptr", DllStructGetPtr($structPROCESS_MEMORY_COUNTERS), "dword", $nSize) ;call GetProcessMemoryInfo
        If $aRet[0] = 0 Then
            ConsoleWrite("(" & @ScriptLineNumber & ") : = Error in GetProcessMemoryInfo call" & @LF)
            SetError(1, 0, $aRet[0])
        EndIf
        Return DllStructGetData($structPROCESS_MEMORY_COUNTERS, "WorkingSetSize")
    Else
        $structPROCESS_MEMORY_COUNTERS_EX = DllStructCreate("dword cb; dword PageFaultCount; dword_ptr PeakWorkingSetSize; dword_ptr WorkingSetSize; " & _
        "dword_ptr QuotaPeakPagedPoolUsage; dword_ptr QuotaPagedPoolUsage; dword_ptr QuotaPeakNonPagedPoolUsage; " & _
        "dword_ptr QuotaNonPagePoolUsage; dword_ptr PagefileUsage; dword_ptr PeakPagefileUsage; " & _
        "dword_ptr PrivateUsage") ;http://msdn.microsoft.com/en-us/library/ms684877%28VS.85%29.aspx
        $nSize = DllStructGetSize($structPROCESS_MEMORY_COUNTERS_EX)
        $aRet = DllCall("Kernel32.dll", "int", "K32GetProcessMemoryInfo", "hwnd", $ProcessHandle, "ptr", DllStructGetPtr($structPROCESS_MEMORY_COUNTERS_EX), "dword", $nSize) ;call GetProcessMemoryInfo
        If $aRet[0] = 0 Then
            ConsoleWrite("(" & @ScriptLineNumber & ") : = Error in GetProcessMemoryInfo call" & @LF)
            SetError(1, 0, $aRet[0])
        EndIf
    ;~ ConsoleWrite("WorkingSetSize: " & Round(DllStructGetData($structPROCESS_MEMORY_COUNTERS_EX, "WorkingSetSize") / 1024, 0) & @CRLF & _
    ;~ "PrivateUsage: " & Round(DllStructGetData($structPROCESS_MEMORY_COUNTERS_EX, "PrivateUsage") / 1024, 0) & @CRLF & @CRLF)
        Return DllStructGetData($structPROCESS_MEMORY_COUNTERS_EX, "PrivateUsage")
    EndIf
EndFunc ;==>_ProcessGetMem

Then follow the program flow and add these lines from time to time to encircle the bugger ...

$iEnmu += 1
ConsoleWrite("Step 3 " & $iEnmu & @TAB & _ProcessGetMem() & @CRLF) ; Replace Step 3 by other unique identifier

[KaFu]

Damn, to shortsighted . In Assign($aData[2], $Variant) the $aData[2] is a variable name, not the variable itself... the reason there's no mem leak is that this code does not work... back to work again.

[JohnOne]

Seems to me that you have narrowed it down though, Its most likely that $variant that is not being disposed of.

Or at least its address space is not being freed after the disposal of the actual bitmap.

[trancexx]

Fix _ScreenCapture_Capture function first, then do the testings. I have written about this function's leaking, others too. If it would make you feel any better I think it's fixed in non-existing future release.

It doesn't make sense, right?

[JohnOne]

Thats a good point.

I ran the code on a laptop which has a virgin install of autoit, when I just tried it on my pc, although the memory leak

is atill present, the gdi problem was gone.

Using winmerge (because I cannot remember altering the udf) I found in the screencapture_capture function around lin 85

thes lines.

_WinAPI_DrawIcon($hCDC, $aCursor[3] - $aIcon[2] - $iLeft, $aCursor[4] - $aIcon[3] - $iTop, $hIcon)
_WinAPI_DestroyIcon($hIcon)

On my pc they were also present, but a couple of other lines directly below them

_WinAPI_DeleteObject($aIcon[4])
_WinAPI_DeleteObject($aIcon[5])

Must have seen it on the forum at some point and added them.

[KaFu]

Seems to me that you have narrowed it down though, Its most likely that $variant that is not being disposed of.

Yep , the actual prod code now works fine. The trick was to replace

Assign($aData[2], $Variant)

with something like this

Switch $aData[2]
    Case "_TNP_P_Busy"
        $_TNP_P_Busy = $Variant
    Case "_TNP_P_BMP_Info"
        $_TNP_P_BMP_Info = $Variant
    Case "_TNP_P_BMP_Bits"
        $_TNP_P_BMP_Bits = $Variant
    Case "_TNP_P_BMP_ID"
        $_TNP_P_BMP_ID = $Variant
    Case "_TNP_C_Requested_Thumbs"
        $_TNP_C_Requested_Thumbs = $Variant
EndSwitch
$Variant = 0

You have to know of course the exact possible values of $aData[2] (the var names to transfer via the UDF) and add them to #Obfuscator_Ignore_Variables= to obfuscate the script properly.

Fix _ScreenCapture_Capture function first, then do the testings.

Realized that it's broken, but as my prod code works fine . I'll threw a working example together tomorrow .

I have written about this function's leaking, others too. If it would make you feel any better I think it's fixed in non-existing future release.

It doesn't make sense, right?

Oh, never read that, that issue's new too me . It's been reported already?

Thanks a bunch for taking a look !

Edit:

Just realized that trancexx has been talking about the GDI leakage in _ScreenCapture_Capture() . Has the possible memory leak with Assign() been reported yet?

Edit2:

Strange , can't reproduce the memory leak... maybe a coincidence with the ReadMemory?

expandcollapse popup

#region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_UseUpx=n
#AutoIt3Wrapper_UseX64=n
#endregion ;**** Directives created by AutoIt3Wrapper_GUI ****
Global $iEnmu = 0
Local Const $Process_All_Access = 0x1F0FFF
$ph = DllCall("kernel32.dll", "hwnd", "OpenProcess", "dword", $Process_All_Access, "int", False, "dword", @AutoItPID)
Global $ProcessHandle = $ph[0]
Func _ProcessGetMem() ;get physical Memory of the process -> http://msdn.microsoft.com/en-us/library/ms683219%28VS.85%29.aspx
    ; by UEZ
    ; http://www.autoitscript.com/forum/topic/121389-memory-consumption-of-an-executable/page__view__findpost__p__843018
    Local $structPROCESS_MEMORY_COUNTERS, $structPROCESS_MEMORY_COUNTERS_EX, $nSize, $aRet
    If @OSBuild < 7600 Then
        $structPROCESS_MEMORY_COUNTERS = DllStructCreate("dword cb; dword PageFaultCount; dword_ptr PeakWorkingSetSize; ulong_ptr WorkingSetSize; " & _
                "dword_ptr QuotaPeakPagedPoolUsage; dword_ptr QuotaPagedPoolUsage; dword_ptr QuotaPeakNonPagedPoolUsage; " & _
                "dword_ptr QuotaNonPagePoolUsage; dword_ptr PagefileUsage; dword_ptr PeakPagefileUsage") ;http://msdn.microsoft.com/en-us/library/ms684877%28VS.85%29.aspx
        $nSize = DllStructGetSize($structPROCESS_MEMORY_COUNTERS)
        $aRet = DllCall("psapi.dll", "int", "GetProcessMemoryInfo", "hwnd", $ProcessHandle, "ptr", DllStructGetPtr($structPROCESS_MEMORY_COUNTERS), "dword", $nSize) ;call GetProcessMemoryInfo
        If $aRet[0] = 0 Then
            ConsoleWrite("(" & @ScriptLineNumber & ") : = Error in GetProcessMemoryInfo call" & @LF)
            SetError(1, 0, $aRet[0])
        EndIf
        Return DllStructGetData($structPROCESS_MEMORY_COUNTERS, "WorkingSetSize")
    Else
        $structPROCESS_MEMORY_COUNTERS_EX = DllStructCreate("dword cb; dword PageFaultCount; dword_ptr PeakWorkingSetSize; dword_ptr WorkingSetSize; " & _
                "dword_ptr QuotaPeakPagedPoolUsage; dword_ptr QuotaPagedPoolUsage; dword_ptr QuotaPeakNonPagedPoolUsage; " & _
                "dword_ptr QuotaNonPagePoolUsage; dword_ptr PagefileUsage; dword_ptr PeakPagefileUsage; " & _
                "dword_ptr PrivateUsage") ;http://msdn.microsoft.com/en-us/library/ms684877%28VS.85%29.aspx
        $nSize = DllStructGetSize($structPROCESS_MEMORY_COUNTERS_EX)
        $aRet = DllCall("Kernel32.dll", "int", "K32GetProcessMemoryInfo", "hwnd", $ProcessHandle, "ptr", DllStructGetPtr($structPROCESS_MEMORY_COUNTERS_EX), "dword", $nSize) ;call GetProcessMemoryInfo
        If $aRet[0] = 0 Then
            ConsoleWrite("(" & @ScriptLineNumber & ") : = Error in GetProcessMemoryInfo call" & @LF)
            SetError(1, 0, $aRet[0])
        EndIf
;~ ConsoleWrite("WorkingSetSize: " & Round(DllStructGetData($structPROCESS_MEMORY_COUNTERS_EX, "WorkingSetSize") / 1024, 0) & @CRLF & _
;~ "PrivateUsage: " & Round(DllStructGetData($structPROCESS_MEMORY_COUNTERS_EX, "PrivateUsage") / 1024, 0) & @CRLF & @CRLF)
        Return DllStructGetData($structPROCESS_MEMORY_COUNTERS_EX, "PrivateUsage")
    EndIf
EndFunc   ;==>_ProcessGetMem




Global $tVar
Global $tVarCopy = DllStructCreate("char[10000]")
Global $sString = ""
For $i = 0 To 9999
    $sString &= "a"
Next
DllStructSetData($tVarCopy, 1, $sString)

For $i = 0 To 1000000
    Assign("tVar", $tVarCopy)
    If not Mod($i, 10) Then
        $iEnmu += 1
        ConsoleWrite("Step " & $iEnmu & @TAB & StringLen(DllStructGetData($tVar, 1)) & @TAB & _ProcessGetMem() & @CRLF)
    EndIf
Next

Edit3: Update example above...

Edited January 6, 2011 by KaFu

[trancexx]

From the memory leak point of view Assign() could very well be, to use JohnOne's wordings, a virgin.

[KaFu]

Here's a working example, compile both (start Example_GDI_Receive.exe) and you'll see the mem usage and GDI object count stable . The screenshot is actually create by Example_GDI_Send.exe and send via a combo of windows messages for notification and reading the bitmap from the childs exe memory directly...