Memory Leak

[ChrisL]

I can't figure out why but this application has a memory leak.

It has about 1500 machines login each morning but for each machine that logs in it takes a bit of memory but never seems to give it back. I can't figure out why though.

Any ideas?

[evilertoaster]

from a quick glace all seems in order... does the memory simply not return after the script runs? how are you knowing this?

What if you set the log file to open and write to a handle, then close it rather and filewriteline() on a path?

Edited April 30, 2007 by evilertoaster

[ChrisL]

from a quick glace all seems in order... does the memory simply not return after the script runs? how are you knowing this?

What if you set the log file to open and write to a handle, then close it rather and filewriteline() on a path?

The error log very rarely gets written to, there is a viewer to view the data and the only error you ever get is that the database is locked when you are doing a query with the viewer and the server tries to update an entry.

When it launches it has about 8,000 k memory and you can see it creep up as connections come in after a day or 2 it goes up to 50mb or more. Most of the computers start logging in at about 7am and this is when you can visibly see the memory going up and up.

I have seen this before with a script where you call a funtion by using a variable name the same as the one you call in the function but I have been careful to avoid this since I made this mistake before.

_Myfunction($var)

Func _Myfunction ($var)

;do stuff

Endfunc

[evilertoaster]

you may be able to see if the problem lies in the TCP/IP layout if you add some code to maybe shutdown the servies every so often (Just for a test) and see if the resources return to normal when this happens or not.

Also maybe you could track your variables and see if one of them is growing in size for some reason (perhaps doing a binary stringlen() will accomplish this?)

quite abit of code to go over so its tough (i cant test it either )

[ChrisL]

you may be able to see if the problem lies in the TCP/IP layout if you add some code to maybe shutdown the servies every so often (Just for a test) and see if the resources return to normal when this happens or not.

Also maybe you could track your variables and see if one of them is growing in size for some reason (perhaps doing a binary stringlen() will accomplish this?)

quite abit of code to go over so its tough (i cant test it either )

Mmmm well I've shut down the TCP and restarted it and this has had no effect, I reakon it's in the sqlite stuff somewhere

Cheers for your advice

[ChrisL]

Right I have narrowed it down to this function

If I fake the return values with the dummy array I get no memory issues, so it is getting the table column names that is the prolem for some reason??

Func GetColumnHeaders($Table, $AsArray=1)
     #cs
        Local $dummy[8] 
        $Dummy[0]=7
        $dummy[1]="ID"
        $dummy[2]="ComputerName"
        $dummy[3]="Hardware"
        $dummy[4]="MicroVersion"
        $dummy[5]="RemoteDatetime"
        $dummy[6]="GatewayDateTime"
        $dummy[7]="LastLoggedTime"
         
        Return $Dummy
    #ce
          
    Local $hQuery, $aNames, $ColumnHeaders= ""

        _SQlite_Query (-1, "SELECT ROWID,* FROM " & $Table & " LIMIT 1;", $hQuery)
        If NOT _SQLite_FetchNames ($hQuery, $aNames) = $SQLITE_OK then LogError("Sqlite Error on get column headers " & _SQLite_ErrMsg ()); Read out Column Names
        
        If isArray ($aNames) then $aNames[0] = Ubound($aNames)-1

        If $AsArray = 1 then Return $aNames;if use array is set return an array
        
    ;if not using array then return a pipe deliminated string
        For $i = 1 to Ubound($aNames) -1
            $ColumnHeaders&=$aNames[$i] & ","
        Next

        $ColumnHeaders = StringTrimRight($ColumnHeaders,1)
                        
        Return $ColumnHeaders
EndFunc

[WeMartiansAreFriendly]

Right I have narrowed it down to this function

If I fake the return values with the dummy array I get no memory issues, so it is getting the table column names that is the prolem for some reason??

Func GetColumnHeaders($Table, $AsArray=1)
     #cs
        Local $dummy[8] 
        $Dummy[0]=7
        $dummy[1]="ID"
        $dummy[2]="ComputerName"
        $dummy[3]="Hardware"
        $dummy[4]="MicroVersion"
        $dummy[5]="RemoteDatetime"
        $dummy[6]="GatewayDateTime"
        $dummy[7]="LastLoggedTime"
         
        Return $Dummy
    #ce
          
    Local $hQuery, $aNames, $ColumnHeaders= ""

        _SQlite_Query (-1, "SELECT ROWID,* FROM " & $Table & " LIMIT 1;", $hQuery)
        If NOT _SQLite_FetchNames ($hQuery, $aNames) = $SQLITE_OK then LogError("Sqlite Error on get column headers " & _SQLite_ErrMsg ()); Read out Column Names
        
        If isArray ($aNames) then $aNames[0] = Ubound($aNames)-1

        If $AsArray = 1 then Return $aNames;if use array is set return an array
        
;if not using array then return a pipe deliminated string
        For $i = 1 to Ubound($aNames) -1
            $ColumnHeaders&=$aNames[$i] & ","
        Next

        $ColumnHeaders = StringTrimRight($ColumnHeaders,1)
                        
        Return $ColumnHeaders
EndFunc

autoit doesnt seem to have the best memory manage.. try resetting the arrays Dim, local, .. after they been used.

[ChrisL]

Well looking at what I did it was a pretty inefficient(sp) way of doing it anyway.

I have changed it to only get the headers once at the begining and store them in an array and if the database needs to be modified by the client then the array gets modified at the point that the database gets altered.

It is running now so I guess only time will tell.

[Jon]

autoit doesnt seem to have the best memory manage.. try resetting the arrays Dim, local, .. after they been used.

That's not particularly helpful. If there is a case where you have to reset arrays, variables manually because there is a memory leak there is a BUG that should be reported so we can fix it.

In this case I'd like to get confirmation that it's not some aspect of the SQLLite calls that are causing a leak before I start spending time investigating

[ChrisL]

That's not particularly helpful. If there is a case where you have to reset arrays, variables manually because there is a memory leak there is a BUG that should be reported so we can fix it.

In this case I'd like to get confirmation that it's not some aspect of the SQLLite calls that are causing a leak before I start spending time investigating

I think this crudely shows the problem

expandcollapse popup

#include <SQLite.au3>
#include <SQLite.dll.au3>

$start = GetMemory()

_SQLite_Startup ()
_SQLite_Open (); open :memory: Database
_SQLite_Exec (-1, "CREATE TABLE aTest (a,b,c);")
_SQLite_Exec (-1, "INSERT INTO aTest(a,b,c) VALUES ('c','2','World');")
_SQLite_Exec (-1, "INSERT INTO aTest(a,b,c) VALUES ('b','3',' ');")
_SQLite_Exec (-1, "INSERT INTO aTest(a,b,c) VALUES ('a','1','Hello');")


For $i = 1 to 2000
    $a = GetColumnHeaders("aTest")
Next

$Finish = GetMemory()

Msgbox(0,"Memory","Memory usage at start was " & $Start & @crlf & "Memory usage at Finish was " & $Finish) 

_SQLite_Exec (-1, "DROP TABLE aTest;")
_SQLite_Close ()
_SQLite_Shutdown ()

Func GetColumnHeaders($Table, $AsArray=1)
 
    Local $hQuery, $aNames, $ColumnHeaders= ""

        _SQlite_Query (-1, "SELECT ROWID,* FROM " & $Table & " LIMIT 1;", $hQuery)
        If NOT _SQLite_FetchNames ($hQuery, $aNames) = $SQLITE_OK then LogError("Sqlite Error on get column headers " & _SQLite_ErrMsg ()); Read out Column Names
        
        If isArray ($aNames) then $aNames[0] = Ubound($aNames)-1

        If $AsArray = 1 then Return $aNames;if use array is set return an array
        
;if not using array then return a pipe deliminated string
        For $i = 1 to Ubound($aNames) -1
            $ColumnHeaders&=$aNames[$i] & ","
        Next

        $ColumnHeaders = StringTrimRight($ColumnHeaders,1)
                        
        Return $ColumnHeaders
EndFunc
    
Func LogError($error)
        
EndFunc
    
    
    Func GetMemory()
        $wbemFlagReturnImmediately = 0x10
$wbemFlagForwardOnly = 0x20
$colItems = ""
$strComputer = "localhost"

$Output=""
$Output = $Output & "Computer: " & $strComputer  & @CRLF
$Output = $Output & "==========================================" & @CRLF
$objWMIService = ObjGet("winmgmts:\\" & $strComputer & "\root\CIMV2")
$colItems = $objWMIService.ExecQuery('SELECT * FROM Win32_Process WHERE ProcessID= "' & @AutoitPid & '"', "WQL", _
                                          $wbemFlagReturnImmediately + $wbemFlagForwardOnly)

If IsObj($colItems) then
   For $objItem In $colItems
    
    Return $objItem.WorkingSetSize /1024

      if Msgbox(1,"WMI Output",$Output) = 2 then ExitLoop
      $Output=""
   Next
Else
   Msgbox(0,"WMI Output","No WMI Objects Found for class: " & "Win32_Process" )
Endif

Endfunc

Func WMIDateStringToDate($dtmDate)

    Return (StringMid($dtmDate, 5, 2) & "/" & _
    StringMid($dtmDate, 7, 2) & "/" & StringLeft($dtmDate, 4) _
    & " " & StringMid($dtmDate, 9, 2) & ":" & StringMid($dtmDate, 11, 2) & ":" & StringMid($dtmDate,13, 2))
EndFunc

[ChrisL]

If you add in to finalize the query then the memory is stable

_SQlite_Query (-1, "SELECT ROWID,* FROM " & $Table & " LIMIT 1;", $hQuery)
        If NOT _SQLite_FetchNames ($hQuery, $aNames) = $SQLITE_OK then LogError("Sqlite Error on get column headers " & _SQLite_ErrMsg ()); Read out Column Names
        _SQLite_QueryFinalize($hQuery)

But the help file does not say you need to do this.

Edited May 1, 2007 by ChrisL

[Silverkorn]

Thanks, this helped me to realize that, in my case, the memory leak was due by calling _SQLite_ErrCode/_SQLite_ErrMsg after the _SQLite_QueryFinalize function.