Jump to content

RAM Usage of Scripts doing external program calls

rudi
 Share Followers 0

Recommended Posts

rudi

rudi 40

Posted
Posted

Hi.

The final question first: Is there a reason, that Autoit doesn't clean up RAM usage as a standard feature?

 

This posting by @guinness was pointing me to the solution for solving my problem:

https://www.autoitscript.com/forum/topic/131315-accumulating-memory-usage/?do=findComment&comment=914208 

DllCall("psapi.dll", "int", "EmptyWorkingSet", "long", -1)

this simple, single line called on a regular basis stopped the script consuming more and more RAM.

CU, Rudi.

Earth is flat, pigs can fly, and Nuclear Power is SAFE!

Link to post
Share on other sites
  • Moderators
Melba23

Melba23 3,726

Posted
  • Moderators
Posted

rudi,

As I understand it, Windows does its own RAM clearing over time and using that particular  line of code merely pushes a little more of the less-often-used pages in RAM onto a Windows file on the harddisk. So it is a bit of a 2-edged sword as it will slow down your program if it suddenly decides it needs some pages back in RAM for usage.

In most cases I have seen over the years, ever-increasing RAM means you have a memory leak in your program - usually a handle or object not correctly disposed of once no longer needed - and fixing that underlying cause, rather than the symptom, has been the key to preventing it.

M23

 

Public_Domain.png.2d871819fcb9957cf44f4514551a2935.png Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind

Open spoiler to see my UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Link to post
Share on other sites
rudi

rudi 40

Posted
  • Author
Posted (edited)

Hi @Melba23,

 

thanks for your reply.

As I couldn't see any such program leaks, I played with these two scripts. I used CMD.EXE, as it's available on every Windows PC, the external program I had the problems with is a tiny console  prog (no GUI) to do connectivity checks.

 

I noticed, that killing the prog, when the output has the info I'm looking for for speed reasons, the RAM consumption is much higher then when I wait for natural program exit.

 

In the original script the connectivity test is done just every 2 minutes. Anyway, when I let that script run (compiled) when I leave my desk, I have even seen RAM consuptions beyond 1GB, when I came back next day or on Monday.

 

Setting $CleanupRAM = True will activate the RAM cleanup for both sample scripts.

Code sample 1: waiting for natural program exit 

; RAM-USAGE-with-ProcessWaitClose.au3

$sDone = "control.exe"
$loops = 0

$CleanupRAM = False

$aWS = ProcessGetStats()
$WSram = $aWS[0]
$WSramPeak = $aWS[1]

While 1
    $loops += 1
    $Timer = TimerInit()
    $PID = Run(@ComSpec & " /c dir c:\windows /s", @TempDir, @SW_HIDE, $STDERR_CHILD + $STDOUT_CHILD)
    ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $PID = ' & $PID & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console
    $sOut = ""
    ProcessWaitClose($PID)
    $sOut = StdoutRead($PID)
    UpdToolTip()
WEnd


Func UpdToolTip()
    $sResult = "Loops: " & $loops & ", Duration: " & Round(TimerDiff($Timer) / 1000, 2) & " sec." & @CRLF & StringLen($sOut) & " Chars received"
    $aWS = ProcessGetStats()
    $WS_TXT = "    WorkingSetSize (4k)= " & Round($aWS[0] / 1024 / 4, 2) & @CRLF & "PeakWorkingSetSize (4k)= " & Round($aWS[1] / 1024 / 4, 2) & @CRLF & _
            "RAM  Delta 4k pages : " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _
            "Peak Delta 4k pages : " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2)

    If $CleanupRAM Then
        DllCall("psapi.dll", "int", "EmptyWorkingSet", "long", -1)
        $aWS = ProcessGetStats()

        $WS_TXT &= @CRLF & _
                "RAM  Delta (after cleanup) = " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _
                "Peak Delta (after cleanup) = " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2)
    EndIf
    $WSram = $aWS[0]
    $WSramPeak = $aWS[1]
    If StringInStr($sOut, $sDone) Then
        ; ConsoleWrite($sOut & @CRLF)
        ToolTip("process ended: " & $PID & @CRLF & $WS_TXT & @CRLF & $sResult, 500, 100, "String found: " & $sDone)
    Else
        ToolTip("Process ended." & @CRLF & "Search string *NOT* found! " & $sDone & @CRLF & $WS_TXT & @CRLF & $sResult, 500, 100, "String not found: " & $sDone)
    EndIf
EndFunc   ;==>UpdToolTip

 

code sample 2: watching out for "my result", killing the PID ASAP 

; RAM-USAGE-with-ProcessClose.au3

#include <AutoItConstants.au3>
$sDone = "control.exe"
$loops = 0

$CleanupRAM = False

$aWS = ProcessGetStats()
$WSram = $aWS[0]
$WSramPeak = $aWS[1]

While 1
    $loops += 1
    $Timer=TimerInit()
    $PID = Run(@ComSpec & " /c dir c:\windows\system32 /s", @TempDir, @SW_HIDE, $STDERR_CHILD + $STDOUT_CHILD)
    ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $PID = ' & $PID & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console
    $sOut = ""
    While 1
        $sOut &= StdoutRead($PID)
        If StringInStr($sOut, $sDone) Then
            ProcessClose($PID)
            ExitLoop
        EndIf
    WEnd
    UpdToolTip()
WEnd

Func UpdToolTip()
    $sResult = "Loops: " & $loops & ", Duration: " & Round(TimerDiff($Timer) / 1000, 2) & " sec." & @CRLF & StringLen($sOut) & " Chars received"
    $aWS = ProcessGetStats()
    $WS_TXT = "    WorkingSetSize (4k)= " & Round($aWS[0] / 1024 / 4, 2) & @CRLF & "PeakWorkingSetSize (4k)= " & Round($aWS[1] / 1024 / 4, 2) & @CRLF & _
            "RAM  Delta 4k pages : " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _
            "Peak Delta 4k pages : " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2)

    If $CleanupRAM Then
        DllCall("psapi.dll", "int", "EmptyWorkingSet", "long", -1)
        $aWS = ProcessGetStats()

        $WS_TXT &= @CRLF & _
                "RAM  Delta (after cleanup) = " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _
                "Peak Delta (after cleanup) = " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2)
    EndIf
    $WSram = $aWS[0]
    $WSramPeak = $aWS[1]
    If StringInStr($sOut, $sDone) Then
        ; ConsoleWrite($sOut & @CRLF)
        ToolTip("process ended: " & $PID & @CRLF & $WS_TXT & @CRLF & $sResult, 500, 100, "String found: " & $sDone)
    Else
        ToolTip("Process ended." & @CRLF & "Search string *NOT* found! " & $sDone & @CRLF & $WS_TXT & @CRLF & $sResult, 500, 100, "String not found: " & $sDone)
    EndIf
EndFunc   ;==>UpdToolTip

 

Edited by rudi

Earth is flat, pigs can fly, and Nuclear Power is SAFE!

Link to post
Share on other sites
rudi

rudi 40

Posted
  • Author
Posted

@Melba23 

<cite>
... using that particular  line of code merely pushes a little more of the less-often-used pages in RAM onto a Windows file on the harddisk
</cite>

Hm. That means the RAM not freed correctly for some reason is just swapped to disk, if I get you correctly? (I missed that aspect before)

To the Windows swap file, or to some dedicated "Swap-file-for-this-app"?

Hm. Then this is definitely no option, as spoiling disk space, especially the Windows PAGEFILE.SYS to free RAM is not a good idea.

 

Can you see in the sample scripts how to avoid the continuosly growing RAM consumption?

TIA Rudi.

Earth is flat, pigs can fly, and Nuclear Power is SAFE!

Link to post
Share on other sites
rudi

rudi 40

Posted
  • Author
Posted (edited)

I think I've caught the problem:

 

It happens, when I use Run() with $STDERR_MERGED to catch the Output.

 

This script is demonstrating the situation. Press F12 to toggle between using and not using $STDERR_MERGED 

; *** Start added by AutoIt3Wrapper ***
#include <AutoItConstants.au3>
; *** End added by AutoIt3Wrapper ***
#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_Add_Constants=n
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****
#include <Misc.au3>



$loops = 0

$CleanupRAM = False

$aWS = ProcessGetStats()
$WSram = $aWS[0]
$WSramPeak = $aWS[1]

$xMax=9999
$yMax=99
Dim $aTest[$xMax+1][$yMax+1]


AdlibRegister(UpdToolTip)

$PID=False

$GrabOutput=False




While 1
    if _IsPressed("7B") Then
        if $GrabOutput=False Then
            $GrabOutput=True
        else
            $GrabOutput=False
        EndIf
        Sleep(100)
    EndIf

    $RandX=Random(0,$xMax,1)
    $RandY=Random(0,$yMax,1)
    $aTest[$RandX][$RandY]=$RandX * $RandY * Random(1,100)
    if ProcessExists($PID) then ProcessClose($PID)
    if $GrabOutput Then
    $PID=Run(@ComSpec & " /c echo hello",@TempDir,@SW_HIDE, $STDERR_MERGED)
    Else
    $PID=Run(@ComSpec & " /c echo hello",@TempDir,@SW_HIDE)
    EndIf
    $loops+=1
WEnd

Func UpdToolTip()
    $sResult = "Loops: " & $loops & @CRLF
    $aWS = ProcessGetStats()
    $WS_TXT = "    WorkingSetSize (4k)= " & Round($aWS[0] / 1024 / 4, 2) & @CRLF & "PeakWorkingSetSize (4k)= " & Round($aWS[1] / 1024 / 4, 2) & @CRLF & _
            "RAM  Delta 4k pages : " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _
            "Peak Delta 4k pages : " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2)

    If $CleanupRAM Then
        DllCall("psapi.dll", "int", "EmptyWorkingSet", "long", -1)
        $aWS = ProcessGetStats()

        $WS_TXT &= @CRLF & _
                "RAM  Delta (after cleanup) = " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _
                "Peak Delta (after cleanup) = " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2)
    EndIf
    $WSram = $aWS[0]
    $WSramPeak = $aWS[1]
        if $GrabOutput Then
        ToolTip($WS_TXT & @CRLF & $sResult & @CRLF & "With $STDERR_MERGED", 500, 100, "Random Array Fill, loops: " & $loops)
        Else
        ToolTip($WS_TXT & @CRLF & $sResult & @CRLF & "Without $STDERR_MERGED", 500, 100, "Random Array Fill, loops: " & $loops)
        EndIf
EndFunc   ;==>UpdToolTip

 

Helpfile for Run():

$STDERR_MERGED (0x8) = Provides the same handle for STDOUT and STDERR. Implies both $STDOUT_CHILD and $STDERR_CHILD.
 

Shot in the dark: This handle is not destroyed automatically.

 

Howto do that explicitely? Maybe StdioClose() , on my way to try that one.

Edited by rudi
edit typos. 2nd edit about "handle"

Earth is flat, pigs can fly, and Nuclear Power is SAFE!

Link to post
Share on other sites
rudi

rudi 40

Posted
  • Author
Posted (edited)

Okay, I suspect, that it's this combination:

 

  1. Start a Programm using Run() with the opt_flag $STDERR_MERGED
  2. killing this process

 

When doing so, it doesn't matter whether StdioClose() is used or not, and in case it *IS* used, whether it's used before or after the ProcessClose($PID), the RAM consumption is increasing with each and every Run() used.

 

 

Edited by rudi

Earth is flat, pigs can fly, and Nuclear Power is SAFE!

Link to post
Share on other sites
KaFu

KaFu 334

Posted
Posted (edited)

This works for me 

If ProcessExists($PID) Then
    While ProcessExists($PID)
        ProcessClose($PID)
    WEnd
    StdioClose($PID)
EndIf

Seems like StdioClose must be called to flush the stream, and processclose alone does not ensure that process is closed when StdioClose is called, so fragments pile up. Ensure that process is closed before the StdioClose call seems to fix this.

Whole modified script 

; *** Start added by AutoIt3Wrapper ***
#include <AutoItConstants.au3>
; *** End added by AutoIt3Wrapper ***
#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_Add_Constants=n
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****
#include <Misc.au3>

$loops = 0

$CleanupRAM = False

$aWS = ProcessGetStats()
$WSram = $aWS[0]
$WSramPeak = $aWS[1]

$xMax = 9999
$yMax = 99
Dim $aTest[$xMax + 1][$yMax + 1]


AdlibRegister(UpdToolTip)

$PID = False
$sOutput = ""

$GrabOutput = True


While 1
    If _IsPressed("7B") Then
        If $GrabOutput = False Then
            $GrabOutput = True
        Else
            $GrabOutput = False
        EndIf
        Sleep(100)
    EndIf

    $RandX = Random(0, $xMax, 1)
    $RandY = Random(0, $yMax, 1)
    $aTest[$RandX][$RandY] = $RandX * $RandY * Random(1, 100)

    If ProcessExists($PID) Then
        While ProcessExists($PID)
            ProcessClose($PID)
        WEnd
        StdioClose($PID)
    EndIf

    If $GrabOutput Then
        $PID = Run(@ComSpec & " /c echo hello", @TempDir, @SW_HIDE, $STDERR_MERGED)
    Else
        $PID = Run(@ComSpec & " /c echo hello", @TempDir, @SW_HIDE)
    EndIf
    $loops += 1
WEnd

Func UpdToolTip()
    $sResult = "Loops: " & $loops & @CRLF
    $aWS = ProcessGetStats()
    $WS_TXT = "    WorkingSetSize (4k)= " & Round($aWS[0] / 1024 / 4, 2) & @CRLF & "PeakWorkingSetSize (4k)= " & Round($aWS[1] / 1024 / 4, 2) & @CRLF & _
            "RAM  Delta 4k pages : " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _
            "Peak Delta 4k pages : " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2)

    If $CleanupRAM Then
        DllCall("psapi.dll", "int", "EmptyWorkingSet", "long", -1)
        $aWS = ProcessGetStats()

        $WS_TXT &= @CRLF & _
                "RAM  Delta (after cleanup) = " & Round(($aWS[0] - $WSram) / 1024 / 4, 2) & @CRLF & _
                "Peak Delta (after cleanup) = " & Round(($aWS[1] - $WSramPeak) / 1024 / 4, 2)
    EndIf
    $WSram = $aWS[0]
    $WSramPeak = $aWS[1]
    If $GrabOutput Then
        ToolTip($WS_TXT & @CRLF & $sResult & @CRLF & "With $STDERR_MERGED", 500, 100, "Random Array Fill, loops: " & $loops)
    Else
        ToolTip($WS_TXT & @CRLF & $sResult & @CRLF & "Without $STDERR_MERGED", 500, 100, "Random Array Fill, loops: " & $loops)
    EndIf
EndFunc   ;==>UpdToolTip

 

Edited by KaFu

de.png OS: Win10-21H1 - 64bit - German, AutoIt Version: 3.3.14.5, AutoIt Editor: SciTE, Website: https://funk.eu
prog_icon_amt.png AMT - Auto-Movie-Thumbnailer (2019-Dec-21) prog_icon_bic.png BIC - Batch-Image-Cropper (2019-Dec-11) prog_icon_cop.pngCOP - Color Picker (2009-May-21)
prog_icon_hmw.pngHMW - Hide my Windows (2018-Sep-16) prog_icon_hrc.pngHRC - HotKey Resolution Changer (2012-May-16)  prog_icon_icu.pngICU - Icon Configuration Utility (2018-Sep-16)
prog_icon_smf.pngSMF - Search my Files (2021-Oct-10) - THE file info and duplicates search tool prog_icon_ssd.pngSSD - Set Sound Device (2017-Sep-16)

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Followers 0
Go to topic listing

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • Tersion
      Huge RAM usage with GuiListView.au3 include.
      By Tersion
      Here test example of a dummy program with random added controls to the main form:
      If #include <GuiListView.au3> is commented out, then this simple program uses around 3,5 MB of RAM. When #include <GuiListView.au3> NOT commented out - RAM usage is around 13-14 MB.
      How can I reduce memory usage? Even if I'm not using GuiListView.au3 - 3,5 MB quite a bit for a such dummy program!
      I found out that using this DLLCall in main loop:
      DllCall("psapi.dll", "int", "EmptyWorkingSet", "long", -1) Significantly reduces RAM usage (even with GuiListView.au3 included, from 13-14 MB to 600 KB !!! ) but I'm not sure if it's doesn't have any impact to common workflow of a program...
      So, give me any advice about that, please.
    • simy8891
      Get network usage per process
      By simy8891
      Hi guys,
      It's been a while since I wrote my last message here and a while since I used AutoIt. I'm currently sort of desperate and I'm trying to find some help in regards of getting the network usage per process!
      I'm not interested in the total network usage of the NIC, but only on a specific PID's network utilization. They idea is to collect the amount of traffic uploaded and downloaded by a list of specific processes. So far Process Hacker and Process Explorer are capable of getting what I need, but I need to use these numbers in another script so they're sort of useless to me. I can't seem to find a way around it.
      Any idea, help is greatly appreciated.
      Thanks
    • VIP
      CPU_Widget_TimeSync
      By VIP
      The simple widget shows the percentage of cpu, ram used and the CPU temperature (WMI).
      - Every 30s check, sync and update time for computer.
      - Drag adn Drop any files on GUI for reset Attribute (-RAHS) and set Full Access to Everyone !
      - Righ Click on x and click Un/Install app to boot with Windows!
      CPU_Widget_TimeSync.exe CRC32: 1956339E MD5: BFC2B596BD1EDA35D6B02B97F8FB57AE SHA-1: 796A663739EA5AC9890D9FA324A98E987F35E7C8
      Function code from:
      CPU: 
      RAM:  MemGetStats ( )
      Memory:   _WinAPI_EmptyWorkingSet ()
       
    • Wicked_Caty
      Getting some statistics
      By Wicked_Caty
      I've written a small surveillance tool for one of my programs. It shows the time that has already passed, the estimated time until the end, the average time of the steps, and the progress in %.
      It works pretty well, but I want to add some stats. That'd be the usages of the CPU, RAM, network and hard-drive of all threads in %. Basically what you see in the taskmanager, but without the list of every single process.
      I'm aware that there's no in-built function for that, and that I'd need an additional library. Are there any for that? Simply functions that return the stats mentioned above. Thanks!
      Edit:   I run an ASUS R752M with an Intel BayTrail M3540 on Windows 10 64-bits.
    • dzlee
      best way to reduse memory usage of my script ?
      By dzlee
      hello autoit developers community
      i have some question , i hope to find the answers of it :
      - in large project or even in small ones
      how to make my compiled script use less memory ?
      - how to test my script and be sure that there is no memory leak , or whatsoever reduce my script performance ?
      - where can i find anything talks about autoit script architecture to avoid bad script design ?

      last thing i find these script
      can anyone describe how these scripts work and is these script actually work fine
      thanks
      Func _ReduceMemory() Local $ai_GetCurrentProcessId = DllCall('kernel32.dll', 'int', 'GetCurrentProcessId') Local $ai_Handle = DllCall("kernel32.dll", 'int', 'OpenProcess', 'int', 0x1f0fff, 'int', False, 'int', $ai_GetCurrentProcessId[0]) Local $ai_Return = DllCall("psapi.dll", 'int', 'EmptyWorkingSet', 'long', $ai_Handle[0]) DllCall('kernel32.dll', 'int', 'CloseHandle', 'int', $ai_Handle[0]) Return $ai_Return[0] EndFunc Func _SelfReduceMemory() DllCall("psapi.dll", "int", "EmptyWorkingSet", "long", -1) EndFunc 
×
×
  • Create New...