My script is eating all the RAM away.

rudi · July 5, 2011

Hi.

This script runs fine, but consumes more and more RAM, until it finally (after several days) takes all the RAM. Windows complains about "no more RAM", and ProcessExplorer shows, that the "dropper.exe" is the guilty party :-(

This is a longer script. It scans a input folder on a regular time schedule for files, or, subfolder(s) with files. These are packed (7zip), then packed again (7zip with random password). Then they resulting ZIP file is dropped into a local Dropbox Public Folder. After that's done, a TXT file is placed in an Output Folder, that has all the information in it to download and ZIP-decrypt-unpack the archive. As I have no clue, where it takes all the RAM, I didn't shorten the script, but pasted it "as is".

May be it's due to some infinite looping to deeper and deeper level, that I can't see?

#include <Date.au3>

If Not @Compiled Then Opt("trayicondebug", 1)

; Einpacken für Dropbox Freigabe und Löschen nach 10 Tagen

#region Variables and Prerequirements
$VorhalteTage=5
$obj = ""
$zip = ""
$pass = ""
$DropPublic = "C:\Dropbox-4-Attachments\Dropbox\Public\"
$DropURL = "http://dl.dropbox.com/u/<some-dropbox-user-id-goes-here>/"
$BaseDir = "h:\daten\public\Email-Postlagernd"
If StringRight($BaseDir, 1) = "\" Then $BaseDir = StringTrimRight($BaseDir, 1) ; @scriptdir liefert für Drive Roots einen abschliessenden "\", sonst nicht...
$InDir = $BaseDir & "\Drop-In\"
DirCreate($InDir)
$OutDir = $BaseDir & "\URL-out\"
DirCreate($OutDir)
$ZipWorkDir = $BaseDir & "\ZIP-Work\"
DirCreate($OutDir)
$SevenZip = "c:\programme\7-Zip\7z.exe"
$SevenZipDir = "c:\programme\7-Zip"
If Not FileExists($SevenZip) Then
    MsgBox(48, "Fehler", "7-Zip ist nicht installiert, ist aber *ZWINGEND* erforderlich für die Funktion dieses Scriptes!")
    Exit
EndIf
#endregion Variables and Prerequirements
$z = 1
While 1
    ConsoleWrite(".")
    $z += 1
    If $z > 100 Then
        ConsoleWrite(@CRLF)
        $z = 1
    EndIf
    $h = FileFindFirstFile($InDir & "*")
    If @error <> 0 Then ; nix zu tun, warten, dann weiter loopen.
        Sleep(2000)
    Else
        While 1 ; es gibt was, alles abgrasen.
            $Next = FileFindNextFile($h)
            If @error Then ExitLoop
            If Not FileExists($InDir & $Next) Then ContinueLoop ; schon wieder gelöscht: weiter suchen / warten...
            Compress($InDir, $Next)
        WEnd
    EndIf
    CheckFileAge()
WEnd

Func CheckFileAge() ; Dateien in DropPublic löschen, die älter sind als $VorhalteTage
    Local $Used=DirGetSize($DropPublic)
    if FileExists($BaseDir & "\NochFrei=*.txt") then FileDelete($BaseDir & "\NochFrei=*.txt")
    $Used = Round($Used / 1000 / 1000)
    Local $FreeTxt=(2000-$Used) & "MB von 2000MB"
    Local $FreeFile=$BaseDir & "\NochFrei=" & $FreeTxt & ".TXT"
    FileWriteLine($FreeFile,_NowCalc() & ": " & $FreeTxt)
    Local $hPub = FileFindFirstFile($DropPublic & "*")
    If @error <> 0 Then Return
    Local $hNext, $tStamp, $Alter
    While 1
        $hNext = FileFindNextFile($hPub)
        If @error Then Return
        $tStamp = FileGetTime($DropPublic & $hNext, 0, 1) ; ,1) = als yyyyMMddhhmmss und nicht als Array.
        $tStamp = StringLeft($tStamp, 4) & "/" & StringMid($tStamp, 5, 2) & "/" & StringMid($tStamp, 7, 2) & " " & StringMid($tStamp, 9, 2) & ":" & StringMid($tStamp, 11, 2) & ":" & StringRight($tStamp, 2)
        $Alter = _DateDiff("D", $tStamp, _NowCalc())
        if $Alter > $VorhalteTage Then FileDelete ($DropPublic & $hNext)
    WEnd
EndFunc   ;==>CheckFileAge



Func Compress($path, $obj)
    $pass = Password()
    If Not FileExists($path & $obj) Then Return 1 ; ist schon wieder weg... (???)
    If StringInStr(FileGetAttrib($path & $obj), "d") Then ; das ist ein Verzeichnis.
        ConsoleWrite("Verzeichnis erkannt" & @CRLF)
        $zip = StringRegExpReplace($obj, "[^a-zA-Z0-9-_]", "") ; alle non-ascii Zeichen einfach kicken. Das Erstellen der URL wird sonst komplizierter.
        ConsoleWrite($zip & " = zip" & @CRLF)
        $zip1 = $zip & ".ZIP"
        ConsoleWrite($zip1 & " = zip1" & @CRLF)
        $zip2 = TimeStamp() & ".ZIP"
        ConsoleWrite($zip2 & " = zip2" & @CRLF)

        If FileExists($InDir & $zip1) Then FileDelete($InDir & $zip1)
        If FileExists($InDir & $zip2) Then FileDelete($InDir & $zip2)
        $LastSize = 0

        While 1 ; warten, bis nix mehr dazu kommt: Könnte ein Verzeichnis mit viel Inhalt sein
            Sleep(5000)
            $NowSize = DirGetSize($path & $obj)
            If $LastSize = $NowSize Then
                ConsoleWrite("Größe gleich geblieben: " & $NowSize & @CRLF)
                ExitLoop
            Else
                ConsoleWrite("Angewachsen von " & $LastSize & " auf " & $NowSize & @CRLF)
                $LastSize = $NowSize
            EndIf
        WEnd

        $cmd = $SevenZip & ' a "' & $ZipWorkDir & $zip1 & '" "' & $path & $obj & '" -mx=7 -r' ; erst recursiv ohne Verschlüsselung.
        ConsoleWrite($cmd & @CRLF)
        $result = RunWait($cmd, $SevenZipDir, @SW_HIDE)
        $result *= 10
        $cmd = $SevenZip & ' a "' & $ZipWorkDir & $zip2 & '" "' & $ZipWorkDir & $zip1 & '" -p' & $pass & " -mx=0" ; dann verschlüsseln ohne weitere Kompression. (so sieht man den Verzeichnisbaum nicht ohne Passwort.)
        $result += RunWait($cmd, $SevenZipDir, @SW_HIDE)
        If FileExists($ZipWorkDir & $zip1) Then
            ConsoleWrite("71 zip1 ist da: " & $ZipWorkDir & $zip1 & @CRLF)
            FileDelete($ZipWorkDir & $zip1)
        EndIf
        ConsoleWrite($result & " = Result 7z.exe" & @CRLF)
        If $result = 0 Then
            ConsoleWrite("hier 76" & @CRLF)
            Write_DL_Info($obj, $zip2)
            ConsoleWrite($result & " = Result 7z.exe - 77" & @CRLF)
            FileCopy($ZipWorkDir & $zip2, $DropPublic & $zip2, 1)
            ConsoleWrite("copy " & $ZipWorkDir & $zip2 & " " & $DropPublic & $zip2 & @CRLF)
            ; ConsoleWrite(" für filemove: " & $InDir & $obj & @CRLF)
            DirRemove($InDir & $obj, 1)
            ; erst mal im ZIPWorkDir liegen lassen, verwirt die User weniger, denke ich...
            ; FileMove($ZipWorkDir & $zip2, $OutDir & $obj & "_" & $zip2, 1)
        Else
            ConsoleWrite("Fehler: 7z.exe Result = " & $result & @CRLF)
        EndIf
    Else ; kein Verzeichnis, sondern eine Datei
        $File = StringLeft($obj, StringInStr($obj, ".", 0, -1) - 1) ; File Extension abschneiden.
        $zip = StringRegExpReplace($File, "[^a-zA-Z0-9-_]", "") ; alle non-ascii Zeichen einfach kicken. Das Erstellen der URL wird sonst komplizierter.
        $zip &= "_" & TimeStamp() & ".ZIP"
        ConsoleWrite($zip & @CRLF)
        $LastSize = 0
        While 1 ; warten, bis nix mehr dazu kommt: Könnte eine grosse Datei sein...
            Sleep(5000)
            $NowSize = FileGetSize($path & $obj)
            If $LastSize = $NowSize Then
                ExitLoop
            Else
                $LastSize = $NowSize
            EndIf
        WEnd
        ConsoleWrite("Fertig mit warten auf ""Alles Da""" & @CRLF)
        If FileExists($ZipWorkDir & $zip) Then FileDelete($ZipWorkDir & $zip)
        $cmd = $SevenZip & ' a "' & $ZipWorkDir & $zip & '" ' & $path & $obj & " -p" & $pass & " -mx=7"
        ConsoleWrite($cmd & @CRLF)
        $result = RunWait($cmd, $SevenZipDir, @SW_HIDE)
        If $result = 0 Then
            Write_DL_Info($obj, $zip)
            ConsoleWrite($result & " = Result 7z.exe" & @CRLF)
            ConsoleWrite("copy " & $InDir & $zip & " " & $DropPublic & $zip & @CRLF)
            FileCopy($ZipWorkDir & $zip, $DropPublic & $zip, 1)
            FileDelete($path & $obj)
            ; erst mal im ZIPWorkDir liegen lassen, verwirt die User weniger, denke ich...
            ; FileMove($ZipWorkDir & $zip, $OutDir & $zip, 1)
        Else
            ConsoleWrite("Fehler: 7z.exe Result = " & $result & @CRLF) ; 7z.exe Exit Codes s.u.
            ; abfangen, fehlermeldung produzieren. Evtl. per Email?
        EndIf
    EndIf
EndFunc   ;==>Compress

#cs
    7z.exe exit codes:
    0   No error
    1   Warning (Non fatal error(s)). For example, some files were locked by other application during compressing. So they were not compressed.
    2   Fatal error
    7   Command line error
    8   Not enough memory for operation
    255 User stopped the process
#ce


Func Write_DL_Info($OrgName, $ZipName)
    $f = FileOpen($OutDir & $OrgName & ".TXT", 2 + 8) ; überschreiben, pfad anlegen.
    FileWriteLine($f, "URL für den Download des Archives """ & $OrgName & """: " & @CRLF & $DropURL & $ZipName & @CRLF & @CRLF)
    FileWriteLine($f, "Passwort der ZIP Datei: " & $pass)
    FileWriteLine($f, "")
    FileWriteLine($f, "Sollten Sie einen ""Error (404)"" erhalten, warten Sie bitte einige Minuten,")
    FileWriteLine($f, "und versuchen Sie es dann erneut.")
    FileWriteLine($f, "Es kann sein, dass das Bereitstellen Ihres Downloads noch etwas dauert.")
    FileClose($f)
EndFunc   ;==>Write_DL_Info

Func TimeStamp()

    Local $foo = _NowCalc()
    $foo = StringRegExpReplace($foo, "[/ :]", "")
    Return $foo
EndFunc   ;==>TimeStamp


Func Password()
    $Letter = ""
    For $i = 1 To 10
        $Rnd = Random()
        If $Rnd < 0.3 Then
            $Buchstabe = Chr(Random(Asc("0"), Asc("9"), 1))
            $Letter &= $Buchstabe
        ElseIf $Rnd < 0.7 Then
            ;Capitals
            $Buchstabe = Chr(Random(Asc("A"), Asc("Z"), 1))
            $Letter &= $Buchstabe
        Else
            ;Lower case
            $Buchstabe = Chr(Random(Asc("a"), Asc("z"), 1))
            $Letter &= $Buchstabe
        EndIf
    Next
    ConsoleWrite("Pass = " & $Letter & @CRLF)
    Return $Letter
EndFunc   ;==>Password

Anybody, who can see, what I miss?

Regards, Rudi.

trancexx · July 5, 2011

Add FileClose() for $hPub.

whim · July 5, 2011

Hi rudi,

I think you missed closing the "FileFindFirstFile" search handle

While 1
    ConsoleWrite(".")
    $z += 1
    If $z > 100 Then
        ConsoleWrite(@CRLF)
        $z = 1
    EndIf
    $h = FileFindFirstFile($InDir & "*")
    If @error <> 0 Then ; nix zu tun, warten, dann weiter loopen.
        Sleep(2000)
    Else
        While 1 ; es gibt was, alles abgrasen.
            $Next = FileFindNextFile($h)
            If @error Then ExitLoop
            If Not FileExists($InDir & $Next) Then ContinueLoop ; schon wieder gelöscht: weiter suchen / warten...
            Compress($InDir, $Next)
        WEnd
    EndIf
FileClose($h)  ;<<<<<<<<<<<<<<< something like this here ?
    CheckFileAge()
WEnd

cheers,

wim

rudi · July 5, 2011

Hi.

I added the two missing "fileclose()", now the script doesn't consume 8kB every some seconds. Thanks to both of you!

#include <Date.au3>

If Not @Compiled Then Opt("trayicondebug", 1)

; Einpacken für Dropbox Freigabe und Löschen nach x Tagen

#region Variables and Prerequirements
$VorhalteTage=5
$obj = ""
$zip = ""
$pass = ""
$DropPublic = "C:\Dropbox-4-Attachments\Dropbox\Public\"
$DropURL = "http://dl.dropbox.com/u/28375637/"
$BaseDir = "h:\daten\public\Email-Postlagernd"
If StringRight($BaseDir, 1) = "\" Then $BaseDir = StringTrimRight($BaseDir, 1) ; @scriptdir liefert für Drive Roots einen abschliessenden "\", sonst nicht...
$InDir = $BaseDir & "\Drop-In\"
DirCreate($InDir)
$OutDir = $BaseDir & "\URL-out\"
DirCreate($OutDir)
$ZipWorkDir = $BaseDir & "\ZIP-Work\"
DirCreate($OutDir)
$SevenZip = "c:\programme\7-Zip\7z.exe"
$SevenZipDir = "c:\programme\7-Zip"
If Not FileExists($SevenZip) Then
    MsgBox(48, "Fehler", "7-Zip ist nicht installiert, ist aber *ZWINGEND* erforderlich für die Funktion dieses Scriptes!")
    Exit
EndIf
#endregion Variables and Prerequirements
$z = 1
While 1
    ConsoleWrite(".")
    $z += 1
    If $z > 100 Then
        ConsoleWrite(@CRLF)
        $z = 1
    EndIf
    $h = FileFindFirstFile($InDir & "*")
    If @error <> 0 Then ; nix zu tun, warten, dann weiter loopen.
        Sleep(2000)
    Else
        While 1 ; es gibt was, alles abgrasen.
            $Next = FileFindNextFile($h)
            If @error Then ExitLoop
            If Not FileExists($InDir & $Next) Then ContinueLoop ; schon wieder gelöscht: weiter suchen / warten...
            Compress($InDir, $Next)
        WEnd
    EndIf
    FileClose($h)
    CheckFileAge()
WEnd

Func CheckFileAge() ; Dateien in DropPublic löschen, die älter sind als $VorhalteTage
    Local $Used=DirGetSize($DropPublic)
    if FileExists($BaseDir & "\NochFrei=*.txt") then FileDelete($BaseDir & "\NochFrei=*.txt")
    $Used = Round($Used / 1000 / 1000)
    Local $FreeTxt=(2000-$Used) & "MB von 2000MB"
    Local $FreeFile=$BaseDir & "\NochFrei=" & $FreeTxt & ".TXT"
    FileWriteLine($FreeFile,_NowCalc() & ": " & $FreeTxt)
    Local $hPub = FileFindFirstFile($DropPublic & "*")
    If @error <> 0 Then
        FileClose($hPub)
        Return
    EndIf

    Local $hNext, $tStamp, $Alter
    While 1
        $hNext = FileFindNextFile($hPub)
        If @error Then Return
        $tStamp = FileGetTime($DropPublic & $hNext, 0, 1) ; ,1) = als yyyyMMddhhmmss und nicht als Array.
        $tStamp = StringLeft($tStamp, 4) & "/" & StringMid($tStamp, 5, 2) & "/" & StringMid($tStamp, 7, 2) & " " & StringMid($tStamp, 9, 2) & ":" & StringMid($tStamp, 11, 2) & ":" & StringRight($tStamp, 2)
        $Alter = _DateDiff("D", $tStamp, _NowCalc())
        if $Alter > $VorhalteTage Then FileDelete ($DropPublic & $hNext)
    WEnd
    FileClose($hPub)
EndFunc   ;==>CheckFileAge



Func Compress($path, $obj)
    $pass = Password()
    If Not FileExists($path & $obj) Then Return 1 ; ist schon wieder weg... (???)
    If StringInStr(FileGetAttrib($path & $obj), "d") Then ; das ist ein Verzeichnis.
        ConsoleWrite("Verzeichnis erkannt" & @CRLF)
        $zip = StringRegExpReplace($obj, "[^a-zA-Z0-9-_]", "") ; alle non-ascii Zeichen einfach kicken. Das Erstellen der URL wird sonst komplizierter.
        ConsoleWrite($zip & " = zip" & @CRLF)
        $zip1 = $zip & ".ZIP"
        ConsoleWrite($zip1 & " = zip1" & @CRLF)
        $zip2 = TimeStamp() & ".ZIP"
        ConsoleWrite($zip2 & " = zip2" & @CRLF)

        If FileExists($InDir & $zip1) Then FileDelete($InDir & $zip1)
        If FileExists($InDir & $zip2) Then FileDelete($InDir & $zip2)
        $LastSize = 0

        While 1 ; warten, bis nix mehr dazu kommt: Könnte ein Verzeichnis mit viel Inhalt sein
            Sleep(5000)
            $NowSize = DirGetSize($path & $obj)
            If $LastSize = $NowSize Then
                ConsoleWrite("Größe gleich geblieben: " & $NowSize & @CRLF)
                ExitLoop
            Else
                ConsoleWrite("Angewachsen von " & $LastSize & " auf " & $NowSize & @CRLF)
                $LastSize = $NowSize
            EndIf
        WEnd

        $cmd = $SevenZip & ' a "' & $ZipWorkDir & $zip1 & '" "' & $path & $obj & '" -mx=7 -r' ; erst recursiv ohne Verschlüsselung.
        ConsoleWrite($cmd & @CRLF)
        $result = RunWait($cmd, $SevenZipDir, @SW_HIDE)
        $result *= 10
        $cmd = $SevenZip & ' a "' & $ZipWorkDir & $zip2 & '" "' & $ZipWorkDir & $zip1 & '" -p' & $pass & " -mx=0" ; dann verschlüsseln ohne weitere Kompression. (so sieht man den Verzeichnisbaum nicht ohne Passwort.)
        $result += RunWait($cmd, $SevenZipDir, @SW_HIDE)
        If FileExists($ZipWorkDir & $zip1) Then
            ConsoleWrite("71 zip1 ist da: " & $ZipWorkDir & $zip1 & @CRLF)
            FileDelete($ZipWorkDir & $zip1)
        EndIf
        ConsoleWrite($result & " = Result 7z.exe" & @CRLF)
        If $result = 0 Then
            ConsoleWrite("hier 76" & @CRLF)
            Write_DL_Info($obj, $zip2)
            ConsoleWrite($result & " = Result 7z.exe - 77" & @CRLF)
            FileCopy($ZipWorkDir & $zip2, $DropPublic & $zip2, 1)
            ConsoleWrite("copy " & $ZipWorkDir & $zip2 & " " & $DropPublic & $zip2 & @CRLF)
            ; ConsoleWrite(" für filemove: " & $InDir & $obj & @CRLF)
            DirRemove($InDir & $obj, 1)
            ; erst mal im ZIPWorkDir liegen lassen, verwirt die User weniger, denke ich...
            ; FileMove($ZipWorkDir & $zip2, $OutDir & $obj & "_" & $zip2, 1)
        Else
            ConsoleWrite("Fehler: 7z.exe Result = " & $result & @CRLF)
        EndIf
    Else ; kein Verzeichnis, sondern eine Datei
        $File = StringLeft($obj, StringInStr($obj, ".", 0, -1) - 1) ; File Extension abschneiden.
        $zip = StringRegExpReplace($File, "[^a-zA-Z0-9-_]", "") ; alle non-ascii Zeichen einfach kicken. Das Erstellen der URL wird sonst komplizierter.
        $zip &= "_" & TimeStamp() & ".ZIP"
        ConsoleWrite($zip & @CRLF)
        $LastSize = 0
        While 1 ; warten, bis nix mehr dazu kommt: Könnte eine grosse Datei sein...
            Sleep(5000)
            $NowSize = FileGetSize($path & $obj)
            If $LastSize = $NowSize Then
                ExitLoop
            Else
                $LastSize = $NowSize
            EndIf
        WEnd
        ConsoleWrite("Fertig mit warten auf ""Alles Da""" & @CRLF)
        If FileExists($ZipWorkDir & $zip) Then FileDelete($ZipWorkDir & $zip)
        $cmd = $SevenZip & ' a "' & $ZipWorkDir & $zip & '" ' & $path & $obj & " -p" & $pass & " -mx=7"
        ConsoleWrite($cmd & @CRLF)
        $result = RunWait($cmd, $SevenZipDir, @SW_HIDE)
        If $result = 0 Then
            Write_DL_Info($obj, $zip)
            ConsoleWrite($result & " = Result 7z.exe" & @CRLF)
            ConsoleWrite("copy " & $InDir & $zip & " " & $DropPublic & $zip & @CRLF)
            FileCopy($ZipWorkDir & $zip, $DropPublic & $zip, 1)
            FileDelete($path & $obj)
            ; erst mal im ZIPWorkDir liegen lassen, verwirt die User weniger, denke ich...
            ; FileMove($ZipWorkDir & $zip, $OutDir & $zip, 1)
        Else
            ConsoleWrite("Fehler: 7z.exe Result = " & $result & @CRLF) ; 7z.exe Exit Codes s.u.
            ; abfangen, fehlermeldung produzieren. Evtl. per Email?
        EndIf
    EndIf
EndFunc   ;==>Compress

#cs
    7z.exe exit codes:
    0   No error
    1   Warning (Non fatal error(s)). For example, some files were locked by other application during compressing. So they were not compressed.
    2   Fatal error
    7   Command line error
    8   Not enough memory for operation
    255 User stopped the process
#ce


Func Write_DL_Info($OrgName, $ZipName)
    $f = FileOpen($OutDir & $OrgName & ".TXT", 2 + 8) ; überschreiben, pfad anlegen.
    FileWriteLine($f, "URL für den Download des Archives """ & $OrgName & """: " & @CRLF & $DropURL & $ZipName & @CRLF & @CRLF)
    FileWriteLine($f, "Passwort der ZIP Datei: " & $pass)
    FileWriteLine($f, "")
    FileWriteLine($f, "Sollten Sie einen ""Error (404)"" erhalten, warten Sie bitte einige Minuten,")
    FileWriteLine($f, "und versuchen Sie es dann erneut.")
    FileWriteLine($f, "Es kann sein, dass das Bereitstellen Ihres Downloads noch etwas dauert.")
    FileClose($f)
EndFunc   ;==>Write_DL_Info

Func TimeStamp()

    Local $foo = _NowCalc()
    $foo = StringRegExpReplace($foo, "[/ :]", "")
    Return $foo
EndFunc   ;==>TimeStamp


Func Password()
    $Letter = ""
    For $i = 1 To 10
        $Rnd = Random()
        If $Rnd < 0.3 Then
            $Buchstabe = Chr(Random(Asc("0"), Asc("9"), 1))
            $Letter &= $Buchstabe
        ElseIf $Rnd < 0.7 Then
            ;Capitals
            $Buchstabe = Chr(Random(Asc("A"), Asc("Z"), 1))
            $Letter &= $Buchstabe
        Else
            ;Lower case
            $Buchstabe = Chr(Random(Asc("a"), Asc("z"), 1))
            $Letter &= $Buchstabe
        EndIf
    Next
    ConsoleWrite("Pass = " & $Letter & @CRLF)
    Return $Letter
EndFunc   ;==>Password

Regards, Rudi.

Edited July 5, 2011 by rudi

Sign In

My script is eating all the RAM away.

Recommended Posts

rudi

Link to comment

Share on other sites

trancexx

Link to comment

Share on other sites

whim

Link to comment

Share on other sites

rudi

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members

Browse

AutoIt Resources

Release

Beta