[SOLVED] Run scripts in sequence

[water]

Our helpdesk system imports data from our HR system every night and starts a compiled AutoIt script for every new record found. Unfortunately it doesn't do a "RunWait" but just a "Run" so many instances of the script might be started at the same time.

As the AutoIt script does extensive logging I get a nearly unreadable log when many scripts write to the log at the same time.

My question is:

Is there an easy way to make sure that only one instance of all started scripts is processing at any time. All other scripts should wait until the active script has finished. Then another one of the scripts waiting should start processing.

The helpdesk system runs as a service on a server so the solution can not rely on any GUI function.

Risk of a deadlock should be 0%

Edited January 20, 2012 by water

[JohnOne]

Could the compiled AutoIt script not first look for an instance of its namesake existing

and wait for it to close before continuing?

Assuming of course you have rights to amend it.

[smartee]

yup, lookup _Singleton() in the help file, its there for just this sort of thing providing the scripts being run are yours, plugging in an implementation should be trivial.

Edit: Spelling

Edited January 20, 2012 by smartee

[water]

I tried before but it doesn't work. The first instance runs but when it ends none of the other instances starts.

To recreate compile this script:

#include <Misc.au3>
#include <File.au3>

While 1
    If _Singleton("NeuerMA", 1) = 0 Then
        _FileWriteLog("C:temptest.log", "Sleep: " & @AutoItPID)
        Sleep(2000)
    Else
        _FileWriteLog("C:temptest.log", "Got mutex: " & @AutoItPID)
        ExitLoop
    EndIf
WEnd
_FileWriteLog("C:temptest.log", "Start: " & @AutoItPID)
Sleep(2000)
_FileWriteLog("C:temptest.log", "End: " & @AutoItPID)
exit

and then run this to create 20 instances. 1 starts and ends the remaining 19 hang.

For $i = 1 To 20
    Run("test.exe")
    Sleep(50)
Next

I'm running Windows 7 64 bit. Edited January 20, 2012 by water

[JohnOne]

#include <Misc.au3>
#include <File.au3>
While ProcessExists(@ScriptDir & 'test.exe')
    Sleep(10)
WEnd
_FileWriteLog(@ScriptDir & "test.log", "Start: " & @AutoItPID)
Sleep(2000)
_FileWriteLog(@ScriptDir & "test.log", "End: " & @AutoItPID)
exit

[water]

Doesn't work either.

As ProcessExists returns the PID of the first started process all waiting scripts start at once as soon as the first script ends.

[JohnOne]

It is a poser.

I tried a _FileInUse Func which made no difference either

expandcollapse popup

;#cs
#include <Misc.au3>
#include <File.au3>
$Sleep = 50
While ProcessExists(@ScriptDir & 'scriptA.exe') Or _FileInUse(@ScriptDir & 'scriptA.exe')
    Sleep($Sleep)
WEnd
_FileWriteLog(@ScriptDir & "test.log", "Start: " & @AutoItPID)
Sleep(2000)
_FileWriteLog(@ScriptDir & "test.log", "End: " & @AutoItPID)
exit
;#ce
#cs
For $i = 1 To 20
Run("ScriptA.exe")
   Sleep(40)
Next
#ce
;===============================================================================
; http://www.autoitscript.com/forum/topic/125469-create-table-of-files-in-use/page__view__findpost__p__870939
; Function Name:    _FileInUse()
; Description:    Checks if file is in use
; Parameter(s):  $sFilename = File name
; Return Value(s):  1 - file in use (@error contains system error code)
;                  0 - file not in use
;===============================================================================
Func _FileInUse($sFilename)
    Local $aRet, $hFile
    $aRet = DllCall("Kernel32.dll", "hwnd", "CreateFile", _
                                    "str", $sFilename, _ ;lpFileName
                                    "dword", 0x80000000, _ ;dwDesiredAccess = GENERIC_READ
                                    "dword", 0, _ ;dwShareMode = DO NOT SHARE
                                    "dword", 0, _ ;lpSecurityAttributes = NULL
                                    "dword", 3, _ ;dwCreationDisposition = OPEN_EXISTING
                                    "dword", 128, _ ;dwFlagsAndAttributes = FILE_ATTRIBUTE_NORMAL
                                    "hwnd", 0) ;hTemplateFile = NULL
    $hFile = $aRet[0]
    If $hFile = -1 Then ;INVALID_HANDLE_VALUE = -1
        $aRet = DllCall("Kernel32.dll", "int", "GetLastError")
        SetError($aRet[0])
        Return 1
    Else
        ;close file handle
        DllCall("Kernel32.dll", "int", "CloseHandle", "hwnd", $hFile)
        Return 0
    EndIf
EndFunc

[Melba23]

water,

I would use ProcessList to find out where each instance is in the list. As the running processes are returned in the order in which they started it is easy to find out which one is just ahead and wait until it stops:

#include <File.au3>

; Get initial process list to see if other instances are running
$aProcess_List = ProcessList("test.exe")
For $i = 1 To $aProcess_List[0][0]
    If $aProcess_List[$i][1] = @AutoItPID Then
        $iIndex = $i
        ExitLoop
    EndIf
Next

; If instance is not first
If $iIndex > 1 Then
    $iPrevious_Instance_PID = $aProcess_List[$iIndex - 1][1]
    ; Wait until the previous instance ends
    While ProcessExists($iPrevious_Instance_PID)
        Sleep(10)
    WEnd
EndIf

; Write to log file
_FileWriteLog(@ScriptDir & "test.log", "Start: " & @AutoItPID & " - Instance: " & $iIndex)
Sleep(2000)
_FileWriteLog(@ScriptDir & "test.log", "End: " & @AutoItPID & " - Instance: " & $iIndex)

Exit

Works fine for me when I test it.

M23

[water]

Hi Melba,

thanks a lot for your assistance!

Your scripts works perfect!

Do I somehow misunderstand how _Singleton works? Or is it a bug?

[Melba23]

water,

I have always found _Singleton very iffy (British technical term for "does not work very well" ) and so do not have a great deal of experience with it. However, I do not believe that you have found a bug.

In this case I do not think it would work as I will now try to explain: The first instance starts, the second instance starts a little later and finds the first running - that should be no problem. But the second instance can only find out when the first ends by polling the _Singleton function in a loop. What happens if a third instance starts before the first ends? I presume that the function will still say that there is an instance running and so the second instance will never run.

Makes sense to me - I hope it does to you.

M23

[water]

Melba,

let me try to explain how I understand _Singleton.

Instance 1 creates the mutex and the function returns the handle. All other instances get 0 from the function because the mutex already exists. When instance 1 ends the mutex is released. The next instance calling the function should recreate the mutex and return the handle. All other instances still get 0 and wait.

But as the tests show this doesn't happen. Even when instance 1 ends all other instances still wait.

[Melba23]

water,

Like I said: "iffy"!

M23

[AdamUL]

Water, try this at the top of your script. It usually works for me in situation.

Global $sProgramName = "Running " & StringTrimRight(@ScriptName, 4)
If WinExists($sProgramName) Then Exit;
AutoItWinSetTitle($sProgramName)

Adam

[water]

Thanks for the suggestion. But as the script is being run by a service on our server I can't use the window related functions.

The solution Melba posted is working just fine and I will implement his code.

[smartee]

@M23: _Singleton is indeed iffy (now you will know why ) and as such, it might not be as useful in this situation (waiting) as I suggested. That's a very creative solution however

@water: Sorry for the late response.

To future run in sequence searchers, and _Singleton's confused,

How it works

Here's how it works, it creates a mutex, however if the named mutex already exists, it gets a handle to it (that can be used with WaitForSingleObject) but it does nothing with this handle, it instead returns 0 and sets @error to ERROR_ALREADY_EXISTS. Great for a one time check at the beginning of a script, but it essentially kills its own post-run functionality by leaving the handle open.

Can I wait by looping it?

Umm No, not a good idea, you just create many more handles to the mutex while it exists, so even when the original thread terminates and that handle to the mutex is freed, _Singleton would still return 0 (and set @error to ERROR_ALREADY_EXISTS)

The solution?

When the mutex exists, close the handle you get to it by trying to create it, before returning 0 and setting @error.

Heres how:

Copy the _Singleton function from the "Misc.au3" include-file into your script then near the end where you see

If $lastError[0] = $ERROR_ALREADY_EXISTS Then

Simply drop in these two lines

DllCall("kernel32.dll", "bool", "CloseHandle", "handle", $handle[0])
If @error Then Return SetError(@error, @extended, 0)

so now overall your function should look like

expandcollapse popup

; #FUNCTION# ====================================================================================================================
; Name...........: _Singleton
; Description ...: Enforce a design paradigm where only one instance of the script may be running.
; Syntax.........: _Singleton($sOccurenceName[, $iFlag = 0])
; Parameters ....: $sOccurenceName - String to identify the occurrence of the script.  This string may not contain the  character unless you are placing the object in a namespace (See Remarks).
;                 $iFlag          - Behavior options.
;                 |0 - Exit the script with the exit code -1 if another instance already exists.
;                 |1 - Return from the function without exiting the script.
;                 |2 - Allow the object to be accessed by anybody in the system. This is useful if specifying a "Global" object in a multi-user environment.
; Return values .: Success    - The handle to the object used for synchronization (a mutex).
;                 Failure     - 0
; Author ........: Valik
; Modified.......:
; Remarks .......: You can place the object in a namespace by prefixing your object name with either "Global" or "Local".  "Global" objects combined with the flag 2 are useful in multi-user environments.
; Related .......:
; Link ..........:
; Example .......: Yes
; ===============================================================================================================================
Func _Singleton($sOccurenceName, $iFlag = 0)
Local Const $ERROR_ALREADY_EXISTS = 183
Local Const $SECURITY_DESCRIPTOR_REVISION = 1
Local $tSecurityAttributes = 0
If BitAND($iFlag, 2) Then
  ; The size of SECURITY_DESCRIPTOR is 20 bytes.  We just
  ; need a block of memory the right size, we aren't going to
  ; access any members directly so it's not important what
  ; the members are, just that the total size is correct.
  Local $tSecurityDescriptor = DllStructCreate("byte;byte;word;ptr[4]")
  ; Initialize the security descriptor.
  Local $aRet = DllCall("advapi32.dll", "bool", "InitializeSecurityDescriptor", _
    "struct*", $tSecurityDescriptor, "dword", $SECURITY_DESCRIPTOR_REVISION)
  If @error Then Return SetError(@error, @extended, 0)
  If $aRet[0] Then
   ; Add the NULL DACL specifying access to everybody.
   $aRet = DllCall("advapi32.dll", "bool", "SetSecurityDescriptorDacl", _
     "struct*", $tSecurityDescriptor, "bool", 1, "ptr", 0, "bool", 0)
   If @error Then Return SetError(@error, @extended, 0)
   If $aRet[0] Then
    ; Create a SECURITY_ATTRIBUTES structure.
    $tSecurityAttributes = DllStructCreate($tagSECURITY_ATTRIBUTES)
    ; Assign the members.
    DllStructSetData($tSecurityAttributes, 1, DllStructGetSize($tSecurityAttributes))
    DllStructSetData($tSecurityAttributes, 2, DllStructGetPtr($tSecurityDescriptor))
    DllStructSetData($tSecurityAttributes, 3, 0)
   EndIf
  EndIf
EndIf
Local $handle = DllCall("kernel32.dll", "handle", "CreateMutexW", "struct*", $tSecurityAttributes, "bool", 1, "wstr", $sOccurenceName)
If @error Then Return SetError(@error, @extended, 0)
Local $lastError = DllCall("kernel32.dll", "dword", "GetLastError")
If @error Then Return SetError(@error, @extended, 0)
If $lastError[0] = $ERROR_ALREADY_EXISTS Then
  DllCall("kernel32.dll", "bool", "CloseHandle", "handle", $handle[0])
  If @error Then Return SetError(@error, @extended, 0)
  If BitAND($iFlag, 1) Then
   Return SetError($lastError[0], $lastError[0], 0)
  Else
   Exit -1
  EndIf
EndIf
Return $handle[0]
EndFunc   ;==>_Singleton

There you go, now it works as expected

Alternatively you can make this change directly in Misc.au3.

Now you can loop to wait etc and all is well.

Example:

expandcollapse popup

#include <Date.au3>

; Create a unique string that will be used to identify this script
Local $sScriptId = "ImportScript"

; Just for fun logging purposes
Local $sTimeStamp = _NowCalc()

$hMutex = _Singleton($sScriptId, 1)
While $hMutex = 0
    Sleep(250)
    $hMutex = _Singleton($sScriptId, 1)
WEnd

; Simulate some tasks that take a variable time to complete and writes to a file
Local $hFile = FileOpen(@ScriptDir & "testing.txt", 1)
FileWriteLine($hFile, "<!-- " & $sScriptId & " PID: " & @AutoItPID & " Started: " & $sTimeStamp & " Waited: " & _DateDiff('s', $sTimeStamp, _NowCalc()) & "s -->")
Local $iTasks = Random(3, 5, 1)
For $i = 1 To $iTasks
    FileWriteLine($hFile, _NowCalc() & ": Writing line " & $i & ", PID: " & @AutoItPID)
    Sleep(Random(500, 1000, 1))
Next
FileClose($hFile)

; #FUNCTION# ====================================================================================================================
; Name...........: _Singleton
; Description ...: Enforce a design paradigm where only one instance of the script may be running.
; Syntax.........: _Singleton($sOccurenceName[, $iFlag = 0])
; Parameters ....: $sOccurenceName - String to identify the occurrence of the script.  This string may not contain the  character unless you are placing the object in a namespace (See Remarks).
;                  $iFlag          - Behavior options.
;                  |0 - Exit the script with the exit code -1 if another instance already exists.
;                  |1 - Return from the function without exiting the script.
;                  |2 - Allow the object to be accessed by anybody in the system. This is useful if specifying a "Global" object in a multi-user environment.
; Return values .: Success      - The handle to the object used for synchronization (a mutex).
;                  Failure      - 0
; Author ........: Valik
; Modified.......:
; Remarks .......: You can place the object in a namespace by prefixing your object name with either "Global" or "Local".  "Global" objects combined with the flag 2 are useful in multi-user environments.
; Related .......:
; Link ..........:
; Example .......: Yes
; ===============================================================================================================================
Func _Singleton($sOccurenceName, $iFlag = 0)
    Local Const $ERROR_ALREADY_EXISTS = 183
    Local Const $SECURITY_DESCRIPTOR_REVISION = 1
    Local $tSecurityAttributes = 0

    If BitAND($iFlag, 2) Then
        ; The size of SECURITY_DESCRIPTOR is 20 bytes.  We just
        ; need a block of memory the right size, we aren't going to
        ; access any members directly so it's not important what
        ; the members are, just that the total size is correct.
        Local $tSecurityDescriptor = DllStructCreate("byte;byte;word;ptr[4]")
        ; Initialize the security descriptor.
        Local $aRet = DllCall("advapi32.dll", "bool", "InitializeSecurityDescriptor", _
                "struct*", $tSecurityDescriptor, "dword", $SECURITY_DESCRIPTOR_REVISION)
        If @error Then Return SetError(@error, @extended, 0)
        If $aRet[0] Then
            ; Add the NULL DACL specifying access to everybody.
            $aRet = DllCall("advapi32.dll", "bool", "SetSecurityDescriptorDacl", _
                    "struct*", $tSecurityDescriptor, "bool", 1, "ptr", 0, "bool", 0)
            If @error Then Return SetError(@error, @extended, 0)
            If $aRet[0] Then
                ; Create a SECURITY_ATTRIBUTES structure.
                $tSecurityAttributes = DllStructCreate($tagSECURITY_ATTRIBUTES)
                ; Assign the members.
                DllStructSetData($tSecurityAttributes, 1, DllStructGetSize($tSecurityAttributes))
                DllStructSetData($tSecurityAttributes, 2, DllStructGetPtr($tSecurityDescriptor))
                DllStructSetData($tSecurityAttributes, 3, 0)
            EndIf
        EndIf
    EndIf

    Local $handle = DllCall("kernel32.dll", "handle", "CreateMutexW", "struct*", $tSecurityAttributes, "bool", 1, "wstr", $sOccurenceName)
    If @error Then Return SetError(@error, @extended, 0)
    Local $lastError = DllCall("kernel32.dll", "dword", "GetLastError")
    If @error Then Return SetError(@error, @extended, 0)
    If $lastError[0] = $ERROR_ALREADY_EXISTS Then
        DllCall("kernel32.dll", "bool", "CloseHandle", "handle", $handle[0])
        If @error Then Return SetError(@error, @extended, 0)
        If BitAND($iFlag, 1) Then
            Return SetError($lastError[0], $lastError[0], 0)
        Else
            Exit -1
        EndIf
    EndIf
    Return $handle[0]
EndFunc   ;==>_Singleton

Missed the bus with you water , but hopefully this'll be helpful to someone.

Maybe this is a bugfix for the bug (unclosed handle) water suspected. I'll leave that to the greats though

Regards,

smartee

[Melba23]

smartee,

Thanks for that explanation.

Are you intending to open a Trac ticket with your suggested change? I think it would be a good idea to do so.

M23

[water]

Smartee,

thanks for your reply and taking the time to explain what happens in such great detail.

I nearly had the solution you now propose found by accident But I inserted the close handle DLLCall at the end of the script because I got the impression that the mutex wasn't freed (which would lead to a small memory leak if the script was ended but the mutex still existed). Luckily Melba showed up with a working solution so I didn't need to do more research.

I think it would be a good idea to create a Trac ticket so AutoIt can be enhanced

[kylomas]

smartee,

I believe that you only want the owning process to be able to close the file handle but do NOT know how to determine the owner.

Was working on this last night before a party got in the way. Trying to get a rudimentary file locking mechanism in place such that processes "a", "b" and "c" serilaize on a resource ("myfilelock" in this case).

Unless I am completely misunderstanding this, releasing the mutex handle everytime it exists is the same as not using the function at all. Is this correct?

kylomas

edit: correction - mutex handle, not file handle

Edited January 21, 2012 by kylomas

[smartee]

ok so guys I created a Trac Ticket #2108

@kylomas, no no, CloseHandle as used above does not release the mutex it just closes the newly acquired handle to it that you get whenever you try to create a mutex that already exists by using the CreateMutex function (in _Singleton).

We close this new handle so when the original mutex creating process closes, and its handle to the mutex automatically closed, being the only handle to the mutex, the mutex object will be destroyed.

Hope this clears it up for you

[kylomas]

smartee,

and previous scripts were hanging because there was always an open handle to the mutex..I think I see it now..

Thanks,

kylomas