Howto avoid parallel I/O conflicts, setSerial ?

[EdDyreen]

'

Hi, I started my question on a different forum and I know a solution unfortunately it only works in DOS

http://www.dostips.com/forum/viewtopic.php?f=3&t=3107&start=0

@echo off &setlocal enableDelayedExpansion

<:return_ <nul echo. &call :setSerial_ fileLock return_ &goto :proceed_
echo.setSerial ok
pause
exit /b

:proceed_
echo.setParallel ok
pause
exit

:setSerial_ ( fileLock, return )
:: (
   2>nul (
      3>"%~1" (
         call :%~2
         exit /b
      ) || (
         >nul 2>&1 ping 0.0.0.0
         goto :setSerial_ "()"
      )
   )
:: )

I'm looking for a similar solution in autoIT, I tried using fileOpen but I get two different handles ?!

Local $file = FileOpen("test.txt", 10) ; which is similar to 2 + 8 (erase + create dir)
If $file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf
MsgBox ( 0, '', 'The file is open for writing')
FileClose($file)
exit

So basically what I'm trying to do is to make sure that different processes or multiple instances of the same process don't run parallel when it's important like when they access the same registry keys/files remember them or not, restore them later or not...

Just click the hyperlink for a more detail explanation of the problem and solution.

Edited March 24, 2012 by EdDyreen

[water]

If all the possible processes are AutoIt scripts you can use function _Singleton to make sure that only one process runs at a time.

[EdDyreen]

'

Can you show me how to implement it, so that two instances of the same or different processes will patiently wait for each other for the file to become available ?

setSerial ()
MsgBox ( 0, '', 'The process is serial')
exit

func setSerial ( $fileLock = 'dummy.TMP' )
   Local $file
   while not $file = FileOpen ( $fileLock, 10 )
      sleep ( 1000 )
   wend
endfunc

Edited March 24, 2012 by EdDyreen

[water]

func setSerial($function, $fileLock = 'dummy.TMP')
    If function = 1 Then
        While FileExists($fileLock) ; Wait until the lock file created by another process has been deleted
           Sleep(1000)
        WEnd
        FileWrite($fileLock) ; Create the lock file
    Else
        FileDelete($fileLock); Delete the lock file
    EndIf
    If @error Then ... ; Error handling if something went wrong while creating/deleting the lock file
endfunc

Call function serialize with parameter 1 set to 1 to create the lock file and with parameter 1 set to anything else to delete the lock file and let another process start to run.

[EdDyreen]

'

This is what I used before, the downside is that it's overly complex and that it stalls if CPU is used intensively.

expandcollapse popup

; ----------------------------------------------------------------------------------------------------------------------------------
Func Chrono_Set_SCRIPT ()
; ----------------------------------------------------------------------------------------------------------------------------------
;  last updated    : 16/03/2012
;
; Run simultanious SCRIPT instances in chronological order .
;
;Debug_On_ $Dfunc
; (
   Debug_ ( 'valid ' &$Dfunc &' ( ' &$Dargs &' )' )
   ; (
  Local $debug = False
   ; )
   Debug_ ( 'perform ' &$Dfunc &' ( ' &$Dargs &' )', 'pause' )
   ; (
  RegWrite ( 'HKLMSoftwareEDAU3', 'Crono_Report_' &@AutoItPID, 'REG_SZ', 'Waiting' )
  Local $AllReady   = False
  Local $ILock  = False
  Local $Locking_Detect= False
  ;
  SleepTiming_ReSet ()
  While 1
   ;
   SleepTiming_Go ()
   Wait_Is_IDLE ()
   Debug_ ( "Get All PID's of all other instances of me &store under $Array_PID", 0, 1 )
   ; (
    Local $Array_LST   = ProcessList ( @ScriptName )
    Local $Array_PID[1]
    Local $LowestPID   = @AutoItPID
    Local $Count    = 0
    ;
    For $Element = 1 To $Array_LST[0][0]
     ;
     $PID = $Array_LST[$Element][1]
     ;
     If $PID <> '' And $PID <> @AutoItPID Then
      ;
      $Count += 1
      $Array_PID[0] = $Count
      ReDim $Array_PID[ $Count + 1 ]
      $Array_PID[$Count] = $PID
      ;
     EndIf
     ;
    Next
    Local $Array_debug_[5] = [                   _
     ''                       ,  _
     "Get All PID's of all other instances of me &store under $Array_PID" &@CRLF ,  _
     ''                    &@CRLF ,  _
     '@AutoItPID: '   &@AutoItPID            &@CRLF ,  _
     '$Array_PID[0]: '  &$Array_PID[0]          &@CRLF &  _
     ''                         _
    ]
    ;
    If $Count > 0 Then
     ;
     ReDim $Array_debug_[6]
     $Array_debug_[5] =         _
     ''             &  _
     '$Array_PID[1]: '  &$Array_PID[1]  &@CRLF &  _
     ''
     ;
    EndIf
    If $Count > 1 Then
     ;
     ReDim $Array_debug_[7]
     $Array_debug_[6] =         _
     ''             &  _
     '$Array_PID[2]: '  &$Array_PID[2]  &@CRLF &  _
     ''
     ;
    EndIf
    If $debug Then
     ;
     Select
      ;
      Case $Count < 1
       ;
       msgBox_ ( $Array_debug_[0] &$Array_debug_[1] &$Array_debug_[2] &$Array_debug_[3] &$Array_debug_[4] )
       ;
      Case $Count < 2
       ;
       msgBox_ ( $Array_debug_[0] &$Array_debug_[1] &$Array_debug_[2] &$Array_debug_[3] &$Array_debug_[4] &$Array_debug_[5] )
       ;
      Case $Count >= 2
       ;
       msgBox_ ( $Array_debug_[0] &$Array_debug_[1] &$Array_debug_[2] &$Array_debug_[3] &$Array_debug_[4] &$Array_debug_[5] &$Array_debug_[6] )
       ;
     EndSelect
     ;
    EndIf
   ; )
  Debug_ ( 'Get $AllReady, $Locking_Detect &$LowestPID', 0, 1 )
  ; (
   $AllReady   = True
   $Locking_Detect= False
   $Locked_Detect = False
   ;
   For $Element = 1 To $Array_PID[0]
    ;
    $PID    = $Array_PID[$Element]
    $ReadState  = RegRead ( 'HKLMSoftwareEDAU3', 'Crono_Report_' &$PID )
    ;
    Select
     ;
     Case $ReadState = 'Locking'
      ;
      $Locking_Detect = True
      ExitLoop
      ;
     Case $ReadState = 'Locked'
      ;
      $Locked_Detect = True
      ExitLoop
      ;
     Case $ReadState = 'Waiting' Or $ReadState = 'Freezed' Or $ReadState = 'Detected'
      ;
      If $PID < $LowestPID Then $LowestPID = $PID
      ;
     Case $ReadState = ''
      ;
      $AllReady = False
      ExitLoop
      ;
    EndSelect
    ;
   Next
   If $debug Then msgBox_ (           _
    ''                &  _
    'Get $AllReady, $Locking_Detect &$LowestPID' &@CRLF &  _
    ''             &@CRLF &  _
    '@AutoItPID: '   &@AutoItPID     &@CRLF &  _
    '$AllReady: '   &$AllReady    &@CRLF &  _
    '$Locking_Detect: '  &$Locking_Detect   &@CRLF &  _
    '$Locked_Detect: '  &$Locked_Detect   &@CRLF &  _
    '$LowestPID: '   &$LowestPID          _
   )
  ; )
  Debug_ ( 'Write my status to registry', 0, 1 )
  ; (
   If $AllReady Then
    ;
    Select
     ;
     Case $Locked_Detect
      ;
      RegWrite ( 'HKLMSoftwareEDAU3', 'Crono_Report_' &@AutoItPID, 'REG_SZ', 'Freezed' )
      ;
     Case $Locking_Detect
      ;
      RegWrite ( 'HKLMSoftwareEDAU3', 'Crono_Report_' &@AutoItPID, 'REG_SZ', 'Detected' )
      ;
     Case $LowestPID = @AutoItPID
      ;
      RegWrite ( 'HKLMSoftwareEDAU3', 'Crono_Report_' &@AutoItPID, 'REG_SZ', 'Locking' )
      $ILock = True
      ;
     Case Else
      ;
      RegWrite ( 'HKLMSoftwareEDAU3', 'Crono_Report_' &@AutoItPID, 'REG_SZ', 'Waiting' )
      $ILock = False
      ;
    EndSelect
    ;
   EndIf
   If $debug Then msgBox_ (         _
    ''              &  _
    'Write my status to registry'   &@CRLF &  _
    ''           &@CRLF &  _
    '@AutoItPID: '   &@AutoItPID   &@CRLF &  _
    '$AllReady: '   &$AllReady  &@CRLF &  _
    '$Locking_Detect: '  &$Locking_Detect &@CRLF &  _
    '$Locked_Detect: '  &$Locked_Detect &@CRLF &  _
    '$LowestPID: '   &$LowestPID   &@CRLF &  _
    '$ILock: '   &$ILock          _
   )
  ; )
  Debug_ ( 'If everyone ready &I am the lowest Locking then lock me', 0, 1 )
  ; (
   $AllReady = True
   ;
   If $ILock Then
    ;
    For $Element = 1 To $Array_PID[0]
     ;
     $PID = $Array_PID[$Element]
     $ReadState = RegRead ( 'HKLMSoftwareEDAU3', 'Crono_Report_' &$PID )
     If $ReadState <> 'Detected' Then
      ;
      $AllReady = False
      If $ReadState = 'Locking' Then
       ;
       If $PID < $LowestPID Then $LowestPID = $PID
       If $LowestPID <> @AutoItPID Then
        $ILock = False
        ExitLoop
       EndIf
       ;
      EndIf
      ;
     EndIf
     ;
    Next
    ;
   EndIf
   If $debug Then msgBox_ (               _
    ''                    &  _
    'If everyone ready &I am the lowest Locking then lock me'&@CRLF &  _
    ''                 &@CRLF &  _
    '@AutoItPID: '   &@AutoItPID         &@CRLF &  _
    '$AllReady: '   &$AllReady        &@CRLF &  _
    '$Locking_Detect: '  &$Locking_Detect      &@CRLF &  _
    '$LowestPID: '   &$LowestPID         &@CRLF &  _
    '$ILock: '   &$ILock                _
   )
  ; )
  Debug_ ( 'If everyone ready &I am Locked, then freeze the others, exit this loop &start running', 0, 1 )
  ; (
   If $AllReady And $ILock Then
    ;
    $msg = Debug_ ( ' Entering chronological mode', 0, 1 )
    RegWrite ( 'HKLMSoftwareEDAU3', 'Crono_Report_' &@AutoItPID, 'REG_SZ', 'Locked' )
    ;
    For $Element = 1 To $Array_PID[0]
     ;
     $PID = $Array_PID[$Element]
     RegWrite ( 'HKLMSoftwareEDAU3', 'Crono_Report_' &$PID, 'REG_SZ', 'Freezed' )
     ;
    Next
    ;
    $msg = Debug_ ( ' Entering chronological mode', 0, 1 )
    ExitLoop
    ;
   EndIf
  ; )
WEnd
If $debug Then msgBox_ (    _
  ''        &  _
  'Running'   &@CRLF &  _
  ''     &@CRLF &  _
  '@AutoItPID: ' &@AutoItPID  _
)
;
EndFunc
; )
; ----------------------------------------------------------------------------------------------------------------------------------

Thanks water !

I will have to test it extensively using many instances of the process for prolonged periods of time.

It just looks too simple, to be bulletproof, I hope it will work perfectly, otherwise I'll be back,

Ed

Edited March 24, 2012 by EdDyreen

[water]

Another approach would be to use a modified 8and therefore non-standard) version of function _Singleton.

If you have any problems I will be glad to send you this modified version.

I use it to serialize the execution of a script which is mass started by another application.

[EdDyreen]

'

Oops, I investigated your code by the eye and I already foresee a glitch.

In theory it's possible that two different process fall through the loop simultaniously and that they will both FileWrite.

While FileExists($fileLock) ; Wait until the lock file created by another process has been deleted
           Sleep(1000)
        WEnd
        FileWrite($fileLock) ; Create the lock file

If that happens they would be still running parallel, the chances are slim but it can happen and I can't risk it Edited March 24, 2012 by EdDyreen

[EdDyreen]

Another approach would be to use a modified 8and therefore non-standard) version of function _Singleton.

If you have any problems I will be glad to send you this modified version.

I use it to serialize the execution of a script which is mass started by another application.

'

Yesyes please, look my previous post it has to be bulletproof, no matter how much code it takes.

I assume it will always be better than my tampering.

Edited March 24, 2012 by EdDyreen

[water]

I will post my code as soon as I return to my office on monday.

[kylomas]

Ed,

Look at the WinAPIEx UDF for a function named _WINAPI_FIleInUse($File).

$File is the resource that you are serializing. You can find the UDF by searching the forum...

Good Luck,

kylomas

[EdDyreen]

'

Thanks kylomas, I looked up the magic part and

Func _WinAPI_CreateFileEx($sFile, $iCreation, $iAccess = 0, $iShare = 0, $iFlagsAndAttributes = 0, $tSecurity = 0, $hTemplate = 0)
Local $Ret = DllCall('kernel32.dll', 'ptr', 'CreateFileW', 'wstr', $sFile, 'dword', $iAccess, 'dword', $iShare, 'ptr', DllStructGetPtr($tSecurity), 'dword', $iCreation, 'dword', $iFlagsAndAttributes, 'ptr', $hTemplate)
If (@error) Or ($Ret[0] = Ptr(-1)) Then
  Return SetError(1, 0, 0)
EndIf
Return $Ret[0]
EndFunc   ;==>_WinAPI_CreateFileEx

It looks really complex !!, unfortunately, I don't understand dll's

but I don't have to I guess, it works

O and water, I'm still interested in your solution too !

ps, I will also use the Filelock on registry keys, that may seem strange but I don't know any better.

Edited March 25, 2012 by EdDyreen

[kylomas]

Ed,

I am trying to write a "proof of concept" script using _winapi_fileinuse and it is NOT working...wait for a response from Water before electing a serialization technique. I'll keep you posted with what I find.

kylomas

[kylomas]

Ed,

Yashied created this excellent UDF to take the complexity out of calls to win api's. The UDF is complete with a help file.

At this point it might be safest for you to use Water's solution from post #4, given that you are more concerned with serialization than additional coding (as you should be).

Good Luck,

kylomas

[EdDyreen]

kylomas,

I don't think the solution from post #4 is bullet proof

But my solution 'post #5' is. I've used it for over a year, but it's so complex I was looking to see how more experienced people solve it. It stalls on high CPU levels too

I will use my own solution until I hear from water then....

[kylomas]

Ed,

I worked out the bugs in the "proof of concept" script that I was writing. The point was to demonstrate that you can serialize on a file for read/write access exclusively.

Copy the following script to your desktop and start multiple instances of it (up to 6). The script figures out where to put the gui's so you can see them all at the same time. Try various combinations of opening the test file across multiple processes.

expandcollapse popup

;
;
;

#include <winapiex.au3>
#include <file.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <StaticConstants.au3>
#include <EditConstants.au3>

Opt("GUIOnEventMode", 1)

Local $a_winlist = WinList('File Lock Test'), $top = 10, $left = 10

Switch $a_winlist[0][0]
    Case 0
    Case 1
        $left = 410
    Case 2
        $left = 820
    Case 3
        $top  = 550
        $left = 10
    Case 4
        $top  = 550
        $left = 410
    Case 5
        $top  = 550
        $left = 820
    Case Else
        MsgBox(0,'*** Error ***','Only 6 instances allowed')
        Exit
endswitch

;_arraydisplay($a_winlist)
;ConsoleWrite(@DesktopWidth & ' ' & @DesktopHeight & @lf)

Local $g010 = GUICreate('File Lock Test',300,400,$left,$top)
Local $e010 = GUICtrlCreateEdit('',10,10,280,300,$ES_AUTOVSCROLL + $WS_VSCROLL + $ws_hscroll)
              GUICtrlSetFont(-1,8.5,800,default,'courier new')
              GUISetFont(8.5,600,default,'times new roman')
Local $b010 = GUICtrlCreateButton('Open File (Read)  ',10,325,110,20)
Local $b020 = GUICtrlCreateButton('Open File (Write) ',180,325,110,20)
Local $b030 = GUICtrlCreateButton('Release File ',100,355,95,20)
Local $s010 = GUICtrlCreateLabel('',10,380,280,15,$ss_sunken)
              GUICtrlSetColor(-1,0xff0000)
              GUICtrlSetFont(-1,8.5,800)
              GUISetState()

              GUICtrlSetData($s010,'Running file lock test')
              GUISetOnEvent($GUI_EVENT_CLOSE, "_close")

              GUICtrlSetOnEvent($b010,"_openR")
              GUICtrlSetOnEvent($b020,"_openW")
              GUICtrlSetOnEvent($b030,"_release")

              AdlibRegister("_status",5000)

If Not FileExists(@TempDir & 'file_lock_test') Then FileWrite(@TempDir & 'file_lock_test',"")

Local $fn10 = @TempDir & "file_lock_test", $str, $st, $fl_lock, $hfl, $sw_w, $sw_r

While 1
    Sleep(1000)
wend

Func _close()
    Exit
endfunc

Func e010($str)
    GUICtrlSetData($e010,@MIN & ':' & @sec & ' ' & $str & @crlf,1)
endfunc

Func _openW()

    ;ConsoleWrite($fn10 & @lf)

    $fl_lock = _winapi_fileinuse($fn10)

    If @error <> 0 Then
        If MsgBox(17,'File Lock Error','Error = ' & @error) = 2 Then Exit
    endif

    If $fl_lock = 1 Then
        e010('Waiting for Write Access...')
        $sw_w = true
    Else
        $hfl = FileOpen($fn10,1)
        If $hfl = -1 Then
            e010('*** error opening file ***')
        Else
            e010('File Opened for Write Access')
        endif
    endif

endfunc

Func _release()

    FileClose($hfl)
    $hfl     = 0
    $fl_lock = 0
    e010('File released')

endfunc

Func _openR()

    ;ConsoleWrite($fn10 & @lf)

    $fl_lock = _winapi_fileinuse($fn10)

    If @error <> 0 Then
        If MsgBox(17,'File Lock Error','Error = ' & @error) = 2 Then Exit
    endif

    If $fl_lock = 1 Then
        e010('Waiting for Read Access...')
        $sw_r = true
    Else
        $hfl = FileOpen($fn10)
        If $hfl = -1 Then
            e010('*** error opening file ***')
        Else
            e010('File Opened for Read Access')
        endif
    endif

endfunc

Func _status()

    e010('waiting....lock status = ' & $fl_lock)

    If $sw_r Then
        $sw_r = False
        _openR()
    endif

    If $sw_w Then
        $sw_w = False
        _openW()
    endif

endfunc

kylomas

[water]

Hi Ed,

here is the code I use to make sure that scripts started in a bulk run serialized.

It's a modified and stripped down version of _Singleton. you can find the whole discussion.

expandcollapse popup

;---------------------------------------------------------------------
; Make sure that only one script runs at a time
;---------------------------------------------------------------------
While 1
    If _SingletonEx("New Employee", 1) = 0 Then
        Sleep(2000)
    Else
        ExitLoop
    Endif
WEnd

; #FUNCTION# ====================================================================================================================
; Name...........: _SingletonEx
; Description ...: Enforce a design paradigm where only one instance of the script may be running.
; Syntax.........: _SingletonEx($sOccurenceName[, $iFlag = 0])
; Parameters ....: $sOccurenceName - String to identify the occurrence of the script.  This string may not contain the  character unless you are placing the object in a namespace (See Remarks).
;                 $iFlag          - Behavior options.
;                 |0 - Exit the script with the exit code -1 if another instance already exists.
;                 |1 - Return from the function without exiting the script.
;                 |2 - Allow the object to be accessed by anybody in the system. This is useful if specifying a "Global" object in a multi-user environment.
; Return values .: Success  - The handle to the object used for synchronization (a mutex).
;                 Failure   - 0
; Author ........: Valik
; Modified.......: water based on code modified by smartee
; Remarks .......: You can place the object in a namespace by prefixing your object name with either "Global" or "Local".  "Global" objects combined with the flag 2 are useful in multi-user environments.
; Related .......:
; Link ..........:
; Example .......: Yes
; ===============================================================================================================================
Func _SingletonEx($sOccurenceName, $iFlag = 0)

    Local Const $ERROR_ALREADY_EXISTS = 183
    Local $tSecurityAttributes = 0
    Local $handle = DllCall("kernel32.dll", "handle", "CreateMutexW", "struct*", $tSecurityAttributes, "bool", 1, "wstr", $sOccurenceName)
    If @error Then Return SetError(@error, @extended, 0)
    Local $lastError = DllCall("kernel32.dll", "dword", "GetLastError")
    If @error Then Return SetError(@error, @extended, 0)
    If $lastError[0] = $ERROR_ALREADY_EXISTS Then
        DllCall("kernel32.dll", "bool", "CloseHandle", "handle", $handle[0])
        If @error Then Return SetError(@error, @extended, 0)
        If BitAND($iFlag, 1) Then
            Return SetError($lastError[0], $lastError[0], 0)
        Else
            Exit -1
        EndIf
    EndIf
    Return $handle[0]

EndFunc   ;==>_SingletonEx

Edited March 26, 2012 by water

[EdDyreen]

'

Unfortunately I won't have time for testing this week, but no news is good news

Thanks for the excellent help !,

Ed

[water]

Guinness posted a UDF for locking/unlocking files Maybe this does what you need.

[slbmeh]

I think the most effective solution would be to work with Semaphores and IPC. As far as I have seen does great IPC IMO.

Edited March 27, 2012 by slbmeh

[kylomas]

slbmeh,

I believe _singleton uses either semaphores or a mutex.

kylomas