Jump to content

This site uses cookies. By continuing to browse the site you are agreeing to our use of cookies. Find out more here. X
X


Photo

File Locking with Cooperative Semaphores


  • Please log in to reply
14 replies to this topic

#1 willichan

willichan

    Go ahead. You know you want to scan it.

  • Active Members
  • PipPipPipPipPipPip
  • 728 posts

Posted 12 July 2010 - 06:49 PM

This is a UDF I created for myself, to handle file locking on an SQLite database that was being shared on a network. Since I did not want to run a process on the server to handle the file locking, I needed a way to avoid write conflicts from the clients themselves.

**********

Thanks to an ISP problem, the ZIP download is not available.  The original code in post #2.

**********
 

I have an updated version that works in 64-bit Win7

Spoiler

If you want to see it in action, before implementing it in your own production scripts, I have written up a little demo here:

Spoiler

Make sure to set $logpath and $logfile to a proper location on your PC or network.

Run the demo as many times as you like on a single machine to simulate multiple machines, or run on several machines.
You will be able to see when each session gets its "lock" on the log file. You can also look at the log file itself to see what sessions/machines got access.

NOTE: Because of the way Windows handles multitasking, you will probably see groupings of the same session getting its lock several times before the next one. This is because Windows does not do a very good job of sharing the processor between applications, so one session may make several iterations before Windows pulls processor time away for the next session. You will see a better distribution if you actually use multiple machines rather than simulated machines.

 

For those who are wanting them, here are the "help" files that were included in the original ZIP file.

Nothing spectacular.  They are just a cut back demo designed to look like what would be in the help file if this were included.

Spoiler

 


Edited by willichan, 15 August 2013 - 07:49 PM.

  • VelvetElvis likes this







#2 willichan

willichan

    Go ahead. You know you want to scan it.

  • Active Members
  • PipPipPipPipPipPip
  • 728 posts

Posted 13 October 2010 - 08:57 PM

For those who don't want to trust downloading the UDF from my server, here is the code for it. The download does have the help/example files included in it, however.

CFS.au3
Plain Text         
#include-once #include <date.au3> ; Using:  _DateAdd(), _NowCalc(), _DateDiff() ; #INDEX# ========================================================================================= ; Title .........: CFS ; AutoIt Version : 3.2.3++ ; Language ..... : English ; Description ...: Functions for resource locking using cooperative, file-based semaphores ; Author(s) .....: willichan ; Note ..........: No files are actually locked using these functions.  Cooperative semaphores are ;                : dependant upon all access to the resource following the same locking mechanism. ;                : This satisfies resource locking needs when the resource is on a shared network ;                : location, but a server based semaphore service is not practical. ;                : For a good explanation of this methodology, see Sean M. Burke's article, ;                : "Resource Locking with Semaphore Files" at <a href='http://interglacial.com/tpj/23/' class='bbc_url' title='External link' rel='nofollow external'>http://interglacial.com/tpj/23/</a> and ;                : "Resource Locking Over Networks" at <a href='http://interglacial.com/tpj/24/' class='bbc_url' title='External link' rel='nofollow external'>http://interglacial.com/tpj/24/</a> ; ;================================================================================================ ; ------------------------------------------------------------------------------ ; This software is provided 'as-is', without any express or ; implied warranty.  In no event will the authors be held liable for any ; damages arising from the use of this software. ; #CURRENT# ======================================================================================= ; _CFS_RequestSemaphore ; _CFS_ReleaseSemaphore ; ;================================================================================================ ; #INTERNAL_USE_ONLY# ============================================================================= ; _CFS_ResourseLockName ; _CFS_CreateLock ; _CFS_RemoveLock ; _CFS_LockAge ; ;================================================================================================ ; #FUNCTION# ;===================================================================================== ; ; Name...........: _CFS_RequestSemaphore ; Description ...: Requests cooperative semaphore lock for requested resource ; Syntax.........: _CFS_RequestSemaphore($sRequestedResource[, $iTimeout[, $iClean]]) ; Parameters ....: $sRequestedResource - Path/name of the requested resource ;                  $iTimeout - Optional, timeout in seconds for before lock request fails ;                  default = 300s (5 min) ;                  $iClean - Optional, age in seconds for lock to exist before it is ;                            assumed dead and forcably removed. default = 900s (15 min) ; Return values .: Success - Returns 1 ;                  Failure - Returns 0 ; Author ........: willichan ; Modified.......: ; Remarks .......: Since we assume the resource exists, we also assume the path to it exists ; Related .......: ; Link ..........: ; Example .......: Yes ; ; ;================================================================================================ Func _CFS_RequestSemaphore($sRequestedResource, $iTimeout = 300, $iClean = 900)     Local $sCFS_Semaphore = _CFS_ResourseLockName($sRequestedResource)     Local $iStart = TimerInit()     While True         If _CFS_LockAge($sCFS_Semaphore, $iClean) Then _CFS_RemoveLock($sCFS_Semaphore) ; Too old?         If TimerDiff($iStart) > ($iTimeout * 1000) Then ; have we timed out?             Return 0         Else             If _CFS_CreateLock($sCFS_Semaphore) Then Return 1 ; did we get the lock?         EndIf         Sleep(100) ; didn't get a lock, pause and try again     WEnd EndFunc   ;==>_CFS_RequestSemaphore ; #FUNCTION# ;===================================================================================== ; ; Name...........: _CFS_ReleaseSemaphore ; Description ...: Releases cooperative semaphore lock for requested resource ; Syntax.........: _CFS_ReleaseSemaphore($sRequestedResource) ; Parameters ....: $sRequestedResource - Path/name of the requested resource ; Return values .: Success - Returns 1 ;                  Failure - Returns 0 ; Author ........: willichan ; Modified.......: ; Remarks .......: Never call this unless you hold a successful request ; Related .......: ; Link ..........: ; Example .......: Yes ; ; ;================================================================================================ Func _CFS_ReleaseSemaphore($sRequestedResource)     Return _CFS_RemoveLock(_CFS_ResourseLockName($sRequestedResource)) EndFunc   ;==>_CFS_ReleaseSemaphore ; #INTERNAL_USE_ONLY# ============================================================================= ; ; Name...........: _CFS_ResourseLockName ; Description ...: Returns the name of the resource lock ; Syntax.........: _CFS_ResourseLockName($sRequestedResource) ; Parameters ....: $sRequestedResource - Path/name of the requested resource ; Return values .: Name of resource lock ; Author ........: willichan ; Modified.......: ; Remarks .......: ; Related .......: ; Link ..........: ; Example .......: No ; ; ;================================================================================================ Func _CFS_ResourseLockName($sRequestedResource)     Local Const $sInvalidChars = "\ "     Local $sCFS_Semaphore = StringStripWS($sRequestedResource, 3)     While StringInStr($sInvalidChars, StringRight($sCFS_Semaphore, 1))         $sCFS_Semaphore = StringLeft($sCFS_Semaphore, StringLen($sCFS_Semaphore) - 1)     WEnd     Return $sCFS_Semaphore & "_Sem" EndFunc   ;==>_CFS_ResourseLockName ; #INTERNAL_USE_ONLY# ============================================================================= ; ; Name...........: _CFS_CreateLock ; Description ...: Creates the semaphore lock ; Syntax.........: _CFS_CreateLock($sCFS_Semaphore) ; Parameters ....: $sCFS_Semaphore - Name of the resource lock ; Return values .: Success - Returns 1 ;                  Failure - Returns 0 ; Author ........: willichan ; Modified.......: ; Remarks .......: ; Related .......: ; Link ..........: ; Example .......: No ; ; ;================================================================================================ Func _CFS_CreateLock($sCFS_Semaphore)     ; Since DirCreate() does not return an error if the directory already exists, we use mkdir     Local $iResult = RunWait('mkdir "' & $sCFS_Semaphore & '"', @TempDir, @SW_HIDE)     If @error Then Return 0     If $iResult = 0 Then         Return 1     Else         Return 0     EndIf EndFunc   ;==>_CFS_CreateLock ; #INTERNAL_USE_ONLY# ============================================================================= ; ; Name...........: _CFS_RemoveLock ; Description ...: Removes the semaphore lock ; Syntax.........: _CFS_RemoveLock($sCFS_Semaphore) ; Parameters ....: $sCFS_Semaphore - Name of the resource lock ; Return values .: Success - Returns 1 ;                  Failure - Returns 0 ; Author ........: willichan ; Modified.......: ; Remarks .......: ; Related .......: ; Link ..........: ; Example .......: No ; ; ;================================================================================================ Func _CFS_RemoveLock($sCFS_Semaphore)     Return DirRemove($sCFS_Semaphore, 1) EndFunc   ;==>_CFS_RemoveLock ; #INTERNAL_USE_ONLY# ============================================================================= ; ; Name...........: _CFS_LockAge ; Description ...: Checks to see if the Lock is too old to be valid ; Syntax.........: _CFS_LockAge($sCFS_Semaphore, $iClean) ; Parameters ....: $sCFS_Semaphore - Name of the resource lock ;                  $iClean - Age in seconds for lock to exist before it is assumed dead and ;                  forcably removed ; Return values .: Returns 1 if lock is to old ;                  Returns 0 if lock is still valid ; Author ........: willichan ; Modified.......: ; Remarks .......: ; Related .......: ; Link ..........: ; Example .......: No ; ; ;================================================================================================ Func _CFS_LockAge($sCFS_Semaphore, $iClean)     Local $asDate = FileGetTime($sCFS_Semaphore, 1, 0)     If @error Then Return 0     Local $sExpires = _DateAdd("s", $iClean, $asDate[0] & "/" & $asDate[1] & "/" & $asDate[2] & " " & $asDate[3] & ":" & $asDate[4] & ":" & $asDate[5])     If _DateDiff("s", _NowCalc(), $sExpires) > 0 Then         Return 0     Else         Return 1     EndIf EndFunc   ;==>_CFS_LockAge


#3 footswitch

footswitch

    Polymath

  • Active Members
  • PipPipPipPip
  • 206 posts

Posted 25 November 2010 - 09:50 PM

Thanks for posting your idea, I'll definitely give it a try :graduated:

[EDIT]
First remarks:
- You need to have the desired network path mapped with a drive letter (you also need to be authenticated to that path, but mapping rules out this requirement).

Second remarks:
- I believe it works as advertised! Only suggestion is that you change the following in CFS.au3
From this:
Local $iResult = RunWait('mkdir "' .........
To this:
Local $iResult = RunWait(@ComSpec & " /c " &'mkdir "' ............

The first option doesn't work under Windows 7 64-bit


Cheers,
footswitch

Edited by footswitch, 26 November 2010 - 12:54 AM.


#4 willichan

willichan

    Go ahead. You know you want to scan it.

  • Active Members
  • PipPipPipPipPipPip
  • 728 posts

Posted 29 November 2010 - 09:47 PM

The first option doesn't work under Windows 7 64-bit

Thank you. I will test that out.

You need to have the desired network path mapped with a drive letter

I wonder if that is also a Win7 64-bit limitation. I am able to work with UNCs under XP Pro SP3

you also need to be authenticated to that path

That goes without saying. :graduated:

Thanks for the feedback. I'll start doing some testing under 64-bit.

#5 footswitch

footswitch

    Polymath

  • Active Members
  • PipPipPipPip
  • 206 posts

Posted 30 November 2010 - 07:49 PM

I am able to work with UNCs under XP Pro SP3

Really? Because as far as I know, the command line doesn't support UNCs. I could be wrong.

About the authentication, yes, it's kind of obvious, but once you start working directly with UNCs you run the risk of needing to authenticate on-the-fly, and this script wouldn't allow that also.

So... does mkdir work flawlessly? I mean, does mkdir really return an error upon creating an existing folder, even when it "thinks" it doesn't exist?
Let me give you an example, based on the way that I THINK mkdir works:
- mkdir checks for the directory to exist
a) if it exists, return error
:graduated: if it doesn't exist, create it and return successfully - now, at this time, imagine another instance of mkdir is doing the exact same thing, but some miliseconds earlier. Question is: does mkdir check for the successful CREATION of the folder or, in the other hand, successful ISSUING of that command?
So as you can see I'm worried about a racing event which could happen or not depending on the way mkdir works.


footswitch

#6 willichan

willichan

    Go ahead. You know you want to scan it.

  • Active Members
  • PipPipPipPipPipPip
  • 728 posts

Posted 01 December 2010 - 07:10 PM

RE: Mkdir
Yes, mkdir does support UNCs. Many of the command line utilities will support UNC parameters. You just can't change to/land on a UNC.

I won't go into conjecture as to how Microsoft goes about doing it. My testing, however, shows mkdir to be the most reliable method as far as returning a proper error code. I tested various methods on 30 machines, running 24 hours, with 100 processes each, all simultaneously attempting mkdir commands. In that time, I did not get a single false 'success' report.

If a more reliable method is found, I will definitely test/implement it.

RE: authentication
Since the semaphore directory is in the same location as the resource your are trying to "lock" it is assumed that you will already be handling the authentication to the resource location before attempting to get a lock on it.

#7 footswitch

footswitch

    Polymath

  • Active Members
  • PipPipPipPip
  • 206 posts

Posted 02 December 2010 - 12:44 PM

Yes, mkdir does support UNCs. Many of the command line utilities will support UNC parameters. You just can't change to/land on a UNC.

You are absolutely right, I just tested it under Windows 7 x64 and it did work. but the script didn't.
So I retested the script and the path wasn't an issue after all, it was the mkdir call all along. As I mentioned earlier, for 64-bit compatibility:
Local $iResult = RunWait(@ComSpec & " /c " & 'mkdir "' & $sCFS_Semaphore & '"', @TempDir, @SW_HIDE)
Probably this compatibility issue is related to the command tools' location not being mapped to the PATH environment variable. But this fixes it.
Turns out it was my fault that I didn't test for UNC support after adjusting that single line of code. Sorry about that.


I tested various methods on 30 machines, running 24 hours, with 100 processes each, all simultaneously attempting mkdir commands. In that time, I did not get a single false 'success' report.

Wow, didn't realise there was so much study involved in it. That's... something :graduated:

Since the semaphore directory is in the same location as the resource your are trying to "lock" it is assumed that you will already be handling the authentication to the resource location before attempting to get a lock on it.

Okay, that's a fair assumption :D


Next step: File Locking with Cooperative Queuing Semaphores :(

#8 willichan

willichan

    Go ahead. You know you want to scan it.

  • Active Members
  • PipPipPipPipPipPip
  • 728 posts

Posted 02 December 2010 - 02:54 PM

So I retested the script and the path wasn't an issue after all, it was the mkdir call all along. As I mentioned earlier, for 64-bit compatibility:

I've got that added to my ToDo list. ALl of the systems at the office are XP Pro, and all my systems at home are Linux, but I do have a license for Win 7 Ultimate. I will need to set up a machine and do some testing to make sure the error returns still work right using @ComSpec.

Next step: File Locking with Cooperative Queuing Semaphores :D

:graduated: That will be a nice trick. If I figure that one out, does someone buy me lunch? :(

#9 footswitch

footswitch

    Polymath

  • Active Members
  • PipPipPipPip
  • 206 posts

Posted 02 December 2010 - 04:59 PM

I guess you're right, @ComSpec could change the results. "your mileage may vary" :(

It's not like I can't live without queuing, but over time I'm thinking of implementing it myself :graduated:
If xyz_Sem exists, find all folders named xyz_Sem_*, get the last folder number and create xyz_Sem_n+1, where n is the Queue order.
Something like that.
But I'm predicting some issues here: There's already a timeout condition for the current semaphores. Now when you have lots of Queues and you don't know whether they will succeed or not, you could be placing a huge delay in database response time.

#10 willichan

willichan

    Go ahead. You know you want to scan it.

  • Active Members
  • PipPipPipPipPipPip
  • 728 posts

Posted 02 December 2010 - 10:37 PM

It's not like I can't live without queuing, but over time I'm thinking of implementing it myself :(


It would be nice. Unfortunately, the cooperative semaphore scheme muddles it a bit. It was intended for where running server services were impractical. When you reach the need for queuing, it may become more practical to go with a server service.

I'll still keep my brain chugging on it :graduated:, but I don't see it materializing too soon.

#11 Megabird

Megabird

    Seeker

  • Normal Members
  • 3 posts

Posted 27 April 2011 - 09:28 PM

I'm thinking of something that may give you something to work on: DirCreate won't throw an error if the directory already exists, but DirMove does if you leave it's flag value to 0. So the idea is:

1: create a unique semafore (ex. using computername + time)
2: try to rename that semafore to the real semafore name until success

It can also help you queue the requests, since you can see every uniques semafores...

just an idea... :unsure:

#12 willichan

willichan

    Go ahead. You know you want to scan it.

  • Active Members
  • PipPipPipPipPipPip
  • 728 posts

Posted 28 April 2011 - 02:43 PM

I'm thinking of something that may give you something to work on: DirCreate won't throw an error if the directory already exists, but DirMove does if you leave it's flag value to 0. So the idea is:

I actually use the DOS mkdir command, because it does return an error for directory exists. Directory creation is the most stable for cross-platform server compatibility. Even though AutoIt only runs from Windows, normal file locking is not handled consistently by servers running on various platforms (Mac, Linux, Unix, Novel, etc...).

This is an interesting idea. I will look into how consistent directory moves/renames are on other platforms. Definately something to play around with.

----Edit----

Ok. I've been playing around with queuing. Since there is no master process monitoring the queue and cleaning up, it gets a little tricky. I have a basic process for cooperative cleaning up, but it is not perfect. Without the queuing, I was able to run several hundred simulated users across 30-50 machines without any collisions. Unfortunately, with the queuing, I don't have it running quite so cleanly. I get collisions every 20-30 accesses with only 30 simulated users. Still needs some work. Feel free to play around with it if you want to.

Spoiler

Edited by willichan, 28 April 2011 - 09:52 PM.


#13 willichan

willichan

    Go ahead. You know you want to scan it.

  • Active Members
  • PipPipPipPipPipPip
  • 728 posts

Posted 15 August 2013 - 07:39 PM

New version now works in 64-bit Windows 7.

 

See post #1



#14 orbs

orbs

    Universalist

  • Active Members
  • PipPipPipPipPipPip
  • 676 posts

Posted 12 October 2014 - 09:28 AM

willichan, very nice work! simple implementation of a very smart concept!

 

question about parameter $iClean in function _CFS_RequestSemaphore() :

description: "age in seconds for lock to exist before it is assumed dead and forcably removed."

 

it seems that this is not quite so.

 

$iClean is used in function _CFS_LockAge() to determine if the existing semaphore exceeds the time specified for the calling instance, not for the instance which created the semaphore.

 

consider this:

 

PROCESS1 creates a semaphore with $iClean=60. PROCESS1 thinks that this means that the resource will remain at its disposal for at least one minute.

now PROCESS2 wants to lock the resource for itself, but needs it for only 15 seconds, so calling _CFS_RequestSemaphore() with $iClean=15.

_CFS_LockAge() called by PROCESS2 checks if the semaphore is older than 15 seconds, while any common sense would think it should check if the semaphore is older than 60 seconds, as it is currently locked by PROCESS1 which requested it for 60 seconds.

 

admittedly, usually the same resource is locked by different instances of the same process for the same maximum time, so this is not a problem. but occasionally this is not the case, and anyway for proper coding, this should be addressed.

 

possible solutions would be to write a text file inside the semaphore directory, which contains the maximum age (and perhaps also the hostname and PID of the process who locked it, so before force kill, the calling process can query the condition of the locking process).

 

or am i getting this completely wrong?

 

EDIT:

 

also, what is the X86 condition good for? @ComSpec works just as well for X86, you can just use it regardless of architecture. i'm referring to this line:

If @OSArch <> 'X86' Then $cmd = @ComSpec & ' /c ' & $cmd

which can be simply:

$cmd = @ComSpec & ' /c ' & $cmd

Edited by orbs, 12 October 2014 - 06:50 PM.


#15 orbs

orbs

    Universalist

  • Active Members
  • PipPipPipPipPipPip
  • 676 posts

Posted 15 minutes ago

using mkdir instead of DirCreate() is understood. but a better way may exist: using direct call to CreateDirectoryW in kernel32.dll (which also fails if the directory already exists).

 

a timing test shows mkdir takes ~25ms while CreateDirectoryW takes ~1ms

 

testing locally on quite a decent machine:

Windows 7 64-bit, CPU i5, 16GB RAM

testing on SSD, SATA3, and USB3 drives - similar results.

 

this testing script does not use the UDF, instead it has 2 versions of the _CFS_CreateLock() function - one the original (well, almost - i got rid of the architecture condition, as i mentioned in previous post), one calling the dll.

AutoIt         
#include <Array.au3> ; for _ArrayDisplay() Global $sCFS_Semaphore = @TempDir & '\CFS_Test_Sem' ; direclty define the semaphore directory name Global $aTime[4] ; an array to hold the timing results DirRemove($sCFS_Semaphore) ; initial cleanup. comment this line out (and create the semaphore before next run) to make sure the function failes when semaphore already exists $aTime[0] = @SEC & '.' & @MSEC $nResult_comspec = __CFS_CreateLock_comspec($sCFS_Semaphore) $aTime[1] = @SEC & '.' & @MSEC MsgBox(0, $nResult_comspec, '$nResult_comspec') ; see the return value DirRemove($sCFS_Semaphore) ; initial cleanup. comment this line out to make sure the function failes when semaphore already exists $aTime[2] = @SEC & '.' & @MSEC $nResult_dll = __CFS_CreateLock_dll($sCFS_Semaphore) $aTime[3] = @SEC & '.' & @MSEC MsgBox(0, $nResult_dll, '$nResult_dll') ; see the return value DirRemove($sCFS_Semaphore) ; final cleanup _ArrayDisplay($aTime) ; display results Func __CFS_CreateLock_comspec($sCFS_Semaphore)     ; Since DirCreate() does not return an error if the directory already exists, we use mkdir     Local $cmd = 'mkdir "' & $sCFS_Semaphore & '"'     $cmd = @ComSpec & ' /c ' & $cmd     Local $iResult = RunWait($cmd, @TempDir, @SW_HIDE)     If @error Then Return 0     If $iResult = 0 Then         Return 1     Else         Return 0     EndIf EndFunc   ;==>__CFS_CreateLock_comspec Func __CFS_CreateLock_dll($sCFS_Semaphore)     ; Since DirCreate() does not return an error if the directory already exists, we use WinAPI     Local $aRet = DllCall('kernel32.dll', 'bool', 'CreateDirectoryW', 'wstr', $sCFS_Semaphore, 'struct*', 0)     If @error Then Return 0     If $aRet[0] = 0 Then         Return 0     Else         Return 1     EndIf EndFunc   ;==>__CFS_CreateLock_dll  

i will test on a network share when i get the chance. unless i experience some hiccups in further tests, i think the conclusion is clear - it is better to use the dll version (last function in the testing script).






2 user(s) are reading this topic

1 members, 1 guests, 0 anonymous users


    steve8tch