Jump to content
Sign in to follow this  
Zaxon

Implementation of file semaphores

Recommended Posts

Zaxon

When writing out to a file, you need to guarantee you have exclusive access (a lock) on that file. By contrast, many processes may read from a file simultaneously.

File systems have locks or semaphores built into them. However, I'm unable to form any sort of lock in AutoIT.

AutoIT will cheerfully open the same file for writing from two different processes without an error. And you can open the same file for writing as many times as you like within a process. So in short, there's no way I can see to lock a file for exclusive access.

I've tried several ways of "rolling my own" semaphore system which is implemented successfully, but like all software semaphores, there's always the problem that a semaphore takes several steps:

1) Is this file in use?

2) No? Then writing out a temp file to represent a lock

The problem is that there are two steps to this process, which allows multiple processes to collide and both think they've got the lock.

So, how can we get exclusive writes to files in AutoIT?

Share this post


Link to post
Share on other sites
Valik

Semaphore's have nothing to do with locking a file. All you are doing with that method is serializing I/O in the scripts that are under your control. To lock a file for exclusive access, the "LockFile" Windows API function must be called. In order to use this, you need the file handle of the opened file (Not the handle returned by AutoIt, that is a pseudo-handle). That means you're basically going to have to implement all the file functions through DllCall() including LockFile(). Larry has wrapped some of the API functions so you might search for a library by him for a good starting location.

Share this post


Link to post
Share on other sites
Sokko

You could do something like this:

$file = FileOpen("filename",1)

If $file <> -1 Then
     FileSetAttrib("filename","+R")
     ;Insert file-writing stuff here
     FileClose($file)
     FileSetAttrib("filename","-R")
Else
     MsgBox(0,"Error","File opening failed!")
EndIf

The key is that you can't open a file for writing after it's been set to read-only, but you can do it the other way around. AutoIt will happily write to a read-only file if it was opened prior to setting the attribute.

Edited by Sokko

Share this post


Link to post
Share on other sites
Zaxon

Semaphore's have nothing to do with locking a file.  All you are doing with that method is serializing I/O in the scripts that are under your control.

<{POST_SNAPBACK}>

They are both related in achieving this purpose. Consider the extensive use of flock.

Having found no way of locking files in AutoIT - a pretty big hole that needs filling with inbuilt support in the language - I looked to implementing semaphores as a work around. But neither are supported by the language, so it would seem.

Share this post


Link to post
Share on other sites
Zaxon

The key is that you can't open a file for writing after it's been set to read-only, but you can do it the other way around. AutoIt will happily write to a read-only file if it was opened prior to setting the attribute.

<{POST_SNAPBACK}>

I gave your method a go. Here is your method with minor added steps:

$file = FileOpen("filename.txt",1)

If $file <> -1 Then
     FileSetAttrib("filename","+R")
    ;Insert file-writing stuff here
     sleep(5000)
     FileClose($file)
     FileSetAttrib("filename","-R")
     MsgBox(0,"Success","You did it!")
Else
     MsgBox(0,"Error","File opening failed!")
EndIf

The script pauses after (allegedly) securing the file to give us some testing space. If I run this script twice, one second after the other, I get two "success" outcomes, even though the two processes were clashing with access the file at the same time, whereas the second invocation should fail.

Share this post


Link to post
Share on other sites
Sokko

If I run this script twice, one second after the other, I get two "success" outcomes, even though the two processes were clashing with access the file at the same time, whereas the second invocation should fail.

I did the exact same thing as you, with the exact same script, and it worked. The second instance gave the "fail" message right away, and the first instance gave the "succeed" message after waiting out the delay. How are you doing the "running the script twice" bit? I tried both running the same script file twice, and running two different script files with the same content. Both times, it worked as expected.

Also, are you accounting for the fact that the "filename" in the FileSetAttrib commands would need to be changed to "filename.txt" if you also change it in the FileOpen command?

Edited by Sokko

Share this post


Link to post
Share on other sites
Valik

They are both related in achieving this purpose.  Consider the extensive use of flock. 

No, not really. Synchronization objects are better designed for serializing access than outright denying access. It also won't protect against applications outside your control accessing the file since you obviously can't inject the synchronization checks into it.

Having found no way of locking files in AutoIT - a pretty big hole that needs filling with inbuilt support in the language -

Let's not turn the entrance to an ant-hill into the mouth of a volcano. Yes, perhaps this is a hole, however just because you need the functionality doesn't mean the entire world does. As best I can remember, you are the first person to ever mention locking a file so this isn't a "pretty big hole" or there would be multiple people requesting it.

I looked to implementing semaphores as a work around.  But neither are supported by the language, so it would seem.

Semaphores/Mutexes/Events (Synchronization objects) are easily supported in the language through DllCall(). Virtually every script I write implements the singleton pattern by using a name semaphore to allow only a single instance of that script to run. I've also successfully created a script which communicates between two instances of itself via an event object. So I would say that it's technically true that they aren't implemented as native functions but synchronization is quite possible in AutoIt.

My suggestion is bring up in the idea lab that a FileLock() function needs to be added. It should be trivial to add and it does make sense to add so I can't fathom a reason why somebody wouldn't take the 2 minutes to implement it.

Share this post


Link to post
Share on other sites
Zaxon

Also, are you accounting for the fact that the "filename" in the FileSetAttrib commands would need to be changed to "filename.txt" if you also change it in the FileOpen command?

<{POST_SNAPBACK}>

No. Oops. OK. Works perfectly now. What is outstanding to make use of this method is:

1) What happens if a process crashes leaving the +R in place? How do we recover?

2) We need to allow multiple process access if reading. But how does a "soon-to-write" process know whether a file is currently being read if we don't lock down read access too?

But your method certainly starts out in the right direction. And I'd ideally like to find an elegant solution without having to investigate the DLL aspect of this. After all, if I'm going to call native windows functions, I may as well throw out AutoIT and code everything in C++. I thought the whole purpose of AutoIT was to abstract the user from low-level system functions by creating a simple, inbuilt interface?

Share this post


Link to post
Share on other sites
Valik

After all, if I'm going to call native windows functions, I may as well throw out AutoIT and code everything in C++.  I thought the whole purpose of AutoIT was to abstract the user from low-level system functions by creating a simple, inbuilt interface?

So you're saying that if you were then using C++ and then came upon a problem that needed implemented in assembly, rather than use inline assembly from C++ you'd throw away C++ and write the entire application in assembly? Doesn't that sound absurd?

Not every single thing can or should be implemented in AutoIt. If that were the case we could swell it to be several MB in a matter of hours. The power of DllCall lies in the ability to access to the native window functions for the things you need but still allows the bulk of an application to be written in AutoIt. Taking the attitude that "If I have to use native functions then I might as well use C" is just, frankly, stupid.

The goal of any language should be to provide the tools necessary for the developer to get the job done. AutoIt does this in spades. However, what AutoIt also does is provides low level access to people who need to go beyond what the language currently offers and allows them to extend the language through user-defined functions which call system API functions how they need. In other words, AutoIt offers both the convenience of any good scripting language but also provides the ability to have low-level access to get things done where the language's built-in features don't suffice.

I can guarantee you that it'll be harder and more work to implement the entire project in C++ (Even if you use AutoItX in the probject) than it will be to write the bulk in AutoIt and use DllCall() to fill in the gaps.

Share this post


Link to post
Share on other sites
Sokko

1) What happens if a process crashes leaving the +R in place?  How do we recover?

You could add a check at the beginning of the script using FileGetAttrib. If the desired file is read-only and there is exactly one script running, unset the attribute and continue normally. Depending on how you're running the scripts, there are three ways to check whether others are running:

1) If all the scripts are compiled and have different names, use a bunch of ProcessExists functions and slightly alter each script to check for all of the other script names, but not itself.

2) If all the scripts are compiled and have the same name (they would have to be in different locations), use $array[0][0] from ProcessList("scriptname.exe").

3) If you're running the script or scripts directly, use $array[0][0] from ProcessList("autoit.exe").

If you want other processes to be able to continue if one of them goes down, add a read timeout. Each time a process tries and fails to access the desired file, increment a counter and sleep for 1000. When the counter gets high enough, zero it and unset the attribute on the file. The limit on the counter should be significantly higher than the maximum number of seconds you will ever be accessing the file for.

2) We need to allow multiple process access if reading.  But how does a "soon-to-write" process know whether a file is currently being read if we don't lock down read access too?

I guess you could just use FileGetAttrib to disallow a script from both reading and writing if the file has read-only set. I don't know what could be done if you need to control write and read access separately.

Share this post


Link to post
Share on other sites
Zaxon

So you're saying that if you were then using C++ and then came upon a problem that needed implemented in assembly, rather than use inline assembly from C++ you'd throw away C++ and write the entire application in assembly?  Doesn't that sound absurd?

No, what I'm saying is that for a large majority of AutoIT users, creating structs and calling DLLs is really about mimicking C rather than about using AutoIT in a familiar way.

Taking the attitude that "If I have to use native functions then I might as well use C" is just, frankly, stupid.

I've never considered myself stupid, and I don't think you should be throwing around such labels in this forum.

I can guarantee you that it'll be harder and more work to implement the entire project in C++ (Even if you use AutoItX in the probject) than it will be to write the bulk in AutoIt and use DllCall() to fill in the gaps.

<{POST_SNAPBACK}>

Possibly. Having said that, I've successfully implemented the LockFile and UnlockFile system routines via DllCall.

However, I'm at a bit of a loss of how to integrate them with useful file access, such as iniwrite, etc.

I suspect I can't, and that once I've open up a file and locked it using system calls, that all my file access would have to be via system calls from then on in. Which negates having power functions like iniread/iniwrite in the first place.

In my programming world using PERL, PHP etc, a function like flock() is not a true file lock, but rather a negotiated semaphore (or probably a mutexe). I can call for a (pretend) lock, and then read/write to the file however I want to, and then simply release it once I've done.

I need this exact, same functionality here. Essentially, I'm wanting to lock a file whether via system or mutexe, and then be able to fully utilize AutoIt's native file accessing routines, before releasing the lock.

Although I've successfully implemented working calls to window's LockFile and UnlockFile system routines, I'm not sure that it's taking me where I need to be.

Share this post


Link to post
Share on other sites
Zaxon

You could add a check at the beginning of the script using FileGetAttrib. If the desired file is read-only and there is exactly one script running, unset the attribute and continue normally. Depending on how you're running the scripts, there are three ways to check whether others are running:

I don't know what could be done if you need to control write and read access separately.

<{POST_SNAPBACK}>

I like the approach in that it's using AutoIT native commands. But I think there's an inherent weakness in the basic first three lines.

1. $file = FileOpen("filename.txt",1)

2. If $file <> -1 Then
3.   FileSetAttrib("filename","+R")

It's predictable that you can get two processes which happen to run line 1 at around the same time. Both process A & B attempt to open the file simultaneously. Successful. They then both set the +R file attribute, leaving process A & B to believe that they now have exclusive use of the file. Which of course, isn't the case.

We need a way to check and reserve access in one statement, or else there are gaps for other processes to slide into.

Share this post


Link to post
Share on other sites
Valik

I've never considered myself stupid, and I don't think you should be throwing around such labels in this forum.

Did I call you stupid? No, I did not say, "Zaxon is stupid". I feel that the logic, "I have to do one thing in this language so I might as well do it all in this language" is stupid and that is what I said. There is a difference.

Possibly.  Having said that, I've successfully implemented the LockFile and UnlockFile system routines via DllCall.

However, I'm at a bit of a loss of how to integrate them with useful file access, such as iniwrite, etc.

I suspect I can't, and that once I've open up a file and locked it using system calls, that all my file access would have to be via system calls from then on in.  Which negates having power functions like iniread/iniwrite in the first place.

INI functions are "atomic"; you do not need to serialize or lock them. Even then they do not directly use the underlying file functions but rather use specific Windows API functions designed for working with INI files. It Windows handles simultaenous writes/reads to an INI file is up to itself but as I said, we are using underlying INI-specific functions, not homebrew code.

I have also already stated that if you wish to implement file related functions to have locking that you would have to implement them all via DllCall. I also said to search for a post by Larry which already implements most of them for you. The alternative was for you to put in a feature request to get a native file locking function which will work with AutoIt's pseudo-handle returned by FileOpen().

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×