simple semaphore solution for concurrent file writing

[Imbuter2000]

I simultaneously run (with F5 in multiple istances of SciTE, not as .EXE) multiple scripts that (among other things) writes to the same XML file, like these:

Script 1:

[...]
$fileFullPath = "c:\test.xml"
$o = _XMLFileOpen($fileFullPath)
_XMLCreateChildNode($o, "/xml", "abc", "text1")
_XMLCreateChildNode($o, "/xml", "def", "text2")
_XMLCreateChildNode($o, "/xml", "ghi", "text3")
_XMLSaveDoc($o, $fileFullPath)
[...]

Script 2:

[...]
$fileFullPath = "c:\test.xml"
$o = _XMLFileOpen($fileFullPath)
_XMLCreateChildNode($o, "/xml", "xxxxx", "teeeext4")
_XMLCreateChildNode($o, "/xml", "yyyyy", "teeeext5")
_XMLCreateChildNode($o, "/xml", "zzzzz", "teeeext6")
_XMLSaveDoc($o, $fileFullPath)
[...]

How can I avoid conflicts, thus how can I ensure that the block of instructions from _XMLFileOpen to _XMLSaveDoc in script 1 never starts while the similar block of istructions in script 2 is running, that would cause data loss?

I searched this forum 2 hours with keywords "semaphore", "singleton", "mutex" and tried 5-6 "solutions" but nothing worked...   or may be I simply didn't understand how to use them...

 

Edited November 1, 2013 by Imbuter2000

[water]

First question: Is necessary to start multiple instances of the same script?

[jdelaney]

Might be easier to create a queue folder, where one script reads in the files of that folder, and performs the necessary writes.

So, each script, each time you want to add the xml nodes, will create some sort of distinct file, in the above folder, with instructions for the other script to know what to write...like a CSV file...with only one script writing, there should be no conflicts

sample:

expandcollapse popup

#include <Array.au3>
#include <File.au3>
#include <Misc.au3>
$sQueueDir = @DesktopDir & "\queue"
$sXMLFile = $sQueueDir & "\output\some.xml"
$iFilesToAddPerRun = 100
$iConcurrentScripts = 10

If _Singleton(@ScriptName,1) Then
    DirCreate($sQueueDir)
    DirCreate($sQueueDir & "\output")
    If Not FileExists($sXMLFile) Then
        _FileCreate($sXMLFile)
        FileWrite($sXMLFile, '<?xml version="1.0"?>' & @CRLF & "<catalog><Parent></Parent></catalog>")
    EndIf
    Run(@AutoItExe & " /AutoIt3ExecuteScript " & @ScriptFullPath & " a XMLCreator")
    ; create 10 instances of script to add to queue
    For $i = 1 To $iConcurrentScripts
        $iPID = Run(@AutoItExe & " /AutoIt3ExecuteScript " & @ScriptFullPath & " " & $i)
        Sleep(100)
    Next
    ; Run until last script called is complete...make sure not singleton when calling others
    While ProcessExists($iPID)
        Sleep (10)
    WEnd
    Sleep(500)
    Exit
Else
    If UBound($CmdLine)>2 Then
        ; For 20 seconds, write to xml while looping for new queue entries
        ; where xml file is:
;~ <?xml version="1.0"?>
;~ <catalog><Parent></Parent></catalog>

        $oXML = ObjCreate("Microsoft.XMLDOM")
        $oXML.Load($sXMLFile)
        $oXMLParentNode = $oXML.selectSingleNode("//catalog/Parent")
        ; continue looping until no new files for 5 seconds
        $iTimer = TimerInit()
        While TimerDiff($iTimer) < 5000

            $aFiles = _FileListToArray_Limited($sQueueDir,"*",1,50)
            For $i = 1 To UBound($aFiles) - 1
                $iTimer = TimerInit()
                $text = FileRead($sQueueDir & "\" & $aFiles[$i])
                If StringLen($text) = 0 Then ContinueLoop
                $aData = StringSplit($text, ",", 2)
                $oNode = $oXMl.createElement($aData[0])
                ; adding in datetime, for fun
                $oNode.text = $aData[1]
                $oNode.setAttribute("FileCreated",$aData[2])
                $oNode.setAttribute("XMLWrite",@HOUR & ":" & @MIN & ":" & @SEC & "." & @MSEC)
                $oXMLParentNode.appendChild($oNode)
                FileDelete($sQueueDir & "\" & $aFiles[$i])
            Next
            $oXML.Save($sXMLFile)
            Sleep(50)
        WEnd
        Exit
    EndIf
EndIf

; each script creates 5 nodes
For $i = 1 To $iFilesToAddPerRun
    $sTime = $CmdLine[1] & "_" & @MIN & @SEC & @MSEC & ".txt"
    _FileCreate($sQueueDir & "\" & $sTime)
    $hFile = FileOpen($sQueueDir & "\" & $sTime,1)
    FileWrite($hFile, "NodeName,NodeValue_" & $CmdLine[1] & "_" & $i & "," & @HOUR & ":" & @MIN & ":" & @SEC & "." & @MSEC  )
    FileClose($hFile)
    Sleep(100)
Next

Func _FileListToArray_Limited($sPath, $sFilter = "*", $iFlag = 0, $iMaxUbound = Default)
    Local $hSearch, $sFile, $sFileList, $sDelim = "|"
    If $iMaxUbound = Default Then $iMaxUbound = -1 ; no limit
    $sPath = StringRegExpReplace($sPath, "[\\/]+\z", "") & "\" ; ensure single trailing backslash
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRegExp($sFilter, "[\\/:><\|]|(?s)\A\s*\z") Then Return SetError(2, 2, "")
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2) Then Return SetError(3, 3, "")
    $hSearch = FileFindFirstFile($sPath & $sFilter)
    If @error Then Return SetError(4, 4, "")
    $iCounter = 0
    While 1
        $sFile = FileFindNextFile($hSearch)
        If @error Then ExitLoop
        If ($iFlag + @extended = 2) Then ContinueLoop
        $sFileList &= $sDelim & $sFile
        If $iMaxUbound > -1 Then
            $iCounter+=1
            If $iCounter >= $iMaxUbound Then ExitLoop
        EndIf
    WEnd
    FileClose($hSearch)
    If Not $sFileList Then Return SetError(4, 4, "")
    Return StringSplit(StringTrimLeft($sFileList, 1), "|")
EndFunc   ;==>_FileListToArray

move the $oXML.Save prior to the filedelete to update the file real-time

Edited November 1, 2013 by jdelaney

[Gianni]

Nice the jdelaney's idea to write many separate files and let only one process merge them in one file.
Also you could check this 2 post that shows how to check if a file is in use or not,

so you could write to a file only if is not in use by checking before writing
but i do not know if it is a bullet proof way,
Or maybe you could use SQLite to write concurrently and safely data to a database (>here an example from Mr. jchd) and  when processing is complete read from database and write to xml all at once

[jdelaney]

I've also just found out ways of writing to an excel file as a database via ADO...don't even need excel installed!  When I find the link, i'll post it:

'?do=embed' frameborder='0' data-embedContent>>

this can read and write

Edited November 1, 2013 by jdelaney