Jump to content
Sign in to follow this  
Imbuter2000

simple semaphore solution for concurrent file writing

Recommended Posts

Imbuter2000

I simultaneously run (with F5 in multiple istances of SciTE, not as .EXE) multiple scripts that (among other things) writes to the same XML file, like these:

Script 1:

[...]
$fileFullPath = "c:\test.xml"
$o = _XMLFileOpen($fileFullPath)
_XMLCreateChildNode($o, "/xml", "abc", "text1")
_XMLCreateChildNode($o, "/xml", "def", "text2")
_XMLCreateChildNode($o, "/xml", "ghi", "text3")
_XMLSaveDoc($o, $fileFullPath)
[...]

Script 2:

[...]
$fileFullPath = "c:\test.xml"
$o = _XMLFileOpen($fileFullPath)
_XMLCreateChildNode($o, "/xml", "xxxxx", "teeeext4")
_XMLCreateChildNode($o, "/xml", "yyyyy", "teeeext5")
_XMLCreateChildNode($o, "/xml", "zzzzz", "teeeext6")
_XMLSaveDoc($o, $fileFullPath)
[...]

How can I avoid conflicts, thus how can I ensure that the block of instructions from _XMLFileOpen to _XMLSaveDoc in script 1 never starts while the similar block of istructions in script 2 is running, that would cause data loss?

I searched this forum 2 hours with keywords "semaphore", "singleton", "mutex" and tried 5-6 "solutions" but nothing worked...   or may be I simply didn't understand how to use them...

 

Edited by Imbuter2000

Share this post


Link to post
Share on other sites
water

First question: Is necessary to start multiple instances of the same script?


My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2018-12-03 - Version 1.4.11.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (2018-10-31 - Version 1.3.4.1) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2017-07-21 - Version 0.4.0.1) - Download - General Help & Support - Example Scripts
PowerPoint (2017-06-06 - Version 0.0.5.0) - Download - General Help & Support
Excel - Example Scripts - Wiki
Word - Wiki
 
Tutorials:

ADO - Wiki

 

Share this post


Link to post
Share on other sites
jdelaney

Might be easier to create a queue folder, where one script reads in the files of that folder, and performs the necessary writes.

So, each script, each time you want to add the xml nodes, will create some sort of distinct file, in the above folder, with instructions for the other script to know what to write...like a CSV file...with only one script writing, there should be no conflicts

sample:

#include <Array.au3>
#include <File.au3>
#include <Misc.au3>
$sQueueDir = @DesktopDir & "\queue"
$sXMLFile = $sQueueDir & "\output\some.xml"
$iFilesToAddPerRun = 100
$iConcurrentScripts = 10

If _Singleton(@ScriptName,1) Then
    DirCreate($sQueueDir)
    DirCreate($sQueueDir & "\output")
    If Not FileExists($sXMLFile) Then
        _FileCreate($sXMLFile)
        FileWrite($sXMLFile, '<?xml version="1.0"?>' & @CRLF & "<catalog><Parent></Parent></catalog>")
    EndIf
    Run(@AutoItExe & " /AutoIt3ExecuteScript " & @ScriptFullPath & " a XMLCreator")
    ; create 10 instances of script to add to queue
    For $i = 1 To $iConcurrentScripts
        $iPID = Run(@AutoItExe & " /AutoIt3ExecuteScript " & @ScriptFullPath & " " & $i)
        Sleep(100)
    Next
    ; Run until last script called is complete...make sure not singleton when calling others
    While ProcessExists($iPID)
        Sleep (10)
    WEnd
    Sleep(500)
    Exit
Else
    If UBound($CmdLine)>2 Then
        ; For 20 seconds, write to xml while looping for new queue entries
        ; where xml file is:
;~ <?xml version="1.0"?>
;~ <catalog><Parent></Parent></catalog>

        $oXML = ObjCreate("Microsoft.XMLDOM")
        $oXML.Load($sXMLFile)
        $oXMLParentNode = $oXML.selectSingleNode("//catalog/Parent")
        ; continue looping until no new files for 5 seconds
        $iTimer = TimerInit()
        While TimerDiff($iTimer) < 5000

            $aFiles = _FileListToArray_Limited($sQueueDir,"*",1,50)
            For $i = 1 To UBound($aFiles) - 1
                $iTimer = TimerInit()
                $text = FileRead($sQueueDir & "\" & $aFiles[$i])
                If StringLen($text) = 0 Then ContinueLoop
                $aData = StringSplit($text, ",", 2)
                $oNode = $oXMl.createElement($aData[0])
                ; adding in datetime, for fun
                $oNode.text = $aData[1]
                $oNode.setAttribute("FileCreated",$aData[2])
                $oNode.setAttribute("XMLWrite",@HOUR & ":" & @MIN & ":" & @SEC & "." & @MSEC)
                $oXMLParentNode.appendChild($oNode)
                FileDelete($sQueueDir & "\" & $aFiles[$i])
            Next
            $oXML.Save($sXMLFile)
            Sleep(50)
        WEnd
        Exit
    EndIf
EndIf

; each script creates 5 nodes
For $i = 1 To $iFilesToAddPerRun
    $sTime = $CmdLine[1] & "_" & @MIN & @SEC & @MSEC & ".txt"
    _FileCreate($sQueueDir & "\" & $sTime)
    $hFile = FileOpen($sQueueDir & "\" & $sTime,1)
    FileWrite($hFile, "NodeName,NodeValue_" & $CmdLine[1] & "_" & $i & "," & @HOUR & ":" & @MIN & ":" & @SEC & "." & @MSEC  )
    FileClose($hFile)
    Sleep(100)
Next

Func _FileListToArray_Limited($sPath, $sFilter = "*", $iFlag = 0, $iMaxUbound = Default)
    Local $hSearch, $sFile, $sFileList, $sDelim = "|"
    If $iMaxUbound = Default Then $iMaxUbound = -1 ; no limit
    $sPath = StringRegExpReplace($sPath, "[\\/]+\z", "") & "\" ; ensure single trailing backslash
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If StringRegExp($sFilter, "[\\/:><\|]|(?s)\A\s*\z") Then Return SetError(2, 2, "")
    If Not ($iFlag = 0 Or $iFlag = 1 Or $iFlag = 2) Then Return SetError(3, 3, "")
    $hSearch = FileFindFirstFile($sPath & $sFilter)
    If @error Then Return SetError(4, 4, "")
    $iCounter = 0
    While 1
        $sFile = FileFindNextFile($hSearch)
        If @error Then ExitLoop
        If ($iFlag + @extended = 2) Then ContinueLoop
        $sFileList &= $sDelim & $sFile
        If $iMaxUbound > -1 Then
            $iCounter+=1
            If $iCounter >= $iMaxUbound Then ExitLoop
        EndIf
    WEnd
    FileClose($hSearch)
    If Not $sFileList Then Return SetError(4, 4, "")
    Return StringSplit(StringTrimLeft($sFileList, 1), "|")
EndFunc   ;==>_FileListToArray

move the $oXML.Save prior to the filedelete to update the file real-time

Edited by jdelaney

IEbyXPATH-Grab IE DOM objects by XPATH IEscriptRecord-Makings of an IE script recorder ExcelFromXML-Create Excel docs without excel installed GetAllWindowControls-Output all control data on a given window.

Share this post


Link to post
Share on other sites
Chimp

Nice the jdelaney's idea to write many separate files and let only one process merge them in one file.
Also you could check this 2 post that shows how to check if a file is in use or not,

so you could write to a file only if is not in use by checking before writing
but i do not know if it is a bullet proof way,
Or maybe you could use SQLite to write concurrently and safely data to a database (>here an example from Mr. jchd) and  when processing is complete read from database and write to xml all at once


small minds discuss people average minds discuss events great minds discuss ideas.... and use AutoIt....

Share this post


Link to post
Share on other sites
jdelaney

I've also just found out ways of writing to an excel file as a database via ADO...don't even need excel installed!  When I find the link, i'll post it:

'?do=embed' frameborder='0' data-embedContent>>

this can read and write

Edited by jdelaney

IEbyXPATH-Grab IE DOM objects by XPATH IEscriptRecord-Makings of an IE script recorder ExcelFromXML-Create Excel docs without excel installed GetAllWindowControls-Output all control data on a given window.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×