Folder watcher doesn't work when folder has more than a few files

[emendelson 8]

Using some very ingenious scripts that I found on this forum, I've put together a special-purpose folder watcher that watches a specified folder for printer output files and then either send them to a printer or converts them to a PDF and prints or opens the PDF. The page about this utility is here:

http://www.columbia.edu/~em36/printfileprinter.html

The one serious problem that it has is that it seems not to detect a new file if there are more than two or three files already in the watched folder. I can't figure out what I'm doing wrong, and will be grateful for any help. Here is the relevant part of the script. I've left out the functions that test whether the file is in use or not, and that send the raw data to the printer or create a PDF, etc. I hope there's enough code here to make sense of it, and will be very grateful for any help. Again, the problem is that the script doesn't detect newly-created files in watched folders with more than a very few files already in it.

My totally ignorant guess is that the problem is in the line $iID = _WinAPI_WaitForMultipleObjects(2, $paObj, 0, 0) - but I don't know how to change it and of course I'm only guessing whether it's relevant or not. Many thanks

 

expandcollapse popup

Global $g_ahObj[2]
$g_ahObj[0] = _WinAPI_FindFirstChangeNotification($watchPath, $FILE_NOTIFY_CHANGE_FILE_NAME)
$g_ahObj[1] = _WinAPI_FindFirstChangeNotification($watchPath, $FILE_NOTIFY_CHANGE_DIR_NAME)

If (Not $g_ahObj[0]) Or (Not $g_ahObj[1]) Then
    MsgBox(BitOR($MB_ICONERROR, $MB_SYSTEMMODAL), 'Error', 'Unable to create change notification.')
    Exit
EndIf

Local $tObjs = DllStructCreate('ptr;ptr')
Local $paObj = DllStructGetPtr($tObjs)
For $i = 0 To 1
    DllStructSetData($tObjs, $i + 1, $g_ahObj[$i])
Next

Local $iID
While 1
    Sleep(100)
    $select = 0
    $print = 0
    $format = ""
    Local $tempPDF
    $tempPDF = 0
    $iID = _WinAPI_WaitForMultipleObjects(2, $paObj, 0, 0)
    Switch $iID
        Case 0 ; WAIT_OBJECT_0
            ; ConsoleWrite('A file was created, renamed, or deleted in the directory.' & @CRLF)

            Local $hSearch = FileFindFirstFile($watchPath & "\*")
            Local $sFileName = "", $iResult = 0
            Local $sFilePath = ""
            While 1
                $sFileName = FileFindNextFile($hSearch)
                ; If there is no more file matching the search.
                If @error Then ExitLoop

                $sFilePath = $watchPath & "\" & $sFileName

                Local $fileUsed
                $fileUsed = 0
                $fileUsed = _FileIsUsed($sFilePath)

                If $fileUsed = 1 Then
                    While 1
                        Sleep(100)
                        $fileUsed = _FileIsUsed($sFilePath)
                        If $fileUsed = 0 Then ExitLoop
                    WEnd
                EndIf
                
                If StringInStr($sFileName, "raw") Then
                    If StringInStr($sFileName, "select") Then $select = 1
                    PrintRawFile($sFilePath, $select)
                Else
                    If StringInStr($sFileName, "lg.") Then
                        $pageSize = "legal"
                    ElseIf StringInStr($sFileName, "a4.") Then
                        $pageSize = "a4"
                    ElseIf StringInStr($sFileName, "us.") Then
                        $pageSize = "letter"
                    EndIf

                    $pdfTemp = 0
                    If StringLower(StringLeft($sFileName, 7) = "pdftemp") Then $pdfTemp = 1

                    If StringInStr($sFileName, ".pcl") Then
                        $format = "pcl"
                        $print = 1
                        If StringLeft($sFileName, 6) = "select" Then
                            $select = 1
                            $print = 1
                        ElseIf StringLeft($sFileName, 3) = "pdf" Then
                            $select = 0
                            $print = 0
                        EndIf
                        MakePDF($sFilePath, $format, $print, $select, $pageSize, $pdfTemp)
                    ElseIf StringInStr($sFileName, ".ps") Then
                        $format = "ps"
                        $print = 1
                        If StringLeft($sFileName, 6) = "select" Then
                            $select = 1
                            $print = 1
                        ElseIf StringLeft($sFileName, 3) = "pdf" Then
                            $select = 0
                            $print = 0
                        EndIf
                        MakePDF($sFilePath, $format, $print, $select, $pageSize, $pdfTemp)
                    ElseIf StringInStr($sFileName, ".esc") Then
                        Sleep(200)
                        $format = "epson"
                        $print = 1
                        If StringLeft($sFileName, 6) = "select" Then
                            $select = 1
                            $print = 1
                        ElseIf StringLeft($sFileName, 3) = "pdf" Then
                            $select = 0
                            $print = 0
                        EndIf
                        MakePDF($sFilePath, $format, $print, $select, $pageSize, $pdfTemp)
                    ElseIf StringInStr($sFileName, ".prn") Then
                        $format = GetFileFormat($sFilePath)
                        $print = 1
                        If StringLeft($sFileName, 6) = "select" Then
                            $select = 1
                            $print = 1
                        ElseIf StringLeft($sFileName, 3) = "pdf" Then
                            $select = 0
                            $print = 0
                        EndIf
                        ConsoleWrite("PRN test format: " & $format & @CRLF)
                        MakePDF($sFilePath, $format, $print, $select, $pageSize, $pdfTemp)
                    EndIf
                EndIf

                ; Display the file name.
                ; $iResult = MsgBox(BitOR($MB_SYSTEMMODAL, $MB_OKCANCEL), "", $watchPath & "\" & $sFileName)
;~              RunWait(@ComSpec & " /c notepad.exe " & $watchPath & "\" & $sFileName)
;~              FileDelete($watchPath & "\" & $sFileName)
                If $iResult <> $IDOK Then ExitLoop ; If the user clicks on the cancel/close button.
            WEnd

            ; Close the search handle.
            FileClose($hSearch)


        Case 1 ; WAIT_OBJECT_0 + 1
            ; ConsoleWrite('A directory was created, renamed, or deleted.' & @CRLF)
        Case Else
            ContinueLoop
    EndSwitch
    If Not _WinAPI_FindNextChangeNotification($g_ahObj[$iID]) Then
        MsgBox(BitOR($MB_ICONERROR, $MB_SYSTEMMODAL), 'Error', 'Unexpected error.')
        Exit
    EndIf
WEnd

 

[argumentum 835]

2 minutes ago, emendelson said:

Using some very ingenious scripts that I found on this forum

then try 

there is an "Example_Fork_FileMonitor" you can use  

[emendelson 8]

Thank you! I will try this, and it looks extremely useful and careful - but it would clearly require a total rewrite of what I've been using. I may need to do that in the end, but I wonder if there's something else that could fix the existing code?

[argumentum 835]

1 minute ago, emendelson said:

wonder if there's something else that could fix the existing code

wouldn't. Forking is better for this case. I've tested the example with fast file create in a RAM drive and works well. Working stopping the "file watcher" may will introduce "miss catch" of an event, so, adapt the example to your code. Should be simple.

[emendelson 8]

OK, I'll go to work! Thank you!

[emendelson 8]

@argumentum I started work and found that I had to make one change in Fork.au3:

It tries to use the function _MailSlot_Write(), but the actual function in MailSlot.au3 is _MailSlotWrite().

Have I got the right versions of everything? Thanks again...

[argumentum 835]

1 minute ago, emendelson said:

but the actual function in MailSlot.au3 is _MailSlotWrite()

yep, my bad. You've got it  
I'll look at it and fix it. Thanks for the catch 

[emendelson 8]

This is obviously terrific, but it's far beyond my abilities to reduce the example folder watcher to something that will silently watch for a new or renamed file in a folder and then perform an action on that file. Do you know of any scripts based on your code that I can work with? I've been staring at your code, and I'm just not expert enough to figure out how to use it without the GUI...

Apologies for responding to your help with a question!

Edited February 3, 2020 by emendelson

[argumentum 835]

3 minutes ago, emendelson said:

but it's far beyond my abilities

lol, is not. It can get cumbersome looking at someone else's code.
..you'd have to start the forked watcher by 

$pidFileMon = _Fork_Func('_Forked_ReadDirectoryChanges("' & GUICtrlRead($guiFFM_Path) & '")', Default, '/ErrorStdOut "note to self to see in Task Manager OR extra CmdLine you may need"')

and to stop it, using the PID from $pidFileMon, just ProcessClose($pidFileMon).
( If you wanna make your own idea, run the _Forked_ReadDirectoryChanges($sPath) from another file. )

The fork will send the events via the MailSlot. The events are "*" separated ( as no filename can have that character ), so, StringSplit() each line from the mailslot and that would give that you the data ( or close to it ) of _WinAPI_FindFirstChangeNotification()

If you get used to passing data with an IPC ( inter process communication ) scheme, in this case MailSlots, you'll free your main loop to do the logic you envision, without the hindering that a process ( in this case, watching file/folder changes ) may do ( lock or slow down your main loop ).

Fear not citizen, help has arrived  

[emendelson 8]

OK, I'll give this a try this week - I can see that I'll need a couple of hours at least to concentrate on it.

Meanwhile, I'm assuming that you're telling me that I don't need to open the GUI in order to make this work?

Thank you for this!

[argumentum 835]

9 hours ago, emendelson said:

I don't need to open the GUI in order to make this work

nope, the GUI is a visual thing to see what goes on. Testing. The GUI is just a GUI, no need for it  

[Nine 1,515]

Or you may want to take a look at this :

#include <APIFilesConstants.au3>
#include <Array.au3>
#include <MsgBoxConstants.au3>
#include <WinAPIError.au3>
#include <WinAPIFiles.au3>
#include <WinAPIMem.au3>

HotKeySet ("{ESC}", _Exit)

Global $g_sPath = "C:\Apps\Temp\", $nBufferLen = 1048576
If Not FileExists($g_sPath) Then Exit MsgBox($MB_SYSTEMMODAL, 'Error', 'Unable to access folder')

Local $hDirectory = _WinAPI_CreateFileEx($g_sPath, $OPEN_EXISTING, $GENERIC_READ+$GENERIC_WRITE, $FILE_SHARE_READ+$FILE_SHARE_WRITE, $FILE_FLAG_BACKUP_SEMANTICS)
If @error Then Exit MsgBox($MB_SYSTEMMODAL, 'Error', 'Unable to get handle')

Local $pBuffer = _WinAPI_CreateBuffer($nBufferLen), $aData
While True
  $aData = _WinAPI_ReadDirectoryChanges($hDirectory, _
    $FILE_NOTIFY_CHANGE_FILE_NAME + _
    $FILE_NOTIFY_CHANGE_DIR_NAME + _
    $FILE_NOTIFY_CHANGE_ATTRIBUTES + _
    $FILE_NOTIFY_CHANGE_SIZE + _
    $FILE_NOTIFY_CHANGE_LAST_WRITE + _
    $FILE_NOTIFY_CHANGE_LAST_ACCESS + _
    $FILE_NOTIFY_CHANGE_CREATION + _
    $FILE_NOTIFY_CHANGE_SECURITY, $pBuffer, $nBufferLen, 1)
  If Not @error Then
    _ArrayDisplay($aData, '_WinAPI_ReadDirectoryChanges')
  Else
    _WinAPI_ShowLastError('', 1)
  EndIf
WEnd

Func _Exit ()
  Exit
EndFunc

I have been using it successfully for quite some times now.  Downside, it is a blocking function.

Edited February 4, 2020 by Nine

[emendelson 8]

@Nine - I think this gives me exactly what I need; thank you! Questions

1. When you say this is a blocking function, I think that means that it blocks keystrokes to the script while the function is running. But my script is supposed to run completely silently, displaying error messages and prompts to overwrite an existing file, and nothing else. Am I right in thinking that the blocking function won't get in the way?

2. And this is a beginner's question (and I'm embarrassed to be such a beginner that I ask it). I'm trying to use _ArrayExtract to get the filename into a variable, but the best I can do is this (based on someone else's code), which displays the filename in an array but doesn't extract it to a variable.

If some generous person is willing to let me know how to extract that cell to a variable, I'll be very grateful.

For $i = 0 To (UBound($aData) - 1)
            For $j = 0 To 1
                $aData[$i][$j] = $i & $j
            Next
            Local $aExtract = _ArrayExtract($aData, 1, 1, 0, 0)
            _ArrayDisplay($aExtract, "Filename")
Next

 

[Nine 1,515]

1- yes you are right if it is to run silently

2- Just do what you want to do in your loop, something like this :

Local $sFileName 
For $i = 1 To $aData[0][0]
  $sFileName = $g_sPath & $aData[$i][0]
  If FileExists ($sFileName) Then
    _PrintThatFile ($sFileName)
    FileDelete ($sFileName)
  EndIf
Next

 

Edited February 4, 2020 by Nine
added folder path

[emendelson 8]

@Nine - Thank you! That is exactly what I was struggling to figure out. (And I realize that anyone who knows somethingabout arrays in AutoIt could have figured it out very quickly. It's time I learned more...)

[emendelson 8]

@Nine - One more question. I've got my script working with your method, and doing exactly what I want it to do - with one exception. My original script used a ToolTip that allowed the user to Exit the script and also to give information on which folder is being watched, etc. As far as I can tell, the ToolTip doesn't work with your file watcher. Is that because it's a blocking function? If so, do you (or anyone) know a way to use a ToolTip with this? It's not crucial that I have a ToolTip, but it's a convenience for other users who will have this script.

Thank you again for this.

[argumentum 835]

31 minutes ago, emendelson said:

My original script used a ToolTip

...this is the reason for the IPC. Run @Nine's script and send the array via MailSlot to another "main" script, where you can be free to do anything you'd like.

[emendelson 8]

11 minutes ago, argumentum said:

...this is the reason for the IPC. Run @Nine's script and send the array via MailSlot to another "main" script, where you can be free to do anything you'd like.

Thank you for this. I was beginning to see that I would need to do this. Does this mean that I'll need two executables, or that it's possible to to have them both in the same script/executable file.

[Nine 1,515]

Yes it is because it is a blocking function. But how do you trigger that tooltip ?  You said you want it to be ran silently, but I believe it could be very easy to implement.

[emendelson 8]

@Nine - I was being ignorant when I wrote that. I simply didn't realize that having a ToolTip meant that my script was NOT silent... Apologies for the confusion! I tried a few ways to implement it, but wasn't able to make it work...