ARB Posted February 1, 2011 Share Posted February 1, 2011 Hopefuly this is the correct place to post this question. I am attempting to loop through a directory opening each file (xml) reading the second line, reformat the content into SQLlite table, closing the file and going back aroud the loop. There could be 30000+ files in the directory. The problem the file fails to open at file 1568. There are several thousand mor to process. Is this bug or limitaion of the fileopen keyword? AlanB Link to comment Share on other sites More sharing options...
Moderators Melba23 Posted February 1, 2011 Moderators Share Posted February 1, 2011 ARB,Welcome to the AutoIt forum. Without seeing your code, I would suggest that you are not closing all the file handles after you have finished with the files. A quick way to check would be to use ConsoleWrite to see the handle returned by FileOpen - if it increases, you are leaving handles open.Or post your code we can take a closer look, but please use Code tags. Put [autoit ] before and [/autoit ] after your posted code (but omit the trailing space - it is only there so the tags display here). M23 Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind Open spoiler to see my UDFs: Spoiler ArrayMultiColSort ---- Sort arrays on multiple columnsChooseFileFolder ---- Single and multiple selections from specified path treeview listingDate_Time_Convert -- Easily convert date/time formats, including the language usedExtMsgBox --------- A highly customisable replacement for MsgBoxGUIExtender -------- Extend and retract multiple sections within a GUIGUIFrame ---------- Subdivide GUIs into many adjustable framesGUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView itemsGUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeViewMarquee ----------- Scrolling tickertape GUIsNoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxesNotify ------------- Small notifications on the edge of the displayScrollbars ----------Automatically sized scrollbars with a single commandStringSize ---------- Automatically size controls to fit textToast -------------- Small GUIs which pop out of the notification area Link to comment Share on other sites More sharing options...
ARB Posted February 1, 2011 Author Share Posted February 1, 2011 Hi MVP Thanks for your comments here is a code extract. The functions have not been included, _write is just a consolewrite with linefeed, _xmldata() just returns the value for a tag. Cheers AlanB *** Code *** $fld = StringSplit('date,time,ticket,value',",") $filename = "EX3_SCHSTS*.xml" $FileList =_FileListToArray($ftpinlcd , $filename) _write("Filelist = " & $filelist[0]) ;#cs $search = FileFindFirstFile($ftpinlcd & "*.xml") ; Check if the search was successful If $search = -1 Then MsgBox(0, "Error", "No files/directories matched the search pattern") Exit EndIf $cnt = 1 While 1 $xml = "" $os = "" $fileo = "" $file = FileFindNextFile($search) If @error Then ExitLoop $fileo = FileOpen($file, 0) ; Check if file opened for reading OK If $fileo = -1 Then MsgBox(0, "Error", "Unable to open file." & @lf & $file) Exit EndIf _write("processing file " & $file) ; Read in 1 character at a time until the EOF is reached $xml = FileRead($fileo) ;_write("Xml value " & $xml) for $i = 1 to $fld[0] $r = _XMLData($xml,$fld[$i]) if $i = 3 Then $p = StringInStr($r,">") ;_write($p) $s = StringMid($r,$p + 1) ;_write($s) $p = StringInStr($s,"<") ;_write($p) $s = StringLeft($s,$p - 1) ;_write("ticket[" & $fld[$i] & "] = " & $s) $os &= $s & "," Else $os &= $r & "," EndIf Next $os &= $file _write($cnt & "os = " & $os) ;_fillline($os) FileClose($fileo) $cnt += 1 WEnd FileClose($search) Exit Link to comment Share on other sites More sharing options...
ARB Posted February 1, 2011 Author Share Posted February 1, 2011 Hi Here is a console extract showing file handle up to the point of failure. Cheers AlanB processing file EX3_SCHSTS_001732538.xml 2166os = 0,0,,0,EX3_SCHSTS_001732538.xml Fileo Handle = 2 processing file EX3_SCHSTS_001732540.xml 2167os = 0,0,,0,EX3_SCHSTS_001732540.xml Fileo Handle = 2 processing file EX3_SCHSTS_001732542.xml 2168os = 0,0,,0,EX3_SCHSTS_001732542.xml Fileo Handle = 2 processing file EX3_SCHSTS_001732544.xml 2169os = 0,0,,0,EX3_SCHSTS_001732544.xml Fileo Handle = 2 processing file EX3_SCHSTS_001732546.xml 2170os = 0,0,,0,EX3_SCHSTS_001732546.xml Fileo Handle = -1 Link to comment Share on other sites More sharing options...
Moderators Melba23 Posted February 1, 2011 Moderators Share Posted February 1, 2011 ARB,With those values you are definitely closing the handles - as your script also shows. Remember that a return of -1 merely means that the FileOpen failed - it gives you no idea why. As the Help file points out: "A file may fail to open due to access rights or attributes"Could this be the answer? I notice that in your post above it was file #2170 that caused the problem - not #1568 as you indicated in your first post. Was this in a different folder?There certainly is no bug in FileOpen - I use it regularly on 7000+ files with no problem. M23P.S. And on an entirely different note, why are you using FileFindFirst/NextFile anyway? Why not loop through the result of a simple _FileListToArray call? Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind Open spoiler to see my UDFs: Spoiler ArrayMultiColSort ---- Sort arrays on multiple columnsChooseFileFolder ---- Single and multiple selections from specified path treeview listingDate_Time_Convert -- Easily convert date/time formats, including the language usedExtMsgBox --------- A highly customisable replacement for MsgBoxGUIExtender -------- Extend and retract multiple sections within a GUIGUIFrame ---------- Subdivide GUIs into many adjustable framesGUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView itemsGUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeViewMarquee ----------- Scrolling tickertape GUIsNoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxesNotify ------------- Small notifications on the edge of the displayScrollbars ----------Automatically sized scrollbars with a single commandStringSize ---------- Automatically size controls to fit textToast -------------- Small GUIs which pop out of the notification area Link to comment Share on other sites More sharing options...
MvGulik Posted February 1, 2011 Share Posted February 1, 2011 (edited) whatever Edited February 7, 2011 by MvGulik "Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions.""The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014) "Believing what you know ain't so" ... Knock Knock ... Link to comment Share on other sites More sharing options...
Moderators Melba23 Posted February 1, 2011 Moderators Share Posted February 1, 2011 MvGulik,what script?There is enough in that short extract to show that every file opened is closed - besides the handle is not increasing, as demonstrated in the printout: 1 = search handle from FileFindFirstFile, 2 = handle from FileOpen. M23 Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind Open spoiler to see my UDFs: Spoiler ArrayMultiColSort ---- Sort arrays on multiple columnsChooseFileFolder ---- Single and multiple selections from specified path treeview listingDate_Time_Convert -- Easily convert date/time formats, including the language usedExtMsgBox --------- A highly customisable replacement for MsgBoxGUIExtender -------- Extend and retract multiple sections within a GUIGUIFrame ---------- Subdivide GUIs into many adjustable framesGUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView itemsGUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeViewMarquee ----------- Scrolling tickertape GUIsNoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxesNotify ------------- Small notifications on the edge of the displayScrollbars ----------Automatically sized scrollbars with a single commandStringSize ---------- Automatically size controls to fit textToast -------------- Small GUIs which pop out of the notification area Link to comment Share on other sites More sharing options...
MvGulik Posted February 1, 2011 Share Posted February 1, 2011 (edited) whatever Edited February 7, 2011 by MvGulik "Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions.""The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014) "Believing what you know ain't so" ... Knock Knock ... Link to comment Share on other sites More sharing options...
MvGulik Posted February 2, 2011 Share Posted February 2, 2011 (edited) whatever Edited February 7, 2011 by MvGulik "Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions.""The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014) "Believing what you know ain't so" ... Knock Knock ... Link to comment Share on other sites More sharing options...
ARB Posted February 2, 2011 Author Share Posted February 2, 2011 Hi Guys Many thanks for your comments, here are my replies to the points raised. I have written many interface programs over the last 4 years using AutoIT and this is the first time that I have encountered a problem I could not resolve. It is also the first time that I have attempted to process a large number of files at once. I have written a version using _FileListToArray() but I had the same problem. The data files are imported to my local drive via ftp. I am an admin, therfore have full control of all data. The confussion over the file entry failure was in my haste to post I just copied the last entries. Extract of the Console output: #1566 $os = 28-01-2011,1544,42006696,0500,EX3_SCHSTS_001727449.xml Fileo Handle = 2 processing file EX3_SCHSTS_001727450.xml #1567 $os = 28-01-2011,1544,42006700,0500,EX3_SCHSTS_001727450.xml Fileo Handle = 2 processing file EX3_SCHSTS_001727451.xml #1568 $os = 0,0,,0,EX3_SCHSTS_001727451.xml Below are the files prior to and the point of failure. Prior File content: <?xml version="1.0" encoding="utf-8" ?> - <rmc> <date>28-01-2011</date> <time>1544</time> <summit_co>01</summit_co> <summit_module>jtent</summit_module> <mode>A</mode> - <ticket> <field id="ticket_no">42006714</field> - <field type="mtext" id="jtca_notes"> <text /> </field> - <field id="status"> <value>0500</value> </field> </ticket> </rmc> Error File content: <?xml version="1.0" encoding="utf-8" ?> - <rmc> <date>28-01-2011</date> <time>1544</time> <summit_co>01</summit_co> <summit_module>jtent</summit_module> <mode>A</mode> - <ticket> <field id="ticket_no">42006723</field> - <field type="mtext" id="jtca_notes"> <text /> </field> - <field id="status"> <value>0500</value> </field> </ticket> </rmc> Maybe the fileopen error is a symptom if my previous calls to extract the values or the tags from the xml. here is the function, it may help ;********************************************************************************************* ;$s = String ;$t = Tag ; returns array of values between start and end tags ;********************************************************************************************* Func _XMLData($s,$t) dim $a[1] $nOffset = 1 While 1 $array = StringRegExp($s, '<(?i)' & $t & '>(.*?)</(?i)' & $t & '>', 1, $nOffset) If @error = 0 Then $nOffset = @extended Else ExitLoop EndIf _ArrayAdd($a,$array[0]) return($array[0]) WEnd _ArrayDelete($a,0) ;_ArrayDisplay($a) EndFunc Cheers AlanB Link to comment Share on other sites More sharing options...
Moderators Melba23 Posted February 2, 2011 Moderators Share Posted February 2, 2011 ARB,Please, please use code tags as I suggested in my first post - put [autoit ] before and [/autoit ] after your posted code (but omit the trailing space - it is only there so the tags display here. It makes the whole thread much easier to read. I have just created 30000 files using the format you have just posted - only the <field id="ticket_no">42006723</field> line changed to reflect the ID number of the file. Using the following code (based on what you have posted so far) I got all 30000 to open/close without problem - several times:expandcollapse popup#include <Array.au3> #include <File.au3> $fld = StringSplit('date,time,ticket,value', ",") $filename = "*.txt" $FileList = _FileListToArray(@ScriptDir, $filename) MsgBox(0, "Found", $FileList[0]) For $k = 1 To $FileList[0] $xml = "" $os = "" $fileo = "" $file = $FileList[$k] $fileo = FileOpen($file, 0) ; Check if file opened for reading OK If $fileo = -1 Then MsgBox(0, "Error", "Unable to open file " & $k & " " & $file) Exit EndIf ;ConsoleWrite("processing file " & $file & @CRLF) If $fileo <> 1 Then ConsoleWrite("File handle = " & $fileo & @CRLF) ; Read in 1 character at a time until the EOF is reached $xml = FileRead($fileo) ;_write("Xml value " & $xml) For $i = 1 To $fld[0] $r = _XMLData($xml, $fld[$i]) If $i = 3 Then $p = StringInStr($r, ">") ;_write($p) $s = StringMid($r, $p + 1) ;_write($s) $p = StringInStr($s, "<") ;_write($p) $s = StringLeft($s, $p - 1) ;_write("ticket[" & $fld[$i] & "] = " & $s) $os &= $s & "," Else $os &= $r & "," EndIf Next $os &= $file ConsoleWrite($k & " os = " & $os & @CRLF) ;_fillline($os) FileClose($fileo) Next Exit ;********************************************************************************************* ;$s = String ;$t = Tag ; returns array of values between start and end tags ;********************************************************************************************* Func _XMLData($s, $t) Dim $a[1] $nOffset = 1 While 1 $array = StringRegExp($s, '<(?i)' & $t & '>(.*?)</(?i)' & $t & '>', 1, $nOffset) If @error = 0 Then $nOffset = @extended Else ExitLoop EndIf _ArrayAdd($a, $array[0]) Return ($array[0]) WEnd _ArrayDelete($a, 0) ;_ArrayDisplay($a) EndFunc ;==>_XMLDataHere is the final section of the printout:29991 os = 28-01-2011,1544,,0500,File_00029991.txt 29992 os = 28-01-2011,1544,,0500,File_00029992.txt 29993 os = 28-01-2011,1544,,0500,File_00029993.txt 29994 os = 28-01-2011,1544,,0500,File_00029994.txt 29995 os = 28-01-2011,1544,,0500,File_00029995.txt 29996 os = 28-01-2011,1544,,0500,File_00029996.txt 29997 os = 28-01-2011,1544,,0500,File_00029997.txt 29998 os = 28-01-2011,1544,,0500,File_00029998.txt 29999 os = 28-01-2011,1544,,0500,File_00029999.txt 30000 os = 28-01-2011,1544,,0500,File_00030000.txt +>11:10:36 AutoIT3.exe ended.rc:0 >Exit code: 0 Time: 24.467So you must have something else which is preventing the FileOpen from working - certainly it does not appear to be AutoIt related.Is it always the same file that blocks? If you amend your code to skip over the file that does not want to open, can you still open later files in the queue? That would narrow it down a bit. I will keep thinking about other possibilites. M23 Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind Open spoiler to see my UDFs: Spoiler ArrayMultiColSort ---- Sort arrays on multiple columnsChooseFileFolder ---- Single and multiple selections from specified path treeview listingDate_Time_Convert -- Easily convert date/time formats, including the language usedExtMsgBox --------- A highly customisable replacement for MsgBoxGUIExtender -------- Extend and retract multiple sections within a GUIGUIFrame ---------- Subdivide GUIs into many adjustable framesGUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView itemsGUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeViewMarquee ----------- Scrolling tickertape GUIsNoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxesNotify ------------- Small notifications on the edge of the displayScrollbars ----------Automatically sized scrollbars with a single commandStringSize ---------- Automatically size controls to fit textToast -------------- Small GUIs which pop out of the notification area Link to comment Share on other sites More sharing options...
ARB Posted February 2, 2011 Author Share Posted February 2, 2011 Hi Guys Many thanks for all you help. I have found the problem, and it was in my code. I was not adding the full path and the filename in the file open statement. The other thing that was causing consfusion was that I was getting firewall issues while ftp'ing the files to my PC I think it thought that it was getting a denial of service attack. This resulted in a quantity of files ending up in the script directory, many with zero bytes thus the script was stalling with the file open error. Anyway, thanks again for all your comments. Regards Alan Link to comment Share on other sites More sharing options...
Moderators Melba23 Posted February 2, 2011 Moderators Share Posted February 2, 2011 ARB, Glad you got it running! M23 Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind Open spoiler to see my UDFs: Spoiler ArrayMultiColSort ---- Sort arrays on multiple columnsChooseFileFolder ---- Single and multiple selections from specified path treeview listingDate_Time_Convert -- Easily convert date/time formats, including the language usedExtMsgBox --------- A highly customisable replacement for MsgBoxGUIExtender -------- Extend and retract multiple sections within a GUIGUIFrame ---------- Subdivide GUIs into many adjustable framesGUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView itemsGUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeViewMarquee ----------- Scrolling tickertape GUIsNoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxesNotify ------------- Small notifications on the edge of the displayScrollbars ----------Automatically sized scrollbars with a single commandStringSize ---------- Automatically size controls to fit textToast -------------- Small GUIs which pop out of the notification area Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now