Sign in to follow this  
Followers 0
ARB

Is this a Bug? - FileOpen

13 posts in this topic

Hopefuly this is the correct place to post this question.

I am attempting to loop through a directory opening each file (xml) reading the second line, reformat the content into SQLlite table, closing the file and going back aroud the loop.

There could be 30000+ files in the directory.

The problem the file fails to open at file 1568. There are several thousand mor to process.

Is this bug or limitaion of the fileopen keyword?

AlanB

Share this post


Link to post
Share on other sites



ARB,

Welcome to the AutoIt forum. :)

Without seeing your code, I would suggest that you are not closing all the file handles after you have finished with the files. ;)

A quick way to check would be to use ConsoleWrite to see the handle returned by FileOpen - if it increases, you are leaving handles open.

Or post your code we can take a closer look, but please use Code tags. Put [autoit ] before and [/autoit ] after your posted code (but omit the trailing space - it is only there so the tags display here). :idiot:

M23


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

Hi MVP

Thanks for your comments here is a code extract. The functions have not been included, _write is just a consolewrite with linefeed, _xmldata() just returns the value for a tag.

Cheers

AlanB

*** Code ***

$fld = StringSplit('date,time,ticket,value',",")

$filename = "EX3_SCHSTS*.xml"

$FileList =_FileListToArray($ftpinlcd , $filename)

_write("Filelist = " & $filelist[0])

;#cs

$search = FileFindFirstFile($ftpinlcd & "*.xml")

; Check if the search was successful

If $search = -1 Then

MsgBox(0, "Error", "No files/directories matched the search pattern")

Exit

EndIf

$cnt = 1

While 1

$xml = ""

$os = ""

$fileo = ""

$file = FileFindNextFile($search)

If @error Then ExitLoop

$fileo = FileOpen($file, 0)

; Check if file opened for reading OK

If $fileo = -1 Then

MsgBox(0, "Error", "Unable to open file." & @lf & $file)

Exit

EndIf

_write("processing file " & $file)

; Read in 1 character at a time until the EOF is reached

$xml = FileRead($fileo)

;_write("Xml value " & $xml)

for $i = 1 to $fld[0]

$r = _XMLData($xml,$fld[$i])

if $i = 3 Then

$p = StringInStr($r,">")

;_write($p)

$s = StringMid($r,$p + 1)

;_write($s)

$p = StringInStr($s,"<")

;_write($p)

$s = StringLeft($s,$p - 1)

;_write("ticket[" & $fld[$i] & "] = " & $s)

$os &= $s & ","

Else

$os &= $r & ","

EndIf

Next

$os &= $file

_write($cnt & "os = " & $os)

;_fillline($os)

FileClose($fileo)

$cnt += 1

WEnd

FileClose($search)

Exit

Share this post


Link to post
Share on other sites

Hi

Here is a console extract showing file handle up to the point of failure.

Cheers

AlanB

processing file EX3_SCHSTS_001732538.xml

2166os = 0,0,,0,EX3_SCHSTS_001732538.xml

Fileo Handle = 2

processing file EX3_SCHSTS_001732540.xml

2167os = 0,0,,0,EX3_SCHSTS_001732540.xml

Fileo Handle = 2

processing file EX3_SCHSTS_001732542.xml

2168os = 0,0,,0,EX3_SCHSTS_001732542.xml

Fileo Handle = 2

processing file EX3_SCHSTS_001732544.xml

2169os = 0,0,,0,EX3_SCHSTS_001732544.xml

Fileo Handle = 2

processing file EX3_SCHSTS_001732546.xml

2170os = 0,0,,0,EX3_SCHSTS_001732546.xml

Fileo Handle = -1

Share this post


Link to post
Share on other sites

ARB,

With those values you are definitely closing the handles - as your script also shows. ;)

Remember that a return of -1 merely means that the FileOpen failed - it gives you no idea why. As the Help file points out: "A file may fail to open due to access rights or attributes"

Could this be the answer? I notice that in your post above it was file #2170 that caused the problem - not #1568 as you indicated in your first post. Was this in a different folder?

There certainly is no bug in FileOpen - I use it regularly on 7000+ files with no problem. :idiot:

M23

P.S. And on an entirely different note, why are you using FileFindFirst/NextFile anyway? Why not loop through the result of a simple _FileListToArray call? :)


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

#6 ·  Posted (edited)

whatever Edited by MvGulik

"Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions."
"The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014)

"Believing what you know ain't so" ...

Knock Knock ...
 

Share this post


Link to post
Share on other sites

MvGulik,

what script?

There is enough in that short extract to show that every file opened is closed - besides the handle is not increasing, as demonstrated in the printout: 1 = search handle from FileFindFirstFile, 2 = handle from FileOpen. :)

M23


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

whatever Edited by MvGulik

"Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions."
"The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014)

"Believing what you know ain't so" ...

Knock Knock ...
 

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

whatever Edited by MvGulik

"Straight_and_Crooked_Thinking" : A "classic guide to ferreting out untruths, half-truths, and other distortions of facts in political and social discussions."
"The Secrets of Quantum Physics" : New and excellent 2 part documentary on Quantum Physics by Jim Al-Khalili. (Dec 2014)

"Believing what you know ain't so" ...

Knock Knock ...
 

Share this post


Link to post
Share on other sites

Hi Guys

Many thanks for your comments, here are my replies to the points raised. I have written many interface programs over the last 4 years using AutoIT and this is the first time that I have encountered a problem I could not resolve. It is also the first time that I have attempted to process a large number of files at once.

I have written a version using _FileListToArray() but I had the same problem.

The data files are imported to my local drive via ftp. I am an admin, therfore have full control of all data.

The confussion over the file entry failure was in my haste to post I just copied the last entries.

Extract of the Console output:

#1566 $os = 28-01-2011,1544,42006696,0500,EX3_SCHSTS_001727449.xml

Fileo Handle = 2

processing file EX3_SCHSTS_001727450.xml

#1567 $os = 28-01-2011,1544,42006700,0500,EX3_SCHSTS_001727450.xml

Fileo Handle = 2

processing file EX3_SCHSTS_001727451.xml

#1568 $os = 0,0,,0,EX3_SCHSTS_001727451.xml

Below are the files prior to and the point of failure.

Prior File content:

<?xml version="1.0" encoding="utf-8" ?>

- <rmc>

<date>28-01-2011</date>

<time>1544</time>

<summit_co>01</summit_co>

<summit_module>jtent</summit_module>

<mode>A</mode>

- <ticket>

<field id="ticket_no">42006714</field>

- <field type="mtext" id="jtca_notes">

<text />

</field>

- <field id="status">

<value>0500</value>

</field>

</ticket>

</rmc>

Error File content:

<?xml version="1.0" encoding="utf-8" ?>

- <rmc>

<date>28-01-2011</date>

<time>1544</time>

<summit_co>01</summit_co>

<summit_module>jtent</summit_module>

<mode>A</mode>

- <ticket>

<field id="ticket_no">42006723</field>

- <field type="mtext" id="jtca_notes">

<text />

</field>

- <field id="status">

<value>0500</value>

</field>

</ticket>

</rmc>

Maybe the fileopen error is a symptom if my previous calls to extract the values or the tags from the xml.

here is the function, it may help

;*********************************************************************************************

;$s = String

;$t = Tag

; returns array of values between start and end tags

;*********************************************************************************************

Func _XMLData($s,$t)

dim $a[1]

$nOffset = 1

While 1

$array = StringRegExp($s, '<(?i)' & $t & '>(.*?)</(?i)' & $t & '>', 1, $nOffset)

If @error = 0 Then

$nOffset = @extended

Else

ExitLoop

EndIf

_ArrayAdd($a,$array[0])

return($array[0])

WEnd

_ArrayDelete($a,0)

;_ArrayDisplay($a)

EndFunc

Cheers

AlanB

Share this post


Link to post
Share on other sites

ARB,

Please, please use code tags as I suggested in my first post - put [autoit ] before and [/autoit ] after your posted code (but omit the trailing space - it is only there so the tags display here. It makes the whole thread much easier to read. ;)

I have just created 30000 files using the format you have just posted - only the <field id="ticket_no">42006723</field> line changed to reflect the ID number of the file. Using the following code (based on what you have posted so far) I got all 30000 to open/close without problem - several times:

#include <Array.au3>
#include <File.au3>

$fld = StringSplit('date,time,ticket,value', ",")

$filename = "*.txt"
$FileList = _FileListToArray(@ScriptDir, $filename)

MsgBox(0, "Found", $FileList[0])

For $k = 1 To $FileList[0]

    $xml = ""
    $os = ""
    $fileo = ""
    $file = $FileList[$k]
    $fileo = FileOpen($file, 0)
    ; Check if file opened for reading OK
    If $fileo = -1 Then
        MsgBox(0, "Error", "Unable to open file " & $k & " " & $file)
        Exit
    EndIf
    ;ConsoleWrite("processing file " & $file & @CRLF)
    If $fileo <> 1 Then ConsoleWrite("File handle = " & $fileo & @CRLF)
    ; Read in 1 character at a time until the EOF is reached
    $xml = FileRead($fileo)
    ;_write("Xml value " & $xml)
    For $i = 1 To $fld[0]
        $r = _XMLData($xml, $fld[$i])
        If $i = 3 Then
            $p = StringInStr($r, ">")
            ;_write($p)
            $s = StringMid($r, $p + 1)
            ;_write($s)
            $p = StringInStr($s, "<")
            ;_write($p)
            $s = StringLeft($s, $p - 1)
            ;_write("ticket[" & $fld[$i] & "] = " & $s)
            $os &= $s & ","
        Else
            $os &= $r & ","
        EndIf
    Next
    $os &= $file
    ConsoleWrite($k & " os = " & $os & @CRLF)
    ;_fillline($os)
    FileClose($fileo)
Next

Exit

;*********************************************************************************************
;$s = String
;$t = Tag
; returns array of values between start and end tags
;*********************************************************************************************
Func _XMLData($s, $t)
    Dim $a[1]
    $nOffset = 1
    While 1
        $array = StringRegExp($s, '<(?i)' & $t & '>(.*?)</(?i)' & $t & '>', 1, $nOffset)
        If @error = 0 Then
            $nOffset = @extended
        Else
            ExitLoop
        EndIf
        _ArrayAdd($a, $array[0])
        Return ($array[0])
    WEnd
    _ArrayDelete($a, 0)
    ;_ArrayDisplay($a)
EndFunc   ;==>_XMLData

Here is the final section of the printout:

29991 os = 28-01-2011,1544,,0500,File_00029991.txt
29992 os = 28-01-2011,1544,,0500,File_00029992.txt
29993 os = 28-01-2011,1544,,0500,File_00029993.txt
29994 os = 28-01-2011,1544,,0500,File_00029994.txt
29995 os = 28-01-2011,1544,,0500,File_00029995.txt
29996 os = 28-01-2011,1544,,0500,File_00029996.txt
29997 os = 28-01-2011,1544,,0500,File_00029997.txt
29998 os = 28-01-2011,1544,,0500,File_00029998.txt
29999 os = 28-01-2011,1544,,0500,File_00029999.txt
30000 os = 28-01-2011,1544,,0500,File_00030000.txt
+>11:10:36 AutoIT3.exe ended.rc:0
>Exit code: 0    Time: 24.467

So you must have something else which is preventing the FileOpen from working - certainly it does not appear to be AutoIt related.

Is it always the same file that blocks? If you amend your code to skip over the file that does not want to open, can you still open later files in the queue? That would narrow it down a bit. :idiot:

I will keep thinking about other possibilites. :)

M23


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

Hi Guys

Many thanks for all you help.

I have found the problem, and it was in my code. I was not adding the full path and the filename in the file open statement. The other thing that was causing consfusion was that I was getting firewall issues while ftp'ing the files to my PC I think it thought that it was getting a denial of service attack. This resulted in a quantity of files ending up in the script directory, many with zero bytes thus the script was stalling with the file open error.

Anyway, thanks again for all your comments.

Regards

Alan

Share this post


Link to post
Share on other sites

ARB,

Glad you got it running! :)

M23


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0