Is this a Bug? - FileOpen

ARB · February 1, 2011

Hopefuly this is the correct place to post this question.

I am attempting to loop through a directory opening each file (xml) reading the second line, reformat the content into SQLlite table, closing the file and going back aroud the loop.

There could be 30000+ files in the directory.

The problem the file fails to open at file 1568. There are several thousand mor to process.

Is this bug or limitaion of the fileopen keyword?

AlanB

Melba23 · February 1, 2011

ARB,

Welcome to the AutoIt forum.

Without seeing your code, I would suggest that you are not closing all the file handles after you have finished with the files.

A quick way to check would be to use ConsoleWrite to see the handle returned by FileOpen - if it increases, you are leaving handles open.

Or post your code we can take a closer look, but please use Code tags. Put [autoit ] before and [/autoit ] after your posted code (but omit the trailing space - it is only there so the tags display here). :idiot:

M23

ARB · February 1, 2011

Hi MVP

Thanks for your comments here is a code extract. The functions have not been included, _write is just a consolewrite with linefeed, _xmldata() just returns the value for a tag.

Cheers

AlanB

*** Code ***

$fld = StringSplit('date,time,ticket,value',",")

$filename = "EX3_SCHSTS*.xml"

$FileList =_FileListToArray($ftpinlcd , $filename)

_write("Filelist = " & $filelist[0])

;#cs

$search = FileFindFirstFile($ftpinlcd & "*.xml")

; Check if the search was successful

If $search = -1 Then

MsgBox(0, "Error", "No files/directories matched the search pattern")

Exit

EndIf

$cnt = 1

While 1

$xml = ""

$os = ""

$fileo = ""

$file = FileFindNextFile($search)

If @error Then ExitLoop

$fileo = FileOpen($file, 0)

; Check if file opened for reading OK

If $fileo = -1 Then

MsgBox(0, "Error", "Unable to open file." & @lf & $file)

Exit

EndIf

_write("processing file " & $file)

; Read in 1 character at a time until the EOF is reached

$xml = FileRead($fileo)

;_write("Xml value " & $xml)

for $i = 1 to $fld[0]

$r = _XMLData($xml,$fld[$i])

if $i = 3 Then

$p = StringInStr($r,">")

;_write($p)

$s = StringMid($r,$p + 1)

;_write($s)

$p = StringInStr($s,"<")

;_write($p)

$s = StringLeft($s,$p - 1)

;_write("ticket[" & $fld[$i] & "] = " & $s)

$os &= $s & ","

Else

$os &= $r & ","

EndIf

_write($cnt & "os = " & $os)

;_fillline($os)

FileClose($fileo)

$cnt += 1

WEnd

FileClose($search)

Exit

ARB · February 1, 2011

Hi

Here is a console extract showing file handle up to the point of failure.

Cheers

AlanB

processing file EX3_SCHSTS_001732538.xml

2166os = 0,0,,0,EX3_SCHSTS_001732538.xml

Fileo Handle = 2

processing file EX3_SCHSTS_001732540.xml

2167os = 0,0,,0,EX3_SCHSTS_001732540.xml

Fileo Handle = 2

processing file EX3_SCHSTS_001732542.xml

2168os = 0,0,,0,EX3_SCHSTS_001732542.xml

Fileo Handle = 2

processing file EX3_SCHSTS_001732544.xml

2169os = 0,0,,0,EX3_SCHSTS_001732544.xml

Fileo Handle = 2

processing file EX3_SCHSTS_001732546.xml

2170os = 0,0,,0,EX3_SCHSTS_001732546.xml

Fileo Handle = -1

Melba23 · February 1, 2011

ARB,

With those values you are definitely closing the handles - as your script also shows.

Remember that a return of -1 merely means that the FileOpen failed - it gives you no idea why. As the Help file points out: "A file may fail to open due to access rights or attributes"

Could this be the answer? I notice that in your post above it was file #2170 that caused the problem - not #1568 as you indicated in your first post. Was this in a different folder?

There certainly is no bug in FileOpen - I use it regularly on 7000+ files with no problem. :idiot:

M23

P.S. And on an entirely different note, why are you using FileFindFirst/NextFile anyway? Why not loop through the result of a simple _FileListToArray call?

MvGulik · February 1, 2011

whatever Edited February 7, 2011 by MvGulik

Melba23 · February 1, 2011

MvGulik,

what script?

There is enough in that short extract to show that every file opened is closed - besides the handle is not increasing, as demonstrated in the printout: 1 = search handle from FileFindFirstFile, 2 = handle from FileOpen.

M23

MvGulik · February 1, 2011

whatever Edited February 7, 2011 by MvGulik

MvGulik · February 2, 2011

whatever Edited February 7, 2011 by MvGulik

ARB · February 2, 2011

Hi Guys

Many thanks for your comments, here are my replies to the points raised. I have written many interface programs over the last 4 years using AutoIT and this is the first time that I have encountered a problem I could not resolve. It is also the first time that I have attempted to process a large number of files at once.

I have written a version using _FileListToArray() but I had the same problem.

The data files are imported to my local drive via ftp. I am an admin, therfore have full control of all data.

The confussion over the file entry failure was in my haste to post I just copied the last entries.

Extract of the Console output:

#1566 $os = 28-01-2011,1544,42006696,0500,EX3_SCHSTS_001727449.xml

Fileo Handle = 2

processing file EX3_SCHSTS_001727450.xml

#1567 $os = 28-01-2011,1544,42006700,0500,EX3_SCHSTS_001727450.xml

Fileo Handle = 2

processing file EX3_SCHSTS_001727451.xml

#1568 $os = 0,0,,0,EX3_SCHSTS_001727451.xml

Below are the files prior to and the point of failure.

Prior File content:

<?xml version="1.0" encoding="utf-8" ?>

- <rmc>

<summit_co>01</summit_co>

<summit_module>jtent</summit_module>

- <ticket>

- <field type="mtext" id="jtca_notes">

</field>

- <field id="status">

</field>

</ticket>

</rmc>

Error File content:

<?xml version="1.0" encoding="utf-8" ?>

- <rmc>

<summit_co>01</summit_co>

<summit_module>jtent</summit_module>

- <ticket>

- <field type="mtext" id="jtca_notes">

</field>

- <field id="status">

</field>

</ticket>

</rmc>

Maybe the fileopen error is a symptom if my previous calls to extract the values or the tags from the xml.

here is the function, it may help

;*********************************************************************************************

;$s = String

;$t = Tag

; returns array of values between start and end tags

;*********************************************************************************************

Func _XMLData($s,$t)

dim $a[1]

$nOffset = 1

While 1

$array = StringRegExp($s, '<(?i)' & $t & '>(.*?)</(?i)' & $t & '>', 1, $nOffset)

If @error = 0 Then

$nOffset = @extended

Else

ExitLoop

EndIf

_ArrayAdd($a,$array[0])

return($array[0])

WEnd

_ArrayDelete($a,0)

;_ArrayDisplay($a)

EndFunc

Cheers

AlanB

Melba23 · February 2, 2011

ARB,

Please, please use code tags as I suggested in my first post - put [autoit ] before and [/autoit ] after your posted code (but omit the trailing space - it is only there so the tags display here. It makes the whole thread much easier to read.

I have just created 30000 files using the format you have just posted - only the <field id="ticket_no">42006723</field> line changed to reflect the ID number of the file. Using the following code (based on what you have posted so far) I got all 30000 to open/close without problem - several times:

#include <Array.au3>
#include <File.au3>

$fld = StringSplit('date,time,ticket,value', ",")

$filename = "*.txt"
$FileList = _FileListToArray(@ScriptDir, $filename)

MsgBox(0, "Found", $FileList[0])

For $k = 1 To $FileList[0]

    $xml = ""
    $os = ""
    $fileo = ""
    $file = $FileList[$k]
    $fileo = FileOpen($file, 0)
    ; Check if file opened for reading OK
    If $fileo = -1 Then
        MsgBox(0, "Error", "Unable to open file " & $k & " " & $file)
        Exit
    EndIf
    ;ConsoleWrite("processing file " & $file & @CRLF)
    If $fileo <> 1 Then ConsoleWrite("File handle = " & $fileo & @CRLF)
    ; Read in 1 character at a time until the EOF is reached
    $xml = FileRead($fileo)
    ;_write("Xml value " & $xml)
    For $i = 1 To $fld[0]
        $r = _XMLData($xml, $fld[$i])
        If $i = 3 Then
            $p = StringInStr($r, ">")
            ;_write($p)
            $s = StringMid($r, $p + 1)
            ;_write($s)
            $p = StringInStr($s, "<")
            ;_write($p)
            $s = StringLeft($s, $p - 1)
            ;_write("ticket[" & $fld[$i] & "] = " & $s)
            $os &= $s & ","
        Else
            $os &= $r & ","
        EndIf
    Next
    $os &= $file
    ConsoleWrite($k & " os = " & $os & @CRLF)
    ;_fillline($os)
    FileClose($fileo)
Next

Exit

;*********************************************************************************************
;$s = String
;$t = Tag
; returns array of values between start and end tags
;*********************************************************************************************
Func _XMLData($s, $t)
    Dim $a[1]
    $nOffset = 1
    While 1
        $array = StringRegExp($s, '<(?i)' & $t & '>(.*?)</(?i)' & $t & '>', 1, $nOffset)
        If @error = 0 Then
            $nOffset = @extended
        Else
            ExitLoop
        EndIf
        _ArrayAdd($a, $array[0])
        Return ($array[0])
    WEnd
    _ArrayDelete($a, 0)
    ;_ArrayDisplay($a)
EndFunc   ;==>_XMLData

Here is the final section of the printout:

29991 os = 28-01-2011,1544,,0500,File_00029991.txt
29992 os = 28-01-2011,1544,,0500,File_00029992.txt
29993 os = 28-01-2011,1544,,0500,File_00029993.txt
29994 os = 28-01-2011,1544,,0500,File_00029994.txt
29995 os = 28-01-2011,1544,,0500,File_00029995.txt
29996 os = 28-01-2011,1544,,0500,File_00029996.txt
29997 os = 28-01-2011,1544,,0500,File_00029997.txt
29998 os = 28-01-2011,1544,,0500,File_00029998.txt
29999 os = 28-01-2011,1544,,0500,File_00029999.txt
30000 os = 28-01-2011,1544,,0500,File_00030000.txt
+>11:10:36 AutoIT3.exe ended.rc:0
>Exit code: 0    Time: 24.467

So you must have something else which is preventing the FileOpen from working - certainly it does not appear to be AutoIt related.

Is it always the same file that blocks? If you amend your code to skip over the file that does not want to open, can you still open later files in the queue? That would narrow it down a bit. :idiot:

I will keep thinking about other possibilites.

M23

ARB · February 2, 2011

Hi Guys

Many thanks for all you help.

I have found the problem, and it was in my code. I was not adding the full path and the filename in the file open statement. The other thing that was causing consfusion was that I was getting firewall issues while ftp'ing the files to my PC I think it thought that it was getting a denial of service attack. This resulted in a quantity of files ending up in the script directory, many with zero bytes thus the script was stalling with the file open error.

Anyway, thanks again for all your comments.

Regards

Alan

Melba23 · February 2, 2011

ARB,

Glad you got it running!

M23

Is this a Bug? - FileOpen

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members