FileReadLine() Performance Issue

[flyingboz]

Sample code follows

I start losing around 4 sec for every 100 lines of text I process ; and the amount of time it takes goes up by around

3 seconds for each additional 100 lines of text.

Using TimerDiff and TimerInit, I've traced the source of the problem to the FileReadLine() function.

Suggestions? Can others verify this behavior on a (preferably large) dataset? I am using 3.2.2.0

$fh_in = FileOpen($file_in,0) ; opens input file for reading
$fh_out = FileOpen($file_out,1) ; opens output file for writing

$i = 1
while 1
  $line = FileReadLine($fh_in,$i)
  if @error = -1 then ExitLoop
  $line = StringSplit($line)
   For $j = 1 to $line[0]
   ; do stuff to the fields of line to create record variable
   Next
   FileWriteLine($fh_out,$record)
   $i +=1
Wend

[herewasplato]

From the help file under FileReadLine:

From a performance standpoint it is a bad idea to read line by line specifying "line" parameter whose value is incrementing by one. This forces AutoIt to reread the file from the beginning until it reach the specified line.

Might I suggest FileRead?

Edit:

$fh_in = FileOpen($file_in, 0) ; opens input file for reading
$fh_out = FileOpen($file_out, 1) ; opens output file for writing

$file_array = StringSplit(FileRead($fh_in), @CR, 1)
;or maybe
;$file_array = StringSplit(FileRead($fh_in), @CRLF, 1)

For $i = 1 To $file_array[0]
    $line = StringSplit($file_array[$i], ",", 1)
    For $j = 1 To $line[0]
        ; do stuff to the fields of line to create record variable
    Next
    FileWriteLine($fh_out, $record)
Next

...and there are faster ways than using FileWriteLine... but you will need to provide a sample input file and the desired output file for us to give you a working snippet of code. Edited April 20, 2007 by herewasplato

[GEOSoft]

If you read the help file you will find the reason for that. FileReadLineis not really a good choice.

Better solution

#include<File.au3>

$Array = ''

_FileReadToArray($FileIn,$Array)

While 1

Etc

[WeMartiansAreFriendly]

From the help file under FileReadLine:

Might I suggest FileRead?

Edit:

$fh_in = FileOpen($file_in, 0) ; opens input file for reading
$fh_out = FileOpen($file_out, 1) ; opens output file for writing

$file_array = StringSplit(FileRead($fh_in), @CR, 1)
;or maybe
;$file_array = StringSplit(FileRead($fh_in), @CRLF, 1)

For $i = 1 To $file_array[0]
    $line = StringSplit($file_array[$i], ",", 1)
    For $j = 1 To $line[0]
        ; do stuff to the fields of line to create record variable
    Next
    FileWriteLine($fh_out, $record)
Next

...and there are faster ways than using FileWriteLine... but you will need to provide a sample input file and the desired output file for us to give you a working snippet of code.

if you wanted.. you can turn filewriteline() to $string &= $record & @crlf, and after the loop write the $string variable.. to output file.. from my experience that speeds things up consetterbly..

i have yet to see why you would need filewriteline(), readline() in a loop.... my guess is autotit counts the text lines from the top, so every 1+ increment that surely slows things down

Edited April 20, 2007 by mrRevoked

[xcal]

If you read the help file you will find the reason for that. FileReadLineis not really a good choice.

Better solution

#include<File.au3>

$Array = ''

_FileReadToArray($FileIn,$Array)

While 1

Etc

You may want to have a closer look at the _FileReadToArray function (in the include file), since you're calling that a 'better' solution. Just saying, since you're following herewasplato's post.

Edited April 20, 2007 by xcal

[flyingboz]

i have yet to see why you would need filewriteline(), readline() in a loop....

<paraphrased> Just use an array, stupid!

the answer to both the above is because the datasets are so large that storing them in memory arrays is impractical for the hardware I have to work with.

<edit: found the scroll bar on the help file...> Will try w/o specifying the line number.

Edited April 20, 2007 by flyingboz

[flyingboz]

When the line number is removed from the code, each iteration performs linearly, without a dependency on file size.

[WeMartiansAreFriendly]

ahh i see, well you could always add it to an array, once it reaches a certain length write that to the disk, and so on.

[WeMartiansAreFriendly]

like so.. i havent tested but it should work..

$maxlen = 10000
$fh_in = FileOpen($file_in, 0); opens input file for reading
$fh_out = FileOpen($file_out, 1); opens output file for writing

$file_array = StringSplit(StringStripCR(FileRead($fh_in)), @LF, 1)



For $i = 1 To $file_array[0]
    $line = StringSplit($file_array[$i], ",", 1)
    For $j = 1 To $line[0]
       ; do stuff to the fields of line to create record variable
    $record &= $file_array[$i]
    Next
    If StringLen($record) >= $maxlen Then
        FileWriteLine($fh_out, $record)
        $record = ''";flush $record
    EndIf
Next