Jump to content
Sign in to follow this  
flyingboz

FileReadLine() Performance Issue

Recommended Posts

flyingboz

Sample code follows

I start losing around 4 sec for every 100 lines of text I process ; and the amount of time it takes goes up by around

3 seconds for each additional 100 lines of text.

Using TimerDiff and TimerInit, I've traced the source of the problem to the FileReadLine() function.

Suggestions? Can others verify this behavior on a (preferably large) dataset? I am using 3.2.2.0

$fh_in = FileOpen($file_in,0) ; opens input file for reading
$fh_out = FileOpen($file_out,1) ; opens output file for writing

$i = 1
while 1
  $line = FileReadLine($fh_in,$i)
  if @error = -1 then ExitLoop
  $line = StringSplit($line)
   For $j = 1 to $line[0]
   ; do stuff to the fields of line to create record variable
   Next
   FileWriteLine($fh_out,$record)
   $i +=1
Wend

Reading the help file before you post... Not only will it make you look smarter, it will make you smarter.

Share this post


Link to post
Share on other sites
herewasplato

From the help file under FileReadLine:

From a performance standpoint it is a bad idea to read line by line specifying "line" parameter whose value is incrementing by one. This forces AutoIt to reread the file from the beginning until it reach the specified line.

Might I suggest FileRead?

Edit:

$fh_in = FileOpen($file_in, 0) ; opens input file for reading
$fh_out = FileOpen($file_out, 1) ; opens output file for writing

$file_array = StringSplit(FileRead($fh_in), @CR, 1)
;or maybe
;$file_array = StringSplit(FileRead($fh_in), @CRLF, 1)

For $i = 1 To $file_array[0]
    $line = StringSplit($file_array[$i], ",", 1)
    For $j = 1 To $line[0]
        ; do stuff to the fields of line to create record variable
    Next
    FileWriteLine($fh_out, $record)
Next
...and there are faster ways than using FileWriteLine... but you will need to provide a sample input file and the desired output file for us to give you a working snippet of code. Edited by herewasplato

[size="1"][font="Arial"].[u].[/u][/font][/size]

Share this post


Link to post
Share on other sites
GEOSoft

If you read the help file you will find the reason for that. FileReadLineis not really a good choice.

Better solution

#include<File.au3>

$Array = ''

_FileReadToArray($FileIn,$Array)

While 1

Etc


George

Question about decompiling code? Read the decompiling FAQ and don't bother posting the question in the forums.

Be sure to read and follow the forum rules. -AKA the AutoIt Reading and Comprehension Skills test.***

The PCRE (Regular Expression) ToolKit for AutoIT - (Updated Oct 20, 2011 ver:3.0.1.13) - Please update your current version before filing any bug reports. The installer now includes both 32 and 64 bit versions. No change in version number.

Visit my Blog .. currently not active but it will soon be resplendent with news and views. Also please remove any links you may have to my website. it is soon to be closed and replaced with something else.

"Old age and treachery will always overcome youth and skill!"

Share this post


Link to post
Share on other sites
WeMartiansAreFriendly

From the help file under FileReadLine:

Might I suggest FileRead?

Edit:

$fh_in = FileOpen($file_in, 0) ; opens input file for reading
$fh_out = FileOpen($file_out, 1) ; opens output file for writing

$file_array = StringSplit(FileRead($fh_in), @CR, 1)
;or maybe
;$file_array = StringSplit(FileRead($fh_in), @CRLF, 1)

For $i = 1 To $file_array[0]
    $line = StringSplit($file_array[$i], ",", 1)
    For $j = 1 To $line[0]
        ; do stuff to the fields of line to create record variable
    Next
    FileWriteLine($fh_out, $record)
Next
...and there are faster ways than using FileWriteLine... but you will need to provide a sample input file and the desired output file for us to give you a working snippet of code.

if you wanted.. you can turn filewriteline() to $string &= $record & @crlf, and after the loop write the $string variable.. to output file.. from my experience that speeds things up consetterbly..

i have yet to see why you would need filewriteline(), readline() in a loop.... my guess is autotit counts the text lines from the top, so every 1+ increment that surely slows things down

Edited by mrRevoked

Don't bother, It's inside your monitor!------GUISetOnEvent should behave more like HotKeySet()

Share this post


Link to post
Share on other sites
xcal

If you read the help file you will find the reason for that. FileReadLineis not really a good choice.

Better solution

#include<File.au3>

$Array = ''

_FileReadToArray($FileIn,$Array)

While 1

Etc

You may want to have a closer look at the _FileReadToArray function (in the include file), since you're calling that a 'better' solution. :shocked: Just saying, since you're following herewasplato's post.

Edited by xcal

Share this post


Link to post
Share on other sites
flyingboz

i have yet to see why you would need filewriteline(), readline() in a loop....

<paraphrased> Just use an array, stupid!

the answer to both the above is because the datasets are so large that storing them in memory arrays is impractical for the hardware I have to work with.

<edit: found the scroll bar on the help file...> Will try w/o specifying the line number.

Edited by flyingboz

Reading the help file before you post... Not only will it make you look smarter, it will make you smarter.

Share this post


Link to post
Share on other sites
flyingboz

When the line number is removed from the code, each iteration performs linearly, without a dependency on file size.


Reading the help file before you post... Not only will it make you look smarter, it will make you smarter.

Share this post


Link to post
Share on other sites
WeMartiansAreFriendly

like so.. i havent tested but it should work..

$maxlen = 10000
$fh_in = FileOpen($file_in, 0); opens input file for reading
$fh_out = FileOpen($file_out, 1); opens output file for writing

$file_array = StringSplit(StringStripCR(FileRead($fh_in)), @LF, 1)



For $i = 1 To $file_array[0]
    $line = StringSplit($file_array[$i], ",", 1)
    For $j = 1 To $line[0]
       ; do stuff to the fields of line to create record variable
    $record &= $file_array[$i]
    Next
    If StringLen($record) >= $maxlen Then
        FileWriteLine($fh_out, $record)
        $record = ''";flush $record
    EndIf
Next

Don't bother, It's inside your monitor!------GUISetOnEvent should behave more like HotKeySet()

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.