flyingboz Posted April 19, 2007 Share Posted April 19, 2007 Sample code follows I start losing around 4 sec for every 100 lines of text I process ; and the amount of time it takes goes up by around 3 seconds for each additional 100 lines of text. Using TimerDiff and TimerInit, I've traced the source of the problem to the FileReadLine() function. Suggestions? Can others verify this behavior on a (preferably large) dataset? I am using 3.2.2.0 $fh_in = FileOpen($file_in,0) ; opens input file for reading $fh_out = FileOpen($file_out,1) ; opens output file for writing $i = 1 while 1 $line = FileReadLine($fh_in,$i) if @error = -1 then ExitLoop $line = StringSplit($line) For $j = 1 to $line[0] ; do stuff to the fields of line to create record variable Next FileWriteLine($fh_out,$record) $i +=1 Wend Reading the help file before you post... Not only will it make you look smarter, it will make you smarter. Link to comment Share on other sites More sharing options...
herewasplato Posted April 19, 2007 Share Posted April 19, 2007 (edited) From the help file under FileReadLine:From a performance standpoint it is a bad idea to read line by line specifying "line" parameter whose value is incrementing by one. This forces AutoIt to reread the file from the beginning until it reach the specified line. Might I suggest FileRead? Edit:$fh_in = FileOpen($file_in, 0) ; opens input file for reading $fh_out = FileOpen($file_out, 1) ; opens output file for writing $file_array = StringSplit(FileRead($fh_in), @CR, 1) ;or maybe ;$file_array = StringSplit(FileRead($fh_in), @CRLF, 1) For $i = 1 To $file_array[0] $line = StringSplit($file_array[$i], ",", 1) For $j = 1 To $line[0] ; do stuff to the fields of line to create record variable Next FileWriteLine($fh_out, $record) Next...and there are faster ways than using FileWriteLine... but you will need to provide a sample input file and the desired output file for us to give you a working snippet of code. Edited April 20, 2007 by herewasplato [size="1"][font="Arial"].[u].[/u][/font][/size] Link to comment Share on other sites More sharing options...
GEOSoft Posted April 20, 2007 Share Posted April 20, 2007 If you read the help file you will find the reason for that. FileReadLineis not really a good choice. Better solution #include<File.au3> $Array = '' _FileReadToArray($FileIn,$Array) While 1 Etc George Question about decompiling code? Read the decompiling FAQ and don't bother posting the question in the forums.Be sure to read and follow the forum rules. -AKA the AutoIt Reading and Comprehension Skills test.*** The PCRE (Regular Expression) ToolKit for AutoIT - (Updated Oct 20, 2011 ver:3.0.1.13) - Please update your current version before filing any bug reports. The installer now includes both 32 and 64 bit versions. No change in version number. Visit my Blog .. currently not active but it will soon be resplendent with news and views. Also please remove any links you may have to my website. it is soon to be closed and replaced with something else. "Old age and treachery will always overcome youth and skill!" Link to comment Share on other sites More sharing options...
WeMartiansAreFriendly Posted April 20, 2007 Share Posted April 20, 2007 (edited) From the help file under FileReadLine: Might I suggest FileRead? Edit:$fh_in = FileOpen($file_in, 0) ; opens input file for reading $fh_out = FileOpen($file_out, 1) ; opens output file for writing $file_array = StringSplit(FileRead($fh_in), @CR, 1) ;or maybe ;$file_array = StringSplit(FileRead($fh_in), @CRLF, 1) For $i = 1 To $file_array[0] $line = StringSplit($file_array[$i], ",", 1) For $j = 1 To $line[0] ; do stuff to the fields of line to create record variable Next FileWriteLine($fh_out, $record) Next...and there are faster ways than using FileWriteLine... but you will need to provide a sample input file and the desired output file for us to give you a working snippet of code. if you wanted.. you can turn filewriteline() to $string &= $record & @crlf, and after the loop write the $string variable.. to output file.. from my experience that speeds things up consetterbly.. i have yet to see why you would need filewriteline(), readline() in a loop.... my guess is autotit counts the text lines from the top, so every 1+ increment that surely slows things down Edited April 20, 2007 by mrRevoked Don't bother, It's inside your monitor!------GUISetOnEvent should behave more like HotKeySet() Link to comment Share on other sites More sharing options...
xcal Posted April 20, 2007 Share Posted April 20, 2007 (edited) If you read the help file you will find the reason for that. FileReadLineis not really a good choice.Better solution#include<File.au3>$Array = ''_FileReadToArray($FileIn,$Array)While 1EtcYou may want to have a closer look at the _FileReadToArray function (in the include file), since you're calling that a 'better' solution. Just saying, since you're following herewasplato's post. Edited April 20, 2007 by xcal How To Ask Questions The Smart Way Link to comment Share on other sites More sharing options...
flyingboz Posted April 20, 2007 Author Share Posted April 20, 2007 (edited) i have yet to see why you would need filewriteline(), readline() in a loop....<paraphrased> Just use an array, stupid!the answer to both the above is because the datasets are so large that storing them in memory arrays is impractical for the hardware I have to work with.<edit: found the scroll bar on the help file...> Will try w/o specifying the line number. Edited April 20, 2007 by flyingboz Reading the help file before you post... Not only will it make you look smarter, it will make you smarter. Link to comment Share on other sites More sharing options...
flyingboz Posted April 20, 2007 Author Share Posted April 20, 2007 When the line number is removed from the code, each iteration performs linearly, without a dependency on file size. Reading the help file before you post... Not only will it make you look smarter, it will make you smarter. Link to comment Share on other sites More sharing options...
WeMartiansAreFriendly Posted April 20, 2007 Share Posted April 20, 2007 ahh i see, well you could always add it to an array, once it reaches a certain length write that to the disk, and so on. Don't bother, It's inside your monitor!------GUISetOnEvent should behave more like HotKeySet() Link to comment Share on other sites More sharing options...
WeMartiansAreFriendly Posted April 20, 2007 Share Posted April 20, 2007 like so.. i havent tested but it should work.. $maxlen = 10000 $fh_in = FileOpen($file_in, 0); opens input file for reading $fh_out = FileOpen($file_out, 1); opens output file for writing $file_array = StringSplit(StringStripCR(FileRead($fh_in)), @LF, 1) For $i = 1 To $file_array[0] $line = StringSplit($file_array[$i], ",", 1) For $j = 1 To $line[0] ; do stuff to the fields of line to create record variable $record &= $file_array[$i] Next If StringLen($record) >= $maxlen Then FileWriteLine($fh_out, $record) $record = ''";flush $record EndIf Next Don't bother, It's inside your monitor!------GUISetOnEvent should behave more like HotKeySet() Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now