dabus Posted June 21, 2005 Share Posted June 21, 2005 I have a little problem. I want to parse a biiiiiiig file that I created by a dos-pipe. It is 15.000 lines long. First 1000 lines take 12sec, next 28 sec, next 58 ... and here I quit. Why does this thing slow down so badly? Do I have to split the file? Any ideas appreciated. expandcollapse popupAutoItSetOption("WinTitleMatchMode", 2) AutoItSetOption("TrayIconDebug", 1) AutoItSetOption("TrayIconHide", 0) #include <file.au3> #include <array.au3> #include <math.au3> $begin = TimerInit() $Date=@MDAY&"."&@Mon&"."&@YEAR RunWait ( @comspec &" /c dir /A:-D /Q /T:W /S /N V:\ > "&@ScriptDir&"\Dir.txt" , "", @SW_HIDE ) $dif = TimerDiff($begin) ConsoleWrite ( $dif&@CR ) $begin = TimerInit() $begin2 = TimerInit() $FileDirOutPut = @ScriptDir&"\Dir.txt" $Lines = _FileCountLines($FileDirOutPut) $DirOutPut = FileOpen ( $FileDirOutPut,0 ) For $Line=1 To $Lines Step 1 $Out=_MathCheckDiv ( $Line, 1000 ) If $Out=2 Then $difold=$dif $dif = TimerDiff($begin) $dif=Round ( ($dif/1000)-$difold , 0 ) ConsoleWrite ($Line&": "&$dif ) EndIf $File = FileReadLine($DirOutPut, $Line) If $File = "" Then ContinueLoop TrayTip ( "Delete old" , "", 0 ) TrayTip ( "Arbeite", $Line , 5) $SplitLine=StringSplit ( $File, " ") If $SplitLine[1] = "" Then ContinueLoop If $SplitLine[1] <> $Date Then ContinueLoop ConsoleWrite ( $Line&": "&$File&" "&$File&@CR ) Next FileClose ( $DirOutPut ) $dif2=TimerDiff ( $begin2 ) ConsoleWrite ( $dif2&@CR ) Link to comment Share on other sites More sharing options...
jpm Posted June 21, 2005 Share Posted June 21, 2005 (edited) at least 2 improvements 1. $File = FileReadLine($DirOutPut) will not force the rereading of the file from the beginning each time you read a line. 2. _FileCountLines is reading the file entirely too. You better consider reading until you reach end of file. Edited June 21, 2005 by jpm Link to comment Share on other sites More sharing options...
dabus Posted June 21, 2005 Author Share Posted June 21, 2005 (edited) Cool, replaced For ... Next with While 1 ... Wend and the FileReadLine-stuff. Now it goes within 80 sec. Great. Could you guys be so kind and add your first reply to the help-file? Just for people like me... Edited June 21, 2005 by dabus Link to comment Share on other sites More sharing options...
blink314 Posted June 21, 2005 Share Posted June 21, 2005 I also have this problem... except some of my files are 400,000 lines plus. I read the entire file into memory with _FileToArray (only takes 1-2 seconds) but my problem is that even looping through the array takes so long (~15 seconds per 100,000 lines). Are there any general efficiency tips (not specific to this problem? For example are For... Next loops less efficient than while loops? Is it faster to use If.. elseif constructs or Case... Select? Kevin Link to comment Share on other sites More sharing options...
dabus Posted June 21, 2005 Author Share Posted June 21, 2005 Ok, without this stupid linecounter, it takes 15 sec. Man, that's speed. Link to comment Share on other sites More sharing options...
jpm Posted June 21, 2005 Share Posted June 21, 2005 Cool, replaced For ... Next with While 1 ... Wend and the FileReadLine-stuff. Now it goes within 80 sec. Great. Could you guys be so kind and add your first reply to the help-file? Just for people like me... <{POST_SNAPBACK}>I will add a remark on this speed impact when using the line parameter Link to comment Share on other sites More sharing options...
JSThePatriot Posted June 21, 2005 Share Posted June 21, 2005 I also have this problem... except some of my files are 400,000 lines plus. I read the entire file into memory with _FileToArray (only takes 1-2 seconds) but my problem is that even looping through the array takes so long (~15 seconds per 100,000 lines). Are there any general efficiency tips (not specific to this problem? For example are For... Next loops less efficient than while loops? Is it faster to use If.. elseif constructs or Case... Select? Kevin<{POST_SNAPBACK}>If I am not mistaken Select...Case...EndSelect is 1-3% faster than If...ElseIf...EndIf statements.JS AutoIt Links File-String Hash Plugin Updated! 04-02-2008 Plugins have been discontinued. I just found out. ComputerGetInfo UDF's Updated! 11-23-2006 External Links Vortex Revolutions Engineer / Inventor (Web, Desktop, and Mobile Applications, Hardware Gizmos, Consulting, and more) Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now