Parse a big file

[dabus]

I have a little problem.

I want to parse a biiiiiiig file that I created by a dos-pipe.

It is 15.000 lines long.

First 1000 lines take 12sec, next 28 sec, next 58 ... and here I quit.

Why does this thing slow down so badly?

Do I have to split the file?

Any ideas appreciated.

expandcollapse popup

AutoItSetOption("WinTitleMatchMode", 2)
AutoItSetOption("TrayIconDebug", 1)
AutoItSetOption("TrayIconHide", 0)

#include <file.au3>
#include <array.au3>
#include <math.au3>

$begin = TimerInit()
$Date=@MDAY&"."&@Mon&"."&@YEAR
RunWait ( @comspec &" /c dir /A:-D /Q /T:W /S /N V:\ > "&@ScriptDir&"\Dir.txt" , "", @SW_HIDE )
$dif = TimerDiff($begin)
ConsoleWrite ( $dif&@CR )
$begin = TimerInit()
$begin2 = TimerInit()
$FileDirOutPut = @ScriptDir&"\Dir.txt"
$Lines = _FileCountLines($FileDirOutPut)
$DirOutPut = FileOpen ( $FileDirOutPut,0 )
For $Line=1 To $Lines Step 1
    $Out=_MathCheckDiv ( $Line, 1000 )
    If $Out=2 Then
        $difold=$dif
        $dif = TimerDiff($begin)
        $dif=Round ( ($dif/1000)-$difold , 0 )
        ConsoleWrite ($Line&": "&$dif )
    EndIf
    $File = FileReadLine($DirOutPut, $Line)
    If $File = "" Then ContinueLoop
    TrayTip ( "Delete old" , "", 0 )
    TrayTip ( "Arbeite",  $Line , 5)
    $SplitLine=StringSplit ( $File, " ")
    If $SplitLine[1] = "" Then ContinueLoop
    If $SplitLine[1] <> $Date Then ContinueLoop
    ConsoleWrite ( $Line&": "&$File&" "&$File&@CR )
Next
FileClose ( $DirOutPut )
$dif2=TimerDiff ( $begin2 )
ConsoleWrite ( $dif2&@CR )

[jpm]

at least 2 improvements

1. $File = FileReadLine($DirOutPut) will not force the rereading of the file from the beginning each time you read a line.

2. _FileCountLines is reading the file entirely too. You better consider reading until you reach end of file.

Edited June 21, 2005 by jpm

[dabus]

Cool, replaced For ... Next with While 1 ... Wend and the FileReadLine-stuff.

Now it goes within 80 sec. Great.

Could you guys be so kind and add your first reply to the help-file? Just for people like me...

Edited June 21, 2005 by dabus

[blink314]

I also have this problem... except some of my files are 400,000 lines plus. I read the entire file into memory with _FileToArray (only takes 1-2 seconds) but my problem is that even looping through the array takes so long (~15 seconds per 100,000 lines). Are there any general efficiency tips (not specific to this problem? For example are For... Next loops less efficient than while loops? Is it faster to use If.. elseif constructs or Case... Select?

Kevin

[dabus]

Ok, without this stupid linecounter, it takes 15 sec.

Man, that's speed.

[jpm]

Cool, replaced For ... Next with While 1 ... Wend and the FileReadLine-stuff.

Now it goes within 80 sec. Great.

Could you guys be so kind and add your first reply to the help-file? Just for people like me...

<{POST_SNAPBACK}>

I will add a remark on this speed impact when using the line parameter

[JSThePatriot]

I also have this problem... except some of my files are 400,000 lines plus.  I read the entire file into memory with _FileToArray (only takes 1-2 seconds) but my problem is that even looping through the array takes so long (~15 seconds per 100,000 lines).  Are there any general efficiency tips (not specific to this problem?  For example are For... Next loops less efficient than while loops?  Is it faster to use If.. elseif constructs or Case... Select? 

Kevin

<{POST_SNAPBACK}>

If I am not mistaken Select...Case...EndSelect is 1-3% faster than If...ElseIf...EndIf statements.

JS