Sign in to follow this  
Followers 0
dabus

Parse a big file

7 posts in this topic

I have a little problem.

I want to parse a biiiiiiig file that I created by a dos-pipe.

It is 15.000 lines long. :)

First 1000 lines take 12sec, next 28 sec, next 58 ... and here I quit. :evil:

Why does this thing slow down so badly?

Do I have to split the file?

Any ideas appreciated.

AutoItSetOption("WinTitleMatchMode", 2)
AutoItSetOption("TrayIconDebug", 1)
AutoItSetOption("TrayIconHide", 0)

#include <file.au3>
#include <array.au3>
#include <math.au3>

$begin = TimerInit()
$Date=@MDAY&"."&@Mon&"."&@YEAR
RunWait ( @comspec &" /c dir /A:-D /Q /T:W /S /N V:\ > "&@ScriptDir&"\Dir.txt" , "", @SW_HIDE )
$dif = TimerDiff($begin)
ConsoleWrite ( $dif&@CR )
$begin = TimerInit()
$begin2 = TimerInit()
$FileDirOutPut = @ScriptDir&"\Dir.txt"
$Lines = _FileCountLines($FileDirOutPut)
$DirOutPut = FileOpen ( $FileDirOutPut,0 )
For $Line=1 To $Lines Step 1
    $Out=_MathCheckDiv ( $Line, 1000 )
    If $Out=2 Then
        $difold=$dif
        $dif = TimerDiff($begin)
        $dif=Round ( ($dif/1000)-$difold , 0 )
        ConsoleWrite ($Line&": "&$dif )
    EndIf
    $File = FileReadLine($DirOutPut, $Line)
    If $File = "" Then ContinueLoop
    TrayTip ( "Delete old" , "", 0 )
    TrayTip ( "Arbeite",  $Line , 5)
    $SplitLine=StringSplit ( $File, " ")
    If $SplitLine[1] = "" Then ContinueLoop
    If $SplitLine[1] <> $Date Then ContinueLoop
    ConsoleWrite ( $Line&": "&$File&" "&$File&@CR )
Next
FileClose ( $DirOutPut )
$dif2=TimerDiff ( $begin2 )
ConsoleWrite ( $dif2&@CR )

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

at least 2 improvements

1. $File = FileReadLine($DirOutPut) will not force the rereading of the file from the beginning each time you read a line.

2. _FileCountLines is reading the file entirely too. You better consider reading until you reach end of file. :)

Edited by jpm

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

Cool, replaced For ... Next with While 1 ... Wend and the FileReadLine-stuff.

Now it goes within 80 sec. Great. :)

Could you guys be so kind and add your first reply to the help-file? Just for people like me... :evil:

Edited by dabus

Share this post


Link to post
Share on other sites

I also have this problem... except some of my files are 400,000 lines plus. I read the entire file into memory with _FileToArray (only takes 1-2 seconds) but my problem is that even looping through the array takes so long (~15 seconds per 100,000 lines). Are there any general efficiency tips (not specific to this problem? For example are For... Next loops less efficient than while loops? Is it faster to use If.. elseif constructs or Case... Select?

Kevin

Share this post


Link to post
Share on other sites

Ok, without this stupid linecounter, it takes 15 sec.

Man, that's speed.

Share this post


Link to post
Share on other sites

Cool, replaced For ... Next with While 1 ... Wend and the FileReadLine-stuff.

Now it goes within 80 sec. Great. :)

Could you guys be so kind and add your first reply to the help-file? Just for people like me... :evil:

<{POST_SNAPBACK}>

I will add a remark on this speed impact when using the line parameter :D

Share this post


Link to post
Share on other sites

I also have this problem... except some of my files are 400,000 lines plus.  I read the entire file into memory with _FileToArray (only takes 1-2 seconds) but my problem is that even looping through the array takes so long (~15 seconds per 100,000 lines).  Are there any general efficiency tips (not specific to this problem?  For example are For... Next loops less efficient than while loops?  Is it faster to use If.. elseif constructs or Case... Select? 

Kevin

<{POST_SNAPBACK}>

If I am not mistaken Select...Case...EndSelect is 1-3% faster than If...ElseIf...EndIf statements.

JS


AutoIt Links

File-String Hash Plugin Updated! 04-02-2008 Plugins have been discontinued. I just found out.

ComputerGetInfo UDF's Updated! 11-23-2006

External Links

Vortex Revolutions Engineer / Inventor (Web, Desktop, and Mobile Applications, Hardware Gizmos, Consulting, and more)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0