Read Large Text (Log) File to Array

[Morel]

Hi,

i am try to Read A Text (LOG) File to Array But the file is 15MB And i want to read it to Array And then Write Only Line That Start With The Current

Date to ListView but The Function _FileReadToArray Stuck the Application What Can I Do the File Contain hundreds of Lines

Morel

[randallc]

Hi,

[EDIT; I agree with HardCopy; your script should have worked; ? strange characters or some permission problem on file?]

If you are using winXp, you can use DOS FindStr on the file; eg in func;

Func _FindLinesDOSsearch(ByRef $s_File, ByRef $s_AnswerFile, $s_Searches, $i_Append = 0, $i_Case = 0, $i_Array = 0, $i_Literal = 0)

;Syntax; _FindLinesDOSsearch($s_file,$s_AnswerFile,$s_Searches, $i_Case , $i_Array ,$i_Literal); $s_AnswerFile ByRef needs not be defined

; Parameters; $i_Literal=1 implies spaces are delimiters for a number of search strings in $s_Searches, rather than spaces included in search

Local $asList

If FileExists($s_File) Then

If $i_Append Then

$i_Append = ">>"

Else

$i_Append = ">"

EndIf

If $i_Literal = 1 Then

$i_Literal = "/c:"

Else

$i_Literal = ""

EndIf

If $i_Case = 1 Then

$i_Case = "/I"

Else

$i_Case = ""

EndIf

$Position = StringInStr($s_File, "\", 0, -1)

$s_Path = StringLeft($s_File, $Position)

FileChangeDir($s_Path)

$s_Searches = StringReplace($s_Searches, ".", "\."); to use as RegExp in "FindLines"

$s_Searches = StringReplace($s_Searches, "*", ".*"); to use as RegExp in "FindLines"

If StringInStr($s_File, " ") Then $s_File = '"' & $s_File & '"'

If StringInStr($s_AnswerFile, " ") Then $s_AnswerFile = '"' & $s_AnswerFile & '"'

$s_Command = 'findstr /R' & $i_Case & ' ' & $i_Literal & $s_Searches & ' ' & $s_File & ' ' & $i_Append & ' ' & $s_AnswerFile; rem /c: for literal?

$i_Pid = RunWait(@ComSpec & " /c " & $s_Command, @ScriptDir, @SW_HIDE)

ProcessClose($i_Pid)

$s_AnswerFile = StringReplace($s_AnswerFile, '"', '')

$s_File = StringReplace($s_File, '"', "")

If $i_Array Then

$sList = FileRead($s_AnswerFile, FileGetSize($s_AnswerFile))

$sList = StringTrimRight(StringReplace($sList, @CRLF, @LF), 1)

$asList = StringSplit($sList, @LF)

EndIf

Else

SetError(1)

EndIf

Return $asList

EndFunc ;==>_FindLinesDOSsearch

This will give an array with only the lines you have requested.

This is in the UDF in the zip from the link in my signature; "_GUICtrlListView"

Best, Randall

[Edit; removed dysfunctional vbs script, sorry]

Edited December 13, 2006 by randallc

[randallc]

Hi again,

Another fast option if the lines are all at the end of the log file, is to read the last 10 or 100 or 500 lines using TailRW.au3 [so reads fast] from my signature; make a new file and use FileReadtoArray (or the vbs version if still a lot of lines) on that file as you originally wanted.

Reading line by line with au3 script "filereadline" and checking if they have today's date, and only adding those lines to an Array would be more traditional, but guaranteed slower!

Best, randall

[HardCopy]

Hi,

i am try to Read A Text (LOG) File to Array But the file is 15MB And i want to read it to Array And then Write Only Line That Start With The Current

Date to ListView but The Function _FileReadToArray Stuck the Application What Can I Do the File Contain hundreds of Lines

Morel

I regularly Parse large Text (30megs +), csv files, using fileread to array.

The code below is just a mockup of read file to array, then splits each line into an array for whatever parsing.

to read a 32meg file containing 210,000 lines takes this script 3.1 seconds on my mid range spec AMD pc

to read the file to an array.

and 9 more seconds to parse ALL the 210k lines. So i dont think your problem lies in the _filereadto array part

Hope it helps you solve your problem.

However Populating a Listview with each item sequentially, esp if a many many unique items could be slow, and sorting them once populated even slower.

HardCopy

expandcollapse popup

#include<File.au3>
#include<array.au3>

Global $dummydata = ""

_ReadFile()



Func _ReadFile()
    $begin = TimerInit()
    _FileReadToArray("C:\inv.txt", $dummydata);read in a BIG tab delim text file to the array.
    If @error Then
        MsgBox(0,'Fatal Error', 'File Not Found Error, - File NOT read to ARRAY',3)
        Exit
    EndIf
        
    
    $dif = TimerDiff($begin)
    ConsoleWrite("File to Array Complete - Time (ms) = " & $dif & @LF & @LF)
    
    $inc = 0

    For $x=1 To UBound($dummydata)-1
        $SplitArr = StringSplit($dummydata[$x],@TAB)
        $inc +=1
        If $inc = 50000 Then
            $inc = 0
            ConsoleWrite("50000 Lines Completed" & @LF)
        EndIf
    Next
    ;;;info only
    $SplitArr = StringSplit($dummydata[4],@TAB)
    ;;;; ^^^^^^^^^^^^^^^^^^^^^^^
    $dif = TimerDiff($begin)
    ConsoleWrite("Time to complete - Lines Total = " & $x & " // TotalTime (ms) = " & $dif & @LF)
EndFunc