Sign in to follow this  
Followers 0
Morel

Read Large Text (Log) File to Array

4 posts in this topic

Hi,

i am try to Read A Text (LOG) File to Array But the file is 15MB And i want to read it to Array And then Write Only Line That Start With The Current

Date to ListView but The Function _FileReadToArray Stuck the Application What Can I Do the File Contain hundreds of Lines

Morel

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

Hi,

[EDIT; I agree with HardCopy; your script should have worked; ? strange characters or some permission problem on file?]

If you are using winXp, you can use DOS FindStr on the file; eg in func;

Func _FindLinesDOSsearch(ByRef $s_File, ByRef $s_AnswerFile, $s_Searches, $i_Append = 0, $i_Case = 0, $i_Array = 0, $i_Literal = 0)

;Syntax; _FindLinesDOSsearch($s_file,$s_AnswerFile,$s_Searches, $i_Case , $i_Array ,$i_Literal); $s_AnswerFile ByRef needs not be defined

; Parameters; $i_Literal=1 implies spaces are delimiters for a number of search strings in $s_Searches, rather than spaces included in search

Local $asList

If FileExists($s_File) Then

If $i_Append Then

$i_Append = ">>"

Else

$i_Append = ">"

EndIf

If $i_Literal = 1 Then

$i_Literal = "/c:"

Else

$i_Literal = ""

EndIf

If $i_Case = 1 Then

$i_Case = "/I"

Else

$i_Case = ""

EndIf

$Position = StringInStr($s_File, "\", 0, -1)

$s_Path = StringLeft($s_File, $Position)

FileChangeDir($s_Path)

$s_Searches = StringReplace($s_Searches, ".", "\."); to use as RegExp in "FindLines"

$s_Searches = StringReplace($s_Searches, "*", ".*"); to use as RegExp in "FindLines"

If StringInStr($s_File, " ") Then $s_File = '"' & $s_File & '"'

If StringInStr($s_AnswerFile, " ") Then $s_AnswerFile = '"' & $s_AnswerFile & '"'

$s_Command = 'findstr /R' & $i_Case & ' ' & $i_Literal & $s_Searches & ' ' & $s_File & ' ' & $i_Append & ' ' & $s_AnswerFile; rem /c: for literal?

$i_Pid = RunWait(@ComSpec & " /c " & $s_Command, @ScriptDir, @SW_HIDE)

ProcessClose($i_Pid)

$s_AnswerFile = StringReplace($s_AnswerFile, '"', '')

$s_File = StringReplace($s_File, '"', "")

If $i_Array Then

$sList = FileRead($s_AnswerFile, FileGetSize($s_AnswerFile))

$sList = StringTrimRight(StringReplace($sList, @CRLF, @LF), 1)

$asList = StringSplit($sList, @LF)

EndIf

Else

SetError(1)

EndIf

Return $asList

EndFunc ;==>_FindLinesDOSsearch

This will give an array with only the lines you have requested.

This is in the UDF in the zip from the link in my signature; "_GUICtrlListView"

Best, Randall

[Edit; removed dysfunctional vbs script, sorry]

Edited by randallc

Share this post


Link to post
Share on other sites

Hi again,

Another fast option if the lines are all at the end of the log file, is to read the last 10 or 100 or 500 lines using TailRW.au3 [so reads fast] from my signature; make a new file and use FileReadtoArray (or the vbs version if still a lot of lines) on that file as you originally wanted.

Reading line by line with au3 script "filereadline" and checking if they have today's date, and only adding those lines to an Array would be more traditional, but guaranteed slower!

Best, randall

Share this post


Link to post
Share on other sites

Hi,

i am try to Read A Text (LOG) File to Array But the file is 15MB And i want to read it to Array And then Write Only Line That Start With The Current

Date to ListView but The Function _FileReadToArray Stuck the Application What Can I Do the File Contain hundreds of Lines

Morel

I regularly Parse large Text (30megs +), csv files, using fileread to array.

The code below is just a mockup of read file to array, then splits each line into an array for whatever parsing.

to read a 32meg file containing 210,000 lines takes this script 3.1 seconds on my mid range spec AMD pc

to read the file to an array.

and 9 more seconds to parse ALL the 210k lines. So i dont think your problem lies in the _filereadto array part

Hope it helps you solve your problem.

However Populating a Listview with each item sequentially, esp if a many many unique items could be slow, and sorting them once populated even slower.

HardCopy

#include<File.au3>
#include<array.au3>

Global $dummydata = ""

_ReadFile()



Func _ReadFile()
    $begin = TimerInit()
    _FileReadToArray("C:\inv.txt", $dummydata);read in a BIG tab delim text file to the array.
    If @error Then
        MsgBox(0,'Fatal Error', 'File Not Found Error, - File NOT read to ARRAY',3)
        Exit
    EndIf
        
    
    $dif = TimerDiff($begin)
    ConsoleWrite("File to Array Complete - Time (ms) = " & $dif & @LF & @LF)
    
    $inc = 0

    For $x=1 To UBound($dummydata)-1
        $SplitArr = StringSplit($dummydata[$x],@TAB)
        $inc +=1
        If $inc = 50000 Then
            $inc = 0
            ConsoleWrite("50000 Lines Completed" & @LF)
        EndIf
    Next
    ;;;info only
    $SplitArr = StringSplit($dummydata[4],@TAB)
    ;;;; ^^^^^^^^^^^^^^^^^^^^^^^
    $dif = TimerDiff($begin)
    ConsoleWrite("Time to complete - Lines Total = " & $x & " // TotalTime (ms) = " & $dif & @LF)
EndFunc

Contributions: UDF _DateYearFirstChildren are like Farts, you can just about stand your own.Why am I not a Vegetarian?...Well...my ancestors didn't fight & evolve to the Top of the food chain for me to survive on Salad

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0