_FileWriteToLine is too slow

[Bert]

You can send it to me... I'll have a look, I just need a detailed outline of one of them to see what the exact output you hope to achieve would be.

Here is the file after I have ran it through my script, and I have hand edited the first 10 questions so the formatting is correct. The reasoning with the carriage return being a pain is trandumper is a stickler when it comes to formatting and reading the test.

http://www.autoitscript.com/fileman/index.php?act=list&op=get&target=N10-003_1.txt

[SmOke_N]

Here is the file after I have ran it through my script, and I have hand edited the first 10 questions so the formatting is correct. The reasoning with the carriage return being a pain is trandumper is a stickler when it comes to formatting and reading the test.

http://www.autoitscript.com/fileman/index.php?act=list&op=get&target=N10-003_1.txt

I was talking about without running it through your script. I'm sure there's a more efficient way to do everything.

[Bert]

I gave it to everyone both ways. The first example is the raw txt file. The second example is ran through my script. The first 10 questions have also been edited by hand so you can see the final result of what I'm trying to get to. You can go past question 10 to see how it is working otherwise. Sorry if I wasn't clear.

[Bert]

Here is the txt file I'm attempting to fix. For those who want to take a stab at it....http://www.autoitscript.com/fileman/index.php?act=list&op=get&target=N10-003.txt

this one is the raw txt file

[SmOke_N]

this one is the raw txt file

hmm, funny I don't have access to it.

[Bert]

That's odd. I put it on fileman...hmmm

[Bert]

Try this:

This is the old one

http://www.autoitscript.com/fileman/users/Vollyman/N10-003.txt

edited one:

http://www.autoitscript.com/fileman/users/Vollyman/N10-003_1.txt

[MHz]

I'm attempting to reformat a text file. Here is a small snippet of what I'm dealing with:

Now, I need to reformat the information and remove lines here or there to make everything work the way I need it to. For example, I need to remove every reference of "Actualtests.com - The Power of Knowing". Here is the script I'm using to just remove this

#include <file.au3>
$file = FileOpen("N10-003.txt", 0)
$CountLines = _FileCountLines("N10-003.txt")
; Check if file opened for reading OK
TraySetToolTip("0 of "&$CountLines&" lines checked")
If $file = -1 Then
    MsgBox(0, "Error", "Unable to open file.")
    Exit
EndIf

; Read in lines of text until the EOF is reached
$x=1
While 1
    $line = FileReadLine($file, $x)
    If @error = -1 Then ExitLoop
    $string = StringInStr($line, "Actualtests.com")
    if $line = "Actualtests.com - The Power of Knowing" then    
        _FileWritetoLine(@ScriptDir&"/N10-003.txt", $x,"", 1)
    endif
    TraySetToolTip($x&" of "&$CountLines&" lines checked.")
    $x= $x+1
Wend

FileClose($file)

The file has about 13000 lines. It takes about 20 minutes to run through the thing. Any way to speed it up?

Bugged code is all I can say as many things wrong with it. _FileWritetoLine() is a very slow depending how it is used but you are using it on a file that already has a handle open on it which does not help.

Out of interest try this.

expandcollapse popup

#include <file.au3>
$time = TimerInit()
$handle_read = FileOpen("N10-003.txt", 0); read
If $handle_read = -1 Then
    MsgBox(0, "Error", "Unable to open file to read.")
    Exit
EndIf

$tempfile = _tempfile()
$handle_write = FileOpen($tempfile, 1); write (append mode)
If $handle_read = -1 Then
    MsgBox(0, "Error", "Unable to open file to write.")
    FileClose($handle_read)
    Exit
EndIf

While 1
    $line = FileReadLine($handle_read);, $x
    If @error = -1 Then ExitLoop
    if $line = "Actualtests.com - The Power of Knowing" then     
        FileWriteLine($handle_write, "")
        ContinueLoop
    endif
    FileWriteLine($handle_write, $line)
Wend

FileClose($handle_read)
FileClose($handle_write)

If FileMove('N10-003.txt', 'N10-003.txt.bak') Then
    If FileMove($tempfile, 'N10-003.txt') Then
        FileDelete('N10-003.txt.bak')
    Else
        FileMove('N10-003.txt.bak', 'N10-003.txt')
    EndIf
EndIf

MsgBox(0, '', TimerDiff($time)/1000)oÝ÷ Ù8b²Ç+[zZ0¶æ­zË_W­¢·tÚ-zØZ¶+p®+^²Ú*ºM×M4ÞÜmªê-jëh×6#include <file.au3>
$destination = 'N10-003.txt'
FileDelete($destination)
$handle_write = FileOpen($destination, 1); write, append
If $handle_write <> -1 Then
    For $i = 1 To 6500
        FileWriteLine($handle_write, 'Some text here to read')
        FileWriteLine($handle_write, 'Actualtests.com - The Power of Knowing')
    Next
    FileClose($handle_write)
EndIf

MsgBox(0, '13000 lines?', _FileCountLines("N10-003.txt"))

Running the main script on the generated text file completes in 1.4 seconds as Scite tells me.

Edit:

Added TimeInit() and TimeDiff() to time with and now timed at 0.17 seconds so the timing mentioned by Scite is far longer then ought to be.

Edited May 9, 2008 by MHz

[Bert]

Taking a (need for speed) stab at it.

1.7sec @ 2.4GHz.

expandcollapse popup

Dim Const $NL = @CRLF
Dim $Gtime1 = -1, $Gtime2 = -1
Dim $sFileIN, $sFileOUT
Dim $sFIND, $sREPL, $sPATERN
Dim $sTEXTDUMP
$Gtime1 = TimerInit()
$sFileIN = 'FILE_IN.txt'
$sFileOUT = 'FILE_OUT.txt'
If FileExists($sFileOUT) Then FileRecycle($sFileOUT)
$sTEXTDUMP = FileRead($sFileIN)

SR($NL, ' ') ;; first things first, lets get rid of all those pesty line breaks.
SR('Actualtests.com - The Power of Knowing', ' ')
SRR('QUESTION (\d+)\:', $NL & $NL & '\1. ')
SR('Answer:', $NL & $NL & 'Answer:')
SR('Explanation:', $NL & $NL & '<B>Explanation:</B><br>' & $NL)
SR('Incorrect Answers:', $NL & $NL & '<B>Incorrect Answers:</B><br>')
SRR('([ABCD])\, ([ABCD])\: ', $NL & '<B>\1\2 -</B> ') ;; exception!
SRR('([ABCD])\: ', $NL & '<B>\1 -</B> ')
SRR(' ([A-D]\. )', $NL & '\1 ')
SR('A. ', $NL & $NL & 'A. ')
SR('References:', $NL & $NL & '<B>References:</B><br>' & $NL)

;; GENERAL CLEANUP ;;
$sFIND = '  '
$sREPL = ' '
While StringInStr($sTEXTDUMP, $sFIND)
    SR($sFIND, $sREPL)
WEnd
SR($NL & ' ', $NL)
SR(' ' & $NL, $NL)
$sFIND = $NL & $NL & $NL
$sREPL = $NL & $NL
While StringInStr($sTEXTDUMP, $sFIND)
    SR($sFIND, $sREPL)
WEnd

FileWrite($sFileOUT, $sTEXTDUMP)

$Gtime2 = TimerDiff($Gtime1)
ConsoleWrite('--) ' & Int($Gtime2) / 1000 & $NL)
Exit

Func SR($sFIND, $sREPL)
    $sTEXTDUMP = StringReplace($sTEXTDUMP, $sFIND, $sREPL, 0, 2)
EndFunc
Func SRR($sPATERN, $sREPL)
    $sTEXTDUMP = StringRegExpReplace($sTEXTDUMP, $sPATERN, $sREPL, 0)
EndFunc

M.v.G

- - -

Tip for really big files.

Sometimes its better to split a file into smaller chunks and process those instead of the whole file.

! How big a chunk is a matter of experimenting.

This is perfect! I'm able to add the extra stuff I need easily, and it is real fast! Thanks!