Fast File handle? accelerate the script

[Xenobiologist]

Hi,

is there a way, to make this script faster? It is a competition between my co-workers and me. (Okay, I tried Java, Perl and Ruby)

1. Ruby 90 millisec

2. Perl 180 millisec

3. Java 240 millisec

4. AutoIt 300 millisec

Task: read an xml-file with 3000 lines. Replace filter by valhalla (330 times) and write it in a new File.

Thx

So long,

Mega

#include <file.au3>

_FileCreate("V.xml")

$f = FileOpen("V.xml", 2)

$b = TimerInit()

;For $i = 1 to 9

Dim $a

_FileReadToArray("model.xml", $a)

For $x = 1 To $a[0]

$t = StringReplace($a[$x], "filter", "valhalla")

FileWriteLine($f, $t)

Next

;Next

$d = TimerDiff($

MsgBox(0,"Zeit", "Es wurden " & $d & " Millisekunden benötigt") ;/ 10 average

Exit (0)

Edited August 20, 2005 by th.meger

[LxP]

local $data, $start, $end

$start = timerInit()
$data = fileRead("model.xml", fileGetSize("model.xml"))
$data = stringReplace($data, "filter", "valhalla")
fileWrite("V.xml", $data)
$end = timerDiff($start)

msgBox(0x40, "Test", "Operation took " & $end & "ms.")

How does that go?

Edited August 19, 2005 by LxP

[Valuater]

Nice code.... LxP

"Anything less would be uncivilized"

I hope he replies

8)

s/z

[w0uter]

"Anything less would be uncivilized"

<{POST_SNAPBACK}>

dim $t = timerInit()
fileWrite("V.xml", stringReplace(fileRead("model.xml", fileGetSize("model.xml")), "filter", "valhalla"))
msgBox(0x40, "Test", "Operation took " & timerDiff($t) & " ms.")

[edit] you forgot a space before ms. [/edit]

[edit2] i missed a brace [/edit2]

Edited August 19, 2005 by w0uter

[LxP]

Nice code....  LxP

Thanks!

[edit] you forgot a space before ms. [/edit]

Nah... that's just how I write things -- 1.44MB, 20s...

[Xenobiologist]

Hi,

thank u. But the scripts you wrote took more than 1200 ms. Maybe StringReplace has a problem with the number of data.

It think in my script the function replaces just the words in one line at writes it into the new V.xml file.

Maybe in your scripts: You read the whole model.xml file, and the you start to replace. So far so good. That idea has to be faster.

But it is possible that StringReplace starts again after replacing one hit?

I don´t know, but your scripts are definitely slower than mine. ( 4 times!)

So long,

Mega

[LxP]

You are right -- there's definitely a huge decrease in speed with my script, which takes almost two full seconds to execute here.

Devs, why is this so?

Edit: I tested Mega's theory of StringReplace() starting from the beginning after each replace, but that would imply that the following script outputs 'x':

local $test = "xxxxxxxxxx"
msgBox(0x40, "StringReplace()", stringReplace($test, "xx", "x"))

which it doesn't. It outputs 'xxxxx'. So what's going on?

Without seeing the code (I doubt that I could understand it at this stage anyway), my initial guess is that it must have to rewrite the entire string in memory each time it makes a replacement, and since the resulting string is larger it requires more memory and therefore it must be reallocated.

Therefore the 1MB-long string is rewritten in memory over 300 times. No wonder it takes so long with my method!

Edited August 20, 2005 by LxP

[MHz]

The file handle was not closed in the script in the 1st post. There would have to be a time penalty for that. You may find the file is not readable for not closing it. That is a incorrect time reading because of it.

[Jos]

\dev\null has submitted an UDF to do this function and we have been doing some test what is the fastest way to do this,

The string replace is relative slow on a big string so the fastest way is to read the file into an array, then process the array entries and write the records to the file.

here's the fastest version so try this to see if that helps:

expandcollapse popup

func _ReplaceStringInFile($szFileName, $szSearchString, $szReplaceString, $fCaseness = 0, $fOccurance = 1)

    local $iRetVal = 0

    if Not FileExists($szFileName)then 
        SetError(1)
        return -1
    endif

    $szTempFile = _TempFile()
    $hWriteHandle = FileOpen($szTempFile,2)
    if $hWriteHandle = -1 then 
        SetError(2)
        return -1
    endif

    Local $T_Out = ""
    Local $T_recs
    _FileReadToArray($szFileName,$T_recs)

    For $x = 1 to $T_recs[0] 
    ;======================================================================
    ;== check if the search string is found in the current file line
    ;======================================================================
        If $fOccurance = 0 and $iRetVal > 0 then
            FileWriteLine($hWriteHandle,$T_recs[$x])
        Else
            if StringInStr($T_recs[$x],$szSearchString,$fCaseness) then
                $iRetVal = $iRetVal + 1
                FileWriteLine($hWriteHandle, StringReplace($T_recs[$x],$szSearchString,$szReplaceString,0,$fCaseness))
            else 
                FileWriteLine($hWriteHandle,$T_recs[$x])
            ;$T_Out = $T_Out & $T_recs[$x] & @CRLF
            endif 
        EndIf
    Next

    FileClose($hWriteHandle)
    
;======================================================================
;== Delete the original file
;======================================================================
    if FileDelete($szFileName) = 0 then
        SetError(4)
        return -1
    endif

;======================================================================
;== Rename the temp file to the original file
;======================================================================
    if FileMove($szTempFile,$szFileName) = 0 then
         SetError(5)
        return -1
    endif

    return $iRetVal
endfunc;==> _ReplaceStringInFile

Edited August 21, 2005 by JdeB

[Xenobiologist]

Hello,

I tried your function, but it takes nearly 900 ms.

Did I use your function right?

Why is AutoIt so slow?

I just started to learn AuoiIt and now I´m searching for some reasonable tasks.

Thanks @all : It´s really fun dealing with AutoIt

So long,

Mega

#include <file.au3>

$szFileName = "model.xml"

$szSearchString = "filter"

$szReplaceString = "valhalla"

$fCaseness = 0

$fOccurance = 1

$b = TimerInit()

Call(_ReplaceStringInFile($szFileName, $szSearchString, $szReplaceString, $fCaseness = 0, $fOccurance = 1))

MsgBox(0,"Time", "The Operation took " & TimerDiff($ & " ms")

Exit(0)

func _ReplaceStringInFile($szFileName, $szSearchString, $szReplaceString, $fCaseness = 0, $fOccurance = 1)

local $iRetVal = 0

if Not FileExists($szFileName)then

SetError(1)

return -1

endif

$szTempFile = _TempFile()

$hWriteHandle = FileOpen($szTempFile,2)

if $hWriteHandle = -1 then

SetError(2)

return -1

endif

Local $T_Out = ""

Local $T_recs

_FileReadToArray($szFileName,$T_recs)

For $x = 1 to $T_recs[0]

;======================================================================

;== check if the search string is found in the current file line

;======================================================================

If $fOccurance = 0 and $iRetVal > 0 then

FileWriteLine($hWriteHandle,$T_recs[$x])

Else

if StringInStr($T_recs[$x],$szSearchString,$fCaseness) then

$iRetVal = $iRetVal + 1

FileWriteLine($hWriteHandle, StringReplace($T_recs[$x],$szSearchString,$szReplaceString,0,$fCaseness))

else

FileWriteLine($hWriteHandle,$T_recs[$x])

;$T_Out = $T_Out & $T_recs[$x] & @CRLF

endif

EndIf

Next

FileClose($hWriteHandle)

;======================================================================

;== Delete the original file

;======================================================================

if FileDelete($szFileName) = 0 then

SetError(4)

return -1

endif

;======================================================================

;== Rename the temp file to the original file

;======================================================================

if FileMove($szTempFile,$szFileName) = 0 then

SetError(5)

return -1

endif

return $iRetVal

endfunc;==> _ReplaceStringInFile

Edited August 22, 2005 by th.meger