Sign in to follow this  
Followers 0
Xenobiologist

Fast File handle? accelerate the script

10 posts in this topic

#1 ·  Posted (edited)

Hi,

is there a way, to make this script faster? It is a competition between my co-workers and me. (Okay, I tried Java, Perl and Ruby)

1. Ruby 90 millisec

2. Perl 180 millisec

3. Java 240 millisec

4. AutoIt 300 millisec

Task: read an xml-file with 3000 lines. Replace filter by valhalla (330 times) and write it in a new File.

Thx

So long,

Mega

#include <file.au3>

_FileCreate("V.xml")

$f = FileOpen("V.xml", 2)

$b = TimerInit()

;For $i = 1 to 9

Dim $a

_FileReadToArray("model.xml", $a)

For $x = 1 To $a[0]

$t = StringReplace($a[$x], "filter", "valhalla")

FileWriteLine($f, $t)

Next

;Next

$d = TimerDiff($:whistle:

MsgBox(0,"Zeit", "Es wurden " & $d & " Millisekunden benötigt") ;/ 10 average

Exit (0)

Edited by th.meger

Scripts & functions Organize Includes Let Scite organize the include files

Yahtzee The game "Yahtzee" (Kniffel, DiceLion)

LoginWrapper Secure scripts by adding a query (authentication)

_RunOnlyOnThis UDF Make sure that a script can only be executed on ... (Windows / HD / ...)

Internet-Café Server/Client Application Open CD, Start Browser, Lock remote client, etc.

MultipleFuncsWithOneHotkey Start different funcs by hitting one hotkey different times

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

local $data, $start, $end

$start = timerInit()
$data = fileRead("model.xml", fileGetSize("model.xml"))
$data = stringReplace($data, "filter", "valhalla")
fileWrite("V.xml", $data)
$end = timerDiff($start)

msgBox(0x40, "Test", "Operation took " & $end & "ms.")

How does that go?

Edited by LxP

Share this post


Link to post
Share on other sites

Nice code.... LxP

"Anything less would be uncivilized"

I hope he replies

8)

s/z


NEWHeader1.png

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

"Anything less would be uncivilized"

<{POST_SNAPBACK}>

dim $t = timerInit()
fileWrite("V.xml", stringReplace(fileRead("model.xml", fileGetSize("model.xml")), "filter", "valhalla"))
msgBox(0x40, "Test", "Operation took " & timerDiff($t) & " ms.")

[edit] you forgot a space before ms. [/edit]

[edit2] i missed a brace [/edit2]

Edited by w0uter

My UDF's:;mem stuff_Mem;ftp stuff_FTP ( OLD );inet stuff_INetGetSource ( OLD )_INetGetImage _INetBrowse ( Collection )_EncodeUrl_NetStat_Google;random stuff_iPixelSearch_DiceRoll

Share this post


Link to post
Share on other sites

Nice code....  LxP

Thanks! :dance:

[edit] you forgot a space before ms. [/edit]

Nah... that's just how I write things -- 1.44MB, 20s... :whistle:

Share this post


Link to post
Share on other sites

Hi,

thank u. But the scripts you wrote took more than 1200 ms. Maybe StringReplace has a problem with the number of data.

It think in my script the function replaces just the words in one line at writes it into the new V.xml file.

Maybe in your scripts: You read the whole model.xml file, and the you start to replace. So far so good. That idea has to be faster. :whistle:

But it is possible that StringReplace starts again after replacing one hit? :dance:

I don´t know, but your scripts are definitely slower than mine. ( 4 times!)

So long,

Mega


Scripts & functions Organize Includes Let Scite organize the include files

Yahtzee The game "Yahtzee" (Kniffel, DiceLion)

LoginWrapper Secure scripts by adding a query (authentication)

_RunOnlyOnThis UDF Make sure that a script can only be executed on ... (Windows / HD / ...)

Internet-Café Server/Client Application Open CD, Start Browser, Lock remote client, etc.

MultipleFuncsWithOneHotkey Start different funcs by hitting one hotkey different times

Share this post


Link to post
Share on other sites

#7 ·  Posted (edited)

You are right -- there's definitely a huge decrease in speed with my script, which takes almost two full seconds to execute here.

Devs, why is this so? :whistle:

Edit: I tested Mega's theory of StringReplace() starting from the beginning after each replace, but that would imply that the following script outputs 'x':

local $test = "xxxxxxxxxx"
msgBox(0x40, "StringReplace()", stringReplace($test, "xx", "x"))

which it doesn't. It outputs 'xxxxx'. So what's going on? :dance:

Without seeing the code (I doubt that I could understand it at this stage anyway), my initial guess is that it must have to rewrite the entire string in memory each time it makes a replacement, and since the resulting string is larger it requires more memory and therefore it must be reallocated.

Therefore the 1MB-long string is rewritten in memory over 300 times. No wonder it takes so long with my method!

Edited by LxP

Share this post


Link to post
Share on other sites

The file handle was not closed in the script in the 1st post. There would have to be a time penalty for that. You may find the file is not readable for not closing it. That is a incorrect time reading because of it. :whistle:

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

\dev\null has submitted an UDF to do this function and we have been doing some test what is the fastest way to do this,

The string replace is relative slow on a big string so the fastest way is to read the file into an array, then process the array entries and write the records to the file.

here's the fastest version so try this to see if that helps:

func _ReplaceStringInFile($szFileName, $szSearchString, $szReplaceString, $fCaseness = 0, $fOccurance = 1)

    local $iRetVal = 0

    if Not FileExists($szFileName)then 
        SetError(1)
        return -1
    endif

    $szTempFile = _TempFile()
    $hWriteHandle = FileOpen($szTempFile,2)
    if $hWriteHandle = -1 then 
        SetError(2)
        return -1
    endif

    Local $T_Out = ""
    Local $T_recs
    _FileReadToArray($szFileName,$T_recs)

    For $x = 1 to $T_recs[0] 
    ;======================================================================
    ;== check if the search string is found in the current file line
    ;======================================================================
        If $fOccurance = 0 and $iRetVal > 0 then
            FileWriteLine($hWriteHandle,$T_recs[$x])
        Else
            if StringInStr($T_recs[$x],$szSearchString,$fCaseness) then
                $iRetVal = $iRetVal + 1
                FileWriteLine($hWriteHandle, StringReplace($T_recs[$x],$szSearchString,$szReplaceString,0,$fCaseness))
            else 
                FileWriteLine($hWriteHandle,$T_recs[$x])
            ;$T_Out = $T_Out & $T_recs[$x] & @CRLF
            endif 
        EndIf
    Next

    FileClose($hWriteHandle)
    
;======================================================================
;== Delete the original file
;======================================================================
    if FileDelete($szFileName) = 0 then
        SetError(4)
        return -1
    endif

;======================================================================
;== Rename the temp file to the original file
;======================================================================
    if FileMove($szTempFile,$szFileName) = 0 then
         SetError(5)
        return -1
    endif

    return $iRetVal
endfunc;==> _ReplaceStringInFile
Edited by JdeB

Visit the SciTE4AutoIt3 Download page for the latest versions        Beta files                                                          Forum Rules
 
Live for the present,
Dream of the future,
Learn from the past.
  :)

Share this post


Link to post
Share on other sites

#10 ·  Posted (edited)

Hello,

I tried your function, but it takes nearly 900 ms. :dance:

Did I use your function right?

Why is AutoIt so slow?

I just started to learn AuoiIt and now I´m searching for some reasonable tasks. :whistle:

Thanks @all : It´s really fun dealing with AutoIt

So long,

Mega

#include <file.au3>

$szFileName = "model.xml"

$szSearchString = "filter"

$szReplaceString = "valhalla"

$fCaseness = 0

$fOccurance = 1

$b = TimerInit()

Call(_ReplaceStringInFile($szFileName, $szSearchString, $szReplaceString, $fCaseness = 0, $fOccurance = 1))

MsgBox(0,"Time", "The Operation took " & TimerDiff($:dance: & " ms")

Exit(0)

func _ReplaceStringInFile($szFileName, $szSearchString, $szReplaceString, $fCaseness = 0, $fOccurance = 1)

local $iRetVal = 0

if Not FileExists($szFileName)then

SetError(1)

return -1

endif

$szTempFile = _TempFile()

$hWriteHandle = FileOpen($szTempFile,2)

if $hWriteHandle = -1 then

SetError(2)

return -1

endif

Local $T_Out = ""

Local $T_recs

_FileReadToArray($szFileName,$T_recs)

For $x = 1 to $T_recs[0]

;======================================================================

;== check if the search string is found in the current file line

;======================================================================

If $fOccurance = 0 and $iRetVal > 0 then

FileWriteLine($hWriteHandle,$T_recs[$x])

Else

if StringInStr($T_recs[$x],$szSearchString,$fCaseness) then

$iRetVal = $iRetVal + 1

FileWriteLine($hWriteHandle, StringReplace($T_recs[$x],$szSearchString,$szReplaceString,0,$fCaseness))

else

FileWriteLine($hWriteHandle,$T_recs[$x])

;$T_Out = $T_Out & $T_recs[$x] & @CRLF

endif

EndIf

Next

FileClose($hWriteHandle)

;======================================================================

;== Delete the original file

;======================================================================

if FileDelete($szFileName) = 0 then

SetError(4)

return -1

endif

;======================================================================

;== Rename the temp file to the original file

;======================================================================

if FileMove($szTempFile,$szFileName) = 0 then

SetError(5)

return -1

endif

return $iRetVal

endfunc;==> _ReplaceStringInFile

Edited by th.meger

Scripts & functions Organize Includes Let Scite organize the include files

Yahtzee The game "Yahtzee" (Kniffel, DiceLion)

LoginWrapper Secure scripts by adding a query (authentication)

_RunOnlyOnThis UDF Make sure that a script can only be executed on ... (Windows / HD / ...)

Internet-Café Server/Client Application Open CD, Start Browser, Lock remote client, etc.

MultipleFuncsWithOneHotkey Start different funcs by hitting one hotkey different times

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0