CrypticKiwi

Help Make This Slow Function Faster

19 posts in this topic

Hey guys, 
I’m stuck with speed again with StringReplace() but this time it’s as slow as a snail. I realize I might have posted a seemingly similar post before but I choose to start a new topic because circumstances are different.
The function replaces 2 numbers with each other based on another number in another file: 

Scramble.jpg

 

 

$Handle1 = FileOpen("ExampleFile01.txt", 0)
$Example1 = FileRead($Handle1)

$Handle2 = FileOpen("ExampleFile02.txt", 0)
$Example2 = FileRead($Handle2)

_Scramble($Example1,$Example2)
Func _Scramble($CycleData,$Data)
    For $N = 1 To StringLen($CycleData) - 10
        Local $SMid = StringMid($Data, $N, 1)
        If $SMid <> 0 Then
            Local $Right = StringMid($CycleData, ($N + $SMid) - 1, 1)
            Local $Left = StringMid($CycleData, $N, 1)
            $CycleData = StringReplace($CycleData, $N, $Right)
            $CycleData = StringReplace($CycleData, Number(($N + $SMid) - 1), $Left)
        EndIf
    Next
    Return $CycleData
EndFunc   ;==>_Scramble


Now, this function takes around 1 hour 30 mins to finish a ~1mb file  on a core i5. the bigger problem is that I need to loop this function ~100000 times :-/
Any help would be greatly appreciated.  

ExampleFile01.txt

ExampleFile02.txt

Share this post


Link to post
Share on other sites



Hey CrypticKiwi,
I have rewritten your function to gain some performance, hope it helps.

$Handle1 = FileOpen("ExampleFile01.txt", 0)
$Example1 = FileRead($Handle1)

$Handle2 = FileOpen("ExampleFile02.txt", 0)
$Example2 = FileRead($Handle2)

_Scramble($Example1,$Example2)
Func _Scramble($CycleData,$Data)
    $CycleDataArray = StringSplit($CycleData,"")
    $DataArray = StringSplit($Data,"")
    $ReturnString = ""
    For $i = 1 To StringLen($CycleData) - 10
        If $DataArray[$i] <> 0 Then
            $Temp = $CycleDataArray[$i]
            $CycleDataArray[$i] = $CycleDataArray[$i + $DataArray[$i] - 1]
            $CycleDataArray[$i + $DataArray[$i] - 1] = $Temp
        EndIf
    Next
    For $i = 1 To StringLen($CycleData)
        $ReturnString &= $CycleDataArray[$i]
    Next
    return $ReturnString
EndFunc   ;==>_Scramble

 

1 person likes this

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

Even with a regex looping 1 million times will be long...

Edited by mikell

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

Thank you guys for your reply,

Special thanks to Mip101, this is a huge performance boost!! the file got _scrambled in less than 20 secs!! :D 

Even with a regex looping 1 million times will be long...

thats why im thankful its 100000 not a million.

 

Edited by CrypticKiwi

Share this post


Link to post
Share on other sites

Is that for an encryption scheme? Because 20*10000 seconds (> 23 days) for processing 1Mb file isn't too sexy.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#6 ·  Posted (edited)

to be faster make de replace part directly in memory. I rewrite your code using structure and making son changes. I should be at least 96% faster based on 100%.

$Handle1 = FileOpen("ExampleFile01.txt", 0)
$Example1 = FileRead($Handle1)

$Handle2 = FileOpen("ExampleFile02.txt", 0)
$Example2 = FileRead($Handle2)

Local $hTimer = TimerInit()
Local $Salida=_Scramble($Example1,$Example2)
Local $fDiff = TimerDiff($hTimer)
MsgBox(0, "Time Difference", $fDiff)

FileWrite("Output.txt",$Salida)

FileClose($Handle1)
FileClose($Handle2)
Func _Scramble($CycleData,$Data)
    Local $Right=""
    Local $Left=""
    Local $SMid=""
    Local $iSize=StringLen($CycleData)
    Local $tCycleData=DllStructCreate("char Data[" & $iSize & "]")
    $tCycleData.Data=$CycleData
    $CycleData=""
    $iSize=$iSize-10
    For $N = 1 To $iSize
        $SMid = StringMid($Data, $N, 1)
        If $SMid <> 0 Then
            $Right=DllStructGetData($tCycleData,1,($N + $SMid) - 1)
            $Left=DllStructGetData($tCycleData,1,$N)
            DllStructSetData($tCycleData,1,$Right,$N)
            DllStructSetData($tCycleData,1,$Left,Number(($N + $SMid) - 1))
        EndIf
    Next
    Return $tCycleData.Data
EndFunc   ;==>_Scramble

Saludos

 

 

Edited by Danyfirex

Share this post


Link to post
Share on other sites

#7 ·  Posted (edited)

 

Is that for an encryption scheme? Because 20*10000 seconds (> 23 days) for processing 1Mb file isn't too sexy.

Nope theoretical  data compression algo. and indeed it aint sexy at all considering ill wait 23 days check result fix stuff repeat!  

yet Mip101's code halves the 23 days with a whooping 39% on my desktop.

on old core i5 laptop
Mip101's = 22604.22
Dany's = 42909.00

on old core 2 duo Desktop
Mip101's = 9040.88 Best So far
Dany's = 20454.23

 

Edited by CrypticKiwi

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

I forgot that Autoit Memory access is slow. so I write this just for fun and for x86. I should take less that 1 second even old Pentium Machine as mine one.

$Handle1 = FileOpen("ExampleFile01.txt", 0)
$Example1 = FileRead($Handle1)
$Handle2 = FileOpen("ExampleFile02.txt", 0)
$Example2 = FileRead($Handle2)
Local $hTimer = TimerInit()
Local $Salida = _Scramble($Example1, $Example2)
Local $fDiff = TimerDiff($hTimer)
MsgBox(0, "Time Difference", $fDiff)
FileWrite("Output.txt", $Salida)
FileClose($Handle1)
FileClose($Handle2)


Func _Scramble($CycleData, $Data)
    Local $lenCycleData = StringLen($CycleData)
    Local $lenData = StringLen($CycleData)
    Local $tCycleData = DllStructCreate("byte Data[" & $lenCycleData & "]")
    Local $tData = DllStructCreate("byte Data[" & $lenData & "]")
    Local $pData = DllStructGetPtr($tData)
    Local $pCycleData = DllStructGetPtr($tCycleData)
    Local $sByteArray = "0x5589E5BE000000008B4D088B550C8B7D10B800000000BB000000008A04372C303C0074253C0072063C097702EB05B801000000BB0000000001F083E8018A1C028A3C32881C32883C0246E2C55DC20C00"
    Local $tByteArray = DllStructCreate("byte Data[" & BinaryLen($sByteArray) & "]")
    $tByteArray.Data = $sByteArray
    $tData.Data = $Data
    $tCycleData.Data = $CycleData
    DllCallAddress("none", DllStructGetPtr($tByteArray), "int", $lenCycleData - 10, "ptr", $pCycleData, "ptr", $pData)
    Return $tCycleData.Data

EndFunc   ;==>_Scramble

Saludos

 

Edited by Danyfirex
1 person likes this

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

@Danyfirex: Can you show us the details for

Local $sByteArray = "0x5589E5BE000000008B4D088B550C8B7D10B800000000BB000000008A04372C303C0074253C0072063C097702EB05B801000000BB0000000001F083E8018A1C028A3C32881C32883C0246E2C55DC20C00"

please?

 

Thx.

Edited by UEZ

Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

Share this post


Link to post
Share on other sites

@UEZ   it is awful assembly code (is not optimized). but if you insist :) (lol) look:

push ebp
mov ebp, esp
mov esi, 0
mov ecx, [ebp + 08]
mov edx, [ebp + 12]
mov edi, [ebp + 16]
Bucle :
mov eax, 0
mov ebx, 0
mov al, byte[edi + esi]
sub al, 48
cmp al, 0
je Incre
cmp al, 0
jb Clear
cmp al, 9
ja Clear
jmp Mid
Clear :
mov eax, 1
Mid :
mov ebx, 0
add eax, esi
sub eax, 1
mov bl, byte[edx + eax]
mov bh, byte[edx + esi]
mov byte[edx + esi], bl
mov byte[edx + eax], bh
Incre :
inc esi
loop Bucle
pop ebp
ret 12

Saludos 

1 person likes this

Share this post


Link to post
Share on other sites

#11 ·  Posted (edited)

Here again, native AutoIt strings are UTF16-LE (wchar[N]), not ANSI (char[N]). The distinction doesn't seem to make a difference with the little that the OP told us but if one somehow someday has to handle non ASCII digits, then bytes are not enough.

Edited by jchd

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

#12 ·  Posted (edited)

@Danyfirex: When I saw this thread my idea was to code also some ASM lines to increase my skills but I saw that you did it already and I was curious about the code. :)

I'm also an ASM rookie... ;)

Thanks for sharing! :thumbsup:

Edited by UEZ

Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

Share this post


Link to post
Share on other sites

#13 ·  Posted (edited)

@jchd what's your point?  I got it.

You're wellcome @UEZ.You' have nice assembly posts. I had months without write anything in assembly. 

 

Saludos

Edited by Danyfirex

Share this post


Link to post
Share on other sites

@Danyfirex

For an ASM noob like me, that ASM function is great! Thanks for sharing :)

Share this post


Link to post
Share on other sites

OMG Danyfirex this is what they call magic... from 90 minutes to 18 ms im impressed!!

Share this post


Link to post
Share on other sites

Yes. Assembly look magic. That would be more or less the result if you do it in C / C ++. Autoit is a little slow for this type o things. (That does not take away the love I have for it.)  maybe I'll write some assembly alternative for some string funtions as replace when we need to do some in  very long strings.

Saludos

 

Share this post


Link to post
Share on other sites

A more flexible alternative if you accept either installing first or FileInstall-ing things, you could submit your C source to TCC, compile + link it on the fly (to .exe or DLL) and invoke it for a (relatively) comparable speed.

More steps, slightly slower, but finally much more versatile than hardcoding machine code in the application.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

True, but you gain the joy of C obfuscation.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now