Autoit Compiled (EXE) speed issue (solved)

[CFLam]

Hello, I am using Autoit v3.3.12.0 and my project is a Markov Chain calculation which involve millions calculations and several external program (one of them is Excel).

My program logic is like this:

for $i=1 to 400

 for $j=1 to 420

 <do my calculation>

next i,j

then launch external program for further process

I timed that the calculation is around 80 seconds, then the whole calculation eed 156 days!!!!

In order to speed up the process,I compiled the program to exe (x64) as prog.exe and I copy prog.exe t prog1.exe, prog2.exe... prog8.exe. Each with an ini file to control the range of $i e.g.

prog1.exe $i=1 to 50

prog2.exe $i=51 to 100

...

prog8.exe $i=351 to 400

And then I run prog1 to prog8 at the same time. However, with 8 prog execute at the same time, the calculation time increase from 80 seconds to 300 seconds!!!

I know that run so many exe will sacrifice some performance, however, I didn't experience any "slow/jerky" in using Windows while the 8 prog running, this mena that my PC is not working full power on those 8 prog (I already use processsetpriority to set all prog* to 5). My problem is:

How to increase the performance of my prog ? Or, how to make use of all my PC's power on those program ?

I am running Windows 10 home edition with i7 quad core CPU, 16 GB ram. Thanks.

Also, I use traysettooltip to keep track the progress (which $i $j is working on ) and I find that even I run prog1~prog8 and the same time, I find that prog1 can maintain the speed (80 seconds) and prog8 is the slowest. I have the feeling (just feeling, no concrete evidence) that even I compile 8 prog*.exe, they use the same set of resource of my PC instead of different one. Therefore, prog* run slower and slower (prog2 slower than prog1 and prog3 slower than prog2) and other application (e.g. FireFox) seems not affected by those 8 prog(s) (I can even play Flash based online game without any performance degrade!!!). What I expected is:

prog1 ~ prog8 will become slower than 80 seconds but all 8 prog will perform more or less the same, not one slower than the previous one.

--------------------------------------------------------------------

(problem solved)

Thanks for everyone's help, eventually I find out the root issue of my problem.
I am using nested for loop in my program
 for $i=1 to 400
  for $j=1 to 420
in the middle of my program. I need to convert the product of i and j to string
e.g. 2x2=4 will convert to 000004, 3x8=24 will convert to 000024
the conversion make use a temp variable $tmpVar, however, other part of my program
also use this variable!!!!!! (Strange that program just become slow, and no error generated.)

Apart from the above reason, another cause tomy slow problem is memory issue. Autoit will "eat"
certain amount of memory each loop. I make use the _reducememory introduce in this forum to re-gain
memory, but the _reducememory only work in 32 bits version, and I compiled the program to x64 mode
which _reducememory not work!!! Therefore, I am forced to compile to 32 bits.
Any 64 bits version of _reducememory ?

 

Regds

LAM Chi-fung

Edited March 9, 2017 by CFLam
Update info

[Subz]

Are you able to provide more information on your script like what's happening between For and Next, you could use a timer (see TimerInit) so that you can find where the script is slowing down and logging it to a log file for further analysis.

[CFLam]

Hello Subz, the code is over 2000 lines and are reading a csv file into an array, store it in database (SQLite) and use query to perform certain calculations (based on Markov Chains theory) and store the result to another csv file. The output file is around 260,000 lines, inside my calculations, there is certain for loop ($k, $l, $m and $n). The inner loop output 8 lines of result, at first, I store the 8 lines result in memory and use arrayadd to concat the result and then write to file after $klmn. However, arrayadd become very slow when array growth (my calculation take over 1 hr!!!!!). I then change the logic to write out 8 lines to same file each $klmn (calculation become 80 seconds). Seems that I can't further reduce the calculation time, therefore I looking for another ways to improve performance e.g. use more than 1 exe to perform calculation.

 

Why not switch to another language? Because I love the array handling and SQLite of Autoit, also, I need to launch external program before andafter my calculation and control them, Autoit is the best choice in this aspect.

 

Regds

LAM Chi-fung

[mLipok]

3 hours ago, CFLam said:

I store the 8 lines result in memory and use arrayadd to concat the result and then write to file after $klmn. However, arrayadd become very slow when array growth (my calculation take over 1 hr!!!!!). I then change the logic to write out 8 lines to same file each $klmn (calculation become 80 seconds).

Use my Chilkat.au3 UDF to atomatically add lines to CSV without adding them to AutoIt ARRAY.
This should be faster in your case.

 

Edited February 16, 2017 by mLipok

[mLipok]

10 hours ago, CFLam said:

Also, I use traysettooltip to keep track the progress (which $i $j is working on )

Try to use ToolTip() instead  TraySetToolTip(), in my opinion displaying the second one is slower.

[RTFC]

For matrix operations (performing the same calculations on a grid), check out my Eigen4AutoIt  environment (link in my sig).

For increasing processing efficiency more generally, I would suggest placing your entire environment on a RAM disk (if it fits, and assuming you have suficient RAM to begin with; there's no point if the RAM disk's memory access is going to have to swap to disk itself). Much of the bottleneck in large computations is bus speed and I/O more generally, not computational power itself. Furthermore, if you have access to a LAN, install HTCondor (free), and farm out individual computing jobs to all idle CPU cores on the network. It's trivial to set up, and scales up your processing capacity by 2-3 orders of magnitude (at least for me).

Edited February 16, 2017 by RTFC
typos

[CFLam]

53 minutes ago, mLipok said:

Use my Chilkat.au3 UDF to atomatically add lines to CSV without adding them to AutoIt ARRAY.
This should be faster in your case.

 

Thanks a lot. I already use _FileWriteFromArray to output the result to a opened file instead of _ArrayAdd.

[mLipok]

Using Chilkat UDF you can try to add this to CSV as object, not to file.
Saving to file should be done at the end of processing.

 

[CFLam]

56 minutes ago, mLipok said:

Using Chilkat UDF you can try to add this to CSV as object, not to file.
Saving to file should be done at the end of processing.

 

Any example on this ? I just find example on how to read a csv file into csv obj, but not add array to a csv obj and write it out, thanks.

[mLipok]

Edit:

 

 

Edited February 16, 2017 by mLipok

[CFLam]

7 hours ago, mLipok said:

Edit:

When I tried compiled the code, it return the following

>"C:\Program Files (x86)\AutoIt3\SciTE\..\au3check.exe" -q "C:\Users\LAM Chi-fung\Downloads\Chilkat_example.au3"
"C:\Users\LAM Chi-fung\Downloads\Chilkat.au3"(157,93) : warning: $OBJ_PROGID: possibly used before declaration.
Return __ErrorLog_Chilkat('$oChilkat.UnlockComponent() ' & ObjName($oChilkat, $OBJ_PROGID)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Test\Chilkat.au3"(157,93) : error: $OBJ_PROGID: undeclared global variable.
Return __ErrorLog_Chilkat('$oChilkat.UnlockComponent() ' & ObjName($oChilkat, $OBJ_PROGID)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
C:\Test\Chilkat_example.au3 - 1 error(s), 1 warning(s)
>Exit code: 2    Time: 0.05162

It can run inside SciTE but not compiled. Also, seems that when I use setcell to add my 8x37 result array to the csv object, execution time even slower than I write (append) the 8x37 array by _FileWriteFromArray (increase from 80 seconds to 160 seconds).

W

[mLipok]

I just download 0.1.2

and use this as example:
 

expandcollapse popup

#AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w- 4 -w 5 -w 6 -w 7
;~ #AutoIt3Wrapper_Run_Debug_Mode=Y
#Tidy_Parameters=/sort_funcs /reel

#include <Array.au3>
#include <StringConstants.au3>
#include "Chilkat.au3"

_Example()
Func _Example()

    #Region - Chilkat_example.au3 - Chillkat ActiveX object initialization
    _ErrorLog_ChilkatWrapper(ConsoleWrite)
    _Chilkat_StartUp(@ScriptDir & '\ChilkatAx-9.5.0-win32.dll')
    _Chilkat_DllVersion($CHILKATOBJ_VERSION_950)

    Local $oGLOBAL = _Chilkat_GLOBAL_ObjCreate()
    If @error Then MsgBox($MB_ICONERROR, 'GLOBAL Declartion', '@error = ' & @error & @CRLF & '@extended = ' & @extended)

    _Chilkat_GLOBAL_UnlockBundle($oGLOBAL, "Anything for 30-day trial")
    If @error Then MsgBox($MB_ICONERROR, 'UnlockBundle', '@error = ' & @error & @CRLF & '@extended = ' & @extended)
    #EndRegion - Chilkat_example.au3 - Chillkat ActiveX object initialization

    _Example_CSV_CreateFile()
EndFunc   ;==>_Example

Func _Example_CSV_CreateFile()
    _Log_ChilkatExample('_Example_CSV_CreateFile')

    ; https://www.example-code.com/vbscript/csv_create.asp

    ;  Indicate that the 1st row
    ;  should be treated as column names:
    __MY_CSV_API(Null)
    __MY_CSV_API("year", "color", "country", "food")
    __MY_CSV_API("2001", "red", "France", "cheese")
    __MY_CSV_API("2005", "blue", "United States", "hamburger")

    Local $oCSV = __MY_CSV_API()

    ;  Write the CSV to a string and display:
    Local $sCSV_String
    $sCSV_String = $oCSV.SaveToString()
    MsgBox(0, '', $sCSV_String)

    ;  Save the CSV to a file:
    Local $bSuccess = $oCSV.SaveFile("_Example_CSV_CreateFile_Result.csv")
    If Not $bSuccess Then ConsoleWrite($oCSV.LastErrorText)

EndFunc   ;==>_Example_CSV_CreateFile

Func __MY_CSV_API($vCol0 = Default, $vCol1 = Default, $vCol2 = Default, $vCol3 = Default)
    Local Static $oCSV = Null

    If $vCol0 = Null Or $oCSV = Null Then ; Create New Object
        $oCSV = _Chilkat_CSV_ObjCreate()
        $oCSV.HasColumnNames = True
    ElseIf $vCol0 = Default And $vCol1 = Default And $vCol2 = Default And $vCol3 = Default Then
        Return $oCSV
    ElseIf $oCSV.NumColumns = 0 Then
        $oCSV.SetColumnName(0, $vCol0)
        $oCSV.SetColumnName(1, $vCol1)
        $oCSV.SetColumnName(2, $vCol2)
        $oCSV.SetColumnName(3, $vCol3)
    Else
        Local $iRow = $oCSV.NumRows
        $oCSV.SetCell($iRow, 0, $vCol0)
        $oCSV.SetCell($iRow, 1, $vCol1)
        $oCSV.SetCell($iRow, 2, $vCol2)
        $oCSV.SetCell($iRow, 3, $vCol3)
    EndIf

EndFunc   ;==>__MY_CSV_API

Func _Log_ChilkatExample($sData)
    ConsoleWrite('>+ ' & $sData & @CRLF)
EndFunc   ;==>_Log_ChilkatExample

and all is Ok.

 

[CFLam]

Thanks for everyone's help, eventually I find out the root issue of my problem.
I am using nested for loop in my program
 for $i=1 to 400
  for $j=1 to 420
in the middle of my program. I need to convert the product of i and j to string
e.g. 2x2=4 will convert to 000004, 3x8=24 will convert to 000024
the conversion make use a temp variable $tmpVar, however, other part of my program
also use this variable!!!!!! (Strange that program just become slow, and no error generated.)

Apart from the above reason, another cause tomy slow problem is memory issue. Autoit will "eat"
certain amount of memory each loop. I make use the _reducememory introduce in this forum to re-gain
memory, but the _reducememory only work in 32 bits version, and I compiled the program to x64 mode
which _reducememory not work!!! Therefore, I am forced to compile to 32 bits.
Any 64 bits version of _reducememory ?