Jump to content

Pre-compile include-file into an autoit readable binary?


Recommended Posts

I have an au3 include file which contains just one big 2D array with 3 elements in each line, and around 120 000 lines in total. When I include this file in my script and load it, it takes roughly 4 minutes until it's been fully loaded into memory and my main script starts.

Is there any way I could pre-compile, or somehow turn this file (11 MB) containing the array into a binary which is then loaded (faster I assume since it would be in binary format?) by my main script?

Link to post
Share on other sites

Can you post a short sample of how you declare and  initialize your array?

4 minutes is really a huge time for loading 11Mb of data.  This example, including computed init data, runs in under 2s from Scite on my very, really very slow PC:

Local $a[120000][3]

For $i = 0 To UBound($a) - 1
    $a[$i][0] = $i * $i
    $a[$i][1] = ChrW($i + 256)
    $a[$i][2] = Random(0,1)
Next

To answer your last question, AutoIt doesn't offer a way to build object files, like true compilers do.

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to post
Share on other sites

Sounds really long to read a text file and split it into columns

  1. Read the full textfile in a variable
  2. Split it with a regex into an array

example like below based on fixed columns of size 2,2,3 and a CRLF at end of each line

#include <StringConstants.au3>
#include <array.au3>

$strData="ABCDEFG" & @CRLF
$strData&="HIJKLMN" & @CRLF
$strData&="OPQRSTU" & @CRLF
$strData&="VWXYZAB" & @CRLF

;~ $strData=fileread("data.txt")
consolewrite($strdata)

;~ Split fixed line
$aArray=StringRegExp($strData,"(.{2})(.{2})(.{3})(\R)",$STR_REGEXPARRAYGLOBALFullMATCH)
consolewrite(@error&@CRLF)

Local $aMatch = 0

For $iRow = 0 To UBound($aArray) - 1
    $aMatch = $aArray[$iRow]

    For $iCol = 1 To UBound($aMatch) - 1
       consolewrite($iRow & ',' & $icol & ":" & $aMatch[$icol] &@CRLF)
    Next
Next

 

Link to post
Share on other sites
17 hours ago, jchd said:

Can you post a short sample of how you declare and  initialize your array?

Round 2

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to post
Share on other sites
20 hours ago, jchd said:

Can you post a short sample of how you declare and  initialize your array?

In my main script I do an include:

#include "_bigarray.au3"

 

And the contents of this file are as follows:

; _bigarray.au3
    #include-once
    global $aArray1 = [ _
['String1', 'String2', 0.71711], _
['String1', 'String2', 0.70803], _
['String1', 'String2', 0.70042], _
; ... and so on, around 10000 lines more ...
['String1', 'String2', 0.44563]] ; last line
    ; i have around 12 of the above arrays

 

I tried out your sample code and it only took me about one second.

Link to post
Share on other sites

@Danp2 Basically I am using the data stored in the array like a database. But I would really like to avoid using a database to keep my script free of dependencies.

 

@Thopaga @junkew

Thanks, I think this could be a good solution for what I am trying to do.

I tried junkew's script for a text file with 120k sample lines and it took only 1 second to process the file and turn it into an array, which would be way faster than how I am doing it right now with an include.

Edited by yyywww
Link to post
Share on other sites

When the array dimension(s) are not explicitely specified, the interpretor has to parse the whole init part in full detail to determine the actual dimension(s), before actually loading data into memory.

For instance, this:

Global $aArray1 = [ _
    ['String1', 'String2', 0.71711], _
    ['String1', 'String2', 0.70803], _
    ['String1', 'String2', 0.44563]] ; last line

results in:

Array[3][3]
      [0][0] => String (7)         'String1'
      [0][1] => String (7)         'String2'
      [0][2] => Double             0.71711
      [1][0] => String (7)         'String1'
      [1][1] => String (7)         'String2'
      [1][2] => Double             0.70803
      [2][0] => String (7)         'String1'
      [2][1] => String (7)         'String2'
      [2][2] => Double             0.44563

But this:

Global $aArray2 = [ _
    ['String1', 'String2', 0.71711], _
    ['String1', 'String2', 0.70803], _
    ['String1', 'String2', 0.44563, "Gotcha", "bigger than you thought", "and much bigger really!"]] ; last line

yields:

Array[3][6]
      [0][0] => String (7)         'String1'
      [0][1] => String (7)         'String2'
      [0][2] => Double             0.71711
      [0][3] => String (0)         ''
      [0][4] => String (0)         ''
      [0][5] => String (0)         ''
      [1][0] => String (7)         'String1'
      [1][1] => String (7)         'String2'
      [1][2] => Double             0.70803
      [1][3] => String (0)         ''
      [1][4] => String (0)         ''
      [1][5] => String (0)         ''
      [2][0] => String (7)         'String1'
      [2][1] => String (7)         'String2'
      [2][2] => Double             0.44563
      [2][3] => String (6)         'Gotcha'
      [2][4] => String (23)        'bigger than you thought'
      [2][5] => String (23)        'and much bigger really!'

That implies much more work at init time than this:

Global $aArray2[][6] = [ _
    ['String1', 'String2', 0.71711], _
    ['String1', 'String2', 0.70803], _
    ['String1', 'String2', 0.44563, "Gotcha", "bigger than you thought", "and much bigger really!"]] ; last line

because by finding the column dimension, the interpreter only has to count rows and can store data on the fly.
Better still, provide the actual dimensions at declaration [10000][2], since values are known in advance or easily counted.

This slowdown isn't noticeable when arrays are small but can waste enough cycles to make a difference in your use case.

 

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to post
Share on other sites

@jchd Thank you, that was enlightening to me.

But it is strange, when I change each Array to its specific size [10000][3] like you suggested, it still takes roughly the same amount of time? 143 seconds with that size declaration for each array and 143.3 seconds without.

Link to post
Share on other sites

You're right. The following code takes around 65 s on my slow, slow PC.

Local $h = FileOpen("arr12x10kx3.au3", 2)
FileWriteLine($h, '#include-once' & @CRLF)
For $i = 1 To 12
    FileWriteLine($h, 'Local $a' & $i & ' = [ _')
    For $j = 1 To 9999
        FileWriteLine($h, ' ["abcdefghijklm", "nopqrstuvwxyz", 0.123456789], _')
    Next
    FileWriteLine($h, ' ["abcdefghijklm", "nopqrstuvwxyz", 0.123456789] _')
    FileWriteLine($h, ']' & @CRLF)
Next

Local $t = TimerInit()
#include "arr12x10kx3.au3"
ConsoleWrite(TimerDiff($t))

Anyway consider SQLite. Remember you can FileInstall the SQLite library.  Of course accessing an array via index will be faster than a database, but the advantage of a DB goes way beyond accessing a single item thru individual indices.  It all depends on how you'll be accessing data and what you intend to compute (e.g. statistical or partial results).

Edited by jchd
Paste error

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to post
Share on other sites
  • it will really depend on your usecase requirements if include au3 file array or database usage or txtfilebased or ...  approach is best
  • resources can be included in your compiled exe approcah:  so data.txt could be directly get with function _ResourceGetAsString
  • I would suggest to use some kind of integer constant value to indicate string1 or string2 or was it just in your example any random character string?

 

 

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...