Eigen4AutoIt - Matrix computing with Eigen

[RTFC 353]

The Eigen C++ template library is a great environment for matrix computing; it is fast, reliable, extensive, and well-documented. It is also completely free, and does not rely on any external dependencies. Unfortunately for AutoIt users, the term “template library” implies that any functions you call are only instantiated upon compilation (in C++). That means there's nothing to hook into.

To make Eigen ’s most important functionality directly accessible from within AutoIt scripts (version 3.3.12+, download it here), I developed the Eigen4AutoIt environment. It runs on Windows and under Wine also on Linux and Mac (Ubuntu, Debian, Fedora, and MacOS supported by WineHQ), and SUSE, Slackware, and FreeBSD supported by the distros).

>Download the latest version

It consists of:

1) Eigen4AutoIt.au3

• an AutoIt library of wrapper functions that contain extensive bounds checks, matrix management, file I/O, type conversion, and precision control, and two-way data exchange with files and native AutoIt arrays;

2) Eigen-wrapper dlls (EigenDense.dll, EigenDense_x64.dll)

• re-maps matrices created in AutoIt as Eigen Matrix objects, then calls Eigen’s powerful core functions; in the spirit of open-source, the full C++ source code I wrote is included in the bundle (see subdirectory "source"). The basic functions consist of a single Eigen call; decompositions and statistics are more involved.

3) Additional materials:

• the user-interactive, animated Function Selector and MatrixViewer tools
• the MatrixFileConverter to read/write E4A matrices from/to .csv ASCII, Excel, and Xbase files.
• three libraries of scientific and mathematical constants
• online Help, with example code for every function
• Large (.chm) Help file (for offline access)
• Quickstart Manual (11-page pdf, updated)
• Test suite
• Tutorials from Basics to Advanced

Please note:

• none of this is part of Eigen's own distribution
• you only need this bundle; you do not need to install Eigen.

How it works:
No matrix content is ever transferred, only memory pointers, meaning computations in AutoIt are virtually as fast as in Eigen’s native environment, with the added advantage of not having to be compiled first. The drawback is that Eigen's native ad-hoc expression templates (and their internal optimisations) cannot be exploited here; you have to construct your operations with the basic building blocks. These provide matrix creation, I/O, cellwise operations, reduction, multiplication, transformation, decomposition (LU, Householder, Choleski, and Jacobi SVD; these include general linear solvers) and a small statistics module by yours truly.

IMPORTANT: Posting Rules for this thread:
 
1) Do not post or PM me any matrix-, maths-, or Eigen-related questions. Eigen has its own User Forum for that (or try math.stackExchange.com). I am not your maths guru! If you post such questions, I will either ignore your post or remind you of this rule.

2) Do not post or PM me your data sets and/or non-working Eigen4AutoIt scripts; I will not analyse your data or fix your scripts for you! There are many reasons why a linear algebra procedure might fail to produce the answer you expect. You are wielding great mathematical power here, so exploit the fantastic internet resources at your fingertips and learn how to use it. To get you started, I've listed a few video tutorials and other helpful materials in the header remarks of Eigen4AutoIt.au3. Also check out the test scripts, the Tutorials, and the Help file.

3) I do warmly welcome all of the following:

• remarks, advice, suggestions for improvements, encouragement, cash;
• bug reports re. the Eigen4AutoIt interface (individual functions that don't work the way they should) and/or the associated dll code (ditto);
• your own working Eigen4AutoIt templates of general use that you'd like to see implemented in a future release.

Regarding that last item, have a look at my PCA tutorial. After the step-by-step stage, I summarise the entire procedure in a "mini script" of Eigen4AutoIt calls. Then I introduce the two internal PCA functions I developed, which replace that script with two simple calls. You can do the same thing, and submit your own functional Eigen4AutoIt script in this thread. If I consider it of general use and can implement it, it may become a new Eigen4AutoIt function in the next release (with source acknowledgement, of course). This means that you'd get a precompiled dll version of your script that will likely run faster and be simpler to call. Thereby this thread would become an Eigen4AutoIt Example Scripts mini forum. It's just a thought.
 

>Download the latest version (uncompressed size: 38.4 MB)

 

How to Install

You need AutoIt version 3.3.12 or later. Simply run the installer, navigate to the tutorial subdirectory, open the first tutorial ("Basics") in Scite, and start it. This shows basic matrix I/O and mode switching (single versus double precision). If that runs, you're in business.

NB to leverage the full power of x64 features, you'll also need to have the full Scite4AutoIt3 package installed. For more info, see sections "Bitness" and "Shared Memory" in the Help, main topic: "Work Environment" page.

Edited Saturday at 03:26 PM by RTFC

[ValeryVal 8]

It starts running for me with following function _CallStack_Push($procname) in file Eigen4AutoIt.au3:

Func _CallStack_Push($procname)

    If $procname="" Then Return
;   _ArrayAdd($Callstack,$procname,0,Chr(0))
    _ArrayAdd($Callstack,$procname&"0"&Chr(0))
    $ShowErrorMsg=True

EndFunc

and with true path for $EIGEN_DLLPATH - folder with file Eigen4AutoIt.au3.

Thanks

[RTFC 353]

@ValeryVal: Thanks for your response.

Perhaps I should have emphasised that this library is written for AutoIt version 3.3.12; anything array-related will probably fail in earlier versions, because of the recent script-breaking changes in Array.au3. I'll stress this in the first post. Thanks a lot.

You will likely encounter other issues later on if you attempt to run in earlier AutoIt versions, even if the first tutorial runs normally. Please upgrade to AutoIt 3.3.12 before testing. See also the remarks in the "How to Install" section in the QuickStart Manual. Best of luck!

 

Edit: Okay, I've added an AutoIt version check to alert people to this. Thanks again for flagging this, ValeryVal.

Edited July 21, 2014 by RTFC

[ValeryVal 8]

@RTFC

Can you die from laughter? I have your very long test scripts completed with success.

The only note (apart from noted above) in MatrixDisplay.au3 is for me:

;   Local $asHeader = StringSplit($sHeader, $sCurr_Separator, $STR_NOCOUNT) ; No count element
    Local $asHeader = StringSplit($sHeader, $sCurr_Separator, 2) ; No count element

Would you like to add source of Eigen4AutoIt-QuickStart_Manual.pdf in any form (.doc, html, sdt) to simplify the translation it into Russian?

Thank you.

[RTFC 353]

Hi ValeryVal,

That's fast work! Glad you enjoyed the ride.

Regarding the MatrixDisplay.au3 script, the current trend in AutoIt is to move away from "magic numbers" and use sensibly-named constants (this one is in StringConstants.au3 in AutoIt version 3.3.12), and as Eigen4AutoIt is supposed to be run in that environment (or future versions) I'll keep using that constant, to make the code more transparent and legible to future users. You are of course completely free to edit it as you see fit for your own environment; whatever works for you.

I'll PM you the powerpoint source of the manual; a Russian translation would be great!   I hope you'll post a link to the result in this thread so others may benefit from it too. I'm not adding the powerpoint source to the package because I've almost run out of forum file storage space myself, and most users won't need it. Anyone else wishing to provide a translation, just PM me.

Edited July 24, 2014 by RTFC

[RTFC 353]

Beta Version 0.8 released.

Statistics module: several bug fixes plus new functions for skewness and kurtosis (biased and unbiased).

[ValeryVal 8]

@RTFC

Quick Tutorial of EigenAutoIt has been translated.

Introduction into linear algebra algorithm (Russian):

http://www.tdoc.ru/c/programming/programming-theory/matrix-vectors-values.html

Thanks

Edited September 22, 2014 by ValeryVal

[RTFC 353]

Fantastic, ValeryVal, it looks great!

I hope you can advertise it at autoit-script.ru too (if you're a member there).

Your contribution should definitely help to make this environment more accessible to Russian-speakers.

Thank you very much for your hard work!

RT

 

Edit: in a PM to ValeryVal I mentioned that he missed translating my favourite Spoon Boy Matrix quotation, hence the following response...

Edited August 1, 2014 by RTFC

[ValeryVal 8]

@RTFC

Quick Tutorial of EigenAutoIt has been fixed (see above). Now spoon is nowhere in accordance to Spoon boy.

 

 

[RTFC 353]

Ah, yes, now the universe makes sense again!

Stay tuned, folks, the next release with added functionality is coming soon...

[RTFC 353]

Beta version 0.9 released.

Aside from bug fixes in the <alternative operator> lookup, and, following a suggestion by ValeryVal, an extra manual page detailing what outputs are actually produced by each decomposition (and associated with which matrix letter), several new functions are added:

Conditional Cwise Operators:

_Eigen_ConditUnaryOp
_Eigen_ConditScalarOp
_Eigen_ConditBinaryOp
These are Cwise operations acting on the entire matrix that are performed only on cells that satisfy a condtion (equal/unequal/smaller/larger/smaller-or-equal/larger-or-equal ) with respect to a separately parsed scalar value. See new test script _EIgenTest_12_ConditionalCwise.au3 for details.

Conditional evaluation:

_Eigen_ConditAll         ; returns True if all cells satisfy condition, otherwise False
_Eigen_ConditAny       ; returns True if any cell satisfies condition, otherwise False
_Eigen_ConditCount   ; returns number of cells that satisfy condition

See new test script _EIgenTest_12_ConditionalCwise.au3 for details.

These act either on the entire matrix or on a specific Col or Row.

In this context, I should mention that MatrixSpecs ID 16 (nonZeros) does not return the number of cells currently containing the value zero,   but the number of cells that have ever been assigned any value. So you'll need _Eigen_ConditCount to obtain the number of cells containing zero (or any other value).

Data retrieval from $MatrixList:

_Eigen_GetMatrixCols     ; return specified dimension for parsed matrix ID
_Eigen_GetMatrixRows   ; return specified dimension for parsed matrix ID
_Eigen_GetMatrixType     ; return matrix type string for parsed matrix ID

_Eigen_ReDimExistingMatrix

Unlike _Eigen_Transpose, this function leaves data storage unchanged, but changes the (vector or) matrix shape. So one could use it to change a rowvector to a colvector (without transposition). Note that this only works if the total number of cells (given the new user-defined dimensions) remains the same. Example:

Original 3 x 2 matrix

1  4

2  5

3  6
 

Transposed 2 x 3 matrix:

1  2  3

4  5  6

new ReDim  2 x 3 matrix:

1  3  5

2  4  6

I am currently starting work on a separate Import/export utility, to transfer data between matrix files and other file types, but this will take a while yet to finish.

Finally, I am happy to announce that Eigen now has a link to the Eigen4AutoIt thread here.

Edited August 1, 2014 by RTFC

[ValeryVal 8]

Quick Tutorial of EigenAutoIt (Russian - v.0.9) has been fixed (see above).

[RTFC 353]

Version 1.0 released, now with separate MatrixFileConverter utility.

Additionally:

Various minor issues fixed (bounds checks, matrix type conversion).

New UDF: _Eigen_RedimExistingMatrixFile(), works as _Eigen_RedimExistingMatrix(), but acting directly on a matrix file (editing its header), so no need to load the full matrix into memory first.

The MatrixFileConverter is a GUI that allows you to transfer and convert numerical content between binary matrix files as generated by Eigen4AutoIt on the one hand, and three external file formats: 1) ASCII delimited (.txt, .csv, .sdf, ...) with a choice of delimiters, 2) Excel Worksheets (provided you have Excel installed), or 3) Xbase files (dbf database files). Usage is pretty intuitive: define a matrix file and a target file, select the direction of transfer, and press the big Convert button. If a filename is not found, it is created in the specified directory.

Some formatting options (scientific notation, binary storage) are available from the GUI, others (such as the Xbase signature byte) can be altered by editing the auto-generated MatrixFileConverter.ini file.

If you need to convert lots of files, you may find the cmdline options useful (see Remarks in script header for a list of switches).

Options involving complex numbers are not yet functional in this release.

Note that unlike ASCII and Xbase conversions (which transfer directly between files), the Excel option uses an AutoIt array as intermediate buffer, and is thereby restricted to a maximum of 16 MB elements. If you need to import/export larger Excel files, use the CSV (comma-separated values) ASCII format as external intermediate instead. This is also the way to handle data in Matlab matrix files; like Excel, Matlab has its own sophisticated file im/export utility for that.

Finally, Xbase aficionados will be familiar with the limitation on the number of database fields (127 in dBase III / IV, 2047 in Xbase). To circumvent this, I've added a custom feature to store wider matrices (2D) as vectors (1D, ColMajor order); when converting such output back to a matrix file later on (with MatrixFileConverter!), the 2D structure (temporarily squeezed in a few rarely-used header fields) is rebuilt. For details on the Xbase specification, see the new include Xbase.au3 (link in signature).

Edited August 29, 2014 by RTFC

[RTFC 353]

Version 1.1 released.

Mainly a housekeeping update; there were still several inconsistencies between the AU3 wrappers and the dll wrappers.

There are several undocumented experimental functions involving complex datatypes in the dll for which no AU3 wrapper yet exists.

The MatrixFileConverter had a display issue with verylongpathnameswithverylongfilenames that were truncated incorrectly in the display.

The most significant internal change is a substantial expansion in the number of alias wrappers for many UDFs, making it much easier if you sort of approximately remember what a function was called, but not exactly (as happens to me a lot). This also addresses several function naming convention issues (re. section splitting) that are now increasingly standardised (at least the alias is). All original UDFs names will remain functional.

A few (very minor) UDFs were also added:
_Eigen_ShowUnaryOperators            ; list the available options
_Eigen_ShowScalarOperators
_Eigen_ShowBinaryOperators
_Eigen_ShowConditOperators
 

Edited August 29, 2014 by RTFC

[RTFC 353]

Version 1.2 released, with all new, 1.2 MB Help File!

See the Changelog in the Help file for all the latest changes, including a few new functions (mainly for display and MatrixList admin). The most significant change is a major overhaul of the function naming conventions. Writing the Help file made me aware of a number of inconsistencies which I tried to address. However, the old names still work as well (and will do indefinitely), thanks to additional alias wrappers. All test scripts have been edited to reflect the various changes.

The three Tutorials are also in the Help file, with lots of screenshots and some graphics where appropriate.

The >download link has been moved to Miscellaneous downloads, because the package is now far too big for me to host myself.

There are two script-breaking changes in version 1.2:

• Changed: _EigenBinaryOp and all its subfunctions had their parameter order changed so the last two are now always: $operator, $matB. THIS IS A SCRIPT BREAKING CHANGE.
• Changed: the large majority of functions now all behave in the same way in returning either (non-zero) target matrix ID or True upon success and False + nonzero @error flag upon failure. This affects for example all decompositions, which formerly returned an Eigen status code (now incorporated into error message upon failure). Notable exceptions are: all work environment functions (no return value), Show_<some list> functions (no return value), and functions that return a single value directly. THIS IS A SCRIPT BREAKING CHANGE.

 

I am grateful to everyone who contributed to the AutoIt help pages, which I took as my style template, to make it as easy as possible for AutoIt users to delve into Eigen4AutoIt. Having never before written a Help file myself, I never realised how much effort it takes to build one. Mine is of course still in its infancy, and doubtless still contains lots of typos, broken links, etc. Anyway, it's a start.

Edited September 18, 2014 by RTFC

[RTFC 353]

Forgot to mention that I've upgraded Eigen (and my dlls) to its latest release 3.2.2.

As you may have noticed, this causes an issue with Eigen's own assert messages no longer being displayed in some environments. I'm currently fixing this, but it should only affect the TestAssert function I call in the first Tutorial, and any malformed function calls that get past Eigen4AutoIt's own bounds checks (which is not that easy to do). Valid code is not affected.

Edit: Dlls updated, asserts should work now (in Debug mode).

Another detail I forgot to mention yesterday is that none of the Help file topics yet has an Example snippet. These will be added in the near future.

Stay tuned...

Edited September 18, 2014 by RTFC

[RTFC 353]

Well, the future is now, so all Example code snippets have been added to the Help file, in the latest version: 1.3.

Still very much struggling with html-encoding, as you can see from the ridiculous text colours (it looks fine until I generate the final output ) and countless extra links that I never consciously inserted.   Oh well, it's better than nothing, I suppose. I'll be taking a little break from this Hellp file now...

Some handy new CreateMatrix functions with predefined contents have been added to the mix, plus various minor edits and improvements in code and Help file, and a nasty type-conversion bug was "retired" in the dll (Eigen version 3.2.2). Full details in the Changelog in the Help file.

 

Edit: two link-pruning sessions later (no, I couldn't let it lie), the large majority of stray links have now been expunged from the Help file. The ugly colours we'll have to live with for now. The package has been updated under the same version number 1.3.

Edited September 22, 2014 by RTFC

[RTFC 353]

@ValeryVal: Wow,   are you sure you want to do this? I mean, it would be fantastic,   but it's a huge job. I'm very impressed that you'd even consider such an undertaking. One thing though, this afternoon I uploaded a cleaned-up version 1.3, from which all stray hyperlinks in the Example code snippets have been removed (and there were many, many hundreds of them).   You should preferably use this latest version (uploaded today) as your template, because many of those links I removed directed the user to the wrong destination pages.  There were also a few broken internet links I fixed.

The Help file won't undergo such large revisions in future (I hope), only individual functions being added, and maybe another Tutorial or so. First, I'll be focussing on other new parts of this environment...

Again, thanks for the vote of confidence, ValeryVal.   Perhaps other Russian-speaking forum members can be found to lend you a hand?

[ValeryVal 8]

>but it's a huge job

AutoIt Help (Russian edition) was also a huge job (since 1998, thanx Jon!), I think.

>The Help file won't undergo such large revisions in future (I hope)

It will be translated slowly and thoughtfully, I repeat

>Perhaps other Russian-speaking forum members can be found to lend you a hand?

We have here In Russia a motto: John Smith makes his work alone...

Thank you for your hard work and kindly words

:-)

[RTFC 353]

:cheer:

Celebrating 100 downloads since moving v1.2 to section Misc, and releasing version 1.4, a major upgrade.

Two new Tutorials, new advanced statistics, a significant number of new matrix management functions for cutting and pasting, more control over error handling, a complete overhaul of the alias wrappers (which now parse any error back up the calling chain), many new Help file topics, and the MatrixFileConverter utility can now handle free-format space-delimited ASCII, and add new Excel worksheets to store new data. More details in the Changelog in the Help file.

All this adds up to another megabyte of content (total package (uncompressed) ca. 7.5 MB now), and over 260 non-internal _Eigen_ functions.

This release was mainly driven by my first attempt to use this environment for actual work, which brought to light some glaring omissions in basic matrix management (mostly re. creating new matrices from data subsets in other matrices). If you identify similar omissions in basic functionality, please let me know; I may be able to implement a fix fairly quickly.

 

@ValeryVal:great work on the Help file translation into Russian!

Edited October 8, 2014 by RTFC