AutoIt Language Tools

[Mat]

This is a release of something I wrote as a proof of concept a couple of years ago. I've now put it on github and made a few minor changes to it, mostly so that it now reports errors properly.

It's a lexer and parser for the AutoIt language. The idea is that tools could be written in AutoIt and maintained by the community. In practice, it's probably slow enough on large scripts that it could get annoying. Still, could be of a lot of use to the various projects to work on the language and extend it. 

Usage is pretty simple, _Ault_ParseFile parses a file and returns the syntax tree, or an array describing the error if @error flag is set. 

https://github.com/MattDiesel/Ault

Matt

[JohnOne]

Not sure what I would do with this, so just ran test.au3.

Here is the console output, I'm guessing it's not what it's supposed to be.

expandcollapse popup

>Running AU3Check (3.3.14.2)  from:C:\Program Files (x86)\AutoIt3  input:C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\Test.au3
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(30,35) : warning: $_AP_STI_COUNT: possibly used before declaration.
    Local $aSt[100][$_AP_STI_COUNT]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(55,51) : warning: $AP_BR_FILE: possibly used before declaration.
        $iSt = __AuAST_AddBranch($aSt, $AP_BR_FILE,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(65,43) : warning: $AP_STI_LEFT: possibly used before declaration.
        If $i Then $aSt[$iSt][$AP_STI_LEFT]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(130,59) : warning: $AP_BR_GROUP: possibly used before declaration.
            $iStRet = __AuAST_AddBranch($aSt, $AP_BR_GROUP,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(151,56) : warning: $AP_BR_OP: possibly used before declaration.
            $iStRet = __AuAST_AddBranch($aSt, $AP_BR_OP,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(208,40) : warning: $AP_STI_RIGHT: possibly used before declaration.
            $aSt[$iStRet][$AP_STI_RIGHT]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(249,32) : warning: $AP_VARF_LOCAL: possibly used before declaration.
            $i = $AP_VARF_LOCAL
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(251,33) : warning: $AP_VARF_Global: possibly used before declaration.
            $i = $AP_VARF_Global
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(253,30) : warning: $AP_VARF_DIM: possibly used before declaration.
            $i = $AP_VARF_DIM
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(255,32) : warning: $AP_VARF_CONST: possibly used before declaration.
            $i = $AP_VARF_CONST
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(259,32) : warning: $AP_VARF_BYREF: possibly used before declaration.
            $i = $AP_VARF_BYREF
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(387,49) : warning: $AP_STI_BRTYPE: possibly used before declaration.
                    $aSt[$iStRet][$AP_STI_BRTYPE]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(387,64) : warning: $AP_BR_STMT: possibly used before declaration.
                    $aSt[$iStRet][$AP_STI_BRTYPE] = $AP_BR_STMT
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(412,70) : warning: $AP_BR_LOOKUP: possibly used before declaration.
                        If $aSt[$i][$AP_STI_BRTYPE] <> $AP_BR_LOOKUP Then
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(450,66) : warning: $AP_BR_DECL: possibly used before declaration.
                    $iStRet = __AuAST_AddBranch($aSt, $AP_BR_DECL,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(455,68) : warning: $AP_STI_VALUE: possibly used before declaration.
                            Case BitAND($aSt[$iStRet][$AP_STI_VALUE]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(455,84) : warning: $AP_VARF_ENUM: possibly used before declaration.
                            Case BitAND($aSt[$iStRet][$AP_STI_VALUE], $AP_VARF_ENUM)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(466,119) : warning: $AP_VARF_STATIC: possibly used before declaration.
                                    $aSt[$iStRet][$AP_STI_VALUE] = BitOR($aSt[$iStRet][$AP_STI_VALUE], $AP_VARF_STATIC)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(519,90) : warning: $AP_BR_ENUMDEF: possibly used before declaration.
                        $aSt[$iStRet][$AP_STI_BRTYPE] = $AP_BR_ENUMDEF ; Correct the type
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(558,54) : warning: $AP_BR_IF: possibly used before declaration.
    Local $iStRet = __AuAST_AddBranch($aSt, $AP_BR_IF,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(648,57) : warning: $AP_BR_WHILE: possibly used before declaration.
    Local $iStRet = __AuAST_AddBranch($aSt, $AP_BR_WHILE,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(676,55) : warning: $AP_BR_FOR: possibly used before declaration.
    Local $iStRet = __AuAST_AddBranch($aSt, $AP_BR_FOR,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(691,53) : warning: $AP_BR_FORIN: possibly used before declaration.
        $aSt[$iStRet][$AP_STI_BRTYPE] = $AP_BR_FORIN
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(778,54) : warning: $AP_BR_DO: possibly used before declaration.
    Local $iStRet = __AuAST_AddBranch($aSt, $AP_BR_DO,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(808,58) : warning: $AP_BR_SELECT: possibly used before declaration.
    Local $iStRet = __AuAST_AddBranch($aSt, $AP_BR_SELECT,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(827,57) : warning: $AP_BR_CASE: possibly used before declaration.
            $iCase = __AuAST_AddBranch($aSt, $AP_BR_CASE,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(870,58) : warning: $AP_BR_SWITCH: possibly used before declaration.
    Local $iStRet = __AuAST_AddBranch($aSt, $AP_BR_SWITCH,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(945,60) : warning: $AP_BR_FUNCCALL: possibly used before declaration.
    Local $iStRet = __AuAST_AddBranch($aSt, $AP_BR_FUNCCALL,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(1013,53) : warning: $AP_BR_FUNCDEF: possibly used before declaration.
    $iStRet = __AuAST_AddBranch($aSt, $AP_BR_FUNCDEF,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(1067,76) : error: __AuParse_ParseEnumDecls() called by a previous line with 5 arg(s). Max = 4. First previous line calling this Func is 520.
Func __AuParse_ParseEnumDecls(ByRef $lexer, ByRef $aSt, ByRef $tk, $iStRet)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
"C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\ault\Parser.au3"(1214,57) : warning: $AP_BR_ARRAY: possibly used before declaration.
    Local $iStRet = __AuAST_AddBranch($aSt, $AP_BR_ARRAY,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\Test.au3 - 1 error(s), 30 warning(s)
!>22:05:37 AU3Check ended. Press F4 to jump to next error.rc:2
+>22:05:37 AutoIt3Wrapper Finished.
>Exit code: 2    Time: 0.8678

 

[Mat]

Yeah, I'm not using Au3Check at the moment. I'll get AutoIt installed properly and see what Au3Check is showing. For now just not running Au3Check should let it run. 

Edit: It seems that just switching the order of the includes in test.au3 fixes all the warnings.

Edited January 10, 2016 by Mat

[mLipok]

Change in AST.au3

;~ #include "Parser.au3"

but then in Parser.au3  when comes to this line:

$iStRet = __AuParse_ParseEnumDecls($lexer, $aSt, $tk, $i, $iStRet)

 

Fires such errors:

>"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "Z:\TOOLs\Macro\FORUM\___FORUM -- z forum\_AutoIt_Tools\Ault-master\Test.au3" /UserParams    
+>23:23:51 Starting AutoIt3Wrapper v.15.920.938.4 SciTE v.3.6.2.0   Keyboard:00000415  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64  Environment(Language:0415)  CodePage:65001  utf8.auto.check:4    # detect ascii high characters and if none found set default encoding to UTF8 and do not add BOM
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\user\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\user\AppData\Local\AutoIt v3\SciTE 
>Running AU3Check (3.3.15.1)  from:C:\Program Files (x86)\AutoIt3  input:Z:\TOOLs\Macro\FORUM\___FORUM -- z forum\_AutoIt_Tools\Ault-master\Test.au3
"Z:\TOOLs\Macro\FORUM\___FORUM -- z forum\_AutoIt_Tools\Ault-master\ault\Parser.au3"(1067,76) : error: __AuParse_ParseEnumDecls() called by a previous line with 5 arg(s). Max = 4. First previous line calling this Func is 520.
Func __AuParse_ParseEnumDecls(ByRef $lexer, ByRef $aSt, ByRef $tk, $iStRet)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
Z:\TOOLs\Macro\FORUM\___FORUM -- z forum\_AutoIt_Tools\Ault-master\Test.au3 - 1 error(s), 0 warning(s)
!>23:23:51 AU3Check ended. Press F4 to jump to next error.rc:2
+>23:23:51 AutoIt3Wrapper Finished.
>Exit code: 2    Time: 0.7442

So i change this line to:

$iStRet = __AuParse_ParseEnumDecls($lexer, $aSt, $tk, $iStRet)

Now scirpt is working but I do not know if this is proper results.

 

Edited January 10, 2016 by mLipok

[JohnOne]

Yes,  disabling Au3Check done the job.

Interesting.

ran on win 10

[JohnOne]

Only console output I get now it's working is...

>Running:(3.3.14.2):C:\Program Files (x86)\AutoIt3\autoit3.exe "C:\Users\User Name\Documents\PC\AU3CODE\Ault-master\Test.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
+>22:55:48 AutoIt3.exe ended.rc:0
+>22:55:48 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 65.32

 

[mLipok]

run test.au3

Edit:... time to sleep .

Edited January 11, 2016 by mLipok

[Mat]

Sorry, didn't really have time to document what it does.

Given an input file (ExampleScript.au3) of this:

Local $a = 1, $b = 2
Local $foo = _MyFunction($a, $b)

ConsoleWrite($foo & @LF)

It will generate an array that looks like this:

This is the syntax tree for the script, using an odd flat tree format documented here: https://github.com/MattDiesel/Ault/wiki/AST-Format

The "tree" starts at row 1, with a file. For a file branch, the LEFT value is a comma separated list of child branches, in this case "2,5,8,15" which correspond to rows in the array (these are: the definition of $a, the definition of $b, the definition of $foo and the ConsoleWrite call). If we look at row 15 for example, this is the AST for the function call "ConsoleWrite($foo & @LF)":

From this starting point, it would be possible to implement something like Tidy (which is what the messagebox shows afterwards, though a rather naive and not too useful Tidy), or other tools, including executing the code. 

Edited January 11, 2016 by Mat

[guinness]

Are you accepting PRs? I could tidy the files for you and fix a couple of bugs in test.au3.

[Mat]

Of course! That's why it's on github (as well as being incredibly easy to manage the code). "Sociable Coding"  

Let me know if there is anything else I need to explain. I probably won't be able to actively work on anything now term has started, but I'm not going to drop off the internet entirely for another couple of months yet.

Mat

[czardas]

From the examples displayed, this looks like an interesting project.  If you really want to get your head around how code works, this ought to be a great learning tool. Anything more it has to offer is surely a bonus, I put my trust in you, because it makes no sense to invest so much time and effort without reason. BTW, did you invent this Left / Right terminology layout system?

Edit: It's the first time I've come across this, so I'm a bit ignorant on the subject.

Edited January 13, 2016 by czardas

[Mat]

From the examples displayed, this looks like an interesting project.  If you really want to get your head around how code works, this ought to be a great learning tool. Anything more it has to offer is surely a bonus, I put my trust in you, because it makes no sense to invest so much time and effort without reason. BTW, did you invent this Left / Right terminology layout system?

Edit: It's the first time I've come across this, so I'm a bit ignorant on the subject.

An Abstract Syntax Tree (so the left/right tree structure) is how it's always done (well, except LISP maybe, but even that results in a tree like structure). The problem I was trying to solve was that AutoIt is not really that good at storing a tree structure. Nested arrays aren't efficient, and we don't have structures/classes and pointers. So the only new thing, is trying to use one array to describe a tree.

[junkew]

nice. Maybe an alternative to use ANTLR to specify the interpreter http://www.antlr.org/

That would make life easier to port to Java (and as such part of the language can run on other systems)