Alexi 1.0

[WaitingForZion]

The AutoItObject team has been brilliantly made Object Oriented Programming available in AutoIt. Not too long after they were working on this, I tried to implement a lexical analyzer in AutoIt, but the code became unmanageable due to lack of decomposition and other reasons, so I gave up on it and just release the broken the code. However, with the addition of Object Oriented Programming in AutoIt, I was able to implement my library successfully. This new Object Oriented Library for tokenizing strings is called Alexi. Alexi does away with the need to implement the first step in processing structured language. Alexi itself lets you define token types and split up a string into tokens of those types. Since this is only the first version, there are certain limitations, so not every structure can be tokenized, but for me at least, this is a big step headed in that direction.

The library along with documentation and an example is in the zip file attached to this post.

Alexi.zip

Feedback is appreciated, and I implore you kindly to provide it.

Update: Scroll down to other post.

Edited February 28, 2010 by WaitingForZion

[trancexx]

Great!

I like it.

[Yoriz]

It's totaly over my head as to what this actually does , but ive been looking forward to seeing something done with AutoItObject to study

[monoceres]

Nice to see something already written in AutoItObject

Now go write that parser! I've heard Recursive Descent is a good approach.

If you wish to keep your code even more object oriented, then try replacing your arrays with a LinkedList object (available in autoit object dowload)

[WaitingForZion]

This past week I was not in a place where I could work on my lexer, but for the first few days I was filled with ideas on how to improve it. Sad to say, I have only made one minor update to the library. In the earlier version, the _StringTokenize function would simply ignore all occurrences of characters not associated with a token definition. I understand that in many cases it may not be desirable to allow these occurrences. To remedy this problem, I have merely added a parameter and a few more lines of code allowing you to choose whether you want the function to terminate with an error when this happens.

It should be pretty easy to break up all kinds of language structures. Limitations still remain, but they can be worked around.

The only severe limitation I can think of, besides the speed limitation that come with being implemented in a scripting language, is the inability to recognize tokens made of characters whose types vary according to order, and tokens made of differing literal characters. There is a simple remedy for the latter. Simply included all possible characters in the token definition, and then after they have been recognize, process them further by filtering.

So instead of defining a separate token type for each operator when you have operators represented by more than one character, simply define one token type for all operator characters. After you have your list of tokens, merely filter those tokens identified as the given token type.

I'm really sorry for this inconvenience. To make the necessary improvements, I will have to make another implementation taking these things into mind.

Update: Alexi 1.0.1.zip

Now, does anybody have any ideas or suggestions? Or does anyone have feedback?

[Raik]

line 104 in Alexi.au3 throws an error because line 102 is missing "... UBound($aTokenDefs)-1

because of missing comments i cant debug further.

try to tokenize a complete autoit-script, not just one line of pseudo-code.

maybe then you will find the bugs.