DLL Support idea

[Valik]

Alright, I mentioned in another thread that one method of achieving DLL support (Not calling functions from a DLL, but adding functionality to AutoIt via DLL) was to create a hook API in AutoIt and expose that for DLL developers. The main reason for that suggestion was so it would be possible to initialize stuff or set whatever flags needed to be set before the function is called or whatever else needs done at various points outside the scope of the function. Thinking along the lines of how to get that sort of support without the hooks, I finally realized Jon had done a lot of the work already with his design of AutoIt 3.

Jon has alredy set AU3 up with a class structure, which makes things much easier. By building ontop of the AutoIt_Script class (Or more specifically building under it...), DLL support is very possible and shouldn't be a massive rewrite (At least I hope not).

Here's the idea. Make a base class which contains very few critical members and functions. It could maybe have Execute() and the arrays detailing what functions are available. From there, derive a new class (AutoIt_Script currently) which has the implementation of the functions. This would allow the source to be more broken up and allow the AutoIt_Script class to go away and be replaced by several smaller derived classes which house specific functionality (i.e. Window management, string management, et cetera).

By having this base class which takes care of some of the ugly stuff, a DLL author could dervice a class from the base class, write his functionality into the new class, then a simple '#DLLinclude "my.dll"' (Or something similar) would be all it takes to expose the script and functions to one another.

Advantages:

- DLL support

- The DLL will use the same methods as AU3 does internally

- Functions are overidable through the DLL (See below)

- Source code can be better organized and easier to read (No more giant class with all functions in it)

- Some code rewriting, but so much to be a drastic rewrite

Disadvantages:

- Some code rewriting

- Forces DLL developers to use C++ to write the DLL's (although this is essentially the same dilema they would face if they were to write the code straight into AutoIt...)

Problems:

- User functions

I think the best way to store the list and call it would be through an array of pointer's. This serves two purposes, first, it allows the derived classes to be called in the first place. Secondly, it would allow a last-in-first-used order of calling so that DLL's would be called before internal AU3 functions. By doing this, overriding functions is possible.

The problem I see is user functions. I don't quite know what to do with them just yet. They are part of the AutoIt_Script class but I think the way I'm going to set up derivation and such will require them being moved somewhere.

One of my goals is to hide as much of the implementation as possible. Instead of exposing the giant case structure and all the arrays containing the function details, I want to create one or two simple functions that build all that for the author. This would apply for the source as well. This design could be used for future additions to the code as well as support the current way it's written (To prevent a full rewrite, obviously).

I'm going to work on this project because I just want to see if I can do it (I don't think I have near the programming experience as the rest of you who've worked on AU3). We'll see how much success I have. I want this functionality for myself as much as anything, but I would like to see Jon integrate it into the "official" release if I get it working right. Larry's already stated on the Yahoo group one advantage to DLL support (Writing libraries that will work with more than just Microsoft's controls). I may or may not post updates along the way but I may have a question or two on how AutoIt works internally. I don't know how long this will take, either, maybe a couple days, maybe a couple weeks, maybe a month, depends on how much time I spend and whether I can translate the idea to code or not.

Anyway, that's the plan. I may or may not of explained myself very well, so feel free to ask if you didn't get what I said (provided you actually care what I said ).

[Jon]

I had a few ideas on the subject but most of them only worked with C++ and I know a lot of delphi users were interesting in adding functions. I had an idea for that too but I forgot what it was (being busy and all that).

I did notice that the NSIS installer I use has a plugin feature where you just include a dll in your script and the functions become available - I was going to have a look at the source for ideas.

[Valik]

It looks like NSIS opens the DLL and reads the list of exported functions and adds them to a list which it can later access. I don't know how well it would mesh with AutoIt because of how AU3 handles function lookups. It might work, although I think it might be a bit limiting.

Eww, I saw a goto in there, strangely, a while loop using continue and/or break would of worked just as well minus the inherent danger of goto...

[Valik]

Okay, I've revised the initial idea slightly. It's still the same class structure I outlined above except now I'm thinking it might be a better idea to do what NSIS does and read the export table to find the functions. Then the class can generate it's own method's for looking up and calling functions. This should remove any language constraints as any language that will allow you to export a function in a DLL should work, right? Additionally, I think two special purpose functions Init() and Cleanup() should be optional functions that are called during the Constructor (Init()) and Destructor (Cleanup()) that take care of anything the DLL may need to do.

Does this sound like a more useable method to you?

[Valik]

I did some more looking around at the NSIS source and source for the plugins. They are all written in C\C++ and all the exported functions seem to share the same prototype. They include a header (NSIS\Contrib\ExDLL\exdll.h) which defines a few common elements such as a stack and ways to pop it and the like.

In essence, I was going to do this, too. Except I was forcing C++ instead of having the option to use C. The prototype I was thinking of which seems most common is 'AUT_RESULT func(VectorVariant &vParams, uint iNumParams, Variant &vResult)'.

There's still a problem with using pure C, though. Even by doing away with the class inheritance stuff, the prototype is still using C++ elements in the VectorVariant and Variant classes. There seems only two options available, one is to to create some structures in C which can serve the purpose of the two mentioned classes (Not fun, I wouldn't think); option two is to just do it in C++.

This doesn't necessarily doom C programmer's from making DLL's. A good template file (which I was going to make anyway simply for testing and demonstration purposes) should convey the usage enough that they could copy that style and make stuff work.

I admit that I'm unfamiliar with delphi (or any other language, C++ is the only language I've tried to learn (yes, I tried to learn C++, not C, then C++, but C++ from the beginning)). However, I don't think a common interface is currently possible. The plugin systems in the two highly customizable systems I've seen both forced language constraints on the developer. NSIS allows the option of C or C++, the other was an IRC chat server which forced C. I don't think AutoIt can be much different (It's not COM after all...). I think the best we can hope for is to make a good, clean interface for external DLL's and try to make it as easy as possible to write DLL's and easy to use them.

For now, I'm going to continue on with my ideas of class inheritance and see where that leads. As I stated earlier, I wouldn't mind this being in the "official" AutoIt, but at the very least, I want it in some form just to see how it works out. Currently I'm studying the code trying to see how it all works with emphasis on the lexer and parser because both of those will need modifications so they can look in more than one place for functions. I have ideas for how to do most all of this though, so it's just finding the time and patience to tear it all apart without breaking it all.

[jpm]

Valik,

Will our thought support the hook for calling internal debug code?

For implementing the debug we need to add hook in the actual code.

[Valik]

Valik,

Will our thought support the hook for calling internal debug code?

For implementing the debug we need to add hook in the actual code. 

I remembered the debug stuff last night, but I'm not sure about it yet. I don't know how it's set up because I haven't seen the debugging code. To be perfectly honest, I had forgotten about it altogether until after I made my last post. Just have to see how it all goes and see what the debug code looks like.

[jpm]

THe hook for calling the debug code is something like

If (g_bDebugEnabled)

g_oDebug.Debug_DocorrespondingAction(parameter);

Mainly all hooks refer to a class. Just for performance reason few have a wrapper in script object before using the class.

[Valik]

By taking the route of writing the "simplest thing that could possibly work" I'm now able to call a script function which is located in an external DLL . The DLL is hard-coded in, though.

I've just done enough to get DLL's working, I haven't made much progress on restructuring the script engine yet. What I have working now is a hybrid of Jon's original design and my re-structuring. The design I have in mind will be more modular, though, allowing for much easier maintenance (at least it seems that way to me).

I'm trying to seperate the program into logically seperate parts with the "final" design having each part seperate enough that a brand new part could be written and dropped into place without breaking any other part (For example, a free-form lexer and parser could be dropped in in place of the current pair without requiring a change to any other part of the program). At any rate, I think it'll be interesting to see the results.

[redndahead]

Any update on this?

red

[Valik]

Any update on this?

red

Yeah, I'm sick. Heh. I've been sick for about 2 weeks and haven't worked on it very much. I need to add some documentation explaining how to use the framework to write a DLL and make the path not hard-coded in. But I have 95% of the framework done. It's a C++ framework, but it shouldn't be too hard to make it work with C (Read: Hide the class part and let the compiler make the class instead of the user). This would of course still require a C++ compiler, but that's pretty much a requirement anyway because of the variant classes.

Jon, I worked with the goal of NOT modifying the current AutoIt as much as possible. Off the top of my head, the only changes I made to the pre-existing code were to add 3 functions to AutoIt_Script and couple changes to the lexer and parser so they can search DLL's. The DLL's path is hard-coded in, that's the only other thing that needs changed. When I get feeling well enough to write the documentation and make sure there aren't any bugs, I'll give you a demonstration so you can see what you think.

Random technical notes about it:

I wrote a virtual base class (Probably will be a pure virtual base class, though) with a few common functions (Find, GetMinParams, GetMaxParams, FunctionExecute) which all derived classes are expected to have. AutoIt_Script is derived from that base class with the missing functions added. DLL's are also derived from this base class, but I made those 4 virtual functions automatically generated by the compiler. It uses pointer to member functions instead of a case structure for calling functions. Obviously, this method can be skipped and a custom set of functions can be written. Creating a DLL is as easy writing your functions then calling a special RegisterFunction function which takes all the info needed to expose that functionality to scripts.

Pointers to these objects are stored in a list, the lexer searches this list through a function instead of the old method. By the time the parser comes up, it knows which object has the function so it requests the min/max parameters. If everything checks out, then FunctionExecute is called and the correct function is called automagically.

[Jon]

I was considering splitting the lexer into a seperate class to make it easier to move to Aut2Exe when changes are made. This might break stuff you are doing atm so I'll leave it for now.

The even more tricky bit is getting any of this to work with Aut2Exe, although I can think of a way based on your description

[Valik]

I was considering splitting the lexer into a seperate class to make it easier to move to Aut2Exe when changes are made.  This might break stuff you are doing atm so I'll leave it for now.

The even more tricky bit is getting any of this to work with Aut2Exe, although I can think of a way based on your description

Mmm, seperated lexer. Moving it won't break anything. My ScriptList class has a global object which I call things through. I tried to make the design modular so that the lexer and stuff could be moved into seperate classes. I also tried to make it versatile enough that each "family" of related functions could be moved out to DLL's if desired (I can argue that this will reduce the size of compiled scripts, but the amount of work involved would suck).

The tricky part (and the reason I avoided it) of getting it to work with Aut2Exe is telling a compiled script where to find DLL's. I couldn't quite figure out all that was going on in that department, so I just hard-coded the DLL in for testing and focused on getting DLL's to work at all.

[redndahead]

I also tried to make it versatile enough that each "family" of related functions could be moved out to DLL's if desired (I can argue that this will reduce the size of compiled scripts, but the amount of work involved would suck).

Well this is the part I was hoping for. I can't be too pushy because I am certainly not the one to code this. But I like the idea of being able to keep the exe size as small as possible. I know a hundred KB is not that big but as people create dll's it would be nice to still be able to keep the file size down.

Thanks for all the work everyone and I hope you feel better valik. =)

red

Edited January 9, 2004 by redndahead

[Valik]

I've been trying to think of ways to keep size down for the DLL's by minimizing the opportunity for redundant code. One of the things that would seem common would be the utility functions. Because of what they do, it's obvious that the DLL author should have access to them, but to me, it makes no sense for every single DLL and AutoIt itself to have a copy of it. I was thinking of an au3_common.dll which holds some of this common stuff. Then put a cute wrapper around it to take care of LoadLibrary/FreeLibrary/GetProcAddress stuff and we've got something easy to use and it's only in one place instead of being duplicated everywhere.

Another issue I've been trying to figure out how to work around is Execute() can't be called recursively in a DLL because the DLL doesn't know it exists. The idea I had was to make a HandleDelayedFunctions() member in the base class so every object would have an opportunity to use delayed functions if necessary. Then Execute() could call that for all the script objects and if any of them returned true, it would signal Execute() to loop through again until every object returned false.

Any thoughts or better ideas? My two goals are to not breaking anything currently there and make writing a DLL just like adding code directly to AutoIt (i.e. no limitation to what you can do).

[Jon]

It's the sort of thing I'll have to see before having an opinion I still can't quite picture how it works.

Not too bothered about the Util_ functions though, I'd forget about those. They will already be getting a way to plugin functions and they can always lift the ones they want from the source. I certainly wouldn't be expecting access to those functions as a plug in writter (and they are very basic anyway).

[Valik]

It's the sort of thing I'll have to see before having an opinion   I still can't quite picture how it works.

Do you have access to VC 2003 (7.10)? I can send you the work in progress I have based on the 3.0.84 source. It works, but needs some work done to it (Path is hard-coded in for one). If not, I can convert the project files to 7.00.

[Jon]

If all goes well I should have access to 2003 today...

[Valik]

Jon, I just sent off an email to you with the source attached. It was almost 300 kb since it was the whole tree, but that was easier than making you try to WinDiff all the crap in. The project files are for 2003. In a few minutes, I'm going to go make use of AutoIt to whip up a converter to transform 2003 into 2002 versions, so let me know if you can't get access to 2003.

[Valik]

In a few minutes, I'm going to go make use of AutoIt to whip up a converter to transform 2003 into 2002 versions, so let me know if you can't get access to 2003.

Or if you need, you could use this VS 2003 <-> 2002 Converter and that way I don't have to write anything because somebody else already made a program to do it.