Jump to content

Function inclusion in already-compiled script


Recommended Posts

Hello fellow AutoIt-ers,

I'm building a utility to help automate certain tasks, and my approach is to make it as modular as possible. The utility is essentially a framework upon which various task-specific modules can be stacked. I'm defining each module's attributes and actions in its own XML file so that I can make updates to each module independently of one another, as well as update them or create new ones without having to recompile the core application. Which brings me to my question:

Is it possible to define functions in an external file that gets read into memory upon launch of my compiled AutoIt application? In other words, what I'm looking to do is to get the same behavior as using an #include with an .au3 file, but with the external AutoIt functions living in XML instead of au3, and being included by an AutoIt script that's already compiled into an executable. Is this at all possible?

I'd even settle for having the module "import" process include a step to parse the XML file and dump all AutoIt-function-specific elements into a temporary .au3 file, if that would be the difference between this being possible or not.

Thank you in advance for any potential guidance. :-)

An emoticon is worth a dozen words.

Link to comment
Share on other sites

I think I figured it out - the Execute() function! So, I'll have my module functions defined as simple string values within my XML file(s), to then be executed by Execute(). This should not only get me what I'm looking for, but since the Execute argument is plain text that isn't evaluated until the Execute function call is instantiated within the script it also won't raise any "function not defined" flags whenever I compile my master script. :D

I'll test the theory and report back the results.

An emoticon is worth a dozen words.

Link to comment
Share on other sites

@Jeemo,

- assuming Execute() does work on sections of code and not only on a single expression -

be aware that by taking that approach, you are deliberately exposing your script to the "arbitrary code execution" security vulnerability exploit. since your code is blindly executing whatever instructions submitted to it by an external plain text file, anyone with sufficient permissions (and with either malicious intent, or the best of intentions but lack of programming skills) is able to modify your code and replace your functions with their own.

now, you may not think much of security, but this is also an issue of robustness: even if they don't mean any harm, they can easily break your application, and you will be held responsible.

one workaround would be to sign or encrypt your external functions before distribution, and have the master script validate/decrypt them at run time. this will eliminate the breakdown risk, but not the security vulnerability itself.

other than that, probably because i don't know what your script is doing and in what kind of environment, i fail to see the benefit of such an approach. are you expecting sooo much of a diversity in features, to make this worthwhile? how "general-purpose" do you want your script to be?

Signature - my forum contributions:

Spoiler

UDF:

LFN - support for long file names (over 260 characters)

InputImpose - impose valid characters in an input control

TimeConvert - convert UTC to/from local time and/or reformat the string representation

AMF - accept multiple files from Windows Explorer context menu

DateDuration -  literal description of the difference between given dates

Apps:

Touch - set the "modified" timestamp of a file to current time

Show For Files - tray menu to show/hide files extensions, hidden & system files, and selection checkboxes

SPDiff - Single-Pane Text Diff

 

Link to comment
Share on other sites

@orbs, thank you for your reply, you bring up some great points. Security is absolutely a concern, so I was already planning to encrypt the parts of the XML file that would expose functions or code execution, and decrypt them upon import (I omitted that from the original post to keep from confusing things). While I suspect that it might be possible to crack open the master script executable to find its internal unencryption key, if a malicious actor can do that then they would pretty much "own" the master script even if it didn't have any external file dependencies, no?

To your point about Execute() maybe not working on sections of code, I'm looking into a worst-case-scenario of getting multi-line functions into a single line, with some sort of new line delimiter. Not sure about the possibility of that idea, though.

Lastly, I have a couple reasons for my approach:

The first is that the script is a multi-purpose UI. I already have about 5 different modules in mind, and I expect that number to grow. More importantly, I expect there to be somewhat frequent changes and feature enhancements within each module, such that I'd like each one to have its own version. And with each change, I'd prefer to have the option to send the users of the utility a new version of the module as it becomes available, rather than bump the overall application version plus have users do a full rip/reinstall just to affect a potentially minuscule change to a single module. And if someone requests a one-off customization of one of the modules then I'd like to be able to provide that without having to maintain a different version of the entire script for each such custom request to a module.

The other reason is that I expect many (if not all) of the modules will ultimately be migrated to a vehicle other than my my AutoIt script (no offense to AutoIt, but cross-platform usage will eventually be a requirement). I figured that defining the modules in XML would make that migration process easier.

With all that being said, I admittedly have a tendency to over-engineer things. I'm not 100% married to the idea of my approach - I just started mucking around with XML in AutoIt a couple days, and I have to say it's been a pretty big headache so far. If you or anyone else think that the approach will be more effort than it's worth, or if you have any simpler alternatives to get similar flexibility to what I'm looking for then I would eagerly listen. :)

Edited by Jeemo

An emoticon is worth a dozen words.

Link to comment
Share on other sites

@orbs, you were right about Execute() being limited to single-line commands. I have a workaround that should work: upon launching the master script, parse the full-blown functions from the XML file, decrypt them and spit them out to a temp .au3 file, compile to an executable belonging to that module and immediately delete the temp .au3 file. Calls to the functions within the module-specific executables could then employ Execute(). Then destroy the resultant executables upon exiting the script.

After reading what I just wrote, I'm leaning more towards "it's too complex, and too vulnerable" (albeit possible). I'd definitely welcome any suggestions towards getting anything close to the modularity I'm seeking with less complexity.

An emoticon is worth a dozen words.

Link to comment
Share on other sites

@Jeemo,

if there is too much diversity between modules (i.e. not enough common features to justify using only master functions), then consider this approach:

each module should be an individual executable.

first, this answers your development requirements. it also makes your master script job of validation the integrity of the module much easier.

second, you can build your modules with a common framework of communication with the master script, thus make your modules dependent of the master (i.e. a module executable won't work unless called by the master, and it accepts its input from the master, and reports its output to the master).

if a module is configurable by the user, then each module exe can be accompanied by a config file (ini or xml, or whatever). you can hardcode some defaults in the exe (so you don't need to ship a default config file with any new version of a module, just the exe, and you don't override user configuration).

as for security: be aware that compiled AutoIt scripts are not secure. there are discussions aplenty on this subject in the forum. what you can do is, for example, sign your modules (and master), so if anyone tampers with your app, they can't use your signature. you should, of course, alert the users in advance that they should trust only signed modules.

if you dislike the cost of - or the mafia concept behind - the signing solution, then another alternative is to distribute your modules along with their checksum, so users can validate them.

anyway, the responsibility for using genuine master and modules are of the users, and that should be well presented to them. you cannot trust your master to validate the modules programatically, because the validation process itself can be tampered by an attacker.

now, this is all quite theoretical. if you don't mind describing your project in more concrete details, you can get more concrete ideas.

Signature - my forum contributions:

Spoiler

UDF:

LFN - support for long file names (over 260 characters)

InputImpose - impose valid characters in an input control

TimeConvert - convert UTC to/from local time and/or reformat the string representation

AMF - accept multiple files from Windows Explorer context menu

DateDuration -  literal description of the difference between given dates

Apps:

Touch - set the "modified" timestamp of a file to current time

Show For Files - tray menu to show/hide files extensions, hidden & system files, and selection checkboxes

SPDiff - Single-Pane Text Diff

 

Link to comment
Share on other sites

20 hours ago, Jeemo said:

cross-platform usage will eventually be a requirement

so AutoIt will not suit you, as it is Windows-only. admittedly, it can function to some extent under Wine in Linux & Mac, but no-one in their right mind won't recommend using this in production.

if you anticipate cross-platform compatibility sooner than later, then start your entire project in another language. if you do, security also becomes easier in many aspects.

Signature - my forum contributions:

Spoiler

UDF:

LFN - support for long file names (over 260 characters)

InputImpose - impose valid characters in an input control

TimeConvert - convert UTC to/from local time and/or reformat the string representation

AMF - accept multiple files from Windows Explorer context menu

DateDuration -  literal description of the difference between given dates

Apps:

Touch - set the "modified" timestamp of a file to current time

Show For Files - tray menu to show/hide files extensions, hidden & system files, and selection checkboxes

SPDiff - Single-Pane Text Diff

 

Link to comment
Share on other sites

On 7/20/2016 at 9:34 AM, orbs said:

@Jeemo,

if there is too much diversity between modules (i.e. not enough common features to justify using only master functions), then consider this approach:

each module should be an individual executable.

first, this answers your development requirements. it also makes your master script job of validation the integrity of the module much easier.

second, you can build your modules with a common framework of communication with the master script, thus make your modules dependent of the master (i.e. a module executable won't work unless called by the master, and it accepts its input from the master, and reports its output to the master).

if a module is configurable by the user, then each module exe can be accompanied by a config file (ini or xml, or whatever). you can hardcode some defaults in the exe (so you don't need to ship a default config file with any new version of a module, just the exe, and you don't override user configuration).

as for security: be aware that compiled AutoIt scripts are not secure. there are discussions aplenty on this subject in the forum. what you can do is, for example, sign your modules (and master), so if anyone tampers with your app, they can't use your signature. you should, of course, alert the users in advance that they should trust only signed modules.

if you dislike the cost of - or the mafia concept behind - the signing solution, then another alternative is to distribute your modules along with their checksum, so users can validate them.

anyway, the responsibility for using genuine master and modules are of the users, and that should be well presented to them. you cannot trust your master to validate the modules programatically, because the validation process itself can be tampered by an attacker.

now, this is all quite theoretical. if you don't mind describing your project in more concrete details, you can get more concrete ideas.

Thanks again for your detailed response, Orbs. Given everything you've mentioned, I've decided against the approach of defining modules in XML (or any other plain text format). I may end up breaking them up into separate executables, but I'm even on the fence about that. Yes, it would be nice for all modules to have their own versions, but the engineering that would be required to get that all to work might not justify the benefits.

An emoticon is worth a dozen words.

Link to comment
Share on other sites

On 7/20/2016 at 9:41 AM, orbs said:

so AutoIt will not suit you, as it is Windows-only. admittedly, it can function to some extent under Wine in Linux & Mac, but no-one in their right mind won't recommend using this in production.

if you anticipate cross-platform compatibility sooner than later, then start your entire project in another language. if you do, security also becomes easier in many aspects.

I'm almost 3,000 lines into this project in AutoIt, so there's no turning back now. Time is of the essence, so I couldn't afford the time it would take to conquer the learning curve of a different language that would do the same thing as AutoIt. The platform-independence thing would be down the road, likely a year or more. I'm fine with some (or all) of what I'm doing in AutoIt being tossed out at that time, if it's been able to provide value in the meantime.

An emoticon is worth a dozen words.

Link to comment
Share on other sites

  • 2 years later...
On 7/19/2016 at 8:18 PM, Jeemo said:

@orbs, you were right about Execute() being limited to single-line commands.

While I'm almost certain you're no longer looking for a solution, I wanted to add for posterity the Execute() CAN in fact run multi-line commands! It simply requires replacing carriage-return line-feeds with the autoit syntax for such. A simple stringreplace($string,@CRLF,' @CRLF ') applied to an .au3 makes it executable using execute().

Here is a proof of concept converter that converts an .au3 to something that can be run using an execute()

Local $filepath = FileOpenDialog("Open File",@ScriptDir,"AutoIt Source (*.au3)|Plain Text (*.txt)|All Files (*.*)")
Local $filehand = FileOpen($filepath)
Local $filedata = FileRead($filehand)
MsgBox(0,"Init",$filedata)
Local $savedata = StringReplace($filedata,@CRLF,' @CRLF ')
MsgBox(0,"Done",$savedata)
Local $savepath = FileSaveDialog("Save File",@ScriptDir,"Plain Text (*.txt)|All Files (*.*)")
Local $savehand = FileOpen($savepath,2)
FileWrite($savehand,$savedata)

 

Edited by Funtime60
Adding Promised POC
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...