CONCEPT: UDF Updater (+ repo) & Decoupling UDF Files from AutoIt Installer

[WildByDesign]

General Concept:

As we all know, UDF functions from %PROGRAMFILES(X86)%\AutoIt3\Include receive bug fixes, improvements and new additions from the community which is both important and incredible. These bug fixes, improvements and additions then get pushed/accepted into the AutoIt SVN server. The problem is that these UDF files only get updated when AutoIt releases a new installer. Also, as we know, there can potentially be months or even years between AutoIt releases. Please note that the duration between AutoIt releases is not the problem at all. I am thankful for any releases.

I would like to present (and discuss as a community) the following ideas:

• Repo with which to store updated UDF files that have been officially accepted (eg. code changes approved)
• Decoupling the UDF files from the AutoIt installer and having its own source/installer
• Updater with which to check for these updated UDF files

Security:

The UDF files which have been checked for code quality and have been approved would absolutely require hash checking and possibly some sort of versioning. The Updater would, of course, need to verify those hashes and discard them if there was a hash mismatch. This is needed to avoid malicious functionality being inserted.

File Locations:

By default, UDF files are stored based on the installer which is generally: %PROGRAMFILES(X86)%\AutoIt3\Include

This could present a problem for a UDF Updater in the sense that it would require being run as Admin. That is certainly possible. However, another option could be to place the UDF files in a user-writable location such as: %LocalAppData%\AutoIt v3\Include

Possibly some sort of symbolic link could be used here.

Script Breaking Changes:

From time to time, there are changes made to some functions which would present script breaking changes. It would be absolutely necessary for such a UDF updater to present a notification showing which function(s) have script breaking changes and therefore giving the scripter/dev the opportunity (dialog box) to continue updating these UDFs or to cancel the update and read more into those script breaking changes.

Discussion:

I have lots of other ideas around this concept. But I wanted to cover the main points first and now, for anyone who has an interest in this, we can discuss this as a community. If such a concept moves forward, we can build this as a community.

I think that for the future of AutoIt this would be an important decision. It might make it easier to simplify the main AutoIt installer. But also, there could be a time in which there are no more official AutoIt releases. I personally would never want to see that because the community behind AutoIt is so strong.

Please feel free to discuss any ideas, potential problems (and workarounds to those problems) and so on.

[argumentum]

33 minutes ago, WildByDesign said:

Security: ..would absolutely require hash checking..

Nah, other than the file corrupted in transit ( the download ) but not because the hosting site would distribute malware.

33 minutes ago, WildByDesign said:

File Locations: .. could present a problem for a UDF Updater in the sense that it would require being run as Admin..

If the user installed as Admin, then only and Admin should be able to update Admin stuff. Not a problem if you can run as Admin.

33 minutes ago, WildByDesign said:

Script Breaking Changes:

hmm. A new beta, or alpha ? ( is not "released" other than a work in progress ) that may be interrelated to other UDFs. It'd have to be a full download of the whole new UDFs as a package. So not a "live" in development beta. But once the current beta is deemed as complete, because the help file will have changes too, then that could be packed and ... good luck ?, because a less savvy user ( yourself not long ago ) could have a hard time switching between "release", "beta" and "this" ( whatever this is )

AutoIt looks for the includes subfolder, and in SciTE one can declare a custom includes path ( all this I'll have to read up on because I'm not too sure ) so, ...

... how do you propose it would all work for the end user in exact concrete terms ? ( not pushing your buttons but clarifying your vision  )

I like the idea. The AutoIt3 version ( the binaries ) has it's ins and outs, and that is that. But the UDFs could have a more fluid update cycle. There are more corrections that can be easily fixed in a UDF than the those of the stub itself.

Then versioning could be with an added letter ?, like 3.3.18.0a ?, or a date, ..or the internal SVN revision I think would be best. 3.3.18.0 (r13212).

Again, I like the idea, but again, present the whole picture so that @Jon can buy it ( you're the sales person  )

[SOLVE-SMART]

Hi @WildByDesign ✌️ ,

my first thought was: @genius257 already tried/did a similar step in such a direction.
I mean he created a package manager au3pm like npm (JS/Node.js) or pip (Python) or NuGet (.NET), which would/could handle such UDF versioned files.
But your approach or idea is not exactly the some, but similar as genius approach or thoughts. At least if I understood it correctly.

There are also some challenges in case we don't only talk about the UDFs of %PROGRAMFILES(X86)%\AutoIt3\Include. Such a package manager would also (in first place) handle custom UDFs (user created UDFs) and therefore we have to respect their licenses etc. The versioning aspect is also not simple, but feasible by SemVer (as example).

I would love to contribute to such a project - but like @argumentum already mentioned, let's talk about the scope and the vision/big picture - so @Jon could spread his thoughts about that. Thank you so much @WildByDesign for bringing up this topic 🤝 .

Best regards
Sven

[argumentum]

26 minutes ago, SOLVE-SMART said:

...Such a package manager would also (in first place) handle custom UDFs (user created UDFs)...

One thing is to install/update AutoIt and it's UDFs, and another outside of my thinking mind, is to use this site and it's product ( AutoIt3 ) to piggyback custom user UDFs, as if this site would vouch for them.

And I read that npm had their issues with stuff.

My idea is in regards to AutoIt's core components and UDFs. No other UDFs should be included. And don't get me wrong, I'd love to include a whole lot of awesome UDFs that are around but, is not the idea. At least that is something that I would not agree to if I was the owner of this site and project.

[SOLVE-SMART]

40 minutes ago, argumentum said:

And I read that npm had their issues with stuff.

Yes, npm is not the best example, because many security issues arise in the last months, but the example is, that npm is a known package manager.

41 minutes ago, argumentum said:

My idea is in regards to AutoIt's core components and UDFs. No other UDFs should be included. And don't get me wrong, I'd love to include a whole lot of awesome UDFs that are around but, is not the idea. At least that is something that I would not agree to if I was the owner of this site and project.

Understandable, yes, thanks. Maybe the ideas could be combined in any way or at least we discussed this "possibility" here. I hope @genius257 will find the time to comment on that approach/suggestion.

It could be exist an official part (UDF of the include path) and a package manager part (free to use, no dependency to the official part), which can complement each other, but don't have to. Why? Because the listed UDFs in the wiki are not versioned per AutoIt version, which a package manager could do.

Best regards
Sven

[genius257]

Hi all

@WildByDesign I like the idea of separating the UDF files to a separate repository.
It does of curse introduce some issues, for example the need to keep documentation for AutoIt version compatibility, but would allow quick releases, without depending on a future AutoIt version bump.
But in regards to the updater, it could evolve into something more complicated, in regards to keeping track of AutoIt vs UDF version comparability. It's not a huge problem, but the additional time needed to make it work may not be worth it?
There are also the question if all the UDF files should be in the same repository or split into smaller ones based on code relevance?

 

5 hours ago, WildByDesign said:

By default, UDF files are stored based on the installer which is generally: %PROGRAMFILES(X86)%\AutoIt3\Include

This could present a problem for a UDF Updater in the sense that it would require being run as Admin. That is certainly possible. However, another option could be to place the UDF files in a user-writable location such as: %LocalAppData%\AutoIt v3\Include

There is also the case of using AutoIt portable, like from an USB stick. When AutoIt and official UDF's are downloaded together, this is not an issue to do manually, but introduces a problem, if the updater does not support this, and the user now manually need to check compatible versions between the two manually.

 

1 hour ago, SOLVE-SMART said:

my first thought was: @genius257 already tried/did a similar step in such a direction.
I mean he created a package manager au3pm like npm (JS/Node.js) or pip (Python) or NuGet (.NET), which would/could handle such UDF versioned files.
But your approach or idea is not exactly the some, but similar as genius approach or thoughts. At least if I understood it correctly.

@SOLVE-SMART I have indeed a half finished, unofficial AutoIt package manager. It would be able to handle the version dependency checks, if the relevant repositories uphold the same standard with version requirement information files (au3pm.json in my case).
I am happy to talk different options, regarding package manager implementation details, if relevant

 

41 minutes ago, SOLVE-SMART said:

Understandable, yes, thanks. Maybe the ideas could be combined in any way or at least we discussed this "possibility" here. I hope @genius257 will find the time to comment on that approach/suggestion.

The difference between official UDF's and user UDF's could be irrelevant, if they just comply with the same versioning standard, or they are compatible in some manner.
In regards to a site "vouching" for a package, it depends on how the package manager infrastructure is setup. For example i originally planned to have a central package registry for tracking packages and their versions, but ended up supporting GitHub repositories references instead. This does mean that i won't ever have a searchable list of package dependencies, but also means that no one is dependent on if i keep a service running or not.

Ultimately the package manager just need to be able to resolve packages to one or more downloadable links, compare compatibility via a standardized way and resolve/download the dependency tree in a standardized manner for the solution that also works with the relevant coding language runtime.

 

4 hours ago, argumentum said:

AutoIt looks for the includes subfolder, and in SciTE one can declare a custom includes path ( all this I'll have to read up on because I'm not too sure ) so, ...

Not so much SciTE, as creating or modifying the "HKEY_CURRENT_USER\Software\AutoIt v3\AutoIt" semi comma delimited list of User-defined library folders (nitpick i know...). More detail on the Keyword #include documentation page.

[argumentum]

56 minutes ago, SOLVE-SMART said:

Understandable, yes, thanks. Maybe the ideas could be combined in...

No.
And your insistence ruins even the slightest possibility of any change to the current installer.

[argumentum]

@genius257, I see the sad emoji. And there is nothing I can do about that strict "no" to the notion of a broader use of the installer  

Actually, just this notion of making an on-line installer, separating the AutoIt3 stuff from the UDFs, is something that is not wanted from the higher ups as it takes a whole lot of work to keep up.

Convincing them ( am part of them, but not them all, nor the last word ), is going to take a well thought out plan. Something that would invite to invest into changing the way things are, that as you could imagine, is set in stone.

So, we either stay on track in regards of what can be done in regards to having the UDFs updated more often, or sincerely forget that anything will change.

If the idea we come to, is sound, and makes Jon's life easier, and the consensus in the MVPs agree, then maybe, just maybe, we can have it.

[WildByDesign]

There is a lot of great feedback here. I still have to go over a lot of these comments to respond later when I have a bit more time. I appreciate all of the feedback.

7 hours ago, argumentum said:

... how do you propose it would all work for the end user in exact concrete terms ? ( not pushing your buttons but clarifying your vision  )

This is a really good question that requires some more thought, for sure. I've got to put some more thought into how this would all work. I will have to get back to you on this.

7 hours ago, argumentum said:

or the internal SVN revision I think would be best. 3.3.18.0 (r13212)

This would make the most sense and is something that would already be available to script that revision number into the versioning of the UDF files, JSON or however which way we can end up tracking this.

2 hours ago, argumentum said:

and makes Jon's life easier

This, 100%. This would be well deserved.

One issue that I was just thinking about would be the hosting of those UDFs. As we all know, the AutoIt site and forum has been getting hammered the last few months and sometimes not responding because of that. Having this functionality hosted on the AutoIt servers would likely worsen that problem.

I wonder if they can be hosted in a GitHub repo or if that would be against the rules on GH or not. I suppose this is something to consider.

[argumentum]

3 hours ago, WildByDesign said:

One issue that I was just thinking about would be the hosting of those UDFs. As we all know, the AutoIt site and forum has been getting hammered the last few months and sometimes not responding because of that. Having this functionality hosted on the AutoIt servers would likely worsen that problem.

I wonder if they can be hosted in a GitHub repo or if that would be against the rules on GH or not. I suppose this is something to consider.

The site getting hammered..., is just that. But is mostly the DB engine having trouble keeping up with request. Like a DDoS on the DB. But just reading files should not be a problem.

GitHub hosting the SVN may not be accepted because it all works "like a well oiled machine" as is. All the scripts that are made for SVN .... moving to GitHub will not happen. I don't see that happening.

Building the help file, zipping the UDFs and uploading to this site can be done in under 30 minutes ( that given the automation would be 5 to 10 min. of total human work ), so that is not like "OMG, what a pain" when Jon finds the time once the UDFs are finished**.

**We are all working independently without a central coordination. We take "something" and work on it when we have the time. All we'd have to do (MVPs), is to make a post in the MVP area and ask for status, and if the status is "am good" for everyone, then that is a good moment to distribute the \includes\ folder ( with a newly compiled help file ). But, that is my opinion and not anything that the MVPs are aware of because so far, am the only one on this thread, and there is no thread discussing what we are cooking here on this thread elsewhere.

When we get something reasonable here on this thread, then there is something to bring up in MVP chat 🤔

To recap: This site is fine for the UDFs distro. The copy of the files to GitHub is going to be a very hard sell ( to not say impossible ).

Also, AutoIt is not open source. And I don't know what legality would apply if going to M$ owned GitHub. That as disregarding as we may be of how international software legalities work, if kept inhouse, there is no risk of future confrontation due to, ??? something.

Edited 18 hours ago by argumentum
English

[argumentum]

@SOLVE-SMART, I see the confused emoji. You should know that I have you in the highest regards. All you do is well done and you have an amicable demeanor, yet I answered with an abrupt but clear answer. And that is because asking for what the OP is presenting is a "no-go", "not gonna happen" kind of situation.

So, how can we rescue the proposition ?, a compromise.
Don't go overboard with the request. Lets see if the independent UDF portion can be updated more often. If that takes off and becomes a reality, then expanding it to some of the UDFs from the WiKi ?, maybe ?.
But for now, focusing on distributing the standard UDFs more often is a goal that merits attention. And even that has a strong opposition in my view. So, baby steps.

In any case, a package manager for GitHub hosted projects is a simple thing compared to the private ( until full release ) of the standard distribution UDFs. I say that we focus on that if we expect any attention. My 2 cents. Am a forum user just like you in regards to this.

[argumentum]

You know that in "\AutoIt3\Extras\AutoUpdateIt\" there are 2 scripts to update AutoIt and SQLite. Maybe we can ask for a UDFs update script and place the UDF.ZIP file somewhere on this site and that would not require any changes for the current installer nor wait for a next release to get what we wanna see. Just an idea

[SOLVE-SMART]

Kind of off-topic:

Spoiler

9 hours ago, argumentum said:

No.
And your insistence ruins even the slightest possibility of any change to the current installer.

A "no" is totally fine for me, but I don't get how thoughts (or ideas with words like "maybe") should ruin "the slightest possibility of any change to the current installer"?
Please @argumentum keep in mind that this is a platform (forum) about thoughts, ideas, exchange of the community. I also do not agree to every statement of people here, but I cannot set my opinion and experience over others. As usually I don't take this personally in any way. I believe I know your style of writing so don't worry.

💡 I wrote this 👆 before you added the following statement:

1 hour ago, argumentum said:

@SOLVE-SMART, I see the confused emoji. You should know that I have you in the highest regards. All you do is well done and you have an amicable demeanor, yet I answered with an abrupt but clear answer. And that is because asking for what the OP is presenting is a "no-go", "not gonna happen" kind of situation.

Thank you, I can undoubtedly return the compliment. I believe you also have an amicable demeanor, but it's sometimes hiding behind a facade because of disappointments regarding AutoIt specific topics (at least what I am feeling about it). Anyhow, I respect you and what you do is valuable, thank you.

 

1 hour ago, argumentum said:

And that is because asking for what the OP is presenting is a "no-go", "not gonna happen" kind of situation.

How do you know this so strongly? Out of experience of the past of how improvements to the language or to the language ecosystem were handled by Jon (and/or the DEVs)? If so, then okay, it is what is it.

1 hour ago, argumentum said:

Lets see if the independent UDF portion can be updated more often. If that takes off and becomes a reality, then expanding it to some of the UDFs from the WiKi ?, maybe ?. But for now, focusing on distributing the standard UDFs more often is a goal that merits attention. And even that has a strong opposition in my view. So, baby steps.

Yes, baby steps would be good. I was not aware of the way such things will be processed here.

• I mean a thread with thoughts/ideas is created.
• Once a proposal and/or concept has matured, it is forwarded to the MVPs (separate thread).
• Then this can presented to Jon?

Why do we have the "mentioned functionality" @<name> when this is so complicated 🤔 ? Sure, to filter SPAM and crap out, but it's hard to understand what is a "no-go" and what not. Doen't matter ...

1 hour ago, argumentum said:

In any case, a package manager for GitHub hosted projects is a simple thing compared to the private ( until full release ) of the standard distribution UDFs. I say that we focus on that if we expect any attention.

... yes 🤝 .

Let's see what @WildByDesign can rethink and add more answers, like he wrote. I also think that this topic would benefit from @genius257's experience (package manager), even when we don't focus on this approach.

-------

Okay, for now I have to do my working day 👨‍💻 .

Best regards
Sven

Update:

46 minutes ago, argumentum said:

You know that in "\AutoIt3\Extras\AutoUpdateIt\" there are 2 scripts to update AutoIt and SQLite. Maybe we can ask for a UDFs update script and place the UDF.ZIP file somewhere on this site and that would not require any changes for the current installer nor wait for a next release to get what we wanna see. Just an idea

No I wasn't aware of this, but seems like a good start, yes.

Edited 14 hours ago by SOLVE-SMART

[argumentum]

7 minutes ago, SOLVE-SMART said:

Let's see what @WildByDesign can rethink and add more answers, like he wrote. I also think that this topic would benefit from @genius257's experience (package manager), even when we don't focus on this approach.

on that front, evaluating a JSON or INI or something, to hold the distro's info would be beneficial for a standardized updater that would keep in mind "what if" we integrated it an all-in-one in the future and/or common interaction, as a standard.

12 minutes ago, SOLVE-SMART said:

Okay, for now I have to do my working day 👨‍💻 .

Ok, I too have to go, catch some ZZZzzz  

[genius257]

9 hours ago, argumentum said:

@genius257, I see the sad emoji. And there is nothing I can do about that strict "no" to the notion of a broader use of the installer  

The sad emoji was not about the verdict, but the harsh response.
I completely understand why including user UDF's are not in the interest of the AutoIt team.

[SOLVE-SMART]

3 hours ago, genius257 said:

but the harsh response.

All good @genius257, as you can see in my "Kind of off-topic" spoiler section:
https://www.autoitscript.com/forum/topic/213399-concept-udf-updater-repo-decoupling-udf-files-from-autoit-installer/#findComment-1548757

I think it was phrased more harshly than intended.

Best regards
Sven

[WildByDesign]

8 hours ago, argumentum said:

To recap: This site is fine for the UDFs distro. The copy of the files to GitHub is going to be a very hard sell ( to not say impossible ).

After thinking about it some more, the small amount of users checking for UDF updates would likely not cause any issues. The compressed size of the UDF files is much smaller than I had initially realized. For .7z archive ~530 KB and ~760  KB for .zip archive.

5 hours ago, argumentum said:

So, how can we rescue the proposition ?, a compromise.
Don't go overboard with the request. Lets see if the independent UDF portion can be updated more often. If that takes off and becomes a reality, then expanding it to some of the UDFs from the WiKi ?, maybe ?.
But for now, focusing on distributing the standard UDFs more often is a goal that merits attention. And even that has a strong opposition in my view. So, baby steps.

Compromises are an important part of life. Baby steps as well. My main goal is for the updating of the included UDF files from the installer. That is the most important part. Updates from these included UDF files can fix some really important things between releases.

Separately from this, there is nothing to stop anyone from creating some type of community-driven UDF downloader/updater from a community-driven curated list of UDF files. That would likely be something that the user could select UDF files from various categories to download/update. I like that idea. But it would be a separate topic from this one. This one is all about doing a full update of all included UDF files based on whatever is the currently approved SVN revision.

6 hours ago, argumentum said:

You know that in "\AutoIt3\Extras\AutoUpdateIt\" there are 2 scripts to update AutoIt and SQLite. Maybe we can ask for a UDFs update script and place the UDF.ZIP file somewhere on this site and that would not require any changes for the current installer nor wait for a next release to get what we wanna see. Just an idea

So I actually really like the simplicity of this idea. I personally have a problem where I tend to over-complicate things initially at the planning/organizing stage, when in reality things don't always have to be complicated. You are referring to a familiar directory that already contains familiar update-related scripts. I think that this makes very good sense. Simple too which is also good.

[WildByDesign]

5 hours ago, argumentum said:

on that front, evaluating a JSON or INI or something, to hold the distro's info would be beneficial for a standardized updater that would keep in mind "what if" we integrated it an all-in-one in the future and/or common interaction, as a standard.

Originally, I was thinking of adding the versioning to the top of each UDF file for the possibility of updating some (or all) of the UDF files or unselecting some that may have script breaking changes. However, I'm pretty certain that you suggested that it would be an all or nothing type of thing where all UDF files would have to be updated together. I'm assuming likely because of some of those functions relying on other functions within it and so on.

In that regard, I think that having a single versioning file at the top of the UDF directory would be sufficient. Kind of like how it already has that _ReadMe_.txt file, we can add a file to the top of the directory that would contain the SVN revision number along with whatever else might be pertinent information.

Since the UDF files would be packaged in an archive, we really only need to check the hash of the archive. I don't think that there is a need to check the hash of all 150+ UDF files.

I don't know much about the server-side aspect. But it would have to have a value to determine whether the users' update check has the latest UDF revision or not.

[WildByDesign]

1 hour ago, SOLVE-SMART said:

as you can see in my "Kind of off-topic" spoiler section:

You are always a fantastic human being, Sven. A great example. Always positive thinking and I appreciate what you do for this community.

I think that this is going to end up much more simplistic then I had originally thought. Although that is certainly a good thing. I think that if we were to do some sort of separate community-driven project (separate topic also), in that case we could certainly make great use of @genius257's package manager functionality because in that case it would be tracking UDF files on an individual basis. I really like that idea and I can help with the GUI aspect of it. Although that would be a much bigger project with lot of work.

[argumentum]

2 hours ago, WildByDesign said:

For .7z archive ~530 KB and ~760  KB for .zip archive.

A zip file is not a great compressor but the most flexible packer for use from WinXP to Win11. Bandwidth is worse but flexibility is better.

2 hours ago, WildByDesign said:

creating some type of community-driven UDF downloader/updater from a community-driven curated list of UDF files. That would likely be something that the user could select UDF files from various categories to download/update. I like that idea. But it would be a separate topic from this one. This one is all about doing a full update of all included UDF files based on whatever is the currently approved SVN revision.

2 hours ago, WildByDesign said:

I personally have a problem where I tend to over-complicate things initially at the planning/organizing stage, when in reality things don't always have to be complicated.

On that: thinking ahead on what that implementation would need ... . Would be nice if the release.inf ( to give it a name ) was compatible with that unrelated project to future proof the possibility of the integration of both, by the community driven updater. 

And as you can see, I too tend to over engineer / over think stuff. There is no need for the release.inf to be compatible ( but it'd be nice )

1 hour ago, WildByDesign said:

... in that case we could certainly make great use of @genius257's package manager functionality because in that case it would be tracking UDF files on an individual basis. I really like that idea and I can help with the GUI aspect of it. Although that would be a much bigger project with lot of work.

If the release.inf file is a good standard, anyone can make a package manager based on the release.inf standard. So getting that release.inf well thought out ( by those with experience with that type of software, I've never used one ) is important, I believe.
Not important to this project but again, it'd be nice to think ahead.

2 hours ago, WildByDesign said:

I don't know much about the server-side aspect. But it would have to have a value to determine whether the users' update check has the latest UDF revision or not.

Look at the code of the 2 scripts already there. Then we follow the same criteria for this updater.

7 hours ago, argumentum said:

Ok, I too have to go, catch some ZZZzzz  

... I just wanted a separator here  

When texting my kids a question, if they don't respond within a few minutes, I text: "yes,no,maybe" like in, answer !.
Since I don't pay much attention to emotional damage 🤷‍♂️, at times I don't filter myself adequately, or even close. Or just no filter. ...because I process stuff different ( maybe ) and things are "right to the point", is my default trend of thought. My mind works on a "if, then, else" in a loop, to the point that there are so many if,then,elses that I get overwhelmed, and go right to the end result of my if,then.elses, hence a stern yes/no/maybe. I mean no harshness.
The avatar I use is the best I found to show myself in a picture 😅

Edited 7 hours ago by argumentum