AutoIt Package Manager

[genius257 120]

Hello.

So for some time now, I've been thinking about the possibilities with a package manager for AutoIt, like npm for UDF's and such.

I am curious if anyone else would be interested in such a thing, or if it would be useless/problematic in practice?

My personal problem with the idea so far, is the hurdle of legality and hosting options.

Your thought(s) on this matter would be greatly appreciated.

 

Edit:

Package repo web page: https://au3pm.github.io/

Docs: https://genius257.github.io/au3pm/

Project: https://github.com/genius257/au3pm/

Latest release: https://github.com/genius257/au3pm/releases/tag/0.3.0

Download link: https://github.com/genius257/au3pm/releases/download/0.3.0/au3pm.exe

Edited August 18, 2021 by genius257
new version

[Danp2 1,156]

Did you see this recent thread?

 

[genius257 120]

4 minutes ago, Danp2 said:

Did you see this recent thread?

Hi @Danp2.

Yeah, sadly right after i posted this topic.
Story of my life

[Deye 114]

Hi genius257.

I say give it a go if you are ready & willing,  it will surely be interesting to see what you can come up with for this
I was starting it about over here  and then got caught up doing other stuff
personally it can help with the house keeping .. 

There can be 2 versions or ways to go about it :

1. - Doing as I described at the  previous Topic  @ coping the files to @scriptDir

2. - would be to edit the paths for the includes pointing to 1 folder that will be holding all the extras ..

Deye

[genius257 120]

Hi @Deye.

Thanks for the nice feedback

I guess I’ll get started today when I get home from work

About the files themselves I considered the "global" installation would download the module to the "Standard library" folder, in most cases that would be the /Include folder in the installation directory. The local installed modules would be in a folder maybe called au3_modules, and would work by adding a relative path to the registry key "Include" in the path "HKEY_CURRENT_USER\Software\AutoIt v3\AutoIt\".

With conflicting versions I see 2 possibilities:

• Report error and do nothing with the conflict.
• Parse the package and rename all functions with a version prefix.

I also plan on using this "guide" to get me started on the basic workings of the version management and structure.

[Deye 114]

Hi genius257,

I think the idea of grouping some modules into 7z packages can just make it easy to pull out single specific files as needed
* having something that makes a listing of files of @Scriptdir into an ini - listing all production modules and versions

* Anything that cannot be retrieved by this pack system remains siding the UDF along with its file versions stored in the ini .. 
Then maybe  if there can be some kind of switch for instance #AutoIt3Wrapper_module_Override = y : causing anything in @ScriptDir to gain priority over the installed autoit's lib file modules including the autoit.exe version ..  then it ca make thins even easier ..

One example of what this can be good for : For instance someone posts a UDF with its in_production ini file and you need to quickly test the examples attached without going through the hurdle of inspecting all its script breaking changes future-wise ..
another example is @ offering quick fixes or ways to update older UDFs for instance: https://www.autoitscript.com/forum/topic/167184-winapivhd-udf-winapiex-udf-extension/ ..

btw 
your StringRegExp() doesn't do:

#AutoIt3Wrapper_Res_File_Add=C:\Users\me\Desktop\88x31.jpg, RT_ICON ..

@ what i got so far:

expandcollapse popup

#NoTrayIcon
;Place it inside any module dir you want, Update its path from any other module dir paths by executing it stand alone ..
#pragma compile(Out, Get_au3_modules_Cmd.exe)
#AutoIt3Wrapper_Change2CUI=y
#include <File.au3>
If Not @Compiled Then Exit

If $CmdLine[0] = 0 Then
    $address = 'HKEY_LOCAL_MACHINE\SOFTWARE\Classes\AutoIt3Script\Shell\__Get_module_Copies\Command'
    If Not StringInStr(RegRead($address, ""), @ScriptFullPath) Then
        If Not IsAdmin() Then Exit ShellExecute(@ScriptFullPath, "", @ScriptDir, "runas")
        RegWrite($address, '', 'REG_SZ', '"' & @ScriptFullPath & '" "%1"')
        ConsoleWrite(' a new option called: "__Get_module_Copies)" has been added to the explorer *.au3 RC Context Menu ..' & @CRLF)
        Sleep(5000)
    EndIf
    Exit
EndIf

#include <WinAPIShPath.au3>
$sFile = FileRead($CmdLine[1])
$pattern = '(?(DEFINE) (?<string>(["''''])((?!\2).)*\2))(?:#include ((?&string))|FileInstall\(((?&string))|#AutoIt3Wrapper_Res_[a-zA-Z]+_Add=([^\n]+))'
$aResults = StringRegExp($sFile, $pattern, 3)
For $i = 0 To UBound($aResults) - 1
    If StringLen($aResults[$i]) > 0 Then
        $s = StringRegExpReplace($aResults[$i], '^(["''''])(.*)\1', "$2")
        $sPath = @WorkingDir & "\" & $s
        $sCopyDest = @ScriptDir & "\" & $s
        If FileExists($sCopyDest) Then
            FileCopy($sCopyDest, $sPath, 8)
            If Not @error Then
                ConsoleWrite(_WinAPI_PathCompactPathEx($sPath, 65) & "   -Copied" & @CRLF)
                $sFile = FileRead($sPath)
                $aResults1 = StringRegExp($sFile, $pattern, 3)
                For $j = 0 To UBound($aResults1) - 1
                    If StringLen($aResults1[$j]) > 0 Then
                        $s = StringRegExpReplace($aResults1[$j], '^(["''''])(.*)\1', "$2")
                        $sPath1 = StringLeft($sPath, StringInStr($sPath, "\", "", -1))
                        $sCopyDest = @ScriptDir & StringRight($sPath1, StringInStr($sPath1, "\", "", -2))
                        If FileExists($sCopyDest) Then
                            FileCopy($sCopyDest & $s, $sPath1 & $s, 8)
                            If Not @error Then
                                ConsoleWrite(_WinAPI_PathCompactPathEx($sPath1 & $s, 65) & "   -Copied" & @CRLF)
                            Else
                                ConsoleWrite($sFile & "   not found" & @CRLF)
                            EndIf
                        EndIf
                    EndIf
                Next
            Else
                ConsoleWrite($sFile & "   not found" & @CRLF)
            EndIf
        EndIf
    EndIf
Next

Sleep(2500)

Deye

Edited July 13, 2018 by Deye

[genius257 120]

Status update.

I've been stuck on a host for the package registry, and finally found a solution (at least for now), thanks to the introduction of GitHub Actions.

Currently the plan is only to support GitHub hosted repositories (represent packages).

The package manager solution I've started to implement is heavily inspired by NPM-cli and yarn package manager.

I plan on finishing initial code for the install command today, and pusing it to the au3pm repository, if anyone is interested in giving feedback or just for looking into it .

Initial install command will only support getting:

• custom github repo/commit
• au3pm package with semantic versioning support
• local packages

I will be happy to explain anything I've done so far, but i remind you that this is in the alpha stage so nothing is set in stone.

Links:

au3pm repository

au3pm package registry (currently does nothing)

au3pm dev package registry (currently used for the alpha testing with au3pm)

[seadoggie01 285]

I love this idea, I keep needing to update TaskScheduler.au3 from water at home and work and I'm tired of downloading the same thing over and over

Even if GitHub actions doesn't work out or goes away, you could still use releases from the GitHub API... I have some code at home for pulling released binaries from the latest version. I was going to use it for keeping programs up to date.

[genius257 120]

Hi @seadoggie01

Thank you

The GitHub Actions are intended only for the "official" au3pm package registry. It holds only the references to the GitHub repo and SHA of the release. GitHub Actions are needed to automate the registering of package releases on the package registry. This way if the au3pm "butler" script works as intended, members of the au3pm GitHub Organisation with Admin repository permissions will only need to make changes to the package registry in rare cases.

Ideally a proper package registry should be made, but as au3pm is in it's infancy and the number of expected users actually using it is unknown, this current solution is intended as a starting point. The package registry may very likely need to be hosted on a server environment, more akin to the NPM, should needed features and number of users using au3pm turn out to be of that scale.

I looked into using GitHub releases, but GitHub releases are finicky, as some releases will be listed on the releases endpoint and some will be listed in tags. Take into account there is no way to make sure a user does not adhere to expected version naming or renaming of releases. I would love to see your code to pull the binaries, i may simply not understand the GitHub releases API enough.

[seadoggie01 285]

TBH, I don't get all of what you just said, but I figured there was a good reason to use Actions besides it being shiny

I'm not great at programming with APIs, but I found the GitHub API and managed to find this: https://developer.github.com/v3/repos/releases/#get-the-latest-release. I started testing it with some code like this:

Spoiler

#include <CustomDebugging.au3> ; Debug($sMsg) -> Prints to console and log.txt
#include <Array.au3>

Global Const $fileName = @TempDir & "\releases.json"
Global Const $apiUrl = 'https://api.github.com/repos/'

_GitRel_AssetLatestGet('seadoggie01', 'Test')

Func _GitRel_AssetLatestGet($userName, $repository)

    ; build url
    Local $downloadUrl = $apiUrl & $userName & "/" & $repository & "/" & "releases/latest"
    Debug($downloadUrl)

    ; Download the json
    If InetGet($downloadUrl, $fileName, 1, 0) = 0 Then
        Debug("No file get")
        Return SetError(1, 0, False)
    EndIf

    Local $json = FileRead($fileName)
    
    ; Display all downloads
    _ArrayDisplay(StringRegExp($json, '(?i)(?U)"browser_download_url":\s*"([^]]*?)"', 3))

    ; Delete the json file
    ;~FileDelete($fileName)

EndFunc

 

It gets an array of the url of each asset that is added to the release. You can also get the zip/tar of the source code of the release as well, but I didn't write that... I thought I did more, but apparently not. (The repository in the code is real, see it here: https://github.com/seadoggie01/Test/releases)

[genius257 120]

Yeah i tend to ramble, sorry ^^'

Thank you for the code yeah it seems to be working just fine

I might look into using releases/tags with au3pm again at a later time, then

[genius257 120]

Small update post, to let you know I am still very much working on this.

Further development of au3pm, beside bug fixes are on hold, until i develop a better way of working with command line arguments (currently in progress).

pre-release version is available via Github (see link(s) in top post)

initial documentation is available, thanks to Github pages.

Currently the package manager bot is broken after the official launch of GitHub Actions, but i will be more than happy to add packages manually if you create an issue here.

Let me know if something is confusing about the implementation or usage, as i try to keep it as simple and user friendly as possible

Edited September 8, 2020 by genius257

[TheDcoder 1,544]

The project looks interesting, I once thought of making a package manager myself

However I have to say that I dislike npm, and the way it managed packages per folder... I like my packages to be in a central location, being shared with other scripts as needed. I guess everyone has their own preferences.

What I do like is how a GitHub repository can be served as a package (if I am understanding it correctly), it will be interesting to see how this project grows. Best of luck @genius257

[genius257 120]

Small update:

The default package manger lookup repo now has a website for viewing and creating packages https://au3pm.github.io/#/

It is important to note that everything is still pre-release, so this may not reflect the final product.

[genius257 120]

On 9/7/2020 at 9:18 PM, TheDcoder said:

However I have to say that I dislike npm, and the way it managed packages per folder... I like my packages to be in a central location, being shared with other scripts as needed. I guess everyone has their own preferences.

I understand how you feel, when looking at current practices of AutoIt file includes.

But having them globally creates problems when many projects may need the same dependency, but at different versions, and those dependencies, and so on.

A lot of problems with AutoIt includes became apparent for me, as I've been developing this project, and i do have some plans to address most, if not all of them.

It does however drastically change how AutoIt includes will work in the future, and presents a lot of code needed, to get to an ideal solution.

 

On 9/7/2020 at 9:18 PM, TheDcoder said:

What I do like is how a GitHub repository can be served as a package (if I am understanding it correctly)

That is correct As i currently have no plans of throwing money at this, and still want it to be reliably accessible, it seemed like a good approach to me.

 

On 9/7/2020 at 9:18 PM, TheDcoder said:

Best of luck @genius257

Thanks . Sorry for the late reply, i did read your post less than 24 hours after you posted it, but didn't take the time to reply and it got away from me 😓

[TheDcoder 1,544]

No problem.

Since my last post, I have tried out NPM once again and I see how it can be beneficial, one key component that I realized is that you can skip committing the "node modules" directory to the source control, many projects however do this and the result is huge repo sizes with just bloat.

The approach of just committing the package-lock.json (or whatever it is called) and calling npm to download all the deps after a fresh clone is far better IMO

Make sure you implement this in au3pm too

[seadoggie01 285]

Bit of feedback on the website: adding a package request isn't terribly understandable... some hints would be nice. I also had an issue logging in, it was stuck in a loop until I refreshed the page, but I'm not sure about that being your problem. And the plus button shows on all pages, which is a bit weird

The commit was super unclear... I expected to need the short string that GitHub shows, like 'Latest commit 3d29209 on Mar 18'. A dropdown or hint is probably needed
The package name was fine, but I wasn't sure if I should include ".au3"
The version number didn't allow a 4 part version, as commonly (I think anyways) seen on UDFs. Saying why it failed would be nice

I'm sure that it's also pretty beta though, so I like it!

[genius257 120]

1 hour ago, TheDcoder said:

Make sure you implement this in au3pm too

The lock file is not implemented yet, but it is in planning but the concept of only committing the config file, and not the dependecies themselves is working, like on my DllStructEx repo. (the au3pm.json file)

Edited August 14, 2021 by genius257

[genius257 120]

19 minutes ago, seadoggie01 said:

Bit of feedback on the website: adding a package request isn't terribly understandable... some hints would be nice. I also had an issue logging in, it was stuck in a loop until I refreshed the page, but I'm not sure about that being your problem. And the plus button shows on all pages, which is a bit weird

The commit was super unclear... I expected to need the short string that GitHub shows, like 'Latest commit 3d29209 on Mar 18'. A dropdown or hint is probably needed
The package name was fine, but I wasn't sure if I should include ".au3"
The version number didn't allow a 4 part version, as commonly (I think anyways) seen on UDFs. Saying why it failed would be nice

I'm sure that it's also pretty beta though, so I like it!

Thanks for the very useful feedback @seadoggie01

Yeah i agree feedback on the fields are missing

About the versioning it's tricky. I plan to use semantic versioning instead of file versioning used by AutoIt3. This has more to do with standardizing when comparing usage with other package managers for other languages. That being said it should be explained to the user 😓

The logging in getting stuck I'm aware of, and have fixed on my codesandbox (dev env), just haven't pushed and build on GitHub. It occurs only when promted with granting access dialogue, so i missed it until lately, since i had not revoked access before then.

 

But i will take your feedback to heart, and by next iteration, UX will be more in focus

[genius257 120]

version 0.3.0 released.

https://github.com/genius257/au3pm/releases/tag/0.3.0

changes:

• Added update command (a0a95f8)
• Fixed issue where creating the au3pm symbolic link folders would fail, if the path contained a space (6225d71)
• Fixed issue with au3pm package name regex, where some packages could not be installed as a result (4085014)

non code changes:

• started creating au3pm documentation pages (cli, and repo)