AutoIT Development Process / Verification Status

[Draksyl]

Firstly apologies in advance if this isn't the right area of the forums to ask the questions below - but there didn't seem to be anywhere else more appropriate?

I work in the Aerospace industry and we have some pretty strict rules regarding the set of tools we are allowed to use to build our systems.  Part of these rules includes the need to conduct formal qualification on any part of the toolset used to design the system, or to prove that the system meets the necessary functional, safety, security and regulatory requirements.

I understand that the AutoIT Script Compiler (and related tools) are provided on a strict 'As-Is' basis for use by the community (as per the AutoIT Licence) - but would like to understand a little more about the process and lifecycle used to develop and release incremental tool versions over time - particularly with regards to verification / regression testing.

• Is there a standard library of unit tests against which each new release is checked? 
• Would any such unit tests potentially be releasable to a third party company (with or without NDA), or are they proprietary and only to be used by the core development team?
• Could any additional information be released publically or privately about the tools verification status?

I did have a quick search of the forums but didn't find any information about what the development and verification process was for AutoIT?

My hope is that by finding out a little more about the way in which the tool is developed, managed, and verified before release that we can minimise any additional qualification testing required to prove to the regulatory authorities that the tool is 'fit for purpose' for test automation tasks.

Im also sure given the widespread use of AutoIT for test automation in the IT Industry that others within the community will have a similar interest in the tools verification status. 

Happy to take this conversation offline, either via PM or email if that is deemed more appropriate.

Thanks in advance for any information - or for pointing out any relevant forum posts that I have previously overlooked.

Regards, Draksyl

[TheDcoder]

I'm sure the AutoIt team would be happy to help you with consultation and custom terms in exchange for generous amounts of money

[Draksyl]

Fine by me - its not my money 😉

[Melba23]

Draksyl,

Quote

 but would like to understand a little more about the process and lifecycle used to develop and release incremental tool versions over time - particularly with regards to verification / regression testing.

The "process and lifecycle" for releases is whenever Jon has some time to spend on development - which as he has to work to keep a roof over his head and food on the table is not as frequent as some more impatient users might wish!

To the best of my knowledge there is no standard set of tests against which new versions are run. We do try our best to avoid problems but a quick look at the development history of AutoIt shows that major releases are nearly always followed in reasonably short order by updates to fix problems that have been caused - most often in the UDFs that come with the standard install.

As you stated, the AutoIt license states that the software is 'provided "as is" without warranty of any kind' so it is very much up to you to determine if the product is "fit for purpose".

 Having been involved in the aeronautical world all my life (operator rather then designer) I understand your problem - sorry we cannot be any more helpful.

M23

[Jos]

1 hour ago, Melba23 said:

To the best of my knowledge there is no standard set of tests against which new versions are run.

Just some extra info on this: There is an automated buildprocess for Prod/Beta releases which includes a big set of test scripts which are automatically ran and checks a wide range of AutoIt3 functionality, which has been built over the years. 

Edited October 31, 2023 by Jos

[Draksyl]

Thanks Melba and Jon  Jos for the responses.

Unfortunately for us the Verification problem occurs on three separate levels:

1. Does the AutoIT Compiler successfully translate a script command into the appropriate behaviour or Windows API Call?
2. Has a UDF been implemented correctly without the possibility of introducing errors during script execution?   [Assuming that the AutoIT compiler is error-free]?
3. Has AutoIT been integrated in an appropriate manner with the particular software under test  [Assuming the both the AutoIT Compiler and any UDFs are error-free]?

In the absence of any other evidence for (1) we could build up a library of simple scripts that exercise each of the native AutoIT functions used in the main 'Production' scripts, and then use that as evidence of correct function - in exactly the same manner as Jos' Prod/Beta test scripts.

UDFs can similarly be subject to testing using simple scripts in isolation to check that they function as advertised without error in the target test environment.

Even after we manage to gather evidence for (1) and (2), then (3) is the area in which I believe we are always going to have to generate additional application-specific evidence to show that the scripts have been integrated in an appropriate manner and are error free.  For example, which specific methods have been used to perform UI Automation, and via which set of UIA UDF libraries - which is obviously wholly-dependent upon how the software under test was built in the first place?

Has anyone else faced a similar problem when trying to adopt AutoIT into their local toolsets?  How did you overcome it?

Is there likely to be much interest in trying to gather a body of certification/verification evidence for the tool based upon a combination of simple test scripts and UDF testing - so that when people use the tool they can also have a set of accompanying functional tests to use as V&V evidence?

• To be clear here - what I'm talking about is not for Jos or the Devs to change anything about the way they manage or develop the tool (as they have access to the AutoIT Dev environment and we dont), but for any scripts or UDF tests that the community generate purely to assist in establishing the tools verification status to be stored in a central location - and added to over time?

Has anyone previously raised the prospect of generating a shared pool of V&V evidence that people can use to strengthen their arguments as to why AutoIT should be considered for use in the first place?   I suspect that community members may already have a surprising amount of material that may be of use in this regard stashed away  - but not necessarily collected together into one place and organised?  

Any ideas on how / if achieving such a goal is even possible?

Regards, Draksyl

Edited October 31, 2023 by Jos
Fixed name ;)

[Draksyl]

Ooops - sorry Jos

[Andreik]

4 minutes ago, Draksyl said:

Does the AutoIT Compiler successfully translate a script command into the appropriate behaviour or Windows API Call?

Most UDFs are based on Windows API but not every time this functions are limited to the call but how the result is presented to the user or it might involve some data preparation in order to make the call but it tries to keep in line with what each API does. Most close behavior to what are you looking for it's to use DllCall() and call these APIs by yourself but in preparing the call or processing the result same potential errors could be encountered, so I would stick with UDFs since these functions are widely used by many users and it's a better chance that someone will spot a wrong behavior.

I'm not sure if I fully understand your concerns but what I can tell you is that I used AutoIt for some projects in the largest bank from my country and everything was fine. Usually errors appears due to poor programming and in very few cases due to an in appropriate behavior of a programming language. I hope this helps you.

[Draksyl]

11 minutes ago, Andreik said:

I'm not sure if I fully understand your concerns but what I can tell you is that I used AutoIt for some projects in the largest bank from my country and everything was fine. Usually errors appears due to poor programming and in very few cases due to an in appropriate behavior of a programming language. I hope this helps you.

Our systems have to meet the requirements of standards like DO-178 and DO-254 which demand that all elements of the toolsets used to either design, or to test safety critical systems must themselves be qualified as fit for purpose.

AutoIT has not yet undergone any public qualification of its functionality - so the question we are trying to answer is whether we can gather sufficient evidence of its 'fitness for purpose' and 'correct function without error' as to be useful in our projects.

The alternative is to select an alternative commercial test tool which comes with a proven set of verification & validation test evidence - but that is obviously much more expensive not only to licence, and for ongoing consultancy - whilst not being nearly as flexible in its deployment as AutoIT.  

AutoIT is attractive for a number of reasons - but ultimately may not meet the stringent needs for evidence of the regulatory authorities who decide whether our system can be certified for use or not!

Edited October 31, 2023 by Draksyl
Edit - typo

[ioa747]

with window10 ?

[Melba23]

Draksyl,

Quote

AutoIT is attractive for a number of reasons - but ultimately may not meet the stringent needs for evidence of the regulatory authorities who decide whether our system can be certified for use or not!

As an end-user I can completely understand this requirement - for many years my life (and those of the others with me) depended on such strict adherance to standards!

M23

[argumentum]

this post gives a perspective of the situation: https://www.autoitscript.com/forum/topic/201901-we-get-our-thursdays-from-a-banana/

Good luck with the higherups

[Draksyl]

18 hours ago, ioa747 said:

with window10 ?

Potentially Win7 and Win10 instances - some of the hardware we are working with is older and has an embedded Win OS built into it that cant be upgraded.  For other parts of the test environment we are free to decide which windows version to use to host SW services and AutoIT scripts.

[Draksyl]

17 hours ago, argumentum said:

this post gives a perspective of the situation: https://www.autoitscript.com/forum/topic/201901-we-get-our-thursdays-from-a-banana/

Good luck with the higherups

That posts in the Active Members area, but in a happy coincidence this is my tenth post so I will hopefully be granted access shortly 😀

[TheDcoder]

46 minutes ago, Draksyl said:

AutoIT

It's "AutoIt" by the way, as in "Automate it", not "Auto Information Technology"

For some reason almost everyone (including me) gets it wrong initially.

[Draksyl]

5 minutes ago, TheDcoder said:

It's "AutoIt" by the way, as in "Automate it", not "Auto Information Technology"

For some reason almost everyone (including me) gets it wrong initially.

In my case the phone seems to have added the mis-spelled version into its dictionary and now it keeps 'correcting' the correct version incorrectly 🤣

[argumentum]

3 hours ago, Draksyl said:

I will hopefully be granted access shortly 😀

Spoiler

https://www.youtube.com/watch?v=Xe1TZaElTAs&t=96s

...is a post of the above video

[Draksyl]

1 hour ago, argumentum said:

  Reveal hidden contents

https://www.youtube.com/watch?v=Xe1TZaElTAs&t=96s

...is a post of the above video

Thanks - very informative and more or less in line with my opinion of the community given the abundance of UDFs and support freely offered to others on these forums.

Whilst I cant guarantee anything at this point, if we do decide to pursue qualification of AutoIt to DO-178 or DO-330 standards I will try to make sure that as much as possible of the library of V&V tests developed for that purpose gets fed back into the AutoIt community for other people to add to over time - hopefully making it easier in future to prove the 'fitness of purpose' for the tool to decision makers and regulatory authorities alike.