Jump to content
fsteff

How to ease debugging and maintenance of big projects...?

Recommended Posts

fsteff

Hi,

Background:
I'm new to AutoIt, and I've recently taken over the maintenance of an AutoIt script, running 100% console mode. The original developer is no longer available for assistance. :-(
The script is about 24000 lines of code, distributed in about 100 files.
I briefly tried to use SciTE, but being used to Eclipse and other editors, I didn't find it useful. Instead I ended up with a combination of Total Commander (for function lookup) and NotePad++

Versions:
AU3Check (3.3.12.0)
Aut2Exe (3.3.12.0)
Au3Stripper (14.801.1932.0)

Stats:
Au3Stripper v14.801.1932.0 finished merging 24071 lines of code, stripped 11002 comment lines and Merged 2642 Continuation lines.


Problem:
When  executing the compiled script (with command line parameters from the windows command Prompt), and after running for 5 minutes I sometimes (not always) get an Array overflow error (I don't remember the exact wording), the offending line is stated to be in line 9525. I'm then attempting to use "Au3Stripper /MergeOnly ScriptName.au3", (which will generate an error that Scriptname.au3 is an "Invalid Au3Stripper option" and allow me to select the same Scriptname.au3, which then completes as it should.) to generate the expected ScriptName_stripped.au3 file.
Looking at ScriptName_stripped.au3 line 9525 and the surrounding 100+ lines it's very evident that they can't be causing the problem as there are absolutely no arrays used in that code - there are usage of global enum values, but the same enum values have been used before.

I'm now wondering:

  1. Are the Au3Stripper really performing exactly the same actions as the Aut2Exe? (I would feel better is it was possible to generate the stripped version with the Aut2Exe tool)
  2. What actions can I take to ease debugging this?
  3. In general, what recommendations do you have to ease maintenance of this project?  (I'm thinking better tools, so suggestions are welcome.)
     

I'm looking forward to hear from you.

Share this post


Link to post
Share on other sites
JLogan3o13

Hi, fsteff, welcome to the forum. Unfortunately, whether it is Java in Eclipse or AutoIt in SciTE, the methodology is the same; you need to know your code so you can do proper error checking. Unfortunately, you are in the unenviable position of being screwed by the previous developer, and this is going to take some time. I would suggest familiarizing yourself with the architecture of the script(s) first, even if you have to spend some time mapping out how each is connected.

Once you become familiar with the architecture, if you have questions about specific portions of code, please post that portion (as close to runnable as possible) and we will do what we can to assist. Good luck!


√-1 2^3 ∑ π, and it was delicious!

Share this post


Link to post
Share on other sites
minxomat

24000 LOC? Are they organic? I suspect the prev. dev. just shipped a whole bunch of includes to prevent missing definitions, hence the heavy stripping of what i think are UDF headers. If so, you might just clean it up yourself and compress as much as you can. This is the first step of debugging project this big.

Edited by minx

I will answer every single PM, and you are free to ask anything anytime.

Share this post


Link to post
Share on other sites
mLipok

I'm now wondering:

  1. Are the Au3Stripper really performing exactly the same actions as the Aut2Exe? (I would feel better is it was possible to generate the stripped version with the Aut2Exe tool)
  2. What actions can I take to ease debugging this?
  3. In general, what recommendations do you have to ease maintenance of this project?  (I'm thinking better tools, so suggestions are welcome.)
     

I'm looking forward to hear from you.

ad1.
Should work properly but you must know that it is only compared in compiled script, because when you try to run using F5 then this line is related to each UDF file separate.

ad2.
Compiled or running ?

ad3.

Start by reading Best Coding Practices in Wiki (look in my sign for link)

 

mLipok

 

btw. Welcome to the forum..

EDIT:

your code is not so big, just look here for my stats:

+> 0 Au3Stripper v15.503.1200.1 finished merging 116790 lines of code, stripped 44473 comment lines and Merged 3492 Continuation lines.
+>        Created:Z:\TOOLs\Macro\PROGRAMY\************_stripped.au3 with 68825 lines.

For your information, I clean \my codes, for almost a year.
Just in the last 2 years I have acquired experience, and now I can no longer look at my old bad habits in my 20 major programs.

From day to day I am more and more pleased with the progress.
And they are just so slowly that simply everyone UDF, I try to write from scratch, and then I publish it on this forum.

 

 

Edited by mLipok

Signature beginning:   Wondering who uses AutoIT and what it can be used for ?
* GHAPI UDF - modest begining - comunication with GitHub REST API Forum Rules *
ADO.au3 UDF     POP3.au3 UDF     XML.au3 UDF    How to use IE.au3  UDF with  AutoIt v3.3.14.x  for other useful stuff click the following button

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST APIErrorLog.au3 UDF - A logging Library
 

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF SMTP Mailer UDF * Dual Monitor resolution detection *

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * 

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * IE in TaskScheduler

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2018-10-15

Share this post


Link to post
Share on other sites
fsteff

Thank you for your replies, JLogan3o13, minx and mLipok, your welcome is appreciated.

 

@JLogan3o13,
Thank you for your comment. I fully agree that it's needed to know the code, and I am fairly familiar with it now. I have been making optimisations  to it the last week or so, and do feel reasonable at home already. (I'm used to much bigger projects in primarily C and assembly, and have 25+ years of coding experience.) What I lack in SciTE - or perhaps just haven't yet found out - is how multiple-files-projects are working, how to jump back and forth between functions and declarations etc.

 

@minx, At least none of the official AutoIT includes are in the project files, but they might be part of the output from Au3Stripper? However, I was under the impression that they were left out when using the /MO switch, and included when using the /SO switch.

 

@mLipok Thank you for your more specific answers to my questions, my responce below.

Ad 1, I'm aware the line numbers in the output from Au3Stripper can only be compared to the line numbers reported on errors in the compiled versions of the script.

Ad 2, Ease debugging the compiled version. Are there any options to get something like a core-dump and a stack trace (or whatever equivalent exists in AutoIt)

Ad 3, I'll look into that link, thanks.. However my question was more aimed at how to deal with navigating the many lines and files of code, as I really feel SciTE is very limited.

 

A few extra questions.

4. How do I make Au3Stripper accept an argument containing the script file name. I've tried the the script name with both relative and absolute paths and also just the filename, and in all cases I get the  "Invalid Au3Stripper option", followed by a file requester to select the file I want to start with. (I need this to better use the tool from a batch file)

5. I tried to just execute the script using "full-path/autoit3.exe /autoit3.exe-commandline_switches my_script.au3 my_script_commandline_options" (where /autoit3.exe-commandline_switches are all four combinations of the two switches described in the documentation), but in all cases nothing, absolutely nothing, happens, I'm just returned straight to the command line. 

 

I'm sure I'll have more questions as I attempt more.

Share this post


Link to post
Share on other sites
Melba23

fsteff,

I am moving this to the "General Help" section.

M23


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites
reb
Ad 3, I'll look into that link, thanks.. However my question was more aimed at how to deal with navigating the many lines and files of code, as I really feel SciTE is very limited.

​use SciTE Jump from the Tools Menu in SciTE

Edited by reb

MEASURE TWICE - CUT ONCE

Share this post


Link to post
Share on other sites
mLipok

(I'm used to much bigger projects in primarily C and assembly, and have 25+ years of coding experience.) What I lack in SciTE - or perhaps just haven't yet found out - is how multiple-files-projects are working, how to jump back and forth between functions and declarations etc.

​So now we know something more about you.
You are using Au3Stripper so I guess you are also using SciTE4AutoIt - just press CTRL+F1 in S4AU3

Use SciTEJump      just press ALT+Q   in S4AU3

Use Calltips .

 

Read this : 

http://www.autoitscript.com/autoit3/scite/docs/SciTE4AutoIt3/SciTE4AutoIt3-getting-started.html

https://www.autoitscript.com/wiki/SciTE4AutoIt3

https://www.autoitscript.com/wiki/Adding_UDFs_to_AutoIt_and_SciTE

 

 

Ad 2, Ease debugging the compiled version. Are there any options to get something like a core-dump and a stack trace (or whatever equivalent exists in AutoIt)

You can use Trace from "Tools" Menu 
Or make your own solution to trace functions (like I did).

 

Ad 3, I'll look into that link, thanks.. However my question was more aimed at how to deal with navigating the many lines and files of code, as I really feel SciTE is very limited.

Use #region #end-region to group code.

use CTRL+J to jump in to the selected funtion, or jump back CTRL+SHIFT+J 

Use Tidy to sort functions

 

4. How do I make Au3Stripper accept an argument containing the script file name. I've tried the the script name with both relative and absolute paths and also just the filename, and in all cases I get the  "Invalid Au3Stripper option", followed by a file requester to select the file I want to start with. (I need this to better use the tool from a batch file)

Paste here the entire command which you use.

 

5. I tried to just execute the script using "full-path/autoit3.exe /autoit3.exe-commandline_switches my_script.au3 my_script_commandline_options" (where /autoit3.exe-commandline_switches are all four combinations of the two switches described in the documentation), but in all cases nothing, absolutely nothing, happens, I'm just returned straight to the command line. 

Paste here the entire command which you use.

 

I'm sure I'll have more questions as I attempt more.

Good.​


mLipok

 

 


Signature beginning:   Wondering who uses AutoIT and what it can be used for ?
* GHAPI UDF - modest begining - comunication with GitHub REST API Forum Rules *
ADO.au3 UDF     POP3.au3 UDF     XML.au3 UDF    How to use IE.au3  UDF with  AutoIt v3.3.14.x  for other useful stuff click the following button

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST APIErrorLog.au3 UDF - A logging Library
 

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF SMTP Mailer UDF * Dual Monitor resolution detection *

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * 

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * IE in TaskScheduler

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2018-10-15

Share this post


Link to post
Share on other sites
jchd

fsteff,

Try to forget stripper and get back to "normal" code, i.e. applicative code split in sensible sub-domains invoking standard UDFs.

What I lack in SciTE - or perhaps just haven't yet found out - is how multiple-files-projects are working, how to jump back and forth between functions and declarations etc.

That's where you shoot yourself in the foot, if you allow me to say that. SciTE is  just an editor with a (large) number of way to ease your job. Eclipse is a complete develpoment framework with much, much more features built-in.

Let's stop at this: I see this lack of advanced debugging tools both a pain and a blessing. Let me explain a bit more why.

If I ask you to develop a real-world app all in assembly, you're going to demand a very powerful integrated development suite ... and a lot of time to reinvent all the wheels of a long train. Say that now I offer you to use instead a very high-level language. The efforts saved by using an advanced language have to translate in coding defensively and carefully (think before code) instead of relying on a line-by-line, step-by-step, instruction-by-instruction debugger to pinpoint where exactly you, as a coder, made a blemish in a 45-level stack. That means coding rigorously (testin,g for errors where they may occur) rather than "if ever this corner case happens, we'll fix it".

Of course AutoIt is a simple interpreted language mostly created by one individual with a large number of additions made by concerned people. That can't compete with Eclipse + <whatever fancy language is in the headlines today>. But is this really true? I find it revealing that many newcomers discovering AutoIt spontaneously declare that they are surprised how powerful the language is to achieve their goal, whatever it is (well, barring rewritting SAP or such, of course).

If I were in your place, I'd try to put back standard UDFs functions into where they came from, that is standard UDFs. That should simplify things greatly (given the number of lines you cite). Then I'd ask: is the overall architecture suited to both the needs and what AutoIt is today (it has significantly evolved over time). Obviously it's hard to give you useful advices without any knowledge of the code and what it is supposed to do.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites
jchd

Forget that, unexpected double post ending with some SQL or more global IPB error.... sh*tty update.

Edited by jchd
IPB's period!

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×