How to ease debugging and maintenance of big projects...?

[fsteff]

Hi,

Background:
I'm new to AutoIt, and I've recently taken over the maintenance of an AutoIt script, running 100% console mode. The original developer is no longer available for assistance. :-(
The script is about 24000 lines of code, distributed in about 100 files.
I briefly tried to use SciTE, but being used to Eclipse and other editors, I didn't find it useful. Instead I ended up with a combination of Total Commander (for function lookup) and NotePad++

Versions:
AU3Check (3.3.12.0)
Aut2Exe (3.3.12.0)
Au3Stripper (14.801.1932.0)

Stats:
Au3Stripper v14.801.1932.0 finished merging 24071 lines of code, stripped 11002 comment lines and Merged 2642 Continuation lines.

Problem:
When  executing the compiled script (with command line parameters from the windows command Prompt), and after running for 5 minutes I sometimes (not always) get an Array overflow error (I don't remember the exact wording), the offending line is stated to be in line 9525. I'm then attempting to use "Au3Stripper /MergeOnly ScriptName.au3", (which will generate an error that Scriptname.au3 is an "Invalid Au3Stripper option" and allow me to select the same Scriptname.au3, which then completes as it should.) to generate the expected ScriptName_stripped.au3 file.
Looking at ScriptName_stripped.au3 line 9525 and the surrounding 100+ lines it's very evident that they can't be causing the problem as there are absolutely no arrays used in that code - there are usage of global enum values, but the same enum values have been used before.

I'm now wondering:

1. Are the Au3Stripper really performing exactly the same actions as the Aut2Exe? (I would feel better is it was possible to generate the stripped version with the Aut2Exe tool)
2. What actions can I take to ease debugging this?
3. In general, what recommendations do you have to ease maintenance of this project?  (I'm thinking better tools, so suggestions are welcome.)
    

I'm looking forward to hear from you.

[JLogan3o13]

Hi, fsteff, welcome to the forum. Unfortunately, whether it is Java in Eclipse or AutoIt in SciTE, the methodology is the same; you need to know your code so you can do proper error checking. Unfortunately, you are in the unenviable position of being screwed by the previous developer, and this is going to take some time. I would suggest familiarizing yourself with the architecture of the script(s) first, even if you have to spend some time mapping out how each is connected.

Once you become familiar with the architecture, if you have questions about specific portions of code, please post that portion (as close to runnable as possible) and we will do what we can to assist. Good luck!

[minxomat]

24000 LOC? Are they organic? I suspect the prev. dev. just shipped a whole bunch of includes to prevent missing definitions, hence the heavy stripping of what i think are UDF headers. If so, you might just clean it up yourself and compress as much as you can. This is the first step of debugging project this big.

Edited June 8, 2015 by minx

[mLipok]

  On 6/8/2015 at 11:22 AM, fsteff said:

I'm now wondering:

1. Are the Au3Stripper really performing exactly the same actions as the Aut2Exe? (I would feel better is it was possible to generate the stripped version with the Aut2Exe tool)
2. What actions can I take to ease debugging this?
3. In general, what recommendations do you have to ease maintenance of this project?  (I'm thinking better tools, so suggestions are welcome.)
    

I'm looking forward to hear from you.

​

ad1.
Should work properly but you must know that it is only compared in compiled script, because when you try to run using F5 then this line is related to each UDF file separate.

ad2.
Compiled or running ?

ad3.

Start by reading Best Coding Practices in Wiki (look in my sign for link)

 

mLipok

 

btw. Welcome to the forum..

EDIT:

your code is not so big, just look here for my stats:

  Quote

+> 0 Au3Stripper v15.503.1200.1 finished merging 116790 lines of code, stripped 44473 comment lines and Merged 3492 Continuation lines.
+>        Created:Z:\TOOLs\Macro\PROGRAMY\************_stripped.au3 with 68825 lines.

For your information, I clean \my codes, for almost a year.
Just in the last 2 years I have acquired experience, and now I can no longer look at my old bad habits in my 20 major programs.

From day to day I am more and more pleased with the progress.
And they are just so slowly that simply everyone UDF, I try to write from scratch, and then I publish it on this forum.

 

 

Edited June 8, 2015 by mLipok

[fsteff]

Thank you for your replies, JLogan3o13, minx and mLipok, your welcome is appreciated.

 

@JLogan3o13,
Thank you for your comment. I fully agree that it's needed to know the code, and I am fairly familiar with it now. I have been making optimisations  to it the last week or so, and do feel reasonable at home already. (I'm used to much bigger projects in primarily C and assembly, and have 25+ years of coding experience.) What I lack in SciTE - or perhaps just haven't yet found out - is how multiple-files-projects are working, how to jump back and forth between functions and declarations etc.

 

@minx, At least none of the official AutoIT includes are in the project files, but they might be part of the output from Au3Stripper? However, I was under the impression that they were left out when using the /MO switch, and included when using the /SO switch.

 

@mLipok Thank you for your more specific answers to my questions, my responce below.

Ad 1, I'm aware the line numbers in the output from Au3Stripper can only be compared to the line numbers reported on errors in the compiled versions of the script.

Ad 2, Ease debugging the compiled version. Are there any options to get something like a core-dump and a stack trace (or whatever equivalent exists in AutoIt)

Ad 3, I'll look into that link, thanks.. However my question was more aimed at how to deal with navigating the many lines and files of code, as I really feel SciTE is very limited.

 

A few extra questions.

4. How do I make Au3Stripper accept an argument containing the script file name. I've tried the the script name with both relative and absolute paths and also just the filename, and in all cases I get the  "Invalid Au3Stripper option", followed by a file requester to select the file I want to start with. (I need this to better use the tool from a batch file)

5. I tried to just execute the script using "full-path/autoit3.exe /autoit3.exe-commandline_switches my_script.au3 my_script_commandline_options" (where /autoit3.exe-commandline_switches are all four combinations of the two switches described in the documentation), but in all cases nothing, absolutely nothing, happens, I'm just returned straight to the command line. 

 

I'm sure I'll have more questions as I attempt more.

[Melba23]

fsteff,

I am moving this to the "General Help" section.

M23

[reb]

  On 6/8/2015 at 7:39 PM, fsteff said:

Ad 3, I'll look into that link, thanks.. However my question was more aimed at how to deal with navigating the many lines and files of code, as I really feel SciTE is very limited.

​use SciTE Jump from the Tools Menu in SciTE

Edited June 8, 2015 by reb

[mLipok]

  On 6/8/2015 at 7:39 PM, fsteff said:

(I'm used to much bigger projects in primarily C and assembly, and have 25+ years of coding experience.) What I lack in SciTE - or perhaps just haven't yet found out - is how multiple-files-projects are working, how to jump back and forth between functions and declarations etc.

​So now we know something more about you.
You are using Au3Stripper so I guess you are also using SciTE4AutoIt - just press CTRL+F1 in S4AU3

Use SciTEJump      just press ALT+Q   in S4AU3

Use Calltips .

 

Read this : 

http://www.autoitscript.com/autoit3/scite/docs/SciTE4AutoIt3/SciTE4AutoIt3-getting-started.html

https://www.autoitscript.com/wiki/SciTE4AutoIt3

https://www.autoitscript.com/wiki/Adding_UDFs_to_AutoIt_and_SciTE

 

 

  On 6/8/2015 at 7:39 PM, fsteff said:

Ad 2, Ease debugging the compiled version. Are there any options to get something like a core-dump and a stack trace (or whatever equivalent exists in AutoIt)

You can use Trace from "Tools" Menu 
Or make your own solution to trace functions (like I did).

 

  On 6/8/2015 at 7:39 PM, fsteff said:

Ad 3, I'll look into that link, thanks.. However my question was more aimed at how to deal with navigating the many lines and files of code, as I really feel SciTE is very limited.

Use #region #end-region to group code.

use CTRL+J to jump in to the selected funtion, or jump back CTRL+SHIFT+J 

Use Tidy to sort functions

 

  On 6/8/2015 at 7:39 PM, fsteff said:

4. How do I make Au3Stripper accept an argument containing the script file name. I've tried the the script name with both relative and absolute paths and also just the filename, and in all cases I get the  "Invalid Au3Stripper option", followed by a file requester to select the file I want to start with. (I need this to better use the tool from a batch file)

Paste here the entire command which you use.

 

  On 6/8/2015 at 7:39 PM, fsteff said:

5. I tried to just execute the script using "full-path/autoit3.exe /autoit3.exe-commandline_switches my_script.au3 my_script_commandline_options" (where /autoit3.exe-commandline_switches are all four combinations of the two switches described in the documentation), but in all cases nothing, absolutely nothing, happens, I'm just returned straight to the command line. 

Paste here the entire command which you use.

 

  On 6/8/2015 at 7:39 PM, fsteff said:

I'm sure I'll have more questions as I attempt more.

Good.​

mLipok

 

 

[jchd]

fsteff,

Try to forget stripper and get back to "normal" code, i.e. applicative code split in sensible sub-domains invoking standard UDFs.

  Quote

What I lack in SciTE - or perhaps just haven't yet found out - is how multiple-files-projects are working, how to jump back and forth between functions and declarations etc.

That's where you shoot yourself in the foot, if you allow me to say that. SciTE is  just an editor with a (large) number of way to ease your job. Eclipse is a complete develpoment framework with much, much more features built-in.

Let's stop at this: I see this lack of advanced debugging tools both a pain and a blessing. Let me explain a bit more why.

If I ask you to develop a real-world app all in assembly, you're going to demand a very powerful integrated development suite ... and a lot of time to reinvent all the wheels of a long train. Say that now I offer you to use instead a very high-level language. The efforts saved by using an advanced language have to translate in coding defensively and carefully (think before code) instead of relying on a line-by-line, step-by-step, instruction-by-instruction debugger to pinpoint where exactly you, as a coder, made a blemish in a 45-level stack. That means coding rigorously (testin,g for errors where they may occur) rather than "if ever this corner case happens, we'll fix it".

Of course AutoIt is a simple interpreted language mostly created by one individual with a large number of additions made by concerned people. That can't compete with Eclipse + <whatever fancy language is in the headlines today>. But is this really true? I find it revealing that many newcomers discovering AutoIt spontaneously declare that they are surprised how powerful the language is to achieve their goal, whatever it is (well, barring rewritting SAP or such, of course).

If I were in your place, I'd try to put back standard UDFs functions into where they came from, that is standard UDFs. That should simplify things greatly (given the number of lines you cite). Then I'd ask: is the overall architecture suited to both the needs and what AutoIt is today (it has significantly evolved over time). Obviously it's hard to give you useful advices without any knowledge of the code and what it is supposed to do.

[jchd]

Forget that, unexpected double post ending with some SQL or more global IPB error.... sh*tty update.

Edited June 8, 2015 by jchd
IPB's period!