Sign in to follow this  
Followers 0
Robinson1

Making AutoItSC.bin smaller

30 posts in this topic

#1 ·  Posted (edited)

Recently I was playing around a little with the Visual C++ Compiler and saw that you can do much off from the standard to make the compiled code smarter & more compacted.

...

And beside that really nice Preprocessor I just discover by now:

http://www.autoitscript.com/forum/index.php?showtopic=30573&hl=cleanscript

It also happened that I had a look upon AutoItSC.bin. (The script interpreter stub that's attached to every compiled AutoIt-Exe.)

  • Wow nice no '.reloc' so I see you already 'found' the /DYNAMICBASE:NO' linker switch.

    Just because I read the first reply:

    So why is that good? - Guess what it's the *,exe that loads the *.dll and not the other way. So since it comes first in the new process adress space - nearly everything is free. So it can always load at it's preferred address and never need to 'move'/relocated to somewhere else.

  • However it happened that I loaded AutoItSC.bin into in IDA, And saw that the Common RunTime's are statically linked:Posted Image^- That's the whole adress space of AutoItSC.bin, the cyan area's are statically linked library functions.

    By using the Compiler switch /MD these areas will stay out of the AutoItSC.bin and load from msvcrt.dll that there in the system32 dir on ever windows installation.

    That might shrink the size something about 25%.

    Maybe I'm really not really telling any news about that /MultithreadDll switch. And in detail things are not that perfect. The performance 'loss' between calling a library function or a function to a loaded Dll is in my eyes minimal.

    The main reason is that depending on what VC you're using that MSVCRT.dll is sometimes names MSVCR70.dll, MSVCR71.dll, or with the 'brand new' VC++ 2011 MSVCR100.dll. and I'll need to include these *.dll since you can't really assume that they are already in the system.

    Well I for the other part of the reply I just checked it msvcrt.dll is even present in Win98 (Setupfile WIN98_57.CAB) Windows itself is written in C++ and will make itself use of it - like for ex. calc.exe or ftp.exe does.)

    Maybe you get it compiled with the old'n gold'n includes from VC6:

    http://www.sendspace.com/file/gaw39k

    that really includes MSVCRT.dll and not that new crap from the MSVCRXX-version-zoo where I can't see what better whem using. Something it need undocumented under the hood gribs to like -d2QIftol to say to the actual compiler no I don't want my FloatToLong convertions with SSE. Or adjusting stdio.h

    #ifndef _STDIO_DEFINED

    _CRTIMP extern FILE _iob[];

    #endif /* _STDIO_DEFINED */

    ...

    #define stdin (&_iob[0])

    #define stdout (&_iob[1])

    #define stderr (&_iob[2])

    so I get my 'stdin' handle directly via '_iob' array and not that fucking __iob_func that is not present in MSVCRT.dll

    ...okay next

  • Framepointers

    As C-Coder how landed from time to time in the ASM-Debug on nasty bug's you're probably know this:

    PUSH EBP

    MOV EBP, ESP

    SUB ESP, 10

    [EBP+8] ->Arg0

    [EBP+C] ->Arg1

    ...

    [EBP-4] ->LocalVar0

    [EBP-8] ->LocalVar1

    ...

    LEAVE

    RETN

    These framepointer based functions are somehow nice for debugging (and Reverse engineering) easier and quicker compiling, but for compiling optimized release code it's better to remove them with the /Oy compiler switch. That saves some space and bring minimalistic performance gains.

  • More Optimizations options

    I just name the in short without going to much in detail. Use the documentation at first place or ask for more details

    /GL - Full optimization

    /Oi - use Compiler build in asm functions like ROL instead of 'importing' them

    /Os - small code - no alignment gaps between functions.

    /GF - String pooling store identical strings only once

    /Gr - try to use __fastcall calling convention

    /Gs - no Buffer overflow protections - saves space and performance. newer saw the sense of enabling that by default. (AutoIT is not viewer-or-player-only-Standard app like a browser, pictureviewer or Audio/Video player where a buffer overflow attack makes sense )

  • Linker Tweaks

    Ever notice that mysterious 'Rich'-signature between the MZ-Dos Header and the PE-COFF Header on all files compiled by the Microsoft incremental Linker? Well after removing the light encryption from that an looking where that data comes from, it get's more harmless than you might though about when you where still in the mystery about what it could be.

    It even get's useless - so I feel better not to have that crap in my release.

    So I patched my linker to keep that 'rich'- thing out + shorten the MZ-Dos header ("This program cannot be run in DOS mode.") a little so that both(MZ+PE-Section header) stays under 0x200 bytes. So the .text section start at 0x200 instead of 0x400.

    Well that 512 byte less is really not worth to talk about- but it's nice to get rid of that 'rich'-crap.

  • Esoteric stuff

    Linker option

    /MERGE:.rdata=.text

    /MERGE:.data=.rdata

    sometimes the one, the other or both makes sence.

    /ALIGN:16 <- newer thought the windows loader will load such PE's. However I had I closer look into the ntldr-internals an if filealign = sectionalign the PE may seen as valid, when if filealign is not aligned to 0x200.

    ^- I don't recommand / like this kind of tweak since files just end up in a unpageable mess of code&data. However it's nice to try it out.

Anyway if someone of the Devs couid make a 'private' build for 32-bit version of AutoItSC.bin with

/MD /Oy /GL /Oi /Os /GF /Gr /Gs

and release it here !

Would but be really nice. :mellow:

Edited by Robinson1

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

msvcrt.dll is not found on every system AutoIt supports natively.

Why would current absence of relocation directory be a good thing?

Edited by trancexx

♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

No shit!? There are compiler options!?

Umm, yeah, we aren't stupid. We know what they are and how they work. Dynamically linking to CRT requires an appropriate version of the CRT runtime that matches the compiler version and processor architecture to be installed. The runtimes do not appear on a virgin installation Thus dynamic linking breaks AutoIt's feature of being stand-alone requiring no additional components to be installed on a virgin Windows system.

Edit: I just saw your edit. Umm... yeah, the forum auto-saves your post and you can restore it later in the event you lose it. Otherwise you should probably compose in a text editor because now I've replied to something that may change entirely.

Edited by Valik

Share this post


Link to post
Share on other sites

If you are worried about small size and performance gains you are using the wrong language.

That preprocessors functionality is already in Obfuscator with SciTE.

Share this post


Link to post
Share on other sites

Anyway if someone of the Devs couid make a 'private' build for 32-bit version of AutoItSC.bin with

/MD /Oy /GL /Oi /Os /GF /Gr /Gs

and release it here !

Would but be really nice. :mellow:

No. I'm going to speak for Jon here as well as myself: We have better things to do with our time than do special builds for insignificant reasons. A compiled AutoIt script (compressed) still easily fits on a .3.5" floppy disk despite the fact that you don't see new systems with floppy drives. New systems sport at least half a terabyte HDDs and USB flash drives with GBs of data are absurdly cheap. There's absolutely no reason to worry about a couple hundred KB.

Share this post


Link to post
Share on other sites

So why is that good? - Guess what it's the *,exe that loads the *.dll and not the other way. So since it comes first in the new process adress space - nearly everything is free. So it can always load at it's preferred address and never need to 'move'/relocated to somewhere else.

But what loads the exe? How do you know if the exe itself is loaded first by the loader in it's own memory space?

Loading procedure implementation of the windows loader may be different depending on system (kernel). Making assumptions about what's done first is wrong.

For the reasons of forward compatibility it's very much preferable that every PE have relocation directory. I dare to say that current behavior is wrong and could/would lead to malfunctioning of compiled scripts in future.


♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites

No. I'm going to speak for Jon here as well as myself: We have better things to do with our time than do special builds for insignificant reasons. A compiled AutoIt script (compressed) still easily fits on a .3.5" floppy disk despite the fact that you don't see new systems with floppy drives. New systems sport at least half a terabyte HDDs and USB flash drives with GBs of data are absurdly cheap. There's absolutely no reason to worry about a couple hundred KB.

I agree, today size isn't that relevant anymore..


Best regards,Emiel Wieldraaijer

Share this post


Link to post
Share on other sites

Valik is one of the few people still around here that will remember when the full AutoIt download would fit on a 1.44 meg floppy. In those days I didn't allow my Dev system to connect to the internet at all so I used to download AutoIt to a floppy on a different box and then use the floppy to put the files on my Dev system.


George

Question about decompiling code? Read the decompiling FAQ and don't bother posting the question in the forums.

Be sure to read and follow the forum rules. -AKA the AutoIt Reading and Comprehension Skills test.***

The PCRE (Regular Expression) ToolKit for AutoIT - (Updated Oct 20, 2011 ver:3.0.1.13) - Please update your current version before filing any bug reports. The installer now includes both 32 and 64 bit versions. No change in version number.

Visit my Blog .. currently not active but it will soon be resplendent with news and views. Also please remove any links you may have to my website. it is soon to be closed and replaced with something else.

"Old age and treachery will always overcome youth and skill!"

Share this post


Link to post
Share on other sites

Hell, I still remember having discussions with Jon when we'd add a kickass new feature that bumped the size up by 20kb or something. Now-a-days I wouldn't be surprised if it took 30 seconds to talk him into using the C++ Standard Library because even though it would add quite a bit to AutoIt that space is irrelevant to my $20 16GB USB flash drive.

Share this post


Link to post
Share on other sites

I even seem to recall bitching commenting about the fact it wouldn't fit on a floppy anymore.

That's progress. My smallest USB sticks are 16G and they will hold several versions of AutoIt at the size it is now. If I had to switch to using my USB drives you couldn't write enough AutoIt core code to fill them up.


George

Question about decompiling code? Read the decompiling FAQ and don't bother posting the question in the forums.

Be sure to read and follow the forum rules. -AKA the AutoIt Reading and Comprehension Skills test.***

The PCRE (Regular Expression) ToolKit for AutoIT - (Updated Oct 20, 2011 ver:3.0.1.13) - Please update your current version before filing any bug reports. The installer now includes both 32 and 64 bit versions. No change in version number.

Visit my Blog .. currently not active but it will soon be resplendent with news and views. Also please remove any links you may have to my website. it is soon to be closed and replaced with something else.

"Old age and treachery will always overcome youth and skill!"

Share this post


Link to post
Share on other sites

Well I understand ya practical point of view.

Of course it's a little bit nerdy to care nowadays about some KB more or less.

Me myself can't really say why I like small and light stuff. May it's some habit inspired by the demoszene.

I got ya point -If I can take Valik as speaker for the dev's- :)

Anyway thanks for the quick reply's - and sorry for bringing up that maybe old topic.

Well I'm not that kind of guy that likes to take part in discussions to fight for opinions, to discuss, justify or convince.

I enjoy more to watch things listen to ppl - to create understanding (for them and myself).

...and when there the time also to speak/teach.

:mellow:

Share this post


Link to post
Share on other sites

AutoIt is light. If you think otherwise then you are insane.

Share this post


Link to post
Share on other sites

You can most definitely take Valik as the speaker for the Devs. In fact often times outspoken speaker for the Devs.


George

Question about decompiling code? Read the decompiling FAQ and don't bother posting the question in the forums.

Be sure to read and follow the forum rules. -AKA the AutoIt Reading and Comprehension Skills test.***

The PCRE (Regular Expression) ToolKit for AutoIT - (Updated Oct 20, 2011 ver:3.0.1.13) - Please update your current version before filing any bug reports. The installer now includes both 32 and 64 bit versions. No change in version number.

Visit my Blog .. currently not active but it will soon be resplendent with news and views. Also please remove any links you may have to my website. it is soon to be closed and replaced with something else.

"Old age and treachery will always overcome youth and skill!"

Share this post


Link to post
Share on other sites

I do not always speak for the developers. I tend to speak for me and me alone.

Share this post


Link to post
Share on other sites

#15 ·  Posted (edited)

Normally you are trying to make some things bigger (at least for men) but it is cool to have an exe which is only e.g. 100kb and it is doing the job.

Imho the size aspect here is not due to any space problem but rather a psychological stuff. Me too, some people are using UPX with highest compression settings to get some kb smaller files but for what?

@Robinson1: are you a reverser?

Br,

UEZ

Edited by UEZ

Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

Share this post


Link to post
Share on other sites

AutoIt is light. If you think otherwise then you are insane.

Compared to what?

How much "heft" does statically linking msvr library actually add to AutoIt?

Share this post


Link to post
Share on other sites

#17 ·  Posted (edited)

Compared to what?

Any equivalent language.

How much "heft" does statically linking msvr library actually add to AutoIt?

For x86, about 100KB or ~22%.

Edit: The Visual Studio 2010 runtime redistributable is 4MB. Given a choice between having a stand-alone application that will run on any version of Windows since Windows 2000 at the cost of having 100KB tacked onto every binary or having to bundle every script with an installer and the redistributable package for the runtime - which as mentioned is 4MB - I think I'll stick with static linking.

Edited by Valik

Share this post


Link to post
Share on other sites

While we are on this topic, may I use this oppertunity to enquire how the Autoit build process works? I intend to do something similar (having base code, then adding info onto it and finalizing the executable)

1. When you made your Autoit interpreter how did you get it to compile into .bin format?

2. How does the Compiler take the .bin and 'make' an executable? I understand there are a number of 'sections' in a PE file but how does this compare to compiling straight to an EXE?

3. How is the script tokens stored? are they added as a resource? what winAPI functions did you use?

4. How does the Interpreter base code find where the tokens are and all the settings, and then understand how to read them and readin the settings into variables?

Thankyou.


ongoing projects:-firestorm: Largescale P2P Social NetworkCompleted Autoit Programs/Scripts: Variable Pickler | Networked Streaming Audio (in pure autoIT) | firenet p2p web messenger | Proxy Checker | Dynamic Execute() Code Generator | P2P UDF | Graph Theory Proof of Concept - Breadth First search

Share this post


Link to post
Share on other sites

I'm going to take a stab at this and say the bin file is actually an executable file that has been renamed. Then the script is broken down into whatever it is broken down into and inserted as a resource.

Share this post


Link to post
Share on other sites

While we are on this topic, may I use this oppertunity to enquire how the Autoit build process works? I intend to do something similar (having base code, then adding info onto it and finalizing the executable)

I know AutoHotkey is quite bad to mention on this forum

anyway look here

http://www.autohotkey.com/download/OtherDownloads.htm

there is "stolen" princip of Aut2Exe also with sources Ahk2Exe

Note: There is also msvcrt related stuff

Reduce the size of compiled scripts by about 20 KB: Place this smaller version of the AutoHotkeySC.bin file in your AutoHotkey\Compiler folder (overwriting the existing file of the same name). Any compiled script produced in this fashion will be dependent on MSVCRT.dll. Although this DLL is always present on Windows 2000/XP or later, older operating systems do not necessarily have it.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0