29 replies to this topic

[Robinson1]

Recently I was playing around a little with the Visual C++ Compiler and saw that you can do much off from the standard to make the compiled code smarter & more compacted.
...
And beside that really nice Preprocessor I just discover by now:
http://www.autoitscript.com/forum/index.php?showtopic=30573&hl=cleanscript

It also happened that I had a look upon AutoItSC.bin. (The script interpreter stub that's attached to every compiled AutoIt-Exe.)

• Wow nice no '.reloc' so I see you already 'found' the /DYNAMICBASE:NO' linker switch.
  
  
  Just because I read the first reply:
  So why is that good? - Guess what it's the *,exe that loads the *.dll and not the other way. So since it comes first in the new process adress space - nearly everything is free. So it can always load at it's preferred address and never need to 'move'/relocated to somewhere else.
• However it happened that I loaded AutoItSC.bin into in IDA, And saw that the Common RunTime's are statically linked:^- That's the whole adress space of AutoItSC.bin, the cyan area's are statically linked library functions.
  
  
  
  
  By using the Compiler switch /MD these areas will stay out of the AutoItSC.bin and load from msvcrt.dll that there in the system32 dir on ever windows installation.
  That might shrink the size something about 25%.
  
  Maybe I'm really not really telling any news about that /MultithreadDll switch. And in detail things are not that perfect. The performance 'loss' between calling a library function or a function to a loaded Dll is in my eyes minimal.
  The main reason is that depending on what VC you're using that MSVCRT.dll is sometimes names MSVCR70.dll, MSVCR71.dll, or with the 'brand new' VC++ 2011 MSVCR100.dll. and I'll need to include these *.dll since you can't really assume that they are already in the system.
  Well I for the other part of the reply I just checked it msvcrt.dll is even present in Win98 (Setupfile WIN98_57.CAB) Windows itself is written in C++ and will make itself use of it - like for ex. calc.exe or ftp.exe does.)
  
  Maybe you get it compiled with the old'n gold'n includes from VC6:
  http://www.sendspace.com/file/gaw39k
  that really includes MSVCRT.dll and not that new crap from the MSVCRXX-version-zoo where I can't see what better whem using. Something it need undocumented under the hood gribs to like -d2QIftol to say to the actual compiler no I don't want my FloatToLong convertions with SSE. Or adjusting stdio.h
  #ifndef _STDIO_DEFINED
  _CRTIMP extern FILE _iob[];
  #endif /* _STDIO_DEFINED */
  ...
  #define stdin (&_iob[0])
  #define stdout (&_iob[1])
  #define stderr (&_iob[2])
  so I get my 'stdin' handle directly via '_iob' array and not that fucking __iob_func that is not present in MSVCRT.dll
  
  ...okay next
• Framepointers
  
  
  As C-Coder how landed from time to time in the ASM-Debug on nasty bug's you're probably know this:
  PUSH EBP
  MOV EBP, ESP
  SUB ESP, 10
  
  [EBP+8] ->Arg0
  [EBP+C] ->Arg1
  ...
  [EBP-4] ->LocalVar0
  [EBP-8] ->LocalVar1
  ...
  LEAVE
  RETN
  These framepointer based functions are somehow nice for debugging (and Reverse engineering) easier and quicker compiling, but for compiling optimized release code it's better to remove them with the /Oy compiler switch. That saves some space and bring minimalistic performance gains.
• More Optimizations options
  
  
  I just name the in short without going to much in detail. Use the documentation at first place or ask for more details
  /GL - Full optimization
  /Oi - use Compiler build in asm functions like ROL instead of 'importing' them
  
  /Os - small code - no alignment gaps between functions.
  /GF - String pooling store identical strings only once
  
  /Gr - try to use __fastcall calling convention
  /Gs - no Buffer overflow protections - saves space and performance. newer saw the sense of enabling that by default. (AutoIT is not viewer-or-player-only-Standard app like a browser, pictureviewer or Audio/Video player where a buffer overflow attack makes sense )
• 
  Linker Tweaks
  Ever notice that mysterious 'Rich'-signature between the MZ-Dos Header and the PE-COFF Header on all files compiled by the Microsoft incremental Linker? Well after removing the light encryption from that an looking where that data comes from, it get's more harmless than you might though about when you where still in the mystery about what it could be.
  It even get's useless - so I feel better not to have that crap in my release.
  So I patched my linker to keep that 'rich'- thing out + shorten the MZ-Dos header ("This program cannot be run in DOS mode.") a little so that both(MZ+PE-Section header) stays under 0x200 bytes. So the .text section start at 0x200 instead of 0x400.
  
  Well that 512 byte less is really not worth to talk about- but it's nice to get rid of that 'rich'-crap.
• Esoteric stuff
  
  Linker option
  /MERGE:.rdata=.text
  /MERGE:.data=.rdata
  sometimes the one, the other or both makes sence.
  
  /ALIGN:16 <- newer thought the windows loader will load such PE's. However I had I closer look into the ntldr-internals an if filealign = sectionalign the PE may seen as valid, when if filealign is not aligned to 0x200.
  ^- I don't recommand / like this kind of tweak since files just end up in a unpageable mess of code&data. However it's nice to try it out.

Anyway if someone of the Devs couid make a 'private' build for 32-bit version of AutoItSC.bin with
/MD /Oy /GL /Oi /Os /GF /Gr /Gs
and release it here !

Would but be really nice.

Edited by Robinson1, 08 September 2011 - 09:57 PM.

[trancexx]

msvcrt.dll is not found on every system AutoIt supports natively.

Why would current absence of relocation directory be a good thing?

Edited by trancexx, 08 September 2011 - 05:25 PM.

[Valik]

No shit!? There are compiler options!?

Umm, yeah, we aren't stupid. We know what they are and how they work. Dynamically linking to CRT requires an appropriate version of the CRT runtime that matches the compiler version and processor architecture to be installed. The runtimes do not appear on a virgin installation Thus dynamic linking breaks AutoIt's feature of being stand-alone requiring no additional components to be installed on a virgin Windows system.

Edit: I just saw your edit. Umm... yeah, the forum auto-saves your post and you can restore it later in the event you lose it. Otherwise you should probably compose in a text editor because now I've replied to something that may change entirely.

Edited by Valik, 08 September 2011 - 06:51 PM.

[Mat]

If you are worried about small size and performance gains you are using the wrong language.

That preprocessors functionality is already in Obfuscator with SciTE.

[Valik]

Anyway if someone of the Devs couid make a 'private' build for 32-bit version of AutoItSC.bin with
/MD /Oy /GL /Oi /Os /GF /Gr /Gs
and release it here !

Would but be really nice.

No. I'm going to speak for Jon here as well as myself: We have better things to do with our time than do special builds for insignificant reasons. A compiled AutoIt script (compressed) still easily fits on a .3.5" floppy disk despite the fact that you don't see new systems with floppy drives. New systems sport at least half a terabyte HDDs and USB flash drives with GBs of data are absurdly cheap. There's absolutely no reason to worry about a couple hundred KB.

[trancexx]

So why is that good? - Guess what it's the *,exe that loads the *.dll and not the other way. So since it comes first in the new process adress space - nearly everything is free. So it can always load at it's preferred address and never need to 'move'/relocated to somewhere else.

But what loads the exe? How do you know if the exe itself is loaded first by the loader in it's own memory space?
Loading procedure implementation of the windows loader may be different depending on system (kernel). Making assumptions about what's done first is wrong.
For the reasons of forward compatibility it's very much preferable that every PE have relocation directory. I dare to say that current behavior is wrong and could/would lead to malfunctioning of compiled scripts in future.

[Emiel Wieldraaijer]

No. I'm going to speak for Jon here as well as myself: We have better things to do with our time than do special builds for insignificant reasons. A compiled AutoIt script (compressed) still easily fits on a .3.5" floppy disk despite the fact that you don't see new systems with floppy drives. New systems sport at least half a terabyte HDDs and USB flash drives with GBs of data are absurdly cheap. There's absolutely no reason to worry about a couple hundred KB.

I agree, today size isn't that relevant anymore..

[GEOSoft]

Valik is one of the few people still around here that will remember when the full AutoIt download would fit on a 1.44 meg floppy. In those days I didn't allow my Dev system to connect to the internet at all so I used to download AutoIt to a floppy on a different box and then use the floppy to put the files on my Dev system.

[Valik]

Hell, I still remember having discussions with Jon when we'd add a kickass new feature that bumped the size up by 20kb or something. Now-a-days I wouldn't be surprised if it took 30 seconds to talk him into using the C++ Standard Library because even though it would add quite a bit to AutoIt that space is irrelevant to my $20 16GB USB flash drive.

[GEOSoft]

I even seem to recall bitching commenting about the fact it wouldn't fit on a floppy anymore.
That's progress. My smallest USB sticks are 16G and they will hold several versions of AutoIt at the size it is now. If I had to switch to using my USB drives you couldn't write enough AutoIt core code to fill them up.

[Robinson1]

Well I understand ya practical point of view.
Of course it's a little bit nerdy to care nowadays about some KB more or less.
Me myself can't really say why I like small and light stuff. May it's some habit inspired by the demoszene.
I got ya point -If I can take Valik as speaker for the dev's-

Anyway thanks for the quick reply's - and sorry for bringing up that maybe old topic.

Well I'm not that kind of guy that likes to take part in discussions to fight for opinions, to discuss, justify or convince.
I enjoy more to watch things listen to ppl - to create understanding (for them and myself).
...and when there the time also to speak/teach.

[Valik]

AutoIt is light. If you think otherwise then you are insane.

[GEOSoft]

You can most definitely take Valik as the speaker for the Devs. In fact often times outspoken speaker for the Devs.

[Valik]

I do not always speak for the developers. I tend to speak for me and me alone.

[UEZ]

Normally you are trying to make some things bigger (at least for men) but it is cool to have an exe which is only e.g. 100kb and it is doing the job.

Imho the size aspect here is not due to any space problem but rather a psychological stuff. Me too, some people are using UPX with highest compression settings to get some kb smaller files but for what?


@Robinson1: are you a reverser?

Br,
UEZ

Edited by UEZ, 09 September 2011 - 10:36 PM.

[WinkleDoodle]

AutoIt is light. If you think otherwise then you are insane.

Compared to what?


How much "heft" does statically linking msvr library actually add to AutoIt?

[Valik]

Compared to what?

Any equivalent language.

How much "heft" does statically linking msvr library actually add to AutoIt?

For x86, about 100KB or ~22%.

Edit: The Visual Studio 2010 runtime redistributable is 4MB. Given a choice between having a stand-alone application that will run on any version of Windows since Windows 2000 at the cost of having 100KB tacked onto every binary or having to bundle every script with an installer and the redistributable package for the runtime - which as mentioned is 4MB - I think I'll stick with static linking.

Edited by Valik, 09 September 2011 - 09:50 PM.

[twitchyliquid64]

While we are on this topic, may I use this oppertunity to enquire how the Autoit build process works? I intend to do something similar (having base code, then adding info onto it and finalizing the executable)

1. When you made your Autoit interpreter how did you get it to compile into .bin format?
2. How does the Compiler take the .bin and 'make' an executable? I understand there are a number of 'sections' in a PE file but how does this compare to compiling straight to an EXE?
3. How is the script tokens stored? are they added as a resource? what winAPI functions did you use?
4. How does the Interpreter base code find where the tokens are and all the settings, and then understand how to read them and readin the settings into variables?

Thankyou.

[Richard Robertson]

I'm going to take a stab at this and say the bin file is actually an executable file that has been renamed. Then the script is broken down into whatever it is broken down into and inserted as a resource.

[Zedna]

While we are on this topic, may I use this oppertunity to enquire how the Autoit build process works? I intend to do something similar (having base code, then adding info onto it and finalizing the executable)

I know AutoHotkey is quite bad to mention on this forum
anyway look here
http://www.autohotkey.com/download/OtherDownloads.htm
there is "stolen" princip of Aut2Exe also with sources Ahk2Exe

Note: There is also msvcrt related stuff

Reduce the size of compiled scripts by about 20 KB: Place this smaller version of the AutoHotkeySC.bin file in your AutoHotkey\Compiler folder (overwriting the existing file of the same name). Any compiled script produced in this fashion will be dependent on MSVCRT.dll. Although this DLL is always present on Windows 2000/XP or later, older operating systems do not necessarily have it.