Compiling for machine code, need help

[wraithdu]

Ok... need tips for something else. I'm trying to move over to Visual Studio Express 2012... but the damn compiler is just not producing usable object code. I had to go back to the sStr[] = {'a', 'b', 'c'}; form to get strings out of the .data / .rdata sections. But now it's producing function relocation tables and relative jumps in the code:

push    edx                                     ; 0343 _ 52
        mov     dword ptr [eax+34H], ebx                ; 0344 _ 89. 58, 34
        call    _CopySections@12                        ; 0347 _ E8, 00000000(rel)
        mov     eax, ebx                                ; 034C _ 8B. C3
        sub     eax, dword ptr [esi+34H]                ; 034E _ 2B. 46, 34
        jz      ?_005                                   ; 0351 _ 74, 07
        push    eax                                     ; 0353 _ 50
        push    edi                                     ; 0354 _ 57
        call    _PerformBaseRelocation@8                ; 0355 _ E8, 00000000(rel)
?_005:  push    edi                                     ; 035A _ 57
        call    _BuildImportTable@4                     ; 035B _ E8, 00000000(rel)
        test    eax, eax                                ; 0360 _ 85. C0
        jz      ?_006                                   ; 0362 _ 74, 2D

I can't find any way to force the compiler not to do that. Any ideas?

[trancexx]

You could use __forceinline keyword to inline functions like CopySections and similar.

Also this can be done by selecting correct optimizations in project properties.

String literals are indeed implementation detail. That's why you get different results using different compilers. Characters array is always safe then. The only thing that should be checked is what's done by the compiler if different alignment is used. But that shouldn't be your concern here.

[wraithdu]

You could use __forceinline keyword to inline functions like CopySections and similar.

Also this can be done by selecting correct optimizations in project properties.

 

I tried changing every optimization setting (even totally disabled) and code generation setting I could think of / find in the project properties, and I couldn't remove the function relocations. The closest I got was turning off Whole Program Optimization and String Pooling.

I'll try out __forceinline tonight. Though I'm sure that will result in larger code? I might just stick with VS as the editor, and compile in Codeblocks / MinGW.

[Shaggi]

Ok... need tips for something else. I'm trying to move over to Visual Studio Express 2012... but the damn compiler is just not producing usable object code. I had to go back to the sStr[] = {'a', 'b', 'c'}; form to get strings out of the .data / .rdata sections. But now it's producing function relocation tables and relative jumps in the code:

push    edx                                     ; 0343 _ 52
        mov     dword ptr [eax+34H], ebx                ; 0344 _ 89. 58, 34
        call    _CopySections@12                        ; 0347 _ E8, 00000000(rel)
        mov     eax, ebx                                ; 034C _ 8B. C3
        sub     eax, dword ptr [esi+34H]                ; 034E _ 2B. 46, 34
        jz      ?_005                                   ; 0351 _ 74, 07
        push    eax                                     ; 0353 _ 50
        push    edi                                     ; 0354 _ 57
        call    _PerformBaseRelocation@8                ; 0355 _ E8, 00000000(rel)
?_005:  push    edi                                     ; 035A _ 57
        call    _BuildImportTable@4                     ; 035B _ E8, 00000000(rel)
        test    eax, eax                                ; 0360 _ 85. C0
        jz      ?_006                                   ; 0362 _ 74, 2D

I can't find any way to force the compiler not to do that. Any ideas?

To get the relative jumps & calls resolved, you can just load the code into an debugger and copy paste.

[trancexx]

I tried changing every optimization setting (even totally disabled) and code generation setting I could think of / find in the project properties, and I couldn't remove the function relocations. The closest I got was turning off Whole Program Optimization and String Pooling.

I'll try out __forceinline tonight. Though I'm sure that will result in larger code? I might just stick with VS as the editor, and compile in Codeblocks / MinGW.

I don't see how would you get larger code. More likely it would be smaller.

Sure obj file would/could be bigger because it would contain inlined function and function itself but considering you use those functions only once the real output should be smaller because there wouldn't be at least stack adjustments function call requires.

Microsoft has the most advanced compiler, I see no reason to choose anything else when the best thing is right there.

The secret is to use correct combination of options. The same code can be compiled to 3 or 100 kB depending on developers and their understanding of the tools they work with.

[Ascend4nt]

wraithdu, have you tried using yasm or nasm as a flat assembler?  You can use the output from objconv, or use the /Fa switch in Visual C++ to create the assembly output to start from.  I'm not sure which would work better here.  Still, if using mingw gets you the code you need, there's no reason to use Visual Studio.

As far as trancexx's assertion that Microsoft has the most advanced compiler, that's quite untrue.  It is in fact the worst C compiler out there (and what you have is C code).  However, as for Windows C++ code it does have quite a measurable advantage over other compilers.  The link time code optimization, as well as the modularization of certain parts of C++ to reduce code size, and compatibility with the Windows SDK, all make for a nice code development platform.

Still, GCC and Clang are the most advanced compilers in the industry, and unlike Microsoft they almost fully comply to all current standards.  Plus the code output is in some cases far superior to Microsoft's (some, not all).

*edit: I should also point out that Pelles C compiler is probably one of the best C11-compilant C compilers for Windows.

Edited July 31, 2013 by Ascend4nt

[wraithdu]

Well compiler quality aside, I got everything working on both platforms. MinGW produces smaller code though, and requires FAR FAR FAR less coercing to output something in assembly I can use.
 
So here are the bugs I found, let's see if I got them all.
 
1) In CopySections he was overwriting Misc.PhysicalAddress in the newly copied headers. He should have been saving that value to the old headers for use in FinalizeSections.
2) In FinalizeSections he should NOT be freeing the IMAGE_SCN_MEM_DISCARDABLE memory. Just ignore that flag.
3) In FinalizeSections the VirtualProtect pointer now needs to be read from the old headers.

4) In MemoryLoadLibrary he was copying too much data for the headers. SizeOfHeaders includes everything.

memcpy(headers, dos_header, dos_header->e_lfanew + old_header->OptionalHeader.SizeOfHeaders);

*5) In BuildImportTable change the for loop

for (; importDesc->FirstThunk && importDesc->Name; importDesc++) 

*6) In Visual Studio, ASLR must be ENABLED.

[jchd]

Intel's compiler used to have top reputation but only for Intel targets.

[Ascend4nt]

Intel's compiler used to have top reputation but only for Intel targets.

jchd, this is true, I've heard great things about Intel's compiler, although its one I haven't tested myself..

Additionally, I used to enjoy Borland C++ compilers back in the day, but the technology is now owned by Embarcadero (odd name?).  I had actually given their Embarcadero XE 3 compiler a shot, but was quite appalled at their decision to use two separate compiler technologies - one based on old Borland technology, and one based on Clang.  It was completely insane to me that you had to write code differently depending on whether you wanted to target 32-bit or 64-bit code. Clang is fully C++11 compliant now, but the 32-bit Borland compiler supports maybe 3 different C++11 features, and has additional language extensions that aren't supported by the 64-bit compiler.. absolutely ridiculous.  That is just a sh*t thing to do to customers who expect one solid development framework which is touted as being C++11 ready.

I've tried other compilers, but most of it is freeware stuff.. lcc-win32 is surprisingly good for a C compiler but also surprisingly buggy.. Tiny C produces the most crap code ever, but creates the smallest executables also... Digital Mars compilers choke on so many windows things it makes it difficult to compile anything..

[trancexx]

Well compiler quality aside, I got everything working on both platforms. MinGW produces smaller code though, and requires FAR FAR FAR less coercing to output something in assembly I can use.

So here are the bugs I found, let's see if I got them all.

1) In CopySections he was overwriting Misc.PhysicalAddress in the newly copied headers. He should have been saving that value to the old headers for use in FinalizeSections.

2) In FinalizeSections he should NOT be freeing the IMAGE_SCN_MEM_DISCARDABLE memory. Just ignore that flag.

3) In FinalizeSections the VirtualProtect pointer now needs to be read from the old headers.

4) In MemoryLoadLibrary he was copying too much data for the headers. SizeOfHeaders includes everything.

memcpy(headers, dos_header, dos_header->e_lfanew + old_header->OptionalHeader.SizeOfHeaders);

*5) In BuildImportTable change the for loop

for (; importDesc->FirstThunk && importDesc->Name; importDesc++)

*6) In Visual Studio, ASLR must be ENABLED.

In CopySections section->Misc is union. For PE files VirtualSize member of that union is used. Interpretting it as PhysicalAddress could directly cause crash. My advice is just to comment out the two lines where new values are set, nothing more.

Number 4 is correct but it makes no difference because all the same he's overwritting superfluous data later.

5 is correct because PE documentation says that imports will end up with zero-ed chunk, much like null-terminated strings end up with null character. Still checking both FirstThunk and Name fields is maybe a bit overchecking.

You missed one more thing but probably everything works rergardless then.

Still, I know a person who knows a person who knows a person and some more who knows someone from "Un4seen Developments" and I was given access to some parts of the code there. Going through it I saw that they use some sort of check to see whether their dlls were being touched in indecent manner and forbid execution if that's the case. Jon does something similar with AutoIt where he doesn't allow debuggers attached. It's BS really but whatever.

Anyway un4seens check OptionalHeader.ImageBase and if it's changed they will abandon loading of the module and even display message saying that something is wrong.

I'm not sure why you don't see that thing and going throgh the code (and I will repeat again, in my head only because I have no decent Windows computers here) the only way is that dll is managed to be loaded at OptionalHeader.ImageBase address. That means initial allocation at OptionalHeader.ImageBase address succeeds and "code" isn't NULL (arbitrary position allocation comment).

However that may not be the case with other dlls, or even this one, ever or just next time. So check what happens if you comment out :

code = (unsigned char *)virtualAlloc((LPVOID)(old_header->OptionalHeader.ImageBase),old_header->OptionalHeader.SizeOfImage,MEM_RESERVE | MEM_COMMIT,PAGE_READWRITE);

...and set "code" to NULL instead so that allocation happens at another address.

What is OptionalHeader.ImageBase and why it's used one may ask. That data is used by PE loader primarly to resolve relocations at load time. It will substract the real addres of allocated memory space for the module and OptionalHeader.ImageBase value and use the calculated difference to adjust values inside relocation directory. If the difference is 0 it will do nothing, of course.

So the question is why to change OptionalHeader.ImageBase value? You see, you are manually doing relocations, nobody else does nothing. I would comment out the line where OptionalHeader.ImageBase is changed and not allow possible failures due to internal checks done by nervous developers.

Edited August 1, 2013 by trancexx

[trancexx]

Hehe, I like a challenge, don't mind her teaching method. If I get really stuck, I'll ask for help.

Thanks for this.

Lots of people around see me as arrogant or whatever because the things I say they interpret as me "teaching" them. And who am I to teach them? This is because they constantly compare themselves to me and really have troubles accepting me because of their inferiority. I can imagine how much that hurts their ego and their feelings. Much of them were here before me or became members around the time I did. Many can't accept fact that there are individuals who learn faster, who understand concepts better, hell - who can become developers of the language before they even began to grasp advanced elements of the language.

However I have no intention of apologizing for knowing more before I say something they don't know about or can't understand just yet. It doesn't make sense to say "I'm sorry that I know better, please don't be offended by what I will say because you don't expect me to know things because you don't know them and you were around here more than I was". That's retarded. And politically correct too, lol.

[wraithdu]

In CopySections section->Misc is union. For PE files VirtualSize member of that union is used. Interpretting it as PhysicalAddress could directly cause crash. My advice is just to comment out the two lines where new values are set, nothing more.

What's actually going on there, is he's saving the new address of the copied section for later use in the FinalizeSections function, so he knows what to VirtualProtect. The problem is, he is overwriting the newly copied data and corrupting it. I think he got confused which copy of the headers he was reading / writing, and really meant to save that value to the old header copy.

 

You missed one more thing but probably everything works rergardless then.

Care to clue me in? At least a hint maybe?

 

Anyway un4seens check OptionalHeader.ImageBase and if it's changed they will abandon loading of the module and even display message saying that something is wrong.

I'm not sure why you don't see that thing and going throgh the code

Ha! I did see that! Big scary message about tampering and viruses. I thought it was coming from AutoIt actually.

 

(and I will repeat again, in my head only because I have no decent Windows computers here) the only way is that dll is managed to be loaded at OptionalHeader.ImageBase address. That means initial allocation at OptionalHeader.ImageBase address succeeds and "code" isn't NULL (arbitrary position allocation comment).

However that may not be the case with other dlls, or even this one, ever or just next time. So check what happens if you comment out :

code = (unsigned char *)virtualAlloc((LPVOID)(old_header->OptionalHeader.ImageBase),old_header->OptionalHeader.SizeOfImage,MEM_RESERVE | MEM_COMMIT,PAGE_READWRITE);

...and set "code" to NULL instead so that allocation happens at another address.

 

Interesting...

 

So the question is why to change OptionalHeader.ImageBase value? You see, you are manually doing relocations, nobody else does nothing. I would comment out the line where OptionalHeader.ImageBase is changed and not allow possible failures due to internal checks done by nervous developers.

So you're saying... do the relocation if necessary, but don't update OptionalHeader.ImageBase?

 

Thanks for this.

I like to learn. It comes with the territory sometimes I know it's not personal. Some people just can't separate that.

[wraithdu]

So I tried what you suggested... I forced the memory allocation to happen at an arbitrary address and I got the error message. However if I did NOT update ImageBase with the new location, then the DLL loaded fine. Very interesting. So I'll keep the fallback arbitrary memory allocation, will NOT update ImageBase in the new headers, and make sure nothing else in the code tries to use the ImageBase value (instead use codeBase, which is the same).

Edited August 1, 2013 by wraithdu

[wraithdu]

Still, GCC and Clang are the most advanced compilers in the industry, and unlike Microsoft they almost fully comply to all current standards.  Plus the code output is in some cases far superior to Microsoft's (some, not all).

 

Bah, you're comment made me curious about Clang. I just spent the better part of a day building v3.4 from source, cause Harry Denholm's binaries ( http://www.ishani.org/web/articles/code/clang-win32/ ) still have some compilation bug that was fixed in a later revision. An aside, ClangVSx is pretty sweet.

So I finally got it built, after several Visual Studio compiler crashes and one BSOD. Don't ask how building something crashed the compiler and caused a BSOD on WIndows 8, cause I have no idea. But it ended up compiling a nicely working DLL. But my lack of familiarity with it, I couldn't resolve the old .data sections problems so gave it up for the assembly stuff. Additionally I couldn't get a 64-bit version to compile. That all said, it was a neat exercise at the very least.

[Ascend4nt]

Finding a decent Clang port for Windows is definitely an issue. I'm surprised you got anything working at all to be honest.  One of the issues, besides compatibility with the Windows SDK, is the reliance on PThreads for the C++11 stuff.  That last thing has been an issue with G++ actually, too.

With ClangVSx I've gotten mixed results..  the Microsoft C++ headers are such a mess of compiler-specific hacks and macros that I'm surprised any external tools would be able to use them and work correctly.  And unsurprisingly, much of my stuff wouldn't compile because of that.  However, from what I've read about the Visual C++ 2013 compiler, its headers are much better now since the compiler natively supports things like initializer_list's and variadic templates..   I'll have to consider trying the preview out..

Ah, and I hadn't looked at Harry Denholm's stuff.. I might take a look now, although you've got me a bit worried about the pain and frustration of creating a working build!

*edit: I had looked, but just thought of that guy as 'ishani' haha

Edited August 2, 2013 by Ascend4nt

[wraithdu]

After getting BASSMOD to load, I went back and tried another DLL I had issues with, the last v5.8.8 Eraser.dll from the Heidi Eraser project - http://sourceforge.net/projects/eraser/files/Eraser%205/5.8.8/Eraser%205.8.8%20Portable.zip/download

This one is still crashing. I'm using Ward's full AutoIt implementation for testing. Any ideas, maybe another bug we missed? Or is this one of those that just isn't going to work? The DLL is in the linked zip file.

I'm going to try trancexx's subrogation UDF shortly, and report back on the results.

Edited August 5, 2013 by wraithdu

[trancexx]

Maybe something blew up . I would feel terrible knowing my UDF terminated his poor soul :cry:

wraithdu don't go into the light!

[wraithdu]

Ha! Got sidetracked... no long bright tunnels for this guy.

[wraithdu]

Of course the subrogation UDF works Loaded OK, made a few function calls. Sigh, computers are hard.

[trancexx]

Of course the subrogation UDF works Loaded OK, made a few function calls. Sigh, computers are hard.

This may seem unrelated but could you run help file example for _Security__ImpersonateSelf and post what you get printed to console together with bacis info about your system.