PE File Overlay Extraction

[Ascend4nt 117]

PE File Overlay Extraction

(and Certificate info)

---

Executable and other PE files can contain "overlays", which is data that is appended to the end of the file.  This data can be important, such as setup packages, Authenticode signatures*, and overlays for AutoIt scripts. Or it could just be extra unneeded cruft (sometimes).  Whatever the case is, I wanted to find a way to detect if this data was present.

This project is actually a result of dealing with so-called 'File Optimizer' programs that would strip Overlay information from Executables (leaving compiled AutoIt scripts crippled!). And also a legit >answer to my topic in Help and Support.

While future versions of AutoIt (new beta releases and any official release after v3.3.8.1) are putting tokenized scripts into a resource within the executable, all current compiled scripts are still put together with the tokenized script appended as an overlay.

The UDF here allows you to detect any overlay a PE (Portable Executable) file may have, and allows you to extract the Overlay into a separate file - or alternatively extract the exe without the overlay.  You can actually extract AutoIt scripts and write them to .A3X files using this method, if you so desire.  But don't be a hacker! Mommy will scold you..

If you separate both the exe and overlay, you can combine them again using a simple file-append, something like:

copy /b stripped.exe+script.a3x myscript.exe

_

Anyway, the method to detect overlays is relatively simple - we need to look through the PE file's various headers and find out where the last section of data/code is and its size.  If that last section doesn't reach the end of the file, then you will find an Overlay waiting at the end of the final section. However, there's an issue with Certificate Tables (or signatures) which makes it a bit more tricky to detect - basically the end of the last section and the beginning of the Certificate must be examined to find the sandwiched-in overlay.

*Auhenticode signatures note: These and other certificates are actually linked to in the PE Data Directory, which I had missed in earlier versions.  Now they are accounted for however, and not considered overlays nor are they allowed to be extracted (well, you could extract them but the signature is bound to the unique checksum of the file and needs to be referenced from the Data Directory).

IMPORTANT: The example now queries which part to save, and "No" button means 'yes' to Exe extract. (I didn't want to mess around with creating dialog windows, sorry).

So, here's the UDF with a working example (note the 128MB limit can easily be worked around):

expandcollapse popup

; ========================================================================================================
; <FilePEOverlayExtract.au3>
;
; UDF and Example of getting Overlay info and optionally extracting that info to a file.
;
; NOTE that this isn't intended to be used to hack or decompile AutoIt executables!!
;  It's main purpose is to find Overlays and Certificates and extract/save or just report the info
;
; Functions:
;    _PEFileGetOverlayInfo()    ; Returns a file offset for overlay data (if found), and the size
;
; Author: Ascend4nt
; ========================================================================================================

; Arry indexing
Global Enum $PEI_OVL_START = 0, $PEI_OVL_SIZE, $PEI_CERT_START, $PEI_CERT_SIZE, $PEI_FILE_SIZE

; ---------------------- MAIN CODE -------------------------------

Local $sFile, $sLastDir, $sLastFile, $aOverlayInfo
$sLastDir = @ScriptDir
While 1
    $sFile=FileOpenDialog("Select PE File To Find Overlay Data In",$sLastDir,"PE Files (*.exe;*.dll;*.drv;*.scr;*.cpl;*.sys;*.ocx;*.tlb;*.olb)|All Files (*.*)",3,$sLastFile)
    If @error Or $sFile="" Then Exit
    $sLastFile=StringMid($sFile,StringInStr($sFile,'\',1,-1)+1)
    $sLastDir=StringLeft($sFile,StringInStr($sFile,'\',1,-1)-1)

    $aOverlayInfo = _PEFileGetOverlayInfo($sFile)

    If $aOverlayInfo[$PEI_OVL_START] = 0 Then
        ConsoleWrite("Failed Return from _PEGetOverlayOffset(), @error = " & @error & ", @extended = " & @extended & @CRLF)
        MsgBox(64, "No Overlay Found", "No overlay found in " & $sLastFile)
        ContinueLoop
    EndIf
    ConsoleWrite("Return from _PEFileGetOverlayInfo() = " & $aOverlayInfo[$PEI_OVL_START] & ", @extended = " & $aOverlayInfo[$PEI_OVL_SIZE] & @CRLF)

    If $aOverlayInfo[$PEI_OVL_START] Then
        Local $hFileIn = -1, $hFileOut = -1, $sOutFile, $iMsgBox, $bBuffer, $bSuccess = 0
        $iMsgBox = MsgBox(35, "Overlay found in " & $sLastFile, "Overlay Found. File size: " & $aOverlayInfo[$PEI_FILE_SIZE] & ", Overlay size: " & $aOverlayInfo[$PEI_OVL_SIZE] & @CRLF & @CRLF & _
            "Would you like to:" & @CRLF & _
            "[Yes]: extract and save Overlay" & @CRLF & _
            "[No]: extract Exe without Overlay" & @CRLF & _
            "[Cancel]: Do Nothing")

        If $iMsgBox = 6 Then
            If $aOverlayInfo[$PEI_OVL_SIZE] > 134217728 Then
                MsgBox(48, "Overlay is too huge", "Overlay is > 128MB, skipping..")
                ContinueLoop
            EndIf
            $sOutFile = FileSaveDialog("Overlay - SAVE: Choose a file to write Overlay data to (from " & $sLastFile&")", $sLastDir, "All (*.*)", 2 + 16)
            If Not @error Then

                While 1
                    $hFileOut = FileOpen($sOutFile, 16 + 2)
                    If $hFileOut = -1 Then ExitLoop
                    $hFileIn = FileOpen($sFile, 16)
                    If $hFileIn = -1 Then ExitLoop
                    If Not FileSetPos($hFileIn, $aOverlayInfo[$PEI_OVL_START], 0) Then ExitLoop
                    ; AutoIt 2/3 Signature check requires 32 bytes min.
                    If $aOverlayInfo[$PEI_FILE_SIZE] > 32 Then
                        $bBuffer = FileRead($hFileIn, 32)
                        If @error Then ExitLoop
                        ; AutoIt2 & AutoIt3 signatures
                        If BinaryMid($bBuffer, 1, 16) = "0xA3484BBE986C4AA9994C530A86D6487D" Or _
                            BinaryMid($bBuffer, 1 + 16, 4) = "0x41553321" Then    ; "AU3!"
                            ConsoleWrite("AutoIt overlay file found" & @CRLF)
                        EndIf
                        FileWrite($hFileOut, $bBuffer)
                        ; subtract amount we read in above
                        $bSuccess = FileWrite($hFileOut, FileRead($hFileIn, $aOverlayInfo[$PEI_OVL_SIZE] - 32))
                    Else
                        $bSuccess = FileWrite($hFileOut, FileRead($hFileIn, $aOverlayInfo[$PEI_OVL_SIZE]))
                    EndIf
                    ExitLoop
                WEnd
                If $hFileOut <> -1 Then FileClose($hFileOut)
                If $hFileIn <> -1 Then FileClose($hFileIn)

            EndIf
        ElseIf $iMsgBox = 7 Then
            If $aOverlayInfo[$PEI_FILE_SIZE] - $aOverlayInfo[$PEI_OVL_SIZE] > 134217728 Then
                MsgBox(48, "EXE is too huge", "EXE (minus overlay) is > 128MB, skipping..")
                ContinueLoop
            EndIf
            $sOutFile = FileSaveDialog("EXE {STRIPPED} - SAVE: Choose a file to write EXE (minus Overlay) to. (from " & $sLastFile&")", $sLastDir, "All (*.*)", 2 + 16)
            If Not @error Then
                $bSuccess = FileWrite($sOutFile, FileRead($sFile, $aOverlayInfo[$PEI_OVL_START]))
            EndIf
        Else
            ContinueLoop
        EndIf
        If $bSuccess Then
            ShellExecute(StringLeft($sOutFile,StringInStr($sOutFile,'\',1,-1)-1))
        Else
            MsgBox(64, "Error Opening or writing to file", "Error opening, reading or writing overlay info")
        EndIf
    EndIf
WEnd
Exit


; ------------------------  UDF Function ----------------------------


; ===================================================================================================================
; Func _PEFileGetOverlayInfo($sPEFile)
;
; Returns information on Overlays present in a Windows PE file (.EXE, .DLL etc files), as well as Certificate Info.
;
; Only certain executables contain Overlays, and these are always located after the last PE Section,
; and most times before any Certificate info. Setup/install programs typically package their data in Overlays,
; and AutoIt compiled executables (at least up to v3.3.8.1) contain an overlay in .A3X tokenized format.
;
; Certificate info is available with or without an overlay, and comes after the last section and typically after
; an Overlay. Certificates are included with signed executables (such as Authenticode-signed)
;
; The returned info can be used to examine or extract the Overlay or Certificate, or just to examine the data
; (for example, to see if its an AutoIt tokenized script).
;
; NOTE: Any Overlays packaged into Certificate blocks are ignored, and the methods to extract this info may
; fail if the Certificate Table entries have their sizes modified to include the embedded Overlay.
;
; The returned information can be useful in preventing executable 'optimizers' from stripping the Overlay info,
;  which was the primary intent in creating this UDF.
;
;
; Returns:
;  Success: A 5-element array, @error = 0
;    [0] = Overlay Start (if any)
;    [1] = Overlay Size
;    [2] = Certificate Start (if any)
;    [3] = Certificate Size
;    [4] = File Size
;
;  Failure: Same 5-element array as above (with all 0's), and @error set:
;    @error = -1 = Could not open file
;    @error = -2 = FileRead error (most likely an invalid PE file). @extended = FileRead() @error
;    @error = -3 = FileSetPos error (most likely an invalid PE file)
;    @error =  1 = File does not exist
;    @error =  2 = 'MZ' signature could not be found (not a PE file)
;    @error =  3 = 'PE' signature could not be found (not a PE file)
;    @error =  4 = 'Magic' number not recognized (not PE32, PE32+, could be 'ROM (0x107), or unk.) @extended=number
;
; Author: Ascend4nt
; ===================================================================================================================

Func _PEFileGetOverlayInfo($sPEFile)
;~     If Not FileExists($sPEFile) Then Return SetError(1,0,0)
    Local $hFile, $nFileSize, $bBuffer, $iOffset, $iErr, $iExit, $aRet[5] = [0, 0, 0, 0]
    Local $nTemp, $nSections, $nDataDirectories, $nLastSectionOffset, $nLastSectionSz
    Local $iSucces=0, $iCertificateAddress = 0, $nCertificateSz = 0, $stEndian = DllStructCreate("int")

    $nFileSize = FileGetSize($sPEFile)
    $hFile = FileOpen($sPEFile, 16)
    If $hFile = -1 Then Return SetError(-1,0,$aRet)

    ; A once-only loop helps where "goto's" would be helpful
    Do
        ; We keep different exit codes for different operations in case of failure (easier to track down what failed)
        ;    The function can be altered to remove these assignments of course
        $iExit = -2
        $bBuffer = FileRead($hFile, 2)
        If @error Then ExitLoop

        $iExit = 2
;~     'MZ' in hex (endian-swapped):
        If $bBuffer <> 0x5A4D Then ExitLoop
        ;ConsoleWrite("MZ Signature found:"&BinaryToString($bBuffer)&@CRLF)

        $iExit = -3
;~     Move to Windows PE Signature Offset location
        If Not FileSetPos($hFile, 0x3C, 0) Then ExitLoop

        $iExit = -2
        $bBuffer = FileRead($hFile, 4)
        If @error Then ExitLoop

        $iOffset = Number($bBuffer)    ; Though the data is in little-endian, because its a binary variant, the conversion works
         ;ConsoleWrite("Offset to Windows PE Header="&$iOffset&@CRLF)

        $iExit = -3
;~     Move to Windows PE Header Offset
        If Not FileSetPos($hFile, $iOffset, 0) Then ExitLoop

        $iExit = -2
;~     Read in IMAGE_FILE_HEADER + Magic Number
        $bBuffer = FileRead($hFile, 26)
        If @error Then ExitLoop

        $iExit = 3
        ; "PE/0/0" in hex (endian swapped)
        If BinaryMid($bBuffer, 1, 4) <> 0x00004550 Then ExitLoop

        ; Get NumberOfSections (need to use endian conversion)
        DllStructSetData($stEndian, 1, BinaryMid($bBuffer, 6 + 1, 2))
        $nSections = DllStructGetData($stEndian, 1)
        ; Sanity check
        If $nSections * 40 > $nFileSize Then ExitLoop
;~         ConsoleWrite("# of Sections: " & $nSections & @CRLF)

        $bBuffer = BinaryMid($bBuffer, 24 + 1, 2)

        ; Magic Number check (0x10B = PE32, 0x107 = ROM image, 0x20B = PE32+ (x64)
        If $bBuffer = 0x10B Then
            ; Adjust offset to where "NumberOfRvaAndSizes" is on PE32 (offset from IMAGE_FILE_HEADER)
            $iOffset += 116
        ElseIf $bBuffer = 0x20B Then
            ; Adjust offset to where "NumberOfRvaAndSizes" is on PE32+ (offset from IMAGE_FILE_HEADER)
            $iOffset += 132
        Else
            $iExit = 4
            SetError(Number($bBuffer))        ; Set the error (picked up below and set in @extended) to the unrecognized Number found
            ExitLoop
        EndIf

;~     'Optional' Header Windows-Specific fields

        $iExit = -3
;~     -> Move to "NumberOfRvaAndSizes" at the end of IMAGE_OPTIONAL_HEADER
        If Not FileSetPos($hFile, $iOffset, 0) Then ExitLoop

        $iExit = -2
;~     Read in NumberOfRvaAndSizes
        $nDataDirectories = Number(FileRead($hFile, 4))
        ; Sanity and error check
        If $nDataDirectories <= 0 Or $nDataDirectories > 16 Then ExitLoop

;~         ConsoleWrite("# of IMAGE_DATA_DIRECTORY's: " & $nDataDirectories & @CRLF)

;~     Read in IMAGE_DATA_DIRECTORY's (also moves file position to IMAGE_SECTION_HEADER)
        $bBuffer = FileRead($hFile, $nDataDirectories * 8)
        If @error Then ExitLoop

;~     IMAGE_DIRECTORY_ENTRY_SECURITY entry is special - it's "VirtualAddress" is actually a file offset
        If $nDataDirectories >= 5 Then
            DllStructSetData($stEndian, 1, BinaryMid($bBuffer, 4 * 8 + 1, 4))
            $iCertificateAddress = DllStructGetData($stEndian, 1)
            DllStructSetData($stEndian, 1, BinaryMid($bBuffer, 4 * 8 + 4 + 1, 4))
            $nCertificateSz = DllStructGetData($stEndian, 1)
            If $iCertificateAddress Then ConsoleWrite("Certificate Table address found, offset = " & $iCertificateAddress & ", size = " & $nCertificateSz & @CRLF)
        EndIf

        ; Read in ALL sections
        $bBuffer = FileRead($hFile, $nSections * 40)
        If @error Then ExitLoop

;~     DONE Reading File info..

        ; Now to traverse the sections..

        ; $iOffset Now refers to the location within the binary data
        $iOffset = 1
        $nLastSectionOffset = 0
        $nLastSectionSz = 0
        For $i = 1 To $nSections
            ; Within IMAGE_SECTION_HEADER: RawDataPtr = offset 20, SizeOfRawData = offset 16
            DllStructSetData($stEndian, 1, BinaryMid($bBuffer, $iOffset + 20, 4))
            $nTemp = DllStructGetData($stEndian, 1)
            ;ConsoleWrite("RawDataPtr, iteration #"&$i&" = " & $nTemp & @CRLF)
            ; Is it further than last section offset?
            ;  AND - check here for rare situation where section Offset may be outside Filesize bounds
            If $nTemp > $nLastSectionOffset And $nTemp < $nFileSize Then
                $nLastSectionOffset = $nTemp
                DllStructSetData($stEndian, 1, BinaryMid($bBuffer, $iOffset + 16, 4))
                $nLastSectionSz = DllStructGetData($stEndian, 1)
            EndIf
            ; Next IMAGE_SECTION_HEADER
            $iOffset += 40
        Next
;~         ConsoleWrite("$nLastSectionOffset = " & $nLastSectionOffset & ", $nLastSectionSz = " & $nLastSectionSz & @CRLF)

        $iSucces = 1    ; Everything was read in correctly
    Until 1
    $iErr = @error
    FileClose($hFile)
    ; No Success?
    If Not $iSucces Then Return SetError($iExit, $iErr, $aRet)

;~     Now to calculate the last section offset and size to get the 'real' Executable end-of-file
    ; [0] = Overlay Start
    $aRet[0] = $nLastSectionOffset + $nLastSectionSz

    ; Less than FileSize means there's Overlay info
    If $aRet[0] And $aRet[0] < $nFileSize Then
        ; Certificate start after last section? It should
        If $iCertificateAddress >= $aRet[0]  Then
            ; Get size of overlay IF Certificate doesn't start right after last section
            ; 'squeezed-in overlay'
            $aRet[1] = $iCertificateAddress - $aRet[0]
        Else
            ; No certificate, or < last section - overlay will be end of last section -> end of file
            $aRet[1] = $nFileSize - $aRet[0]
        EndIf
        ; Size of Overlay = 0 ?  Reset overlay start to 0
        If Not $aRet[1] Then $aRet[0] = 0
    EndIf
    $aRet[2] = $iCertificateAddress
    $aRet[3] = $nCertificateSz
    $aRet[4] = $nFileSize
    Return $aRet
EndFunc

FilePEOverlayExtract.au3 ~prev downloads: 34

Updates:

2013-08-09-rev2:

Fixed: Didn't detect 'sandwiched-in' Overlays - Overlays appearing between the end of code/data and before a Certificate section

Changed: UDF now returns an array of information: Overlay offset and size, Certificate offset and size, and filesize

Fixed: Overlays < 32 bytes may have been written incorrectly

2013-08-09:

Fix: Certificate Table now identified and excluded from false detection as Overlay.

2013-08-07:

Fix: Section Offsets that start beyond the filesize are now accounted for.  I'm not sure when this happens, but it's been reported to happen on other sites.

Modified: A more reasonable filesize limit.

Modified: Option to Extract just the Executable without Overlay, or the Overlay itself

2013-08-03:

Fixed: Calculation of FileRead data was off by 16 (which still worked okay, but was not coded correctly!)

Fixed: @extended checking after calls to other code

Edited August 10, 2013 by Ascend4nt

[0xdefea7 10]

Very nice UDF. Works perfectly. Thanks for sharing

Edited August 3, 2013 by 0xdefea7

[Ascend4nt 117]

I just noticed a glaring error with the code - I was subtracting 16 from the final FileRead instead of 32.. which actually worked since it read to end-of-file.

Also, I was using @extended after it was reset by other code.

Update is in first post.

[joakim 4]

Nice. Btw, authenticode signatures is another example of "overlays" that thus can be extracted. Regarding the "file optimizers crippling compiled scripts", you can also re-make the exe (PE) with the old overlay as a new section (for instance instead of as a resource).

[Ascend4nt 117]

joakim, thanks for noting that.  I've realized now that there's a lot of executables that include overlays, including setup and install programs. I didn't allow the example to work with 'big' overlays (>256K), but it can easily be rewritten to use a filecopy function that reads and writes the data in small chunks.

[Ascend4nt 117]

Updated. Found some info on the internet about sections offsets sometimes being outside the filesize!  Plus I figured why not add the ability to extract the executable (or other PE file) without the overlay also.

2013-08-07:

Fix: Section Offsets that start beyond the filesize are now accounted for.  I'm not sure when this happens, but it's been reported to happen on other sits.

Modified: A more reasonable filesize limit.

Modified: Option to Extract just the Executable without Overlay, or the Overlay itself

[Ascend4nt 117]

Well I goofed - I missed the information on IMAGE_DIRECTORY_ENTRY_SECURITY in the Data Directory description that describes how the offset is not in fact an Rva, but an actual file offset.  Turns out this offset is more often than not what I was detecting as overlays in some files.  But I've now fixed that.

Also: I'm not sure about overlays mixed with signed executables.  The information seems to indicate that overlays must be appended to the end of the certificate, and the size would be written to the Data Directory entry.  However, this means traversing through all the Certificate Tables (if there's more than one) to see if there's extra data at the end..  I'm wondering if anyone has a file I can check out which has both an overlay and certificate?  It would seem to make more sense to just plop it behind the certificate but maybe that will cause the exe to fail.. to be continued..

Updates:

2013-08-09:

Fix: Certificate Table now identified and excluded from false detection as Overlay.

[joakim 4]

Guess I can run some tests on it regarding signatures and other overlays. Can it can easily be done by self signed certificates. I made an application some time ago that could hide data inside the authentice signature without invalidating it, and actually there is no size limit to it..

[Ascend4nt 117]

Hmm.. yeah I'm not entirely sure what is allowed and whats not.  I was looking at this: "Changing a signed executable without altering Windows digital signature", which hints at possibly changing the checksum as well as the Data Directory size..]

*edit: Just did a test of the typical method of adding overlays, made easier by a simple command prompt line:

copy /b file.exe+overlay.data fileout.exe

Result: Program still runs, but the digital signature info is no longer recognized.  So it appears that yes, an overlay would need to be accounted for in the Data Directory, and probably the Certificate Table array.

2nd edit: Got out my hex editor and went crazy

It seems that I can actually move the signature further back in the file and insert an overlay after the last section, but before the signature.  After adjusting the offset in the Data Directory, the signature was once again recognized, and there's still no ill effects noticed.  I'm not sure this is a good method or even if it's replicable with all signatures..

Basically what I'd really like to see is how people would sign an AutoIt script and where the overlay and signature get placed..

3rd: Okay, forgot about the 'Details' pane of the Digital Signatures info tab.  I've broken it by inserting that overlay between the last section and the overlay.  So perhaps the checksum would need to be modified.  However, I suppose the next thing to try is to append the overlay and adjust sizes instead, to see if that works nicely.

4th edit: Yup, changing the size in both the Data Directory and the Certificate Tables array is not sufficient; the signature appears but still gives 'Not Valid' when going into Details.  That should be all the edits for this post!  At least I know adjusting this stuff isn't as easy as moving stuff around or changing offsets or sizes!

Edited August 9, 2013 by Ascend4nt

[Ascend4nt 117]

Whew, okay.. after all that testing I've realized I wasn't properly padding the overlay to a 64-bit (8-byte) boundary, at the end of the Certificate Table.  Once I did that, I only needed to change the size in the Data Directory.  The signature is now registering as OK!

So, yay for that.. but boo for finding those types of Overlays.  A bit more work in hopping around the file..

*edit: clarified where overlay was - at the end of the Certificate Table (basically just using that copy /b method above, but making sure to add enough padding bytes).

Edited August 9, 2013 by Ascend4nt

[Ascend4nt 117]

Woo boy that was tricky.  I've tracked down executables that contain signatures and overlays to see how it all fits together, and the common situation is this: legitimate overlays are added to an executable first and then the file is signed - putting the Certificate after the overlay.

At first this wasn't clicking but now it makes perfect sense.  So the new update looks for these in-between Overlays too. The UDF function now also returns a complete array of info including Overlay offset and size, Certificate offset and size, and filesize (as a convenience).

Update:

2013-08-09-rev2:

Fixed: Didn't detect 'wedged-in' Overlays - Overlays appearing between the end of code/data and before a Certificate section

Changed: UDF now returns an array of information: Overlay offset and size, Certificate offset and size, and filesize

Fixed: Overlays < 32 bytes may have been written incorrectly

[joakim 4]

One of the strange things I remember from when poking with this, was that by modifying the signature in certain ways would make explorer crash/freeze when attempting to validate it. If I'm not wrong I had to also adjust the authenticode signature header slightly to adapt to new size (in addition to pe checksum, data directory entries, and the signature itself along with alignment issues as you already pointed out).

The funny thing about all this in the autoit world is that a signed executable with overlay added (encoded script), lets call it program A. You could turn that executable into a different program without messing up the signature. You could even put the script overlay within the signature overlay! Try this by signing the AutoItSC* and then hiding the encoded script within the signature

On a side note, I could mention because actually I did not notice until now, that on Windows 7 x64 I can not even run the executables that I've signed with the certificate I made back then when doing that "DigitalSignatureTweaker" tool. So the OS refuses to run those executables, while it will not complain about unsigned ones.. Hmm, maybe it is my test certificate that has been blacklisted somehow..(don't know). But if not, it certainly does not make sense to have higher restriction for self signed than unsigned executables, which seems like strange logic to me. So maybe the trick can still be verified on Windows 7 x64, as a long as different test certificate is used. Anyway, this was a bit off topic. 

Edit: And for those with nothing else to do, and for the fun of it, could try to make signed executables with hidden signed executables with hidden signed executables, until you are fed up with nesting it. Do it on Microsoft signed executables (exe,dll,sys) instead of self signed ones.

Edited August 10, 2013 by joakim

[Ascend4nt 117]

One of the strange things I remember from when poking with this, was that by modifying the signature in certain ways would make explorer crash/freeze when attempting to validate it. If I'm not wrong I had to also adjust the authenticode signature header slightly to adapt to new size (in addition to pe checksum, data directory entries, and the signature itself along with alignment issues as you already pointed out).

Hm, I was wondering why when I modified the Data Directory size that Windows didn't complain about the signature. As I understood it I should have modified the Certificate Table entry too.

From what little I know of the 'Attribute Certificate Table' as Microsoft labels this in their pecoff documentation, the format is basically:

dword dwLength;       // length of the Certificate
word  wRevision;      // Certificate Version #
word wCertificateType;// Type of Certificate
byte bCertificate[];  // Certificate Data

The dwLength field specifies the length of one individual Certificate, and there can actually be an array of these (aligned to 8-bytes).  The reason I didn't want to bother with this is because I'd need to traverse multiple certificates (a rare situation I'd say) to find the last, then adjust the last Certificate's size, as well as the Data Directory size (which can be different as it encompasses the entire table).

It's interesting to mess with this stuff but curious that there's no built-in security for the security certificate itself.  I was considering some type of 'enhanced' overlay or 'payload' detection on certificates, but there's no way to know (with the limited info on certificates I have) if the certificate's size has been changed - unless only the Data Directory entry has changed..

The funny thing about all this in the autoit world is that a signed executable with overlay added (encoded script), lets call it program A. You could turn that executable into a different program without messing up the signature. You could even put the script overlay within the signature overlay! Try this by signing the AutoItSC* and then hiding the encoded script within the signature

Will the script really run appended to the end of a signature?  I figured that the AutoIt code would do a search for the overlay the way I've done it, but were they so lazy that they just look at the end of the file?  Actually, looking at some compiled exe's (as well as .a3x files), the last 8 bytes at the end of the file is a text string that's pretty consistent.. they could easily search for the signature offset from there, or store the size in it.  But if you sign a script with an overlay, then the signature would come last.. so would that cause AutoIt to fail?

I know I've at least been able to 'pop off' the .A3X file and replace it with another and that's really a serious security flaw with AutoIt - no wonder its been flagged by AntiVirus companies for years. Its just now with the new beta's that the script is being put into a resource, which is a bit more secure.

As an odd consequence of me delving more into PE stuff lately, I've actually found a way to detect overlays in old DOS executables.  In fact 'defrag.exe' has another executable appended at the end. So apparently this technique is plenty old..

[joakim 4]

Will the script really run appended to the end of a signature?

If I remember this ok, it can be

• before signature (stub + script is signed).
• after signature (stub is signed).
• within signature (stub is signed, and script is within the scope of signature as defined in data directory and signatures header).

 

Not so sure if it's a security flaw with regards to autoit, as no one would sign the stub before appending the script anyway. And even if you could append a script after or within the signature, the stub will still execute the correct script (assuming you signed the stub + script).

But still, it's fun to mess with

Edited August 10, 2013 by joakim

[Jon 794]

Hm, I was wondering why when I modified the Data Directory size that Windows didn't complain about the signature. As I understood it I should have modified the Certificate Table entry too.

From what little I know of the 'Attribute Certificate Table' as Microsoft labels this in their pecoff documentation, the format is basically:

dword dwLength;       // length of the Certificate
word  wRevision;      // Certificate Version #
word wCertificateType;// Type of Certificate
byte bCertificate[];  // Certificate Data

  I figured that the AutoIt code would do a search for the overlay the way I've done it, but were they so lazy that they just look at the end of the file?  Actually, looking at some compiled exe's (as well as .a3x files), the last 8 bytes at the end of the file is a text string that's pretty consistent.. they could easily search for the signature offset from there, or store the size in it.  But if you sign a script with an overlay, then the signature would come last.. so would that cause AutoIt to fail?

 

It wasn't doing PE related stuff, It just searched for an internal signature near the end of the file which has an offset in it to allow it to then jump to the right location. Signing doesn't break it.

[trancexx 892]

It wasn't doing PE related stuff, It just searched for an internal signature near the end of the file which has an offset in it to allow it to then jump to the right location. Signing doesn't break it.

I changed that at some point because the method that was used previously (the one you mention) was security issue. It allowed third party to simply overtake and compromise compiled script by, for example, adding a3x code to the resource section of the executable. So unless you have removed the code for skipping PE part of the compiled script when looking for attached script AutoIt compiled script no loger have those issues.

Due to backward compatibility compiled script must check for a3x code stored using older method but as I said, it does it a bit smarter now.

[Ascend4nt 117]

It wasn't doing PE related stuff, It just searched for an internal signature near the end of the file which has an offset in it to allow it to then jump to the right location. Signing doesn't break it.

Well, that solves that mystery!  A bit of a hack, but I suppose looking for those 24 bytes of data (if we're talking that signature) hasn't broken anything yet, so that's good. Although as I said it was a bit of a security issue since it could be replaced so easily.  Thanks for the input.

Due to backward compatibility compiled script must check for a3x code stored using older method but as I said, it does it a bit smarter now.

When you talk about code stored using the old method, do you mean there's checks for a3x outside of the resource section?  That doesn't make sense to me since its now placed in resource section.. but perhaps I'm misreading that.

[Jon 794]

Well, that solves that mystery!  A bit of a hack, but I suppose looking for those 24 bytes of data (if we're talking that signature) hasn't broken anything yet, so that's good. Although as I said it was a bit of a security issue since it could be replaced so easily. 

 

Well, not really. If you have write access to the exe then anything goes anyway. The only real way to ensure that an exe has not been modified is a digital signature and that shows as invalid if you tamper after signing.

[trancexx 892]

When you talk about code stored using the old method, do you mean there's checks for a3x outside of the resource section?  That doesn't make sense to me since its now placed in resource section.. but perhaps I'm misreading that.

You aren't misreading. And really it shouldn't not make sense to you.

For example, every compiled script can run other compiled script (in any format) if command line parameter /AutoIt3ExecuteScript is specified. This was by default until few weeks ago when Jon changed default setting for whatever reason.

If the new code doesn't check for older method then that would be script breaking behaviour because documentation nowhere says (or did say) that newer compiled scripts can run only newer compiled scripts and not older ones.

Maintaining backward compatibility with every new feature was the law. Of course it's much easier not to care and do things as you wish, but obligation toward all users is much more important. We (I) had problems with particular individuals even when maintaining full backward compatibility so I won't even try to imagine what would have been otherwise.

[Ascend4nt 117]

You aren't misreading. And really it shouldn't not make sense to you.

For example, every compiled script can run other compiled script (in any format) if command line parameter /AutoIt3ExecuteScript is specified. This was by default until few weeks ago when Jon changed default setting for whatever reason.

If the new code doesn't check for older method then that would be script breaking behaviour because documentation nowhere says (or did say) that newer compiled scripts can run only newer compiled scripts and not older ones.

Maintaining backward compatibility with every new feature was the law. Of course it's much easier not to care and do things as you wish, but obligation toward all users is much more important. We (I) had problems with particular individuals even when maintaining full backward compatibility so I won't even try to imagine what would have been otherwise.

Hm. Honestly, I really don't get why the beta still checks for scripts lying outside of the PE sections.(*see edit)

Aut2Exe compiles a script and puts the script into a resource, which is sensible. Now, to put an overlay on the executable requires 2 steps - deleting the script resource, and then appending an a3x script at the end (and I've just verified this works). 

But tell me, how does that have anything to do with backward compatibility when it requires manipulating a compiled executable, which typical programmers wouldn't do? Or is it just the possibility of them doing it?  To me it doesn't seem like real script-breaking behavior because well, who the heck does that (besides me, just now )? Seems a better choice to use the new Aut2Exe or stick with a3x files.

Maybe I'm missing something in the compile options?  I've already verified that an exe stub is no longer included in the Aut2Exe folder, so manually putting it together outside of Aut2Exe isn't possible without pulling apart the resources inside of that executable..

*edit: D'oh, guess I didn't comprehend the extent of what you were saying - I really had no idea that /AutoIt3ExecuteScript could run other compiled scripts without actually running the other executable.  I just tested it now, and its a bit clearer.  However, why should the compiled exe check itself for an overlay, rather than just check other executables when invoked that way?  Or am I missing a bigger picture here.. ?

Edited August 11, 2013 by Ascend4nt