Jump to content
Sign in to follow this  

Base64 decoder/encoder, Internet header decoders, email subject decoder, UTF-8, ISO-8859-1, telnet negociations

Recommended Posts


I've been lurking around for a loooong time... and I decided to finally share a little. I do a lot of internet stuff, mostly machine to machine for work (instrumentation) so I have quite a few "RFC" scripts.

Disclaimer these work for me... but I sometime use... "shortcuts" based on my particular requirement. An example, the Base64 encoding snippet might not be too good for binary data. I pad the original data with spaces to avoid the "==" padding of base64.

So... first is the base64 encoding snippet. It is not in a function, it was in a sequential program, used only once! It encode $Graph to $SMTPMessage:

; Create the base64 encoding table
Dim $Base64EncodingTable[0]
For $Cpt = Asc("A") to Asc("Z")
    _ArrayAdd($Base64EncodingTable, Chr($Cpt))
For $Cpt = Asc("a") to Asc("z")
    _ArrayAdd($Base64EncodingTable, Chr($Cpt))
For $Cpt = Asc("0") to Asc("9")
    _ArrayAdd($Base64EncodingTable, Chr($Cpt))
_ArrayAdd($Base64EncodingTable, "+")
_ArrayAdd($Base64EncodingTable, "/")

; Pad the SVG Graph to attach with space(s). Lazy way to avoid base64 == pading
While Mod(StringLen($Graph), 3) <> 0
    $Graph &= " "

; Start from the first character
$Cpt = 1

    ; Extract the 3 characters to encode
    $Char1 = Asc(StringMid($Graph, $Cpt, 1))
    $Char2 = Asc(StringMid($Graph, $Cpt+1, 1))
    $Char3 = Asc(StringMid($Graph, $Cpt+2, 1))

    ; Encode them to 4 characters
    $SMTPMessage &= $Base64EncodingTable[BitShift(BitAND($Char1, 252), 2)]
    $SMTPMessage &= $Base64EncodingTable[BitShift(BitAND($Char1, 3), -4) + BitShift(BitAND($Char2, 240), 4)]
    $SMTPMessage &= $Base64EncodingTable[BitShift(BitAND($Char2, 15), -2) + BitShift(BitAND($Char3, 192), 6)]
    $SMTPMessage &= $Base64EncodingTable[BitAND($Char3, 63)]

    ; Increment the counter, and if required, add a @CRLF to split in multiples lines
    $Cpt += 3
    If Mod($Cpt, 57) = 1 Then $SMTPMessage &= @CRLF

; Do this until all the graph has been encoded
Until $Cpt >= StringLen($Graph)


Second... I just finished this one and was allready thinking about sharing it... so it's been encapsulated into function a bit more. I use it to decode email subjects in a system where you can update something by email. I separated the Base64Decode function so it can be grabbed more easily. Please note that it return an hex string so you would still need to convert it if it's a string with BinaryToString or whatever suit your needs.

If can be copied as is and runned directly... it include my test strings! (Yes... I'm french!)

; For the $SB_UTF8 and $SB_ANSI Variable
#include <StringConstants.au3>

; For _ArrayAdd and _ArraySearch used in the Base64 decoder
#include <Array.au3>

; Various test sentences...
;$text = "=?UTF-8?Q?Ce=c3=a7i_est_un_autre_test!_h=c3=a9h=c3=a9!?="         ; Normal UTF-8
;$text = "=?UTF-8?Q?Encore_=3d_un_autre_test_=c3=a9_?="                     ; "=" added
;$text = "=?UTF-8?Q?un_autre_test_=5f_=c3=a9?="                             ; "_" added
;$text = "=?UTF-8?B?Q2XDp2kgZXN0IHVuIGF1dHJlIHRlc3QhID0gXyBow6low6kh?="     ; UTF-8 Base64
$text = "=?UTF-8?B?ZcOnaSBlc3QgdW4gYXV0cmUgdGVzdCEgPSBfIGjDqWjDqSE=?="      ; UTF-8 Base64 with padding
;$text = "=?iso-8859-1?Q?Ce=E7i_est_un_test!?="                             ; iso-8859-1
MsgBox(0, "", DecodeHeader($text))

Func DecodeHeader($lString)

    ; Check and store encoding type
    If StringInStr($lString, "?Q?") Then
        ; Quoted printable content
        $lType = "?Q?"
    ElseIf StringInStr($lString, "?B?") Then
        ; Base64 encoding
        $lType = "?B?"
        ; No encoding (or unknown encoding)

    ; Start of the charset string
    $lStart = StringInStr($lString, "=?") + 2

    ; End of the charset string
    $lStop = StringInStr($lString, $lType)

    ; Charset variable, storing "UTF-8" or "iso-8859-1"
    $lEncoding = StringMid($lString, $lStart, $lStop-$lStart)

    ; Change encoding type for the BinaryToString flag
    If $lEncoding = "UTF-8" Then
        $lEncoding = $SB_UTF8
    ElseIf $lEncoding = "iso-8859-1" Then
        $lEncoding = $SB_ANSI
        MsgBox(0, "", "Unknown character set")

    ; Start of the actual encoded content
    $lStart = $lStop + 3

    ; End of the actual encoded content
    $lStop = StringInStr($lString, "?=")

    ; Actual content to decode
    $lString = StringMid($lString, $lStart, $lStop-$lStart)

    ; For Quoted printable content
    If $lType == "?Q?" Then

        ; Restore underscore encoded spaces
        $lString = StringReplace($lString, "_", " ")

        ; Starting with the first character of the string
        $lCpt = 1

        ; "=XX" search and convert loop
        While 1

            ; There will be 0 characters to convert in that block unless...
            $lConvertableLenght = 0

            ; That character, and another one 3 bytes over... and the next, and the next...
            For $lCpt2 = 0 to 100

                ; Is equal to "="
                If StringMid($lString, $lCpt+($lCpt2*3), 1) == "=" Then

                    ; In that case, yes, we will have to convert 3 more characters
                    $lConvertableLenght += 3

                    ; But if we fail to find or reach the end of a block of encoded characters, exit the search

            ; If we did in fact find some encoded characters
            If $lConvertableLenght > 0 Then

                ; Extract that block of encoded characters
                $lConvertableString = StringMid($lString, $lCpt, $lConvertableLenght)

                ; Convert it
                $lConvertedString = BinaryToString("0x" & StringReplace($lConvertableString, "=", ""), $lEncoding)

                ; Replace it in the original
                $lString = StringReplace($lString, $lConvertableString, $lConvertedString)

            ; Increment the "=XX" search and convert loop counter
            $lCpt += 1

            ; If we reached the end of the string, exit the "=XX" search and convert loop
            If $lCpt >= StringLen($lString) Then ExitLoop

        ; Continue searching in the "=XX" search and convert loop

    ; For Base64 encoded strings

        ; Use the separate Base64Decode function
        $lString = Base64Decode($lString)
        $lString = BinaryToString($lString, $lEncoding)


Func Base64Decode($lEncoded)

    ; Create the base64 encoding table
    Dim $Base64EncodingTable[0]
    For $Cpt = Asc("A") to Asc("Z")
        _ArrayAdd($Base64EncodingTable, Chr($Cpt))
    For $Cpt = Asc("a") to Asc("z")
        _ArrayAdd($Base64EncodingTable, Chr($Cpt))
    For $Cpt = Asc("0") to Asc("9")
        _ArrayAdd($Base64EncodingTable, Chr($Cpt))
    _ArrayAdd($Base64EncodingTable, "+")
    _ArrayAdd($Base64EncodingTable, "/")

    ; Start from the first character
    $Cpt = 1
    $Decoded = "0x"

        ; Extract the 4 characters to encode
        $Char1 = StringMid($lEncoded, $Cpt, 1)
        $Char2 = StringMid($lEncoded, $Cpt+1, 1)
        $Char3 = StringMid($lEncoded, $Cpt+2, 1)
        $Char4 = StringMid($lEncoded, $Cpt+3, 1)

        ; Decode them
        $Decoded &= Hex(BitShift(_ArraySearch($Base64EncodingTable, $Char1, 0, 0, 1), -2) + BitShift(BitAnd(_ArraySearch($Base64EncodingTable, $Char2, 0, 0, 1), 48), 4), 2)
        If $Char3 <> "=" Then $Decoded &= Hex(BitShift(BitAnd(_ArraySearch($Base64EncodingTable, $Char2, 0, 0, 1), 15), -4) + BitShift(BitAnd(_ArraySearch($Base64EncodingTable, $Char3, 0, 0, 1), 60), 2), 2)
        If $Char4 <> "=" Then $Decoded &= Hex(BitShift(BitAnd(_ArraySearch($Base64EncodingTable, $Char3, 0, 0, 1), 3), -6) + _ArraySearch($Base64EncodingTable, $Char4, 0, 0, 1), 2)

        ; Increment the counter
        $Cpt += 4

    ; Do this until all the encoded string has been decoded
    Until $Cpt >= StringLen($lEncoded)



Last thing... I may update it into a better format for you, like a standalone telnet program with GUI. It is my telnet options negociations loops. The basic concept is systematically deny all request for special options and keep it "raw".

If server says "Will", I reply "Don't". If it says "Do", I reply "Wont"... unless it's the terminal type subnegociation, in which case I reply xterm!

$Data for now needs to be Global. You still need to know what you're doing, opening sockets and making a basic communication loop or something.

Global $T_Is = Chr(0)
Global $T_Send = Chr(1)
Global $T_TerminalType = Chr(24)
Global $T_SE = Chr(240)
Global $T_SB = Chr(250)
Global $T_Will = Chr(251)
Global $T_Wont = Chr(252)
Global $T_Do = Chr(253)
Global $T_Dont = Chr(254)
Global $T_IAC = Chr(255)

Func NegotiateTelnetOptions()
    $NegotiationCommandsToSendBack = ""
    While StringInStr($Data, $T_IAC)
        $IACPosition = StringInStr($Data, $T_IAC)
        Switch StringMid($Data, $IACPosition+1, 1)
            Case $T_Will
                $NegotiationCommandsToSendBack &= CraftReply_CleanUpData($IACPosition, $T_Dont)
            Case $T_Do
                If StringMid($Data, $IACPosition+2, 1) = $T_TerminalType Then
                    $NegotiationCommandsToSendBack &= CraftReply_CleanUpData($IACPosition, $T_Will)
                    $NegotiationCommandsToSendBack &= CraftReply_CleanUpData($IACPosition, $T_Wont)
            Case $T_SB
                If StringMid($Data, $IACPosition, 6) = ($T_IAC & $T_SB & $T_TerminalType & $T_Send & $T_IAC & $T_SE) Then
                    $NegotiationCommandsToSendBack &= $T_IAC & $T_SB & $T_TerminalType & $T_Is & "xterm" & $T_IAC & $T_SE
                    $Data = StringReplace($Data, StringMid($Data, $IACPosition, 6), "")
                    MsgBox(0, "", "Unknown Subnegotiation...") ; Should never happen.
    Return $NegotiationCommandsToSendBack

Func CraftReply_CleanUpData($IACPosition, $Reply)
    $PartialCommandToSendBack = $T_IAC & $Reply & StringMid($Data, $IACPosition+2, 1)
    $Data = StringReplace($Data, StringMid($Data, $IACPosition, 3), "")
    Return $PartialCommandToSendBack


Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Similar Content

    • By AXLE
      I am trying to find information on using UTF-8 Strings in AutoIt. After searching extensively I cannot find anything conclusive on this topic. What I need to do is FileRead() into a String variable(or Array) and keep the UTF-8 Encoding. Some articles, and even Help documents on FileOpen() suggest that AutoIT (Current Versions) can read and store UTF-8 internally but my tests on reading a test web page containing UTF-8 encoded characters into a variable fails.
      Does/Can AutoIt use Strings Encoded as UTF-8, and if so how ?
      If Not does anyone know of a UDF, or a C/Win-API routine to allow to use a UTF-8 Array in AutoIt ?
      What does AutoIt use internally for Strings ? Is it converting the UTF-8 file to UCS-2 String in the Variable ?
      The following is an example which fails for me.
      ;UTF-8 Tests #include <FileConstants.au3> #include <MsgBoxConstants.au3> #include <WinAPIFiles.au3> ;https://www.w3.org/2001/06/utf-8-test/UTF-8-demo.html ;Also all checked in Notepad++ UTF-8 Encoding (Many Characters are scrambled) Local $sFile1 = "UTF-8 test file.htm"; 414 Lines | 76,412 characters. "UTF-8 test file.htm" = "/UTF-8-demo.html" Local $sFile2 = "test2.html" Local $hfile1 = FileOpen($sFile1, BitOr($FO_READ, $FO_UTF8_NOBOM)) If @error Then MsgBox($MB_SYSTEMMODAL, "FileOpen1", "Value of @error is: " & @error & @CRLF & "Value of @extended is: " & @extended) EndIf Local $sAm_I_UFT_8 = FileRead($hfile1, -1);Does not appear to read UTF-8 characters correctly from the "UTF-8 test file.htm" If @error Then MsgBox($MB_SYSTEMMODAL, "FileRead", "Value of @error is: " & @error & @CRLF & "Value of @extended is: " & @extended) EndIf FileClose($hfile1) Local $sAm_I_Still_UTF_8 = $sAm_I_UFT_8 ;Are these two strings stored internaly as UTF-8 ? If @error Then MsgBox($MB_SYSTEMMODAL, "String=String", "Value of @error is: " & @error & @CRLF & "Value of @extended is: " & @extended) EndIf Local $iStrLen1 = StringLen($sAm_I_UFT_8) Local $iStrLen2 = StringLen($sAm_I_Still_UTF_8) MsgBox($MB_SYSTEMMODAL, "String Lenght of $sAm_I_UFT_8", $iStrLen1); 414 Lines | 70,174 characters MsgBox($MB_SYSTEMMODAL, "String Lenght of $sAm_I_Still_UTF_8", $iStrLen2); 414 Lines | 70,174 characters Local $hfile2 = FileOpen($sFile2, BitOR($FO_OVERWRITE, $FO_BINARY)) If @error Then MsgBox($MB_SYSTEMMODAL, "FileOpen2", "Value of @error is: " & @error & @CRLF & "Value of @extended is: " & @extended) EndIf FileWrite($hfile2, $sAm_I_Still_UTF_8) ;If $sAm_I_Still_UTF_8 is actual UTF-8 it should be an exact copy of the original "UTF-8 test file.htm" If @error Then MsgBox($MB_SYSTEMMODAL, "FileOpen2", "Value of @error is: " & @error & @CRLF & "Value of @extended is: " & @extended) EndIf FileClose($hfile2)  
    • By Miliardsto
      You can see my script to POST json and the problem where occurs backslashes before quotations. Second problem is the apperance , there is no new lines for every new value.
      LOOK in attached images!
      You can see rest api on the screen and with id = 0 its made my postman and the rest by autoit (with backslashes)
      Func jsonSave() ; Make login request to server JSON POST Local $data, $request Local $obj Local $adress = '' Json_Put($Obj, ".msgName", "Point HPs") Json_Put($Obj, ".msg", "Not pointed correctly") Json_Put($Obj, ".username", "karolek890") Local $Json = Json_Encode($Obj,$JSON_UNESCAPED_SLASHES) ; WITHOUT preetier becouse it makes clashes r n and t only works in autoit $request = _httpRequestBot($adress, 'POST', $Json) ; only get login if correct then success If @error Then Exit MsgBox(0, "Error", "Open method returned @error = " & @error & " and @extended = " & @extended) Else ConsoleWrite("Data to json server posted!") EndIf EndFunc  
      Im using this JSON UDF

    • By Viszna
      I create a function that saves the log to an html file.
      File html are included picture (conversion to Base64)
      Everything works OK.
      But I do the actions:
      screenshot to the png file (smaller than bmp) per disk (  _ScreenCapture_Capture(@ScriptDir & "\screenshot.png")  ) convert image from disk to base64 I need help to optimize the script:
      - screenshot to memory (do not save to disk) - convert this object to png in memory - invoking the conversion of png image from memory to base64 The first step _ScreenCapture_Capture("") create handle to an HBITMAP in memory
      How to convert image in memory to png?
      How to use Func _ConvertToBase64
      I attach my code
      #include <ScreenCapture.au3> #include <Date.au3> Global $RaportFileName = @YEAR & "-" & @MON & "-" & @MDAY & "_" & @HOUR & "_" & @MIN & "_" & @SEC & ".html" $text = "This is first line text" & @CRLF &"and this is next line" _Raport($text, 1) FileWrite(@ScriptDir & "\" & $RaportFileName, "</pre></html>") ; #FUNCTION# ==================================================================================================================== ; Name ..........: _Raport ; Description ...: ; Syntax ........: _Raport($sText1[, $Screen = 0]) ; Parameters ....: $sText1 - a string value. ; $Screen - [optional] an unknown value. Default is 0. ; 0 - Default - do not screenshot ; 1 - added screenshot full desktop ; Return values .: None ; Author ........: Your Name ; Modified ......: ; Remarks .......: ; Related .......: ; Link ..........: ; Example .......: No ; =============================================================================================================================== Func _Raport($sText1, $Screen=0) Local $sText = "" Local $sHead = "" ; Define HTML file header and style $sHead = '<html>' & @CRLF & '<head><meta charset="utf-8"></head>' & @CRLF $sHead = $sHead & '<style>img{border:3px solid #FF0000;}</style>' & @CRLF $sHead = $sHead & '<style>pre{font-family: monospace;}</style>' & @CRLF $sHead = $sHead & '<style>pre{font-size: large;}</style>' & @CRLF $sHead = $sHead & '<pre>' & @CRLF If NOT FileExists(@ScriptDir & "\" & $RaportFileName) Then ; If file Raport not exist then create FileOpen(@ScriptDir & "\" & $RaportFileName, 258) FileWrite(@ScriptDir & "\" & $RaportFileName, $sHead) EndIf If StringInStr($sText1, @CRLF) > 0 Then ; @CRLF (ENTER) change the @CRLF and 11 space (indentation on width "[GG:MM:SS] ") $sText1 = StringReplace($sText1, @CRLF, @CRLF & '&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;') EndIf $sText = $sText & $sText1 If $Screen <> 0 Then _ScreenCapture_Capture(@ScriptDir & "\screenshot.png") $sText = $sText & @CRLF & '<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;' & '<img src="data:image/png;base64,' & _ConvertToBase64(@ScriptDir & "\screenshot.png") & '"/>' EndIf FileWrite(@ScriptDir & "\" & $RaportFileName, "[" & _NowTime(5) & "]&nbsp;" & $sText & "<br><br>"&@CRLF) ; write to file Raport EndFunc Func _ConvertToBase64($fFile) ;Xroot 2011 ;ClipPut("") ;$FN=@ScriptDir & "\screenshot.png" $FN=$fFile $dat=FileRead(FileOpen($FN,16)) $objXML=ObjCreate("MSXML2.DOMDocument") $objNode=$objXML.createElement("b64") $objNode.dataType="bin.base64" $objNode.nodeTypedValue=$dat ClipPut("") $Wynik = "" ;ClipPut($objNode.Text) $Wynik = $objNode.Text Return $Wynik EndFunc P.S.
      Excuse me my not good English.
    • By argumentum
      AutoIt Machine Code Algorithm Collection
      By Ward, November 11, 2010 in AutoIt Example Scripts 
      Posted November 11, 2010 (edited) I have already published a lot of AutoIt UDF about algorithm, but all of them only support 32 bits or so called X86 system. Recently I got a computer with Windows 7 64 bits, so I finally added X64 support to most of my old projects. Besides, I also added some new. For example, some compression algorithm and SHA3 Candidates.
      Following are the algorithms list:
      Checksum   CRC16   CRC32   ADLER32 Compression   FastLZ   LZF   LZMA   LZMAT   MiniLZO   QuickLZ Encode   Base64   ARC4   XXTEA   DES   AES Hash   Checksums (CRC16/CRC32/ADLER32)   MD2   MD4   MD5   SHA1   SHA2 (SHA224/256/384/512)   SHA3 Candidates     BLAKE     BMW (Blue Midnight Wish)     CUBEHASH     ECHO     SHABAL     SKEIN Some points to mention:
      All of the subroutines have one or more examples to demonstrate the usage. Since the function and usage of subroutine are easy to understand. A complete subroutines and parameters list are unavailability now. Sorry for my lazy. All of the subroutines here invoked by Lazycat's method (through CallWindowProc API). My MemoryDLL UDF is not necessary this time. Although MemoryFuncCall (part of MemoryDLL) is still good, but inevitably, it is slower than CallWindowProc. Some subroutines have the same name with my old machine code version UDF. But for some reason, I rearrange the position of the parameters. Please not mix up. If you notice, yes, checksums are duplicated. But they receive different parameters. One is the old style, and another use the same interface as other hashes. Choose what you like, but don't use them in the same time. Some algorithm already supported by the standard UDF "Encryption.au3". But I still provide them, because some system lack of the full support of Windows Crypt Library. If you are looking for only one hash algorithm, for example, used in encryption, I suggested "SHABAL_TINY.au3". Although it is a bit slower then SHABAL, but it is smaller, and it supports different size of output (from 32 to 512 bits).
    • By Ascer
      A couple week ago i've started learning API Interface for different websites. In this fact sometimes you have to encode your "photo" or "document.body" to send request.
      Functions to encode and decode Base64 was already created by others. Unfortunately i have troubles with running it on Windows R 2008, also speed was terrible.
      I try to find alternative way to code data. 
      I've read about Microsoft "XMLDOM" and created a one simple function to Encode / Decode data to base64binary, base64url
      Thanks for Ghads on Wordpress i coverted a part of his lines from VBscript to AutoIT
      ;============================================================================================================================== ; Function: base64($vCode [, $bEncode = True [, $bUrl = False]]) ; ; Description: Decode or Encode $vData using Microsoft.XMLDOM to Base64Binary or Base64Url. ; IMPORTANT! Encoded base64url is without @LF after 72 lines. Some websites may require this. ; ; Parameter(s): $vData - string or integer | Data to encode or decode. ; $bEncode - boolean | True - encode, False - decode. ; $bUrl - boolean | True - output is will decoded or encoded using base64url shema. ; ; Return Value(s): On Success - Returns output data ; On Failure - Returns 1 - Failed to create object. ; ; Author (s): (Ghads on Wordpress.com), Ascer ;=============================================================================================================================== Func base64($vCode, $bEncode = True, $bUrl = False) Local $oDM = ObjCreate("Microsoft.XMLDOM") If Not IsObj($oDM) Then Return SetError(1, 0, 1) Local $oEL = $oDM.createElement("Tmp") $oEL.DataType = "bin.base64" If $bEncode then $oEL.NodeTypedValue = Binary($vCode) If Not $bUrl Then Return $oEL.Text Return StringReplace(StringReplace(StringReplace($oEL.Text, "+", "-"),"/", "_"), @LF, "") Else If $bUrl Then $vCode = StringReplace(StringReplace($vCode, "-", "+"), "_", "/") $oEL.Text = $vCode Return $oEL.NodeTypedValue EndIf EndFunc ;==>base64  
  • Create New...