"Hex dump" conversion to different datatypes?

[nire5]

I'm trying to take different chunks of hex from UDP packets and convert them to their corresponding datatypes, each packet has a different structure, a couple examples here-

Hex: "04 00 E5 30 07 E0 01 01 0 "

and the structure for datatypes would be short;int;short;byte

another type is

Hex: "02 00 76 8D 09 B5 0C 62 51 0B 00 00 01 00 "

short;int;long

I'm thinking that this is what DllStruct functions are for, but I'm not 100% sure?

[Nine]

Not sure exactly what is your request or issue.  But the way you are presenting it, I do not believe you need DllStruct.  Normally, when you receive binary UDP packets (with AutoIt UDP functions), its basic structure is a string starting with 0x.  What you would do is extract from that string the number of characters you need and convert it to numbers using Dec () function. You may need to invert the order of the bytes received depending on how it was sent to you.  For example you would do $FirstShortRecv = Dec ("0004")

 

[nire5]

My issue is that the packets are all in different formats, some are a byte and a short followed by a varying amount of floats, some are a series of ascii.

[Nine]

I see.  I don't know if there is a UDF that could help you perform those kinds of operations.  Nevertheless, you will have to create a different extraction pattern for each packets.  Lot of work in front of you.  I still think that DllStruct will not help you much, but I may be wrong...

[nire5]

Thanks for the help. I suppose I will have to make my own dissector using stringmid or something. Oh well.

[TheXman]

@nire5

Communication between 2 devices almost always has a defined protocol.  Within that definition should be what the data should look like.  In order to for you to even begin to get any help, you will need to provide a much more detailed description of what you need help with.  The more details that you can provide in terms of what you need, the more likely you will be to get someone to offer some help.  What you have provided so far is almost useless in terms of trying to provide some direction or guidance. 

In general, the specifications that define your data packets will help you determine whether DllStructs can be used, whether it may be easier to parse the binary packets more programmatically/granularly, or whether you will need a combination of both techniques.  One of the most important pieces of information is whether the numeric values in the binary packets are coming across as big-endian, little-endian, or a combination of both.  At this point, you haven't provided enough information to be able to tell.

If you are dealing with a well-defined protocol, it would help if you identified it.  It would also help if you showed any your existing attempts to parse the data.  That will allow others to see where you may be having a problem understanding what needs to be done and it will also prevent any unnecessary or duplicate efforts.

Lastly, there are numerous example of binary data processing throughout the forum, assorted UDFs, and help file examples.  I don't want to assume what your level of expertise is.  So I will just say that if you are new to AutoIt, it would greatly benefit you to spend some time familiarizing yourself with the language and its capabilities.  The help file is a great resource as well as the AutoIt Wiki. 

Edited January 5, 2020 by TheXman
fixed a typo

[RTFC]

19 hours ago, Nine said:

I still think that DllStruct will not help you much, but I may be wrong...

Quite wrong; the OP is totally correct in assuming that multiple differently-formatted structs can all be mapped to the same buffer for variant access, as the example below illustrates:

$data1=Binary("0x0400E53007E0010100")
$data2=Binary("0x0200768D09B50C62510B00000100")

; generic buffer for receiving data, larger than the largest packet
$readbuffer=DllStructCreate("align 1;byte[128]")
$mempointer=DllStructGetPtr($readbuffer)

; here we define two different data structs that both map the buffer struct
; note the second parameter of DllStructCreate, where we parse the memory pointer
$struct1=DllStructCreate("align 1;short;int;short;byte",$mempointer)
$struct2=DllStructCreate("align 1;short;int;long",$mempointer)

; writing packet 1
DllStructSetData($readbuffer,1,$data1)

; reading packet interpreted as type 1
; note that in AutoIt, struct indices are base-1
$var1=DllStructGetData($struct1,1)
$var2=DllStructGetData($struct1,2)
$var3=DllStructGetData($struct1,3)
$var4=DllStructGetData($struct1,4)

ConsoleWrite($var1 & ", " & $var2 & ", " & $var3 & ", " & $var4 & @CRLF)

; writing packet 2
DllStructSetData($readbuffer,1,$data2)

; reading packet interpreted as type 2
$var5=DllStructGetData($struct2,1)
$var6=DllStructGetData($struct2,2)
$var7=DllStructGetData($struct2,3)

ConsoleWrite($var5 & ", " & $var6 & ", " & $var7 & @CRLF)

Of course, proper interpretation of the data does require knowing litte/big-endian byte order of the parsed data, as TheXman already pointed out.

Edited January 6, 2020 by RTFC

[Nine]

1 hour ago, RTFC said:

Quite wrong

Ya I was.  Overlooked that way 

[markyrocks]

I was kinda running up against a situation like this.  I have since resolved the situation by basically just making a dummy struct at the same address similar to the above example.  

But it's very fascinating to see the binary sent to the core memory address and then seeing 2 overlapping structs used as interpreters to pull the data out.  

Now riddle me this.  If I had say a pointer to a struct, and did a memory read on the pointer address.  Could i take that hex information,  convert it to binary, store it in a "buffer" and pull it back out if i know the tag information?  This is all redundant bc obviously just making another struct at the pointer is the way to go but I'm just talking about theory.

Like if i pulled out the hex, converted to binary,  got the size of the binary.  Made a new struct as the buffer, made a second struct with just the "struct" tag or maybe "struct (size) ;endstruct" at that pointer.  Sent the binary the first.  If I pulled the data out of the second would it return a variable as type dllstruct? That I could then use that variable and original pointer to look back at the original struct that I only had the pointer to... lol I know it's a paradox. 

I also find this interesting bc it's a way to use c++ style pointers by placing a variable inside a struct pulling out the pointer via dllgetptr(struct,element) then you can read the underlying data provided you know the data type and size via creating another struct at that pointer and just retrieving it via dllstructget.  This is also something I was beating my head up against the wall about. 

 

 

 

Edited January 6, 2020 by markyrocks

[RTFC]

1 hour ago, Nine said:

Overlooked that way

No worries; you were quite careful in stating your reservations.

@markyrocks: a few remarks for clarification:

• a struct stores data in binary, not hex. AutoIt just displays binary data as hex strings, which can be confusing
• you don't need a second struct to map a particular variable inside a multivar struct; if you study the documentation for DllstructCreate, you'll see that each indexed item of data can (in addition to its type definition) have a named tag (text string), that can be used to retrieve its contents later. Paraphrasing the example there:

• ; Assign a Local constant variable the definition of a structure (read carefully the DllStructCreate remarks).
      Local Const $tagSTRUCT1 = "struct;int var1;byte var2;uint var3;char var4[128];endstruct"
  ;...    
       DllStructSetData($tSTRUCT1, "var2", 123)
  ;...
      $var2 = DllStructGetData($tSTRUCT1, "var2")

• You have to distinguish between the struct itself (a contiguous region in memory at a known address and alignment, which can contain data content of any kind, stored in binary form) and the data you write to/read from it. You cannot "retrieve a variable of type dllstruct" when reading (part of) a struct, you're just retrieving data from that memory location, which you can then store in another named variable for further use.

[markyrocks]

I'm talking about nested structs.  I've been playing with them for days so I'm pretty well aware of how they work etc.  Like I said this is theory.

StructA has a variable stored in it's that is a pointer to structB.  Got the size of structB 

If I made $pStructB=dllgetdata(StructA,element) which is a pointer to structB.  

Then I opened the process, module, made a buffer etc, used winapi_readprocessmemory() to read the pointer from start to size of structB.

Could I then convert the result to binary,  create structC, use the correct tag, alignment ect set that data to StuctC.  Then make StructD at the same memory pointer as StructC but with "struct(size);" then retrieve the data and have it return the variable that is of dllstruct type.  Hence I could then use that variable to retrieve data from StructB via dllstructgetdata()?

I have successfully made nested structs using pointers and dummy structs to pull out the data.  What I'm describing is a very round about way to do this but I'm just curious.   I'm not well versed in the ways of hex or binary but I'm catching on.

edit, I'm not trying high jack the thread.  I just wanted to clarify.

Edited January 6, 2020 by markyrocks

[nire5]

On 1/6/2020 at 8:59 AM, RTFC said:

Quite wrong; the OP is totally correct in assuming that multiple differently-formatted structs can all be mapped to the same buffer for variant access, as the example below illustrates:

$data1=Binary("0x0400E53007E0010100")
$data2=Binary("0x0200768D09B50C62510B00000100")

; generic buffer for receiving data, larger than the largest packet
$readbuffer=DllStructCreate("align 1;byte[128]")
$mempointer=DllStructGetPtr($readbuffer)

; here we define two different data structs that both map the buffer struct
; note the second parameter of DllStructCreate, where we parse the memory pointer
$struct1=DllStructCreate("align 1;short;int;short;byte",$mempointer)
$struct2=DllStructCreate("align 1;short;int;long",$mempointer)

; writing packet 1
DllStructSetData($readbuffer,1,$data1)

; reading packet interpreted as type 1
; note that in AutoIt, struct indices are base-1
$var1=DllStructGetData($struct1,1)
$var2=DllStructGetData($struct1,2)
$var3=DllStructGetData($struct1,3)
$var4=DllStructGetData($struct1,4)

ConsoleWrite($var1 & ", " & $var2 & ", " & $var3 & ", " & $var4 & @CRLF)

; writing packet 2
DllStructSetData($readbuffer,1,$data2)

; reading packet interpreted as type 2
$var5=DllStructGetData($struct2,1)
$var6=DllStructGetData($struct2,2)
$var7=DllStructGetData($struct2,3)

ConsoleWrite($var5 & ", " & $var6 & ", " & $var7 & @CRLF)

Of course, proper interpretation of the data does require knowing litte/big-endian byte order of the parsed data, as TheXman already pointed out.

 

This works perfectly. Thank you very much for this, I've worked out which bytes are little/big endian and I am converting them with a function as needed per opcode template.

 

Two things I'm struggling with-

Some packets have a "list", usually one of the first few bytes defines how large it is. I'm not sure you how you use DllStructGetData in a for loop, or what exactly what your $mempointer was doing.

Also trying to figure out how to size strings, the length of the string when not unicode is usually in the "list".

Here is a struct in a hex editor for this specific opcode-

Struct[Destinations]{byte[Type],byte[Separator],int[Value]}
String
byte
int[entry count - list size]
Destinations[Param,-1]

73 6f 63 6b 65 74 44 65 73 74 69 6e 61 74 69 6f 6e 73 00 00 08 00 00 00 01 20 00 01 00 00 01 20 01 01 00 00 01 20 02 01 00 00 01 20 03 01 00 00 01 20 04 01 00 00 01 20 05 01 00 00 01 20 06 01 00 00 01 20 07 01 00 00

 

and in Hex-

[RTFC]

8 hours ago, nire5 said:

Some packets have a "list", usually one of the first few bytes defines how large it is. I'm not sure you how you use DllStructGetData in a for loop,

The 2nd parameter of DllstructGetData is the (base-1) element index, so I would map a separate struct to the start of that list of (same-sized) entries that contains a sufficient number of slots to accommodate however many can get sent, and then use the actual (sent) entry count for a particular list to define the extent of your for-loop, which then iterates through all items to retrieve using the element index of DllstructGetData, from 1 to <entry count>.

8 hours ago, nire5 said:

what exactly what your $mempointer was doing

$mempointer=DllStructGetPtr($readbuffer)

This line simply gets the memory address of the first byte of the read-buffer defined in the line above it. Note that there, DllstructCreate is called with a single parameter (the struct definition string), which causes a new region in memory to be allocated for storage (starting at the address stored thereafter in $mempointer). The key difference with the subsequent DllStructCreate calls is that these do use the (optional) 2nd parameter (a memory pointer), which causes a new struct definition to be mapped to the same physical starting address (stored in $mempointer) as the first struct definition. So once we know where our data stream lives (stored in our read-buffer), we can interpret whatever lives there in any number of ways. So DllStructCreate can perform two different functions, depending on how you call it, which is kinda neat.

[argumentum]

On 1/5/2020 at 1:19 PM, nire5 said:

My issue is that the packets are all in different formats, some are a byte and a short followed by a varying amount of floats, some are a series of ascii.

so the issue was that the flag ( UDPRecv ( socketarray, maxlen [, flag = 0] ) was Zero instead of One ( UDP_DATA_BINARY (1) - return binary data )  

Edit: I'd use _ArrayToString() to send and _ArrayFromString() once received. All you really want to do is IPC via UDP so I believe this is the simplest way to go about it  

Edited January 26, 2020 by argumentum