I'm trying to figure out a method to translate one or multiple USHORT values into one large bitfield and i am slowly getting a headache figuring out an effective way to perform this.
The bitfield tells my routine which columns entities are *not* present in a record in the database structure i am reading (The so-called NULL values).
This also means that the byte-block beloging to that field, is not present within that file. This is a very simple form of low-level database compression technique.
I have several different database files, and they contain different amounts of columns with various mixed data-type forms.
The logic of the bit-range is that for each group of columns, one bitfield is stuffed into a USHORT, once the boundaries of the lower logic are reached, the bitfield continues in a next set of a USHORT value.
The USHORT blocks are stored in low to high order order inside the file and have to be swapped to get the correct binary representation.
Below snippet displays the details of the structure, starting with the (in this case) three USHORT bitfields.
The next range of figures is the binary representation of these bitfields *when* shifted (i have done this all manually) and combined in the proper order.
Some of the bitranges are between parentheses: when a column represents a char range, the database structure has two types of bit definitions:odd (01) for numeric fields and even (10) for char fields.
bit1 bit2 bit3 bit-range representation of bitfields 3-1 Col-Title column-type 0100 0000 0000 00000000000000000000000000000000000000000001 Column1 (DWORD) 0800 0000 0000 (00000000000000000000000000000000000000001000)Column2 (CHAR) 2000 0000 0000 (00000000000000000000000000000000000000100000)Column3 (CHAR) 8000 0000 0000 (00000000000000000000000000000000000010000000)Column4 (CHAR) 0002 0000 0000 (00000000000000000000000000000000001000000000)Column5 (CHAR) 0004 0000 0000 00000000000000000000000000000000010000000000 Column6 (DWORD) 0010 0000 0000 00000000000000000000000000000001000000000000 Column7 (DWORD) 0040 0000 0000 00000000000000000000000000000100000000000000 Column8 (DWORD) 0000 0100 0000 00000000000000000000000000010000000000000000 Column9 (DWORD) 0000 0400 0000 00000000000000000000000001000000000000000000 Columna (DWORD) 0000 1000 0000 00000000000000000000000100000000000000000000 Columnb (BOOLEAN) 0000 4000 0000 00000000000000000000010000000000000000000000 Columnc (BOOLEAN) 0000 0001 0000 00000000000000000001000000000000000000000000 Columnd (BOOLEAN) 0000 0004 0000 00000000000000000100000000000000000000000000 Columne (BOOLEAN) 0000 0010 0000 00000000000000010000000000000000000000000000 Columnf (BOOLEAN) 0000 0040 0000 00000000000001000000000000000000000000000000 Column10 (DWORD) 0000 0000 0100 00000000000100000000000000000000000000000000 Column11 (DWORD) 0000 0000 0800 (00000000100000000000000000000000000000000000)Column12 (CHAR) 0000 0000 1000 00000001000000000000000000000000000000000000 Column13 (DWORD) 0000 0000 8000 (00001000000000000000000000000000000000000000)Column14 (CHAR) 0000 0000 0002 00100000000000000000000000000000000000000000 Column15 (CHAR) Each database does have a fixed column definition setting, but the order and field-types (char/numeric) within that table can vary per database file.
So figuring out which field is or isn´t present without necessarily knowing it is a char/type or numeric i am using a BitAND() construction with bit pairs of "11" which are in the order of 0x03, 0x0c, 0x30, 0xc0, 0x300, 0xc00, etc.
Regardless of the field type, i would get in that case always a difference if this field is marked as "nul"
I do know my way around reading 16-bit binaries and getting past the "no-consecutive 32-bit datablocks" issue when using WINAPI_ReadFile() but regarding how to transform multiple USHORT values into one large bitfield using bitshifting methods: I could use some crash course here since depending on the column number i have to construct either a USHORT, ULONG or even a 64/bit hex value...
I know WINAPI_ReadFile() performs the endian correction for the USHORT value, but when i have multiple of these pairs, i need to perform some multiplications (with risk of crashing the program by exceeding a variable's type value limitation) to add these numbers up and i have the feeling that using bitshifting, works faster and more effective but....
What is the most effective way to combine these USHORT values into one ULONG/SYSTEMLong value?
I was just working on a project that involved decoding a stream of binary data from a serial port in AutoIt. It took me a few hours to figure out how to process the data efficiently in AutoIt and I did not find any helpful examples on how to do so, so I thought I would share my core example and maybe save someone else some time. There may be a more efficient way to do this, but this works well for me.
#cs Author: ToasterKing This is an example of a way to parse streaming binary data that follows a strict format with a header and footer. In this example, each frame is 5 bytes with a 2-byte header of 0xD5AA and a 1-byte footer of 0xAD. The _BinaryParse() function accumulates incoming data in a buffer. Once a footer is found, it searches backward for the header, and if it is in the right position, it extracts the remaining 2 bytes in the middle, then moves on to looking for the next frame. #ce ; The data source might be something asynchronous like serial or TCP, but since this is just an example, I'm just putting the data in a variable. Local $fSomeData $fSomeData = Binary("0xD5AA24B1") ; Binary data constituting almost a complete frame. _BinaryParse($fSomeData) ; Call the function with the received data. It isn't a complete frame, so it is just stored in the buffer until more data is received. $fSomeData = Binary("0xAD62D5AA92E7AD") ; Remainder of the previous frame, one garbage byte (0x62) which should be skipped, and a complete additional frame. _BinaryParse($fSomeData) ; The function should be able to parse both frames now. Func _BinaryParse($fNewData) Local Static $fBinaryReceived = Binary("") ; Buffer for received data ConsoleWrite("Hey, the function is called!" & @CRLF) ; Add new data to the buffer. ; This ridiculous monstrosity is the only way I could find to append binary data to binary data in AutoIt. It must be converted to strings first. ; Both, one, or no substrings will begin with "0x" depending on whether they contained binary data. To be converted back to binary properly, only one instance ; of "0x" must exist at the beginning of the string. $fBinaryReceived = Binary("0x" & StringReplace(String($fBinaryReceived) & String($fNewData),"0x","")) ConsoleWrite("Data in the buffer: " & String($fBinaryReceived) & @CRLF) Local $iLength = BinaryLen($fBinaryReceived) ; Count the bytes in the data If $iLength > 0 Then Local $fBinaryReceivedTemp = $fBinaryReceived ; Create temporary copy to work on Local $fByte1,$fByte2 For $i = 1 To $iLength If BinaryMid($fBinaryReceivedTemp,$i,1) = 0xAD Then ; If the 1-byte footer found ConsoleWrite("Footer found at end of " & $i & " of " & $iLength & " bytes!" & @CRLF) If BinaryMid($fBinaryReceivedTemp,$i - 4,1) = 0xD5 And BinaryMid($fBinaryReceivedTemp,$i - 3,1) = 0xAA Then ; and the 2-byte header is found 4 bytes before that ConsoleWrite("Header found before the footer!" & @CRLF) $fByte1 = BinaryMid($fBinaryReceivedTemp,$i - 2,1) ; Get 1st byte in the body (between header and footer) $fByte2 = BinaryMid($fBinaryReceivedTemp,$i - 1,1) ; Get 2nd byte in the body (between header and footer) ConsoleWrite("Here is the critical data: " & String($fByte1) & " " & String($fByte2) & @CRLF) ; Just display the 2 bytes for demonstration purposes. Normally, you'd do something more useful with it here. EndIf $fBinaryReceived = BinaryMid($fBinaryReceivedTemp,$i + 1) ; Truncate the original data to remove all of the bytes just processed, then continue processing $fBinaryReceivedTemp EndIf Next EndIf EndFunc