Jump to content
Sign in to follow this  

Generating a bit-field from three ushort values

Recommended Posts

Hi lads,

I'm trying to figure out a method to translate one or multiple USHORT values into one large bitfield and i am slowly getting a headache figuring out an effective way to perform this.

In general
The bitfield tells my routine which columns entities are *not* present in a record in the database structure i am reading (The so-called NULL values).
This also means that the byte-block beloging to that field, is not present within that file. This is a very simple form of low-level database compression technique.
I have several different database files, and they contain different amounts of columns with various mixed data-type forms.

The logic of the bit-range is that for each group of columns, one bitfield is stuffed into a USHORT, once the boundaries of the lower logic are reached, the bitfield continues in a next set of a USHORT value.
The USHORT blocks are stored in low to high order order inside the file and have to be swapped to get the correct binary representation.

Below snippet displays the details of the structure, starting with the (in this case) three USHORT bitfields.
The next range of figures is the binary representation of these bitfields *when* shifted (i have done this all manually) and combined in the proper order.
Some of the bitranges are between parentheses: when a column represents a char range, the database structure has two types of bit definitions:odd (01) for numeric fields and even (10) for char fields.

bit1 bit2 bit3  bit-range representation of bitfields 3-1    Col-Title column-type
0100 0000 0000  00000000000000000000000000000000000000000001 Column1  (DWORD) 
0800 0000 0000 (00000000000000000000000000000000000000001000)Column2  (CHAR) 
2000 0000 0000 (00000000000000000000000000000000000000100000)Column3  (CHAR)
8000 0000 0000 (00000000000000000000000000000000000010000000)Column4  (CHAR)
0002 0000 0000 (00000000000000000000000000000000001000000000)Column5  (CHAR)
0004 0000 0000  00000000000000000000000000000000010000000000 Column6  (DWORD)
0010 0000 0000  00000000000000000000000000000001000000000000 Column7  (DWORD)
0040 0000 0000  00000000000000000000000000000100000000000000 Column8  (DWORD)
0000 0100 0000  00000000000000000000000000010000000000000000 Column9  (DWORD)
0000 0400 0000  00000000000000000000000001000000000000000000 Columna  (DWORD)
0000 1000 0000  00000000000000000000000100000000000000000000 Columnb  (BOOLEAN)
0000 4000 0000  00000000000000000000010000000000000000000000 Columnc  (BOOLEAN)
0000 0001 0000  00000000000000000001000000000000000000000000 Columnd  (BOOLEAN)
0000 0004 0000  00000000000000000100000000000000000000000000 Columne  (BOOLEAN)
0000 0010 0000  00000000000000010000000000000000000000000000 Columnf  (BOOLEAN)
0000 0040 0000  00000000000001000000000000000000000000000000 Column10 (DWORD)
0000 0000 0100  00000000000100000000000000000000000000000000 Column11 (DWORD)
0000 0000 0800 (00000000100000000000000000000000000000000000)Column12 (CHAR)
0000 0000 1000  00000001000000000000000000000000000000000000 Column13 (DWORD)
0000 0000 8000 (00001000000000000000000000000000000000000000)Column14 (CHAR)
0000 0000 0002  00100000000000000000000000000000000000000000 Column15 (CHAR)

Each database does have a fixed column definition setting, but the order and field-types (char/numeric) within that table can vary per database file.

So figuring out which field is or isn´t present without necessarily knowing it is a char/type or numeric i am using a BitAND() construction with bit pairs of "11" which are in the order of 0x03, 0x0c, 0x30, 0xc0, 0x300, 0xc00, etc.
Regardless of the field type, i would get in that case always a difference if this field is marked as "nul"

My issue
I do know my way around reading 16-bit binaries and getting past the "no-consecutive 32-bit datablocks" issue when using WINAPI_ReadFile() but regarding how to transform multiple USHORT values into one large bitfield using bitshifting methods: I could use some crash course here since depending on the column number i have to construct either a USHORT, ULONG or even a 64/bit hex value...
I know WINAPI_ReadFile() performs the endian correction for the USHORT value, but when i have multiple of these pairs, i need to perform some multiplications (with risk of crashing the program by exceeding  a variable's type value limitation) to add these numbers up and i have the feeling that using bitshifting, works faster and more effective but....
What is the most effective way to combine these USHORT values into one ULONG/SYSTEMLong value?



Share this post

Link to post
Share on other sites

I'd suggest using DllstructCreate. Something like this (pseudocode, completely untested):

DllstructSetData($shorts,1,<your first short>,1)
DllstructSetData($shorts,1,<your second short>,2)

$remapped=dllstructcreate("uint64",dllstructgetptr($shorts))    ; 4x16 bit = 1x64 bit

<call func _BitTest to test individual bits (NB bit IDs are base-0, right to left)>

Func _BitTest($value, $bit)
    Return BitAND(BitShift($value, $bit),1)


Share this post

Link to post
Share on other sites

I have attempted something like that earlier, but is a real hassle (and slow), i suspect there is no way to go around it as the bitshifter and bitrotater inside AutoIT go nowhere higher than 32-bit rotation (where i need 64-bit rotation).
And slow as in: when i use DllStruct tables, WinAPI_Readfile() picks up in 32-bit blocks. and with blocks of 3 Ushorts there is lacking one ushort and the DLL reads another extra 2 bytes anyway to compensate the lacking two in memory. This means i have the whole hassle of resetting file-pointers the whole time.


This is my test-code i am battling with so far to get everything bitshifted:

#include <_Primes.au3>
Dim $ByteOrder[2]
$ByteOrder[0] = 0x03
$ByteOrder[1] = 0x0c
Dim $ByteConverted[2]

Local $ByteOrderPosition = 0
Local $LowByte, $HighByte
;Consolewrite (_IntToAny(0x03*0x10,2,(6*4))&@CRLF)
Local $Columns = 24, $Bits = $Columns * 2
Local $Shorts = Ceiling($Bits / 16)
ConsoleWrite("Shorts:" & $Shorts & @CRLF)
Dim $BitBlock[$Shorts]
$BitBlock[0] = 0x000
$BitBlock[1] = 0x000
$BitBlock[2] = 0x020

Local $BinBlock, $raiser
Local $Round = 1
For $x = ($Shorts - 1) To 0 Step -1
    If $Round > 1 Then
        $BinBlock = Int(BitRotate($BinBlock, -16,"D"), 2)
        ConsoleWrite("shifted Binblock:[" & Hex($BinBlock) & "]" & @CRLF)
    ;$BinBlock &= String(_IntToAny($BitBlock[$x],2,20))
    ConsoleWrite("shifted Binblock:[" & Hex($BinBlock) & "] + [" & Hex($BitBlock[$x]) & "]" & @CRLF)
    $BinBlock += $BitBlock[$x]
    $Round += 1
ConsoleWrite("Binblock:[" & $BinBlock & "]" & @CRLF)
ConsoleWrite("Binblock:[" & Hex($BinBlock) & "]" & @CRLF)
;$BinBlock = _BinToInt($BinBlock)
$BinBlock = _HexToInt($BinBlock)
$BinBlock = _IntToBin($BinBlock)
ConsoleWrite("Binblock hex:[" & $BinBlock & "]" & @CRLF)

For $x = 1 To $Columns

    If $x < 3 Then
        $ByteConverted[$ByteOrderPosition] = $ByteOrder[$ByteOrderPosition]
        $ByteConverted[$ByteOrderPosition] *= 0x10

    ConsoleWrite("->" & _IntToAny($ByteConverted[$ByteOrderPosition], 2, $Bits) & " | " & _IntToHex($ByteConverted[$ByteOrderPosition]) & @CRLF)
    If BitAND($ByteConverted[$ByteOrderPosition], $BinBlock) Then
        ConsoleWrite(@CRLF & "Column [" & $x & "] not present" & @CRLF)
    $ByteOrderPosition += 1
    If $ByteOrderPosition > 1 Then
        $ByteOrderPosition = 0

I used the Primes UDF from this topic for the binary  convertions:


Share this post

Link to post
Share on other sites

@Rtfc, thanks for the suggestion so far;
I have reconsidered to simply do it the old fashioned way it was done in the time of lacking 64-bit variables: simply segment the identification part as well:

#include <_Primes.au3>
Dim $ByteOrder[2]
$ByteOrder[0] = 0x03
$ByteOrder[1] = 0x0c
Dim $ByteConverted[2]

Local $ByteOrderPosition = 0
Local $LowByte, $HighByte

;Testing block start
Local $Columns = 21, $Bits = $Columns * 2
Local $Shorts = Ceiling($Bits / 16) ; -> how many USHORTS form the bit-range?

Dim $BitBlock[$Shorts + 1]
;Just for testing, change any value of the bitblocks into one of these values: 1,2,4,8 e.gg 0x0020, 0x0400, 0x8000
$BitBlock[1] = 0x0000
$BitBlock[2] = 0x0000
$BitBlock[3] = 0x0200
;Testing block end

Local $BinBlock, $BitSet, $FormerSet

For $x = 1 To $Columns
    $BitSet = Ceiling($x / 8)

    If $FormerSet <> $BitSet Then
        $FormerSet = $BitSet
        $ByteConverted[0] = 0
        $ByteConverted[1] = 0

    If $ByteConverted[$ByteOrderPosition] == 0 Then
        $ByteConverted[$ByteOrderPosition] = $ByteOrder[$ByteOrderPosition]
        $ByteConverted[$ByteOrderPosition] *= 0x10

    ConsoleWrite($BitSet & "->" & _IntToAny($ByteConverted[$ByteOrderPosition], 2, $Bits) & " | " &         _IntToHex($ByteConverted[$ByteOrderPosition]) & @CRLF)

    If BitAND($ByteConverted[$ByteOrderPosition], $BitBlock[$BitSet]) Then
        ConsoleWrite(@CRLF & "Column [" & $x & "] not present" & @CRLF)

    $ByteOrderPosition += 1

    If $ByteOrderPosition > 1 Then
        $ByteOrderPosition = 0


Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By badcoder123
      Hey, all.  
      I've been looking for a way to change cursor colour but not the cursor itself. I've been looking for a couple hours now and can't find anything.  I also don't even know where to start, if anyone has any tips or examples please comment them.
    • By rootx
      I need help to read one file in a HEX mode and go to one offset and than read the value. THX

    • By Altor
      Hi all:
      I nedd to convert a word hex number to an integer 16
      Ex: 0xFFE9 convert to -23
      I would appreciate if they had any idea how I can get it
    • By toasterking
      I was just working on a project that involved decoding a stream of binary data from a serial port in AutoIt.  It took me a few hours to figure out how to process the data efficiently in AutoIt and I did not find any helpful examples on how to do so, so I thought I would share my core example and maybe save someone else some time.  There may be a more efficient way to do this, but this works well for me.
      #cs Author: ToasterKing This is an example of a way to parse streaming binary data that follows a strict format with a header and footer. In this example, each frame is 5 bytes with a 2-byte header of 0xD5AA and a 1-byte footer of 0xAD. The _BinaryParse() function accumulates incoming data in a buffer. Once a footer is found, it searches backward for the header, and if it is in the right position, it extracts the remaining 2 bytes in the middle, then moves on to looking for the next frame. #ce ; The data source might be something asynchronous like serial or TCP, but since this is just an example, I'm just putting the data in a variable. Local $fSomeData $fSomeData = Binary("0xD5AA24B1") ; Binary data constituting almost a complete frame. _BinaryParse($fSomeData) ; Call the function with the received data. It isn't a complete frame, so it is just stored in the buffer until more data is received. $fSomeData = Binary("0xAD62D5AA92E7AD") ; Remainder of the previous frame, one garbage byte (0x62) which should be skipped, and a complete additional frame. _BinaryParse($fSomeData) ; The function should be able to parse both frames now. Func _BinaryParse($fNewData) Local Static $fBinaryReceived = Binary("") ; Buffer for received data ConsoleWrite("Hey, the function is called!" & @CRLF) ; Add new data to the buffer. ; This ridiculous monstrosity is the only way I could find to append binary data to binary data in AutoIt. It must be converted to strings first. ; Both, one, or no substrings will begin with "0x" depending on whether they contained binary data. To be converted back to binary properly, only one instance ; of "0x" must exist at the beginning of the string. $fBinaryReceived = Binary("0x" & StringReplace(String($fBinaryReceived) & String($fNewData),"0x","")) ConsoleWrite("Data in the buffer: " & String($fBinaryReceived) & @CRLF) Local $iLength = BinaryLen($fBinaryReceived) ; Count the bytes in the data If $iLength > 0 Then Local $fBinaryReceivedTemp = $fBinaryReceived ; Create temporary copy to work on Local $fByte1,$fByte2 For $i = 1 To $iLength If BinaryMid($fBinaryReceivedTemp,$i,1) = 0xAD Then ; If the 1-byte footer found ConsoleWrite("Footer found at end of " & $i & " of " & $iLength & " bytes!" & @CRLF) If BinaryMid($fBinaryReceivedTemp,$i - 4,1) = 0xD5 And BinaryMid($fBinaryReceivedTemp,$i - 3,1) = 0xAA Then ; and the 2-byte header is found 4 bytes before that ConsoleWrite("Header found before the footer!" & @CRLF) $fByte1 = BinaryMid($fBinaryReceivedTemp,$i - 2,1) ; Get 1st byte in the body (between header and footer) $fByte2 = BinaryMid($fBinaryReceivedTemp,$i - 1,1) ; Get 2nd byte in the body (between header and footer) ConsoleWrite("Here is the critical data: " & String($fByte1) & " " & String($fByte2) & @CRLF) ; Just display the 2 bytes for demonstration purposes. Normally, you'd do something more useful with it here. EndIf $fBinaryReceived = BinaryMid($fBinaryReceivedTemp,$i + 1) ; Truncate the original data to remove all of the bytes just processed, then continue processing $fBinaryReceivedTemp EndIf Next EndIf EndFunc  
    • By TheDcoder
      Hello Guys!
      I want to convert the binary hash returned by _Crypt_Hash* function to string.
      Thanks in Advance! TD
  • Create New...