Jump to content

Generating a bit-field from three ushort values

Recommended Posts

Hi lads,

I'm trying to figure out a method to translate one or multiple USHORT values into one large bitfield and i am slowly getting a headache figuring out an effective way to perform this.

In general
The bitfield tells my routine which columns entities are *not* present in a record in the database structure i am reading (The so-called NULL values).
This also means that the byte-block beloging to that field, is not present within that file. This is a very simple form of low-level database compression technique.
I have several different database files, and they contain different amounts of columns with various mixed data-type forms.

The logic of the bit-range is that for each group of columns, one bitfield is stuffed into a USHORT, once the boundaries of the lower logic are reached, the bitfield continues in a next set of a USHORT value.
The USHORT blocks are stored in low to high order order inside the file and have to be swapped to get the correct binary representation.

Below snippet displays the details of the structure, starting with the (in this case) three USHORT bitfields.
The next range of figures is the binary representation of these bitfields *when* shifted (i have done this all manually) and combined in the proper order.
Some of the bitranges are between parentheses: when a column represents a char range, the database structure has two types of bit definitions:odd (01) for numeric fields and even (10) for char fields.

bit1 bit2 bit3  bit-range representation of bitfields 3-1    Col-Title column-type
0100 0000 0000  00000000000000000000000000000000000000000001 Column1  (DWORD) 
0800 0000 0000 (00000000000000000000000000000000000000001000)Column2  (CHAR) 
2000 0000 0000 (00000000000000000000000000000000000000100000)Column3  (CHAR)
8000 0000 0000 (00000000000000000000000000000000000010000000)Column4  (CHAR)
0002 0000 0000 (00000000000000000000000000000000001000000000)Column5  (CHAR)
0004 0000 0000  00000000000000000000000000000000010000000000 Column6  (DWORD)
0010 0000 0000  00000000000000000000000000000001000000000000 Column7  (DWORD)
0040 0000 0000  00000000000000000000000000000100000000000000 Column8  (DWORD)
0000 0100 0000  00000000000000000000000000010000000000000000 Column9  (DWORD)
0000 0400 0000  00000000000000000000000001000000000000000000 Columna  (DWORD)
0000 1000 0000  00000000000000000000000100000000000000000000 Columnb  (BOOLEAN)
0000 4000 0000  00000000000000000000010000000000000000000000 Columnc  (BOOLEAN)
0000 0001 0000  00000000000000000001000000000000000000000000 Columnd  (BOOLEAN)
0000 0004 0000  00000000000000000100000000000000000000000000 Columne  (BOOLEAN)
0000 0010 0000  00000000000000010000000000000000000000000000 Columnf  (BOOLEAN)
0000 0040 0000  00000000000001000000000000000000000000000000 Column10 (DWORD)
0000 0000 0100  00000000000100000000000000000000000000000000 Column11 (DWORD)
0000 0000 0800 (00000000100000000000000000000000000000000000)Column12 (CHAR)
0000 0000 1000  00000001000000000000000000000000000000000000 Column13 (DWORD)
0000 0000 8000 (00001000000000000000000000000000000000000000)Column14 (CHAR)
0000 0000 0002  00100000000000000000000000000000000000000000 Column15 (CHAR)

Each database does have a fixed column definition setting, but the order and field-types (char/numeric) within that table can vary per database file.

So figuring out which field is or isn´t present without necessarily knowing it is a char/type or numeric i am using a BitAND() construction with bit pairs of "11" which are in the order of 0x03, 0x0c, 0x30, 0xc0, 0x300, 0xc00, etc.
Regardless of the field type, i would get in that case always a difference if this field is marked as "nul"

My issue
I do know my way around reading 16-bit binaries and getting past the "no-consecutive 32-bit datablocks" issue when using WINAPI_ReadFile() but regarding how to transform multiple USHORT values into one large bitfield using bitshifting methods: I could use some crash course here since depending on the column number i have to construct either a USHORT, ULONG or even a 64/bit hex value...
I know WINAPI_ReadFile() performs the endian correction for the USHORT value, but when i have multiple of these pairs, i need to perform some multiplications (with risk of crashing the program by exceeding  a variable's type value limitation) to add these numbers up and i have the feeling that using bitshifting, works faster and more effective but....
What is the most effective way to combine these USHORT values into one ULONG/SYSTEMLong value?



Link to post
Share on other sites

I'd suggest using DllstructCreate. Something like this (pseudocode, completely untested):

DllstructSetData($shorts,1,<your first short>,1)
DllstructSetData($shorts,1,<your second short>,2)

$remapped=dllstructcreate("uint64",dllstructgetptr($shorts))    ; 4x16 bit = 1x64 bit

<call func _BitTest to test individual bits (NB bit IDs are base-0, right to left)>

Func _BitTest($value, $bit)
    Return BitAND(BitShift($value, $bit),1)


Link to post
Share on other sites

I have attempted something like that earlier, but is a real hassle (and slow), i suspect there is no way to go around it as the bitshifter and bitrotater inside AutoIT go nowhere higher than 32-bit rotation (where i need 64-bit rotation).
And slow as in: when i use DllStruct tables, WinAPI_Readfile() picks up in 32-bit blocks. and with blocks of 3 Ushorts there is lacking one ushort and the DLL reads another extra 2 bytes anyway to compensate the lacking two in memory. This means i have the whole hassle of resetting file-pointers the whole time.


This is my test-code i am battling with so far to get everything bitshifted:

#include <_Primes.au3>
Dim $ByteOrder[2]
$ByteOrder[0] = 0x03
$ByteOrder[1] = 0x0c
Dim $ByteConverted[2]

Local $ByteOrderPosition = 0
Local $LowByte, $HighByte
;Consolewrite (_IntToAny(0x03*0x10,2,(6*4))&@CRLF)
Local $Columns = 24, $Bits = $Columns * 2
Local $Shorts = Ceiling($Bits / 16)
ConsoleWrite("Shorts:" & $Shorts & @CRLF)
Dim $BitBlock[$Shorts]
$BitBlock[0] = 0x000
$BitBlock[1] = 0x000
$BitBlock[2] = 0x020

Local $BinBlock, $raiser
Local $Round = 1
For $x = ($Shorts - 1) To 0 Step -1
    If $Round > 1 Then
        $BinBlock = Int(BitRotate($BinBlock, -16,"D"), 2)
        ConsoleWrite("shifted Binblock:[" & Hex($BinBlock) & "]" & @CRLF)
    ;$BinBlock &= String(_IntToAny($BitBlock[$x],2,20))
    ConsoleWrite("shifted Binblock:[" & Hex($BinBlock) & "] + [" & Hex($BitBlock[$x]) & "]" & @CRLF)
    $BinBlock += $BitBlock[$x]
    $Round += 1
ConsoleWrite("Binblock:[" & $BinBlock & "]" & @CRLF)
ConsoleWrite("Binblock:[" & Hex($BinBlock) & "]" & @CRLF)
;$BinBlock = _BinToInt($BinBlock)
$BinBlock = _HexToInt($BinBlock)
$BinBlock = _IntToBin($BinBlock)
ConsoleWrite("Binblock hex:[" & $BinBlock & "]" & @CRLF)

For $x = 1 To $Columns

    If $x < 3 Then
        $ByteConverted[$ByteOrderPosition] = $ByteOrder[$ByteOrderPosition]
        $ByteConverted[$ByteOrderPosition] *= 0x10

    ConsoleWrite("->" & _IntToAny($ByteConverted[$ByteOrderPosition], 2, $Bits) & " | " & _IntToHex($ByteConverted[$ByteOrderPosition]) & @CRLF)
    If BitAND($ByteConverted[$ByteOrderPosition], $BinBlock) Then
        ConsoleWrite(@CRLF & "Column [" & $x & "] not present" & @CRLF)
    $ByteOrderPosition += 1
    If $ByteOrderPosition > 1 Then
        $ByteOrderPosition = 0

I used the Primes UDF from this topic for the binary  convertions:


Link to post
Share on other sites

@Rtfc, thanks for the suggestion so far;
I have reconsidered to simply do it the old fashioned way it was done in the time of lacking 64-bit variables: simply segment the identification part as well:

#include <_Primes.au3>
Dim $ByteOrder[2]
$ByteOrder[0] = 0x03
$ByteOrder[1] = 0x0c
Dim $ByteConverted[2]

Local $ByteOrderPosition = 0
Local $LowByte, $HighByte

;Testing block start
Local $Columns = 21, $Bits = $Columns * 2
Local $Shorts = Ceiling($Bits / 16) ; -> how many USHORTS form the bit-range?

Dim $BitBlock[$Shorts + 1]
;Just for testing, change any value of the bitblocks into one of these values: 1,2,4,8 e.gg 0x0020, 0x0400, 0x8000
$BitBlock[1] = 0x0000
$BitBlock[2] = 0x0000
$BitBlock[3] = 0x0200
;Testing block end

Local $BinBlock, $BitSet, $FormerSet

For $x = 1 To $Columns
    $BitSet = Ceiling($x / 8)

    If $FormerSet <> $BitSet Then
        $FormerSet = $BitSet
        $ByteConverted[0] = 0
        $ByteConverted[1] = 0

    If $ByteConverted[$ByteOrderPosition] == 0 Then
        $ByteConverted[$ByteOrderPosition] = $ByteOrder[$ByteOrderPosition]
        $ByteConverted[$ByteOrderPosition] *= 0x10

    ConsoleWrite($BitSet & "->" & _IntToAny($ByteConverted[$ByteOrderPosition], 2, $Bits) & " | " &         _IntToHex($ByteConverted[$ByteOrderPosition]) & @CRLF)

    If BitAND($ByteConverted[$ByteOrderPosition], $BitBlock[$BitSet]) Then
        ConsoleWrite(@CRLF & "Column [" & $x & "] not present" & @CRLF)

    $ByteOrderPosition += 1

    If $ByteOrderPosition > 1 Then
        $ByteOrderPosition = 0


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By Colduction
      Hi AutoIt programmers, excuse me for bothering you with multiple topics.

      In AutoIt we can use Number() function to convert Hex string to number but it's output is different of C# output & and i wanna make it's output like AutoIt code.

      For e.g I use this in AutoIt:
      Local $dBinary = Binary("Hello") ; Create binary data from a string. Local $dExtract = Number(BinaryMid($dBinary, 1, 5)) ConsoleWrite($dExtract & @CRLF) And i use this for C#:
      using System; using System.Text; //NameSpace Is Use of Project Name namespace TEST { class Program { public static void Main(string[] args) { //declaring a variable and assigning hex value string dd = ToHex("Hello", Encoding.ASCII); decimal d = Int64.Parse(dd, System.Globalization.NumberStyles.HexNumber); Console.WriteLine("Result: " + d); //hit ENTER to exit Console.ReadLine(); } public static string ToHex(string sInput, Encoding oEncoding, bool b0x_Prefix = false) { byte[] a_binaryOutput = oEncoding.GetBytes(sInput); string sOutput = BitConverter.ToString(a_binaryOutput).Replace("-", ""); if (b0x_Prefix == false) { return sOutput; } else { return "0x" + sOutput; } } } }
      I say once again that excuse me for creating new topic, in fact i'm making a library for GAuthOTP from a topic in AutoIt.
    • By jmor
      For some reason, I can't convert to Hex a Decimal color that has been calculated.
      For example, this works:
      Local $iColor = 3310546 ConsoleWrite("$iColor (dec): " & $iColor & ", $iColor (hex): " & Hex($iColor, 6) & @CRLF) giving the expected output:
      $iColor (dec): 3310546, $iColor (hex): 3283D2
      But if I want to take an average of several colors, I have a problem.  For example this doesn't work:
      $iColor = (3310546 + 3310546) / 2 ConsoleWrite("$iColor (dec): " & $iColor & ", $iColor (hex): " & Hex($iColor, 6) & @CRLF) giving the wrong output:
      $iColor (dec): 3310546, $iColor (hex): 000000
      I observe this behavior after any processing of a decimal color.
      Is this a bug?
    • By badcoder123
      Hey, all.  
      I've been looking for a way to change cursor colour but not the cursor itself. I've been looking for a couple hours now and can't find anything.  I also don't even know where to start, if anyone has any tips or examples please comment them.
    • By rootx
      I need help to read one file in a HEX mode and go to one offset and than read the value. THX

    • By Altor
      Hi all:
      I nedd to convert a word hex number to an integer 16
      Ex: 0xFFE9 convert to -23
      I would appreciate if they had any idea how I can get it
  • Create New...