Internal handling 64-bit integers (uint64)

[wraithdu 69]

I'm having trouble using the VerSetConditionMask() function to prepare the bitmask for the VerifyVersionInfo() function. The mask input and the return value of this function are both 64-bit integers. I think the returned 64-bit integer is being internally mishandled, the high and low parts of the integer look like they're being reversed. If this is really a bug in AutoIt, I'll post a Trac ticket, but I wanted to confirm it first.

Global Const $VER_PLATFORMID = 0x8
Global Const $VER_EQUAL = 1

$dwlConditionalMask = 0
$ret = DllCall("kernel32.dll", "uint64", "VerSetConditionMask", "uint64", $dwlConditionalMask, "dword", $VER_PLATFORMID, "byte", $VER_EQUAL)
$dwlConditionalMask = $ret[0]
;; the following value should be 9223372036854776320 (0x8000000000000200), but returns 2201170739200 (0x20080000000) instead
;; looks like the high and low parts of the 64-bit integer are reversed
ConsoleWrite($dwlConditionalMask & @CRLF)

Edited April 17, 2009 by wraithdu

[wraithdu 69]

I threw together a full script with a hack I just came up with to correct the return value. Seems that ConsoleWrite() doesn't print uint64's correctly either (it will print a uint correctly though).

expandcollapse popup

Global Const $OSVERSIONINFOEXW = "dword dwOSVersionInfoSize;dword dwMajorVersion;dword dwMinorVersion;dword dwBuildNumber;dword dwPlatformId;" & _
                                "wchar szCSDVersion[128];ushort wServicePackMajor;ushort wServicePackMinor;ushort wSuiteMask;byte wProductType;byte wReserved"
Global Const $VER_PLATFORMID = 0x8
Global Const $VER_EQUAL = 1

; initialize structure
$OSVI = DllStructCreate($OSVERSIONINFOEXW)
DllStructSetData($OSVI, "dwOSVersionInfoSize", DllStructGetSize($OSVI))
; set data we want to compare
DllStructSetData($OSVI, "dwPlatformId", 2)

; structs for return value
$longlong1 = DllStructCreate("byte[8]")
$longlong2 = DllStructCreate("uint64", DllStructGetPtr($longlong1))
$longlong3 = DllStructCreate("int;int", DllStructGetPtr($longlong1))

; initialize and set the mask
$dwlConditionalMask = 0
$ret = DllCall("kernel32.dll", "uint64", "VerSetConditionMask", "uint64", $dwlConditionalMask, "dword", $VER_PLATFORMID, "byte", $VER_EQUAL)
$dwlConditionalMask = $ret[0]
;; the following value should be 9223372036854776320 (0x8000000000000200), but returns 2201170739200 (0x20080000000) instead
;; looks like the high and low parts of the 64-bit integer are reversed
ConsoleWrite("wrong value:  " & $dwlConditionalMask & @CRLF)
;; function fails
$ret = DllCall("kernel32.dll", "int", "VerifyVersionInfoW", "ptr", DllStructGetPtr($OSVI), "dword", $VER_PLATFORMID, "uint64", $dwlConditionalMask)
ConsoleWrite("Success?  " & $ret[0] & @CRLF)

ConsoleWrite("==============" & @CRLF)
ConsoleWrite("Conversion??" & @CRLF)
DllStructSetData($longlong1, 1, $dwlConditionalMask)
ConsoleWrite("Binary Data (Little Endian):  " & DllStructGetData($longlong1, 1) & @CRLF)
ConsoleWrite("As uint64:  " & DllStructGetData($longlong2, 1) & @CRLF)
ConsoleWrite("Low Part:  " & Hex(DllStructGetData($longlong3, 1)) & @CRLF)
ConsoleWrite("High Part:  " & Hex(DllStructGetData($longlong3, 2)) & @CRLF)
;; conversion??
$temp = DllStructGetData($longlong3, 1)
DllStructSetData($longlong3, 1, DllStructGetData($longlong3, 2))
DllStructSetData($longlong3, 2, $temp)
;; correct value
$dwlConditionalMask = DllStructGetData($longlong2, 1)
ConsoleWrite("Corrected Binary (Little Endian):  " & DllStructGetData($longlong1, 1) & @CRLF)
ConsoleWrite("Not sure why this is printing as a signed int64..." & @CRLF)
ConsoleWrite("As uint64:  " & $dwlConditionalMask & @CRLF)
ConsoleWrite("==============" & @CRLF)

;; function works now
$ret = DllCall("kernel32.dll", "int", "VerifyVersionInfoW", "ptr", DllStructGetPtr($OSVI), "dword", $VER_PLATFORMID, "uint64", $dwlConditionalMask)
ConsoleWrite("Success?  " & $ret[0] & @CRLF)

Edited April 17, 2009 by wraithdu

[ProgAndy 66]

In my UDF for libmysql i use this func to fix the return values:

;===============================================================================
;
; Function Name:   __MySQL_ReOrderULONGLONG
; Description::    INTERNAL USE
; Parameter(s):
; Requirement(s):  libmysql.dll
; Return Value(s):
; Author(s):       Prog@ndy
;
;===============================================================================
;
Func __MySQL_ReOrderULONGLONG($UINT64)
    Local $int = DllStructCreate("uint64")
    Local $longlong = DllStructCreate("ulong;ulong", DllStructGetPtr($int))
    DllStructSetData($int, 1, $UINT64)
    Return 4294967296 * DllStructGetData($longlong, 1) + DllStructGetData($longlong, 2)
EndFunc   ;==>__MySQL_ReOrderULONGLONG

//Edit: in the forum i saw some more problems with this. I think it's time for a ticket

Edited April 17, 2009 by ProgAndy

[wraithdu 69]

I wonder though, does this happen only on return values from functions, or also when giving a parameter when calling a function? During other internal calculations regarding (u)int64's? There's no way for users to know for sure in the last two scenarios I don't think.

Edited April 17, 2009 by wraithdu

[ProgAndy 66]

Well, you use the uint64 as param in the next func do you?

1) You get the uint64 in wrong order

2) Then You use it as param

.... -> if it was given in wrong order, it should be right again ( change order 2 times -> original order)

.... -> since this is not the case and we have to reorder it manually, so uint64 as param works right.

It is the return value wich has wrong order and not the param: I found in the BASS.dll thread, that for smaller values "int" instead of "int64" returns the correct value.

Edited April 17, 2009 by ProgAndy

[wraithdu 69]

Good point...

[ProgAndy 66]

I have written some code to prove the mishandling:

freebasic-DLL:

#include "windows.bi"
declare Function      test lib "test" alias "test"(zahl as ULongInt) as LongInt
 function test(zahl as ULongInt) as LongInt Export
    MessageBox(0, "Expected: 99123456789" & chr(13, 10) & "Recieved: " & Str(zahl), "INT64 test: param", 0)
    Return -999888777666
 end function

AutoIt:

$ret = DllCall("C:\Programme\FreeBASIC\FbEdit\Projects\test\test.dll","int64", "test@8", "uint64", 99123456789)

MsgBox(0, 'REturn', "Expected: -999888777666" & @CRLF & "Received: " & $ret[0] & @CRLF & "Reordered: " & __MySQL_ReOrderULONGLONG($ret[0]))

Func __MySQL_ReOrderULONGLONG($UINT64)
    ; Prog@ndy
    Local $int = DllStructCreate("uint64")
    Local $longlong = DllStructCreate("ulong;ulong", DllStructGetPtr($int))
    DllStructSetData($int, 1, $UINT64)
    Return 4294967296 * DllStructGetData($longlong, 1) + DllStructGetData($longlong, 2)
EndFunc   ;==>__MySQL_ReOrderULONGLONG

//Edit: ULongInt in freebasic is uint64 in AutoIt, LongInt is int64

Edited April 17, 2009 by ProgAndy

[SkinnyWhiteGuy 2]

To quote the help file:

Language Reference - Datatypes

Datatypes and Ranges

The following table shows the internal variant datatypes and their ranges.

Data Sub-type -- Range and Notes

Int32 -- A 32bit signed integer number.

Int64 -- A 64bit signed integer number

Double -- A double-precision floating point number.

String -- Can contain strings of up to 2147483647 characters.

Binary -- Binary data, can contain up to 2147483647 bytes.

Pointer -- A memory address pointer. 32bit or 64bit depending on the version of AutoIt used.

Some functions in AutoIt only work with 32 bit numbers (e.g. BitAND) and are converted automatically - these functions are documented where required.

The highest integer value an AutoIt variable can hold is 9,223,372,036,854,775,807 (or 0x7FFFFFFFFFFFFFFF in Hex). After that, it wraps to negative numbers. I tried to use unsigned 64-bit integers myself at one point, and the AutoIt variant could hold the value, but not display it correctly for me. If you use that returned value in other Dll functions that expect it (without modifying it in AutoIt), all will work.

If you need to do actual calculations on unsigned 64-bit integers, you either have to do it in your own Dll, or work around it. If you want to work around it, I have a few ways I discovered to do it, but they are all ugly to have to use.

[wraithdu 69]

If you use that returned value in other Dll functions that expect it (without modifying it in AutoIt), all will work.

Hmm, not true though. Try my second example after removing the conversion code. Passing the return from VerSetConditionMask() directly to VerifyVersionInfo() fails because the returned value is wrong. The high and low parts are reversed. Edited April 17, 2009 by wraithdu

[Ascend4nt 118]

I've noticed the same problems with 64-bit numbers as well. AutoIT provides the 'uint64' datatype but only supports signed 64-bit numbers, which might seem to function okay - until the uppermost bit is set, that is. In my sig. there are special 64-bit base-2/base-16 conversions UDF's that 'pop' the top bit off in order to work correctly, and the same basic idea can be used to 'set' the top bit - but that's only useful when you have numbers that don't exceed 9,223,372,036,854,775,807 (max unsigned value as SkinnyWhiteGuy pointed out).

It would be nice to have the 64-bit integer issue either documented or fixed though.

[wraithdu 69]

@ascendant

Is that related to the problem with returning a (u)int64 from a DllCall (reversing the high and low parts)?

Either way, if a dev doesn't chime in with any objections, I'll start a Trac ticket for at least my issue mentioned above, since I'm not sure exactly what you mean when you say they "function okay - until the uppermost bit is set" (I'm unclear on the 'function okay' part). Perhaps you should clarify the problem and submit a ticket as well?

[Valik 471]

Personally I'm not happy with the way numbers are handled. 64-bit is sort of a tacked on extra that feels exactly that. Unsigned versus signed is sometimes an important distinction that needs made but AutoIt stores everything internally as signed and doesn't provide an easy way to work with unsigned numbers.

With that being said, is this an issue with AutoIt's less than stellar implementation of numbers or is this a genuine bug? If it's the former, don't waste time creating a ticket because I'll just close it and/or re-write it completely (and then I will ignore it for years). However, if you can demonstrate there is an issue where the data is being mangled, well, that's something that can probably be fixed.

[Ascend4nt 118]

@wraithdu:

I was commenting to the point of 64-bit 'unsigned' integers not being a possibility, not the reversal of low/high-order parts - which I hadn't encountered myself. I actually went back through code where I've used 'uint64' and realize now that I've never used them on 'ULONGLONG' returns, only for 'ULONGLONG's inside DLLStructs and also in replacing the Windows 'FILETIME' structures (since the structure is arranged in little-endian order). So, while only the latter is accessed through the DLL call's return array, its not *technically* a ULONGLONG, and is passed as a pointer anyway ('uint64*').

Now that I've checked your code though, I do get the same result as you. If this can be verified through another language then I'd definitely submit a ticket for it - but only for the DLLCall()'s returns.

64-bit integers in AutoIT are otherwise working as expected - for signed numbers, anyway. When used in a way where the data isn't modified, it doesn't make a difference if 0x8000000000000000 (or 9223372036854775807+1) displays as -9223372036854775808. Its only when math needs to be done that things can get messy.

@Valik:

What about modifying the documentation for DLLCall and DLLStructCreate to indicate that 'uin64' is treated internally as signed integers? That might help avoid confusion in the future.

As for the "less than stellar implementation of numbers", you've already created ticket 767 to cover 64-bit integer support, so I wouldn't add anything further to that.

[jpm 42]

Unless I am wrong,

Uint64 are always displayed as int64 as explain by Valik that the reason why you get a negative value.

Definetly something wrong with the return of an uint64 by dllcall.

The return value of an updated parameter is well handled.

[ProgAndy 66]

As you see in my example above, even a signed int64 as return value returns the wrong number and has to be reordered. In all other places, (u)int64 works fine, just as return value in DLLCall it does not.

If you need to display an unsigned Integer or use it in calculations, you have to use the BigNum.au3

This is a func to convert the signed int64 to unsigned int64:

#include<BigNum.au3>
Func _Int64TounsignedString($Int64)
    Local Const $UINT64_OFFSET = "18446744073709551616"
    Local $Res = _BigNum_Add(String($Int64), $UINT64_OFFSET)
    If @error Then Return SetError(1,0,"")
    Return $Res
EndFunc

MsgBox(0, '', _Int64TounsignedString("-9223372036854775808"))

Edited April 20, 2009 by ProgAndy

[SkinnyWhiteGuy 2]

Here's some more weirdness...

WinAPI documentation lists the type for the dwlConditionMask as ULONGLONG, which in our documentation translates to a uint64. But, according to another section of WinAPI:

ULONGLONG 64-bit unsigned integer.

This type is declared in Winnt.h as follows:

typedef unsigned __int64 ULONGLONG;

#else

typedef double ULONGLONG

Would the values we are seeing be explained by it returning as a double, and not the unsigned __int64 we think we should be getting?

This seems to show the same thing...

Global Const $VER_PLATFORMID = 0x8
Global Const $VER_EQUAL = 1

$dwlConditionalMask = 0
$ret = DllCall("kernel32.dll", "uint64", "VerSetConditionMask", "uint64", $dwlConditionalMask, "dword", $VER_PLATFORMID, "byte", $VER_EQUAL)
$dwlConditionalMask = $ret[0]
;; the following value should be 9223372036854776320 (0x8000000000000200), but returns 2201170739200 (0x20080000000) instead
;; looks like the high and low parts of the 64-bit integer are reversed
ConsoleWrite($dwlConditionalMask & @CRLF)
$dwlConditionalMask = 0
$ret = DllCall("kernel32.dll", "uint64", "VerSetConditionMask", "double", $dwlConditionalMask, "dword", $VER_PLATFORMID, "byte", $VER_EQUAL)
$dwlConditionalMask = $ret[0]
;; the following value should be 9223372036854776320 (0x8000000000000200), but returns 2201170739200 (0x20080000000) instead
;; looks like the high and low parts of the 64-bit integer are reversed
ConsoleWrite($dwlConditionalMask & @CRLF)

It seems to treat the input the same (if you change either of the output uint64's to a double, you get a wrong answer).

Edit: And even more curiosities from the Documentation:

Some Windows functions use the LARGE_INTEGER structure to represent 64-bit signed integer values, and the ULARGE_INTEGER structure to specify 64-bit unsigned integer values.

This function doesn't seem to, but we are getting two dword's back in a different order, which this structure would seem to be using (LowPart, then HighPart).

Edited April 20, 2009 by SkinnyWhiteGuy

[jpm 42]

In fact 3.3.0 does not return properly the uint64 (high/low swapping). That will be fix in next Beta/release.

Internally Autoit handle as good as Valik likes the int64.

double/float is not a good solution as it loosing precision against int64.

Uint64 cannot be handle properly inside autoit code so uint64 with 0x8.... will be displayed with a negative value.

[wraithdu 69]

Good to hear, and I'm clear now on the display of uint64's as signed. I won't clutter Trac with another ticket since you are aware of the DllCall return problem. Thanks for looking into it!

[jpm 42]

Good to hear, and I'm clear now on the display of uint64's as signed. I won't clutter Trac with another ticket since you are aware of the DllCall return problem. Thanks for looking into it!

no problem the pb is fix for the next delivery

[geww 0]

It seems this bug isn't fixed yet. All 20 digits numbers what I am using always messing up. Do have anyone fix for that?