Jump to content
Sign in to follow this  
dgm5555

Hex and Bitshift calculation errors

Recommended Posts

dgm5555

Am I doing/expecting something wrong?

Hex division is producing incorrect values

Running from SciTE - AutoIt3Wrapper v.2.1.0.33 (oops: autoit version is 3.3.8.1255)

ConsoleWrite(0xFF0000/0xFFFF & @CRLF)

; expect 0xFF or 255

; actual output is 255.003891050584 = INCORRECT[/autoit]

(ignore the bitshift bit of the title, I figured that out)

Thanks to anyone who can help

David

Edited by dgm5555

Share this post


Link to post
Share on other sites
trancexx

Dividing any two numbers results in double precision number type. If you want integer then use function Int() on your result.


♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites
water

We don't need the version of the SciTE editor but the AutoIt version.

Run

MsgBox(0, "", "AutoIt version: " & @AutoItVersion)

My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2018-06-01 - Version 1.4.9.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX (2018-01-27 - Version 1.3.3.1) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2015-04-01 - Version 0.4.0.0) - Download - General Help & Support - Example Scripts
Excel - Example Scripts - Wiki
Word - Wiki
PowerPoint (2015-06-06 - Version 0.0.5.0) - Download - General Help & Support

Tutorials:
ADO - Wiki

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×