Jump to content

Number fuction appears to be rounding small decimals


uncommon
 Share

Recommended Posts

2 hours ago, uncommon said:
11044326.18126327

This value can't be represented exactly using 64-bit double value:

Input value  = 11044326.1812633
FP (hex)     = 0x416510BCC5CCE8A1
Sign         = +
Exponent     = 23
Scaling      = 2^23 = 8388608
Mantissa     = 1/4 + 1/16 + 1/256 + 1/8192 + 1/32768 + 1/65536 + 1/131072 + 1/262144 + 1/2097152 + 1/4194304 + 1/67108864 + 1/268435456 + 1/536870912 + 1/1073741824 + 1/8589934592 + 1/17179869184 + 1/137438953472 + 1/274877906944 + 1/549755813888 + 1/2199023255552 + 1/35184372088832 + 1/140737488355328 + 1/4503599627370496

Nearby exact double values (exact computation from FP bits)
PrePrevious  = +11044326.18126326613128185272216796875
Previous     = +11044326.181263267993927001953125
Value        = +11044326.18126326985657215118408203125
Next         = +11044326.1812632717192173004150390625
NextNext     = +11044326.18126327358186244964599609375

The ULPs (Unit of Least Precision) around 11044326.1812633 are -1.86264514923096e-09 and +1.86264514923096e-09

This isn't due to AutoIt but is the result of limits in floating-point 64 bit representation.

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

1 hour ago, jchd said:

This value can't be represented exactly using 64-bit double value:

Input value  = 11044326.1812633
FP (hex)     = 0x416510BCC5CCE8A1
Sign         = +
Exponent     = 23
Scaling      = 2^23 = 8388608
Mantissa     = 1/4 + 1/16 + 1/256 + 1/8192 + 1/32768 + 1/65536 + 1/131072 + 1/262144 + 1/2097152 + 1/4194304 + 1/67108864 + 1/268435456 + 1/536870912 + 1/1073741824 + 1/8589934592 + 1/17179869184 + 1/137438953472 + 1/274877906944 + 1/549755813888 + 1/2199023255552 + 1/35184372088832 + 1/140737488355328 + 1/4503599627370496

Nearby exact double values (exact computation from FP bits)
PrePrevious  = +11044326.18126326613128185272216796875
Previous     = +11044326.181263267993927001953125
Value        = +11044326.18126326985657215118408203125
Next         = +11044326.1812632717192173004150390625
NextNext     = +11044326.18126327358186244964599609375

The ULPs (Unit of Least Precision) around 11044326.1812633 are -1.86264514923096e-09 and +1.86264514923096e-09

This isn't due to AutoIt but is the result of limits in floating-point 64 bit representation.

Thanks, I do remember seeing a 64bit limitation somewhere in the help file.

No problem can withstand the assault of sustained thinking.Voltaire

_Array2HTMLTable()_IEClassNameGetCollection()_IEquerySelectorAll()

Link to comment
Share on other sites

First of all, what jchd wrote applies: This number cannot be represented exactly as a floating point number according to IEEE 754. And as mentioned,, this is not an AutoIt problem but affects all programs that use native floating point types.

Your problem with rounding to ...33 lies somewhere else. Internally the precision of your number is higher than what is displayed to you. The problem here is with the output of the number and not its internal storage. If you output a variable via consolewrite, it must first be converted into a string. So implicitly ConsoleWrite(String($dNumber1) & @LF) is done.
However, the conversion into a string does not have the task to output exactly up to the last decimal place, but to make a kind of compromise, so that the number does not become too long. Depending on the number of digits before the decimal point, sometimes more and sometimes less digits after the decimal point are displayed. I don't know the exact algorithm for this but it seems to me that the goal is not to use more than 16 characters for the display of the number. So in your case it rounds to the 16th character of the number.
If you would add a digit to the digits before the decimal point, you would have another digit less in the decimal point area.

This behavior can be deliberately avoided and then you will also see that the internal calculation is indeed more accurate and the phenomenon you describe only affects the output:

$dNumber = 11044326.18126327

ConsoleWrite( "String():           " & $dNumber & _
    stringformat("\nStringformat %%.8f: %18.8f\nStringformat %%.16f: %24.16f\n\n", $dNumber, $dNumber))

 

Link to comment
Share on other sites

Yes, that.

If you run this:

Local $s = "11044326.18126327"
ConsoleWrite($s & StringFormat("\t%18.29f\t%s\n", Number($s), Hex(Number($s), 16)))

you'll see that the output goes is exactly what the double means:

11044326.18126326985657215118408203125 exact value represented by the double
11044326.18126326985657215118408203125 output of StringFormat to 29 digits

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...