dealing with unprecision of calculations with decimals

[Imbuter2000]

Try this script:

$original = "130102.123456"
$diff = "0.000001"
$expected_result = "130102.123455"

consolewrite((($original - $diff) = $expected_result) & @CRLF)

You will get a "False" because the internal result is something near to "130102.123455000001".

This is because of unprecision when dealing with decimals in a binary underlying system (all computers).

The thing that I unexpected is that if you show the result in console or msgbox you'll see exactly "130102.123455".  Why?

PS: these numbers are timestamps. I'm just testing to find a timestamp format that balance precision and human readability

[MilesAhead]

Why?

 

 

My guess is that MsgBox could subtract "1" from "6" to get "5"

If you made it more complicated, like with division or multiplication, I don't think it would work out so well. 

[trancexx]

If you force strings for input then forcing string comparison on result could satisfy your needs.

consolewrite((($original - $diff) == $expected_result) & @CRLF)

[ivan]

$original = "130102.123456"
$diff = "0.000001"
$expected_result = "130102.123455"
consolewrite((($original - $diff) == $expected_result) & @CRLF)
consolewrite(($original - $diff)  & @CRLF)
consolewrite($expected_result & @CRLF)

Try ==.

[Imbuter2000]

 

If you force strings for input then forcing string comparison on result could satisfy your needs.

consolewrite((($original - $diff) == $expected_result) & @CRLF)

Interesting!

Anyway I wrongly set the variable declarations as string, I wanted to use this:

$original = 130102.123456
$diff = 0.000001
$expected_result = 130102.123455

consolewrite((($original - $diff) = $expected_result) & @CRLF)

and it still returns "False"    

[Imbuter2000]

To bring this binary unprecision to the extreme, try this:

$original = 1.6
$diff = 0.2
$expected_result = 1.4

consolewrite (($original - $diff) & @CRLF)
consolewrite((($original - $diff) > $expected_result) & @CRLF)

It returns "1.4" and then "True"!

Edited December 29, 2013 by Imbuter2000

[JohnOne]

With such a subject, can a wordsmith clarify that there is such a word as "unprecision"?

Because I cannot find a definition.

[Malkey]

Use the Round function to control the number of deciminal places returned from a calculation.

Local $original        = 130102.123456
Local $diff            = 0.000001
Local $expected_result = 130102.123455

ConsoleWrite($original & " - " & $diff & " = " & $original - $diff & " (expected: " & $expected_result & ")" & @CRLF)
ConsoleWrite(($original - $diff = $expected_result) & @CRLF)
ConsoleWrite((Round($original - $diff, 6) = $expected_result) & @CRLF)

Edit: Added this example. Note Round added to calculation, not to the Hex conversion.

ConsoleWrite("0.2       = " & Hex(0.2) & @LF)
ConsoleWrite("1.6 - 1.4 = " & Hex(Round(1.6 - 1.4, 1)) & @LF)

Edited December 29, 2013 by Malkey

[Imbuter2000]

With such a subject, can a wordsmith clarify that there is such a word as "unprecision"?

Because I cannot find a definition.

 

ahah sorry, I'm Italian.  I meant "imprecision"  

[jchd]

Imbuter2000,

Welcome to the wonderful world of binary floating-point calculations!

There is nothing surprising in what you show, as the following shows:

ConsoleWrite("1.6 = " & Hex(1.6) & @LF)
ConsoleWrite("1.4 = " & Hex(1.4) & @LF)
ConsoleWrite("0.2 = " & Hex(0.2) & @LF)
ConsoleWrite("1.6 - 1.4 = " & Hex(1.6 - 1.4) & @LF)

[guinness]

Welcome to the wonderful world of binary floating-point calculations!

It's a lie, it's not a wonderful world!

[jchd]

What? A surprise every 3 operations, and more with every comparison. Isn't that wonderful?

[Mat]

Once you can explain why this code outputs what it does for both cases, you are not far from the bare minimum you need to know about numeric bases in programming. The problem is less to do with IEEE754 and the details about floating point, but more about binary itself:

Local $i = 1.1

For $n = 1 To 20
    $i = ($i - 1.0) * 11.0

    ConsoleWrite($i & @LF)
Next


ConsoleWrite("--------------------" & @LF)


Local $i = 1.25

For $n = 1 To 20
    $i = ($i - 1.0) * 5.0

    ConsoleWrite($i & @LF)
Next

It's not a lack of precision, if you used a million decimal places then the first loop would eventually diverge from 1.1, whereas the second loop would never diverge (unless something went horribly wrong). Greater precision just hides the problem.

[jchd]

Exactly