FlashpointBlack

Numbers not comparing properly after math operation

4 posts in this topic

#1 ·  Posted (edited)

This took me ages to track down the issue, but I wanted to pass it on to get your thoughts on this very strange error since it's very possible I'm missing something obvious. When dividing the number 110.4 by 192 (resulting in 0.575), it is not seen as equal to 0.575 when arrived at through other methods.

Here's some replication code:

ConsoleWrite(@tab & "NO ROUNDING" & @CRLF)
$Value1 = 110.4/192
$Value2 = 0.575
$Value3 = 0.2875 * 2
ConsoleWrite("Value 1: " & $Value1 & @CRLF)
ConsoleWrite("Value 2: " & $Value2 & @CRLF)
ConsoleWrite("Value 3: " & $Value3 & @CRLF & @CRLF)

ConsoleWrite("Are Values 1 and 2 Equal: " & ($Value1 = $Value2) & @CRLF)
ConsoleWrite("Are Values 2 and 3 Equal: " & ($Value2 = $Value3) & @CRLF)
ConsoleWrite("Are Values 1 and 3 Equal: " & ($Value1 = $Value3) & @CRLF & @CRLF & "-----------------------" & @CRLF & @CRLF)


ConsoleWrite(@tab & "USING ROUNDING" & @CRLF)
$Value1 = Round(110.4/192, 3)
$Value2 = 0.575
$Value3 = 0.2875 * 2
ConsoleWrite("Value 1: " & $Value1 & @CRLF)
ConsoleWrite("Value 2: " & $Value2 & @CRLF)
ConsoleWrite("Value 3: " & $Value3 & @CRLF & @CRLF)

ConsoleWrite("Are Values 1 and 2 Equal: " & ($Value1 = $Value2) & @CRLF)
ConsoleWrite("Are Values 2 and 3 Equal: " & ($Value2 = $Value3) & @CRLF)
ConsoleWrite("Are Values 1 and 3 Equal: " & ($Value1 = $Value3) & @CRLF)

Any light that could be shed on this, I'd appreciate! I have a very ghetto work-around in my code for comparing numbers, but I'd rather have a proper solution if one exists.

Thoughts?

 

Edit--I should have included this before, this is what is output in the console on my end:

Quote

    NO ROUNDING
Value 1: 0.575
Value 2: 0.575
Value 3: 0.575

Are Values 1 and 2 Equal: False
Are Values 2 and 3 Equal: True
Are Values 1 and 3 Equal: False

-----------------------

    USING ROUNDING
Value 1: 0.575
Value 2: 0.575
Value 3: 0.575

Are Values 1 and 2 Equal: True
Are Values 2 and 3 Equal: True
Are Values 1 and 3 Equal: True

 

Edited by FlashpointBlack
Added my console output

Share this post


Link to post
Share on other sites



Welcome to the world of computers and the representation of floating point numbers in binary. For some none integer numbers it is not possible to store the decimal places exactly, hence we have to round to a reasonable number of decimal places or allow a small tolerance when comparing floating point numbers . I think by default consolewrite rounds number to 7 decimal places which makes the numbers look the same. If you  run this modified version of your script it will become clear what is happening.

ConsoleWrite(@tab & "NO ROUNDING" & @CRLF)
$Value1 = 110.4/192
$Value2 = 0.575
$Value3 = 0.2875 * 2
ConsoleWrite("Value 1: " & StringFormat("%0.17f",$Value1) & @CRLF)
ConsoleWrite("Value 2: " & StringFormat("%0.17f",$Value2) & @CRLF)
ConsoleWrite("Value 3: " & StringFormat("%0.17f",$Value3) & @CRLF & @CRLF)

ConsoleWrite("Are Values 1 and 2 Equal: " & ($Value1 = $Value2) & @CRLF)
ConsoleWrite("Are Values 2 and 3 Equal: " & ($Value2 = $Value3) & @CRLF)
ConsoleWrite("Are Values 1 and 3 Equal: " & ($Value1 = $Value3) & @CRLF & @CRLF & "-----------------------" & @CRLF & @CRLF)


ConsoleWrite(@tab & "USING ROUNDING" & @CRLF)
$Value1 = Round(110.4/192, 3)
$Value2 = 0.575
$Value3 = 0.2875 * 2
ConsoleWrite("Value 1: " & StringFormat("%0.17f",$Value1) & @CRLF)
ConsoleWrite("Value 2: " & StringFormat("%0.17f",$Value2) & @CRLF)
ConsoleWrite("Value 3: " & StringFormat("%0.17f",$Value3) & @CRLF & @CRLF)

ConsoleWrite("Are Values 1 and 2 Equal: " & ($Value1 = $Value2) & @CRLF)
ConsoleWrite("Are Values 2 and 3 Equal: " & ($Value2 = $Value3) & @CRLF)
ConsoleWrite("Are Values 1 and 3 Equal: " & ($Value1 = $Value3) & @CRLF)

 

1 person likes this

"Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to build bigger and better idiots. So far, the universe is winning."- Rick Cook

Share this post


Link to post
Share on other sites

Thank you, Bowmore!

In the back of my mind, I thought that this may be the issue. What was throwing me off was ConsoleWrite automatically rounding. Additionally, while treating the results as a string using the == operand, they were seen as equal at all times further adding to my confusion. In retrospect, I'm sure that the value is automatically rounded or truncated in the generation of the strings to compare but it didn't occur to me at the time.

I truly appreciate your response and insight!

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

Indeed.

The following page has been in my favourites for ages: "What Every Computer Scientist Should Know About Floating-Point Arithmetic"

https://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html

/edit: warning: not for people with math-phobia... :sweating:

It is an awesome read and a reality check for people who do important calculations and don't fringe-check their results often enough. I learned it the hard way, many many moons ago, when my then employer almost got in serious trouble with the tax office for detailed financial specifications not matching the totals in financial reports that I had scripted from a Unix machine. Took me about 40 hours in a single sitting to repair all scripts, rerun all reports over the past 5 years and fight off a diabolically angry financial manager... It's that important :) 

Edited by SadBunny

Roses are FF0000, violets are 0000FF... All my base are belong to you.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now