# Numbers not comparing properly after math operation

## Recommended Posts

This took me ages to track down the issue, but I wanted to pass it on to get your thoughts on this very strange error since it's very possible I'm missing something obvious. When dividing the number 110.4 by 192 (resulting in 0.575), it is not seen as equal to 0.575 when arrived at through other methods.

Here's some replication code:

```ConsoleWrite(@tab & "NO ROUNDING" & @CRLF)
\$Value1 = 110.4/192
\$Value2 = 0.575
\$Value3 = 0.2875 * 2
ConsoleWrite("Value 1: " & \$Value1 & @CRLF)
ConsoleWrite("Value 2: " & \$Value2 & @CRLF)
ConsoleWrite("Value 3: " & \$Value3 & @CRLF & @CRLF)

ConsoleWrite("Are Values 1 and 2 Equal: " & (\$Value1 = \$Value2) & @CRLF)
ConsoleWrite("Are Values 2 and 3 Equal: " & (\$Value2 = \$Value3) & @CRLF)
ConsoleWrite("Are Values 1 and 3 Equal: " & (\$Value1 = \$Value3) & @CRLF & @CRLF & "-----------------------" & @CRLF & @CRLF)

ConsoleWrite(@tab & "USING ROUNDING" & @CRLF)
\$Value1 = Round(110.4/192, 3)
\$Value2 = 0.575
\$Value3 = 0.2875 * 2
ConsoleWrite("Value 1: " & \$Value1 & @CRLF)
ConsoleWrite("Value 2: " & \$Value2 & @CRLF)
ConsoleWrite("Value 3: " & \$Value3 & @CRLF & @CRLF)

ConsoleWrite("Are Values 1 and 2 Equal: " & (\$Value1 = \$Value2) & @CRLF)
ConsoleWrite("Are Values 2 and 3 Equal: " & (\$Value2 = \$Value3) & @CRLF)
ConsoleWrite("Are Values 1 and 3 Equal: " & (\$Value1 = \$Value3) & @CRLF)```

Any light that could be shed on this, I'd appreciate! I have a very ghetto work-around in my code for comparing numbers, but I'd rather have a proper solution if one exists.

Thoughts?

Edit--I should have included this before, this is what is output in the console on my end:

Quote

NO ROUNDING
Value 1: 0.575
Value 2: 0.575
Value 3: 0.575

Are Values 1 and 2 Equal: False
Are Values 2 and 3 Equal: True
Are Values 1 and 3 Equal: False

-----------------------

USING ROUNDING
Value 1: 0.575
Value 2: 0.575
Value 3: 0.575

Are Values 1 and 2 Equal: True
Are Values 2 and 3 Equal: True
Are Values 1 and 3 Equal: True

Edited by FlashpointBlack
##### Share on other sites

Welcome to the world of computers and the representation of floating point numbers in binary. For some none integer numbers it is not possible to store the decimal places exactly, hence we have to round to a reasonable number of decimal places or allow a small tolerance when comparing floating point numbers . I think by default consolewrite rounds number to 7 decimal places which makes the numbers look the same. If you  run this modified version of your script it will become clear what is happening.

```ConsoleWrite(@tab & "NO ROUNDING" & @CRLF)
\$Value1 = 110.4/192
\$Value2 = 0.575
\$Value3 = 0.2875 * 2
ConsoleWrite("Value 1: " & StringFormat("%0.17f",\$Value1) & @CRLF)
ConsoleWrite("Value 2: " & StringFormat("%0.17f",\$Value2) & @CRLF)
ConsoleWrite("Value 3: " & StringFormat("%0.17f",\$Value3) & @CRLF & @CRLF)

ConsoleWrite("Are Values 1 and 2 Equal: " & (\$Value1 = \$Value2) & @CRLF)
ConsoleWrite("Are Values 2 and 3 Equal: " & (\$Value2 = \$Value3) & @CRLF)
ConsoleWrite("Are Values 1 and 3 Equal: " & (\$Value1 = \$Value3) & @CRLF & @CRLF & "-----------------------" & @CRLF & @CRLF)

ConsoleWrite(@tab & "USING ROUNDING" & @CRLF)
\$Value1 = Round(110.4/192, 3)
\$Value2 = 0.575
\$Value3 = 0.2875 * 2
ConsoleWrite("Value 1: " & StringFormat("%0.17f",\$Value1) & @CRLF)
ConsoleWrite("Value 2: " & StringFormat("%0.17f",\$Value2) & @CRLF)
ConsoleWrite("Value 3: " & StringFormat("%0.17f",\$Value3) & @CRLF & @CRLF)

ConsoleWrite("Are Values 1 and 2 Equal: " & (\$Value1 = \$Value2) & @CRLF)
ConsoleWrite("Are Values 2 and 3 Equal: " & (\$Value2 = \$Value3) & @CRLF)
ConsoleWrite("Are Values 1 and 3 Equal: " & (\$Value1 = \$Value3) & @CRLF)```

"Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the universe trying to build bigger and better idiots. So far, the universe is winning."- Rick Cook

##### Share on other sites

Thank you, Bowmore!

In the back of my mind, I thought that this may be the issue. What was throwing me off was ConsoleWrite automatically rounding. Additionally, while treating the results as a string using the == operand, they were seen as equal at all times further adding to my confusion. In retrospect, I'm sure that the value is automatically rounded or truncated in the generation of the strings to compare but it didn't occur to me at the time.

I truly appreciate your response and insight!

##### Share on other sites

Indeed.

The following page has been in my favourites for ages: "What Every Computer Scientist Should Know About Floating-Point Arithmetic"

/edit: warning: not for people with math-phobia...

It is an awesome read and a reality check for people who do important calculations and don't fringe-check their results often enough. I learned it the hard way, many many moons ago, when my then employer almost got in serious trouble with the tax office for detailed financial specifications not matching the totals in financial reports that I had scripted from a Unix machine. Took me about 40 hours in a single sitting to repair all scripts, rerun all reports over the past 5 years and fight off a diabolically angry financial manager... It's that important

Roses are FF0000, violets are 0000FF... All my base are belong to you.

## Create an account

Register a new account

• ### Recently Browsing   0 members

×

• Wiki

• Back

• #### Beta

• Git
• FAQ
• Our Picks
×
• Create New...