calculation error

Recommended Posts

I have some numbers like below and the final \$temp amount should be 450 exact but the calculation ended up with 450.00000000001..

i have no idea what's going wrong ?

```\$p = 4592.05
\$1 = 679.41
\$2 = 661.31
\$3 = 660.04
\$4 = 648.06
\$5 = 963.85
\$6 = 529.38

\$childsum = 0

\$childsum += \$1
msgbox(0,"",\$childsum)
\$childsum += \$2
msgbox(0,"",\$childsum)
\$childsum += \$3
msgbox(0,"",\$childsum)
\$childsum += \$4
msgbox(0,"",\$childsum)
\$childsum += \$5
msgbox(0,"",\$childsum)
\$childsum += \$6
msgbox(0,"",\$childsum)

\$temp = \$p -\$childsum
msgbox(0,"",\$temp)```

Share on other sites

This is due to floating point arithmetic. Since none of your numbers has a precision greater than two decimal places, use the following.

`Round(\$temp, 2)`

Share on other sites

i have to check if the sum + 450 = \$p.......

```if \$childsum + 450 = \$p Then
msgbox(0,"","so something")
Else
msgbox(0,"","oops")
EndIf```
Edited by chun914

Share on other sites

Then use the following after your arithmetic.

`\$childsum = Round(\$childsum, 2)`

Share on other sites

chun914,

As all the amounts have 2 decimal places I would convert all of them to integers (i.e. multiply them by 100) so that I could use integer arithmetic to sum them. Then I would divide by 100 and use Round when I wanted the total back in 2 decimal place format.

M23

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind

Open spoiler to see my UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

Share on other sites

thanks.. M23,

finally get it done by *100 and compare using int(\$a) = int(\$

Create an account

Register a new account