Sign in to follow this  
Followers 0
chun914

calculation error

6 posts in this topic

I have some numbers like below and the final $temp amount should be 450 exact but the calculation ended up with 450.00000000001..

i have no idea what's going wrong ?

$p = 4592.05
$1 = 679.41
$2 = 661.31
$3 = 660.04
$4 = 648.06
$5 = 963.85
$6 = 529.38

$childsum = 0

$childsum += $1
msgbox(0,"",$childsum)
$childsum += $2
msgbox(0,"",$childsum)
$childsum += $3
msgbox(0,"",$childsum)
$childsum += $4
msgbox(0,"",$childsum)
$childsum += $5
msgbox(0,"",$childsum)
$childsum += $6
msgbox(0,"",$childsum)

$temp = $p -$childsum
msgbox(0,"",$temp)

Share this post


Link to post
Share on other sites



This is due to floating point arithmetic. Since none of your numbers has a precision greater than two decimal places, use the following.  

Round($temp, 2)

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

i have to check if the sum + 450 = $p.......

if $childsum + 450 = $p Then
   msgbox(0,"","so something")
Else        
   msgbox(0,"","oops")
EndIf
Edited by chun914

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

Then use the following after your arithmetic.

$childsum = Round($childsum, 2)
Edited by AdamUL

Share this post


Link to post
Share on other sites

chun914,

As all the amounts have 2 decimal places I would convert all of them to integers (i.e. multiply them by 100) so that I could use integer arithmetic to sum them. Then I would divide by 100 and use Round when I wanted the total back in 2 decimal place format. ;)

M23


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

thanks.. M23,

finally get it done by *100 and compare using int($a) = int($B)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0