Sign in to follow this  
Followers 0
Matjo

Error using the function Number

2 posts in this topic

Hi, 

I am using the Number function to parse a text file a translate the numbers into Integers. I have run in to a problem with the number presented on the screen are not the same as those in memory. I assume this has something do with how double is represented but should it be like this? 

for $i = 0 to 100000 step 1
    $a = Number("0.86",3)*100
    $b = Number("0.14",3)*100
    $f = 100 - $a - $b
    if $f < 0 Then
        msgbox(0, "", "$a is: " & $a & " $b is " & $b & " Why is 100 - " & $a & " - " & $b & " = " & $f)
        ExitLoop
    EndIf

Next
MsgBox(0, "", "DONE!")

 

error.jpg

Share this post


Link to post
Share on other sites



Your asumption is correct. No matter how many bits are used, any binary floating point represenation of real numbers will have fiinite accuracy, so there will be gaps between representable numbers, and the size of these gaps will expand with distance from zero. Some background info is here.

Consolewrite((.14*100)-(.15*100) & @CRLF)

You can however use functions such as Int, Round, Floor, and Ceiling to control the result of computations, despite using real numbers as inputs.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0