Jump to content
Sign in to follow this  

Error using the function Number

Recommended Posts


I am using the Number function to parse a text file a translate the numbers into Integers. I have run in to a problem with the number presented on the screen are not the same as those in memory. I assume this has something do with how double is represented but should it be like this? 

for $i = 0 to 100000 step 1
    $a = Number("0.86",3)*100
    $b = Number("0.14",3)*100
    $f = 100 - $a - $b
    if $f < 0 Then
        msgbox(0, "", "$a is: " & $a & " $b is " & $b & " Why is 100 - " & $a & " - " & $b & " = " & $f)

MsgBox(0, "", "DONE!")



Share this post

Link to post
Share on other sites

Your asumption is correct. No matter how many bits are used, any binary floating point represenation of real numbers will have fiinite accuracy, so there will be gaps between representable numbers, and the size of these gaps will expand with distance from zero. Some background info is here.

Consolewrite((.14*100)-(.15*100) & @CRLF)

You can however use functions such as Int, Round, Floor, and Ceiling to control the result of computations, despite using real numbers as inputs.

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Create New...