Matjo Posted January 21, 2016 Posted January 21, 2016 Hi, I am using the Number function to parse a text file a translate the numbers into Integers. I have run in to a problem with the number presented on the screen are not the same as those in memory. I assume this has something do with how double is represented but should it be like this? for $i = 0 to 100000 step 1 $a = Number("0.86",3)*100 $b = Number("0.14",3)*100 $f = 100 - $a - $b if $f < 0 Then msgbox(0, "", "$a is: " & $a & " $b is " & $b & " Why is 100 - " & $a & " - " & $b & " = " & $f) ExitLoop EndIf Next MsgBox(0, "", "DONE!")
RTFC Posted January 21, 2016 Posted January 21, 2016 Your asumption is correct. No matter how many bits are used, any binary floating point represenation of real numbers will have fiinite accuracy, so there will be gaps between representable numbers, and the size of these gaps will expand with distance from zero. Some background info is here. Consolewrite((.14*100)-(.15*100) & @CRLF) You can however use functions such as Int, Round, Floor, and Ceiling to control the result of computations, despite using real numbers as inputs. My Contributions and Wrappers Spoiler BitMaskSudokuSolver BuildPartitionTable CodeCrypter CodeScanner DigitalDisplay Eigen4AutoIt FAT Suite HighMem MetaCodeFileLibrary OSgrid Pool RdRand SecondDesktop SimulatedAnnealing Xbase I/O
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now