bradley1 Posted March 23, 2006 Posted March 23, 2006 Hey guys/girls, semi new user here, sorry if this question has been asked before but I couldn't find a similar topic: Sometimes when I am working with numbers as variables, such as subtracting or adding, something strange happens. A 1 will automatically become a .99999, or a 4 - 2 will become a 2.000001 instead of just plain 2. Where does this come from and is there an easy way to correct it? Thank you, Brad
BigDod Posted March 23, 2006 Posted March 23, 2006 Hey guys/girls, semi new user here, sorry if this question has been asked before but I couldn't find a similar topic: Sometimes when I am working with numbers as variables, such as subtracting or adding, something strange happens. A 1 will automatically become a .99999, or a 4 - 2 will become a 2.000001 instead of just plain 2. Where does this come from and is there an easy way to correct it? Thank you, BradI do not know why other than the default no of decimal places being 6. If you know that you are going to have whole nos type 0.9999 into $input = InputBox('enter a number', 'moo') $input = StringFormat("%.0f", $input) MsgBox(0, '', $input) Time you enjoyed wasting is not wasted time ......T.S. Elliot Suspense is worse than disappointment................Robert Burns God help the man who won't help himself, because no-one else will...........My Grandmother
Moderators SmOke_N Posted March 23, 2006 Moderators Posted March 23, 2006 Hey guys/girls, semi new user here, sorry if this question has been asked before but I couldn't find a similar topic:Sometimes when I am working with numbers as variables, such as subtracting or adding, something strange happens. A 1 will automatically become a .99999, or a 4 - 2 will become a 2.000001 instead of just plain 2. Where does this come from and is there an easy way to correct it? Thank you,BradMust admit... I've never had 4 - 2 become 2.000001 Common sense plays a role in the basics of understanding AutoIt... If you're lacking in that, do us all a favor, and step away from the computer.
Valik Posted March 23, 2006 Posted March 23, 2006 It's perfectly normal and you can learn to correct it by researching floating point numbers which explains why the issue occurs and provides some solutions to some common problems when dealing with floating point numbers.
bradley1 Posted March 25, 2006 Author Posted March 25, 2006 It's perfectly normal and you can learn to correct it by researching floating point numbers which explains why the issue occurs and provides some solutions to some common problems when dealing with floating point numbers.Thank you for your help everyone, I think I have figured it out.
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now