Sign in to follow this  
Followers 0
bradley1

Numbers - Rounding Off ?

5 posts in this topic

Hey guys/girls, semi new user here, sorry if this question has been asked before but I couldn't find a similar topic:

Sometimes when I am working with numbers as variables, such as subtracting or adding, something strange happens. A 1 will automatically become a .99999, or a 4 - 2 will become a 2.000001 instead of just plain 2. Where does this come from and is there an easy way to correct it?

Thank you,

Brad

Share this post


Link to post
Share on other sites



Hey guys/girls, semi new user here, sorry if this question has been asked before but I couldn't find a similar topic:

Sometimes when I am working with numbers as variables, such as subtracting or adding, something strange happens. A 1 will automatically become a .99999, or a 4 - 2 will become a 2.000001 instead of just plain 2. Where does this come from and is there an easy way to correct it?

Thank you,

Brad

I do not know why other than the default no of decimal places being 6. If you know that you are going to have whole nos type 0.9999 into

$input = InputBox('enter a number', 'moo')
$input = StringFormat("%.0f", $input)
MsgBox(0, '', $input)


Get Beta versions Here Get latest SciTE editor Here AutoIt 1-2-3 by Valuater - A great starting point.

Time you enjoyed wasting is not wasted time ......T.S. Elliot
Suspense is worse than disappointment................Robert Burns
God help the man who won't help himself, because no-one else will...........My Grandmother

Share this post


Link to post
Share on other sites

Hey guys/girls, semi new user here, sorry if this question has been asked before but I couldn't find a similar topic:

Sometimes when I am working with numbers as variables, such as subtracting or adding, something strange happens. A 1 will automatically become a .99999, or a 4 - 2 will become a 2.000001 instead of just plain 2. Where does this come from and is there an easy way to correct it?

Thank you,

Brad

Must admit... I've never had 4 - 2 become 2.000001 :)

[center]Common sense plays a role in the basics of understanding AutoIt... If you're lacking in that, do us all a favor, and step away from the computer.[/center]

Share this post


Link to post
Share on other sites

It's perfectly normal and you can learn to correct it by researching floating point numbers which explains why the issue occurs and provides some solutions to some common problems when dealing with floating point numbers.

Share this post


Link to post
Share on other sites

It's perfectly normal and you can learn to correct it by researching floating point numbers which explains why the issue occurs and provides some solutions to some common problems when dealing with floating point numbers.

Thank you for your help everyone, I think I have figured it out.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0