Sign in to follow this  
Followers 0
Quinch

Failing basic math

4 posts in this topic

I have a problem with writing a script - seemingly not related to functions, syntax or anything relating to actual code... but rather, basic math errors. After a run of simple addition/subtraction routines, the script suddenly starts spouting gibberish, missing the mark by a tiny fraction {10^-12 ish}; normally, for whatever reason that happened, I'd simply round the number and to hell with it, but it happens even then.

The attached script and output file should explain the results better - all three values are supposed to gradually increase {by a factor of 0.1} to the rate of 1 per line, and after 100 steps, decrease by the same method. Unfortunately, a bit after the dropoff point, things pretty much get shot to hell and I can't figure out why.

Any ideas?

Desktop.rar

Share this post


Link to post
Share on other sites

When reporting a problem, it's best to present the smallest error producing code possible. I haven't the time to sit and understand this example, sorry.

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

Okay, so I'm guessing that I should round it down because... stuff. What is the "tolerance of a Double Word Floating Point number" in AutoIt, i.e., what number of decimal places should I round it down to? Googling says 11, but I should probably double-check.

EDIT: Also, is there a way to force the script to calculate only up to a set precision? Adding Round(....) to every math function in the script is going to get real messy real fast.

Edited by Quinch

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0