Sign in to follow this  
Followers 0
cycotron

unexpected decimal places

3 posts in this topic

#1 ·  Posted (edited)

Please forgive me if this has been answered before, because I'm sure it has, but i couldn't find the information.

Why is it that when I do the following:

ConsoleWrite(.817 - .79 & @crlf)

The result is "0.0269999999999999"

Should the result not be "0.027"? Is there a standard way of working with decimals other than having to Round() every one of my results?

Edited by cycotron

Share this post


Link to post
Share on other sites



cycotron,

Welcome to the wonderful world of "binary floating point". This is not an AutoIt problem - all computers suffer the same way. Google will tell you more than you ever wanted to know if you look. ;)

I am afraid you will have to Round your results - at least you have that function to use. :)

M23


Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind._______My UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Share this post


Link to post
Share on other sites

Thank you Melba23. I thought I remembered something along those lines that I read a year or two ago.

I will use Round, it's simple enough :)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0