Jump to content

[Solved] strange floating point behaviour


 Share

Recommended Posts

Hi.

I run this script:

Local $num = 0.7
While $num < 11
    $num += 0.1
    ConsoleWrite($num & @CRLF)
WEnd

Console output shows:

0.8
0.9
1
1.1
1.2
1.3
1.4
1.5
1.6
1.7
1.8
1.9
2
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9
3
3.1
3.2
3.3
3.4
3.5
3.6
3.7
3.8
3.9
4
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
4.9
5
5.1
5.2
5.3
5.4
5.5
5.6
5.7
5.8
5.9
5.99999999999999
6.09999999999999
6.19999999999999
6.29999999999999
6.39999999999999
6.49999999999999
6.59999999999999
6.69999999999999
6.79999999999999
6.89999999999999
6.99999999999999
7.09999999999999
7.19999999999999
7.29999999999999
7.39999999999999
7.49999999999999
7.59999999999999
7.69999999999999
7.79999999999999
7.89999999999999
7.99999999999999
8.09999999999999
8.19999999999999
8.29999999999999
8.39999999999999
8.49999999999999
8.59999999999999
8.69999999999999
8.79999999999998
8.89999999999998
8.99999999999998
9.09999999999998
9.19999999999998
9.29999999999998
9.39999999999998
9.49999999999998
9.59999999999998
9.69999999999998
9.79999999999998
9.89999999999998
9.99999999999998
10.1
10.2
10.3
10.4
10.5
10.6
10.7
10.8
10.9
11
11.1

After 5.9 follows 5.99999999999999 not 6. And later on it goes back from 9.99999999999998 to 10.1.

Why is that and how to avoid this. And with avoiding I don't mean:

ConsoleWrite(Round($num, 1) & @CRLF)

Regards, Conrad

Edited by Simpel
[Solved]
SciTE4AutoIt = 3.7.3.0   AutoIt = 3.3.14.2   AutoItX64 = 0   OS = Win_10   Build = 19044   OSArch = X64   Language = 0407/german
H:\...\AutoIt3\SciTE     H:\...\AutoIt3      H:\...\AutoIt3\Include     (H:\ = Network Drive)

   88x31.png  Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind.

Link to comment
Share on other sites

  • Moderators

Simpel,

Search the forum for "floating point arithmetic" - this subject has been covered many, many times before.

How to avoid? Do not use floats and try to stick with integers - for example with currency do the calculations in pennies/cents/whatever and convert back into pounds/dollars/slushniks when you have the final answer.

M23

Public_Domain.png.2d871819fcb9957cf44f4514551a2935.png Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind

Open spoiler to see my UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

 

Link to comment
Share on other sites

Thanks Melba,

should have known that. For those who want to know more about it there seems to be a good starting point here: http://floating-point-gui.de/

Conrad

SciTE4AutoIt = 3.7.3.0   AutoIt = 3.3.14.2   AutoItX64 = 0   OS = Win_10   Build = 19044   OSArch = X64   Language = 0407/german
H:\...\AutoIt3\SciTE     H:\...\AutoIt3      H:\...\AutoIt3\Include     (H:\ = Network Drive)

   88x31.png  Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...