# Decimal errors on exact mathematical formules

## Recommended Posts

Hi all,

i'm becoming crazy since i'm finding several errors when relating numeric expressions each others.

Example:

```\$var1 = 2
\$var2 = 0.50
\$sum = \$var1 * \$var2```

Result is NOT exactly 1 but something like 1.000000000000000000123 and using round() doesn't solve.

Generally speaking i realize that too often simple calcs generates micro-decimals that invalid relations.

I also tried string() but with no success.

Is there a way to *exactly* compare numbers with no margin of errors?

Thanks a lot,

Marco

##### Share on other sites

Well, for me it works as expected:

```ConsoleWrite(1 / 2 & @CRLF)
ConsoleWrite(0.5 * 2 & @CRLF)
ConsoleWrite(1 / 3 * 3 & @CRLF)
ConsoleWrite(5 / 9 & @CRLF)
ConsoleWrite(677 / 1375 & @CRLF)
ConsoleWrite(ACos(-1) & @CRLF)
ConsoleWrite(Sin(ACos(-1) / 2) & @CRLF)```

Results:

```0.5
1
1
0.555555555555556
0.492363636363636
3.14159265358979
1```

Br,

UEZ

Edited by UEZ

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

##### Share on other sites

for me it works as expected on xp

##### Share on other sites

This obviously leads to correct values if the computation is simple.

What I say is that after lot of calculations on same iniital value, I loose some decimals.

If I iterate calculations, it happens.

Usually it's multiplying an integer with a 2 decimal value (i.e 2*.25)

M.

##### Share on other sites

that gives a wrong number

```#include <MsgBoxConstants.au3>

\$var1 = 2
\$var2 = 0.00000000001
\$sum = \$var1 * \$var2
MsgBox(\$MB_SYSTEMMODAL, "Title", StringFormat ("%.40f", \$sum))```

##### Share on other sites

marko001,

This is a problem that affects ALL computers regardless of the language used - google "floating point arithmetic" to learn all about it.

One of the normal workarounds when, as you say you are, dealing with 2 decimal place values (often currency) is to convert all the decimal values into integer values and then reconvert the result once the calculation has been made.

M23

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind

Open spoiler to see my UDFs:

Spoiler

ArrayMultiColSort ---- Sort arrays on multiple columns
ChooseFileFolder ---- Single and multiple selections from specified path treeview listing
Date_Time_Convert -- Easily convert date/time formats, including the language used
ExtMsgBox --------- A highly customisable replacement for MsgBox
GUIExtender -------- Extend and retract multiple sections within a GUI
GUIFrame ---------- Subdivide GUIs into many adjustable frames
GUIListViewEx ------- Insert, delete, move, drag, sort, edit and colour ListView items
GUITreeViewEx ------ Check/clear parent and child checkboxes in a TreeView
Marquee ----------- Scrolling tickertape GUIs
NoFocusLines ------- Remove the dotted focus lines from buttons, sliders, radios and checkboxes
Notify ------------- Small notifications on the edge of the display
Scrollbars ----------Automatically sized scrollbars with a single command
StringSize ---------- Automatically size controls to fit text
Toast -------------- Small GUIs which pop out of the notification area

##### Share on other sites

This is what I thought but hoped in a possible workaround.

I'll try what you suggested and I'll be back with results.

(I knew an aswer would have come from you   )

##### Share on other sites

For a workaround it depends what result you want from the calculation. 1.000000000000000000123 = 1? or 0.000000000019999999999999999 = 0?

Can you provide some examples what the calculation is and what result you are expecting.

Br,

UEZ

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

##### Share on other sites

I need precision on 2nd decimal.

1.000000000000000000123 = 1

0.000000000019999999999999999 = 0

I'm trying to replicate errors and I'll post you values and functions.

M.

##### Share on other sites

Then this should do your job:

```ConsoleWrite(StringFormat ("%.2f", 0.065934857935793) & @CRLF)
ConsoleWrite(StringFormat ("%.2f", 10.000023) & @CRLF)
ConsoleWrite(StringFormat ("%.2f", ACos(-1)) & @CRLF)
ConsoleWrite(StringFormat ("%.2f", 1.009) & @CRLF)```

Br,

UEZ

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯  ٩(●̮̮̃•̃)۶ ٩(-̮̮̃-̃)۶ૐ

##### Share on other sites

As Melba just said, there is nothing wrong here. You just can't expect an exact representation of most real values, let alone the result of any computation involving those values that can't represent exactly in limited precision binary floating point.

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

## Create an account

Register a new account

×

• Wiki

• Back

• #### Beta

• Git
• FAQ
• Our Picks
×
• Create New...