Having a Problem with some calculations (inaccuracies)

[nullschritt]

I've been playing with compressing data by using matricies. In case you don't know a matrix is like a 2d set of numbers, it has rows and columns(much like a 2 dimensional array full of numbers).

 

However I've ran into a problem. It seems that the more data I overlay onto the same matrix, the less accurate the output is. If I understand this is because the numbers are being represented in scientific notation, and digits are chopped off after the decimal point, meaning that when the math is reversed, you don't have the same number you started with.

 

My solution was to use the bignum UDF to try to retain the full value of the numbers, but it seems the decimal points are still being truncated, I might be using the UDF wrong but it seems pretty straight forward, all it does for me though is slow down the script with no more accurate results.

 

I need a solution for math that allows me to calculate with the FULL value of a number(no trimming of the decimal places, at all)

 

See my example script below and test/advise.

 

Thanks ahead of time. 

 

#include "bignum.au3"

dim $numbers[51]
for $i=0 to 50
$numbers[$i] = Random(111111111, 999999999, 1)
Next

$output= 2 ;original number

for $i=0 to 50
$output = _BigNum_Parse($output*$numbers[$i])
Next

ConsoleWrite($output&@CRLF) ;this shouldn't be truancated

for $i=0 to 50
$output = _BigNum_Parse($output/$numbers[50-$i])
Next

ConsoleWrite($output&@CRLF) ;this should be 2

Edited March 9, 2015 by nullschritt

[czardas]

Meh, this is hard. Int64 maxes out. The only thing I can think of would be to check that this will not happen by first doing the calculation using doubles. If the limit is reached then either you continue to use doubles, or you switch to using strings. Perhaps BigNum needs updating because it was created before Int64 was implemented, I'm not sure.

[markyrocks]

i think this says it best.... from the help file

Arrays in Arrays

You may save an array in an array element (item). The thing is, there is no way to directly access that array stored in the element. You have to go through a variable to get access to the embedded array which may make your code overly complicated and difficult to debug.

Remember that there may be issues if you pass an array containing arrays into a function and the embedded array is changed inside that function. So, to conclude this, try not to embed arrays within arrays unless you absolutely need to and are prepared to do rigorous testing to make sure your code will always work as expected.

 

what i really think is that your trying to break your computer or get out on the system.  or maybe you took one of those pills that unlocked 100% of your brain idk.  

Edited March 8, 2015 by markyrocks

[nullschritt]

@czardas
 
Thanks for the quick response, but how would I switch to using doubles, or how would I process a string as a number?(if I go to do math on the string it will convert it to an int and have the same limitation)

@markyrocks

The problem isn't occurring as a result of overlaying the arrays, if I were to multiply a single by other random integers in a loop, eventually I would end up with the same sort of data loss.

The array overlapping is essential for my compression algorithm, as it lays several layers of data into one, to later be reversed.

[SmOke_N]

?do=embed' frameborder='0' data-embedContent>>

Edited March 9, 2015 by SmOke_N

[czardas]

What I was trying to imply was that calculations using Int64 data types have to remain within range, otherwise the result can be wildly inaccurate - in exactly the way you described your results. It was a recent idea I had to first test the output of a calculation using doubles to check if it goes out of range and becomes too large for Int64 to handle. If this occurs then an alternative method has to be employed - such as bignum.

[czardas]

Just playing with the idea:

;

Local $nNum1 = 987654321098765432, _
      $nNum2 = 9, _
      $sOperator = "*"

Local $bOverflow = _Int64OverflowDetect($nNum1, $nNum2, $sOperator)
MsgBox(0, "Overflow will occur", $bOverflow)

Func _Int64OverflowDetect($nNum1, $nNum2, $sOperator)
    If Not StringRegExp($sOperator, '\A[\+\-\*\^/]\z') Then Return SetError(1) ; Operator not recognized
    $nNum1 = Number($nNum1, 3)
    $nNum2 = Number($nNum2, 3)
    Local $nTest = Execute('$nNum1' & $sOperator &'$nNum2')
    Return  $nTest < 0x8000000000000000 Or $nTest > 0x7FFFFFFFFFFFFFFF
EndFunc ;==> _Int64OverflowDetect

;

Actually the division operator and power operator always return doubles, so the function's utility will be affected in some instances. I'm not sure of the best way to deal with the limitation yet. You can always remove the three characters ^/ from the regular expression and add checks to only allow integer input.

Edited March 9, 2015 by czardas

[nullschritt]

?do=embed' frameborder='0' data-embedContent>>

 

Please read the full thread before replying, I specifically stated I've already tried this solution.

[nullschritt]

Just playing with the idea:

;

Local $nNum1 = 987654321098765432, _
      $nNum2 = 9, _
      $sOperator = "*"

Local $bOverflow = _Int64OverflowDetect($nNum1, $nNum2, $sOperator)
MsgBox(0, "Overflow will occur", $bOverflow)

Func _Int64OverflowDetect($nNum1, $nNum2, $sOperator)
    If Not StringRegExp($sOperator, '\A[\+\-\*\^/]\z') Then Return SetError(1) ; Operator not recognized
    $nNum1 = Number($nNum1, 3)
    $nNum2 = Number($nNum2, 3)
    Local $nTest = Execute('$nNum1' & $sOperator &'$nNum2')
    Return  $nTest < 0x8000000000000000 Or $nTest > 0x7FFFFFFFFFFFFFFF
EndFunc ;==> _Int64OverflowDetect

;

Actually the division operator and power operator always return doubles, so the function's utility will be affected in some instances. I'm not sure of the best way to deal with the limitation yet. You can always remove the three characters ^/ from the regular expression and add checks to only allow integer input.

Thanks for this solution, to check if the digit overflows, but....

Please NOTE. I have already mentioned that the bignum UDF returns the SAME results as not using it at all, in fact, the example script at the top of this thread impliments the bignum udf just to show exactly that.

So I need an alternative to the BigNum UDF.

[SmOke_N]

If you have any experience in forums, you'll know that writing a book is the least popular way to get anyone to "read" your entire thread or to get a response you're hoping for.

I stopped at your first paragraph to be honest, and decided to read more hands on with czardas (and to be honest, totally missed his bignum suggestion).

Now that I've at least read your 3rd paragraph, I'll say that you're SOL unless you fix the bignum udf or have some other method that you've found that does what you want but isn't written in this language that maybe someone can translate.

Edited March 9, 2015 by SmOke_N

[czardas]

Here's a method to get 17 digit accuracy (or there abouts) out of a double. It took me way too long to discover this.

;

Local $nDouble = 2^63
MsgBox(0, "17 digit accuracy", $nDouble & @CRLF & _DoubleToInt($nDouble)) ; prints 9223372036854775800

Func _DoubleToInt($fDouble)
    Local $sType = VarGetType($fDouble)
    If $sType = "Int32" Or $sType = "Int64" Then Return $fDouble
    If $sType <> "Double" Then Return SetError(1) ; Input is not a number.
    If StringRegExp($fDouble, '(\A\d+\z)') Then Return $fDouble
    If StringRegExp($fDouble, '(e\-|\.\d+\z)') Then Return Round($fDouble)
    Local $iExponent = Number(StringRight($fDouble, 3))
    Return StringLeft($fDouble, 1) & StringMid(StringFormat('%.' & $iExponent & 'e', $fDouble), 3, $iExponent)
EndFunc

Edited March 9, 2015 by czardas

[jchd]

If/when a simple characterization of bug(s) in the BigNum UDF then a fix is probably easy and that would be a great thing for everyone.

[czardas]

The innaccuracy of double precision is illustrated in this next example, which might (or might not) be useful to you. It is simpler to understand at least.

;

Local $fPi = 3.1415926535897932
MsgBox(0, "", StringFormat('%.16e', $fPi)) ; prints 3.1415926535897931e+000

Edited March 9, 2015 by czardas

[nullschritt]

Here's a method to get 17 digit accuracy (or there abouts) out of a double. It took me way too long to discover this.

;

Local $nDouble = 2^63
MsgBox(0, "17 digit accuracy", $nDouble & @CRLF & _DoubleToInt($nDouble)) ; prints 9223372036854775800

Func _DoubleToInt($fDouble)
    Local $sType = VarGetType($fDouble)
    If $sType = "Int32" Or $sType = "Int64" Then Return $fDouble
    If $sType <> "Double" Then Return SetError(1) ; Input is not a number.
    If StringRegExp($fDouble, '(\A\d+\z)') Then Return $fDouble
    If StringRegExp($fDouble, '(e\-|\.\d+\z)') Then Return Round($fDouble)
    Local $iExponent = Number(StringRight($fDouble, 3))
    Return StringLeft($fDouble, 1) & StringMid(StringFormat('%.' & $iExponent & 'e', $fDouble), 3, $iExponent)
EndFunc

Thanks!, but how exactly would I go about implementing this?

I tried wrapping all my mathematical calculations inside the function but it doesn't seem to work in that way.

 

See below:

$a[$pivrow][$i] = _DoubleToInt($a[$pivrow][$i] / $pivelt)

$a[$tarrow][$i] = _DoubleToInt($a[$tarrow][$i] - $a[$pivrow][$i] * $tarelt)

etc....

Also tried it like this but still not getting it.

$a[$pivrow][$i] = _DoubleToInt($a[$pivrow][$i]) / _DoubleToInt($pivelt)
$a[$tarrow][$i] = _DoubleToInt($a[$tarrow][$i]) - _DoubleToInt($a[$pivrow][$i]) * _DoubleToInt($tarelt)

Edited March 9, 2015 by nullschritt

[czardas]

There's so much code and I couldn't see all of the values in the images actually I could, but I thought there might have been some exponentials off screen - I see now.  What I posted in #11 converts large doubles to integers. I'm not sure how you intend the output to be formatted. Let me look again. Did you see my my last post?

Edited March 9, 2015 by czardas

[nullschritt]

There's so much code and I couldn't see all of the values in the images actually I could, but I thought there might have been some exponentials off screen - I see now.  What I posted in #11 converts large doubles to integers. I'm not sure how you intend the output to be formatted. Let me look again. Did you see my my last post?

Yes but I don't understand how to implement your examples, I'm not really understanding.

My output? I just want to be able to preform math operations with improved accuracy, I need a way to process my values as floats or strings, but I have no idea how to go about even making my output doubles.

I'm not understanding how you're getting the 17 digit accuracy. What I'm seeing is regular math done by autoit(which results in a double, because of the exponent), then being processed to an int.

I need to know how to get autoit to work with doubles on all my calculations, not just when I divide or use an exponent, but with all mathematical operations I'm preforming.

Do I need to preform stringformat() on all my numbers in the operation to convert them to doubles?

Sorry if I'm seeming a little thick, I've just never had to convert anything from an int to a double. (this is what I need to do right? I'm hoping I'm not missing the entire point)

Or is it the script automatically using doubles on certain operations that is causing my inaccuracy? 

Edited March 9, 2015 by nullschritt

[jchd]

Binary FP is inherently prone to approximation and truncation to values that can actually be represented internally.

What you say about BigNum surprises me. For instance you can get pretty close to 15 when doing this:

#include "..\include\bignum.au3"

Local $sqrt3 = _BigNum_SQRT("3", 140)
Local $res = _BigNum_Mul($sqrt3, _BigNum_Mul($sqrt3, "5"))
ConsoleWrite($res & @LF)

You can still increase required accuracy if you need more than 140 decimal digits...

Can you give an example where BigNum fails to give a correct answer?

[czardas]

is it the script automatically using doubles on certain operations that is causing my inaccuracy? 

 

When you divide, AutoIt automatically creates a double. The accuracy is limited, as shown above. What you do really depends on the intended purpose. I don't know what you are trying to achieve with your script.

[nullschritt]

I'm trying to multiply sets of numbers over top of each other.

Then reverse the process by dividing them back to their original values.

The problem is that at some point, either in multiplying the numbers, and their decimal points being truancated, or at dividing them and them being coverted to doubles and losing their real value.

Edited March 9, 2015 by nullschritt

[nullschritt]

Like I said the purpose is to compress several layers of data into one single layer, by using matrix multiplication then to get the original data back, by using matrix division. But the output of the division is not the same as the original data, and it should be. Like I said I'm not sure if the problem is that multiplying the numbers so many times over causes them to lost vital decimal points, and as such the resulting reverse process (dividing) doesn't return the proper data because it doesn't have all the decimal places(my original thought of what the problem is), OR if the problem is as simple as the doubles returned by dividing being inaccurate(an idea you presented).