Jump to content

Calculation Mistakes in Large Arrays

Dana86

By Dana86
in AutoIt General Help and Support

 Share

Recommended Posts

Dana86 Wayfarer

Dana86

Posted
Posted 
Func C(ByRef $a,$Label);Returns Index Num Col of Seek Label in Array
    Local $lr=UBound($a,1)-1
    Local $lc=UBound($a,2)-1
    Local $iLabel="None"
    For $c=0 To $lc
        $iLabel=$a[0][$c]
        If $iLabel=$Label Then Return $c
    Next
    Debug("[ERROR]Label Not Found! "&$Label)
    Return "[ERROR]"
EndFunc

Func ByCn(ByRef $a,$Row,$Label)
    Return Number($a[$Row][C($a,$Label)],3)
EndFunc

Func ByC(ByRef $a,$Row,$Label)
    Return $a[$Row][C($a,$Label)]
EndFunc

 

image.png.73bdd8eb0f26ebf3fd598f7e889df4d6.png

 

When I run complex calculations in large arrays derived from .csv files with ByC() (returned without Number()) I get calculation errors. The calculation errors (straight up bogus outputs) seem to be gone after running the returns with Number(). Normally these errors aren't there when running smaller and less complex calculations. I was running 16 parallel processors and calculations still take 30 minutes, when I add the Number() function calculations take up to 24 hours... Is there a faster alternative to Number() or a way to import the arrays as float values & bypass these calculation errors?

 

Thanks!

  • Replies 40
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

Earthshine
Earthshine

it's not the number of lines you write, it's what they do. C and C++ are FAR superior to what the OP is doing and not using the appropriate language is a huge mistake and waste of time/energy/money

JockoDundee
JockoDundee

I think it only fair to count the lines inserted by any new #include statement, recursively of course

RTFC
RTFC

and (at the fear of repeating myself) E4A relies on internal dlls written (by yours truly)  in C++, based upon Eigen and boost. Eigen is used extensively in science, engineering, robotics, and CG. And

Posted Images

Nine Universalist

Nine

Posted
Posted

Although it is not clear what is your issue (since we do not have a replicable script to test on), there might be a possible solution using scripting dictionary (or maps - beta version).  Provide a full test environment if you want us to propose a precise solution...

“They did not know it was impossible, so they did it” ― Mark Twain

  Reveal hidden contents

 

Nine Universalist

Nine

Posted
Posted
  On 8/10/2021 at 11:01 PM, pseakins said:

I suspect that Dana is trolling us.

Expand  

666 !

“They did not know it was impossible, so they did it” ― Mark Twain

  Reveal hidden contents

 

Dana86 Wayfarer

Dana86

Posted
  • Author
Posted (edited)
  On 8/10/2021 at 12:03 PM, Nine said:

Although it is not clear what is your issue (since we do not have a replicable script to test on), there might be a possible solution using scripting dictionary (or maps - beta version).  Provide a full test environment if you want us to propose a precise solution...

Expand   
 

The data file is too large to upload into this forum. If there any functions that you need I'll provide them. But really I am just getting different calculation results between the use of with and without the use of the function Number(). The miscalculations are reproduceable across a large set of simulations I do, this is just one example. I get completely different results when I replace ByC() with ByCn(). This is a less complex simulation I do for forex, there are more complex simulations for equities that uses parallel processing. 

Edited by Dana86
636C65616E Wayfarer

636C65616E

Posted
Posted (edited)

Well, i'm sorry about that, maybe not the answer you're looking for, but dealing with high computation in AutoIt is not really the point of the langague.

So if i was you, i would just code stuff in C ... AutoIt is intended to be a Scripting/Automation language ... I guess some people gonna hate me for that haha

And if you're afraid of C (some people are), you can still get really good perf with any language oriented for calculus (Python, R, Maple, Matlab, ...) or perfs (Ada, Fortran, ... (but if you're afraid of C ... well hahahaha)) but not a scripting language ...

Also, interpreted languages usually s*ck for computation when not explicitly designed for it ...

Edited by 636C65616E
Dana86 Wayfarer

Dana86

Posted
  • Author
Posted
  On 8/11/2021 at 9:06 PM, 636C65616E said:

Well, i'm sorry about that, maybe not the answer you're looking for, but dealing with high computation in AutoIt is not really the point of the langague.

So if i was you, i would just code stuff in C ... AutoIt is intended to be a Scripting/Automation language ... I guess some people gonna hate me for that haha

Expand  

Ya, C and C++ is super fast but its way slower & harder to write. I can write a few thousand lines of code on Au3 in a day vs only a few hundred lines with C or C++.

636C65616E Wayfarer

636C65616E

Posted
Posted
  On 8/11/2021 at 9:09 PM, Dana86 said:

Ya, C and C++ is super fast but its way slower & harder to write. I can write a few thousand lines of code on Au3 in a day vs only a few hundred lines with C or C++.

Expand  

Then keep praticing, you will end by coding at the same speed ...

Dana86 Wayfarer

Dana86

Posted
  • Author
Posted (edited)
  On 8/11/2021 at 9:06 PM, 636C65616E said:

Well, i'm sorry about that, maybe not the answer you're looking for, but dealing with high computation in AutoIt is not really the point of the langague.

So if i was you, i would just code stuff in C ... AutoIt is intended to be a Scripting/Automation language ... I guess some people gonna hate me for that haha

And if you're afraid of C (some people are), you can still get really good perf with Python or R ...

Expand  

Thanks for being honest, I've tried R and Python the process of going from theory to simulation to implementation is much slower. Tho, I changed a few Au3 funcs to reflect python a bit to make my life easier coding between the two languages. 

Edited by Dana86
Dana86 Wayfarer

Dana86

Posted
  • Author
Posted
  On 8/11/2021 at 9:10 PM, 636C65616E said:

Then keep praticing, you will end by coding at the same speed ...

Expand  

ByC() was meant to be like the python equivalent of df.loc[] with labels instead of col numbers. Makes experiments more robust, scalable and possible for future data manipulation. 

iProfit=round((Price-df.loc[r,'EntryPrice'])*df.loc[r,'Size'],2)      
JockoDundee Universalist

JockoDundee

Posted
Posted
  On 8/11/2021 at 9:09 PM, Dana86 said:

I can write a few thousand lines of code on Au3 in a day…

Expand  

That is fast typing, if nothing else…

3000 lines (few thousand) * 80 chars (avg chars per line) / 5 (avg chars in word) / 8 hours (avg working day) / 60 minutes (avg minutes in an hour) =

100 WPM

Impressive!

Code hard, but don’t hard code...

Dana86 Wayfarer

Dana86

Posted
  • Author
Posted
  On 8/11/2021 at 11:18 PM, JockoDundee said:

That is fast typing, if nothing else…

3000 lines (few thousand) * 80 chars (avg chars per line) / 5 (avg chars in word) / 8 hours (avg working day) / 60 minutes (avg minutes in an hour) =

100 WPM

Impressive!

Expand  

In an experimental data scientist type environment, there is a lot of copy and pasting to get things done faster. 99.9% of experiments fail, so rapid development is very important. Scalability and stability comes later after successful initial experiments. Some days I work for 16 hours on longer experiments. An experiment is like an idea, but a very long drawn out idea that takes many hours to develop and test. Most ideas are are versions of older failed experiments and they get reused.

pseakins Polymath

pseakins

Posted
Posted
  On 8/11/2021 at 9:09 PM, Dana86 said:

I can write a few thousand lines of code on Au3 in a day

Expand  

An experienced coder will write 60, maybe 80 lines of debugged code in a day. For each of those lines 50 - 100 may have been written, tested, edited and whole sections deleted. Copying and pasting untested experimental code is not exactly "coding" IMO. But, I am retired and earn nothing. You, if you are real and it looks like you may be, may be being paid over $100k so who am I to judge?

Phil Seakins

RTFC Universalist

RTFC

Posted
Posted
  On 8/11/2021 at 9:09 PM, Dana86 said:

C++ is super fast

Expand  

and (at the fear of repeating myself) E4A relies on internal dlls written (by yours truly)  in C++, based upon Eigen and boost. Eigen is used extensively in science, engineering, robotics, and CG. And in x64-mode, data set size is limited only by your virtual memory. Yet to engage with it, you only need a few lines of AutoIt.

My Contributions and Wrappers

  Reveal hidden contents
Earthshine Universalist

Earthshine

Posted
Posted (edited)

it's not the number of lines you write, it's what they do. C and C++ are FAR superior to what the OP is doing and not using the appropriate language is a huge mistake and waste of time/energy/money

the argument that you "get more done" is completely false. If you focus on C++ you will become expert in no time and whipping out solutions will be as fast or faster than using an autmation scripting tool.

Basically, in a nutshell--USE THE RIGHT TOOL FOR THE JOB. If speed and accuracy are your main considerations then don't use an interpreted language at all.

 

but if you must use AutoIt then use the UDF that @RTFC created.

Edited by Earthshine

My resources are limited. You must ask the right questions

 

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...