Jump to content

Calculation Mistakes in Large Arrays


Recommended Posts

Func C(ByRef $a,$Label);Returns Index Num Col of Seek Label in Array
    Local $lr=UBound($a,1)-1
    Local $lc=UBound($a,2)-1
    Local $iLabel="None"
    For $c=0 To $lc
        $iLabel=$a[0][$c]
        If $iLabel=$Label Then Return $c
    Next
    Debug("[ERROR]Label Not Found! "&$Label)
    Return "[ERROR]"
EndFunc

Func ByCn(ByRef $a,$Row,$Label)
    Return Number($a[$Row][C($a,$Label)],3)
EndFunc

Func ByC(ByRef $a,$Row,$Label)
    Return $a[$Row][C($a,$Label)]
EndFunc

 

image.png.73bdd8eb0f26ebf3fd598f7e889df4d6.png

 

When I run complex calculations in large arrays derived from .csv files with ByC() (returned without Number()) I get calculation errors. The calculation errors (straight up bogus outputs) seem to be gone after running the returns with Number(). Normally these errors aren't there when running smaller and less complex calculations. I was running 16 parallel processors and calculations still take 30 minutes, when I add the Number() function calculations take up to 24 hours... Is there a faster alternative to Number() or a way to import the arrays as float values & bypass these calculation errors?

 

Thanks!

Link to post
Share on other sites
  • Replies 40
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

it's not the number of lines you write, it's what they do. C and C++ are FAR superior to what the OP is doing and not using the appropriate language is a huge mistake and waste of time/energy/money

and (at the fear of repeating myself) E4A relies on internal dlls written (by yours truly)  in C++, based upon Eigen and boost. Eigen is used extensively in science, engineering, robotics, and CG. And

I think it only fair to count the lines inserted by any new #include statement, recursively of course

Posted Images

Although it is not clear what is your issue (since we do not have a replicable script to test on), there might be a possible solution using scripting dictionary (or maps - beta version).  Provide a full test environment if you want us to propose a precise solution...

Link to post
Share on other sites
Posted (edited)
On 8/10/2021 at 5:03 AM, Nine said:

Although it is not clear what is your issue (since we do not have a replicable script to test on), there might be a possible solution using scripting dictionary (or maps - beta version).  Provide a full test environment if you want us to propose a precise solution...

 

The data file is too large to upload into this forum. If there any functions that you need I'll provide them. But really I am just getting different calculation results between the use of with and without the use of the function Number(). The miscalculations are reproduceable across a large set of simulations I do, this is just one example. I get completely different results when I replace ByC() with ByCn(). This is a less complex simulation I do for forex, there are more complex simulations for equities that uses parallel processing. 

Edited by Dana86
Link to post
Share on other sites
20 hours ago, Nine said:

666 !

 

22 hours ago, pseakins said:

I suspect that Dana is trolling us.

Not, the type of demeanor that results in a better community of developers???

Link to post
Share on other sites

Well, i'm sorry about that, maybe not the answer you're looking for, but dealing with high computation in AutoIt is not really the point of the langague.

So if i was you, i would just code stuff in C ... AutoIt is intended to be a Scripting/Automation language ... I guess some people gonna hate me for that haha

And if you're afraid of C (some people are), you can still get really good perf with any language oriented for calculus (Python, R, Maple, Matlab, ...) or perfs (Ada, Fortran, ... (but if you're afraid of C ... well hahahaha)) but not a scripting language ...

Also, interpreted languages usually s*ck for computation when not explicitly designed for it ...

Edited by 636C65616E
Link to post
Share on other sites
12 hours ago, JockoDundee said:

Fool me once, shame on me…

 

I make my living with Algo-Trading and due to the nature of my work I can not share my code with everyone. 

Link to post
Share on other sites
1 minute ago, 636C65616E said:

Well, i'm sorry about that, maybe not the answer you're looking for, but dealing with high computation in AutoIt is not really the point of the langague.

So if i was you, i would just code stuff in C ... AutoIt is intended to be a Scripting/Automation language ... I guess some people gonna hate me for that haha

Ya, C and C++ is super fast but its way slower & harder to write. I can write a few thousand lines of code on Au3 in a day vs only a few hundred lines with C or C++.

Link to post
Share on other sites
Just now, Dana86 said:

Ya, C and C++ is super fast but its way slower & harder to write. I can write a few thousand lines of code on Au3 in a day vs only a few hundred lines with C or C++.

Then keep praticing, you will end by coding at the same speed ...

Link to post
Share on other sites
Posted (edited)
6 minutes ago, 636C65616E said:

Well, i'm sorry about that, maybe not the answer you're looking for, but dealing with high computation in AutoIt is not really the point of the langague.

So if i was you, i would just code stuff in C ... AutoIt is intended to be a Scripting/Automation language ... I guess some people gonna hate me for that haha

And if you're afraid of C (some people are), you can still get really good perf with Python or R ...

Thanks for being honest, I've tried R and Python the process of going from theory to simulation to implementation is much slower. Tho, I changed a few Au3 funcs to reflect python a bit to make my life easier coding between the two languages. 

Edited by Dana86
Link to post
Share on other sites
11 minutes ago, 636C65616E said:

Then keep praticing, you will end by coding at the same speed ...

ByC() was meant to be like the python equivalent of df.loc[] with labels instead of col numbers. Makes experiments more robust, scalable and possible for future data manipulation. 

iProfit=round((Price-df.loc[r,'EntryPrice'])*df.loc[r,'Size'],2)      
Link to post
Share on other sites
2 hours ago, Dana86 said:

I can write a few thousand lines of code on Au3 in a day…

That is fast typing, if nothing else…

3000 lines (few thousand) * 80 chars (avg chars per line) / 5 (avg chars in word) / 8 hours (avg working day) / 60 minutes (avg minutes in an hour) =

100 WPM

Impressive!

Code hard, but don’t hard code...

Link to post
Share on other sites
1 hour ago, JockoDundee said:

That is fast typing, if nothing else…

3000 lines (few thousand) * 80 chars (avg chars per line) / 5 (avg chars in word) / 8 hours (avg working day) / 60 minutes (avg minutes in an hour) =

100 WPM

Impressive!

In an experimental data scientist type environment, there is a lot of copy and pasting to get things done faster. 99.9% of experiments fail, so rapid development is very important. Scalability and stability comes later after successful initial experiments. Some days I work for 16 hours on longer experiments. An experiment is like an idea, but a very long drawn out idea that takes many hours to develop and test. Most ideas are are versions of older failed experiments and they get reused.

Link to post
Share on other sites
8 hours ago, Dana86 said:

I can write a few thousand lines of code on Au3 in a day

An experienced coder will write 60, maybe 80 lines of debugged code in a day. For each of those lines 50 - 100 may have been written, tested, edited and whole sections deleted. Copying and pasting untested experimental code is not exactly "coding" IMO. But, I am retired and earn nothing. You, if you are real and it looks like you may be, may be being paid over $100k so who am I to judge?

Phil Seakins

Link to post
Share on other sites
9 hours ago, Dana86 said:

C++ is super fast

and (at the fear of repeating myself) E4A relies on internal dlls written (by yours truly)  in C++, based upon Eigen and boost. Eigen is used extensively in science, engineering, robotics, and CG. And in x64-mode, data set size is limited only by your virtual memory. Yet to engage with it, you only need a few lines of AutoIt.

Link to post
Share on other sites

it's not the number of lines you write, it's what they do. C and C++ are FAR superior to what the OP is doing and not using the appropriate language is a huge mistake and waste of time/energy/money

the argument that you "get more done" is completely false. If you focus on C++ you will become expert in no time and whipping out solutions will be as fast or faster than using an autmation scripting tool.

Basically, in a nutshell--USE THE RIGHT TOOL FOR THE JOB. If speed and accuracy are your main considerations then don't use an interpreted language at all.

 

but if you must use AutoIt then use the UDF that @RTFC created.

Edited by Earthshine

My resources are limited. You must ask the right questions

 

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...