Dana86

In an experimental data scientist type environment, there is a lot of copy and pasting to get things done faster. 99.9% of experiments fail, so rapid development is very important. Scalability and stability comes later after successful initial experiments. Some days I work for 16 hours on longer experiments. An experiment is like an idea, but a very long drawn out idea that takes many hours to develop and test. Most ideas are are versions of older failed experiments and they get reused.

ByC() was meant to be like the python equivalent of df.loc[] with labels instead of col numbers. Makes experiments more robust, scalable and possible for future data manipulation. iProfit=round((Price-df.loc[r,'EntryPrice'])*df.loc[r,'Size'],2)

Thanks for being honest, I've tried R and Python the process of going from theory to simulation to implementation is much slower. Tho, I changed a few Au3 funcs to reflect python a bit to make my life easier coding between the two languages.

Ya, C and C++ is super fast but its way slower & harder to write. I can write a few thousand lines of code on Au3 in a day vs only a few hundred lines with C or C++.

I make my living with Algo-Trading and due to the nature of my work I can not share my code with everyone.

Not, the type of demeanor that results in a better community of developers???

  The data file is too large to upload into this forum. If there any functions that you need I'll provide them. But really I am just getting different calculation results between the use of with and without the use of the function Number(). The miscalculations are reproduceable across a large set of simulations I do, this is just one example. I get completely different results when I replace ByC() with ByCn(). This is a less complex simulation I do for forex, there are more complex simulations for equities that uses parallel processing.

I wish I had more time to game! But been too obsessed with writing new simulations & testing theories with autoit and python.

I use $CmdLine for keeping my python APIs from crashing. There are a lot of weird unexpected crashes in other people's wrappers. #include-once #include "E:\Autoit_FibFinder\Include\MyFunctions.au3" Globals() Func Globals() Global $lr=$CmdLine[0] Global $ProgramDir=$CmdLine[1] Global $PingDir=$CmdLine[2] EndFunc Main() Func Main() While 1 Sleep(5000) AlwaysOn(10) WEnd EndFunc Func AlwaysOn($MaxWait) XWriteTxt($PingDir,0) For $i=1 To $MaxWait If Number(XReadTxt($PingDir))=1 Then ;~ MsgBox($MB_SYSTEMMODAL,"$PingDir",$PingDir) ;~ MsgBox($MB_SYSTEMMODAL,"Ping",Number(XReadTxt($PingDir))) Return 1 EndIf Sleep(1000) Next If Number(XReadTxt($PingDir))=0 Then ;~ MsgBox($MB_SYSTEMMODAL,"$PingDir",$PingDir) ;~ MsgBox($MB_SYSTEMMODAL,"Ping",Number(XReadTxt($PingDir))) XWriteTxt($PingDir,1) Sleep(500) Run($ProgramDir) Sleep(5000) Return 0 EndIf EndFunc As stated above it wouldn't work for my case use AlgoTrading because the data files are too large. And new args need to be inserted whilst the large data csv is loaded into a global array once.

Func C(ByRef $a,$Label);Returns Index Num Col of Seek Label in Array Local $lr=UBound($a,1)-1 Local $lc=UBound($a,2)-1 Local $iLabel="None" For $c=0 To $lc $iLabel=$a[0][$c] If $iLabel=$Label Then Return $c Next Debug("[ERROR]Label Not Found! "&$Label) Return "[ERROR]" EndFunc Func ByCn(ByRef $a,$Row,$Label) Return Number($a[$Row][C($a,$Label)],3) EndFunc Func ByC(ByRef $a,$Row,$Label) Return $a[$Row][C($a,$Label)] EndFunc When I run complex calculations in large arrays derived from .csv files with ByC() (returned without Number()) I get calculation errors. The calculation errors (straight up bogus outputs) seem to be gone after running the returns with Number(). Normally these errors aren't there when running smaller and less complex calculations. I was running 16 parallel processors and calculations still take 30 minutes, when I add the Number() function calculations take up to 24 hours... Is there a faster alternative to Number() or a way to import the arrays as float values & bypass these calculation errors? Thanks!

[Update]Ported everything over to my workstation & it's using 35% of my 64 threads with 16 PP(parallel processors)! I am processing data that usually takes 3-4 hours on my laptop in 10 minutes!

I could go that route but things get complicated as the csv file is then used with another python app. CSV hasn't failed me yet.

Thank you, that was actually my main question. I wanted to use this method to create AI vs logic through .csv and .json files which are limiting. Parallel processing is a must for AI with Au3 vs Python has built in multi threading capabilities for AI & ML.

For my particular use case, command-line arguments wouldn't work because all 4 processors run off of the same csv file with 500k-1mill rows. The CSV load time is around 10-30 seconds each load. It is much faster to issue new arguments and executions through .txt files. The processors maintain global csv arrays in a while loop while waiting for new commands. ;~ Globals() Func Globals() Global $a CSVToArray($a,"E:\Data.csv") Global $lr=UBound($a,1)-1 Global $done[1] Global $UniqueCt=0 ;<-----------------------SETTINGS Global $sd=0.10 Global $mrt=1 Global $mp=1 Global $mpt=1 EndFunc ;Main(ReadTxt("PID")) Main(1) Func Main($Num) Globals() While 1 If ReadTxt("P"&$Num&"_NewJob")=1 Then WriteTxt("P"&$Num&"_NewJob",0) WriteTxt("P"&$Num,1) WriteTxt("Accepted",1) Local $Row=ReadTxt("ProcessRow") ;Debug("Processing Row: "&$Row) If $Row>10 Then $Row=$Row-10 Tracker(ReadTxt("ProcessID"),$Row) WriteTxt("P"&$Num,0) EndIf Sleep(250) WEnd EndFunc

It objectively saves time in my particular use case. It has to process 500k-1mill rows of .csv data in multiple layers of nested for loops. Tho it would be nice if I could make build a library for easier & faster execution in the future.