Multi-processing discussion

[kinch]

Hi all,

While I know NOT to mention the words "multi-threading" here, because AutoIT is not multi-threaded nor will it be according to the developers, I'm still interested in multi-processing - running multiple processes to achieve a more efficient result.

This is made very easy in AutoIT thanks to things like "Run" and "ShellExecute" which could allow you to spawn whatever applications or processes you want.  Plus, the wonderful ease of GUI creation means it's easy to give a simple user interface for spawning, modifying, and monitoring results.

I've tried to do my due diligence with the forum, but the best I found was '?do=embed' frameborder='0' data-embedContent>> which unfortunately doesn't seem to work anymore (I tried to fix up the UDF but I had no idea what I was doing or why).

And to get the awkward part out of the way first - I know an interpreted language like AutoIT will be slow, so it's not ideal for intense calculations and loops.  However, I believe using it as a wraparound for the "fast" functions could be of great benefit - heck, you could even dump numerical data into Excel, run the calculations and then grab the result.  The best of both worlds!

So anyway, I'd like to discuss multi-processing in a classic "master/slave" relationship, and get some ideas on possible implementation.

Idea #1: Named Pipes

So, you have a process running as the master which opens the pipe, and then have however many slave processes which connect to the master through the named pipe, to download "tasks" to execute.  Whatever your particular application is.  My understanding is that all PCs/processes will need to be able to connect to where the pipe is created though.  I've never used pipes for this kind of thing, are they any good?

Idea #2: Sockets

Similar to the above, but the master process opens a socket and receives incoming connections from the slaves.  Apart from that, it's similar to named pipes.  I guess this method has the advantage of (theoretically) being used by any PC on the internet/network, instead of having to have access to the server the pipe is created on.

Idea #3: SQLite

While probably the weakest (and, I suspect, slowest), you could have an SQLite table stored on a network drive, which could have the jobs to be completed, and then each process could lock the DB, update the row for "being worked on by me" and then unlock the DB.  It could also store it's results in the table, so that it's accessible from any process (including the master).  This does limit it to applications that could connect to that network resource though.

Idea #4: "Overseers"

I couldn't think of what else to call this, but overseers seems semi-relevant.  The idea being that on a PC you have a master, which accepts requests for work from another PC.  The concept of overseers is that it's a "mini-master" running on a (non-master) PC.  It connects to the master PC, downloads a bunch of jobs, and then creates sub-processes based on the number of processes/CPU cores.  This would mean that a user of that PC has more control over how much work the application(s) are doing.

Idea #5: Clipboard

(Idea stolen from the link pasted above)  This one is straightforward, you copy data to the clipboard, and the processes grab it and action it.  It's presumably limited to one PC, although I suspect it'd be pretty easy to implement (especially if it was just a small number of processes).

Discussion:

So then we get to the discussion.  I'm curious to know if you guys know of better ways (using AutoIT - I don't want to write a DLL in C++ and use DllCall+CreateThread() ) for multi-processing.  Given the above, is one  likely to be easier to implement than another?  Are there any efficiency concerns to take note of?  Can anyone tell me if I've made a glaring error or mistake in my reasoning?  I've never written a program using pipes, but I have done sockets in Perl/C a long time ago.

Or am I just wasting everyone's time including my own?

I have an upcoming project where I'll have to crunch massive amounts of data - farming it out to multiple PCs/CPUs sounds like a fine idea to me... and even if it never is efficient enough for that, I would probably use it for calculating primes or something.  It's just fun to multi process

Has anyone written something similar and would like to share their thoughts on the topic?

Cheers.

[FireFox]

Hi,

>This may interest you, though sockets are not a bad idea

Br, FireFox.

[jchd]

MailSlots are good too.

You keep mixing up IPC and multiprocessing. Also local IPC and remote IPC don't compare.

Don't use raw SQLite on a remote DB.

[fat]

>This   can help you

[RTFC]

@kinch: If you can wait a a few more weeks (once preliminary LAN testing is completed), I'm about to release the first test version of my Pool environment, which gives you almost total control over all your AutoIt processes (that contain the Pool #include) running anywhere on a LAN; this includes exchanging predefined and user-defined messages, LANchat, file/data transfers, shared structs, arrays, and variables, remote execution (single line, batch/separate script, Run/Shellexecute), and HTC synchronisation. I wrote it primarily for grid multiprocessing with my Eigen4AutoIt matrix computing environment (see signature), but it's become more generic and is still growing. It's not superfast and only partially event-driven. I think it'll be worth the wait, but then, I'm probably biased...

[cal]

I've used sockets a couple times when I needed scripts to talk to other scripts across the network.

I use mailslots when I want scripts to talk to each other on the same machine.  Works well and easy to do.   I use this udf found here.

'?do=embed' frameborder='0' data-embedContent>>

mailslots can only send strings to each other.   So I wrote an array converter so I could send arrays back and forth between running scripts.  I think most of that functionality is now built in.

 

Its old but it still works fine.

I've considered the overseers idea but I've never needed to get that complex that can handle general requests from various scripts.

Often I do need a script to spawn a helper script.  But its always related to the first script...  So I combine them into one script that calls itself.  I've been doing it this way for years.  I combine that method with mailslots so not only can one call itself to do other things but it can then talk further to it as needed.

The idea behind calling yourself is to check for command line params when the script first starts.  If your being called without command line params then you know your the first copy.   You can spawn your helper (call yourself but with further parm) and then continue to do your job.  Or do nothing and move on to the main part of your job.

If you do see you were called with parm.  Then your the helper.  so you jump to the func called by the parm and start doing that job.

Any running copy can also call further copies to do specific jobs simply by calling itself with the correct command line parms.

I have my mulit thread structure as a case statement near the top before anything else gets set.

I also have a couple lines that sets its own windows name according to the script name.  This is great when I'm still writing a script as I can force a newly started version to kill the other from memory before it takes its place.  This way I don't have to keep killing running scripts to see the effect of my new changes.

However this can play havoc when spawning helpers.  So dont forget to set another name related to the helper process.  An added benefit is that any script can check running apps to see if it has the proper helper is already in place or if it needs to spawn them.

I used to have a big long detailed post on that a few years ago.  But i cant find it or I'd link to it.

There are many ways to accomplish what your after.

[Guest]

My Idea #6 - Communicate through hidden GUI:

But there's a limitation - I could not completely hide the GUI for the communication process. But this problem should be easy to fix for who knows better than me about the GUI coding.

 

Main process:

; ----------------------------------------------------- Process 1 - Main Process ----------------------------------------------------
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>


$MHGCommunication = GUICreate("MHGCommunication", 0, 0) ; Main Hidden GUI Communication ..
WinSetTrans($MHGCommunication,'',0)
$MHEditCommunication = GUICtrlCreateEdit("", 8, 8, 97, 41) ; $MHEditCommunication Always equal to 3 in this simple case.
GUISetState(@SW_SHOW)

$TestData = 1


$timer = TimerInit()
Do
    If TimerDiff($timer) >= 1000 Then
        GUICtrlSetData($MHEditCommunication,$TestData)
        $TestData += 1
        $timer = TimerInit()
    EndIf
Until GUIGetMsg() = $GUI_EVENT_CLOSE

process 2:

; ----------------------------------------------------- Process 2 ----------------------------------------------------
; Run this when Process 1 is running!

$MainProcessGUI = WinGetHandle('MHGCommunication')
If @error Then Exit MsgBox(16,'Error','Error in WinGetHandle')

Do
    ToolTip('Process 2:'&@CRLF&'The data in Process 1 is:'&@CRLF&WinGetText($MainProcessGUI))
    Sleep(50)
Until Not WinExists($MainProcessGUI)

You must first run Process 1 and then Process 2.

You will see that Process 2 get the data in $TestData from Process 1.

I do not know if anyone thought of this method in the past.
Anyway, I hope this idea and method will be used by someone to develop some advanced UDF which is based on this method

[BrewManNH]

Every AutoIt script already has a hidden GUI when you run them. You can get the title of the GUI using AutoItWinGetTitle and setting it using AutoItWinSetTitle.

[Guest]

Every AutoIt script already has a hidden GUI when you run them. You can get the title of the GUI using AutoItWinGetTitle and setting it using AutoItWinSetTitle.

Ok.

I did not know about it.

But is there a limit of characters number? if not then I think that this hidden GUI is a better option.

In my method you can disable the characters limit because it use EDIT control

Edited February 25, 2015 by Guest

[zalomalo]

I am working in this moment wiht the better InterProccessCommunication way for simple tasks becuse is reliable and the fastest: NamedPipes.

By example a GUI from the proccess A shows progress and results from proccess B, C... etc.

[BrewManNH]

Ok.

I did not know about it.

But is there a limit of characters number? if not then I think that this hidden GUI is a better option.

In my method you can disable the characters limit because it use EDIT control

The AutoIt hidden GUI has an edit control on it as well. Run the example for AutoItWinGetTitle (settitle) and it will show you the window.

[zalomalo]

Ok.

I did not know about it.

 

Neither me knew about this default hidden window exist.

We learn new things all days, every day...