Sign in to follow this  
Followers 0
kinch

Multi-processing discussion

12 posts in this topic

Hi all,

While I know NOT to mention the words "multi-threading" here, because AutoIT is not multi-threaded nor will it be according to the developers, I'm still interested in multi-processing - running multiple processes to achieve a more efficient result.

This is made very easy in AutoIT thanks to things like "Run" and "ShellExecute" which could allow you to spawn whatever applications or processes you want.  Plus, the wonderful ease of GUI creation means it's easy to give a simple user interface for spawning, modifying, and monitoring results.

I've tried to do my due diligence with the forum, but the best I found was '?do=embed' frameborder='0' data-embedContent>> which unfortunately doesn't seem to work anymore (I tried to fix up the UDF but I had no idea what I was doing or why).

And to get the awkward part out of the way first - I know an interpreted language like AutoIT will be slow, so it's not ideal for intense calculations and loops.  However, I believe using it as a wraparound for the "fast" functions could be of great benefit - heck, you could even dump numerical data into Excel, run the calculations and then grab the result.  The best of both worlds!

So anyway, I'd like to discuss multi-processing in a classic "master/slave" relationship, and get some ideas on possible implementation.

Idea #1: Named Pipes

So, you have a process running as the master which opens the pipe, and then have however many slave processes which connect to the master through the named pipe, to download "tasks" to execute.  Whatever your particular application is.  My understanding is that all PCs/processes will need to be able to connect to where the pipe is created though.  I've never used pipes for this kind of thing, are they any good?

Idea #2: Sockets

Similar to the above, but the master process opens a socket and receives incoming connections from the slaves.  Apart from that, it's similar to named pipes.  I guess this method has the advantage of (theoretically) being used by any PC on the internet/network, instead of having to have access to the server the pipe is created on.

Idea #3: SQLite

While probably the weakest (and, I suspect, slowest), you could have an SQLite table stored on a network drive, which could have the jobs to be completed, and then each process could lock the DB, update the row for "being worked on by me" and then unlock the DB.  It could also store it's results in the table, so that it's accessible from any process (including the master).  This does limit it to applications that could connect to that network resource though.

Idea #4: "Overseers"

I couldn't think of what else to call this, but overseers seems semi-relevant.  The idea being that on a PC you have a master, which accepts requests for work from another PC.  The concept of overseers is that it's a "mini-master" running on a (non-master) PC.  It connects to the master PC, downloads a bunch of jobs, and then creates sub-processes based on the number of processes/CPU cores.  This would mean that a user of that PC has more control over how much work the application(s) are doing.

Idea #5: Clipboard

(Idea stolen from the link pasted above)  This one is straightforward, you copy data to the clipboard, and the processes grab it and action it.  It's presumably limited to one PC, although I suspect it'd be pretty easy to implement (especially if it was just a small number of processes).

Discussion:

So then we get to the discussion.  I'm curious to know if you guys know of better ways (using AutoIT - I don't want to write a DLL in C++ and use DllCall+CreateThread() ) for multi-processing.  Given the above, is one  likely to be easier to implement than another?  Are there any efficiency concerns to take note of?  Can anyone tell me if I've made a glaring error or mistake in my reasoning?  I've never written a program using pipes, but I have done sockets in Perl/C a long time ago.

Or am I just wasting everyone's time including my own?

I have an upcoming project where I'll have to crunch massive amounts of data - farming it out to multiple PCs/CPUs sounds like a fine idea to me... and even if it never is efficient enough for that, I would probably use it for calculating primes or something.  It's just fun to multi process ;)

Has anyone written something similar and would like to share their thoughts on the topic?

Cheers.

Share this post


Link to post
Share on other sites



Hi,

>This may interest you, though sockets are not a bad idea ;)

Br, FireFox.


 

OS : Win XP SP2 (32 bits) / Win 7 SP1 (64 bits) / Win 8 (64 bits) | Autoit version: latest stable / beta.
Hardware : Intel(R) Core(TM) i5-2400 CPU @ 3.10Ghz / 8 GiB RAM DDR3.

My UDFs : Skype UDF | TrayIconEx UDF | GUI Panel UDF | Excel XML UDF | Is_Pressed_UDF

My Projects : YouTube Multi-downloader | FTP Easy-UP | Lock'n | WinKill | AVICapture | Skype TM | Tap Maker | ShellNew | Scriptner | Const Replacer | FT_Pocket | Chrome theme maker

My Examples : Capture toolIP Camera | Crosshair | Draw Captured Region | Picture Screensaver | Jscreenfix | Drivetemp | Picture viewer

My Snippets : Basic TCP | Systray_GetIconIndex | Intercept End task | Winpcap various | Advanced HotKeySet | Transparent Edit control

 

Share this post


Link to post
Share on other sites

MailSlots are good too.

You keep mixing up IPC and multiprocessing. Also local IPC and remote IPC don't compare.

Don't use raw SQLite on a remote DB.


This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

@kinch: If you can wait a a few more weeks (once preliminary LAN testing is completed), I'm about to release the first test version of my Pool environment, which gives you almost total control over all your AutoIt processes (that contain the Pool #include) running anywhere on a LAN; this includes exchanging predefined and user-defined messages, LANchat, file/data transfers, shared structs, arrays, and variables, remote execution (single line, batch/separate script, Run/Shellexecute), and HTC synchronisation. I wrote it primarily for grid multiprocessing with my Eigen4AutoIt matrix computing environment (see signature), but it's become more generic and is still growing. It's not superfast and only partially event-driven. I think it'll be worth the wait, but then, I'm probably biased... ;)

1 person likes this

Share this post


Link to post
Share on other sites

I've used sockets a couple times when I needed scripts to talk to other scripts across the network.

I use mailslots when I want scripts to talk to each other on the same machine.  Works well and easy to do.   I use this udf found here.

'?do=embed' frameborder='0' data-embedContent>>

mailslots can only send strings to each other.   So I wrote an array converter so I could send arrays back and forth between running scripts.  I think most of that functionality is now built in.

 

Its old but it still works fine.

I've considered the overseers idea but I've never needed to get that complex that can handle general requests from various scripts.

Often I do need a script to spawn a helper script.  But its always related to the first script...  So I combine them into one script that calls itself.  I've been doing it this way for years.  I combine that method with mailslots so not only can one call itself to do other things but it can then talk further to it as needed.

The idea behind calling yourself is to check for command line params when the script first starts.  If your being called without command line params then you know your the first copy.   You can spawn your helper (call yourself but with further parm) and then continue to do your job.  Or do nothing and move on to the main part of your job.

If you do see you were called with parm.  Then your the helper.  so you jump to the func called by the parm and start doing that job.

Any running copy can also call further copies to do specific jobs simply by calling itself with the correct command line parms.

I have my mulit thread structure as a case statement near the top before anything else gets set.

I also have a couple lines that sets its own windows name according to the script name.  This is great when I'm still writing a script as I can force a newly started version to kill the other from memory before it takes its place.  This way I don't have to keep killing running scripts to see the effect of my new changes.

However this can play havoc when spawning helpers.  So dont forget to set another name related to the helper process.  An added benefit is that any script can check running apps to see if it has the proper helper is already in place or if it needs to spawn them.

I used to have a big long detailed post on that a few years ago.  But i cant find it or I'd link to it.

There are many ways to accomplish what your after.

Share this post


Link to post
Share on other sites

My Idea #6 - Communicate through hidden GUI:

But there's a limitation - I could not completely hide the GUI for the communication process. But this problem should be easy to fix for who knows better than me about the GUI coding.

 

Main process:

; ----------------------------------------------------- Process 1 - Main Process ----------------------------------------------------
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>


$MHGCommunication = GUICreate("MHGCommunication", 0, 0) ; Main Hidden GUI Communication ..
WinSetTrans($MHGCommunication,'',0)
$MHEditCommunication = GUICtrlCreateEdit("", 8, 8, 97, 41) ; $MHEditCommunication Always equal to 3 in this simple case.
GUISetState(@SW_SHOW)

$TestData = 1


$timer = TimerInit()
Do
    If TimerDiff($timer) >= 1000 Then
        GUICtrlSetData($MHEditCommunication,$TestData)
        $TestData += 1
        $timer = TimerInit()
    EndIf
Until GUIGetMsg() = $GUI_EVENT_CLOSE

process 2:

; ----------------------------------------------------- Process 2 ----------------------------------------------------
; Run this when Process 1 is running!

$MainProcessGUI = WinGetHandle('MHGCommunication')
If @error Then Exit MsgBox(16,'Error','Error in WinGetHandle')

Do
    ToolTip('Process 2:'&@CRLF&'The data in Process 1 is:'&@CRLF&WinGetText($MainProcessGUI))
    Sleep(50)
Until Not WinExists($MainProcessGUI)

You must first run Process 1 and then Process 2.

You will see that Process 2 get the data in $TestData from Process 1.

I do not know if anyone thought of this method in the past.
Anyway, I hope this idea and method will be used by someone to develop some advanced UDF which is based on this method

Share this post


Link to post
Share on other sites

Every AutoIt script already has a hidden GUI when you run them. You can get the title of the GUI using AutoItWinGetTitle and setting it using AutoItWinSetTitle.


If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

Every AutoIt script already has a hidden GUI when you run them. You can get the title of the GUI using AutoItWinGetTitle and setting it using AutoItWinSetTitle.

Ok.

I did not know about it.

But is there a limit of characters number? if not then I think that this hidden GUI is a better option.

In my method you can disable the characters limit because it use EDIT control

Edited by gil900

Share this post


Link to post
Share on other sites

I am working in this moment wiht the better InterProccessCommunication way for simple tasks becuse is reliable and the fastest: NamedPipes.

By example a GUI from the proccess A shows progress and results from proccess B, C... etc.


My english shucks, i know it.

Share this post


Link to post
Share on other sites

Ok.

I did not know about it.

But is there a limit of characters number? if not then I think that this hidden GUI is a better option.

In my method you can disable the characters limit because it use EDIT control

The AutoIt hidden GUI has an edit control on it as well. Run the example for AutoItWinGetTitle (settitle) and it will show you the window.


If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Share this post


Link to post
Share on other sites

Ok.

I did not know about it.

 

Neither me knew about this default hidden window exist.

We learn new things all days, every day...  :thumbsup:


My english shucks, i know it.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0