22 replies to this topic

[thomasl]

I know this is going to earn me an earful or two but I'll say it nevertheless.

I find it bizarre and needlessly constraining that AutoIt3 scripts can't be used as simple command line tools.* There is absolutely no technical reason why this has to be so. There are perfectly valid ways and means to eat that cake and have it.

Are there any rational arguments (ie not of the "we simply don't like to do it like that" sort) why AU3 scripts won't work as command line tools? After all, whole OSes grew around command line tools and they still have a place, even in today's GUI-centred world.

Why close so useful a door, why restrict AU3 to windows only?

* They can, of course. But to do this one has to go through a small series of contortions that are probably beyond most users.

[SmOke_N]

Edit:
maybe I'm not understanding completely. I think this is stemming off the azu thread after re-reading it.

Edited by SmOke_N, 15 September 2006 - 12:02 PM.

[thomasl]

Edit:
maybe I'm not understanding completely. I think this is stemming off the azu thread after re-reading it.

I am not sure what "azu thread" you are referring to. If you mean that talk about a Linux version of AutoIt3 -- well, re-read what I wrote once more. My post is NOT about Linux. It is about using AU3 scripts in a console window. (Yeah, Windows does have command lines and command interpreters, stuff like cmd.exe or 4NT.)

Perhaps it helps if I repeat that: I am NOT talking about Linux.

(Edit: windows with a small "w" means these useful holes in the wall, people sometimes jump out of them. Or perhaps it means those funny rectangles on a PC screen.

Windows with a capital W is an OS from Microsoft.)

Edited by thomasl, 15 September 2006 - 02:55 PM.

[Valik]

I find it bizarre and needlessly constraining that AutoIt3 scripts can't be used as simple command line tools.* There is absolutely no technical reason why this has to be so. There are perfectly valid ways and means to eat that cake and have it.

Oh? And what knowledge makes you think there is no technical reason?

Are there any rational arguments (ie not of the "we simply don't like to do it like that" sort) why AU3 scripts won't work as command line tools? After all, whole OSes grew around command line tools and they still have a place, even in today's GUI-centred world.

Yes, there is a rational argument, it's a technical reason.

Why close so useful a door, why restrict AU3 to windows only?

The last time I checked, the door wasn't closed. The door was half open since AutoIt can natively redirect streams of other applications.

* They can, of course. But to do this one has to go through a small series of contortions that are probably beyond most users.

Too bad for most users, then. It's not nearly as hard as you might think, though.

This has been answered about a billion times. It's even signature fodder for at least two people. Windows applications, as I'm sure you know, are not connected to a console when they are run, even from a console. That's a fact of Windows life. If you want to bridge the gap between a GUI and CUI application, you need a tool to reflect the streams. I've written just such a tool and it can be found by searching the forum (It's called stub.exe).

There are no plans to do what VBS does and provide GUI and CUI compiled versions. IMO, that's a waste. In fact that's 4x the waste since I already feel we don't need AutoItSC.bin any longer. It's very simple to use stub.exe to invoke an AutoIt script to make an AutoIt script behave like a console application. I even found out that on Windows XP you can use AttachConsole() (Windows API) and gain access directly to the frame buffer of the console, all from a native GUI compiled application.

[thomasl]

Oh? And what knowledge makes you think there is no technical reason?

Hm, let me think... my ~20 years with Windows SDKs perhaps?

Yes, there is a rational argument, it's a technical reason.

Go ahead, I am all ears. What exactly then is your "technical reason"? (Never fear, I can stand technical details.)

The door was half open since AutoIt can natively redirect streams of other applications.

True enough, and a nice feature. But as you say, door's only half open (at best). I want my doors fully open

Windows applications, as I'm sure you know, are not connected to a console when they are run, even from a console. That's a fact of Windows life.

You should know that this is bollocks. The execution environment that's chosen by the OS loader depends solely on one linker switch. So-called GUI apps can call all CUI functions and vice versa. The stream of executable code simply has neither any knowledge of nor cares whether it's a GUI or a CUI app.

There are no plans to do what VBS does and provide GUI and CUI compiled versions.

What a pity. All the more so as there is absolutely no code difference between the two, it all boils down to that linker switch I just mentioned.

IMO, that's a waste.

Yes, in a very narrow, bean-counting sense, it's a waste. (And Microsoft should have seen this coming... as they should've seen many other things coming. Sigh.) And yet, who cares whether there's one executable or two or four? I'd hazard a guess that many (if not most) people don't even check what gets installed on their machines (or question why it's there).

I even found out that on Windows XP you can use AttachConsole() (Windows API) and gain access directly to the frame buffer of the console, all from a native GUI compiled application.

Good boy I already told you that there's effectively no difference between CUI and GUI, so it's no wonder that a GUI app can use AttachConsole(). Or a CUI app call CreateWindow()...

Two remarks though. 1) AttachConsole() works only in newer versions of Windows, ie XP, Vista and Server 2003. 2) This call is not needed in the context we're discussing anyway, as AllocConsole() returns without allocating a fresh console if the calling program is already attached to a console.

To sum up: I am still waiting for technical reasons; up till now there was only your waste argument whose merit is debatable.

Anyway, I'll have a look into that stub you mentioned; if that does what I want then so be it. I am not dogmatic about the how's and why's; main thing is that I think it should be possible to use AU3 as a fully-fledged command line tool.

[Valik]

The technical reason, you know already. GUI applications do not attach to a console. We do not provide a CUI compiled version of AutoIt. Since we do not provide a CUI version, aside from hacking the binary, there is no way to get AutoIt to talk to the console it was spawned from without the help of another program.

Regarding point #2, how can AllocConsole() return the attached console for a GUI application? It doesn't have a console attached, otherwise it would be CUI and we wouldn't be having this conversation.

And you'll find that stub.exe does everything you need. It allows writing to std streams from AutoIt to echo correctly to the console. It accepts stdin input. Also, as I mentioned, on XP and later, you can use AttachConsole() to attach to the console of the the parent process (That would be stub.exe) to gain frame buffer access for the console. If there's anything it can't do, I don't know about it.

The other alternative is to hack the binary to change the flag. I've never condoned this. It may be true, and I can see as much by looking at crtexe.c, that both entry points use the same startup code. Still, I'd rather not take the chance of introducing a subtle bug as a result of switching the flag after compilation.

[thomasl]

The technical reason, you know already. GUI applications do not attach to a console.

That's not a reason, that's stating the obvious. A GUI-linked version of an app is treated by the loader as a GUI app; whereas a console-linked version (which, let's not forget, would be identical as far as the actual stream of executable bytes is concerned) would either be attached to an existing console or have one created. (That, BTW, was what I meant with my AlloConsole() remark. After thinking through the issue I agree that this call is irrelevant in what we're discussing here.)

We do not provide a CUI compiled version of AutoIt.

Yes, now we're getting to the crux of the matter. The exe, as supplied, is linked as a GUI app. But linking the whole as a CUI exe would be a trivial exercise (it only requires a different linker switch).

So effectively, the question is not a technical one. Rather, it all boils down to your (or the developers', I am not sure whether you're speaking for all of them here) visceral dislike of a second executable, for a CUI scripts.

Where does that visceral dislike come from? What is the rationale behind that? What is so horribly bad about a second exe that you went to the trouble of writing stub.exe? Which brings me to...

And you'll find that stub.exe does everything you need.

As long as we're looking at simple stdin-stdout type programs, it does almost all... the two things it doesn't do are minor niggles. "Real" console stuff is a different story though.

However, stub.exe is an admittedly ingenious way of achieving something that could be more easily (and IMHO in a much cleaner and, dare I say it, more elegant way) achieved with an console-linked version of the exe. (I am tempted to write that stub.exe is a clever hack but as I know how you dislike that dirty word I won't.)

The other alternative is to hack the binary to change the flag.

Not true. What you disparagingly call "hacking the binary" is bit for bit equivalent to linking the obj files against the console subsystem.

It may be true, and I can see as much by looking at crtexe.c, that both entry points use the same startup code. Still, I'd rather not take the chance of introducing a subtle bug as a result of switching the flag after compilation.

Valik, will you ever get it into this head of yours that this flag is purely a linker thing? It has nothing whatsoever to do with the code that's compiled. You can compile a set of C files and link the resulting obj files into two exes: one for GUI and the other for the console -- and both will work flawlessly in their respective execution environment. They have to, because at compile time nobody knows (well, at least the compiler doesn't know) whether the code it's spitting out will finally be linked into a GUI or a console app. Or even, come to think of it, a DLL.

That's what you should do, instead of hiding behind stub.exe and dark mutterings about subtle bugs: produce two exes, one for GUI scripts and one for the console. Then define and set up a second file extension: .au3 remains with the GUI app, for backwards compatibility, while another extension (.a3c or whatever) connects scripts with the console exe. That's as clean and straightforward as it gets.

(Not that a second extension would be necessary: AutoIt could be trivially extended so that the script simple announces via a #directive that it's a console app: the GUI exe would just restart the console exe.)

But if you can't stomach that, if that visceral dislike of two exes I mentioned earlier is just too strong... well, fair enough. I am not a developer and I can resort to stub.exe (or "hack" the AutoIt3 exe and define that second extension locally).

I fully realise that I am with the minority in my yearnings for full console support. However, I still think your attitude is wrong and, as I wrote in my original post, unnecessarily constraining: AU3 has great potential -- and not only for scripting GUI apps.

[Valik]

As long as we're looking at simple stdin-stdout type programs, it does almost all... the two things it doesn't do are minor niggles. "Real" console stuff is a different story though.

And what would those be?

[Valik]

Not true. What you disparagingly call "hacking the binary" is bit for bit equivalent to linking the obj files against the console subsystem.
Valik, will you ever get it into this head of yours that this flag is purely a linker thing? It has nothing whatsoever to do with the code that's compiled. You can compile a set of C files and link the resulting obj files into two exes: one for GUI and the other for the console -- and both will work flawlessly in their respective execution environment. They have to, because at compile time nobody knows (well, at least the compiler doesn't know) whether the code it's spitting out will finally be linked into a GUI or a console app. Or even, come to think of it, a DLL.

I'll get it through my head when I confirm things and not when I listen to some random person on the internet I've never met. It just so happens I've grown tired of this subject so I did check. You are correct, I can compile the same application using both subsystems and produce (nearly) identical binaries. Naturally the subsystem flag is different and the only other differences are the timestamp and checksum which are inconsequential.

That's what you should do, instead of hiding behind stub.exe and dark mutterings about subtle bugs: produce two exes, one for GUI scripts and one for the console.

I'm by no means a defensive programmer but I also don't needlessly go out asking for trouble, either. "Hiding" as you call it is much safer than saying something is okay when I did not know it was. I never said changing the subsystem *would* cause problems, I merely said it *could*. This is a much safter attitude to take until I know for sure one way or the other. As it stands, I've had one person with experience tell me it was bad and I've had one person tell me it was safe. Since I have now confirmed for myself, I will admit that it's safe. I merely erred on the side of caution. Also, it wasn't until recently I discovered crtexe.c and could see how the startup code works. Even then, up until a couple days ago, I still hadn't thoroughly read the code to see how main/WinMain were invoked.

Then define and set up a second file extension: .au3 remains with the GUI app, for backwards compatibility, while another extension (.a3c or whatever) connects scripts with the console exe. That's as clean and straightforward as it gets.

(Not that a second extension would be necessary: AutoIt could be trivially extended so that the script simple announces via a #directive that it's a console app: the GUI exe would just restart the console exe.)

That's not really a good idea. First, since the GUI is already detached from the console that presumably started it, the console script would also be detached, instead getting its own isolated console. Your idea is only useful when starting the script from a non-console source in which case it's going to have it's own console anyway.

But if you can't stomach that, if that visceral dislike of two exes I mentioned earlier is just too strong... well, fair enough. I am not a developer and I can resort to stub.exe (or "hack" the AutoIt3 exe and define that second extension locally).

I fully realise that I am with the minority in my yearnings for full console support. However, I still think your attitude is wrong and, as I wrote in my original post, unnecessarily constraining: AU3 has great potential -- and not only for scripting GUI apps.

I've thought about ideas since your last post. And now that I've confirmed that changing the subsystem post-linking is safe, I can now think about other things as well. I would be willing to make the concession that if AutoItSC.bin goes, we can replace it with AutoItC.exe which would be a console version of AutoIt. Of course, I'd like a more clever solution but it's hard to fight the loader...

[thomasl]

I'll get it through my head when I confirm things and not when I listen to some random person on the internet I've never met.

I'm by no means a defensive programmer but I also don't needlessly go out asking for trouble, either. "Hiding" as you call it is much safer than saying something is okay when I did not know it was. I never said changing the subsystem *would* cause problems, I merely said it *could*. This is a much safter attitude to take until I know for sure one way or the other.

Full ack. I am not criticising this attitude as such; I am (or was) criticising that you continued to parade technical reasons where I could see no such thing. There may well be other reasons for not enabling a console mode... but I'd like to know and understand them.

Rational discussion and safe coding practices are different sides of the same thing.

I've thought about ideas since your last post. And now that I've confirmed that changing the subsystem post-linking is safe, I can now think about other things as well. I would be willing to make the concession that if AutoItSC.bin goes, we can replace it with AutoItC.exe which would be a console version of AutoIt. Of course, I'd like a more clever solution but it's hard to fight the loader...

Hm... So I help you to bin the bin stuff and in turn you help me to get a console exe? Sancho Pansa and Don Quixote against the windmills of exe flab...!

Okay, seriously. There have to be two exes, one GUI, the other console. There's simply no reasonable, safe way around that, given how the OS loader works. Then again, the exes would (or at least could) be identical (with the exception of the subsystem code and some housekeeping stuff).

What I would do if I had to solve that dilemma for myself is to resort to two tiny loaders that "somehow" load the bulk of the code. The obvious solution would be a separate DLL though there are certainly other possibilities. In the end, it's all just binary code and that stuff can be manipulated in all sorts of ways.

(Come to think of it, the main DLL (if it were indeed a DLL) could perhaps done in such a way that it would also work in the AutoItX/DLL environment. Now that would be a neat solution.)

Then there's the compiled environment. I know nothing about the internals of compiled scripts so I am on rather thin ice here, but I think it should be possible to design these tiny loaders in such a way that they can do one of two things: after loading the DLL/big chunk of code with the bulk of AU3 and initialising everything they either execute the given script file or load, from their own executable, the compiled stuff that's piggybacking.

This hypothetical DLL/bin could probably be coded in such a way (ie with wrappers and the like) that the resulting code works the same in all environments (GUI, console, DLL, COM, ...). There would be three (EDIT: or four, if you count the compiler as well) files, then: two small exes and one big DLL (or bin) file. The DLL/bin carries the bulk of AU3 and is the same whatever the environment. The two exes work as controlling loaders; the AU3 compiler would take the appropriate loader (GUI or console), attach the compiled script and that would be that. (A perfect solution would have the compiler pack the DLL/bin itself into the compiled exe as well and somehow make it loadable on the remote machine.)

As I don't know the first thing about the AU3 source code all this is pretty much off the top of my head. Still, perhaps this idea doesn't sound too weird.

Edited by thomasl, 19 September 2006 - 11:37 AM.

[Uten]

@thomasl. Sounds like you should be able to do some proof of concept work on this issue? A limited version of AutoIt is available as source . I'm confident anyone skilled enough to make significant changes to the code will be welcomed by the developers.

Looking forward to get the console only opportunity although I personally would like to see the time spent in other areas of AutoIt.

Happy coding to all of you

[thomasl]

@thomasl. Sounds like you should be able to do some proof of concept work on this issue? A limited version of AutoIt is available as source . I'm confident anyone skilled enough to make significant changes to the code will be welcomed by the developers.

If you have followed the discussion so far you will have realised that the question of GUI vs console has nothing whatsoever to do with the AU3 source code. There's nothing I or anyone else would need to change there* to make AU3 schmooze up to the console.

As to the other idea -- of getting rid of AutoItSC.bin etc. --, this is deeply connected with the AU3 compiler and the way this beast produces its output. I have neither seen any sources for that (I am big fan of UTSL) nor any in-depth idea how it works.

At any rate, this second thing cuts so deeply through (or into) AU3 internals that it's probably not feasible for a newcomer to supply even something as basic as a proof of concept. In fact, I think if we ever see something coming out of this discussion, it might well take the better part of a year to actually get it done and ready for prime time. It'll crawl and not pop out.

* Contradicting myself, there might be. ConsoleWrite() does work as expected, but ConsoleRead() doesn't seem to connect to the console stdin as it is right now (or if it does, it's reading input in a rather strange manner). This would need a (probably small) change... unfortunately, this part is not in the source code I have here (possibly because ConsoleRead() was added after the public sources were frozen).

[Valik]

Until I (or another developer) writes their own LoadLibrary() that can map DLLs without requiring it to be stored in DLL files on disk, that probably won't happen. IMO, DLLs need to be 100% transparent to users when they use a compiled script. Having DLLs to carry along make things much harder both for us and for users. I can also imagine a new DLL-Hell where somebody says "But I'm really using version 3.4.5.6" which is true but the DLL version they use is really "3.4.2.1".

What is it that ConsoleRead() does that you consider strange?

[thomasl]

Until I (or another developer) writes their own LoadLibrary() that can map DLLs without requiring it to be stored in DLL files on disk, that probably won't happen.

Given the potential of this, I think further investigating that area could well bear fruit. As I wrote in another post, that issue may be more of a long-term idea.

IMO, DLLs need to be 100% transparent to users when they use a compiled script.

Absolutely. What exactly that requires in this context is far from clear, however. Perhaps Uten was right after all and I should try to do a sort of proof of concept. Not today though.

What is it that ConsoleRead() does that you consider strange?

I didn't quite write that. But I won't argue a few words.

ConsoleWrite() writes to the console all right, but ConsoleRead() returns without reading a single character, so I assume it connects to Santa Claus or God knows who but not to the actual console. Or if it does, the way it reads the console is non-standard... without seeing the sources I can't say. Try this with a suitably linked exe:

ConsoleWrite(ConsoleRead(20))
ConsoleWrite("@error: "&@error)

It's NBD right now. A simple DllCall() will remedy that.

[Valik]

Do you mean when its called from a CUI version of AutoIt? If so, then that's not supported. Since we don't provide a console version of AutoIt, AutoIt doesn't support FILE_TYPE_CHAR handles. In other words, AutoIt is only expecting emulated stdin streams (Files, pipes) and not a real console (This may apply to Stub.exe, I haven't actually tested it since I made changes to ConsoleRead() awhile back). It should be a simple change so I'll go ahead and add support for that.

[thomasl]

Do you mean when its called from a CUI version of AutoIt?

Why do you think I wrote "try this with a suitably linked exe"? So in a word, yes.

It should be a simple change so I'll go ahead and add support for that.

That's what I was thinking as well. Thanks.

[Valik]

I forgot rule #3287 of being a programmer: Never say something is simple until after you've done it... twice just to make sure the first time wasn't a fluke.

This is far from simple. The one thing I didn't consider was how input is put into the stream. When the stream is a file or a pipe, something else has a write end and can fill it at their leisure. When the stream is attached to a console, nothing can write to it until we let it by calling a read function on the stream. That causes AutoIt to block, of course. It also means peeking becomes impossible because the only way to allow input to be put onto the stream is to block while it's input.

The choices are: Make ConsoleRead() block when it's attached to a real console. Go the much more complicated route and make a separate thread monitoring stdin all the time (When run under a real console) and then ConsoleRead() peeks/reads from a buffer instead of the stream directly.

Personally, I don't like either choice. The former makes ConsoleRead() behave inconsistently compared to when the stdin stream is a file or pipe. The second maintains consistency, which I think is important, but is a lot of work although perhaps not as much as I think. However, there's still the excuse that a console version of AutoIt is still unsupported so making big changes to support functionality in a configuration of AutoIt we don't support is a bit... odd.

At any rate, this is not a simple issue like I thought. I'll have to think about it some more. Sticking with DllCall() for now is your best bet.

[thomasl]

I forgot rule #3287 of being a programmer: Never say something is simple until after you've done it

He, he

Personally, I don't like either choice. The former makes ConsoleRead() behave inconsistently compared to when the stdin stream is a file or pipe. The second maintains consistency, which I think is important, but is a lot of work although perhaps not as much as I think.

I think in the long run the thread solution is preferable. All the more as you sort of have the bare bones in stub.exe.

However, there's still the excuse that a console version of AutoIt is still unsupported so making big changes to support functionality in a configuration of AutoIt we don't support is a bit... odd.

I agree. I am perfectly happy as things stand right now. I just wanted to point out that there's a small problem, before you do a supported version and then it pops up and someone ends up with a bloody nose.

[Richard Robertson]

I love these arguments. Very interesting reading material.

I've looked into a hybrid GUI/CUI dependant on how it starts. I don't know how to find the process that started my/your/our program. If anyone knows how, I'd love to know that even for outside this GUI/CUI conflict.

[Fossil Rock]

because at compile time nobody knows (well, at least the compiler doesn't know) whether the code it's spitting out will finally be linked into a GUI or a console app. Or even, come to think of it, a DLL.

I'm in way over my head here but I enjoy learning. While I have no immediate need of a CUI version of AutoIT, from the quote above, would procuding an AutoIT DLL be possible ? Something that could be used by other languages (C++, C#, VB... ) ?

I appreciate how civil this conversation has been and hope that my lack of knowledge won't trigger any "you're an idiot" type responses.