GZM

Machine Text Parser

18 posts in this topic

Hi everybody! Very happy to finally join you all.

I´m starting to work in a project as a hobbie. Well, I've made some scripts a while ago, but let´s say I'm a complete newbie, both to AutoIt and programming itself, so please don't be that hard on me. Just feel free to correct me if I make any mistake.

I want to translate a program's verbose output (console) into meaningful information on a GUI (Progress bar, ratio, estimated time to finish, etc). Eventually, I would want my prog to become a somewhat universal parser for other command-line tools. The program is precomp. It´s output is pretty simple (but in verbose mode is a nightmare). Should I start reading the simple output or must consider the worst case to begin with?

Sadly, I have basically no clue about where to start... Could you please give some ideas on the script structure?

Thanks in advance!

Share this post


Link to post
Share on other sites



@GZM You can follow Best Coding Practices to start with... Use #Regions to fold code... Avoid using Magic Numbers etc etc...

I also made a UDF called "Process UDF" (Link in my signature) which contains functions to get the command line output from a program :) (Though, its  a bit outdated :P)

 

Good luck ;)


AutoIt.4.Life Clubrooms - Life is like a Donut (secret key)

Spoiler

My contributions to the AutoIt Community

Some messages & Apologizes:

If I hurt you, Please accept my apologies, I never (regardless of the situation) mean to hurt anybody!!!

Also, I am very busy with my project so I will appear in the last row of the online list, if you want to contact me: Email@TheDcoder.xyz

Or you can have a nice chat with me in freenode, I use the same nick on freenode too!

3fHNZJ.gif

PLEASE JOIN ##AutoIt AND HELP THE IRC AUTOIT COMMUNITY!

Share this post


Link to post
Share on other sites
1 minute ago, TheDcoder said:

@GZM You can follow Best Coding Practices to start with... Use #Regions to fold code... Avoid using Magic Numbers etc etc...

I also made a UDF called "Process UDF" (Link in my signature) which contains functions to get the command line output from a program :) (Though, its  a bit outdated :P)

 

Good luck ;)

Yeah, this is useful, indeed. Thank you for the quick answer! :)

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

Start by looking at the following functions in the help file and trying the examples.

Run()
STDOutRead()

You can click them to go to online help, but better to use help file locally, so you can directly open the example scripts.

Edited by JohnOne

AutoIt Absolute Beginners    Require a serial    Pause Script    Video Tutorials by Morthawt   ipify 

Monkey's are, like, natures humans.

Share this post


Link to post
Share on other sites

@GZM,

sorry to be a party crusher, but would you mind ease our concern about that console program of yours, "precomp". it seems to be game-related, and this is handled according to these directives.

if it is not the "precomp.exe" as described here, you can get a better assistance if you post an example of the output.

Share this post


Link to post
Share on other sites

@JohnOne:Thank you. Seems like what I need.

@orbs: Yes, I understand your concern. Precomp is a tool for 'un-doing' some weak compression techniques, (like zlib's deflate, or GIG's LZW) to finally improve the overall ratio. It is a serious tool looking for state-of-the-art results on data compression.

But, given its functionality, it has been widely used on the game "repacking" scene (aka Warez). Annoying, I know. Well, I assure you, this is not what I´m doing. Actually, I dont even like games. It's just I really like this little piece of software. 

Official GitHub repo

Threads on encode.ru

Normal output:

Precomp v0.4.5 - DEVELOPMENT version - USE AT YOUR OWN RISK!
Free for non-commercial use - Copyright 2006-2016 by Christian Schneider

Input file: 01 - Elemental electr¾nica.pdf
Output file: 01 - Elemental electr¾nica.pcf

Using PackJPG for JPG recompression, PackMP3 for MP3 recompression.
--> packJPG library v2.5j (01/15/2014) by Matthias Stirner / Se <--
--> packMP3 library v1.0f (11/21/2014) by Matthias Stirner <--
More about PackJPG and PackMP3 here: http://www.matthiasstirner.com

100.00% - New size: 26157203 instead of 31064028

Done.
Time: 3 minute(s), 33 second(s)

Recompressed streams: 144/244
PDF streams: 98/149
JPG streams: 46/46
zLib streams (intense mode): 0/49

You can speed up Precomp for THIS FILE with these parameters:
-zl11,67,85,97,98,99

Verbose output:

 

LOG.txt

Share this post


Link to post
Share on other sites

@orbs: BTW, I'm just as pissed of as anybody else looking at how science is turned into toys used by irresponsible grown-ups children...

Another great tool went into Warez scene is srep. The same apply to Freearc, both by Bulat Ziganshin. Or Zpaq, the very strongest compressor known by this time. 

1 person likes this

Share this post


Link to post
Share on other sites

Ok... Tried "Process UDF", but it only reads the output once the command is done. I need realtime feedback.

Is there any way to include a GUI artifact inside the "ConsoleRead()" loop? I made a minor modification on the example and partially worked. At least ran an MsgBox each time the last is closed. Is it possible to have a single GUI updating itself?

I also tried Run() with option $STDOUT_CHILD and STDOutRead() but this also reads everything at the end.

Share this post


Link to post
Share on other sites
33 minutes ago, JohnOne said:

Show code.

This is a copy-paste from StdoutRead.au3 (offline help):

; Wait until the process has closed using the PID returned by Run.
    ProcessWaitClose($iPID)

    ; Read the Stdout stream of the PID returned by Run. This can also be done in a while loop. Look at the example for StderrRead.
    Local $sOutput = StdoutRead($iPID)

This is from example.au3 (TheDcoder's Process UDF):

MsgBox($MB_ICONINFORMATION, "Demo", 'Demo for _Process_RunCommand in RUNWAIT mode, This is the function line: _Process_RunCommand($PROCESS_RUNWAIT, $PROCESS_COMMAND & "ping 127.0.0.1")' & @CRLF & @CRLF & "(This will take some time)")
$sOutput = _Process_RunCommand($PROCESS_RUNWAIT, $PROCESS_COMMAND & "ping 127.0.0.1") ; Record the output
$iExitCode = @extended ; Note the exit code
MsgBox($MB_ICONINFORMATION, "Demo", "This is the result: " & @CRLF & @CRLF & $sOutput & @CRLF & @CRLF & "The exit code was: " & $iExitCode)

The only case in which the output is shown real-time is using _Process_DebugRunCommand () as below:

MsgBox($MB_ICONINFORMATION, "Demo", 'Demo for _Process_RunCommand in RUN mode and Demo for _Process_DebugRunCommand, This is the function line: _Process_RunCommand($PROCESS_RUN, $PROCESS_COMMAND & "tracert google.com")' & @CRLF & @CRLF & "This will be a fun ride :D")
$hProcessHandle = _Process_RunCommand($PROCESS_RUN, $PROCESS_COMMAND & "tracert google.com") ; Capture the Process Handle
$iPID = @extended ; Note the PID
_Process_DebugRunCommand($hProcessHandle, $iPID) ; Display the results in real-time

The first two scripts waits until the program is done to show the output. The last one open up a pseudo-cmd window, just like if I wite "cmd /k my_program"...

Share this post


Link to post
Share on other sites
28 minutes ago, JohnOne said:

Looks like someone removed an important example from the help file.

pseudo 

$pid = Run()
    While ProcessExists($pid)
        StdOutRead($pid) ; each line here
    WEnd

 

Well, I try to keep up... Not easy yet. So thank you for being patient.

Could you post a more specific example?

Thanks in advance!

Share this post


Link to post
Share on other sites

Gzm,

Post your code so we can correct/build on it.

Kylomas


Forum Rules         Procedure for posting code

"I like pigs.  Dogs look up to us.  Cats look down on us.  Pigs treat us as equals."

- Sir Winston Churchill

Share this post


Link to post
Share on other sites
1 minute ago, kylomas said:

Gzm,

Post your code so we can correct/build on it.

Kylomas

Let's see another example of what I'm trying to do. This is from StderrRead.au3, just slightly modified to run a custom command instead of the original:

#include <AutoItConstants.au3>
#include <MsgBoxConstants.au3>

Example()

Func Example()
    Local $iPID = Run(@ComSpec & " /k Bat.bat", "", @SW_HIDE, $STDERR_CHILD + $STDOUT_CHILD)
    Local $sOutput = ""
    While 1
        $sOutput = StdoutRead($iPID)
        If @error Then ; Exit the loop if the process closes or StdoutRead returns an error.
            ExitLoop
        EndIf
        MsgBox($MB_SYSTEMMODAL, "Stdout Read:", $sOutput)
    WEnd

    While 1
        $sOutput = StderrRead($iPID)
        If @error Then ; Exit the loop if the process closes or StderrRead returns an error.
            ExitLoop
        EndIf
        MsgBox($MB_SYSTEMMODAL, "Stderr Read:", $sOutput)
    WEnd
EndFunc   ;==>Example

What I get from this is an empty MsgBox, another one with the first line on the batch and then an infinite loop of empty MsgBoxes... 

Sin título.png

Share this post


Link to post
Share on other sites

Now I replaced

@ComSpec & " /k Bat.bat"

with

fazip lz4:hc TEST TEST.lz4

 

Again empty msgboxes with no text

Share this post


Link to post
Share on other sites

Gmz,

You are completely missing the point.  Unfortunately i am not at a pc and dont have time right now to get into it.  However, if you want to process the outout stream as it is written then put you code within the loop that gets it.

Note, some things do not produce console output.

Kylomas


Forum Rules         Procedure for posting code

"I like pigs.  Dogs look up to us.  Cats look down on us.  Pigs treat us as equals."

- Sir Winston Churchill

Share this post


Link to post
Share on other sites

#17 ·  Posted (edited)

#include <AutoItConstants.au3>
#include <MsgBoxConstants.au3>

Example()

Func Example()
    Local $iPID = Run(@ComSpec & ' /C DIR ', "", @SW_HIDE, $STDERR_CHILD + $STDOUT_CHILD)
    Local $sOutput = ""
    While 1
        $sOutput = StdoutRead($iPID)
        If @error Then ; Exit the loop if the process closes or StdoutRead returns an error.
            ExitLoop
        EndIf
        ConsoleWriteLine($sOutput)
    WEnd
EndFunc   ;==>Example

Func ConsoleWriteLine($str)
    If $str <> "" Then
        ConsoleWrite($str & @LF)
    EndIf
EndFunc   ;==>ConsoleWriteLine

 

Edited by JohnOne
1 person likes this

AutoIt Absolute Beginners    Require a serial    Pause Script    Video Tutorials by Morthawt   ipify 

Monkey's are, like, natures humans.

Share this post


Link to post
Share on other sites

@all: Thank you for the feedback.

Well, this is a starting. By changing StdoutRead to StderrRead in @JohnOne script, I've managed to read the console output... The only drawback, it appears every ~16 seconds. Let´s see if it can be tuned.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now