StdoutRead not retrieving what i need

[JackDinn]

@water

yea i know it does, ,, i use that code in my speed tester app for one of the little "diagnostic "side apps" , i also use something very similar to calculate the MTU , so im pretty well versed in using this stuff :-/

Also you said earlier that you were unaware that there was a limit to the upload speed when using AI FTP upload commands , its a really odd one as iv tested all this some 4 or 5 months ago on a local server and its fine, i can measure up to the 100Mb/s my lan can handle , but i cant argue with all the people who are contacting me atm who are saying that it just dont go any higher than a MAX of 5Mb/s over there fiber WAN ?? dont ask me :-/ but when i get them to try it with a code i did that uses either the microsoft FTP command line commands or this ncFTP they all say it works fine and does pretty much max out there ftp upload speeds . So all just really wearied.

Edited May 23, 2012 by JackDinn

[JackDinn]

I just tested the help example for the StdoutRead function and it appears that certain outputted data is "groups". I just confused myself.

#include <Constants.au3>

$net = Run(@ComSpec & " /c dir c:temp", @SystemDir, @SW_HIDE, $STDERR_CHILD + $STDOUT_CHILD)

While 1
    $line = StdoutRead($net)
    If @error Then ExitLoop
    If $line <> "" Then ConsoleWrite("STDOUT read:" & $line & " -- " & @Sec & "." & @MSEC & @CRLF)
Wend

Exit

I then changed the directory to c:windows and there were many more "groups" of outputted data.

Hmm, so what does this all mean then, ? are we not going to be able to get the data in "real time" ? even though it is updated on the console window in real time ?

[water]

I get the impression that it is a mixture of size and time.

If the StdOut buffer is full (Dir example) or x milliseconds have passed (Ping example) the buffer is written to StdOut.

Just my 0.02 $

According to C/C++ docu: "stdin, stdout, stderr are buffered I/O streams."

Edited May 23, 2012 by water

[JackDinn]

well unless someone can figure a way to get the stdout for this, i was wondering what about reading the actual text that is put onto the console window ? is this possible ? is this possible when the console window is hidden ?

This was why i mentioned the "screen_Scrape" , i was kinda hoping that this is what it does.

I suspect that 7-Zip progress output has exactly the same problem !!

Edited May 23, 2012 by JackDinn

[water]

I just wonder why the application writes the text immediately to the console but AutoIt only gets it when the transfer has finished.

How long does your FTP transfer run?

[JackDinn]

atm im testing with a 100MB file, so as im testing over a Wlan to a local filezilla server it lasts approx a minute or so.

But yea im also wondering the exact same thing ^^

Edited May 23, 2012 by JackDinn

[water]

Just to be sure we all are talking about the same code, can you please post the code you are testing with?

[JackDinn]

$tcrtout = ""
$iPID = Run('ncftpput.exe -v -p password -u greg -C newlap "C:Program FilesJDASTUPtestfile.bin" /test/test.upt', @TempDir, @SW_SHOW, 0x2)
While 1
    GUIGetMsg()
    $tcrtout &= StdoutRead($iPID)
    If @error Then ExitLoop
    If $tcrtout <> "" Then ;$tcrcatch Then
        ConsoleWrite($tcrtout & @LF)
    EndIf
WEnd
Exit

Edited May 23, 2012 by JackDinn

[water]

GUIGetMsg is not needed.

$tcrtout = ""
$iPID = Run('ncftpput.exe -v -p password -u greg -C newlap "C:Program FilesJDASTUPtestfile.bin" /test/test.upt', @TempDir, @SW_SHOW, 0x2)
While 1
    $tcrtout = StdoutRead($iPID)
    If @error Then ExitLoop
    If $tcrtout <> "" Then ConsoleWrite($tcrtout & @LF)
WEnd
Exit

[JackDinn]

aye well i put it in to stop the processor getting maxed out by the loop,

Its just handled better than a sleep(10) or whatever ^^

Edited May 23, 2012 by JackDinn

[water]

Sleep(10) is the better choice:

$tcrtout = ""
$iPID = Run('ncftpput.exe -v -p password -u greg -C newlap "C:\Program Files\JDAST\UPtestfile.bin" /test/test.upt', @TempDir, @SW_SHOW, 0x2)
While 1
    $tcrtout = StdoutRead($iPID)
    If @error Then ExitLoop
    If $tcrtout <> "" Then ConsoleWrite($tcrtout & @LF)
    Sleep(10)
WEnd
Exit

[JackDinn]

No problem

im reading more about im really thinking this is a solution to a very similar problem with a console command for 7-Zip and its progress output. Although I'm really stretching the guess work here , lol

Iv got the code for it in my Scite and i firstly need to figure out where to get _WinAPI_MakeDWord() as it dont seem to be included in the <WinAPI.au3> or <WinAPIEx> , where is this func ?

Edited May 23, 2012 by JackDinn

[lowray1975]

It looks like _WinAPI_MakeDWord() was removed several versions ago.

[JackDinn]

yeap, im sure it was , and im also pretty sure its nothing special or complex. maybe something like

Returns a Dword value from two int values, so its a unsigned 32 bit int created from 2 16 bit ones , maybe ?

[JackDinn]

well im having another go at making a second connection to the remote server with standard AI _FTP() commands and getting the remote file size every half a sec. I can calculate the throughput of the ncFTP this way but last time i tried this the requests for the filesize were causing a slowdown of the upload but i was using the microsoft command line FTP that time, maybe it'll work better with ncFTP. dunno :-/

Its all a bit of a fudge , especially when that info is right there already on the output , just cant darn well get at it

[lowray1975]

Mind you, I have no idea was it was removed in v3.3.1.0:

Func _WinAPI_MakeDWord($HiWord, $LoWord)
    Return BitOR($LoWord * 0x10000, BitAND($HiWord, 0xFFFF))
EndFunc

[JackDinn]

well i must admit iv not a clue how this works but with a little gigging about iv got it to read the last 4 characters , i assume he was after just a % and so only needed 4 chr's , i can move these 4 characters to the left but im really lost with most of this coding and cant see how to expand the number of characters it returns

if you'd like a look you can take what iv done so far, might save a little time :-/, excuse the mess lol

expandcollapse popup

#Region Includes
#Region ;************ Includes ************
#Include <Debug.au3>
#include <WinAPI.au3>
#EndRegion ;************ Includes ************
#EndRegion Includes
#Region Options/Singleton
Opt("MustDeclareVars", True)
#EndRegion Options/Singleton
_DebugSetup("debug",true)
#Region Global Variables
Global Const $STD_INPUT_HANDLE = -10
Global Const $STD_OUTPUT_HANDLE = -11
Global Const $STD_ERROR_HANDLE = -12
;~ Global Const $INVALID_HANDLE_VALUE = -1
Global Const $_CONSOLE_SCREEN_BUFFER_INFO = "short dwSizeX; short dwSizeY;" & _
"short dwCursorPositionX; short dwCursorPositionY; short wAttributes;" & _
"short Left; short Top; short Right; short Bottom; short dwMaximumWindowSizeX; short dwMaximumWindowSizeY"
Global Const $_COORD = "short X; short Y"
Global Const $_CHAR_INFO = "wchar UnicodeChar; short Attributes"
Global Const $_SMALL_RECT = "short Left; short Top; short Right; short Bottom"
#EndRegion Global Variables
#Region Main body of code
Global $g_nExitCode = Main()
Exit $g_nExitCode
#EndRegion Main body of code
#Region Main()
; ===================================================================
; Main()
;
; The main program body.
; Parameters:
; None.
; Returns:
; None.
; ===================================================================
Func Main()
; Sanity check to ensure the script is compiled.
;~  If Not @Compiled Then
;~   MsgBox(4096 + 16, "Error", "This script must be compiled.")
;~   Return -1
;~  EndIf
; The 7-Zip command line.
; The show flag, change to @SW_HIDE to prove it works while hidden.
Local Const $nShow = @SW_SHOW
; Run the 7-Zip command line.
Local $pid = Run('ncftpput.exe -v -p password -u greg -C newlap "C:Program FilesJDASTUPtestfile.bin" /test/test.upt', @TempDir, @SW_SHOW)
;~ MsgBox(0,"","")
; Brief pause to let 7-Zip get started.
Sleep(1000)
; Create a progress meter.
;~  ProgressOn("7-Zip percent", "Backing up...", "0%")
; Open the 7-Zip percent reader.
  _DebugReportVar("pid",$pid)
Local $hPercent = Open7ZipPercent($pid)
_DebugReportVar("$hPercent",$hPercent)
;~ SplashTextOn("ww","")
;~ ControlSetText("ww","","Static1",$hPercent)
sleep(1000)
; Loop on the process existence.
While ProcessExists($pid)
  ; Read the percent and update the progress bar.
  Local $nPercent = Read7ZipPercent($hPercent)
  _DebugReportVar("Capture =",$nPercent)
;~   ControlSetText("ww","","Static1",$nPercent)
;~   If $nPercent >= 0 Then ProgressSet($nPercent, $nPercent & "%")
  Sleep(50)
WEnd
; We're finished.
;~  ProgressSet(100, "100%", "Done")
; Give time to see we're finished.
Sleep(1000)
; Cleanup.
Close7ZipPercent($hPercent)
FileDelete(@ScriptDir & "Backup.7z")
EndFunc ; Main()
#EndRegion Main()
#Region Primary Functions
Func Open7ZipPercent($pid)
; Try to attach to the console of the PID.
If _AttachConsole($pid) Then
  ; The user should treat this as an opaque handle, but internally it contains a handle
  ; and some structures.
  Local $vHandle[4]
  $vHandle[0] = _GetStdHandle($STD_OUTPUT_HANDLE) ; STDOUT Handle
  $vHandle[1] = DllStructCreate($_CONSOLE_SCREEN_BUFFER_INFO) ; Screen Buffer structure
  $vHandle[2] = DllStructCreate("dword[4]") ; Data buffer
  $vHandle[3] = DllStructCreate($_SMALL_RECT) ; SMALL_RECT structure
  ; Return the handle on success.
  Return $vHandle
EndIf
; Return 0 on failure.
Return 0
EndFunc ; Open7ZipPercent()
Func Close7ZipPercent(ByRef $vHandle)
; Basic sanity check to validate the handle.
If UBound($vHandle) <> 4 Then Return False
; Detach the console.
_FreeConsole()
; Destroy the handle.
$vHandle = 0
; Return success.
Return True
EndFunc ; Close7ZipPercent()
Func _WinAPI_MakeDWord($HiWord, $LoWord)
    Return BitOR($LoWord * 0x10000, BitAND($HiWord, 0xFFFF))
EndFunc
Func Read7ZipPercent(ByRef $vHandle)
; Basic sanity check to validate the handle.
If UBound($vHandle) = 4 Then
  ; Create some variables for convenience.
  Local Const $hStdOut = $vHandle[0]
  Local Const $pConsoleScreenBufferInfo = $vHandle[1]
  Local Const $pBuffer = $vHandle[2]
  Local Const $pSmallRect = $vHandle[3]
  ; Try to get the screen buffer information.
  If _GetConsoleScreenBufferInfo($hStdOut, $pConsoleScreenBufferInfo) Then
   ; Set up the coordinate structures.
   Local $coordBufferSize = _WinAPI_MakeDWord(4, 1)
   Local $coordBufferCoord = _WinAPI_MakeDWord(0, 0)
   ; Load the SMALL_RECT with the projected text position.
   DllStructSetData($pSmallRect, "Left", DllStructGetData($pConsoleScreenBufferInfo, "dwCursorPositionX") - 13)
   DllStructSetData($pSmallRect, "Top", DllStructGetData($pConsoleScreenBufferInfo, "dwCursorPositionY"))
   DllStructSetData($pSmallRect, "Right", DllStructGetData($pConsoleScreenBufferInfo, "dwCursorPositionX"))
   DllStructSetData($pSmallRect, "Bottom", DllStructGetData($pConsoleScreenBufferInfo, "dwCursorPositionY"))
   ; Read the console output.
   If _ReadConsoleOutput($hStdOut, $pBuffer, $coordBufferSize, $coordBufferCoord, $pSmallRect) Then
    ; This variable holds the output string.
    Local $sPercent = ""
    ; We iterate over 3 characters because that's all we read.
    For $i = 0 To 3
     ; We offset the buffer each iteration by 4 bytes because that is the size of the CHAR_INFO
     ; structure.  We do this so we can read each individual character.
     Local $pCharInfo = DllStructCreate($_CHAR_INFO, DllStructGetPtr($pBuffer) + ($i * 4))
     ; Append the character.
     $sPercent &= DllStructGetData($pCharInfo, "UnicodeChar")
    Next
    ; Ensure we read a valid percentage.  If so return the cast to a number.
    Return $sPercent
   EndIf
  EndIf
EndIf
; On failure we return -1 which is obviously not a valid percentage.
Return -1
EndFunc ; Read7ZipPercent()
#EndRegion Primary Functions
#Region Helper Functions
Func _GetStdHandle($nHandle)
Local $aRet = DllCall("kernel32.dll", "hwnd", "GetStdHandle", "dword", $nHandle)
If @error Then Return SetError(@error, @extended, $INVALID_HANDLE_VALUE)
Return $aRet[0]
EndFunc ; _GetStdHandle()
Func _AttachConsole($nPid)
Local $aRet = DllCall("kernel32.dll", "int", "AttachConsole", "dword", $nPid)
If @error Then Return SetError(@error, @extended, False)
Return $aRet[0]
EndFunc ; _AttachConsole()
Func _FreeConsole()
Local $aRet = DllCall("kernel32.dll", "int", "FreeConsole")
If @error Then Return SetError(@error, @extended, False)
Return $aRet[0]
EndFunc ; _FreeConsole()
Func _GetConsoleScreenBufferInfo($hConsoleOutput, $pConsoleScreenBufferInfo)
Local $aRet = DllCall("kernel32.dll", "int", "GetConsoleScreenBufferInfo", "hwnd", $hConsoleOutput, _
  "ptr", _SafeGetPtr($pConsoleScreenBufferInfo))
If @error Then Return SetError(@error, @extended, False)
Return $aRet[0]
EndFunc ; _GetConsoleScreenBufferInfo()
Func _ReadConsoleOutput($hConsoleOutput, $pBuffer, $coordBufferSize, $coordBufferCoord, $pSmallRect)
; We lie about the types for the COORD structures.  Since they are the size of an int we expect a packed
; int.  Otherwise we may crash or just pass garbage.
Local $aRet = DllCall("kernel32.dll", "int", "ReadConsoleOutputW", "ptr", $hConsoleOutput, _
  "ptr", _SafeGetPtr($pBuffer), "int", $coordBufferSize, "int", $coordBufferCoord, _
  "ptr", _SafeGetPtr($pSmallRect))
If @error Then SetError(@error, @extended, False)
Return $aRet[0]
EndFunc ; _ReadConsoleOutput()
Func _SafeGetPtr(Const ByRef $ptr)
Local $_ptr = DllStructGetPtr($ptr)
If @error Then $_ptr = $ptr
Return $_ptr
EndFunc ; _SafeGetPtr()
#EndRegion Helper Functions

Edited May 23, 2012 by JackDinn

[kylomas]

JD,

Regarding post #'s 28 and 29, you are concatenating the output whereas water is not. This would account for the "appearance" of grouping.

kylomas

[JackDinn]

yea we know, but either way it would have shown if we were getting any info that were after at all, would have cleaned it up after if we had got anything from it.

the "grouping" refereed to is the fact that there is no stdout at all until after the upload has finished at which point the whole lot of data was coming through.

cheers.

[kylomas]

JD,

Sorry, thought that I might have stumbled on something obvious...is there any way to know whether or not 7z is bufferring it's output?

kylomas