Reading large (70MB+) Text with _IEBodyReadText

[AlbertC]

First... I've used AutoIT for many  a moon, just stayed in the background.

I'm trying to automate retrieving some files from a web-based system that are only delivered to me via https, following a login exchange.

I've got the login exchange working (easy), and can use _IENavigate() to get to the file that I need to download.  And _IEReadBodyText() works for reasonable sized requests.

But some of the files are over 70MB, and I always end up with a series like:

Navigated to: https://eschool40.esp.k12.ar.us/eSchoolPLUS40/Reports/Users/springdlsms/4209fjones/4209000/schd_stu_course.txt
"C:\Program Files (x86)\AutoIt3\Include\IE.au3" (1600) : ==> The requested action with this object has failed.:
Return SetError($_IESTATUS_Success, 0, $oObject.document.body.innerText)
Return SetError($_IESTATUS_Success, 0, $oObject.document.body^ ERROR

The first line is returned by my program and is the URL to the file.  Again, this works if the file is smaller, up to around 20MB definitely works.

I've tried using InetGet() after the code that deals with logging in, and part of the time I can actually get the 74MB file... I'm not sure why that is, but it at least hints that the server is able to complete the delivery.,

I may have to try using WebDriver with Chrome for this... but I'm puzzled by the size/ timeout issue.  I have tried setting _IELoadWaitTimeout(6000000) ...  And I trried adding 5 minutes of Sleep() between _IENavigate($sURL) and doing the _IEReadBodyText() call...

The source snippet really is simply:

    _IENavigate($oIE,$sURL)
    $sText=_IEBodyReadText($oIE)

[mLipok]

Did you try winhttp ?

Btw.

Welcome to the forum. 

 

Edited July 17, 2018 by mLipok

[mLipok]

using winhttp I download file about 110MB without any problem.

 

[AlbertC]

I shifted the task to winhttp and it seems to be working well.

Had to do a bit of work to figure out getting the SSL request right.  The _WinHTTPSimpleFormFill is a great helper function.

I tried using the _WinHttpSimpleReadData() function, but it was receiving differing amounts of data processing the same large file.  (I have no control over the far-end server or its configuration, and a bunch of us beg the State to let us have FTP access to the user home directories...)  Using _WinHttpReadData() and the approach of continuing to grab chunks until nothing more remains seems to work fine.

expandcollapse popup

#include "WinHttp.au3"

; Chunk of code using WinHttp to login to a Sunguard/Pentamation eSchool Plus server and retrieve a file. There are a number 
;  of variables defined outside of this section that should be fairly self-explanatory...

Global $hOpen = _WinHttpOpen()
Global $hConnect = _WinHttpConnect($hOpen, "https://eschool40.esp.k12.ar.us/eSchoolPLUS40/Account/LogOn", $INTERNET_DEFAULT_HTTPS_PORT)

Global $sRead = _WinHttpSimpleFormFill($hConnect, _ ; connection handle
        "/eSchoolPLUS40/Account/LogOn", _ ; target page
        "index:0", _ ; form identifier
        "name:UserName", $sUserName, _ 
        "name:Password", $sPassword) 
        
; You can check the HTML returned in $sRead to determine if the login was processed successfully,  or take other action if needed.
if StringInStr($sRead, "<title>Login</title>", 1, 1, 1, 500) Then Exit (2) ; If you're returned to the login page, you've got bad creds.

; Even though a browser login may have another page to navigaate, once authenticated you can downlod eSchool+ reports immediately.
Global $hRequest = _WinHttpSimpleSendSSLRequest($hConnect, "GET", $sUserHome & $sFileName)
ConsoleWrite("Requesting " & $sUserHome & $sFileName & @CRLF)

; See if there is data to read
Global $sChunk, $sData
If _WinHttpQueryDataAvailable($hRequest) Then
    While 1
        $sChunk = _WinHttpReadData($hRequest) ; ReadData gets 8K chunks by default
        If @error Then ExitLoop ; Exit the loop if ReadData returns an error (there's no more data...)
        $sData &= $sChunk
    WEnd
Else
    ConsoleWrite("Error - Site is experiencing problems." & @CRLF)
    _WinHttpCloseHandle($hRequest)
    _WinHttpCloseHandle($hConnect)
    _WinHttpCloseHandle($hOpen)
    Exit (2)
EndIf

; Primitive version of file handling next...  You should really do something better based on your needs.
; I actually check the $sData variable for 404 errors, and compare size to the existing old file before 
; keeping the new one.
FileMove($sFileName,$sFileName & ".old",$FC_OVERWRITE)
FileWrite($sFileName,$sData)

If this code fragment helps someone else, or you've got a better thought, please share it!

Edited July 18, 2018 by AlbertC