Jump to content
NicSol

IE11 "Save as" for file download

Recommended Posts

NicSol

I know this issue has been brought up a few times, but most of the solutions I've found in the forum involve mouse clicks and key sending that I'd like to avoid if possible. Basically, I am using the "_IE" UDFs to automate some very repetitive tasks on a website that runs scientific calculations. When the calculations are done, a CSV file is offered for download. The problem is, a box like this pops up:Capture.thumb.JPG.c3beaccd2ce048b6c1c88c and I have not been able to find a reliable way of selecting "Save as". I read that the UI Automate scripts can be used for this, but for the life of me, I cannot find the example scripts to which people keep referring. Can anyone provide me a link? Or does anyone have another reliable way of doing this?

Thanks in advance!

Just to save time on discussion:
- I can't use InetGet() because I have to log in to the site, and it's not a simple HTTP AUTH
- I've tried using cURL and wget with cookies, but the site uses an apparently complex CSRF scheme (it runs the Yii framework), and I'm having trouble getting cURL and wget to play nicely with it. I've also tried WinHTTP, but I seem to be running into the same problems. That's why I've gone back to trying to use IE directly.

Share this post


Link to post
Share on other sites
NicSol

It would greatly help your cause if you were to post the code you tried with each of the methods you noted.

Thanks, John. I was actually trying to keep the scope of the question narrow: specifically in IE11, how do you "Save as", and is there a way to use UI Auotmate? (I've seen a couple references to an example in the forum, but after hours of searching, I cannot find it).

Here's an example of my attempt with WinHttp, based on code I found elsewhere on the forum:

#include "WinHttp.au3"

Global Const $sUserName = "name"
Global Const $sPassword = "pass"

; Initialize and get session handle
$hOpen = _WinHttpOpen()
; Get connection handle
$hConnect = _WinHttpConnect($hOpen, "site.com", $INTERNET_DEFAULT_HTTPS_PORT)

; Fill login form:
$sRead = _WinHttpSimpleFormFill($hConnect, _
        "basic/web/index.php\?r=site%2Flogin", _
        "login-form", _
        "name:_csrf", "akQtTExDbUUJHmMrKgdYIw0TbDsYDiYdKQZdA2EQCy4FLBopOjYALA%3D%3D", _
        "name:LoginForm[username]", $sUserName, _
        "name:LoginForm[password]", $sPassword, _
        "name:LoginForm[rememberMe]", 1, _
        "name:login-button", "submit" _
        )

; You can check both @error and inspect $sRead to see if login was good
ConsoleWrite($sRead & @CRLF)
ConsoleWrite(@error & @CRLF)

; Close connection handle
_WinHttpCloseHandle($hConnect)

; Open new connection handle
$hConnect = _WinHttpConnect($hOpen, "mysite.com", $INTERNET_DEFAULT_HTTPS_PORT)

$vXLS = _WinHttpSimpleSSLRequest($hConnect, _
        Default, _
        "basic/web/index.php\?r=calculation%2Fdownload&id=5900", _
        Default, _
        Default, _
        Default, _
        Default, _
        Default) ; BINARY

; Close connection handle
_WinHttpCloseHandle($hConnect)
; Close session handle
_WinHttpCloseHandle($hOpen)

; Do whatever with this data, save to file or just print to console like I'm doing here
ConsoleWrite($vXLS & @CRLF)

The _csrf value was obtained by visiting the login page in a browser and finding the value in the hidden form field in the page  This takes me either to the site's home page or to an error 400 page (saying "Unable to verify your data submission.") . I can get into the wget and cURL alternatives I've tried, but the idea was essentially the same (i.e. load the page, get the csrf and cookies, and then make another request [sending the cookies back, along with the csrf token] to login). So far, I have not even been able to log in, let alone download a file with either of these methods. But this really isn't an AutoIT question at that point, it's a cURL and wget issue...

Regarding my attempts with IE, I don't think they'd add much to the discussion. Basically, I have been clicking at the coordinates of the button, but it's a terrible hack. The problem is, I am not able to find any way using the autoIT window info tool to get a handle to the control.

Share this post


Link to post
Share on other sites
mLipok

Is this file have always the same name ?
Or do you can check the file name before you download the file ?

 


Signature beginning:   Wondering who uses AutoIT and what it can be used for ?
* GHAPI UDF - modest begining - comunication with GitHub REST API *
ADO.au3 UDF     POP3.au3 UDF     XML.au3 UDF    How to use IE.au3  UDF with  AutoIt v3.3.14.x  for other useful stuff click the following button

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST API *

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF SMTP Mailer UDF *

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * 

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * 

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2018-03-17

Share this post


Link to post
Share on other sites
NicSol

Hi mLipok

No, the file name offered by the server depends on a number of things and is different each time. I check it before download based on the link offered by the webpage...

Share this post


Link to post
Share on other sites
trancexx

I can help you with WinHttp.au3, but that would be off topic. Create new thread if you want. _csrf isn't static value, it's generated per session, that's why your code doesn't work, I assume.


♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites
LarsJ

Share this post


Link to post
Share on other sites
PACaleala

You could ask Windows to help. Read this : 

http://helpdeskgeek.com/how-to/re-enable-always-ask-before-opening-this-type-of-file-check-box-in-windows/

 

Share this post


Link to post
Share on other sites
NicSol

I can help you with WinHttp.au3, but that would be off topic. Create new thread if you want. _csrf isn't static value, it's generated per session, that's why your code doesn't work, I assume.

Thank you for offering. I would be interested to know regardless, so if you'd be willing to work on it with me i would be highly appreciative. I'll open a new thread.

I think it's this UI Automation example, you're looking for.

Thanks, this indeed seems to be the thread I could not find (links in old posts were broken, and somehow I did not come across this as I combed through the pages). It seems to work perfectly for the "Save" function. I will try to better understand how it works and see if I can modify/extend it to work on the "Save as" option.

You could ask Windows to help. Read this : 

http://helpdeskgeek.com/how-to/re-enable-always-ask-before-opening-this-type-of-file-check-box-in-windows/

 

Thanks. I'd considered going that route at one point also, but it seems that it was made much more difficult in IE11, and then outright impossible. See this page, and the comments on it saying it no longer works (which was also my experience):

http://jwcooney.com/2014/03/31/remove-internet-explorer-open-or-save-popup/

Edited by NicSol

Share this post


Link to post
Share on other sites
NicSol

Thank you for offering. I would be interested to know regardless, so if you'd be willing to work on it with me i would be highly appreciative. I'll open a new thread.

On second thought, discussing how to login to a site with csrf tokens (which would have to be part of the discussion) seems to violate the forum rules...or am I misinterpreting?

Share this post


Link to post
Share on other sites
trancexx

You are misinterpreting, because you are overthinking. Your code is like that too.
Things are often simple.

Edited by trancexx

♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites
NicSol

Things are often simple.

Heheh, you inspired me to try again, and I think I finally got it working. I'm not 100% certain what my problem was before (it was not that the token was static -- I was just manually inserting it from loading the page in a browser right before running the script), but the solutions below seem to work at least for downloading a file requiring login (in my case it also must be SSL). I wouldn't say this is terribly original, but hopefully this will help someone out.

So far I have not been able to wrap my head around the code for IE, but I'm interested in figuring that out at some point, so I will post here if I figure it out.

Thanks to everyone for your help, and thanks to trancexx for the WinHTTP UDF.

----

For me, the key to getting this working was using the Live HTTP Headers plugin for Firefox (http://livehttpheaders.mozdev.org/). Go to the options menu and delete all the cookies from the site so you're starting with a clean slate. Then visit the login page and log in. Then very carefully examine what info is being sent, how it is encoded, etc. Using the two methods below (cURL command line and WinHTTP), you don't really need to worry about the cookies, but it's helpful to at least see what's going so you can see how far you're getting during trouble shooting. Note that many of the cookies set may be "HTTPOnly" cookies that are short-lived and will not show up in your browser's list of cookies. So this is really the only way you can see them.

Method 1 - cURL

  • Get the cURL command line exe (http://curl.haxx.se/download.html); there is also an autoIT interface to the cURL libraries you could use (https://www.autoitscript.com/forum/topic/137456-curl-udf-a-udf-for-transferring-data-with-url-syntax/) with the same strategy as below, but you'd obviously have to code it yourself. Put curl.exe on your PATH or in the same folder as your script.
  • The basic idea is to go to the login page, get the CSRF token, login, and then get the file you need
  • command line options: -c saves the cookies to a file. -b loads the cookies from a file. -o writes the page you visit to a file. -D writes the response headers to a file (for debugging). -d specifies the POST data for the login form
  • This is just to give you the concept. You will have to modify how you parse the CSRF token and the POST data for your particular case (based on the headers you find with the Live HTTP Headers plugin).
;get the csrf token by visiting the login page
RunWait("curl -o initrequest.txt -D headers1.txt -c cookies.txt https://site.com/login")
$fh = FileOpen("initrequest.txt", 0);
$s_page1 = FileRead($fh);
FileClose($fh);

;parse the token from the page
$aMatch = StringRegExp($s_page1, 'name="_csrf" value="([^"]+)"', 3)
$s_csrf = StringReplace($aMatch[0], '=', '%3D')
$s_csrf = StringReplace($s_csrf, '/', '%2F')

ConsoleWrite("CSRF = " & $s_csrf & @CRLF)

;build the post data string -- this can basically be copied from the Live HTTP Headers window
$s_post_data = "_csrf=" & $s_csrf & "&LoginForm%5Busername%5D=USERNAME&LoginForm%5Bpassword%5D=PASSWORD" _
               & "&LoginForm%5BrememberMe%5D=0&LoginForm%5BrememberMe%5D=1&login-button=";

;log in
RunWait('curl -o loginpage.txt -D headers2.txt -b cookies.txt -c cookies.txt -d "' & $s_post_data & '"  https://site.com/login')
;download
RunWait('curl -o download.txt -D headers3.txt -b cookies.txt -c cookies.txt  https://site.com/file')

Method 2 - WinHTTP

  • This method only requires WinHTTP.au3 (https://github.com/dragana-r/autoit-winhttp/releases)
  • Same idea as above, but it handles all the cookies internally and you don't need to create files to hold them. However, this may make it a little harder to debug (I'm sure there's a way to look at the cookies, but I don't know it). I found it helpful to get things working with cURL and then translate it to WinHTTP for my actual script.
  • Again this is just showing you the principle. You'll need to modify this for your particular case, and include error checking as necessary
#include "WinHttp.au3"

$s_uname = 'user'
$s_pword = 'pass'
$s_site = 'site.com'
$s_login_path = 'path/to/login'
$s_file_path = 'path/to/file'

; Initialize and get session handle
$hOpen = _WinHttpOpen()
; Get connection handle
$hConnect = _WinHttpConnect($hOpen, $s_site)

; Navigate to the page and get the CSRF token
$s_page1 = _WinHttpSimpleSSLRequest($hConnect, 'GET', $s_login_path)
; will depend on the site what the regexp should be...
$a_match = StringRegExp($s_page1, 'name="_csrf" value="([^"]+)"', 3)
;URL encode the token (this simple method works for my site, you may need to be more thorough)
$s_csrf = StringReplace($a_match[0], '=', '%3D')
$s_csrf = StringReplace($s_csrf, '/', '%2F')

; log In (your POST parameters will depend on your specific website)
$s_post_data = '_csrf=' & $s_csrf & '&LoginForm%5Busername%5D=' & $s_uname & '&LoginForm%5Bpassword%5D=' & $s_pword _
               & '&LoginForm%5BrememberMe%5D=0&LoginForm%5BrememberMe%5D=1&login-button=';
$s_page2 = _WinHttpSimpleSSLRequest($hConnect, 'POST', $s_login_path, Default, $s_post_data)


;get file data
$s_data = _WinHttpSimpleSSLRequest($hConnect, 'GET', $s_file_path)
;write the data to the file
FileWrite('Data.csv', $s_data)

; Close handles
_WinHttpCloseHandle($hConnect)
_WinHttpCloseHandle($hOpen)

 

Edited by NicSol
Forgot to include link to cURL

Share this post


Link to post
Share on other sites
trancexx

That's great. I'm glad you did it without the need to use half of the system's resources :D.

However it's even simpler than that. I bet this works too:

#include "WinHttp.au3"

$s_uname = 'user'
$s_pword = 'pass'
$s_site = 'https://site.com'
$s_login_path = 'path/to/login'
$s_file_path = 'path/to/file'

; Initialize and get session handle
$hOpen = _WinHttpOpen()
; Get connection handle
$hConnect = _WinHttpConnect($hOpen, $s_site)

; Fill the login form:
_WinHttpSimpleFormFill($hConnect, _
        $s_login_path, _
        "login-form", _
        "name:LoginForm[username]", $s_uname, _
        "name:LoginForm[password]", $s_pword)

$s_data = _WinHttpSimpleSSLRequest($hConnect, Default, $s_file_path)

; Close connection handle
_WinHttpCloseHandle($hConnect)
; Close session handle
_WinHttpCloseHandle($hOpen)

; Write the data to the file
FileWrite('Data.csv', $s_data)

 

 

Edited by trancexx
  • Like 1

♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites
NicSol

However it's even simpler than that. I bet this works too:

No kiddin' -- it also works, thanks!

How does it work without explicitly sending the csrf token? I guess it's sent in the cookie automatically? From reading about the website's content-management system, I was under the impression that you had to send the proper token that matches the csrf value in the cookie, but I guess not. Anyway, thanks again for the help!

Share this post


Link to post
Share on other sites
trancexx

No kiddin' -- it also works, thanks!

How does it work without explicitly sending the csrf token? I guess it's sent in the cookie automatically? From reading about the website's content-management system, I was under the impression that you had to send the proper token that matches the csrf value in the cookie, but I guess not. Anyway, thanks again for the help!

Token is send, no worries. _WinHttpSimpleFormFill does it internally for you.
_WinHttpSimpleFormFill is super advanced function.


♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites
NicSol

_WinHttpSimpleFormFill is super advanced function.

Ah yes I see that now looking through the code. Very handy. Many thanks again.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×