Jump to content
Sign in to follow this  
mattw112

Downloading from Secure Website

Recommended Posts

mattw112

There's a website I have access to with credentials. On that website there's a bunch of links to content that can be downloaded, pptx, wmv mostly.

I can use _IECreate, and all the other necessary _IE.. commands to open IE and logon to the site.

I then scan no problem and get a list of the download urls on the page no problem.

I then try to do an Inetget to get the files, but this isn't working.

My guess is that the inetget creates its own session, not using the one I already created and logged onto.

So my question is, what commands would I use to do this?

Is there a better known way?

I looked at one post where someone did a _IECreateembedded to do this, but I tried that and it didn't work either.

Thanks,

Terry

Share this post


Link to post
Share on other sites
omnomnom

We would be able to help you more if you included your current code (minus any sensitive credentials). This is because its hard for us to help you blind, so having a copy of what you have currently got down then we can assist you based on that code.

Thanks

Share this post


Link to post
Share on other sites
Tvern

I'm not sure if it will work for you, but I solved a similar problem by using _IECreateEmbedded () inside an invisible guicreate() window instead of _IECreate().

edit:

I also use inetgetsource, rather than inetget. Not sure if it matters. I think the reason this works is because this way Inetget has the same parent process as the IE instance you used to log in or something like that.

Edited by Tvern

Share this post


Link to post
Share on other sites
DaleHohm

INetGet does create it's own session.

How would you interact with the page interactively in order to download the files? Use something like DebugBar to examine those elements and use more _IE* functions to perform the task.

Dale


Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

Share this post


Link to post
Share on other sites
mattw112

I'm not sure if it will work for you, but I solved a similar problem by using _IECreateEmbedded () inside an invisible guicreate() window instead of _IECreate().

edit:

I also use inetgetsource, rather than inetget. Not sure if it matters. I think the reason this works is because this way Inetget has the same parent process as the IE instance you used to log in or something like that.

I tried the IECreateEmbedded already with no luck. But I used InetGet.

I just tried using Inetgetsource instead and still just didn't do anything. But I'm curious with InetGet you can specify where to put the file you download? I dont see how you do that with inetgetsource? Also isn't Inetgetsource for getting the source html for the page?

Thanks,

Terry

Share this post


Link to post
Share on other sites
mattw112

INetGet does create it's own session.

How would you interact with the page interactively in order to download the files? Use something like DebugBar to examine those elements and use more _IE* functions to perform the task.

Dale

I've used all the _IE* commands I see to get this far using debuggar, ... but for trying to download the files? Looks like you are saying to actually get the link IDs and then 'click' those? But then what? Send commands?

Terry

Share this post


Link to post
Share on other sites
DaleHohm

I don't know. I have no clue what you see and how you interact with this page -- that is why I asked.

Dale


Free Internet Tools: DebugBar, AutoIt IE Builder, HTTP UDF, MODIV2, IE Developer Toolbar, IEDocMon, Fiddler, HTML Validator, WGet, curl

MSDN docs: InternetExplorer Object, Document Object, Overviews and Tutorials, DHTML Objects, DHTML Events, WinHttpRequest, XmlHttpRequest, Cross-Frame Scripting, Office object model

Automate input type=file (Related)

Alternative to _IECreateEmbedded? better: _IECreatePseudoEmbedded  Better Better?

IE.au3 issues with Vista - Workarounds

SciTe Debug mode - it's magic: #AutoIt3Wrapper_run_debug_mode=Y Doesn't work needs to be ripped out of the troubleshooting lexicon. It means that what you tried did not produce the results you expected. It begs the questions 1) what did you try?, 2) what did you expect? and 3) what happened instead?

Reproducer: a small (the smallest?) piece of stand-alone code that demonstrates your trouble

Share this post


Link to post
Share on other sites
Tvern

But I'm curious with InetGet you can specify where to put the file you download? I dont see how you do that with inetgetsource? Also isn't Inetgetsource for getting the source html for the page?

Just checked it and I actually use InetRead, but _INetGetSource does the same.

Both functions behave much like inetget, but instead of a file you get a variable containing all the data wich you could filewrite if you want a file.

It was usefull in my case because I needed evaluate and change the data before creating a file.

Helping you with your actual problem might be difficult as we have no idea of what you need to do to get the data.

Ideally we'd be able to use a username and pw to test the download, but I guess that's out of the question.

Alternatively it might help if you supplied the code you use to navigate to the download (remove sensitive data) so we can respond with suggestions to alter it.

Share this post


Link to post
Share on other sites
ProgAndy

You could do all actions including login and download with the WinHTTP-UDFs. Then the wholöe session would be into you AutoIt-script and you could download the files with the WinHTTP-functions.


*GERMAN* [note: you are not allowed to remove author / modified info from my UDFs]My UDFs:[_SetImageBinaryToCtrl] [_TaskDialog] [AutoItObject] [Animated GIF (GDI+)] [ClipPut for Image] [FreeImage] [GDI32 UDFs] [GDIPlus Progressbar] [Hotkey-Selector] [Multiline Inputbox] [MySQL without ODBC] [RichEdit UDFs] [SpeechAPI Example] [WinHTTP]UDFs included in AutoIt: FTP_Ex (as FTPEx), _WinAPI_SetLayeredWindowAttributes

Share this post


Link to post
Share on other sites
mattw112

Just checked it and I actually use InetRead, but _INetGetSource does the same.

Both functions behave much like inetget, but instead of a file you get a variable containing all the data wich you could filewrite if you want a file.

It was usefull in my case because I needed evaluate and change the data before creating a file.

Helping you with your actual problem might be difficult as we have no idea of what you need to do to get the data.

Ideally we'd be able to use a username and pw to test the download, but I guess that's out of the question.

Alternatively it might help if you supplied the code you use to navigate to the download (remove sensitive data) so we can respond with suggestions to alter it.

Let me try to recap.

There's a webpage you go to to enter credentials.

After credentials are entered it sends you to another page that has some links to powerpoint files mostly.

Instead of spending hours going through and manually right clicking and saying save as... I want a script that just does that for me.

Sounds pretty easy to me. And sounds like something I'm sure others have done. InetGet seems perfect except the fact it creates a new session instead of the one I've already prepped.

The code I have is pretty simple just go to the help guide for the most part.

Use IECreate, IE navigate, IEObj... etc... I'm not having trouble getting to the page with the downloads. Not having problems getting the urls from the page to the downloads... Just trouble getting the downloads to download.

$oIE = _IECreate("https://XXX/default.aspx")
; get pointers to the login form and username, password and signin fields
$o_login = _IEGetObjByName($oIE,"ctl00$cph1$UserLoginControl1$Login")
_IEFormElementSetValue ($o_login, "myusername")
$o_password = _IEGetObjByName($oIE,"ctl00$cph1$UserLoginControl1$Password")
_IEFormElementSetValue ($o_password, "mypassword")
$o_signin=_IEGetObjByName($oIE,"ctl00$cph1$UserLoginControl1$LoginBtn")
_IEAction($o_signin,"click")
sleep(5000)
_IENavigate ($oIE, "https://XXX/Sessions.aspx")

I did also try (with the inetgetsource):

$oIE =_IECreateEmbedded()
GUICreate("Embedded Web control for testing", 640, 580, _
        (@DesktopWidth - 640) / 2, (@DesktopHeight - 580) / 2, _
        $WS_OVERLAPPEDWINDOW + $WS_VISIBLE + $WS_CLIPSIBLINGS + $WS_CLIPCHILDREN)
$GUIActiveX = GUICtrlCreateObj($oIE, 10, 40, 600, 360)

GUISetState()
_IENavigate($oIE, "https://xxx/default.aspx")

This created the gui and the webpage and got to where it needed to be, but I gave up trying to get the download methods to work with it.

So far I have tried four different methods for downloading the files.

1.) Using just IENavigate to the URL. Nothing happened (also tried putting crentials in the url):

_IENavigate ($oIE, $url)

2.) Using send keys, Basically the address bar just flickered a lot.:

ControlFocus("Schedule Builder", "", "[CLASS:ToolbarWindow32; INSTANCE:2]")

ControlSend("Schedule Builder", "", "[CLASS:ToolbarWindow32; INSTANCE:2]", $url)

ControlSend("Schedule Builder", "", "[CLASS:ToolbarWindow32; INSTANCE:2]", "{ENTER}")

3.) Using _Inetgetsource, didn't work. (But I didn't know about writing the variable afterwards, how does that work for a pptx file though not text?)

$hDownload = _INetGetSource($url)

Do

Sleep(250)

Until InetGetInfo($hDownload, 2)

$nBytes = InetGetInfo($hDownload, 0)

InetClose($hDownload)

4.) Using Inetget kind of worked just got 0 KB files downloaded.

$hDownload = InetGet($url, $LastFolder & "\" & $Doc, 1, 1)

Do

Sleep(250)

Until InetGetInfo($hDownload, 2)

$nBytes = InetGetInfo($hDownload, 0)

InetClose($hDownload)

Share this post


Link to post
Share on other sites
mattw112

You could do all actions including login and download with the WinHTTP-UDFs. Then the wholöe session would be into you AutoIt-script and you could download the files with the WinHTTP-functions.

I just downloaded these and will try them out.

Thanks,

Terry

Share this post


Link to post
Share on other sites
GEOSoft

Tip: When using either InetGet or InetRead, be sure to set the flag to force a fresh download.


George

Question about decompiling code? Read the decompiling FAQ and don't bother posting the question in the forums.

Be sure to read and follow the forum rules. -AKA the AutoIt Reading and Comprehension Skills test.***

The PCRE (Regular Expression) ToolKit for AutoIT - (Updated Oct 20, 2011 ver:3.0.1.13) - Please update your current version before filing any bug reports. The installer now includes both 32 and 64 bit versions. No change in version number.

Visit my Blog .. currently not active but it will soon be resplendent with news and views. Also please remove any links you may have to my website. it is soon to be closed and replaced with something else.

"Old age and treachery will always overcome youth and skill!"

Share this post


Link to post
Share on other sites
mattw112

Tip: When using either InetGet or InetRead, be sure to set the flag to force a fresh download.

Not sure I understand? Do I need to force a fresh download? I thought that was my problem that it was doing something in its own session rather than the one I authenticated to?

Terry

Share this post


Link to post
Share on other sites
mattw112

In response to:

ProgAndy, on 24 April 2010 - 05:23 AM, said:

You could do all actions including login and download with the WinHTTP-UDFs. Then the wholöe session would be into you AutoIt-script and you could download the files with the WinHTTP-functions.

I just downloaded these and will try them out.

Thanks,

Terry

I downloaded and read and read your posts about them. I think it's over my head.

Do you have a post or example you can point me to that fits my scenario?

Authenticate to website, wait to be redirected to another, navigate to where the content is, get the urls for the downloads shown on the page, download those pptx, wmv, pdf files to certain directory on system.

Thanks,

Terry

Edited by mattw112

Share this post


Link to post
Share on other sites
GEOSoft

Not sure I understand? Do I need to force a fresh download? I thought that was my problem that it was doing something in its own session rather than the one I authenticated to?

Terry

Look at the flags for those functions. They default to using the local (cached) file first and failing that will get a new copy. So InetRead("http://www.somesite.com", 1) will force a fresh download.

Also remember that regardless of what the descriptions for those functions may say, the url MUST be fully qualified. www.somesite.com won't work. I wrote an update to the docs already but I don't recall if it was ever changed in time for the last version.

EDIT: I just looked and the docs are updated but depending on your version you may still have old docs.

Edited by GEOSoft

George

Question about decompiling code? Read the decompiling FAQ and don't bother posting the question in the forums.

Be sure to read and follow the forum rules. -AKA the AutoIt Reading and Comprehension Skills test.***

The PCRE (Regular Expression) ToolKit for AutoIT - (Updated Oct 20, 2011 ver:3.0.1.13) - Please update your current version before filing any bug reports. The installer now includes both 32 and 64 bit versions. No change in version number.

Visit my Blog .. currently not active but it will soon be resplendent with news and views. Also please remove any links you may have to my website. it is soon to be closed and replaced with something else.

"Old age and treachery will always overcome youth and skill!"

Share this post


Link to post
Share on other sites
mattw112

Look at the flags for those functions. They default to using the local (cached) file first and failing that will get a new copy. So InetRead("http://www.somesite.com", 1) will force a fresh download.

Also remember that regardless of what the descriptions for those functions may say, the url MUST be fully qualified. www.somesite.com won't work. I wrote an update to the docs already but I don't recall if it was ever changed in time for the last version.

EDIT: I just looked and the docs are updated but depending on your version you may still have old docs.

I was using a 1 to begin with..., but I think the problem is still that InetGet creates a new session, not piggy backing off the one I already created and logged into right? Or am I totally missing what you are saying.

Terry

Share this post


Link to post
Share on other sites
GEOSoft

Now I see where you are coming from. Okay, in that sense you are correct. Each download is a separate session. I was thinking along the lines of getting all the file URLs at one time and then downloading each file to your system. Are all the file links contained on a single secure page or do you have to go to a separate secure page for each file?

What I'm getting at is this. If a page contains ALL of the direct download links then you should be able to get each file direct after recovering the links from that page.


George

Question about decompiling code? Read the decompiling FAQ and don't bother posting the question in the forums.

Be sure to read and follow the forum rules. -AKA the AutoIt Reading and Comprehension Skills test.***

The PCRE (Regular Expression) ToolKit for AutoIT - (Updated Oct 20, 2011 ver:3.0.1.13) - Please update your current version before filing any bug reports. The installer now includes both 32 and 64 bit versions. No change in version number.

Visit my Blog .. currently not active but it will soon be resplendent with news and views. Also please remove any links you may have to my website. it is soon to be closed and replaced with something else.

"Old age and treachery will always overcome youth and skill!"

Share this post


Link to post
Share on other sites
ProgAndy

It's difficult to make an example for WinHTTP since all pages have to be handled a bit different and I have no example page using SSL where I can perform a login.


*GERMAN* [note: you are not allowed to remove author / modified info from my UDFs]My UDFs:[_SetImageBinaryToCtrl] [_TaskDialog] [AutoItObject] [Animated GIF (GDI+)] [ClipPut for Image] [FreeImage] [GDI32 UDFs] [GDIPlus Progressbar] [Hotkey-Selector] [Multiline Inputbox] [MySQL without ODBC] [RichEdit UDFs] [SpeechAPI Example] [WinHTTP]UDFs included in AutoIt: FTP_Ex (as FTPEx), _WinAPI_SetLayeredWindowAttributes

Share this post


Link to post
Share on other sites
mattw112

Now I see where you are coming from. Okay, in that sense you are correct. Each download is a separate session. I was thinking along the lines of getting all the file URLs at one time and then downloading each file to your system. Are all the file links contained on a single secure page or do you have to go to a separate secure page for each file?

What I'm getting at is this. If a page contains ALL of the direct download links then you should be able to get each file direct after recovering the links from that page.

All are on the same page.

Terry

Share this post


Link to post
Share on other sites
mattw112

It's difficult to make an example for WinHTTP since all pages have to be handled a bit different and I have no example page using SSL where I can perform a login.

How about just the WinHTTP commands\functions to do certain tasks, doesn't have to be code.

So:

1.) Open IE and Connect to Webpage (if different from how I'm currently doing things) use:

2.) Enter credentials and click logon (if different from how I'm currently doing things)use:

3.) Once I have the url to the download file use:

Really all I need is #3 I think?

If I know which functions to use then I can always research and figure out the code or post if I run into issues with that.

Terry

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×