Downloading from Secure Website

[mattw112]

There's a website I have access to with credentials. On that website there's a bunch of links to content that can be downloaded, pptx, wmv mostly.

I can use _IECreate, and all the other necessary _IE.. commands to open IE and logon to the site.

I then scan no problem and get a list of the download urls on the page no problem.

I then try to do an Inetget to get the files, but this isn't working.

My guess is that the inetget creates its own session, not using the one I already created and logged onto.

So my question is, what commands would I use to do this?

Is there a better known way?

I looked at one post where someone did a _IECreateembedded to do this, but I tried that and it didn't work either.

Thanks,

Terry

[omnomnom]

We would be able to help you more if you included your current code (minus any sensitive credentials). This is because its hard for us to help you blind, so having a copy of what you have currently got down then we can assist you based on that code.

Thanks

[Tvern]

I'm not sure if it will work for you, but I solved a similar problem by using _IECreateEmbedded () inside an invisible guicreate() window instead of _IECreate().

edit:

I also use inetgetsource, rather than inetget. Not sure if it matters. I think the reason this works is because this way Inetget has the same parent process as the IE instance you used to log in or something like that.

Edited April 23, 2010 by Tvern

[DaleHohm]

INetGet does create it's own session.

How would you interact with the page interactively in order to download the files? Use something like DebugBar to examine those elements and use more _IE* functions to perform the task.

Dale

[mattw112]

I'm not sure if it will work for you, but I solved a similar problem by using _IECreateEmbedded () inside an invisible guicreate() window instead of _IECreate().

edit:

I also use inetgetsource, rather than inetget. Not sure if it matters. I think the reason this works is because this way Inetget has the same parent process as the IE instance you used to log in or something like that.

I tried the IECreateEmbedded already with no luck. But I used InetGet.

I just tried using Inetgetsource instead and still just didn't do anything. But I'm curious with InetGet you can specify where to put the file you download? I dont see how you do that with inetgetsource? Also isn't Inetgetsource for getting the source html for the page?

Thanks,

Terry

[mattw112]

INetGet does create it's own session.

How would you interact with the page interactively in order to download the files? Use something like DebugBar to examine those elements and use more _IE* functions to perform the task.

Dale

I've used all the _IE* commands I see to get this far using debuggar, ... but for trying to download the files? Looks like you are saying to actually get the link IDs and then 'click' those? But then what? Send commands?

Terry

[DaleHohm]

I don't know. I have no clue what you see and how you interact with this page -- that is why I asked.

Dale

[Tvern]

But I'm curious with InetGet you can specify where to put the file you download? I dont see how you do that with inetgetsource? Also isn't Inetgetsource for getting the source html for the page?

Just checked it and I actually use InetRead, but _INetGetSource does the same.

Both functions behave much like inetget, but instead of a file you get a variable containing all the data wich you could filewrite if you want a file.

It was usefull in my case because I needed evaluate and change the data before creating a file.

Helping you with your actual problem might be difficult as we have no idea of what you need to do to get the data.

Ideally we'd be able to use a username and pw to test the download, but I guess that's out of the question.

Alternatively it might help if you supplied the code you use to navigate to the download (remove sensitive data) so we can respond with suggestions to alter it.

[ProgAndy]

You could do all actions including login and download with the WinHTTP-UDFs. Then the wholöe session would be into you AutoIt-script and you could download the files with the WinHTTP-functions.

[mattw112]

Just checked it and I actually use InetRead, but _INetGetSource does the same.

Both functions behave much like inetget, but instead of a file you get a variable containing all the data wich you could filewrite if you want a file.

It was usefull in my case because I needed evaluate and change the data before creating a file.

Helping you with your actual problem might be difficult as we have no idea of what you need to do to get the data.

Ideally we'd be able to use a username and pw to test the download, but I guess that's out of the question.

Alternatively it might help if you supplied the code you use to navigate to the download (remove sensitive data) so we can respond with suggestions to alter it.

Let me try to recap.

There's a webpage you go to to enter credentials.

After credentials are entered it sends you to another page that has some links to powerpoint files mostly.

Instead of spending hours going through and manually right clicking and saying save as... I want a script that just does that for me.

Sounds pretty easy to me. And sounds like something I'm sure others have done. InetGet seems perfect except the fact it creates a new session instead of the one I've already prepped.

The code I have is pretty simple just go to the help guide for the most part.

Use IECreate, IE navigate, IEObj... etc... I'm not having trouble getting to the page with the downloads. Not having problems getting the urls from the page to the downloads... Just trouble getting the downloads to download.

$oIE = _IECreate("https://XXX/default.aspx")
; get pointers to the login form and username, password and signin fields
$o_login = _IEGetObjByName($oIE,"ctl00$cph1$UserLoginControl1$Login")
_IEFormElementSetValue ($o_login, "myusername")
$o_password = _IEGetObjByName($oIE,"ctl00$cph1$UserLoginControl1$Password")
_IEFormElementSetValue ($o_password, "mypassword")
$o_signin=_IEGetObjByName($oIE,"ctl00$cph1$UserLoginControl1$LoginBtn")
_IEAction($o_signin,"click")
sleep(5000)
_IENavigate ($oIE, "https://XXX/Sessions.aspx")

I did also try (with the inetgetsource):

$oIE =_IECreateEmbedded()
GUICreate("Embedded Web control for testing", 640, 580, _
        (@DesktopWidth - 640) / 2, (@DesktopHeight - 580) / 2, _
        $WS_OVERLAPPEDWINDOW + $WS_VISIBLE + $WS_CLIPSIBLINGS + $WS_CLIPCHILDREN)
$GUIActiveX = GUICtrlCreateObj($oIE, 10, 40, 600, 360)

GUISetState()
_IENavigate($oIE, "https://xxx/default.aspx")

This created the gui and the webpage and got to where it needed to be, but I gave up trying to get the download methods to work with it.

So far I have tried four different methods for downloading the files.

1.) Using just IENavigate to the URL. Nothing happened (also tried putting crentials in the url):

_IENavigate ($oIE, $url)

2.) Using send keys, Basically the address bar just flickered a lot.:

ControlFocus("Schedule Builder", "", "[CLASS:ToolbarWindow32; INSTANCE:2]")

ControlSend("Schedule Builder", "", "[CLASS:ToolbarWindow32; INSTANCE:2]", $url)

ControlSend("Schedule Builder", "", "[CLASS:ToolbarWindow32; INSTANCE:2]", "{ENTER}")

3.) Using _Inetgetsource, didn't work. (But I didn't know about writing the variable afterwards, how does that work for a pptx file though not text?)

$hDownload = _INetGetSource($url)

Do

Sleep(250)

Until InetGetInfo($hDownload, 2)

$nBytes = InetGetInfo($hDownload, 0)

InetClose($hDownload)

4.) Using Inetget kind of worked just got 0 KB files downloaded.

$hDownload = InetGet($url, $LastFolder & "\" & $Doc, 1, 1)

Do

Sleep(250)

Until InetGetInfo($hDownload, 2)

$nBytes = InetGetInfo($hDownload, 0)

InetClose($hDownload)

[mattw112]

You could do all actions including login and download with the WinHTTP-UDFs. Then the wholöe session would be into you AutoIt-script and you could download the files with the WinHTTP-functions.

I just downloaded these and will try them out.

Thanks,

Terry

[GEOSoft]

Tip: When using either InetGet or InetRead, be sure to set the flag to force a fresh download.

[mattw112]

Tip: When using either InetGet or InetRead, be sure to set the flag to force a fresh download.

Not sure I understand? Do I need to force a fresh download? I thought that was my problem that it was doing something in its own session rather than the one I authenticated to?

Terry

[mattw112]

In response to:

ProgAndy, on 24 April 2010 - 05:23 AM, said:

You could do all actions including login and download with the WinHTTP-UDFs. Then the wholöe session would be into you AutoIt-script and you could download the files with the WinHTTP-functions.

I just downloaded these and will try them out.

Thanks,

Terry

I downloaded and read and read your posts about them. I think it's over my head.

Do you have a post or example you can point me to that fits my scenario?

Authenticate to website, wait to be redirected to another, navigate to where the content is, get the urls for the downloads shown on the page, download those pptx, wmv, pdf files to certain directory on system.

Thanks,

Terry

Edited April 26, 2010 by mattw112

[GEOSoft]

Not sure I understand? Do I need to force a fresh download? I thought that was my problem that it was doing something in its own session rather than the one I authenticated to?

Terry

Look at the flags for those functions. They default to using the local (cached) file first and failing that will get a new copy. So InetRead("http://www.somesite.com", 1) will force a fresh download.

Also remember that regardless of what the descriptions for those functions may say, the url MUST be fully qualified. www.somesite.com won't work. I wrote an update to the docs already but I don't recall if it was ever changed in time for the last version.

EDIT: I just looked and the docs are updated but depending on your version you may still have old docs.

Edited April 26, 2010 by GEOSoft

[mattw112]

Look at the flags for those functions. They default to using the local (cached) file first and failing that will get a new copy. So InetRead("http://www.somesite.com", 1) will force a fresh download.

Also remember that regardless of what the descriptions for those functions may say, the url MUST be fully qualified. www.somesite.com won't work. I wrote an update to the docs already but I don't recall if it was ever changed in time for the last version.

EDIT: I just looked and the docs are updated but depending on your version you may still have old docs.

I was using a 1 to begin with..., but I think the problem is still that InetGet creates a new session, not piggy backing off the one I already created and logged into right? Or am I totally missing what you are saying.

Terry

[GEOSoft]

Now I see where you are coming from. Okay, in that sense you are correct. Each download is a separate session. I was thinking along the lines of getting all the file URLs at one time and then downloading each file to your system. Are all the file links contained on a single secure page or do you have to go to a separate secure page for each file?

What I'm getting at is this. If a page contains ALL of the direct download links then you should be able to get each file direct after recovering the links from that page.

[ProgAndy]

It's difficult to make an example for WinHTTP since all pages have to be handled a bit different and I have no example page using SSL where I can perform a login.

[mattw112]

Now I see where you are coming from. Okay, in that sense you are correct. Each download is a separate session. I was thinking along the lines of getting all the file URLs at one time and then downloading each file to your system. Are all the file links contained on a single secure page or do you have to go to a separate secure page for each file?

What I'm getting at is this. If a page contains ALL of the direct download links then you should be able to get each file direct after recovering the links from that page.

All are on the same page.

Terry

[mattw112]

It's difficult to make an example for WinHTTP since all pages have to be handled a bit different and I have no example page using SSL where I can perform a login.

How about just the WinHTTP commands\functions to do certain tasks, doesn't have to be code.

So:

1.) Open IE and Connect to Webpage (if different from how I'm currently doing things) use:

2.) Enter credentials and click logon (if different from how I'm currently doing things)use:

3.) Once I have the url to the download file use:

Really all I need is #3 I think?

If I know which functions to use then I can always research and figure out the code or post if I run into issues with that.

Terry