Dante_t Posted July 25, 2017 Posted July 25, 2017 Hi Guys, I need help. I have searched the forum before posting and i couldn't find anything. The code below works fine when downloading files from "http" sites, but when trying to download from "https" sites, no files are downloaded. I tried different sites and I experience the same problem everywhere. Is there something I'm missing or doing wrong? Please note that I'm not a programmer and I'm new to this. I'm just using logic wherever i can to get things done. your help will be highly appreciated. #include <InetConstants.au3> #include <MsgBoxConstants.au3> #include <WinAPIFiles.au3> ; Download a file in the background. ; Wait for the download to complete. Example() Func Example() ; Save the downloaded file to the temporary folder. Local $sFilePath = "d:\" ; Download the file in the background with the selected option of 'force a reload from the remote site.' Local $hDownload = InetGet("https://en.wikipedia.org/wiki/HTTPS#/media/File:Internet2.jpg", $sFilePath& "Internet2.jpg", $INET_FORCERELOAD, $INET_DOWNLOADBACKGROUND) ; Wait for the download to complete by monitoring when the 2nd index value of InetGetInfo returns True. Do Sleep(250) Until InetGetInfo($hDownload, $INET_DOWNLOADCOMPLETE) ; Retrieve the number of total bytes received and the filesize. Local $iBytesSize = InetGetInfo($hDownload, $INET_DOWNLOADREAD) Local $iFileSize = FileGetSize($sFilePath&"Internet2.jpg") ; Close the handle returned by InetGet. InetClose($hDownload) ; Display details about the total number of bytes read and the filesize. MsgBox($MB_SYSTEMMODAL, "", "The total download size: " & $iBytesSize & @CRLF & _ "The total filesize: " & $iFileSize) ; Delete the file. ;FileDelete($sFilePath) EndFunc ;==>Example
Developers Jos Posted July 25, 2017 Developers Posted July 25, 2017 Shouldn't that URL be: "https://upload.wikimedia.org/wikipedia/commons/d/da/Internet2.jpg" ? Jos SciTE4AutoIt3 Full installer Download page - Beta files Read before posting How to post scriptsource Forum etiquette Forum Rules Live for the present, Dream of the future, Learn from the past.
MarxBros Posted July 25, 2017 Posted July 25, 2017 Hi, the problem imo is your link ..../File:Internet2.jpg (the File:) Correct Downloadlink for this file seems to be: https://upload.wikimedia.org/wikipedia/commons/thumb/d/da/Internet2.jpg/1024px-Internet2.jpg I changed your script a little bit with error message, etc... hth sample_https_download.au3
Dante_t Posted July 25, 2017 Author Posted July 25, 2017 I was just using Wikipedia as an example because to actually get the files from the site i want i have to login. the logging in and navigating to the correct web page part is sorted. I'm actually trying to download files from https://rda.ucar.edu/. file sizes are between 17 and 20 MB. I tried using your code but still no luck. I've seen people mentioning things like cookies for sites that require authentication, etc. I don't understand what they were saying but maybe that's where my problem is. My login in code is attached. i will pm my credentials. link to the file i want to download is on the code. login&navigate.au3
Danp2 Posted July 25, 2017 Posted July 25, 2017 Maybe this will work for you -- Latest Webdriver UDF Release Webdriver Wiki FAQs
MarxBros Posted July 25, 2017 Posted July 25, 2017 Hi again, just found this tip elsewhere here in the forum... inetget ("https://" & $user & ":" & $pass & "@www.mywebsite.com) can you try something like... Local $hDownload = InetGet("https://" & $User & ":" & $Pwd & "@rda.ucar.edu/data/ds083.2/grib2/2017/2017.07/fnl_20170724_06_00.grib2", $sFilePath & "fnl_20170724_06_00.grib2", $INET_DOWNLOADWAIT);, $INET_FORCERELOAD) ...with your username and password? good luck!
Dante_t Posted July 26, 2017 Author Posted July 26, 2017 (edited) 22 hours ago, Danp2 said: Maybe this will work for you -- Hi Danp2, which method are you referring to? can you please share your code if ystill have? Edited July 26, 2017 by Dante_t spelling error
Dante_t Posted July 26, 2017 Author Posted July 26, 2017 20 hours ago, MarxBros said: Hi again, just found this tip elsewhere here in the forum... inetget ("https://" & $user & ":" & $pass & "@www.mywebsite.com) can you try something like... Local $hDownload = InetGet("https://" & $User & ":" & $Pwd & "@rda.ucar.edu/data/ds083.2/grib2/2017/2017.07/fnl_20170724_06_00.grib2", $sFilePath & "fnl_20170724_06_00.grib2", $INET_DOWNLOADWAIT);, $INET_FORCERELOAD) ...with your username and password? good luck! Will try this when i get home, thank you MarxBros
Dante_t Posted July 26, 2017 Author Posted July 26, 2017 21 hours ago, MarxBros said: Hi again, just found this tip elsewhere here in the forum... inetget ("https://" & $user & ":" & $pass & "@www.mywebsite.com) can you try something like... Local $hDownload = InetGet("https://" & $User & ":" & $Pwd & "@rda.ucar.edu/data/ds083.2/grib2/2017/2017.07/fnl_20170724_06_00.grib2", $sFilePath & "fnl_20170724_06_00.grib2", $INET_DOWNLOADWAIT);, $INET_FORCERELOAD) ...with your username and password? good luck! Hi MarxBros, I tried this method and it doesnt seem to work. the code just runs continually without any file being downloaded.. how can i use IE object as Danp2 suggested above? Regards
Danp2 Posted July 26, 2017 Posted July 26, 2017 1 hour ago, Dante_t said: Hi Danp2, which method are you referring to? can you please share your code if ystill have? You create a hidden GUI containing an embedded IE object. Use this object to log into the website and navigate to the correct page. Retrieve links to be downloaded and then use InetGet to perform the download. Like this -- expandcollapse popup; ******************************************************* ; Download files from website ; ******************************************************* Func RetrieveFiles() Local $oButton, $oElement, $oForm, $oFrame, $oTable, $aTableData, $oLinks, $oLink, $iNumLinks Local $filename, $i, $trs, $href, $created, $array $IsDownloaded = False _FileWriteLog($LogFile, "Logging into website") ; Create embedded IE object $oIE = _IECreateEmbedded() ; Dummy GUI GUICreate("Dummy", 640, 580, _ (@DesktopWidth - 640) / 2, (@DesktopHeight - 580) / 2) GUICtrlCreateObj($oIE, 10, 40, 600, 360) GUISetState(@SW_SHOW) ; Open instance of website _IENavigate ($oIE, $Url) If @error Then ErrNotify("_IENavigate error. Errorcode = " & @error) Return EndIf ; Log into website $oForm = _IEFormGetCollection($oIE,0) If @error Then ErrNotify("_IEFormGetCollection error. Errorcode = " & @error) Return EndIf $oElement = _IEFormElementGetObjByName($oForm, "user") If @error Then ErrNotify("_IEFormElementGetObjByName (user) error. Errorcode = " & @error) Return EndIf _IEFormElementSetValue($oElement,$User) $oElement = _IEFormElementGetObjByName($oForm, "password") If @error Then ErrNotify("_IEFormElementGetObjByName (password) error. Errorcode = " & @error) Return EndIf _IEFormElementSetValue($oElement,$Pass) _IEFormSubmit($oForm) _IELoadWait ($oIE) _FileWriteLog($LogFile, "Switching to Downloads tab") ; Switch to Downloads tab _IELinkClickByText($oIE, "Downloads") If @error Then ErrNotify("_IELinkClickByText (Downloads) error. Errorcode = " & @error) Return EndIf $oForm = _IEFormGetObjByName($oIE, "frmSearch") If @error Then ErrNotify("_IEFormGetObjByName (Downloads) error. Errorcode = " & @error) Return EndIf $oElement = _IEFormElementGetObjByName($oForm, "lstboxes") If @error Then ErrNotify("_IEFormElementGetObjByName (Downloads) error. Errorcode = " & @error) Return EndIf For $i = 0 To $oElement.Length - 1 _IEFormElementOptionSelect($oElement, $i, 1, "byIndex", 1) _IELoadWait($oIE) _FileWriteLog($LogFile, "Retrieving links (" & $i & ")") $oLinks = _IEGetObjByName($oIE, "file_list_link", -1) If @error Then ErrNotify("_IEGetObjByName (Downloads) error. Errorcode = " & @error) Return EndIf $iNumLinks = @extended For $oLink In $oLinks $href = $oLink.href $filename = StringTrimLeft($href, StringInStr($href, "*", 0, -1)) If StringRegExp($filename, "(?i)\.(?:tif|txt)$") Then ; Make sure file wasn't already processed If Not FileExists($ProcessedDir & $filename) Then $IsDownloaded = True _FileWriteLog($LogFile, "Retrieving file: " & $filename) ; Retrieve file ConsoleWrite($InDir & $filename & @CRLF) InetGet($href, $InDir & $filename, 1) If @error Then ErrNotify("InetGet (Downloads) error. Errorcode = " & @error) Return EndIf EndIf EndIf Next ; Reselect the desired objects due to prior page reload $oForm = _IEFormGetObjByName($oIE, "frmSearch") If @error Then ErrNotify("_IEFormGetObjByName (Downloads) error. Errorcode = " & @error) Return EndIf $oElement = _IEFormElementGetObjByName($oForm, "lstboxes") If @error Then ErrNotify("_IEFormElementGetObjByName (Downloads) error. Errorcode = " & @error) Return EndIf Next _IELinkClickByText($oIE, "Log off") ; Release GUI GUIDelete() _FileWriteLog($LogFile, "Completed File Retrieval") EndFunc Latest Webdriver UDF Release Webdriver Wiki FAQs
mikell Posted July 29, 2017 Posted July 29, 2017 Huge credits to trancexx for the wonderful WinHttp.au3 , this works nice - using the *appropriate* credentials expandcollapse popup#include "WinHttp.au3" $sServerAddress = "https://rda.ucar.edu" $sGeneratorLocation = "/cgi-bin/login" ; sorry, these are private $sEmail = "xxx@xxx" $sPassword = "xxxxxxxx" $hOpen = _WinHttpOpen() ; get access cookie $hConnect = _WinHttpConnect($hOpen, $sServerAddress) _WinHttpSimpleSSLRequest($hConnect) _WinHttpCloseHandle($hConnect) ; build and fill the login form $sForm = _ '<form action="' & $sServerAddress & $sGeneratorLocation & '" method="post">' & _ ' <input name="email" />' & _ ' <input name="passwd" />' & _ ' <input name="remember" />' & _ ' <input name="do" />' & _ ' <input name="url" />' & _ '</form>' $hConnect = $sForm $sReturned = _WinHttpSimpleFormFill($hConnect, $hOpen, _ Default, _ "name:email", $sEmail, _ "name:passwd", $sPassword, _ "name:remember", "on", _ "name:do", "login", _ "name:url", "/") If @error Then MsgBox(4096, "Error", @error) ; ref. WinHttp.chm Else MsgBox(4096, "OK", "login successful, now download") ; download $sFile = "fnl_20170724_06_00.grib2" $sTarget = "data/ds083.2/grib2/2017/2017.07/" & $sFile $hConnect = _WinHttpConnect($hOpen, $sServerAddress) $hRequest = _WinHttpOpenRequest($hConnect, "GET", $sTarget) _WinHttpSendRequest($hRequest) _WinHttpReceiveResponse($hRequest) If _WinHttpQueryDataAvailable($hRequest) Then $headers = _WinHttpQueryHeaders($hRequest) $Length = StringRegExpReplace($headers, '(?s).*Content-Length:\h*(\d+).*', "$1") Local $bChunk, $bData = Binary("") ProgressOn($sFile, "Downloading", "0 %") While 1 $bChunk = _WinHttpReadData($hRequest, 2, 8192) ; binary If @error Then ExitLoop $bData &= $bChunk $percent = Int((BinaryLen($bData)/$Length)*100) ProgressSet($percent, "Downloading", $percent & " %") WEnd ProgressSet(100, "Done !", "100 %") FileWrite(@scriptdir & "\" & $sFile, $bData) Sleep(1000) ProgressOff() Else MsgBox(48, "", "connection error") EndIf ; close handles _WinHttpCloseHandle($hRequest) _WinHttpCloseHandle($hConnect) _WinHttpCloseHandle($hOpen) EndIf
Dante_t Posted July 31, 2017 Author Posted July 31, 2017 (edited) Hi Danp2, i wanted to use your code but i just didnt know where to edit it for it to do what i require it to do. Thank you Hi Mikell, I used this code. the logging in was successful. the file i would like to download is 17MB in size but this code only downloads 440bytes file. the content of the file downloaded is as follows: "Your browser sent a request that this server could not understand. Reason: You're speaking plain HTTP to an SSL-enabled server port. Instead use the HTTPS scheme to access this URL, please" Is there something im not doing right? Regards Dante Edited July 31, 2017 by Dante_t big qoutes
Dante_t Posted July 31, 2017 Author Posted July 31, 2017 expandcollapse popup#include "WinHttp.au3" $sServerAddress = "https://rda.ucar.edu" $sGeneratorLocation = "/cgi-bin/login" $sEmail = "xxxxxxx@xxxxxx" $sPassword = "xxxxxxx" $hOpen = _WinHttpOpen() ; collect access cookie first $hConnect = _WinHttpConnect($hOpen, $sServerAddress) _WinHttpSimpleSSLRequest($hConnect) _WinHttpCloseHandle($hConnect) ; build and fill the login form $sForm = _ '<form action="' & $sServerAddress & $sGeneratorLocation & '" method="post">' & _ ' <input name="email" />' & _ ' <input name="passwd" />' & _ ' <input name="remember" />' & _ ' <input name="do" />' & _ ' <input name="url" />' & _ '</form>' $hConnect = $sForm $sReturned = _WinHttpSimpleFormFill($hConnect, $hOpen, _ Default, _ "name:email", $sEmail, _ "name:passwd", $sPassword, _ "name:remember", "on", _ "name:do", "login", _ "name:url", "/") If @error Then MsgBox(4096, "Error", @error) Else MsgBox(4096, "OK", "login successful, now download") ; download $sFile = "fnl_20170724_06_00.grib2" $sTarget = "data/ds083.2/grib2/2017/2017.07/" & $sFile _WinHttpCloseHandle($hConnect) $hConnect = _WinHttpConnect($hOpen, $sServerAddress) $hRequest = _WinHttpSimpleSendSSLRequest($hConnect, "GET", $sTarget) If _WinHttpQueryDataAvailable($hRequest) Then $headers = _WinHttpQueryHeaders($hRequest) $Length = StringRegExpReplace($headers, '(?s).*Content-Length:\h*(\d+).*', "$1") Local $bChunk, $bData = Binary("") ProgressOn($sFile, "Downloading", "0 %") While 1 $bChunk = _WinHttpReadData($hRequest, 2, 8192) ; binary If @error Then ExitLoop $bData &= $bChunk $percent = Int((BinaryLen($bData)/$Length)*100) ProgressSet($percent, "Downloading", $percent & " %") WEnd ProgressSet(100, "Done !", "100 %") FileWrite(@scriptdir & "\" & $sFile, $bData) Sleep(1000) ProgressOff() Else MsgBox(48, "", "connection error") EndIf ; close handles _WinHttpCloseHandle($hRequest) _WinHttpCloseHandle($hConnect) _WinHttpCloseHandle($hOpen) EndIf
Dante_t Posted July 31, 2017 Author Posted July 31, 2017 Just now, Dante_t said: expandcollapse popup#include "WinHttp.au3" $sServerAddress = "https://rda.ucar.edu" $sGeneratorLocation = "/cgi-bin/login" $sEmail = "xxxxxxx@xxxxxx" $sPassword = "xxxxxxx" $hOpen = _WinHttpOpen() ; collect access cookie first $hConnect = _WinHttpConnect($hOpen, $sServerAddress) _WinHttpSimpleSSLRequest($hConnect) _WinHttpCloseHandle($hConnect) ; build and fill the login form $sForm = _ '<form action="' & $sServerAddress & $sGeneratorLocation & '" method="post">' & _ ' <input name="email" />' & _ ' <input name="passwd" />' & _ ' <input name="remember" />' & _ ' <input name="do" />' & _ ' <input name="url" />' & _ '</form>' $hConnect = $sForm $sReturned = _WinHttpSimpleFormFill($hConnect, $hOpen, _ Default, _ "name:email", $sEmail, _ "name:passwd", $sPassword, _ "name:remember", "on", _ "name:do", "login", _ "name:url", "/") If @error Then MsgBox(4096, "Error", @error) Else MsgBox(4096, "OK", "login successful, now download") ; download $sFile = "fnl_20170724_06_00.grib2" $sTarget = "data/ds083.2/grib2/2017/2017.07/" & $sFile _WinHttpCloseHandle($hConnect) $hConnect = _WinHttpConnect($hOpen, $sServerAddress) $hRequest = _WinHttpSimpleSendSSLRequest($hConnect, "GET", $sTarget) If _WinHttpQueryDataAvailable($hRequest) Then $headers = _WinHttpQueryHeaders($hRequest) $Length = StringRegExpReplace($headers, '(?s).*Content-Length:\h*(\d+).*', "$1") Local $bChunk, $bData = Binary("") ProgressOn($sFile, "Downloading", "0 %") While 1 $bChunk = _WinHttpReadData($hRequest, 2, 8192) ; binary If @error Then ExitLoop $bData &= $bChunk $percent = Int((BinaryLen($bData)/$Length)*100) ProgressSet($percent, "Downloading", $percent & " %") WEnd ProgressSet(100, "Done !", "100 %") FileWrite(@scriptdir & "\" & $sFile, $bData) Sleep(1000) ProgressOff() Else MsgBox(48, "", "connection error") EndIf ; close handles _WinHttpCloseHandle($hRequest) _WinHttpCloseHandle($hConnect) _WinHttpCloseHandle($hOpen) EndIf This code worked for me. Thank you Mikell for the solution and thank you everyone who tried to help. much appreciated.
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now