Sign in to follow this  
Followers 0
Herb191

How to detect if a URL is a download link before using _IENavigate?

6 posts in this topic

#1 ·  Posted (edited)

Is there a fast way of check to see if a URL is a download link before I use _IENavigate?

Also, does anyone know how to disable all popup windows (like the are your sure you want to navigate away windows) without disabling scripting?

Thanks

Edited by Herb191

Share this post


Link to post
Share on other sites



#2 ·  Posted (edited)

Hey,

What do the links look like, are they just URI's to a file? Do you know what types of files to expect?

If so you could just use StringRegExp, something like this for example.

$file_url = "http://www.example.com/path/to/file.zip"
If StringRegExp($file_url, "/[^/]+\.(rar|zip)\z") Then
    ConsoleWrite("File URL: " & $file_url & @CRLF) ; do what you need want with the file URL here
EndIf
Edited by Robjong

Share this post


Link to post
Share on other sites

Hey,

What do the links look like, are they just URI's to a file? Do you know what types of files to expect?

If so you could just use StringRegExp, something like this for example.

$file_url = "http://www.example.com/path/to/file.zip"
If StringRegExp($file_url, "/[^/]+\.(rar|zip)\z") Then
    ConsoleWrite("File URL: " & $file_url & @CRLF) ; do what you need want with the file URL here
EndIf

Hi Robjong,

Thanks for the response. I am actually trying to weed out any URL's that are download links. Unfortunately I never know what the URL's are going to look like because I am using a web crawler to get them from completely random websites. I have tried something similar to what you have above but there are thousands of possible download files and my script inevitably finds one and stops working.

Share this post


Link to post
Share on other sites

#4 ·  Posted (edited)

What do you do with valid URL's? show them in an IE window? get the source?

If it is because you want to parse the URL's in a crawler like fashion, or show only html pages for example,

you could check the content-type of the URL by checking the headers...

Here is a rough example:

Global $aAcceptedContentTypes = "text/\w+" ; allows any text content type
;~ Global $aAcceptedContentTypes[2] = ["text/html", "text/plain"] ; if the type string  contains regex meta characters you can escape the string by putting it between \Q and \E(example: \Qtext/foo\E)
 
$url_text = "http://www.autoitscript.com"
$url_file = "http://www.autoitscript.com/cgi-bin/getfile.pl?autoit3/autoit-v3-setup.exe"
 
$result = _CheckContentType($url_text, $aAcceptedContentTypes)
ConsoleWrite("- " & $result & @CRLF)
 
$result = _CheckContentType($url_file, $aAcceptedContentTypes)
ConsoleWrite("- " & $result & @CRLF)
 
 
Func _CheckContentType($sURL, $mContentTypes)
    Local $oHTTP = ObjCreate("winhttp.winhttprequest.5.1")
    $oHTTP.open("HEAD", $sURL)
    $oHTTP.Send()
    If IsArray($mContentTypes) Then
        For $i = 0 To UBound($mContentTypes) - 1
            If StringRegExp($oHTTP.GetAllResponseHeaders, "(?i)content-type:\s*" & $mContentTypes[$i] & ";") Then Return SetError(0, 0, True)
        Next
    ElseIf StringRegExp($oHTTP.GetAllResponseHeaders, "(?i)content-type:\s*" & $mContentTypes & ";") Then
        Return SetError(0, 0, True)
    EndIf
    Return SetError(1, 0, False)
EndFunc

If you want download the files/source you could use 'GET' instead 'HEAD' for the open function and download if it passes the check, this would save you a request.

Edited by Robjong

Share this post


Link to post
Share on other sites

That is nice bit of coding but I need to be able to show the page URL's in an IE window because I am processing some date after the server side scripts run...also I need to be able to run on just about any kind of page (except PDF).

Share this post


Link to post
Share on other sites

In that case the example I provided should help you out since pages are served as HTML?!

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0

  • Similar Content

    • jonson1986
      By jonson1986
      Hey,
      I'm trying to select language from drop-down menu on a webpage by _IEGetObjById but it's  not working, It mean autoit open the page but unable to select my desired value not even by matching text or by value, I'm looking for kind help...I've pastes my Autoit codes below along with source codes(using ie) of the webpage ;
      #include <IE.au3> $site = "http://example.com/upload" $oIE = _IECreate($site) $oDownloadSamples = _IEGetObjById($oIE, "language") $oDownloadSamplesSelect = _IETagNameGetCollection($oDownloadSamples, "select", "language") $oDownloadSamplesSelect.focus _IEFormElementOptionSelect($oDownloadSamplesSelect, "French", 1, "byText") Sleep (1000) _IEFormElementOptionSelect($oDownloadSamplesSelect, "2", 1, "byValue") HTML Codes using IE:
      <html> <head> <div class="form-row"> <div class="col-7"> <label>Language</label> <select name="language" class="select" id="language"> <option value="1">English</option> <option value="2">French</option> <option value="3">German</option> <option value="4">Italian</option> <option value="5">Japanese</option> <option value="6">Spanish</option> <option value="7">Russian</option> <option value="8">Hindi</option> <option value="9">Arabic</option> <option value="10">Chinese</option> <option value="11">Dutch</option> <option value="12">Finnish</option> <option value="13">Korean</option> <option value="14">Norwegian</option> <option value="15">Portuguese</option> <option value="16">Romanian</option> <option value="17">Serbian</option> <option value="18">Croatian</option> <option value="20">Polish</option> <option value="21">Afar</option> <option value="22">Abkhazian</option> <option value="23">Afrikaans</option> <option value="24">Amharic</option> <option value="25">Assamese</option> <option value="26">Aymara</option> <option value="27">Azerbaijani</option> <option value="28">Bashkir</option> <option value="29">Belarusian</option> <option value="30">Bulgarian</option> <option value="31">Bihari</option> <option value="32">Bislama</option> <option value="33">Bengali/Bangla</option> <option value="34">Tibetan</option> <option value="35">Breton</option> <option value="36">Catalan</option> <option value="37">Corsican</option> <option value="38">Czech</option> <option value="39">Welsh</option> <option value="40">Danish</option> <option value="41">Bhutani</option> <option value="42">Greek</option> <option value="43">Esperanto</option> <option value="44">Estonian</option> <option value="45">Basque</option> <option value="46">Persian</option> <option value="47">Fiji</option> <option value="48">Faeroese</option> <option value="49">Frisian</option> <option value="50">Irish</option> <option value="51">Scots/Gaelic</option> <option value="52">Galician</option> <option value="53">Guarani</option> <option value="54">Gujarati</option> <option value="55">Hausa</option> <option value="56">Hungarian</option> <option value="57">Armenian</option> <option value="58">Interlingua</option> <option value="59">Interlingue</option> <option value="60">Inupiak</option> <option value="61">Indonesian</option> <option value="62">Icelandic</option> <option value="63">Hebrew</option> <option value="64">Yiddish</option> <option value="65">Javanese</option> <option value="66">Georgian</option> <option value="67">Kazakh</option> <option value="68">Greenlandic</option> <option value="69">Cambodian</option> <option value="70">Kannada</option> <option value="71">Kashmiri</option> <option value="72">Kurdish</option> <option value="73">Kirghiz</option> <option value="74">Latin</option> <option value="75">Lingala</option> <option value="76">Laothian</option> <option value="77">Lithuanian</option> <option value="78">Latvian/Lettish</option> <option value="79">Malagasy</option> <option value="80">Maori</option> <option value="81">Macedonian</option> <option value="82">Malayalam</option> <option value="83">Mongolian</option> <option value="84">Moldavian</option> <option value="85">Marathi</option> <option value="86">Malay</option> <option value="87">Maltese</option> <option value="88">Burmese</option> <option value="89">Nauru</option> <option value="90">Nepali</option> <option value="91">Occitan</option> <option value="92">(Afan)/Oromoor/Oriya</option> <option value="93">Punjabi</option> <option value="94">Pashto/Pushto</option> <option value="95">Quechua</option> <option value="96">Rhaeto-Romance</option> <option value="97">Kirundi</option> <option value="98">Kinyarwanda</option> <option value="99">Sanskrit</option> <option value="100">Sindhi</option> <option value="101">Sangro</option> <option value="102">Serbo-Croatian</option> <option value="103">Singhalese</option> <option value="104">Slovak</option> <option value="105">Slovenian</option> <option value="106">Samoan</option> <option value="107">Shona</option> <option value="108">Somali</option> <option value="109">Albanian</option> <option value="110">Siswati</option> <option value="111">Sesotho</option> <option value="112">Sundanese</option> <option value="113">Swedish</option> <option value="114">Swahili</option> <option value="115">Tamil</option> <option value="116">Telugu</option> <option value="117">Tajik</option> <option value="118">Thai</option> <option value="119">Tigrinya</option> <option value="120">Turkmen</option> <option value="121">Tagalog</option> <option value="122">Setswana</option> <option value="123">Tonga</option> <option value="124">Turkish</option> <option value="125">Tsonga</option> <option value="126">Tatar</option> <option value="127">Twi</option> <option value="128">Ukrainian</option> <option value="129">Urdu</option> <option value="130">Uzbek</option> <option value="131">Vietnamese</option> <option value="132">Volapuk</option> <option value="133">Wolof</option> <option value="134">Xhosa</option> <option value="135">Yoruba</option> <option value="136">Zulu</option> <option value="19">Other</option> </select> </div> </main> </body> </html>  
    • kawliga751
      By kawliga751
      I am trying to script the opening of multiple tabs in IE with each tab having a separate login/password. I have been able to make a successful script that will open 3 separate IE sessions with the correct webpage and login or a script that will open 3 tabs in one session (which I want) but will not login. I have tried _IECreate, IEAttach etc. This is the script I am currently using . 
      #include <IE.au3> Const $navOpenInNewTab = 0x0800 Dim $oIE = _IECreate('http://asag.xxxxxxx.com/AAAA/index.htm') ;$o_IE.Navigate2('http://asag.xxxxxxx.com/apps/yyyyyyy/LogInSAG/login.asp', $navOpenInNewTab) ;$o_IE.Navigate2('http://asag.xxxxxxx.com/apps/yyyyyyy/LogInSAG/login.asp', $navOpenInNewTab) Call ("YYYYYYYSignIn") Func YYYYYYYSignIn () Local $username = _IEGetObjByName ($oIE,"uname") Local $password = _IEGetObjByName ($oIE,"pword") $oIE.Navigate2('http://asag.xxxxxxx.com/apps/xxxxxxx/LogInSAG/login.asp',2048) _IEFormElementSetValue ($username, "xxxxxxx") _IEFormElementSetValue ($password, "xxxxxxx") Send('{Enter}') EndFunc Call ("xxxxxxxSignIn") Func xxxxxxxSignIn () __IENavigate($oIE, "https://soa1gui.xxxxxxx.biz/gateway/NFServlet;jsessionid=1E3B8F6520DE0A1EC914A8ABB7E86341?NFH_MessageId=&NFH_Page=%2fpages%2fsearch%2fsearch.jsp", 1, 0x800) Local $domain = _IEGetObjByName ($oIE,"NF_CustomerID") Local $User_Name = _IEGetObjByName ($oIE,"NF_UserName") Local $Password = _IEGetObjByName ($oIE,"NF_Password") _IEFormElementSetValue ($domain, xxxxxxx") _IEFormElementSetValue ($User_Name, "xxxxxxx") _IEFormElementSetValue ($Password, "xxxxxxx") ;Send('{Enter}') EndFunc  
    • slidy56fr
      By slidy56fr
      Internet Explorer 11, AutoIt 3.3.14.2
      I want to download many files which are published with hyperlink on a web page. I use AutoIt and when there is a few files, it works even if It's on a website with secured links.
      I have a script to download link items with the object method (I gave a partial sample), but when the number of link exceed the web page, there is not an auto scroll, then the links which are not on displayed without scroll are not downloaded.
      Thank you
      From France
       
       
      EchantillonRecepAnonym.au3
    • SkysLastChance
      By SkysLastChance
      I am not sure on how to grab the innertext I want. Is there a way I can just grab the second line, or is there a better way to do it all together? 
      #include <IE.au3> $oIE = _IEAttach("Form Details") $oDiv = _IEGetObjById($oIE, "Col3") ;Phone MsgBox (0,"Oops",$oDiv.outertext) When I use this code I get

       


       
      html.au3
    • anthonyjr2
      By anthonyjr2
      I have come across a weird issue that I am not sure how to solve. For some reason, whenever I call WinActivate in certain scenarios instead of opening the single window that already exists, it will create a new blank tab. It should be able to be recreated with this:
      AutoItSetOption("WinTitleMatchMode", 2) Sleep(1000) For $i=0 To 10 WinActivate("Internet Explorer") Next I am not actually doing this in my code, it is just to demonstrate the fact that new tabs keep popping up. Does anyone know why this happens?
       
      EDIT: So I am not sure how reproducible this is, because it only happens every once in a while for me. I will update with more info if I find out anything.