Jump to content
Sign in to follow this  
Herb191

How to detect if a URL is a download link before using _IENavigate?

Recommended Posts

Herb191

Is there a fast way of check to see if a URL is a download link before I use _IENavigate?

Also, does anyone know how to disable all popup windows (like the are your sure you want to navigate away windows) without disabling scripting?

Thanks

Edited by Herb191

Share this post


Link to post
Share on other sites
Robjong

Hey,

What do the links look like, are they just URI's to a file? Do you know what types of files to expect?

If so you could just use StringRegExp, something like this for example.

$file_url = "http://www.example.com/path/to/file.zip"
If StringRegExp($file_url, "/[^/]+\.(rar|zip)\z") Then
    ConsoleWrite("File URL: " & $file_url & @CRLF) ; do what you need want with the file URL here
EndIf
Edited by Robjong

Share this post


Link to post
Share on other sites
Herb191

Hey,

What do the links look like, are they just URI's to a file? Do you know what types of files to expect?

If so you could just use StringRegExp, something like this for example.

$file_url = "http://www.example.com/path/to/file.zip"
If StringRegExp($file_url, "/[^/]+\.(rar|zip)\z") Then
    ConsoleWrite("File URL: " & $file_url & @CRLF) ; do what you need want with the file URL here
EndIf

Hi Robjong,

Thanks for the response. I am actually trying to weed out any URL's that are download links. Unfortunately I never know what the URL's are going to look like because I am using a web crawler to get them from completely random websites. I have tried something similar to what you have above but there are thousands of possible download files and my script inevitably finds one and stops working.

Share this post


Link to post
Share on other sites
Robjong

What do you do with valid URL's? show them in an IE window? get the source?

If it is because you want to parse the URL's in a crawler like fashion, or show only html pages for example,

you could check the content-type of the URL by checking the headers...

Here is a rough example:

Global $aAcceptedContentTypes = "text/\w+" ; allows any text content type
;~ Global $aAcceptedContentTypes[2] = ["text/html", "text/plain"] ; if the type string  contains regex meta characters you can escape the string by putting it between \Q and \E(example: \Qtext/foo\E)
 
$url_text = "http://www.autoitscript.com"
$url_file = "http://www.autoitscript.com/cgi-bin/getfile.pl?autoit3/autoit-v3-setup.exe"
 
$result = _CheckContentType($url_text, $aAcceptedContentTypes)
ConsoleWrite("- " & $result & @CRLF)
 
$result = _CheckContentType($url_file, $aAcceptedContentTypes)
ConsoleWrite("- " & $result & @CRLF)
 
 
Func _CheckContentType($sURL, $mContentTypes)
    Local $oHTTP = ObjCreate("winhttp.winhttprequest.5.1")
    $oHTTP.open("HEAD", $sURL)
    $oHTTP.Send()
    If IsArray($mContentTypes) Then
        For $i = 0 To UBound($mContentTypes) - 1
            If StringRegExp($oHTTP.GetAllResponseHeaders, "(?i)content-type:\s*" & $mContentTypes[$i] & ";") Then Return SetError(0, 0, True)
        Next
    ElseIf StringRegExp($oHTTP.GetAllResponseHeaders, "(?i)content-type:\s*" & $mContentTypes & ";") Then
        Return SetError(0, 0, True)
    EndIf
    Return SetError(1, 0, False)
EndFunc

If you want download the files/source you could use 'GET' instead 'HEAD' for the open function and download if it passes the check, this would save you a request.

Edited by Robjong

Share this post


Link to post
Share on other sites
Herb191

That is nice bit of coding but I need to be able to show the page URL's in an IE window because I am processing some date after the server side scripts run...also I need to be able to run on just about any kind of page (except PDF).

Share this post


Link to post
Share on other sites
Robjong

In that case the example I provided should help you out since pages are served as HTML?!

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Similar Content

    • SkysLastChance
      By SkysLastChance
      I have a goofy problem. I am hoping someone could shed some light. The example is not going around the text box. It is way off. 
      I have seen some post blaming IE 11, however I have IE11 on my desktop and it works fine.
      Is there anything I can do that might fix this? 
       
      ; Open a browser with the form example and get a reference to the form ; textarea element. Get the coordinates and dimensions of the text area, ; outline its shape with the mouse and come to rest in the center #include <IE.au3> Local $oIE = _IE_Example("form") Local $oForm = _IEFormGetObjByName($oIE, "ExampleForm") Local $oTextArea = _IEFormElementGetObjByName($oForm, "textareaExample") ; Get coordinates and dimensions of the textarea Local $iScreenX = _IEPropertyGet($oTextArea, "screenx") Local $iScreenY = _IEPropertyGet($oTextArea, "screeny") Local $iWidth = _IEPropertyGet($oTextArea, "width") Local $iHeight = _IEPropertyGet($oTextArea, "height") ; Outline the textarea with the mouse, come to rest in the center Local $iMousespeed = 50 MouseMove($iScreenX, $iScreenY, $iMousespeed) MouseMove($iScreenX + $iWidth, $iScreenY, $iMousespeed) MouseMove($iScreenX + $iWidth, $iScreenY + $iHeight, $iMousespeed) MouseMove($iScreenX, $iScreenY + $iHeight, $iMousespeed) MouseMove($iScreenX, $iScreenY, $iMousespeed) MouseMove($iScreenX + $iWidth / 2, $iScreenY + $iHeight / 2, $iMousespeed)  
       
    • AndyS19
      By AndyS19
      I'm trying to implement a Ctl-F popup box that looks something like the one that Notepad uses, but I'm not havine much luck.  I intend to get it working, then beef up the popup's contents to add several checkboxes, buttons and radio boxes.
      What my example code does is to use InputBox(), but that's not what I want.
      Here is my test code:
      #include <Array.au3> #include <GUIConstantsEx.au3> #AutoIt3Wrapper_Au3Check_Parameters=-d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6 Opt("GUICloseOnESC", 1) Opt("GUIOnEventMode", 1) Opt('MustDeclareVars', 1) OnAutoItExitRegister("ExitStageLeft") Opt("WinTitleMatchMode", -2) Global $hGUI _Main() Func _Main() $hGUI = GUICreate("Test ^F", 300, 200) setupSpecialKeysHandlers() GUISetOnEvent($GUI_EVENT_CLOSE, "Event_GUIClose") GUISetState() While (1) Sleep(157) WEnd EndFunc ;==>_Main Func handle_CTRL_F_key() Local $str $str = InputBox("Search", "Enter the string to search for:") ConsoleWrite("+++: $str ==>" & $str & "<==" & @CRLF) EndFunc ;==>handle_CTRL_F_key Func ExitStageLeft() Exit (99) EndFunc ;==>ExitStageLeft Func Event_GUIClose() Exit (1) EndFunc ;==>Event_GUIClose Func setupSpecialKeysHandlers() Local $ar, $parts, $key, $handler, $id Local $aAccelKeys[1][2] ; Create a table of Special keys and their handlers $ar = StringSplit("", "") _ArrayAdd($ar, "^f - handle_CTRL_F_key ") ReDim $aAccelKeys[UBound($ar) - 1][2] ; Now, create $aAccelKeys array with the table data. ; For each entry, create a Dummy GUI and associate its ; ID with the special key. For $ndx = 1 To UBound($ar) - 1 $parts = StringSplit($ar[$ndx], "-", 2) $key = StringStripWS($parts[0], 8) $handler = StringStripWS($parts[1], 8) $id = GUICtrlCreateDummy() $aAccelKeys[$ndx - 1][0] = $key $aAccelKeys[$ndx - 1][1] = $id GUICtrlSetOnEvent($id, $handler) Next GUISetAccelerators($aAccelKeys) ; Setup the Special keys hooks EndFunc ;==>setupSpecialKeysHandlers  
    • SkysLastChance
      By SkysLastChance
      So I have two things I am trying to click.
      Policy which works.
      $oInputs3 = _IETagNameGetCollection($oIE, "div") For $oInput3 in $oInputs3 If StringStripWS($oInput3.innertext,1) = "Policy" Then $target = $oInput3 _IELoadWait($target,"",70000) ExitLoop EndIf Next _IEAction($target, "click")  

      And Add Insurance which I havent been able to get to work. 
      $oInputs2 = _IETagNameGetCollection($oIE, "div") For $oInput2 in $oInputs2 If StringStripWS($oInput2.innertext,1) = "Add Insurance" Then $target = $oInput2 _IELoadWait($target,"",70000) ExitLoop EndIf Next _IEAction($target, "click")

      Any Ideas on what I am doing wrong?  I feel like it might be the spaces between >  Add Insurance  < but I am not sure. 
    • FMS
      By FMS
      Hello,
      I'm trying to read a div element and wait until it hits 100%.
      The structure is like :
      <div class="progress-bar" style="width: 48.0219%;  overflow: hidden; "></div>
      And want to wait until :
      <div class="progress-bar" style="width: 100%;  overflow: hidden; "></div>
      because afther this there will be an redirection whish i don't know the URL from and want to catsh this URL.
      And want to push a button on this redidertion page.

      Is there a best pratice way how to do this or is there a better way to wait for the redirection?
      Maybe wait until button exist or something?

      Does anybody could give me some tips about this challange?
       
      thnx in advanced.
       
      #include <IE.au3> Global $IE_flvto = _IECreate("https://www.website.com/",0,1,1,1) Global $oForm = _IEFormGetObjByName ($IE_flvto, "convertForm") Global $oText = _IEFormElementGetObjByName ($oForm, "convertUrl") _IEFormElementSetValue ($oText, "some text") _IEFormSubmit($oForm) ;wait for redirection ;if redirection loaded push button  
    • Juvigy
      By Juvigy
       Hi Guys,
      Could you please try out my script on win10 machine (simple site scrape)? One of my users complains it doesnt work on his win10, but it works fine on my win7. The error he gets is on the .FireEvent call. I think it is admin rights or IE / Edge issue, but don't have win10 to test it out.
      Thanks in advance.
       
      #include <IE.au3> #include <Array.au3> #include <Excel.au3> Global $oIE,$string Dim $destination[8] = ["Viña del Mar", "Rancagua", "Pucon", "Copiapo", "Temuco", "La serena", "Puerto Montt", "Valdivia"] Dim $FinalResult[1][3] $FinalResult[0][0] = "Destination" $FinalResult[0][1] = "Ida" $FinalResult[0][2] = "Vuelta" Attach("https://www.turbus.cl/") If IsObj($oIE) = 0 Then ConsoleWrite("IE error??"&@CRLF) $oIE = _IECreate() EndIf For $i=0 to UBound($destination)-1 Step 1 _IENavigate($oIE,"https://www.turbus.cl/") $site1 = _IEGetObjById($oIE,"j_id_id122:cmbCiudadOrigenV2") While @error Sleep(1000) $site1 = _IEGetObjById($oIE,"j_id_id122:cmbCiudadOrigenV2") WEnd $site2 = _IEGetObjById($oIE,"j_id_id122:cmbCiudadDestinoV2") $date1 = _IEGetObjById($oIE,"j_id_id122:calIdaV2InputDate") $date2 = _IEGetObjById($oIE,"j_id_id122:calVueltaV2InputDate") $buttun1 = _IEGetObjById($oIE,"j_id_id122:botonContinuarV2") $site1.Value = "Santiago" $site2.Value = $destination[$i] $date1.Value = @MDAY+1&"/"&@MON&"/"&@YEAR $date2.Value = @MDAY+4&"/"&@MON&"/"&@YEAR _IEAction($buttun1,"click") _IELoadWait($oIE) $array = GetResult(GetData()) If IsArray($array) = 0 OR UBound($array,1) < 1 Or UBound($array,2) < 2 Then MsgBox(0,UBound($array,1), UBound($array,2)) _ArrayDisplay($array,"$array") ContinueLoop EndIf _ArrayAdd($FinalResult,"Santiago-"&$destination[$i]&"|"&$array[0][0]&"|"&$array[0][1]) Next ;~ _ArrayDisplay($FinalResult) Local $oExcel = _Excel_Open() If @error Then Exit MsgBox($MB_SYSTEMMODAL, "Excel UDF: _Excel_RangeWrite Example", "Error creating the Excel application object." & @CRLF & "@error = " & @error & ", @extended = " & @extended) Local $oWorkbook = _Excel_BookNew($oExcel) If @error Then MsgBox($MB_SYSTEMMODAL, "Excel UDF: _Excel_RangeWrite Example", "Error creating the new workbook." & @CRLF & "@error = " & @error & ", @extended = " & @extended) _Excel_Close($oExcel) Exit EndIf _Excel_RangeWrite($oWorkbook, $oWorkbook.Activesheet, $FinalResult, "A1") If @error Then Exit MsgBox($MB_SYSTEMMODAL, "Excel UDF: _Excel_RangeWrite Example 2", "Error writing to worksheet." & @CRLF & "@error = " & @error & ", @extended = " & @extended) Func GetData() Local $string1,$string2 $ida = _IEGetObjById($oIE,"tbIda_lbl") While @error Sleep(1000) $ida = _IEGetObjById($oIE,"tbIda_lbl") WEnd _IEAction($ida,"click") Sleep(2000) $result1 = _IEGetObjById($oIE,"pnlReglaIda:idPersonalizaPasaje:tb") While @error $ida.FireEvent("onmouseover") Sleep(1000) $ida.FireEvent("onclick") Sleep(1000) $result1 = _IEGetObjById($oIE,"pnlReglaIda:idPersonalizaPasaje:tb") WEnd $string1 = $string1 & $result1.innertext&@CRLF $vuelta = _IEGetObjById($oIE,"tbVuelta_lbl") _IEAction($vuelta,"click") Sleep(2000) $result2 = _IEGetObjById($oIE,"pnlReglaVuelta:idPersonalizaPasajeRegreso:tb") While @error $vuelta.FireEvent("onmouseover") Sleep(1000) $vuelta.FireEvent("onclick") Sleep(1000) $result2 = _IEGetObjById($oIE,"pnlReglaVuelta:idPersonalizaPasajeRegreso:tb") WEnd $string2 = $string2 & $result2.innertext&@CRLF Return $string1&"|"&$string2 EndFunc Func Attach($atachadres) Local $i = 1 While 1 $oIE = _IEAttach("", "instance", $i) If @error = $_IEStatus_NoMatch Then $oIE = 0 ExitLoop EndIf If StringLeft(_IEPropertyGet($oIE, "locationurl"),StringLen($atachadres)) = $atachadres Then ExitLoop $i += 1 WEnd EndFunc Func GetResult($String) Dim $Master[0][2] ,$Master2[1][2] $2strings = StringSplit($String,"|",2) $Strings = StringSplit($2strings[0],@CRLF,2) $Strings = _ArrayUnique($Strings) For $i=0 to UBound($Strings)-1 Step 1 $Data = StringSplit($Strings[$i],"$",2) $add = _ArrayToString($Data) _ArrayAdd($Master,$add) Next For $i=UBound($Master,1)-1 to 0 Step -1 If $Master[$i][1] = "" Then _ArrayDelete($Master,$i) Next _ArraySort($Master,0, 0, 0,1) $Master2[0][0] = $Master[0][0]&$Master[0][1] Dim $Master[0][2] $Strings = StringSplit($2strings[1],@CRLF,2) $Strings = _ArrayUnique($Strings) For $i=0 to UBound($Strings)-1 Step 1 $Data = StringSplit($Strings[$i],"$",2) $add = _ArrayToString($Data) _ArrayAdd($Master,$add) Next For $i=UBound($Master,1)-1 to 0 Step -1 If $Master[$i][1] = "" Then _ArrayDelete($Master,$i) Next _ArraySort($Master,0, 0, 0,1) $Master2[0][1] = $Master[0][0]&$Master[0][1] Return $Master2 EndFunc  
×