Jump to content
Sign in to follow this  
t0nZ

How to download files from temporary links

Recommended Posts

t0nZ

So my problem was to download files from temporary links.

If you consider Combofix, an util to clean Windows from malware, it's a program updated daily and I want always the last version.

But the download link is temporary and always changing, and I don't trust other sites static links.

I developed a system to download a list of links, the main idea is to have the list of links in the page and after I go parsing the list to found the correct temporary links.

It's dirty but it's working.

This is a "lite" version for the forum.:

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_Icon=Icone\Full ico\down.ico
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****

;NSC 2014
; download temporary links demo


#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <ProgressConstants.au3>
#include <WindowsConstants.au3>
#include <Array.au3>
#include <GuiEdit.au3>
#include <IE.au3>
#include <string.au3>

Global $uProgress1, $UPDATE, $updateArray, $updateDownload, $downlist, $adown, $conta
Global $downpath = @DesktopDir & "\GetDown"

;_________________

Gui()
Prepare()
downlist()
downloop()
end()

Exit


Func Gui()

    #Region ### START Koda GUI section ### Form=C:\Program Files (x86)\AutoIt3\Examples\NSC_test\formS\update.kxf
    $UForm1 = GUICreate("GET DOWN lite V.1.5", 300, 200, 200, 150)
    GUISetBkColor(0xC0C0C0)
    $uProgress1 = GUICtrlCreateProgress(265, 5, 30, 190, BitOR($PBS_SMOOTH, $PBS_VERTICAL, $WS_BORDER))
    GUICtrlSetColor(-1, 0x0033FF)
    GUICtrlSetBkColor(-1, 0x000066)
    $UPDATE = GUICtrlCreateEdit("", 5, 5, 250, 190, BitOR($ES_AUTOVSCROLL, $ES_AUTOHSCROLL, $ES_WANTRETURN, $WS_BORDER))
    GUICtrlSetData(-1, "Gettin'down temporary links" & @CRLF)
    GUICtrlSetFont(-1, 8, 800, 0, "Verdana")
    GUICtrlSetColor(-1, 0x99FFFF)
    GUICtrlSetBkColor(-1, 0x330000)
    GUICtrlSetCursor(-1, 3)
    GUISetState(@SW_SHOW)
    #EndRegion ### END Koda GUI section ###

EndFunc   ;==>Gui

Func Prepare()
    If Not FileExists($downpath) Then DirCreate($downpath)
EndFunc   ;==>Prepare

Func downlist()
    Dim $adown[3] ; in other scripts I download a txt file from internet.
    $adown[0] = "http://www.bleepingcomputer.com/download/adwcleaner/dl/125/|adwcleaner.exe"
    $adown[1] = "http://www.bleepingcomputer.com/download/combofix/dl/12/|combofix.exe"
    $adown[2] = "http://www.bleepingcomputer.com/download/rkill/dl/10/|rkill.exe"
    _GUICtrlEdit_AppendText($UPDATE, "download list assigned. " & @CRLF)
EndFunc   ;==>downlist



Func downloop()

    While 1
        $filedown = _ArrayPop($adown)
        If @error <> 0 Then ExitLoop

        $arraydown = _StringExplode($filedown, "|") ; example of line processed "www.yourlinkhere.com|yourfile.exe"

        Local $oIE = _IECreate($arraydown[0], 0, 0, 1, 0) ;hidden IE
        Local $oLinks = _IELinkGetCollection($oIE) ; list of links!
        Local $iNumLinks = @extended
        _GUICtrlEdit_AppendText($UPDATE, "Searching in " & $iNumLinks & " links for " & $arraydown[0] & @CRLF)
        Local $sTxt = $iNumLinks & " links found" & @CRLF & @CRLF ; uncomment for list of links (1/3)
        For $oLink In $oLinks
            $sTxt &= $oLink.href & @CRLF ;uncomment for list of links (2/3)
            If StringInStr($oLink.href, $arraydown[1]) Then
                Inetdown($oLink.href, $arraydown[1])
            EndIf
        Next
        MsgBox($MB_SYSTEMMODAL, "Link Info", $sTxt) ;uncomment for list of links (3/3)
    WEnd

EndFunc   ;==>downloop


Func Inetdown($filedown, $nomefiledown)
    $updateDownload = InetGet($filedown, $downpath & "\" & $nomefiledown, 1, 1)
    _GUICtrlEdit_AppendText($UPDATE, "downloading " & $nomefiledown & @CRLF)

    Do
        Sleep(500)
        _GUICtrlEdit_AppendText($UPDATE, "bytes downloaded :" & InetGetInfo($updateDownload, 0) & @CRLF)
        GUICtrlSetData($uProgress1, (100 * InetGetInfo($updateDownload, 0) / InetGetInfo($updateDownload, 1)))
    Until InetGetInfo($updateDownload, 2) ; Check if the download is complete.

    InetClose($updateDownload) ; Close the handle to release resources.
    _GUICtrlEdit_AppendText($UPDATE, "downloaded in " & $downpath & @CRLF)
EndFunc   ;==>Inetdown

Func end()
    Run("explorer.exe " & $downpath)
    MsgBox(64, "NSC GET DOWN", "Files downloaded!")
EndFunc   ;==>end

the Function downloop() contains the trick.

The complete version downloads from a public server the list of the files to download, and for every file there is a command to choose various download methods and  to exucute commands (run the files etc) after the download, and at the end I have a mail with a report.

Edited by t0nZ

Share this post


Link to post
Share on other sites
wakillon

Why use IE ?

You have just to extract links from source and find the one you want (temporary or not)

#include <Array.au3>

$sSrc = BinaryToString ( InetRead ( 'http://www.bleepingcomputer.com/download/combofix/' ) )
$aLinks = StringRegExp ( $sSrc, '(?s)(?i)<a href="(.*?)"', 3 )
$iIndex = _ArraySearch ( $aLinks, 'download/combofix', 0, 0, 0, 1 )
ConsoleWrite ( '+ Link Download Page : ' & $aLinks[$iIndex] & @Crlf )

$sSrc = StringReplace ( BinaryToString ( InetRead ( $aLinks[$iIndex] ) ), "'", '"' )
$aLink = StringRegExp ( $sSrc, '(?s)(?i)<a href="(.*?)"', 3 )
$iIndex = _ArraySearch ( $aLink, 'ComboFix.exe', 0, 0, 0, 1 )
ConsoleWrite ( '! Link Download ComboFix.exe : ' & $aLink[$iIndex] & @Crlf )
  • Like 1

AutoIt 3.3.14.2 X86 - SciTE 3.6.0WIN 8.1 X64 - Other Example Scripts

Share this post


Link to post
Share on other sites
t0nZ

Thank you for the snippet !

I Like this method, it's faster and more efficient (like a lot of things without IE....)

So I have modified my example to try three methods, with the evidence of timings.

Third method ? Your snippet + >this.

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_Icon=Icone\Full ico\down.ico
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****

;NSC 2014
; download temporary links demo


#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <ProgressConstants.au3>
#include <WindowsConstants.au3>
#include <Array.au3>
#include <GuiEdit.au3>
#include <IE.au3>
#include <string.au3>
#include "UrlDownloadEx.au3"

Global $uProgress1, $UPDATE, $updateArray, $updateDownload, $downlist, $adown, $conta
Global $downpath1 = @DesktopDir & "\GetDown1"
Global $downpath2 = @DesktopDir & "\GetDown2"
Global $downpath3 = @DesktopDir & "\GetDown3"

;_________________

Gui()
Prepare()
downlist()
downloop1()
downlist()
downloop2()
downlist()
downloop3()
end()

Exit


Func Gui()

    #Region ### START Koda GUI section ### Form=C:\Program Files (x86)\AutoIt3\Examples\NSC_test\formS\update.kxf
    $UForm1 = GUICreate("GET DOWN lite V.1.7", 300, 200, 200, 150)
    GUISetBkColor(0xC0C0C0)
    $uProgress1 = GUICtrlCreateProgress(265, 5, 30, 190, BitOR($PBS_SMOOTH, $PBS_VERTICAL, $WS_BORDER))
    GUICtrlSetColor(-1, 0x0033FF)
    GUICtrlSetBkColor(-1, 0x000066)
    $UPDATE = GUICtrlCreateEdit("", 5, 5, 250, 190, BitOR($ES_AUTOVSCROLL, $ES_AUTOHSCROLL, $ES_WANTRETURN, $WS_BORDER))
    GUICtrlSetData(-1, "Gettin'down temporary links" & @CRLF)
    GUICtrlSetFont(-1, 8, 800, 0, "Verdana")
    GUICtrlSetColor(-1, 0x99FFFF)
    GUICtrlSetBkColor(-1, 0x330000)
    GUICtrlSetCursor(-1, 3)
    GUISetState(@SW_SHOW)
    #EndRegion ### END Koda GUI section ###

EndFunc   ;==>Gui

Func Prepare()
    If Not FileExists($downpath1) Then DirCreate($downpath1)
    If Not FileExists($downpath2) Then DirCreate($downpath2)
    If Not FileExists($downpath3) Then DirCreate($downpath3)
EndFunc   ;==>Prepare

Func downlist()
    Dim $adown[3] ; in other scripts I download a txt file from internet.
    $adown[0] = "http://www.bleepingcomputer.com/download/adwcleaner/dl/125/|adwcleaner.exe"
    $adown[1] = "http://www.bleepingcomputer.com/download/combofix/dl/12/|combofix.exe"
    $adown[2] = "http://www.bleepingcomputer.com/download/rkill/dl/10/|rkill.exe"
    _GUICtrlEdit_AppendText($UPDATE, "download list assigned. " & @CRLF)
EndFunc   ;==>downlist

Func downloop1()
    MsgBox(64, "getdown1", "start downloading IE mode")
    $iBegin = TimerInit()
    While 1

        $filedown = _ArrayPop($adown)
        If @error <> 0 Then ExitLoop

        $arraydown = _StringExplode($filedown, "|") ; example of line processed "www.yourlinkhere.com|yourfile.exe"

        ;;;___________________
        Local $oIE = _IECreate($arraydown[0], 0, 0, 1, 0) ;hidden IE
        Local $oLinks = _IELinkGetCollection($oIE) ; list of links!
        Local $iNumLinks = @extended
        _GUICtrlEdit_AppendText($UPDATE, "Searching in " & $iNumLinks & " links for " & $arraydown[0] & @CRLF)
        ;Local $sTxt = $iNumLinks & " links found" & @CRLF & @CRLF ; uncomment for list of links (1/3)
        For $oLink In $oLinks
            ;   $sTxt &= $oLink.href & @CRLF ;uncomment for list of links (2/3)
            If StringInStr($oLink.href, $arraydown[1]) Then
                Inetdown($oLink.href, $downpath1 & "\" & $arraydown[1])
            EndIf
        Next
        ;MsgBox($MB_SYSTEMMODAL, "Link Info", $sTxt) ;uncomment for list of links (3/3)
    WEnd
    $iDiff = Int(TimerDiff($iBegin) / 1000)
    $iMin = Int($iDiff / 60)
    $iSec = $iDiff - ($iMin * 60)
    MsgBox(64, "Downloaded in ", $iMin & ":" & StringFormat("%02i", $iSec))

EndFunc   ;==>downloop1


Func downloop2()
    MsgBox(64, "getdown2", "start downloading INETGET mode")
    $iBegin = TimerInit()
    While 1
        $filedown = _ArrayPop($adown)
        If @error <> 0 Then ExitLoop

        $arraydown = _StringExplode($filedown, "|") ; example of line processed "www.yourlinkhere.com|yourfile.exe"


        $sSrc = StringReplace(BinaryToString(InetRead($arraydown[0])), "'", '"')
        $aLink = StringRegExp($sSrc, '(?s)(?i)<a href="(.*?)"', 3)
        $iIndex = _ArraySearch($aLink, $arraydown[1], 0, 0, 0, 1)
        ConsoleWrite('! Link Download  ' & $aLink[$iIndex] & @CRLF)
        Inetdown($aLink[$iIndex], $downpath2 & "\" & $arraydown[1])
    WEnd
    $iDiff = Int(TimerDiff($iBegin) / 1000)
    $iMin = Int($iDiff / 60)
    $iSec = $iDiff - ($iMin * 60)

    MsgBox(64, "Downloaded in ", $iMin & ":" & StringFormat("%02i", $iSec))
EndFunc   ;==>downloop2


Func downloop3()
    MsgBox(64, "getdown3", "start downloading INETGET mode + UrlDownloadEX")
    $iBegin = TimerInit()
    While 1
        $filedown = _ArrayPop($adown)
        If @error <> 0 Then ExitLoop

        $arraydown = _StringExplode($filedown, "|") ; example of line processed "www.yourlinkhere.com|yourfile.exe"


        $sSrc = StringReplace(BinaryToString(InetRead($arraydown[0])), "'", '"')
        $aLink = StringRegExp($sSrc, '(?s)(?i)<a href="(.*?)"', 3)
        $iIndex = _ArraySearch($aLink, $arraydown[1], 0, 0, 0, 1)
        ConsoleWrite('! Link Download ' & $aLink[$iIndex] & @CRLF)
        UrlDownloadEx($aLink[$iIndex], $downpath3 & "\" & $arraydown[1])
    WEnd
    $iDiff = Int(TimerDiff($iBegin) / 1000)
    $iMin = Int($iDiff / 60)
    $iSec = $iDiff - ($iMin * 60)

    MsgBox(64, "Downloaded in ", $iMin & ":" & StringFormat("%02i", $iSec))
EndFunc   ;==>downloop3

Func Inetdown($filedown, $nomefiledown)
    $updateDownload = InetGet($filedown, $nomefiledown, 1, 1)
    _GUICtrlEdit_AppendText($UPDATE, "downloading " & $nomefiledown & @CRLF)

    Do
        Sleep(500)
        _GUICtrlEdit_AppendText($UPDATE, "bytes downloaded :" & InetGetInfo($updateDownload, 0) & @CRLF)
        GUICtrlSetData($uProgress1, (100 * InetGetInfo($updateDownload, 0) / InetGetInfo($updateDownload, 1)))
    Until InetGetInfo($updateDownload, 2) ; Check if the download is complete.

    InetClose($updateDownload) ; Close the handle to release resources.
    _GUICtrlEdit_AppendText($UPDATE, "downloaded " & $nomefiledown & @CRLF)
EndFunc   ;==>Inetdown

Func end()
    Run("explorer.exe " & $downpath1)
    Run("explorer.exe " & $downpath2)
    Run("explorer.exe " & $downpath3)
    MsgBox(64, "NSC GET DOWN", "Files downloaded!")
EndFunc   ;==>end

Share this post


Link to post
Share on other sites
wakillon

 

Thank you for the snippet !

I Like this method, it's faster and more efficient (like a lot of things without IE....)

So I have modified my example to try three methods, with the evidence of timings.

Third method ? Your snippet + >this.

 

When it's possible to avoid using IE, so do it !

Glad to help you !  :)


AutoIt 3.3.14.2 X86 - SciTE 3.6.0WIN 8.1 X64 - Other Example Scripts

Share this post


Link to post
Share on other sites
t0nZ

Surely, but the snippet for downloading from temporary links it would prove useful... soon or later...

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Similar Content

    • careca
      By careca
      Hi, i feel im close to getting it to work, but there must be something im missing that's probably easy for you experts.
      #include <IE.au3> #include <File.au3> #include <MsgBoxConstants.au3> $Visible = 1 Local $oIE $oIE = _IECreate("https://beemp3s.net/mp3-download.html", 0, $Visible, 1, 1) _IELoadWait($oIE) If $oIE <> @error Then ;_ArrayText() _QuerySite('Finding Hope - Without You') ;_IEQuit($oIE) ;Exit Else MsgBox(64 + 262144, 'Fail', 'Could not create a new window') EndIf ;============================================================================= Func _QuerySite($Name) Local $oIE_ObjectToSearch = _IEGetObjById($oIE, 'query') _IEDocInsertText($oIE_ObjectToSearch, $Name) Local $oIE_ObjectToClick = _IEGetObjById($oIE, "search-action") _IEAction($oIE_ObjectToClick, 'click') _IELoadWait($oIE, 2000) ;============================================================================= Local $oSubmitClick = _IEGetObjById($oIE, "col-md-12") _IEAction($oSubmitClick, "click") _IELoadWait($oIE, 2000) ;============================================================================= Local $oBtns = _IETagNameGetCollection($oIE, "div") For $oBtn In $oBtns If String($oBtn.classname) = "col-md-12" Then If String($oBtn.innertext) = "Download / Play" Then ConsoleWrite($oBtn.innertext & @CRLF) _IEAction($oBtn, 'click') EndIf EndIf Next ;============================================================================= ;Local $oElements = _IETagNameAllGetCollection($oIE) ;For $oElement In $oElements ; If $oElement.id Then MsgBox($MB_SYSTEMMODAL, "Element Info", "Tagname: " & $oElement.tagname & @CRLF & "id: " & $oElement.id & @CRLF & "innerText: " & $oElement.innerText) ;Next ;============================================================================= ;_IEQuit($oIE) ;Exit ;============================================================================= EndFunc ;==>_QuerySite ;============================================================================= Func _ArrayText() If FileExists('Titles.txt') Then Local $FO = FileOpen('Titles.txt') Local $R2A = FileReadToArray($FO) Local $iLineCount = @extended FileClose($FO) For $f = 0 To $iLineCount - 1 ConsoleWrite($f & ' - ' & $R2A[$f] & @CRLF) _QuerySite($R2A[$f]) Next Else MsgBox(64 + 262144, 'Fail', 'Could not find the file Titles.txt') EndIf EndFunc ;==>_ArrayText ;============================================================================= The idea is to get into this site, write into the search and click, which i did, but now the problem is clicking in any of the buttons in the results page, im looking for a way to click the first one, it always matches to what i want, so no need to fiddle around.
      Site
      Thanks for help in advance.

    • yoschga
      By yoschga
      is there a possibility to let ie.au3 click into a form? i use on a project where i need to click into a form where items are loaded just writing it in or fill the form is no option
       

    • RamaKrishna9
      By RamaKrishna9
      What are the minimum system requirements(like Windows OS version, IE browser version, etc,.) for IE.au3(3.3.14.2) to work properly?
    • Rskm
      By Rskm
      Hi, for a script of mine, I wish to keep a time frame on till how the program will work.. basically, I want the program to read the date and if the read date < 'a particular date', the program continues, else it exit. I don't want to read the time from the computer as the program can be fooled by resetting the pc time. is there a way to read time from internet - so that the user cannot fool the program by date reset? thanks
    • nacerbaaziz
      By nacerbaaziz
      Hello
      Can we pause and resume the download in the InetGet function?
      If is possible, what is the solution please?
      I used this code To manage the download

      #include <INet.au3> func _downloader($name, $linc, $filepath, $RTLF = false, $link = false) global $downloader = GUICreate("downloader", 400, 200, -1, -1, $WS_CLIPCHILDREn, $RTLF, $link) global $path = $filePath $labelTxt = GUICtrlCreateLabel("downloading " & $name, 50, 10, 200, 20) global $labelTxt0 = GUICtrlCreateLabel("downloaded size 0 MB " & "OF 0 MB", 50, 60, 300, 20) global $Progress = "" global $sText = ""     For $i = 1 To Random(5, 20, 1) ; Return an integer between 5 and 20 to determine the length of the string.         $sText &= Chr(Random(65, 122, 1)) ; Return an integer between 65 and 122 which represent the ASCII characters between a (lower-case) to Z (upper-case). next global $labelTxt2 = GUICtrlCreateInput("0%", 50, 80, 50, 20) _GUICtrlEdit_SetReadOnly(-1, true) GUIStartGroup("") global $beep = GUICtrlCreateCheckBox("use the progress beep notification", 150, 120, 200, 20) GUIStartGroup("") $button = GUICtrlCreateButton("Cancel', 130, 150, 180, 25, 0x01) $iIndex = 0 global $Target global $url GUIStartGroup("") global $Progress = GUICtrlCreateProgress(50, 90, 150, 20) global $Target = $filepath global $url = $linc global $path = $filepath global $hDownloadNo = _RSMWare_GetData($url, $Target) global $status = false AdlibRegister("SetProgress") global $onprogress = false, $curent = false GUISetState(@sw_Show) While 1 Switch GUIGetMsg() Case $GUI_EVENT_CLOSE, $button $asc = MsgBox(4132,"exit download?","if you click yes the downloading will be cancel, do you want to cancel it ?") if $asc = 6 then AdlibUnRegister("SetProgress") GUIDelete() If $hDownloadNo <> 0 Then InetClose($hDownloadNo) exitLoop endIf EndSwitch if $status = -1 then $status = 0 $hDownloadNo = _RSMWare_GetData($url, $Target) $onprogress = false $curent = false elseIf $Status = 1 then $status = $path GUIDelete() AdlibUnRegister("SetProgress") exitLoop endIf WEnd return $status endFunc Func _RSMWare_GetData($url, $Target) Local $hDownload = InetGet($url, $Target, 1, 1) Return $hDownload EndFunc ;==>_RSMWare_GetData Func SetProgress() Local $state If $hDownloadNo <> 0 Then $state = InetGetInfo($hDownloadNo) If @error = 0 Then $infor = "downloaded size " & Round(Execute(InetGetInfo($hDownloadNo, $INET_DOWNLOADREAD) / 1048576), 2) & " MB of " & Round(Execute(InetGetInfo($hDownloadNo, $INET_DOWNLOADSIZE) / 1048576), 2) & " MB " $onprogress = Round(Ceiling(($state[0] / $state[1]) * 100)) if not (InetGetInfo($hDownloadNo, $INET_DOWNLOADSIZE) = 0) then if $onProgress <= 0 then $onProgress = 0 GUICtrlSetData($Progress, $onProgress) GUICtrlSetData($labelTxt0, $infor) GUICtrlSetData($labelTxt2, $onProgress & "%") if _isChecked($beep) then if $onprogress > $curent then beep((100 + $onprogress * 20), 100) $curent = $onprogress endIf endIf endIf If $state[2] Then If $state[3] Then InetClose($hDownloadNo) $status = 1 else InetClose($hDownloadNo) $status = -1 endIf endIf EndIf endIf EndFunc ;==>SetProgress
×