Jump to content
Sign in to follow this  
t0nZ

How to download files from temporary links

Recommended Posts

So my problem was to download files from temporary links.

If you consider Combofix, an util to clean Windows from malware, it's a program updated daily and I want always the last version.

But the download link is temporary and always changing, and I don't trust other sites static links.

I developed a system to download a list of links, the main idea is to have the list of links in the page and after I go parsing the list to found the correct temporary links.

It's dirty but it's working.

This is a "lite" version for the forum.:

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_Icon=Icone\Full ico\down.ico
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****

;NSC 2014
; download temporary links demo


#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <ProgressConstants.au3>
#include <WindowsConstants.au3>
#include <Array.au3>
#include <GuiEdit.au3>
#include <IE.au3>
#include <string.au3>

Global $uProgress1, $UPDATE, $updateArray, $updateDownload, $downlist, $adown, $conta
Global $downpath = @DesktopDir & "\GetDown"

;_________________

Gui()
Prepare()
downlist()
downloop()
end()

Exit


Func Gui()

    #Region ### START Koda GUI section ### Form=C:\Program Files (x86)\AutoIt3\Examples\NSC_test\formS\update.kxf
    $UForm1 = GUICreate("GET DOWN lite V.1.5", 300, 200, 200, 150)
    GUISetBkColor(0xC0C0C0)
    $uProgress1 = GUICtrlCreateProgress(265, 5, 30, 190, BitOR($PBS_SMOOTH, $PBS_VERTICAL, $WS_BORDER))
    GUICtrlSetColor(-1, 0x0033FF)
    GUICtrlSetBkColor(-1, 0x000066)
    $UPDATE = GUICtrlCreateEdit("", 5, 5, 250, 190, BitOR($ES_AUTOVSCROLL, $ES_AUTOHSCROLL, $ES_WANTRETURN, $WS_BORDER))
    GUICtrlSetData(-1, "Gettin'down temporary links" & @CRLF)
    GUICtrlSetFont(-1, 8, 800, 0, "Verdana")
    GUICtrlSetColor(-1, 0x99FFFF)
    GUICtrlSetBkColor(-1, 0x330000)
    GUICtrlSetCursor(-1, 3)
    GUISetState(@SW_SHOW)
    #EndRegion ### END Koda GUI section ###

EndFunc   ;==>Gui

Func Prepare()
    If Not FileExists($downpath) Then DirCreate($downpath)
EndFunc   ;==>Prepare

Func downlist()
    Dim $adown[3] ; in other scripts I download a txt file from internet.
    $adown[0] = "http://www.bleepingcomputer.com/download/adwcleaner/dl/125/|adwcleaner.exe"
    $adown[1] = "http://www.bleepingcomputer.com/download/combofix/dl/12/|combofix.exe"
    $adown[2] = "http://www.bleepingcomputer.com/download/rkill/dl/10/|rkill.exe"
    _GUICtrlEdit_AppendText($UPDATE, "download list assigned. " & @CRLF)
EndFunc   ;==>downlist



Func downloop()

    While 1
        $filedown = _ArrayPop($adown)
        If @error <> 0 Then ExitLoop

        $arraydown = _StringExplode($filedown, "|") ; example of line processed "www.yourlinkhere.com|yourfile.exe"

        Local $oIE = _IECreate($arraydown[0], 0, 0, 1, 0) ;hidden IE
        Local $oLinks = _IELinkGetCollection($oIE) ; list of links!
        Local $iNumLinks = @extended
        _GUICtrlEdit_AppendText($UPDATE, "Searching in " & $iNumLinks & " links for " & $arraydown[0] & @CRLF)
        Local $sTxt = $iNumLinks & " links found" & @CRLF & @CRLF ; uncomment for list of links (1/3)
        For $oLink In $oLinks
            $sTxt &= $oLink.href & @CRLF ;uncomment for list of links (2/3)
            If StringInStr($oLink.href, $arraydown[1]) Then
                Inetdown($oLink.href, $arraydown[1])
            EndIf
        Next
        MsgBox($MB_SYSTEMMODAL, "Link Info", $sTxt) ;uncomment for list of links (3/3)
    WEnd

EndFunc   ;==>downloop


Func Inetdown($filedown, $nomefiledown)
    $updateDownload = InetGet($filedown, $downpath & "\" & $nomefiledown, 1, 1)
    _GUICtrlEdit_AppendText($UPDATE, "downloading " & $nomefiledown & @CRLF)

    Do
        Sleep(500)
        _GUICtrlEdit_AppendText($UPDATE, "bytes downloaded :" & InetGetInfo($updateDownload, 0) & @CRLF)
        GUICtrlSetData($uProgress1, (100 * InetGetInfo($updateDownload, 0) / InetGetInfo($updateDownload, 1)))
    Until InetGetInfo($updateDownload, 2) ; Check if the download is complete.

    InetClose($updateDownload) ; Close the handle to release resources.
    _GUICtrlEdit_AppendText($UPDATE, "downloaded in " & $downpath & @CRLF)
EndFunc   ;==>Inetdown

Func end()
    Run("explorer.exe " & $downpath)
    MsgBox(64, "NSC GET DOWN", "Files downloaded!")
EndFunc   ;==>end

the Function downloop() contains the trick.

The complete version downloads from a public server the list of the files to download, and for every file there is a command to choose various download methods and  to exucute commands (run the files etc) after the download, and at the end I have a mail with a report.

Edited by t0nZ

Share this post


Link to post
Share on other sites

Why use IE ?

You have just to extract links from source and find the one you want (temporary or not)

#include <Array.au3>

$sSrc = BinaryToString ( InetRead ( 'http://www.bleepingcomputer.com/download/combofix/' ) )
$aLinks = StringRegExp ( $sSrc, '(?s)(?i)<a href="(.*?)"', 3 )
$iIndex = _ArraySearch ( $aLinks, 'download/combofix', 0, 0, 0, 1 )
ConsoleWrite ( '+ Link Download Page : ' & $aLinks[$iIndex] & @Crlf )

$sSrc = StringReplace ( BinaryToString ( InetRead ( $aLinks[$iIndex] ) ), "'", '"' )
$aLink = StringRegExp ( $sSrc, '(?s)(?i)<a href="(.*?)"', 3 )
$iIndex = _ArraySearch ( $aLink, 'ComboFix.exe', 0, 0, 0, 1 )
ConsoleWrite ( '! Link Download ComboFix.exe : ' & $aLink[$iIndex] & @Crlf )

AutoIt 3.3.14.2 X86 - SciTE 3.6.0WIN 8.1 X64 - Other Example Scripts

Share this post


Link to post
Share on other sites

Thank you for the snippet !

I Like this method, it's faster and more efficient (like a lot of things without IE....)

So I have modified my example to try three methods, with the evidence of timings.

Third method ? Your snippet + >this.

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_Icon=Icone\Full ico\down.ico
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****

;NSC 2014
; download temporary links demo


#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <ProgressConstants.au3>
#include <WindowsConstants.au3>
#include <Array.au3>
#include <GuiEdit.au3>
#include <IE.au3>
#include <string.au3>
#include "UrlDownloadEx.au3"

Global $uProgress1, $UPDATE, $updateArray, $updateDownload, $downlist, $adown, $conta
Global $downpath1 = @DesktopDir & "\GetDown1"
Global $downpath2 = @DesktopDir & "\GetDown2"
Global $downpath3 = @DesktopDir & "\GetDown3"

;_________________

Gui()
Prepare()
downlist()
downloop1()
downlist()
downloop2()
downlist()
downloop3()
end()

Exit


Func Gui()

    #Region ### START Koda GUI section ### Form=C:\Program Files (x86)\AutoIt3\Examples\NSC_test\formS\update.kxf
    $UForm1 = GUICreate("GET DOWN lite V.1.7", 300, 200, 200, 150)
    GUISetBkColor(0xC0C0C0)
    $uProgress1 = GUICtrlCreateProgress(265, 5, 30, 190, BitOR($PBS_SMOOTH, $PBS_VERTICAL, $WS_BORDER))
    GUICtrlSetColor(-1, 0x0033FF)
    GUICtrlSetBkColor(-1, 0x000066)
    $UPDATE = GUICtrlCreateEdit("", 5, 5, 250, 190, BitOR($ES_AUTOVSCROLL, $ES_AUTOHSCROLL, $ES_WANTRETURN, $WS_BORDER))
    GUICtrlSetData(-1, "Gettin'down temporary links" & @CRLF)
    GUICtrlSetFont(-1, 8, 800, 0, "Verdana")
    GUICtrlSetColor(-1, 0x99FFFF)
    GUICtrlSetBkColor(-1, 0x330000)
    GUICtrlSetCursor(-1, 3)
    GUISetState(@SW_SHOW)
    #EndRegion ### END Koda GUI section ###

EndFunc   ;==>Gui

Func Prepare()
    If Not FileExists($downpath1) Then DirCreate($downpath1)
    If Not FileExists($downpath2) Then DirCreate($downpath2)
    If Not FileExists($downpath3) Then DirCreate($downpath3)
EndFunc   ;==>Prepare

Func downlist()
    Dim $adown[3] ; in other scripts I download a txt file from internet.
    $adown[0] = "http://www.bleepingcomputer.com/download/adwcleaner/dl/125/|adwcleaner.exe"
    $adown[1] = "http://www.bleepingcomputer.com/download/combofix/dl/12/|combofix.exe"
    $adown[2] = "http://www.bleepingcomputer.com/download/rkill/dl/10/|rkill.exe"
    _GUICtrlEdit_AppendText($UPDATE, "download list assigned. " & @CRLF)
EndFunc   ;==>downlist

Func downloop1()
    MsgBox(64, "getdown1", "start downloading IE mode")
    $iBegin = TimerInit()
    While 1

        $filedown = _ArrayPop($adown)
        If @error <> 0 Then ExitLoop

        $arraydown = _StringExplode($filedown, "|") ; example of line processed "www.yourlinkhere.com|yourfile.exe"

        ;;;___________________
        Local $oIE = _IECreate($arraydown[0], 0, 0, 1, 0) ;hidden IE
        Local $oLinks = _IELinkGetCollection($oIE) ; list of links!
        Local $iNumLinks = @extended
        _GUICtrlEdit_AppendText($UPDATE, "Searching in " & $iNumLinks & " links for " & $arraydown[0] & @CRLF)
        ;Local $sTxt = $iNumLinks & " links found" & @CRLF & @CRLF ; uncomment for list of links (1/3)
        For $oLink In $oLinks
            ;   $sTxt &= $oLink.href & @CRLF ;uncomment for list of links (2/3)
            If StringInStr($oLink.href, $arraydown[1]) Then
                Inetdown($oLink.href, $downpath1 & "\" & $arraydown[1])
            EndIf
        Next
        ;MsgBox($MB_SYSTEMMODAL, "Link Info", $sTxt) ;uncomment for list of links (3/3)
    WEnd
    $iDiff = Int(TimerDiff($iBegin) / 1000)
    $iMin = Int($iDiff / 60)
    $iSec = $iDiff - ($iMin * 60)
    MsgBox(64, "Downloaded in ", $iMin & ":" & StringFormat("%02i", $iSec))

EndFunc   ;==>downloop1


Func downloop2()
    MsgBox(64, "getdown2", "start downloading INETGET mode")
    $iBegin = TimerInit()
    While 1
        $filedown = _ArrayPop($adown)
        If @error <> 0 Then ExitLoop

        $arraydown = _StringExplode($filedown, "|") ; example of line processed "www.yourlinkhere.com|yourfile.exe"


        $sSrc = StringReplace(BinaryToString(InetRead($arraydown[0])), "'", '"')
        $aLink = StringRegExp($sSrc, '(?s)(?i)<a href="(.*?)"', 3)
        $iIndex = _ArraySearch($aLink, $arraydown[1], 0, 0, 0, 1)
        ConsoleWrite('! Link Download  ' & $aLink[$iIndex] & @CRLF)
        Inetdown($aLink[$iIndex], $downpath2 & "\" & $arraydown[1])
    WEnd
    $iDiff = Int(TimerDiff($iBegin) / 1000)
    $iMin = Int($iDiff / 60)
    $iSec = $iDiff - ($iMin * 60)

    MsgBox(64, "Downloaded in ", $iMin & ":" & StringFormat("%02i", $iSec))
EndFunc   ;==>downloop2


Func downloop3()
    MsgBox(64, "getdown3", "start downloading INETGET mode + UrlDownloadEX")
    $iBegin = TimerInit()
    While 1
        $filedown = _ArrayPop($adown)
        If @error <> 0 Then ExitLoop

        $arraydown = _StringExplode($filedown, "|") ; example of line processed "www.yourlinkhere.com|yourfile.exe"


        $sSrc = StringReplace(BinaryToString(InetRead($arraydown[0])), "'", '"')
        $aLink = StringRegExp($sSrc, '(?s)(?i)<a href="(.*?)"', 3)
        $iIndex = _ArraySearch($aLink, $arraydown[1], 0, 0, 0, 1)
        ConsoleWrite('! Link Download ' & $aLink[$iIndex] & @CRLF)
        UrlDownloadEx($aLink[$iIndex], $downpath3 & "\" & $arraydown[1])
    WEnd
    $iDiff = Int(TimerDiff($iBegin) / 1000)
    $iMin = Int($iDiff / 60)
    $iSec = $iDiff - ($iMin * 60)

    MsgBox(64, "Downloaded in ", $iMin & ":" & StringFormat("%02i", $iSec))
EndFunc   ;==>downloop3

Func Inetdown($filedown, $nomefiledown)
    $updateDownload = InetGet($filedown, $nomefiledown, 1, 1)
    _GUICtrlEdit_AppendText($UPDATE, "downloading " & $nomefiledown & @CRLF)

    Do
        Sleep(500)
        _GUICtrlEdit_AppendText($UPDATE, "bytes downloaded :" & InetGetInfo($updateDownload, 0) & @CRLF)
        GUICtrlSetData($uProgress1, (100 * InetGetInfo($updateDownload, 0) / InetGetInfo($updateDownload, 1)))
    Until InetGetInfo($updateDownload, 2) ; Check if the download is complete.

    InetClose($updateDownload) ; Close the handle to release resources.
    _GUICtrlEdit_AppendText($UPDATE, "downloaded " & $nomefiledown & @CRLF)
EndFunc   ;==>Inetdown

Func end()
    Run("explorer.exe " & $downpath1)
    Run("explorer.exe " & $downpath2)
    Run("explorer.exe " & $downpath3)
    MsgBox(64, "NSC GET DOWN", "Files downloaded!")
EndFunc   ;==>end

Share this post


Link to post
Share on other sites

 

Thank you for the snippet !

I Like this method, it's faster and more efficient (like a lot of things without IE....)

So I have modified my example to try three methods, with the evidence of timings.

Third method ? Your snippet + >this.

 

When it's possible to avoid using IE, so do it !

Glad to help you !  :)


AutoIt 3.3.14.2 X86 - SciTE 3.6.0WIN 8.1 X64 - Other Example Scripts

Share this post


Link to post
Share on other sites

Surely, but the snippet for downloading from temporary links it would prove useful... soon or later...

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By UEZ
      This project has been discontinued!
       
      Here a small tool I wrote to update my Sysinternal tools collection without the need to download always the whole package or visiting the site to check for updates. I know that there are several tools available (also some tools written in AutoIt) but here another one for the collection. It was good exercise for me to code it.
       
       
        
       
       
      Some files from cannot be downloaded although they are visible on the web site!
       
      Here the download link of the source code only: AutoIt Sysinternal Tools Synchronizer v0.99.6 build 2020-09-23 beta.7z  (1557 downloads previously)
      -=> Requires AutoIt version 3.3.13.20 or higher / tested on Win8.1 real machine and some VMs: Win7 / Vista / Win10
       
      Compiled exe only: @MediaFire
       
      Just select the Sysinternal Tools folder or create one and press the synchronize button to download the selected items. Click on AutoIt label (near to left upper corner) to open menu.
       
      Special thanks to LarsJ, Melba23 and mesale0077 for their help. 
       
      I've still some ideas to implement which are more gimmick related, so it is not finished yet...
      If you want to add your language please check out #Region Language. Thanks. 
       
      Please report any bug or if you have any suggestions.
       
      The language of the tool tip from each of the executable in the left list view were automatically created using Google translator and weren't checked for correctness.
       
      Br,
      UEZ
    • By walec
      Hey
      I have problem with Set Value for control slider on webside.
      Anyone have any idea for a solution?
       
      I tried but it doesn't work.
      Local $Strength = _IEGetObjById($oIE_Strona, "strength_value") Local $oText = _IEFormElementGetObjByName($Strength, "valuee") _IEFormElementSetValue($oText, "6") or
      _IENavigate ($oIE_Strona, "https://guantanamo.prisonwarsonline.com/gym/") Local $oDiv = $oIE_Strona.Document.GetElementByID("strength_value") Local $oAttribute = $oDiv.getAttribute("valuee") If IsObj ( $oAttribute ) Then $oAttribute.value = "6" EndIf Best Regards
    • By mdwerne
      Hello,
      Before Broadcom took over Symantec, I was able to use the following code as the base to scrape daily definition downloads from the web. Since the pages were moved over to the Broadcom web servers, I get a page full of what I believe may be JavaScript instead of the fully rendered page that lies behind it. Does anyone have any suggestions as to how I can read the full rendered webpage using AutoIt?
      #include <IE.au3> $oIE_DEFS = _IECreate("https://www.broadcom.com/support/security-center/definitions/download/detail?gid=sep14", 0, 0, 1, 1) $sString_DEFS = _IEBodyReadHTML($oIE_DEFS) MsgBox(0, "https://www.broadcom.com/support/security-center/definitions/download/detail?gid=sep14", $sString_DEFS) The above code will show the JavaScript, but if you go to the URL in a browser that has JavaScript enabled, you will see the fully rendered page that I would like to access. I hope my question makes sense, and I would appreciate any suggestions to get this working again.
      All the best,
      -Mike
    • By Mariog2000
      I've tried for a few days to simply open a url in IE and click an IE object that I can't "TAB" to and don't want to use a mouse-click if at all possible.  I've read up on things like using IEAction and IETagNameGetCollection to find the object but not smart enough to figure out how that works with my url or how to tell if my url is "basic" or a "form", "frameset", "iframe", etc.
      The other problem is I've tried using the "Run" and "ShellExecute" commands to open the browser, which works, but it opens it in a way that doesn't include my currently logged in user account.  For example, if I just click IE or Chrome for that matter and open my default homepage of google, the page that opens knows I'm logged in.  When I use "Send" "ShellExecute" or "_IE_Create" they always open pages generically which would require a login.
      So questions are, if I have a URL I'm trying to open, that's passes the currently logged on users credentials when opening the page (no clue where that happens)
      1.  Are there any code examples someone is willing to share that opens my IE maximized, with logged in credentials, and
      2.  Once that works, can someone suggest a technique a non-developer can find which IE Objects I need to send a "click" to so it opens the correct menu, pop-up, url, etc., associated with that object.  I've copied what doesn't work and sure I'm way off but it's not for a lack of effort. I'll keep trying and post updates but not looking good 🙂
      #include <IE.au3>
      Call ("selection")
      Func selection ()
      Global $oIE = _IECreate ("https://internal-webpage.aspx")
      Local $selectid = _IEGetObjByName ($oIE, "signageNo")
      Local $button = _IEGetObjByName ($oIE, "GO")

      _IEFormElementOptionSelect ($selectid, "3")
      _IEAction ($button,"click")
      EndFunc
      #include <IE.au3> Call ("selection") Func selection () Global $oIE = _IECreate ("https://internal-webpage.aspx") Local $selectid = _IEGetObjByName ($oIE, "signageNo") Local $button = _IEGetObjByName ($oIE, "GO") _IEFormElementOptionSelect ($selectid, "3") _IEAction ($button,"click") EndFunc 1. 
    • By zuladabef
      I want to set the value of a input, but I only have the input name and input id.  I cannot find a form in the source.  Form collection does not return any results.  
      Is there a way to use FormElementSetValue without a form name? Can I use _IEGetObjByName or _IEGetObjById?  If so, how do I set the value once I have the object? Where do I go from here? Local $oIE = _IEAttach("[Website Goes Here"]) Local $oForm = _IEFormGetCollection($oIE, 1) Local $oQuery = _IEFormElementGetCollection($oForm, 4) _IEFormElementSetValue($oQuery, "Set it to this")  
×
×
  • Create New...