Sign in to follow this  
Followers 0
t0nZ

How to download files from temporary links

6 posts in this topic

#1 ·  Posted (edited)

So my problem was to download files from temporary links.

If you consider Combofix, an util to clean Windows from malware, it's a program updated daily and I want always the last version.

But the download link is temporary and always changing, and I don't trust other sites static links.

I developed a system to download a list of links, the main idea is to have the list of links in the page and after I go parsing the list to found the correct temporary links.

It's dirty but it's working.

This is a "lite" version for the forum.:

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_Icon=Icone\Full ico\down.ico
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****

;NSC 2014
; download temporary links demo


#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <ProgressConstants.au3>
#include <WindowsConstants.au3>
#include <Array.au3>
#include <GuiEdit.au3>
#include <IE.au3>
#include <string.au3>

Global $uProgress1, $UPDATE, $updateArray, $updateDownload, $downlist, $adown, $conta
Global $downpath = @DesktopDir & "\GetDown"

;_________________

Gui()
Prepare()
downlist()
downloop()
end()

Exit


Func Gui()

    #Region ### START Koda GUI section ### Form=C:\Program Files (x86)\AutoIt3\Examples\NSC_test\formS\update.kxf
    $UForm1 = GUICreate("GET DOWN lite V.1.5", 300, 200, 200, 150)
    GUISetBkColor(0xC0C0C0)
    $uProgress1 = GUICtrlCreateProgress(265, 5, 30, 190, BitOR($PBS_SMOOTH, $PBS_VERTICAL, $WS_BORDER))
    GUICtrlSetColor(-1, 0x0033FF)
    GUICtrlSetBkColor(-1, 0x000066)
    $UPDATE = GUICtrlCreateEdit("", 5, 5, 250, 190, BitOR($ES_AUTOVSCROLL, $ES_AUTOHSCROLL, $ES_WANTRETURN, $WS_BORDER))
    GUICtrlSetData(-1, "Gettin'down temporary links" & @CRLF)
    GUICtrlSetFont(-1, 8, 800, 0, "Verdana")
    GUICtrlSetColor(-1, 0x99FFFF)
    GUICtrlSetBkColor(-1, 0x330000)
    GUICtrlSetCursor(-1, 3)
    GUISetState(@SW_SHOW)
    #EndRegion ### END Koda GUI section ###

EndFunc   ;==>Gui

Func Prepare()
    If Not FileExists($downpath) Then DirCreate($downpath)
EndFunc   ;==>Prepare

Func downlist()
    Dim $adown[3] ; in other scripts I download a txt file from internet.
    $adown[0] = "http://www.bleepingcomputer.com/download/adwcleaner/dl/125/|adwcleaner.exe"
    $adown[1] = "http://www.bleepingcomputer.com/download/combofix/dl/12/|combofix.exe"
    $adown[2] = "http://www.bleepingcomputer.com/download/rkill/dl/10/|rkill.exe"
    _GUICtrlEdit_AppendText($UPDATE, "download list assigned. " & @CRLF)
EndFunc   ;==>downlist



Func downloop()

    While 1
        $filedown = _ArrayPop($adown)
        If @error <> 0 Then ExitLoop

        $arraydown = _StringExplode($filedown, "|") ; example of line processed "www.yourlinkhere.com|yourfile.exe"

        Local $oIE = _IECreate($arraydown[0], 0, 0, 1, 0) ;hidden IE
        Local $oLinks = _IELinkGetCollection($oIE) ; list of links!
        Local $iNumLinks = @extended
        _GUICtrlEdit_AppendText($UPDATE, "Searching in " & $iNumLinks & " links for " & $arraydown[0] & @CRLF)
        Local $sTxt = $iNumLinks & " links found" & @CRLF & @CRLF ; uncomment for list of links (1/3)
        For $oLink In $oLinks
            $sTxt &= $oLink.href & @CRLF ;uncomment for list of links (2/3)
            If StringInStr($oLink.href, $arraydown[1]) Then
                Inetdown($oLink.href, $arraydown[1])
            EndIf
        Next
        MsgBox($MB_SYSTEMMODAL, "Link Info", $sTxt) ;uncomment for list of links (3/3)
    WEnd

EndFunc   ;==>downloop


Func Inetdown($filedown, $nomefiledown)
    $updateDownload = InetGet($filedown, $downpath & "\" & $nomefiledown, 1, 1)
    _GUICtrlEdit_AppendText($UPDATE, "downloading " & $nomefiledown & @CRLF)

    Do
        Sleep(500)
        _GUICtrlEdit_AppendText($UPDATE, "bytes downloaded :" & InetGetInfo($updateDownload, 0) & @CRLF)
        GUICtrlSetData($uProgress1, (100 * InetGetInfo($updateDownload, 0) / InetGetInfo($updateDownload, 1)))
    Until InetGetInfo($updateDownload, 2) ; Check if the download is complete.

    InetClose($updateDownload) ; Close the handle to release resources.
    _GUICtrlEdit_AppendText($UPDATE, "downloaded in " & $downpath & @CRLF)
EndFunc   ;==>Inetdown

Func end()
    Run("explorer.exe " & $downpath)
    MsgBox(64, "NSC GET DOWN", "Files downloaded!")
EndFunc   ;==>end

the Function downloop() contains the trick.

The complete version downloads from a public server the list of the files to download, and for every file there is a command to choose various download methods and  to exucute commands (run the files etc) after the download, and at the end I have a mail with a report.

Edited by t0nZ

Share this post


Link to post
Share on other sites

Why use IE ?

You have just to extract links from source and find the one you want (temporary or not)

#include <Array.au3>

$sSrc = BinaryToString ( InetRead ( 'http://www.bleepingcomputer.com/download/combofix/' ) )
$aLinks = StringRegExp ( $sSrc, '(?s)(?i)<a href="(.*?)"', 3 )
$iIndex = _ArraySearch ( $aLinks, 'download/combofix', 0, 0, 0, 1 )
ConsoleWrite ( '+ Link Download Page : ' & $aLinks[$iIndex] & @Crlf )

$sSrc = StringReplace ( BinaryToString ( InetRead ( $aLinks[$iIndex] ) ), "'", '"' )
$aLink = StringRegExp ( $sSrc, '(?s)(?i)<a href="(.*?)"', 3 )
$iIndex = _ArraySearch ( $aLink, 'ComboFix.exe', 0, 0, 0, 1 )
ConsoleWrite ( '! Link Download ComboFix.exe : ' & $aLink[$iIndex] & @Crlf )
1 person likes this

AutoIt 3.3.14.2 X86 - SciTE 3.6.0WIN 8.1 X64 - Other Example Scripts

Share this post


Link to post
Share on other sites

Thank you for the snippet !

I Like this method, it's faster and more efficient (like a lot of things without IE....)

So I have modified my example to try three methods, with the evidence of timings.

Third method ? Your snippet + >this.

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_Icon=Icone\Full ico\down.ico
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****

;NSC 2014
; download temporary links demo


#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <ProgressConstants.au3>
#include <WindowsConstants.au3>
#include <Array.au3>
#include <GuiEdit.au3>
#include <IE.au3>
#include <string.au3>
#include "UrlDownloadEx.au3"

Global $uProgress1, $UPDATE, $updateArray, $updateDownload, $downlist, $adown, $conta
Global $downpath1 = @DesktopDir & "\GetDown1"
Global $downpath2 = @DesktopDir & "\GetDown2"
Global $downpath3 = @DesktopDir & "\GetDown3"

;_________________

Gui()
Prepare()
downlist()
downloop1()
downlist()
downloop2()
downlist()
downloop3()
end()

Exit


Func Gui()

    #Region ### START Koda GUI section ### Form=C:\Program Files (x86)\AutoIt3\Examples\NSC_test\formS\update.kxf
    $UForm1 = GUICreate("GET DOWN lite V.1.7", 300, 200, 200, 150)
    GUISetBkColor(0xC0C0C0)
    $uProgress1 = GUICtrlCreateProgress(265, 5, 30, 190, BitOR($PBS_SMOOTH, $PBS_VERTICAL, $WS_BORDER))
    GUICtrlSetColor(-1, 0x0033FF)
    GUICtrlSetBkColor(-1, 0x000066)
    $UPDATE = GUICtrlCreateEdit("", 5, 5, 250, 190, BitOR($ES_AUTOVSCROLL, $ES_AUTOHSCROLL, $ES_WANTRETURN, $WS_BORDER))
    GUICtrlSetData(-1, "Gettin'down temporary links" & @CRLF)
    GUICtrlSetFont(-1, 8, 800, 0, "Verdana")
    GUICtrlSetColor(-1, 0x99FFFF)
    GUICtrlSetBkColor(-1, 0x330000)
    GUICtrlSetCursor(-1, 3)
    GUISetState(@SW_SHOW)
    #EndRegion ### END Koda GUI section ###

EndFunc   ;==>Gui

Func Prepare()
    If Not FileExists($downpath1) Then DirCreate($downpath1)
    If Not FileExists($downpath2) Then DirCreate($downpath2)
    If Not FileExists($downpath3) Then DirCreate($downpath3)
EndFunc   ;==>Prepare

Func downlist()
    Dim $adown[3] ; in other scripts I download a txt file from internet.
    $adown[0] = "http://www.bleepingcomputer.com/download/adwcleaner/dl/125/|adwcleaner.exe"
    $adown[1] = "http://www.bleepingcomputer.com/download/combofix/dl/12/|combofix.exe"
    $adown[2] = "http://www.bleepingcomputer.com/download/rkill/dl/10/|rkill.exe"
    _GUICtrlEdit_AppendText($UPDATE, "download list assigned. " & @CRLF)
EndFunc   ;==>downlist

Func downloop1()
    MsgBox(64, "getdown1", "start downloading IE mode")
    $iBegin = TimerInit()
    While 1

        $filedown = _ArrayPop($adown)
        If @error <> 0 Then ExitLoop

        $arraydown = _StringExplode($filedown, "|") ; example of line processed "www.yourlinkhere.com|yourfile.exe"

        ;;;___________________
        Local $oIE = _IECreate($arraydown[0], 0, 0, 1, 0) ;hidden IE
        Local $oLinks = _IELinkGetCollection($oIE) ; list of links!
        Local $iNumLinks = @extended
        _GUICtrlEdit_AppendText($UPDATE, "Searching in " & $iNumLinks & " links for " & $arraydown[0] & @CRLF)
        ;Local $sTxt = $iNumLinks & " links found" & @CRLF & @CRLF ; uncomment for list of links (1/3)
        For $oLink In $oLinks
            ;   $sTxt &= $oLink.href & @CRLF ;uncomment for list of links (2/3)
            If StringInStr($oLink.href, $arraydown[1]) Then
                Inetdown($oLink.href, $downpath1 & "\" & $arraydown[1])
            EndIf
        Next
        ;MsgBox($MB_SYSTEMMODAL, "Link Info", $sTxt) ;uncomment for list of links (3/3)
    WEnd
    $iDiff = Int(TimerDiff($iBegin) / 1000)
    $iMin = Int($iDiff / 60)
    $iSec = $iDiff - ($iMin * 60)
    MsgBox(64, "Downloaded in ", $iMin & ":" & StringFormat("%02i", $iSec))

EndFunc   ;==>downloop1


Func downloop2()
    MsgBox(64, "getdown2", "start downloading INETGET mode")
    $iBegin = TimerInit()
    While 1
        $filedown = _ArrayPop($adown)
        If @error <> 0 Then ExitLoop

        $arraydown = _StringExplode($filedown, "|") ; example of line processed "www.yourlinkhere.com|yourfile.exe"


        $sSrc = StringReplace(BinaryToString(InetRead($arraydown[0])), "'", '"')
        $aLink = StringRegExp($sSrc, '(?s)(?i)<a href="(.*?)"', 3)
        $iIndex = _ArraySearch($aLink, $arraydown[1], 0, 0, 0, 1)
        ConsoleWrite('! Link Download  ' & $aLink[$iIndex] & @CRLF)
        Inetdown($aLink[$iIndex], $downpath2 & "\" & $arraydown[1])
    WEnd
    $iDiff = Int(TimerDiff($iBegin) / 1000)
    $iMin = Int($iDiff / 60)
    $iSec = $iDiff - ($iMin * 60)

    MsgBox(64, "Downloaded in ", $iMin & ":" & StringFormat("%02i", $iSec))
EndFunc   ;==>downloop2


Func downloop3()
    MsgBox(64, "getdown3", "start downloading INETGET mode + UrlDownloadEX")
    $iBegin = TimerInit()
    While 1
        $filedown = _ArrayPop($adown)
        If @error <> 0 Then ExitLoop

        $arraydown = _StringExplode($filedown, "|") ; example of line processed "www.yourlinkhere.com|yourfile.exe"


        $sSrc = StringReplace(BinaryToString(InetRead($arraydown[0])), "'", '"')
        $aLink = StringRegExp($sSrc, '(?s)(?i)<a href="(.*?)"', 3)
        $iIndex = _ArraySearch($aLink, $arraydown[1], 0, 0, 0, 1)
        ConsoleWrite('! Link Download ' & $aLink[$iIndex] & @CRLF)
        UrlDownloadEx($aLink[$iIndex], $downpath3 & "\" & $arraydown[1])
    WEnd
    $iDiff = Int(TimerDiff($iBegin) / 1000)
    $iMin = Int($iDiff / 60)
    $iSec = $iDiff - ($iMin * 60)

    MsgBox(64, "Downloaded in ", $iMin & ":" & StringFormat("%02i", $iSec))
EndFunc   ;==>downloop3

Func Inetdown($filedown, $nomefiledown)
    $updateDownload = InetGet($filedown, $nomefiledown, 1, 1)
    _GUICtrlEdit_AppendText($UPDATE, "downloading " & $nomefiledown & @CRLF)

    Do
        Sleep(500)
        _GUICtrlEdit_AppendText($UPDATE, "bytes downloaded :" & InetGetInfo($updateDownload, 0) & @CRLF)
        GUICtrlSetData($uProgress1, (100 * InetGetInfo($updateDownload, 0) / InetGetInfo($updateDownload, 1)))
    Until InetGetInfo($updateDownload, 2) ; Check if the download is complete.

    InetClose($updateDownload) ; Close the handle to release resources.
    _GUICtrlEdit_AppendText($UPDATE, "downloaded " & $nomefiledown & @CRLF)
EndFunc   ;==>Inetdown

Func end()
    Run("explorer.exe " & $downpath1)
    Run("explorer.exe " & $downpath2)
    Run("explorer.exe " & $downpath3)
    MsgBox(64, "NSC GET DOWN", "Files downloaded!")
EndFunc   ;==>end

Share this post


Link to post
Share on other sites

 

Thank you for the snippet !

I Like this method, it's faster and more efficient (like a lot of things without IE....)

So I have modified my example to try three methods, with the evidence of timings.

Third method ? Your snippet + >this.

 

When it's possible to avoid using IE, so do it !

Glad to help you !  :)


AutoIt 3.3.14.2 X86 - SciTE 3.6.0WIN 8.1 X64 - Other Example Scripts

Share this post


Link to post
Share on other sites

Surely, but the snippet for downloading from temporary links it would prove useful... soon or later...

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0

  • Similar Content

    • Tippex
      By Tippex
      I have a problem trying to use _IEAttach to recover control after a Windows Explorer reset (e.g. on Windows 7), since all the 
      InternetExplorer.Application objects appear to be destroyed by this event. An example of this is when a PC has some kind of fault causing a pop-up error message "Windows Explorer has stopped working". I can simulate this event by using Windows Task Manager to end the Explorer.exe process then to file, new task (run...) Explorer.exe
      After this event, _IEAttach gives an @error of: 
      7 ($_IEStatus_NoMatch) - No Match
      However, using WinList() followed by WinGetText() I can recover the URL for an IE session that has lost its 
      InternetExplorer.Application object which could be used to _IECreate a new session (the $iTryAttach fails) and WinKill() the original session but the result is a if a refresh had been done (any form entries are lost) . Unfortunately form entries do not appear as text or hidden text to Windows so I'm looking for the best way to read the contents of an Internet Explorer screen (even if just text without structure). 
      When there is no InternetExplorer.Application object (because of some Windows fault) but still with an Internet Explorer session running, I tried to read its contents by a "Select All" & "Copy to Clipboard" but its form entries are blanked and combo boxes choices are all displayed with no way of finding out which ones were selected.
      Local $ClipBoardSave = _ClipBoard_GetData() ;not ClipGet() which is text only ClipPut("") ;Clear User's Clipboard Local $sText = "" Local $hWnd = WinActivate("Window Title", "") If IsHWnd($hWnd) Then Local $bStatusCtrlSend = ControlSend("Window Title", "", "", "^a^c") If $bStatusCtrlSend Then Local $bStatusCtrlClick = ControlClick("Window Title", "", "") If $bStatusCtrlClick Then $sText = ClipGet() _ClipBoard_SetData($ClipBoardSave) ;not ClipPut($ClipBoardSave) which is text only  
      Is  there a solution to this please?
    • Pintu
      By Pintu
      Good evening guys,
      i'm having a problem, not about the code (i'm only thinking about it at the moment) but about the way i can do it.
      I have a webpage (photo N.1) it has some elements in it.
      I need only the table (photo N.2), looking at the code and with _IEFunctions i can easy find the table but how i can i "copy" it?
      As i said it's really easy to found with a script but what should i do then? Copy the source? And how can i display it to the user?
      In my head i'd like to display it inside a GUI, is that possible?
       
      Thanks in advance
       
      Edit: Posted 2 times same photo. Now should be OK

       

    • UEZ
      By UEZ
      Here a small tool I wrote to update my Sysinternal tools collection without the need to download always the whole package or visiting the site to check for updates. I know that there are several tools available (also some tools written in AutoIt) but here another one for the collection. It was good exercise for me to code it.
       
       
        
       
       
      Some files from cannot be downloaded although they are visible on the web site!
       
      Here the download link of the source code only: AutoIt Sysinternal Tools Synchronizer v0.99.1 build 2017-10-23 beta.7z  (1353 downloads previously)
      -=> Requires AutoIt version 3.3.13.20 or higher / tested on Win8.1 real machine and some VMs: Win7 / Vista / Win10
       
      Compiled exe only: @MediaFire
       
      Just select the Sysinternal Tools folder or create one and press the synchronize button to download the selected items. Click on AutoIt label (near to left upper corner) to open menu.
       
      Special thanks to LarsJ, Melba23 and mesale0077 for their help. 
       
      I've still some ideas to implement which are more gimmick related, so it is not finished yet...
      If you want to add your language please check out #Region Language. Thanks. 
       
      Please report any bug or if you have any suggestions.
       
      The language of the tool tip from each of the executable in the left list view were automatically created using Google translator and weren't checked for correctness.
       
      Br,
      UEZ
    • Tippex
      By Tippex
      I'm having a problem with _IEBodyReadText() which is a function from the IE.au3 UDF include.
      My problem is that randomly it will crash my script with:
      Return SetError($_IESTATUS_Success, 0, $oObject.document.body.innerText) Return SetError($_IESTATUS_Success, 0, $oObject.document.body^ ERROR I don't know why the error isn't caught by __IEErrorHandlerRegister but looking at this function, I can see that there is a check done by IsObj($oObject) to ensure that the object "$oObject" exists but no check that it contains the property "innerText"
      ; #FUNCTION# ==================================================================================================================== ; Author ........: Dale Hohm ; =============================================================================================================================== Func _IEBodyReadText(ByRef $oObject) If Not IsObj($oObject) Then __IEConsoleWriteError("Error", "_IEBodyReadText", "$_IESTATUS_InvalidDataType") Return SetError($_IESTATUS_InvalidDataType, 1, 0) EndIf If Not __IEIsObjType($oObject, "browserdom") Then __IEConsoleWriteError("Error", "_IEBodyReadText", "$_IESTATUS_InvalidObjectType", "Expected document element") Return SetError($_IESTATUS_InvalidObjectType, 1, 0) EndIf ; Return SetError($_IESTATUS_Success, 0, $oObject.document.body.innerText) EndFunc ;==>_IEBodyReadText I'm struggling to find out how to test if "$oObject.document.body.innerText" exists before allowing the following Return statement to reference it with the assumption that it does exist (crashing my script when it doesn't).
      Any ideas please?
    • Nareshm
      By Nareshm
      How can i add sleep command using internet speed.
      If my internet speed is 40-50Kbps then sleep for 5 seconds,
      If my internet speed is 30-40Kbps then sleep for 4 seconds,
      If my internet speed is 20-30Kbps then sleep for 3 seconds.

      *Sleep command used bitween my costom script, Example,
      RunWait("rasdial /disconnect", "", @SW_HIDE) ; Stop connect from start
      Sleep(5000)
      RunWait("rasdial Internet", "", @SW_HIDE) ; Connect again