Trying to DL a file... gets redirected

[Mechaflash]

Trying to download a file from here:

http://omahadailyrecord.com/index.cfm?show=82&mid=188&viewDoc=3734

If you go to the link, you'll see you're redirected to a login. I have the login credentials already, so I've tried passing it to inetget via

http:\\username:password@omahadailyrecord.com/index.cfm?show=82&mid=188&viewDoc=3734

But it doesn't work.

What happens is the page it attempts to download is the login page instead of the pdf in question.

Any thoughts?

Edited March 12, 2013 by Mechaflash

[BrewManNH]

Leave out the second http:// in the URL?

[Mechaflash]

just a typo in this post but not in my code =D

my code iterates over a few different viewDoc=#s

Edited March 12, 2013 by Mechaflash

[FireFox]

This is not a HTTP raw authentication as I can see, so I doubt you can login by the URL.

[Mechaflash]

You are correct. I had to perform a more direct approach with opening each download link in a new ie tab and manipulating the Save As window to download it.

[FireFox]

You are correct. I had to perform a more direct approach with opening each download link in a new ie tab and manipulating the Save As window to download it.

Well, I would retrieve the cookie and use it with HTTP requests through TCP functions.

[Mechaflash]

[FireFox]

Tell me where are you stuck in what I said ?

[Mechaflash]

retrieve the cookie and use it with HTTP requests through TCP functions.

[FireFox]

Retrieve the cookie :

;login to the webpage or directly attach to it when logged.
...
$sCookie = $oIE.document.cookie

HTTP request :

expandcollapse popup

TCPStartup()

Local $sGet = "", $sHost = "", $iSocket = 0, $sReferer = "", $bRecvtmp = Binary(""), $bRecv = $bRecvtmp, $sRecv = "", _
        $blHeaderStripped = False, $iEOH = 0, $hFile = 0

$sGet = "/index.cfm?show=82&mid=188&viewDoc=3734"

$sHost = TCPNameToIP("omahadailyrecord.com")

$iSocket = TCPConnect($sHost, 80)

$sReferer = "http://omahadailyrecord.com/"

TCPSend($iSocket, _
        "GET " & $sGet & " HTTP/1.1" & @CRLF & _
        "User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)" & @CRLF & _
        "Host: " & $sHost & @CRLF & _
        "Accept: */*" & @CRLF & _
        "Referer: " & $sReferer & @CRLF & _
        "Connection: close" & @CRLF & _
        "Cookie: " & $sCookie & @CrLf & _
        @CRLF)

While 1
    $bRecvtmp = TCPRecv($iSocket, 2048, 1)
    If @error Then ExitLoop

    If BinaryLen($bRecvtmp) Then
        If $blHeaderStripped Then
            $bRecv &= $bRecvtmp
        Else
            $sRecv = BinaryToString($bRecvtmp)

            $iEOH = StringInStr($sRecv, @CRLF & @CRLF, 2)
            If $iEOH > 0 Then
                $bRecv = BinaryMid($bRecvtmp, $iEOH + 4)

                $blHeaderStripped = True
            EndIf
        EndIf
    EndIf
WEnd

TCPCloseSocket($iSocket)

TCPShutdown()

$hFile = FileOpen("test.pdf", 18)
FileWrite($hFile, $bRecv)
FileClose($hFile)

Not tested as I'm not registered on that website.

Edit: Note that you can save the cookie in a file/var and use it until this one is not valid.

Br, FireFox.

Edited March 13, 2013 by FireFox

[Mechaflash]

i don't know what other words to express the way I feel about you right now... but..

I love you <3

[FireFox]

I love you <3

Then I consider that it worked.

Glad to help you dear lover

[Mechaflash]

one more question for u sir...

how would I go about verifying the downloaded file? making sure that the file that was downloaded was downloaded fully? (checking file size or something?)

Edited March 14, 2013 by Mechaflash

[FireFox]

checking file size or something?

Sure , here you go : (Not tested.)

expandcollapse popup

TCPStartup()

Local $sGet = "", $sHost = "", $iSocket = 0, $sReferer = "", $bRecvtmp = Binary(""), $bRecv = $bRecvtmp, $sRecv = "", _
        $blHeaderStripped = False, $iEOH = 0, $hFile = 0, $iContLenPos = 0, $iContLenPosEnd = 0, $iFileSize = 0

$sGet = "/index.cfm?show=82&mid=188&viewDoc=3734"

$sHost = TCPNameToIP("omahadailyrecord.com")

$iSocket = TCPConnect($sHost, 80)

$sReferer = "http://omahadailyrecord.com/"

TCPSend($iSocket, _
        "GET " & $sGet & " HTTP/1.1" & @CRLF & _
        "User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)" & @CRLF & _
        "Host: " & $sHost & @CRLF & _
        "Accept: */*" & @CRLF & _
        "Referer: " & $sReferer & @CRLF & _
        "Connection: close" & @CRLF & _
        "Cookie: " & $sCookie & @CrLf & _
        @CRLF)

While 1
    $bRecvtmp = TCPRecv($iSocket, 2048, 1)
    If @error Then ExitLoop

    If BinaryLen($bRecvtmp) Then
        If $blHeaderStripped Then
            $bRecv &= $bRecvtmp
        Else
            $sRecv = BinaryToString($bRecvtmp)

            $iEOH = StringInStr($sRecv, @CRLF & @CRLF, 2)
            If $iEOH > 0 Then
                $bRecv = BinaryMid($bRecvtmp, $iEOH + 4)

                $blHeaderStripped = True

                $iContLenPos = StringInStr($sRecv, "Content-Length:", 2)
                If $iContLenPos = 0 Then
                    $iContLenPosEnd = $iContLenPos + StringLen("Content-Length:")

                    $iFileSize = Number(StringMid($sRecv, $iContLenPosEnd, StringInStr($sRecv, @CR, 2, 1, $iContLenPosEnd) - $iContLenPosEnd))
                EndIf
            EndIf
        EndIf
    EndIf
WEnd

TCPCloseSocket($iSocket)

TCPShutdown()

If BinaryLen($bRecv) = $iFileSize Then
    $hFile = FileOpen("test.pdf", 18)
    FileWrite($hFile, $bRecv)
    FileClose($hFile)
Else
    ConsoleWrite("corrupted download !" & @CrLf)
EndIf

Br, FireFox.

[Mechaflash]

(Haven't test yet) looking at the added code... is

StringLen("Content-Length:")

necessary? Why would I specify a string instead of just inserting the number 15 for the value?

EDIT:

And $iFileSize returns 0 even though the file downloads successfully. I'm looking into why that's happening now.

Edited March 14, 2013 by Mechaflash

[FireFox]

necessary? Why would I specify a string instead of just inserting the number 15 for the value?

No, but I don't like "magic numbers" anymore. I prefer my code to be as clear as possible.

And $iFileSize returns 0 even though the file downloads successfully. I'm looking into why that's happening now.

Check the sRecv var to see if it contains the Content-Length header param.

Br, FireFox.

[Mechaflash]

Instead of doing all that, I waited for the loop to end and just checked the filesize of the completed file FileGetSize() against BinaryLen($bRecv), and they do match if the file was downloaded properly.

[Mechaflash]

expandcollapse popup

TCPStartup()

Local $sGet = "", $sHost = "", $iSocket = 0, $sReferer = "", $bRecvtmp = Binary(""), $bRecv = $bRecvtmp, $sRecv = "", _
        $blHeaderStripped = False, $iEOH = 0, $hFile = 0, $iContLenPos = 0, $iContLenPosEnd = 0, $iFileSize = 0

$sGet = "/index.cfm?show=82&mid=188&viewDoc=3734"

$sHost = TCPNameToIP("omahadailyrecord.com")

$iSocket = TCPConnect($sHost, 80)

$sReferer = "http://omahadailyrecord.com/"

TCPSend($iSocket, _
        "GET " & $sGet & " HTTP/1.1" & @CRLF & _
        "User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)" & @CRLF & _
        "Host: " & $sHost & @CRLF & _
        "Accept: */*" & @CRLF & _
        "Referer: " & $sReferer & @CRLF & _
        "Connection: close" & @CRLF & _
        "Cookie: " & $sCookie & @CrLf & _
        @CRLF)

While 1
    $bRecvtmp = TCPRecv($iSocket, 2048, 1)
    If @error Then ExitLoop

    If BinaryLen($bRecvtmp) Then
        If $blHeaderStripped Then
            $bRecv &= $bRecvtmp
        Else
            $sRecv = BinaryToString($bRecvtmp)

            $iEOH = StringInStr($sRecv, @CRLF & @CRLF, 2)
            If $iEOH > 0 Then
                $bRecv = BinaryMid($bRecvtmp, $iEOH + 4)

                $blHeaderStripped = True
            EndIf
        EndIf
    EndIf
WEnd

TCPCloseSocket($iSocket)

TCPShutdown()

$hFile = FileOpen("test.pdf", 18)
FileWrite($hFile, $bRecv)
FileClose($hFile)

if BinaryLen($bRecv) <> FileGetSize("test.pdf") then
    ConsoleWrite("corrupted download !" & @CrLf)
EndIf

[FireFox]

@Mechaflash

Huh, the binary length of the bRecv var is written in your file, so the binary length of your file will be the same.

So, it's useless.

The only way to know if you received everything is to know the length of the data to receive or to retrieve the Content-Length header value if available. (it's always the case for a binary download, not for e.g a webpage).

Br, FireFox.

Edited March 14, 2013 by FireFox

[Mechaflash]

Ok then... so in your code,

$iContLenPos = StringInStr($sRecv, "Content-Length:", 2) ; return 0

If $iContLenPos = 0 Then
  $iContLenPosEnd = $iContLenPos + StringLen("Content-Length:") ; returns 15
  $iFileSize = Number(StringMid($sRecv, $iContLenPosEnd, StringInStr($sRecv, @CR, 2, 1, $iContLenPosEnd) - $iContLenPosEnd)) ; returns 0
EndIf

If I remove the if statement, and let it write to the file, the file is received and is perfectly readable. So not sure where this code is supposed to be going.