Get huge text from server HTM file

[DaLiMan]

Hi,

I'm having trouble making this work.
And this is probably because of the size of the file. (It's huge...)

If I try another website it is working just fine.

I need all the text from the link to be downloaded into a text file in the PC.
Now this is done all by hand and I like to automate this, because it is done every month for 5 files.
The link is on a server at work.

Anyone have any idea on how to get this to work?
Maybe another way / workaround?

 

PS: InetGet is not working because it is blocked somehow by the server....

 

Local $Link_ET = "http://prod/PRTA/%27WS01.WEBA99.B2JVET.J01.D2016336.HTM%27"

$oHTTP = ObjCreate("winhttp.winhttprequest.5.1")
$oHTTP.Open("GET", $Link_ET, False)
$oHTTP.SetCredentials ("myname", "secretpw",0)
$oHTTP.Send()

$oHTTPtext = $oHTTP.ResponseText
$oStatusCode = $oHTTP.Status
ConsoleWrite("stcde: " & $oStatusCode & @LF)


If $oStatusCode == 200 then
 ConsoleWrite("writing..." & @LF)
 $file = FileOpen(@DesktopDir & "\Received.txt", 18) ; The value of 2 overwrites the file if it already exists
 FileWrite($file, $oHTTPtext)
 FileClose($file)
EndIf

ConsoleWrite("The End" & @LF)

 

[MagnumXL]

I would think that the problem with both this and any inetget script is the link address in the first line. Are you going for apostrophes with the %27's? And of course the whole thing will fail if  local (any) DNS isn't handling the "prod" cname.

[Jfish]

Could you explain exactly what is not working, any error codes you see, and what Scite shows you for output?  That may help folks better understand what is not working.

 

[DaLiMan]

The link is valid, hower I understand the question about the '%27' at the end of the link.
But I have another file on the server which also has the %27 at the end which is working just fine.

As for the error code from SciTe.
When I start the program it just hangs after the $oHTTP.Send()
After that SciTe comes with a pop-up it that it's not working anymore.

+>         SciTEDir => D:\PortableApps\AutoIt\install\SciTE4AutoIt3_Portable   UserDir => D:\PortableApps\AutoIt\install\SciTE4AutoIt3_Portable\AutoIt3Wrapper
>Running AU3Check (3.3.14.2)  from:D:\PortableApps\AutoIt\install  input:D:\Documents\Scripts\_test.au3
+>07:13:42 AU3Check ended.rc:0
>Running:(3.3.14.2):D:\PortableApps\AutoIt\install\autoit3_x64.exe "D:\Documents\Scripts\_test.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
Check1
Check2
Check3
Check4
!>07:14:29 AutoIt3.exe ended.rc:-1073741819
+>07:14:29 AutoIt3Wrapper Finished.
>Exit code: 3221225477    Time: 47.36

 

 

I made some consolewrite checks into the code to see where it goes wrong.

In the above SciTe log it seems $oHTTP.ResponseBody is actually giving the error.
Now I read somewhere that there could occur a timeout-error with very large files??
 

Local $Link_ET = "http://prod/PRTA/%27WS01.WEBA99.B2JVET.J01.D2016336.HTM%27"

$oHTTP = ObjCreate("winhttp.winhttprequest.5.1")
ConsoleWrite("Check1" & @LF)
$oHTTP.Open("GET", $Link_ET, False)
ConsoleWrite("Check2" & @LF)
$oHTTP.SetCredentials ("********", "*******",0)
ConsoleWrite("Check3" & @LF)
$oHTTP.Send()
ConsoleWrite("Check4" & @LF)

$oHTTPtext = $oHTTP.ResponseBody
ConsoleWrite("Check5" & @LF)
$oStatusCode = $oHTTP.Status
ConsoleWrite("stcde: " & $oStatusCode & @LF)


If $oStatusCode == 200 then
 ConsoleWrite("writing..." & @LF)
 $file = FileOpen(@DesktopDir & "\Received.txt", 18) ; The value of 2 overwrites the file if it already exists
 FileWrite($file, $oHTTPtext)
 FileClose($file)
EndIf

ConsoleWrite("The End" & @LF)

 

Edited December 8, 2016 by DaLiMan
typos

[Jfish]

A bit of Googling suggests that the default timeout is 30 seconds.  But there is a method to change it: https://msdn.microsoft.com/en-us/library/windows/desktop/aa384061(v=vs.85).aspx

 

[DaLiMan]

Hi Jfish, thanx for the link.
So I tried setting the timeouts a bit larger but with no result.

$oHTTP.SetTimeouts(0,200000,200000,200000)

 

Still getting a crash.
Maybe there is more to it than meets the eye. I'm stuck.

>"D:\PortableApps\AutoIt\install\SciTE4AutoIt3_Portable\..\AutoIt3.exe" "D:\PortableApps\AutoIt\install\SciTE4AutoIt3_Portable\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "D:\Documents\Scripts\_test.au3" /UserParams    
+>10:03:34 Starting AutoIt3Wrapper v.16.306.1237.0 SciTE v.3.6.2.0   Keyboard:00020409  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64  Environment(Language:0413)  CodePage:0  utf8.auto.check:4    # detect ascii high characters and if none found set default encoding to UTF8 and do not add BOM
+>         SciTEDir => D:\PortableApps\AutoIt\install\SciTE4AutoIt3_Portable   UserDir => D:\PortableApps\AutoIt\install\SciTE4AutoIt3_Portable\AutoIt3Wrapper
>Running AU3Check (3.3.14.2)  from:D:\PortableApps\AutoIt\install  input:D:\Documents\Scripts\_test.au3
+>10:03:34 AU3Check ended.rc:0
>Running:(3.3.14.2):D:\PortableApps\AutoIt\install\autoit3_x64.exe "D:\Documents\Scripts\_test.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
Check1
Check2
Check3
Check4
Check5
!>10:05:03 AutoIt3.exe ended.rc:-1073741819
+>10:05:03 AutoIt3Wrapper Finished.
>Exit code: 3221225477    Time: 89.33

 

[Jfish]

This time you made it to check 5 - last time you only made it to check 4 - it looks like you got the response, no?  Now you need to debug what happens next.  

EDIT: You may be having latency on the response for the status as well.  You could use the wait for response as shown here: https://msdn.microsoft.com/en-us/library/windows/desktop/aa384064(v=vs.85).aspx.

EDIT: I might also suggest dropping the http status check after the response and not messing with the wait for response - just for debugging. That way you can check to see if you actually received the content and if your response status check is the issue.  

 

Edited December 9, 2016 by Jfish

[DaLiMan]

Hi Jfish, the .WaitForResponse() seems to be working. 
Below is the first promising result....
Let's hope it will work next week also.

Thank you very much for the help. 

 

$oHTTP = ObjCreate("winhttp.winhttprequest.5.1")
ConsoleWrite("Check1" & @LF)
$oHTTP.Open("GET", $Link_ET, False)
ConsoleWrite("Check2" & @LF)
$oHTTP.SetCredentials ("****", "******",0)
ConsoleWrite("Check3" & @LF)
$oHTTP.SetTimeouts(0,200000,200000,200000)
ConsoleWrite("Check4" & @LF)
$oHTTP.Send()
ConsoleWrite("Check5" & @LF)
$oHTTP.WaitForResponse()
ConsoleWrite("Check6" & @LF)

$oHTTPtext = $oHTTP.ResponseBody
ConsoleWrite("Check7" & @LF)
$oStatusCode = $oHTTP.Status
ConsoleWrite("stcde: " & $oStatusCode & @LF)

 

>"D:\PortableApps\AutoIt\install\SciTE4AutoIt3_Portable\..\AutoIt3.exe" "D:\PortableApps\AutoIt\install\SciTE4AutoIt3_Portable\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "D:\Documents\Scripts\_test.au3" /UserParams    
+>08:19:53 Starting AutoIt3Wrapper v.16.306.1237.0 SciTE v.3.6.2.0   Keyboard:00020409  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64  Environment(Language:0413)  CodePage:0  utf8.auto.check:4    # detect ascii high characters and if none found set default encoding to UTF8 and do not add BOM
+>         SciTEDir => D:\PortableApps\AutoIt\install\SciTE4AutoIt3_Portable   UserDir => D:\PortableApps\AutoIt\install\SciTE4AutoIt3_Portable\AutoIt3Wrapper
>Running AU3Check (3.3.14.2)  from:D:\PortableApps\AutoIt\install  input:D:\Documents\Scripts\_test.au3
+>08:19:53 AU3Check ended.rc:0
>Running:(3.3.14.2):D:\PortableApps\AutoIt\install\autoit3_x64.exe "D:\Documents\Scripts\_test.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
Check1
Check2
Check3
Check4
Check5
Check6
Check7
stcde: 200
writing...
The End
+>08:20:40 AutoIt3.exe ended.rc:0
+>08:20:40 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 47.4

 

[Jfish]