Optimizing _INetGetSource. is it possible?

[pillbug]

Hi, I've been working on optimizing the _INetGetSource( and I am not seeing much improvement.

If I have a 1 MB download connection, and each html file is about 10K, I should be able to download 100 searched results per second. Unfortunately, I am not even close to this.

Although eventually I don't plan to use this for google searches, but a list of websites, I've created an example of searches in google with the search word "hello"

I suspect the main problem is having to connect to wininet.dll

If anyone sees ways to improve the program, please let me know on how to speed this process.

expandcollapse popup

$begin = TimerInit()

$h_DLL = DllOpen('wininet.dll')
    $ai_IO = DllCall($h_DLL, 'int', 'InternetOpen', 'str', 'AutoIt v3', 'int', 0, 'int', 0, 'int', 0, 'int', 0)
    If @error Or $ai_IO[0] = 0 Then
        SetError(1)
        Return ''
    EndIf
$v_Struct = DllStructCreate('udword'); udword=unsigned 32 bit integer
for $i = 0 to 399

    $s_URL= 'http://www.google.com/search?hl=en&q=hello&start=' & $i*10 & '&sa=N'
    $ai_IOU = DllCall($h_DLL, 'int', 'InternetOpenUrl', 'int', $ai_IO[0], 'str', $s_URL, 'str', '', 'int', 0, 'int', 0x80000000, 'int', 0)
    If @error Or $ai_IOU[0] = 0 Then
        $source= ''
    EndIf

$s_Buf = ''
    DllStructSetData($v_Struct, 1, 1)
    for $z=1 to 3
        $ai_IRF = DllCall($h_DLL, 'int', 'InternetReadFile', 'int', $ai_IOU[0], 'str', '', 'int', 4096, 'ptr', DllStructGetPtr($v_Struct))
        $s_Buf &= StringLeft($ai_IRF[2], DllStructGetData($v_Struct, 1)); 
    next
    DllCall($h_DLL, 'int', 'InternetCloseHandle', 'int', $ai_IOU[0])
    $source= $s_Buf

TrayTip('', 'Source: ' & $i, 3)

if $i = 5 then TrayTip('', 'Source: ' & stringLeft($source, 5), 3)

next

DllCall($h_DLL, 'int', 'InternetCloseHandle', 'int', $ai_IO[0])
DllClose($h_DLL)

$dif =Timerdiff($begin)
msgbox(0,"Time Difference", $dif)

Edited February 1, 2009 by pillbug

[Inverted]

If I have a 1 MB download connection, and each html file is about 10K, I should be able to download 100 searched results per second.

Way to oversimplify ! I don't have the knowledge to help you, maybe someone else will provide more useful info. I just wanted to say that your thinking is wrong, there are are more connection latencies involved. And the fractions of a second needed to interface with a dll is negligible.

[BrettF]

Maybe something like this, but it was still way slower than what you posted. Maybe explore the forums and see if there is any other examples of getting the source of a page...

$begin = TimerInit()

$oHTTP = ObjCreate("winhttp.winhttprequest.5.1")

$term = "hello"

For $i = 0 To 10
    $s_URL = 'http://www.google.com/search?hl=en&q=' & $term & '&start=' & $i * 10 & '&sa=N'
    ClipPut ($s_URL)
    $oHTTP.Open("GET", $s_URL, False)
    $oHTTP.Send()
    $source = $oHTTP.ResponseText
    ConsoleWrite ("Source for document " & $i & @CRLF)
Next

$dif = TimerDiff($begin)
MsgBox(0, "Time Difference", $dif & @CRLF & "Average (ms) = " & Int ($dif/$i))

[AzKay]

I already tried that one Brett, For me, it ended up being 17 seconds for all 10; when his code was like, 4 seconds for 10. So I didnt bother posting.

My only suggestion, would be maybe making your own function using the TCP* stuff.

Something like;

TCPConnect()

For $i = 0 To 10

TCPSend()

While 1

;some other thing to log the source

WEnd

Next

So that your not having to reopen a connection every time, Though I dont know if that works, as ive never tried.

But I think it should work.

[TerarinK]

Remember this is single thread code your working with and it takes time to excute this code so right there your download rate is slowing down or I should say not fully being used. First off how fast is your computer because every mm adds and having 100 at about 10 mm is 1000 mm, oh a second more or less.

Anyways your forgets the handshake that you must pass through then the data your sending each taking up those precious mm, in the end you likely to get 5-25 but no more remember the more the more time it takes to retrieve them unless you go thread wise. Go to search and lookup "Thread" that should get you in the direction you should get to maximize your full download rate

[TerarinK]

I already tried that one Brett, For me, it ended up being 17 seconds for all 10; when his code was like, 4 seconds for 10. So I didnt bother posting.

My only suggestion, would be maybe making your own function using the TCP* stuff.

Something like;

TCPConnect()

For $i = 0 To 10

TCPSend()

While 1

;some other thing to log the source

WEnd

Next

So that your not having to reopen a connection every time, Though I dont know if that works, as ive never tried.

But I think it should work.

Check on a topic concerning that, however it is for http but you want it to act like a ftp

http://www.autoitscript.com/forum/index.ph...p;hl=tcpCONNECT

[pillbug]

So I looked into TCP.

I modified some of the code from http://www.autoitscript.com/forum/index.ph...0617&hl=tcp

However, TCP is 2x slower:

TCP:6035.04243275544 DLL:3014.04479785984

Do any TCP experts see any ways to make TCP faster or the DLL faster?

expandcollapse popup

$begin_tcp = TimerInit()
TCPFUNCTION()
$dif_tcp =Timerdiff($begin_tcp)

$begin_dll = TimerInit()
DLLFUNCTION()
$dif_dll =Timerdiff($begin_dll)

Consolewrite("TCP:" & $dif_tcp & "   DLL:" & $dif_dll)






Func TCPFunction()
    TCPStartup(); initializing service
    
for $i = 0 to 4

    $URL= 'http://www.google.com/search?hl=en&q=hello&start=' & $i*10 & '&sa=N' 
    
    $URL = StringRegExpReplace($URL, '\A(http://|https://)(.*?)/?\Z', '$2'); dropping http// https// if are there e.g. will return www.autoitscript.com/forum/index.php?showforum=9 for us
   

   
    Local $dom = StringRegExpReplace($URL, '\A(.*?)/.*', '$1'); this part is domain name (www.autoitscript.com)
    Local $ip = TCPNameToIP($dom); will need this to connect to server
    If $ip = "" Then Return -1
    Local $get = StringRegExpReplace($URL, '\A(.*?)/(.*)', '$2'); we want this (forum/index.php?showforum=9)
    If $get = $dom Then $get = ''; in case requiring main page
   
    Local $header = 'GET /' & $get & ' HTTP/1.1' & @CRLF _
             & 'User-Agent: Test' & @CRLF _
             & 'Host: 127.0.0.1' & @CRLF & @LF; something about us and what we want  ending with @CRLF & @LF

    Local $socket = TCPConnect($ip, 80); connecting to server
    If $socket = -1 Then Return -2; will not check any more errors from here on (you do it tongue.gif)
   
    TCPSend($socket, $header); sending request

    Local $rcv, $out, $x, $sw, $r, $lenght
   
    While 1
       
        If $rcv <> '' Then
            If $x <> 1 Then

                $lenght = Number(StringRegExpReplace($rcv, '(?s)(.*?)Content-Length: (\d+)(.*)', '$2') + StringLen(StringLeft($rcv, StringInStr($rcv, @CRLF & @CRLF))) + 3)
            EndIf
            $x = 1
        EndIf
       
        $rcv = TCPRecv($socket, 1024); receiving data from server
 
        $out &= $rcv; adding to what we already have
       
        If $x = 1 Then
            If $rcv = '' Then
                $sw += 1
                If $sw = 10000 Then ExitLoop; sometimes there is no end, so we'll have to end it
            Else
                $sw = 0
            EndIf
        EndIf
       
        If $lenght <> 0 Then
            If StringLen($out) = $lenght Then; some servers are done when they send ammount of data that they previously declared
                ExitLoop
            EndIf
        EndIf
       
        If StringRight($rcv, 5) = 0 & @CRLF & @CRLF Then; some servers will end with this
            ExitLoop
        EndIf
       
    WEnd
   

    TCPCloseSocket($socket); closing socket
  ;Return $out
   TrayTip('', 'Source: ' & $i & $out, 3)
next

    TCPShutdown(); stopping service
EndFunc

Func DLLFUNCTION()
$h_DLL = DllOpen('wininet.dll')
    $ai_IO = DllCall($h_DLL, 'int', 'InternetOpen', 'str', 'AutoIt v3', 'int', 0, 'int', 0, 'int', 0, 'int', 0)
    If @error Or $ai_IO[0] = 0 Then
        SetError(1)
        Return ''
    EndIf
$v_Struct = DllStructCreate('udword'); udword=unsigned 32 bit integer
for $i = 0 to 4

    $s_URL= 'http://www.google.com/search?hl=en&q=hello&start=' & $i*10 & '&sa=N'
    $ai_IOU = DllCall($h_DLL, 'int', 'InternetOpenUrl', 'int', $ai_IO[0], 'str', $s_URL, 'str', '', 'int', 0, 'int', 0x80000000, 'int', 0)
    If @error Or $ai_IOU[0] = 0 Then
        $source= ''
    EndIf

$s_Buf = ''
    DllStructSetData($v_Struct, 1, 1)
    for $z=1 to 3
        $ai_IRF = DllCall($h_DLL, 'int', 'InternetReadFile', 'int', $ai_IOU[0], 'str', '', 'int', 4096, 'ptr', DllStructGetPtr($v_Struct))
        $s_Buf &= StringLeft($ai_IRF[2], DllStructGetData($v_Struct, 1)); 
    next
    DllCall($h_DLL, 'int', 'InternetCloseHandle', 'int', $ai_IOU[0])
    $source= $s_Buf

TrayTip('', 'Source: ' & $i, 3)

if $i = 5 then TrayTip('', 'Source: ' & stringLeft($source, 5), 3)

next

DllCall($h_DLL, 'int', 'InternetCloseHandle', 'int', $ai_IO[0])
DllClose($h_DLL)

ENDFUNC

Edited February 6, 2009 by pillbug

[ProgAndy]

Try to split the DLL-function into 3 parts:

1): open DLL and call InternetOpen

2): Open URL, download and Close URL-Handle

2a): repeat step 2 as often as you want.

3): at the end of the Script, close InternetOpen and DLL

[pillbug]

Try to split the DLL-function into 3 parts:

1): open DLL and call InternetOpen

2): Open URL, download and Close URL-Handle

2a): repeat step 2 as often as you want.

3): at the end of the Script, close InternetOpen and DLL

I am not sure how my code is different from what you said...

Could you give me some code to show what you mean?