Jump to content

Many INETGET TCP Request

uncommon

By uncommon,
in AutoIt General Help and Support

 Share Followers 0

Recommended Posts

uncommon

uncommon 3

Posted
Posted

Right now the fastest way I can mine someone's database is by making hundreds of individual executable that all do there on INETGET TCP Request, obviously this take up a lot of processing and RAM resources. Anyone know of a way I can make more requests for pages faster\more efficiently?

My scripts that I run look something like this...

;A setprate script makes a txt file with part of a URL to go to
#include <File.au3>
#include <Array.au3>
$htmlstore = @DesktopCommonDir & "\HTMLstore\"
$FileList = _FileListToArray($htmlstore)
For $count = 2 To $FileList[0] + 1
If FileExists($htmlstore & $count & ".txt") = 1 Then
FileMove($htmlstore & $count & ".txt", $htmlstore & @AutoItPID & @ComputerName & ".txt")
$file = FileOpen($htmlstore & @AutoItPID & @ComputerName & ".txt")
$PID = FileRead($file)
FileClose($file)
$hDownload = InetGet("http://www.ocpafl.org/Searches/ParcelSearch.aspx/PID/" & $PID, $htmlstore & $PID & ".html", 1)
InetClose($hDownload) ; Close the handle to release resources.
FileDelete($htmlstore & @AutoItPID & @ComputerName & ".txt")
Exit
EndIf
Next

If you need more info let me know.

Any recommendations would be appreciated

No problem can withstand the assault of sustained thinking.Voltaire

_Array2HTMLTable(), _IEClassNameGetCollection(), _IEquerySelectorAll()

Link to post
Share on other sites
somdcomputerguy

somdcomputerguy 103

Posted
Posted

..by making hundreds of individual executable..

I think what you mean is you have to run that script many times in order to get done what you're trying to get done. If that's so, try commenting out (or removing) the Exit line from the If loop.

- Bruce /*somdcomputerguy */  If you change the way you look at things, the things you look at change.

Link to post
Share on other sites
uncommon

uncommon 3

Posted
  • Author
Posted

I think what you mean is you have to run that script many times in order to get done what you're trying to get done. If that's so, try commenting out (or removing) the Exit line from the If loop.

Alright here is the real problem, every I fire an Inetget and run it in the background and then run a different Inetget, it cancels the first and uses the second one. So I need to run many (100s) of Inetgets at once. How do I do this?

No problem can withstand the assault of sustained thinking.Voltaire

_Array2HTMLTable(), _IEClassNameGetCollection(), _IEquerySelectorAll()

Link to post
Share on other sites
BrewManNH

BrewManNH 1,315

Posted
Posted

Look at the 4th parameter for InetGet, should be fairly simple to figure out how to do it in the background while you run the next one. Also, it's explained fairly well in the help file that this is how to do multiple downloads at the same time.

If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Link to post
Share on other sites
uncommon

uncommon 3

Posted
  • Author
Posted

I made a dll that will use winhttp in threads to make multiple background requests.

I couldn't get it to work with autoit and the winhttp library or wininet either.

the multithreaded dll worked nicely, maybe you can try that.

I will look into this thanks FlutterShy I was thinking about multithreading.

And thank you BrewManNH I will do more testing on Inetget maybe I need to take a better look at remarks on the Inetget help file.

No problem can withstand the assault of sustained thinking.Voltaire

_Array2HTMLTable(), _IEClassNameGetCollection(), _IEquerySelectorAll()

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Followers 0
Go to topic listing

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • rudi
      inetget() Catch the content of the 404 response returned by the webserver?
      By rudi
      Hello,
      is there a way to use inetget() to catch the content of an 404 error page returned by the web server?
       
      $URL="https://www.autoitscript.com/ThisPathDoesntExist" $content=InetGet($url,"c:\temp\xxx.html",1+2) ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $content = "' & $content & """" & @CRLF & "@Extended: """ & @extended & """" & @CRLF & '>Error code: ' & @error & @CRLF) ;### Debug Console  
      >"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "C:\temp\löschmich\xxx.au3" /UserParams +>15:27:05 Starting AutoIt3Wrapper v.18.708.1148.0 SciTE v.4.1.0.0 Keyboard:00000407 OS:WIN_10/ CPU:X64 OS:X64 Environment(Language:0407) CodePage:0 utf8.auto.check:4 +> SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE UserDir => C:\Users\admin.AD\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper SCITE_USERHOME => C:\Users\admin.AD\AppData\Local\AutoIt v3\SciTE >Running AU3Check (3.3.14.5) from:C:\Program Files (x86)\AutoIt3 input:C:\temp\löschmich\xxx.au3 +>15:27:05 AU3Check ended.rc:0 >Running:(3.3.14.5):C:\Program Files (x86)\AutoIt3\autoit3.exe "C:\temp\löschmich\xxx.au3" --> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop @@ Debug(6) : $content = "0" @Extended: "0" >Error code: 13 +>15:27:05 AutoIt3.exe ended.rc:0 +>15:27:05 AutoIt3Wrapper Finished. >Exit code: 0 Time: 0.9361  
      The browser (I use Chrome) is displaying this 404 page: (That's what I'd like to catch)
      Not Found The requested URL /ThisPathDoesntExist was not found on this server. html code (Browser ctrl+u):
      <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /ThisPathDoesntExist was not found on this server.</p> </body></html>  
      Wireshark response 404 packet:
      Hypertext Transfer Protocol HTTP/1.1 404 Not Found\r\n Server: nginx\r\n Date: Wed, 06 Apr 2022 13:34:26 GMT\r\n Content-Type: text/html; charset=iso-8859-1\r\n Content-Length: 217\r\n Connection: keep-alive\r\n Vary: Accept-Encoding\r\n \r\n [HTTP response 1/1] [Time since request: 0.056074000 seconds] [Request in frame: 1476] [Request URI: http://www.autoitscript.com/ThisPathDoesntExist] File Data: 217 bytes Line-based text data: text/html (7 lines) <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>404 Not Found</title> </head><body> <h1>Not Found</h1> <p>The requested URL /ThisPathDoesntExist was not found on this server.</p> </body></html> any suggestions appreciated,
      <edit> also tried _inetgetsource() and inetread() </edit>
      Rudi
    • Jahar
      InetGet was working previously, but not extracting full html
      By Jahar
      I have an AutoIT script It monitors 2 websites for content that applys to me and the services that I provide. One site is : www.Freelancer.com The other: www.PeoplePerHour.com Both sites publish new jobs on their site hourly or so. My AutoIT app, will view those sites and present new jobs to me in a grid that pops up on my screen. Lately, the app has stopped showing me any jobs from PeoplePerHour.
       
      For freelancer.com,  Inetget is giving full html but for peopleperhour, now its not coming.
      Func _CheckPPH()
          Local Static $hTimer = 0
          Local Static $hDownload = 0
          Local $aTitlesandUrls = 0
          Local Static $sTempFile = ""
          If $hTimer = 0 Then $hTimer = TimerInit()
          If $hDownload = 0 Then
              $sTempFile = _WinAPI_GetTempFileName(@TempDir)
              ConsoleWrite("Checking PPH..." & @CRLF)
              ConsoleWrite(">Downloading..." & @CRLF)
      ;~         $hDownload = InetGet("http://www.peopleperhour.com/freelance-jobs", $sTempFile, $INET_FORCERELOAD, $INET_DOWNLOADBACKGROUND)
              $hDownload = InetGet("http://www.peopleperhour.com/freelance-jobs", $sTempFile, $INET_FORCERELOAD)
      ;~         Return 0
          EndIf
      ;~     Sleep(30)
      ;~     Local $isCompleted = InetGetInfo($hDownload, $INET_DOWNLOADCOMPLETE)
      ;~     Local $isError = InetGetInfo($hDownload, $INET_DOWNLOADERROR)
      ;~     Sleep(30)
      ;~     If TimerDiff($hTimer) > 3000 And $isError Then
      ;~         ConsoleWrite("!PPH Fail" & @CRLF)
      ;~         InetClose($hDownload)
      ;~         $hDownload = 0
      ;~         Return 0
      ;~     EndIf
      ;~     Sleep(30)
          Local $Show = 0
      ;~     If TimerDiff($hTimer) > 3000 And $isCompleted Then
          If $hDownload > 0 Then
              ConsoleWrite("+Downloaded..." & @CRLF)
              Local $sPPHHtml = FileRead($sTempFile)
              $aTitlesandUrls = _StringBetween($sPPHHtml, '"title">' & @LF, 'time>')
      ;~         _ArrayDisplay($aTitlesandUrls)
              Local $aPPH[0][4]
              Local $sTitle = ""
              Local $sUrl = ""
              Local $sID = ""
              Local $sDate = ""
              Local $iRet=0
              Sleep(30)
              For $i = 0 To UBound($aTitlesandUrls) - 1
                  $sTitle = _StringBetween($aTitlesandUrls[$i], '<a title="', '" class')
                  $sUrl = _StringBetween($aTitlesandUrls[$i], 'href="', '">')
                  $sDate = _GetDate($aTitlesandUrls[$i])
                  If IsArray($sTitle) And IsArray($sUrl) Then
                      $sID = _GetID($sUrl[0])
      ;~                 _ArrayAdd($aPPH, $sDate & "|" & $sTitle[0] & "|" & $sUrl[0] & "|" & $sID)
                      $iRet = _BuildPopupsPPH($sID, $sDate, "PPH: " & $sTitle[0], $sUrl[0])
                      If $iRet Then $Show+=1
                  EndIf
              Next
              Sleep(30)
      ;~         If $Show > 0 Then ShowLatestJobs()
      ;~         _ArrayDisplay($aPPH)
              FileDelete($sTempFile)
              InetClose($hDownload)
              $hDownload = 0
              $hTimer = 0
              Return $Show
          EndIf
          Sleep(30)
      EndFunc   ;==>_CheckPPH
    • Tersion
      Why InetGet() allow only two simultaneously background downloads?
      By Tersion
      Why I can't get more than two simultaneously background downloads?
      Simultaneously background downloads with InetGet() works! It was just server limitation from which I downloaded files.
    • Rskm
      time from internet
      By Rskm
      Hi, for a script of mine, I wish to keep a time frame on till how the program will work.. basically, I want the program to read the date and if the read date < 'a particular date', the program continues, else it exit. I don't want to read the time from the computer as the program can be fooled by resetting the pc time. is there a way to read time from internet - so that the user cannot fool the program by date reset? thanks
    • ur
      How to notify a process from internet
      By ur
      I have an aws cloud machine where I kept all the build process to create some set of zip files and uploading them to OneDrive.
      Once it is completed, I have another local machine in our environment which is a acting as a local share server also.
      Is there anyway to notify this machine(without public IP) to start downloading these files.
      I have written an AutoIT exe to download these files and copy them to respective locations.But don't know how to notify this from the remote machine.
       
      Any suggestion??
       
×
×
  • Create New...