Jump to content
Sign in to follow this  
uncommon

This might not be an autoit question but...

Recommended Posts

uncommon

Looking to run some scripts through this site. Source code a bit too big to post but I am looking for a way to search though this database in a systematic fashion but I don't know how to accomplish this.

http://www.ocpafl.org/Searches/ParcelSearch.aspx

I know how to put fields in the search box and search any address then rip the source code off the page. That's easy.

What I don't get is to make sure I don't miss any address when I am searching through the database.

Ideas comments?


No problem can withstand the assault of sustained thinking.Voltaire

_Array2HTMLTable()_IEClassNameGetCollection()_IEquerySelectorAll()

Share this post


Link to post
Share on other sites
Jury

Use 'Property Use Search' and go through each 'Specify Property Type' listed selecting 'All' and then grab the items as you go through the pages.

Share this post


Link to post
Share on other sites
uncommon

I feel kind of dumb for not think thinking of that myself. Thanks

After testing it a few more times I found out you can search for properties via the PID like this http://www.ocpafl.org/Searches/ParcelSearch.aspx/PID/272001000000008

So now its only a matter of time.... a Long time


No problem can withstand the assault of sustained thinking.Voltaire

_Array2HTMLTable()_IEClassNameGetCollection()_IEquerySelectorAll()

Share this post


Link to post
Share on other sites
Jury

uncommon,

You should be able to automate coing through the pages and grabbing the content into array and then into delimited file - hint:

$oNext = _IEGetObjByName($oIE, "pager1", $z)

_IEAction($oNext, "click")

_IELoadWait($oIE)

Sleep(2000)

Local $oTable = _IETableGetCollection($oIE, 0)

Local $aTableData = _IETableWriteToArray($oTable, 1)

_FileWriteFromArray($file, $aTableData, 1, 0)

Share this post


Link to post
Share on other sites
uncommon

uncommon,

You should be able to automate coing through the pages and grabbing the content into array and then into delimited file - hint:

$oNext = _IEGetObjByName($oIE, "pager1", $z)

_IEAction($oNext, "click")

_IELoadWait($oIE)

Sleep(2000)

Local $oTable = _IETableGetCollection($oIE, 0)

Local $aTableData = _IETableWriteToArray($oTable, 1)

_FileWriteFromArray($file, $aTableData, 1, 0)

Thanks was using the IE to get webpages but the Inetget out to file works so much for eficaintly, especailly if you are doing multiple request on 200,000 some pages. My current plan is to only work on getting all the pages into a folder, then use a seprat scrpt aftwards to pharse the HTML for the needed content into SQL or some kind of database.

Hmm I might use IE to naviget the local files once downloaded to make the scraping a but easier, thanks.


No problem can withstand the assault of sustained thinking.Voltaire

_Array2HTMLTable()_IEClassNameGetCollection()_IEquerySelectorAll()

Share this post


Link to post
Share on other sites
uncommon

Right now the fastest way I can mine someone's database is by making hundreds of individual executables that all do there on INETGET TCP Request, obviously this take up a lot of processing and RAM resources. Anyone know of a way I can make more requests for pages faster\more efficiently?

My scripts that I run look something like this...

;A setprate script makes a txt file with part of a URL to go to
#include <File.au3>
#include <Array.au3>
$htmlstore = @DesktopCommonDir & "\HTMLstore\"
$FileList = _FileListToArray($htmlstore)
For $count = 2 To $FileList[0] + 1
If FileExists($htmlstore & $count & ".txt") = 1 Then
FileMove($htmlstore & $count & ".txt", $htmlstore & @AutoItPID & @ComputerName & ".txt")
$file = FileOpen($htmlstore & @AutoItPID & @ComputerName & ".txt")
$PID = FileRead($file)
FileClose($file)
$hDownload = InetGet("http://www.ocpafl.org/Searches/ParcelSearch.aspx/PID/" & $PID, $htmlstore & $PID & ".html", 1)
InetClose($hDownload) ; Close the handle to release resources.
FileDelete($htmlstore & @AutoItPID & @ComputerName & ".txt")
Exit
EndIf
Next

If you need more info let me know.

Any recommendations would be appreciated


No problem can withstand the assault of sustained thinking.Voltaire

_Array2HTMLTable()_IEClassNameGetCollection()_IEquerySelectorAll()

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Similar Content

    • Seminko
      By Seminko
      Is there a way to grab non-hardcoded but rather javascript generated data from a webpage?
      Tried a get request as well as _IEBodyReadHTML but both seem to grab the code without the javascript generated data.
      $oHTTP = ObjCreate("winhttp.winhttprequest.5.1") $oHTTP.Open("GET", "link", False) $oHTTP.Send() $oReceived = $oHTTP.ResponseText $oStatusCode = $oHTTP.Status Global $DataArray[10][5] If $oStatusCode <> 200 Then Exit MsgBox(1, "Error", "Status Code <> 200") EndIf FileWrite(@ScriptDir & "\output.txt", $oReceived) ; //////// #include <IE.au3> Local $FullLink = "link" Local $oIE = _IECreate($FullLink, 0, 0) _IELoadWait($oIE) Local $sText = _IEBodyReadHTML($oIE) FileWrite(@ScriptDir & "\output.txt", $sText)  
    • FMS
      By FMS
      Hello,
      I try to get all the text from a news site around a subject.
      The first run I get all the text inside a attribute in an array.
      When i try to go back and reload another page it chrashes and think it because "$oIE.GoBack"
      I couldn't find anything in the help/forum around this subject.
      Do I need to reload the $oIE or something afther an X.GoBack?
      The error i got is :
      if $oTag2.GetAttribute("class") == "NewsDetail" Then if $oTag2^ ERROR I'm not shure why I got this error, maybe someone could explain?
      Also I'm open for some pointers in this test script because I'm pretty new in working whit the IE.UDF
      Maybe there is an simpler way to get the same results?

      test script:
      #include <IE.au3> #include <MsgBoxConstants.au3> #include <Array.au3> HotKeySet("{ESC}", "Terminate") Global $oIE = _IECreate("https://www.iex.nl/Zoeken/Nieuws.aspx?q=air%20france") ;get first subject Global $oLink1 = _IEGetObjById($oIE, "ctl00_ctl00_Content_LeftContent_NewsSearch_repNews_ctl00_linkNews") Sleep(500) _IEAction($oLink1, "click") Sleep(500) Global $oTags = _IETagNameGetCollection($oIE, "div") Global $aResults[1] For $oTag In $oTags if $oTag.GetAttribute("class") == "NewsDetail" Then _ArrayAdd($aResults, $oTag.innerTEXT) EndIf Next $aResults[0] = UBound($aResults) - 1 _ArrayDisplay($aResults, "Episodelist") ConsoleWrite($aResults[1] & @CRLF) $oIE.GoBack ;get second subject Global $oLink2 = _IEGetObjById($oIE, "ctl00_ctl00_Content_LeftContent_NewsSearch_repNews_ctl01_linkNews") Sleep(500) _IEAction($oLink2, "click") Sleep(500) Local $oTags2 = _IETagNameGetCollection($oIE, "div") Local $aResults2[1] For $oTag2 In $oTags2 if $oTag2.GetAttribute("class") == "NewsDetail" Then _ArrayAdd($aResults2, $oTag2.innerTEXT) EndIf Next $aResults2[0] = UBound($aResults2) - 1 ConsoleWrite($aResults2[1] & @CRLF) Func Terminate() _IEQuit($oIE) Exit EndFunc ;==>Terminate  
    • SkysLastChance
      By SkysLastChance
      I am having a hard time understanding why this is not working. I was hoping some one could help explain it to me. 
      $tags = $oIE.document.GetElementsByTagName("input") For $tag in $tags $class_value = $tag.GetAttribute("class") If string($class_value) = "fTs-p3298-l0 wplEditControl" Then $target = $tag ExitLoop EndIF Next MsgBox(0,"",$target) If $target = "fTs-p3298-l0 wplEditControl" THEN MsgBox(0,"","itworked") I have tried 
       MsgBox(0,"",$target.Attribute)  MsgBox(0,"",$target.Value)  MsgBox(0,"",$target.InnerText) I would expect to see this in the msgbox
      fTs-p3298-l0 wplEditControl  
    • SkysLastChance
      By SkysLastChance
      I am trying to grab the id "in2xk_26" however it the characters before the underscore always change. (in the name too)
      Is there a way I can find a id or name by the last 3 charcters?
      Using something like "stringright?"
      So I would want to search for just "_26" in this case.
      This is assuming that there are no other _26
       

      #include <Excel.au3> #include <IE.au3> #include <GUIConstantsEx.au3> Global $iMousespeed = 25,$target = "",$TagName = "",$Value = "",$Atrribute = "" $oIE = _IEAttach("MEDITECH") $TagName = "input" $Value = "in2xk_26" $Attribute = "id" $tags = $oIE.document.GetElementsByTagName($TagName) ;TagName ------ MAKE SURE TO NAME THESE For $tag in $tags $class_value = $tag.GetAttribute($Attribute) ;Attribute ------ MAKE SURE TO NAME THESE If string($class_value) = $Value Then ;Value ------ MAKE SURE TO NAME THESE $target = $tag $iScreenX = _IEPropertyGet($target, "screenx") $iScreenY = _IEPropertyGet($target, "screeny") $iWidth = _IEPropertyGet($target, "width") $iHeight = _IEPropertyGet($target, "height") $oMouseCords = MouseMove($iScreenX + $iWidth / 2, $iScreenY + $iHeight / 2, $iMousespeed) MouseClick($MOUSE_CLICK_LEFT) ExitLoop EndIf Next  
       
    • SkysLastChance
      By SkysLastChance
      I was wondering how I read data inside of a iframe. I would like to be able to click something inside a iframe. However, I can't even find the tag. 
      The id and name change all the time so I can't use those. 
      The code highlighted in blue is what I am trying to click. (second picture)
      Here is what I have tried.
      $target = "" $tags = $oIE.document.GetElementsByTagName("div") For $tag in $tags $class_value = $tag.GetAttribute("class") If string($class_value) = "s_92 altstyle s_93 s_94" Then $target = $tag ConsoleWrite("Tag Found " & $target.outerText&@CRLF) ExitLoop EndIf Next  

       
       

×