Jump to content

InetGet was working previously, but not extracting full html


Recommended Posts

I have an AutoIT script It monitors 2 websites for content that applys to me and the services that I provide. One site is : www.Freelancer.com The other: www.PeoplePerHour.com Both sites publish new jobs on their site hourly or so. My AutoIT app, will view those sites and present new jobs to me in a grid that pops up on my screen. Lately, the app has stopped showing me any jobs from PeoplePerHour.

 

For freelancer.com,  Inetget is giving full html but for peopleperhour, now its not coming.

Func _CheckPPH()
    Local Static $hTimer = 0
    Local Static $hDownload = 0
    Local $aTitlesandUrls = 0
    Local Static $sTempFile = ""
    If $hTimer = 0 Then $hTimer = TimerInit()
    If $hDownload = 0 Then
        $sTempFile = _WinAPI_GetTempFileName(@TempDir)
        ConsoleWrite("Checking PPH..." & @CRLF)
        ConsoleWrite(">Downloading..." & @CRLF)
;~         $hDownload = InetGet("http://www.peopleperhour.com/freelance-jobs", $sTempFile, $INET_FORCERELOAD, $INET_DOWNLOADBACKGROUND)
        $hDownload = InetGet("http://www.peopleperhour.com/freelance-jobs", $sTempFile, $INET_FORCERELOAD)
;~         Return 0
    EndIf
;~     Sleep(30)
;~     Local $isCompleted = InetGetInfo($hDownload, $INET_DOWNLOADCOMPLETE)
;~     Local $isError = InetGetInfo($hDownload, $INET_DOWNLOADERROR)
;~     Sleep(30)
;~     If TimerDiff($hTimer) > 3000 And $isError Then
;~         ConsoleWrite("!PPH Fail" & @CRLF)
;~         InetClose($hDownload)
;~         $hDownload = 0
;~         Return 0
;~     EndIf
;~     Sleep(30)
    Local $Show = 0
;~     If TimerDiff($hTimer) > 3000 And $isCompleted Then
    If $hDownload > 0 Then
        ConsoleWrite("+Downloaded..." & @CRLF)
        Local $sPPHHtml = FileRead($sTempFile)
        $aTitlesandUrls = _StringBetween($sPPHHtml, '"title">' & @LF, 'time>')
;~         _ArrayDisplay($aTitlesandUrls)
        Local $aPPH[0][4]
        Local $sTitle = ""
        Local $sUrl = ""
        Local $sID = ""
        Local $sDate = ""
        Local $iRet=0
        Sleep(30)
        For $i = 0 To UBound($aTitlesandUrls) - 1
            $sTitle = _StringBetween($aTitlesandUrls[$i], '<a title="', '" class')
            $sUrl = _StringBetween($aTitlesandUrls[$i], 'href="', '">')
            $sDate = _GetDate($aTitlesandUrls[$i])
            If IsArray($sTitle) And IsArray($sUrl) Then
                $sID = _GetID($sUrl[0])
;~                 _ArrayAdd($aPPH, $sDate & "|" & $sTitle[0] & "|" & $sUrl[0] & "|" & $sID)
                $iRet = _BuildPopupsPPH($sID, $sDate, "PPH: " & $sTitle[0], $sUrl[0])
                If $iRet Then $Show+=1
            EndIf
        Next

        Sleep(30)
;~         If $Show > 0 Then ShowLatestJobs()
;~         _ArrayDisplay($aPPH)
        FileDelete($sTempFile)
        InetClose($hDownload)
        $hDownload = 0
        $hTimer = 0
        Return $Show
    EndIf
    Sleep(30)
EndFunc   ;==>_CheckPPH

Link to post
Share on other sites

Is this topic related to your previous topic?  If so, why did you start another topic?  Also, why didn't you answer my question in the previous topic?  Is it because you knew that harvesting data from the sites that you referred to above is prohibited by their terms of use which would also mean that helping you to do so here would be prohibited?

 

Edited by TheXman
Typo
Link to post
Share on other sites
9 minutes ago, Jahar said:

For previous one, you have asked me to go thru scripts given as examples.

No, I asked you why were asking for help to access a non-existent domain.

 

 

Link to post
Share on other sites
  • Moderators

@Jahar As stated above (and in the other thread) both sites you specify have verbiage in their TOS that states scraping or crawling of their site pages is not permitted. Case closed, please do not open another thread on this topic.

Edited by JLogan3o13

"Profanity is the last vestige of the feeble mind. For the man who cannot express himself forcibly through intellect must do so through shock and awe" - Spencer W. Kimball

How to get your question answered on this forum!

Link to post
Share on other sites
Guest
This topic is now closed to further replies.
  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By JohnnyTries
      Hi Fellow Automators,
      Long time listener, first time caller. I've resisted posting on the forums as long as possible for fear of public lynching, but I'm stuck and could really use some help.
      Note: If this is the wrong side of the forum for this topic, I apologize.
      I've built a GUI and script to make our lab data collection easier. The app has a number of input boxes and a 'record' button. The user fills out the input boxes with various notes and then presses 'record', which in turn presses 'record' on two other, separate apps simultaneously, pulls those recordings together into one folder, and then takes the text from the input boxes and adds it in a new row at the bottom of an existing .xlsx spreadsheet. Everything works great, except that every time I open the app to start collecting data for the day, the 'headers' for the $aArray are added to a new row and then the text is added below it. Now, if I don't close the app between collections, subsequent 'recordings' are added to the spreadsheet as expected. If I close the app and open it, the 'first' recording of the day adds the headers to a new row. I don't need new 'header' info because I've already got that in row 1 of the spreadsheet.
      If someone could tell me where I'm !#$%ing up, I would greatly appreciate it. 
       
      #RequireAdmin #include <ButtonConstants.au3> #include <GUIConstantsEx.au3> #include <MsgBoxConstants.au3> #include <WindowsConstants.au3> #include <String.au3> #include <Process.au3> #include <FileConstants.au3> #include <WinAPIFiles.au3> #include <DirConstants.au3> #include <Array.au3> #include <AutoItConstants.au3> #include <File.au3> #include <WinAPIShPath.au3> #include <Excel.au3> ;Declaring the $aArray and location of the speadsheet at the top of the script Local $aArray[1][9] = [["TestID","DateTimeStamp","Tamb_C","BGTemp_C", "GasType", "TrueFlow_slm", "Lens-BGDist_in", "Lens-LeakDist_In", "AddNotes"]] Local $sDataFilePath = @ScriptDir & "\Notes\DualCaptureNotes.xlsx" Func CaptureVideo() Global $TestID = GuiCtrlRead($TestIDInput) Global $timestamp = @YEAR & "-" & @MON & "-" & @MDAY & "-" & @HOUR & "-" & @MIN & "-" & @SEC Global $Cam1Dir = "C:\Archive" Global $Cam2Dir = "C:\ALD" ;Capture Cam1 Data WinActivate("Cam1 App") ControlFocus("Cam1 App", "Save",'WindowsForms10.Window.8.app.0.2099316_r7_ad114') ControlSend("Cam1 App", "Save", 'WindowsForms10.Window.8.app.0.2099316_r7_ad114', "{SPACE}") ;Capture Cam2 Data WinActivate("Cam2 App") ControlClick("Cam2 App", "", 'WindowsForms10.Window.8.app.0.1b0ed41_r7_ad122', '', 1, 10, 10) ;Wait to ensure data files have been fully written to their default locations Sleep(2000) ;Self-explanatorily named functions MoveData() RecordNotesToArray() RecordArraytoExcel() WinActivate("DualCapture") EndFunc ;==>CaptureVideo ;Skipping ahead to the .xlsx part.... Func RecordNotesToArray() _ArrayAdd($aArray, GUICtrlRead($TestIDInput) & "|" & $timestamp & "|" & GUICtrlRead($Tamb_CInput) & "|" & GUICtrlRead($BGTemp_CInput) & "|" & GUICtrlRead($GasTypeInput) & "|" & GUICtrlRead($TrueFlow_slmInput) & "|" & GUICtrlRead($Dist_BG_inInput) & "|" & GUICtrlRead($Dist_Leak_inInput) & "|" & GUICtrlRead($AddNotesInput)) EndFunc ;==>RecordNotesToArray() Func RecordArraytoExcel() Local $oExcel = _Excel_Open() Local $oWorkBook If Not FileExists($sDataFilePath) Then $oWorkBook = _Excel_BookNew($oExcel) Else $oWorkBook = _Excel_BookOpen($oExcel, $sDataFilePath) EndIf $oWorkBook.Worksheets("DataTable").Columns("A:I").AutoFit $LastRow = $oWorkbook.ActiveSheet.Range("A1").SpecialCells($xlCellTypeLastCell).Row $Rowrange = "A"&$LastRow+1 Consolewrite($Rowrange & @crlf) _Excel_RangeWrite($oWorkbook, $oWorkbook.Activesheet, $aArray, $Rowrange) If FileExists($sDataFilePath) Then _Excel_BookSave($oWorkBook) Else _Excel_BookSaveAs($oWorkBook, $sDataFilePath) EndIf _Excel_BookClose($oWorkBook) _Excel_Close($oExcel) EndFunc ;==>RecordArrayToExcel() I appreciate your time and any help you can provide.
       
      Best,
      Johnny
       
    • By LeeSG
      (If translated with a translator, it may be written a little awkwardly. I would be grateful if you could understand my situation)
      As mentioned in the title, the icon file, which was applied well when compiled with exe, does not apply when compiled with a3x, and is displayed as the default autoit icon.
       
      #AutoIt3Wrapper_Icon=icon.ico
       
      I specified the icon file at the top, but why can't it be applied? Is there any way to compile with a3x including icons?
    • By HoangDung
      This is the function that returns the result from cmd, initially i connect to the network wait then i make a call to the above _GetDOSOutput($sCommand) function i want to wait 1 period of time netsh wlan connect name="name" actually but after starting to execute the netsh wlan show interfaces command i tried adding a timeout command it seems to have ignored the timeout command?
      #include <WindowsConstants.au3> #include <Constants.au3> Func _GetDOSOutput($sCommand) Run('"' & @ComSpec & '" /c ' & $sCommand, "", @SW_HIDE, $STDERR_CHILD + $STDOUT_CHILD) Local $sOutput = '' Local $iPID = Run('"' & @ComSpec & '" /c ' & $sCommand, "", @SW_HIDE, $STDERR_CHILD + $STDOUT_CHILD) While 1 $sOutput &= StdoutRead($iPID, False, False) If @error Then ExitLoop EndIf Sleep(10) WEnd Return $sOutput EndFunc Local $sCommand= 'netsh wlan delete profile name="wait" & netsh wlan connect name="name" interface="Wi-fi" & netsh wlan show interfaces' MsgBox(0,0,_GetDOSOutput($sCommand))  
    • By jiaojiaodubai
      I am happy to see that the latest version of AutoIt3 can already be searched, downloaded and installed in the Windows 10+built-in package manager (Winget). Therefore, we can automatically install AutoIt3 itself by writing Powershell scripts, which helps to quickly deploy the working environment of AutoIt3 on new machines.

      Now, although AutoIt3 can be found and installed in Winget, its dedicated editor, SciTE4AutoIt, does not support this. I submitted a package request for SciTE4AutoIt to the Winget development team on GitHub and provided a download link. Then its developer replied to me:
      In other words, Winget developers believe that our website prevents Winget from directly downloading the .exe file of SciTE4AutoIt.

      In fact, I don't know much about the technical details he said, but I think we can make some changes to make SciTE4AutoIt more accessible.
       
    • By D3fr0s7
      I'm trying to make one tray item delete another, but when I do this, all tray items that were created after the deleted item don't work as intended, as if their controlID's were all shifted down one value, and their corresponding tray items now (after deletion) run the code of the tray item before it. Am I missing something? Is there a better way to accomplish what I'm trying to do?
      #include <TrayConstants.au3> #include <Array.au3> HotKeySet ( "{ESC}", "Abort" ) Opt ( "TrayMenuMode", 3 ) TraySetState($TRAY_ICONSTATE_SHOW) ; Show the tray menu. Global $aTray[8] ; Defines array to hold tray items. $aTray[0] = TrayCreateItem ( "Test 1 (Name Test 5)" ) $aTray[1] = TrayCreateItem ( "Test 2 (Delete Test 5)" ) $aTray[2] = TrayCreateItem ( "Test 3 (Restore Test 5)" ) $aTray[3] = TrayCreateItem ( "Test 4 (Check if Test 5 is blank or space)" ) $aTray[4] = TrayCreateItem ( "Test 5 Delete Me" ) $aTray[5] = TrayCreateItem ( "Test 6 (Check Test 5 Text)" ) $aTray[6] = TrayCreateItem ( "Test 7 (Read Values)" ) $aTray[7] = TrayCreateItem ( "Test 8 (Count Blanks)" ) While 1 Switch TrayGetMsg() Case $aTray[0] ; "Test 1" Change Test 5 Text. If TrayItemGetText ( $aTray[0] ) <> "" Then Global $TrayText = InputBox ( "Test", "Choose text for Test 5", "Test 5 Delete Me" ) TrayItemSetText ( $aTray[4], $TrayText) EndIf Case $aTray[1] ; "Test 2" Deletes "Test 5". If TrayItemGetText ( $aTray[1] ) <> "" Then Global $TrayDeletedName = TrayItemGetText ( $aTray[4] ) TrayItemDelete ( $aTray[4] ) _ArrayInsert ( $aTray, 4 ) EndIf Case $aTray[2] ; "Test 3" Restores "Test 5". If TrayItemGetText ( $aTray[2] ) <> "" Then $aTray[4] = TrayCreateItem ( $TrayDeletedName ) EndIf Case $aTray[3] ; "Test 4" Check if Test 5 value is blank, space, or filled. If TrayItemGetText ( $aTray[3] ) <> "" Then If TrayItemGetText ( $aTray[4] ) = "" Then MsgBox ( 0, "Test", "Test 5 is blank" ) ElseIf TrayItemGetText ( $aTray[4] ) = " " Then MsgBox ( 0, "Test", "Test 5 is not blank (space)" ) Else MsgBox ( 0, "Test", "Test 5 is assigned a value" ) EndIf EndIf Case $aTray[4] ; "Test 5" (Item to test for, during, and after deletion). If TrayItemGetText ( $aTray[4] ) <> "" Then MsgBox ( 0, "Test", "I'm here!" ) EndIf Case $aTray[5] ; "Test 6" Displays Text from Test 5 item. If TrayItemGetText ( $aTray[5] ) <> "" Then $Test5Text = TrayItemGetText ( $aTray[4] ) MsgBox ( 0, "Test", "Test 5 Text: " & $Test5Text ) EndIf Case $aTray[6] ; "Test 7" Displays all item values. If TrayItemGetText ( $aTray[6] ) <> "" Then MsgBox ( 0, "Test", "$aTray[0]: " & $aTray[0] & @CRLF & _ "$aTray[1]: " & $aTray[1] & @CRLF & _ "$aTray[2]: " & $aTray[2] & @CRLF & _ "$aTray[3]: " & $aTray[3] & @CRLF & _ "$aTray[4]: " & $aTray[4] & @CRLF & _ "$aTray[5]: " & $aTray[5] & @CRLF & _ "$aTray[6]: " & $aTray[6] & @CRLF & _ "$aTray[7]: " & $aTray[7] & @CRLF ) EndIf Case $aTray[7] ; "Test 8" Counts all blanks in tray values. If TrayItemGetText ( $aTray[7] ) <> "" Then Global $blankCount = _ArrayFindAll ( $aTray, "" ) If $blankCount = -1 Then If @error = 6 Then MsgBox ( 0, "Test", "Error, No blanks present") EndIf Else MsgBox ( 0, "Test", "# of blanks: " & $blankCount ) EndIf EndIf EndSwitch WEnd Func Abort() Exit EndFunc Here is a test script I created to try to troubleshoot the problem on my own, with no luck. pay specific attention to "Test 2" ($aTray[1]), "Test 5" ($aTray[4]), and how every tray item after "Test 5" ($aTray[4]) behaves after deletion. Clicking "Test 2" will delete tray item "Test 5", after deletion every item runs the code of the tray item that was established before it (ex. "Test 3" and "Test 4" run their respective code, "Test 5" no longer exists, "Test 6" runs "Test 7", "Test 7" runs "Test 8"), and the last item ("Test 8" $aTray[7]) has no effect when the tray item is clicked. 

      I understand that deleting the tray item changes the controlID, but I don't know in what way it does, and therefore how I can fix it to be able to achieve what I want it to. I appreciate any help or guidance with this problem.

      To clarify, what I'm ultimately trying to do is create a 'while' loop with switch case functions that can exist without necessarily being linked to a tray item, so that I can add and delete them at liberty using the script's functions, without having to differentiate switch case functions with if functions (if $aTray[x] exists, then use this set of switch case functions, etc.). 

      Please, I am in pain. Water come school me again pls
×
×
  • Create New...