Jump to content

[Help] HTTP requests

xanathos
 Share

Recommended Posts

xanathos Seeker

xanathos

Posted
Posted

Hello folks,

I'm running on a problem lately, I'm trying to send a request on a website (so far no problem), but I'm stuck on trying to retrieve all the links on the next page that would contain profile/XXXXXXX; sending requests (to visit them) to each of them one after one, and once visited, move to page 2.

Any example or help would be greatly appreciated, thanks.

xanathos Seeker

xanathos

Posted
  • Author
Posted

Here is what I have tried:

Func Main()
; The data to be sent
$sPD = 'username=Admin&password=password&remember=on'
$search = 'age%5Bmin%5D=18&age%5Bmax%5D=30&age_step=1&by=distance&country=&region=&distance%5Bmin%5D=&distance%5Bmax%5D=50&distance_step=10&pseudo=&sex=1&size%5Bmin%5D=120&size%5Bmax%5D=220&size_step=5&weight%5Bmin%5D=30&weight%5Bmax%5D=115&weight_step=5&mandatory%5B%5D=shape&mandatory%5B%5D=eyes_color&mandatory%5B%5D=origins&mandatory%5B%5D=hair_color&mandatory%5B%5D=hair_size&mandatory%5B%5D=hair_style&mandatory%5B%5D=styles&mandatory%5B%5D=character&mandatory%5B%5D=features&mandatory%5B%5D=diet&mandatory%5B%5D=favourite_food&mandatory%5B%5D=tobacco&mandatory%5B%5D=alcohol'

; Creating the object
$oHTTP = ObjCreate("winhttp.winhttprequest.5.1")
$oHTTP.Open("POST", "http://127.0.0.1/auth/login", False)
$oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
$oHTTP.Send($sPD)
sleep(1000)
$oHTTP.Open("POST", "http://127.0.0.1/mySearch", False)
$oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
$oHTTP.Send($search)


$oHTTP.Open("GET", "http://127.0.0.1/mySearch", False)
$oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
$oHTTP.Send($search)


; Download the body response if any, and get the server status response code.
$oReceived = $oHTTP.ResponseText
$oStatusCode = $oHTTP.Status
$aLinks = StringRegExp($oReceived, '<\s*?a href=".*?profile/\w+;".*?>', 3)

If $oStatusCode <> 200 then
 MsgBox(4096, "Response code", $oStatusCode)
EndIf

; Saves the body response regardless of the Response code 
 $file = FileOpen("Received.html", 2) ; The value of 2 overwrites the file if it already exists
 $IDs = FileOpen("IDs.txt", 2) ;
 FileWrite($file, $oReceived)
 FileWrite($IDs, $aLinks)
 FileClose($file)
 FileClose($IDs)
 EndFunc

I checked the IDs.txt file and there's only "1" in it...

GMK Universalist

GMK

Posted
Posted (edited)

$aLinks will be an array containing all the links (if the SRE works properly).

Insert

#include <Array.au3>

at the beginning of the script, then after

$aLinks = StringRegExp($oReceived, '<\s*?a href=".*?profile/\w+;".*?>', 3)

insert

_ArrayDisplay($aLinks)

Then you can loop through each value of the array and do what you need with each link.  (FileWriteLine, perhaps?)

Edit: Perhaps you can try this, as I'm unable to test it:

#include <Array.au3>
#include <File.au3>

Func Main()
    ; The data to be sent
    $sPD = 'username=Admin&password=password&remember=on'
    $search = 'age%5Bmin%5D=18&age%5Bmax%5D=30&age_step=1&by=distance&country=&region=&distance%5Bmin%5D=&distance%5Bmax%5D=50&distance_step=10&pseudo=&sex=1&size%5Bmin%5D=120&size%5Bmax%5D=220&size_step=5&weight%5Bmin%5D=30&weight%5Bmax%5D=115&weight_step=5&mandatory%5B%5D=shape&mandatory%5B%5D=eyes_color&mandatory%5B%5D=origins&mandatory%5B%5D=hair_color&mandatory%5B%5D=hair_size&mandatory%5B%5D=hair_style&mandatory%5B%5D=styles&mandatory%5B%5D=character&mandatory%5B%5D=features&mandatory%5B%5D=diet&mandatory%5B%5D=favourite_food&mandatory%5B%5D=tobacco&mandatory%5B%5D=alcohol'

    ; Creating the object
    $oHTTP = ObjCreate("winhttp.winhttprequest.5.1")
    $oHTTP.Open("POST", "http://127.0.0.1/auth/login", False)
    $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
    $oHTTP.Send($sPD)
    Sleep(1000)
    $oHTTP.Open("POST", "http://127.0.0.1/mySearch", False)
    $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
    $oHTTP.Send($search)

    $oHTTP.Open("GET", "http://127.0.0.1/mySearch", False)
    $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
    $oHTTP.Send($search)

    ; Download the body response if any, and get the server status response code.
    $oReceived = $oHTTP.ResponseText
    $oStatusCode = $oHTTP.Status
    $aLinks = StringRegExp($oReceived, '<\s*?a href="(.*?profile/\w+ ;)".*?>', 3)
    _ArrayDisplay($aLinks, "Links")

    If $oStatusCode <> 200 Then MsgBox(4096, "Response code", $oStatusCode)

    ; Saves the body response regardless of the Response code
    $file = FileOpen("Received.html", 2) ; The value of 2 overwrites the file if it already exists
    FileWrite($file, $oReceived)
    FileClose($file)
    _FileWriteFromArray("IDs.txt", $aLinks)
EndFunc
Edited by GMK
xanathos Seeker

xanathos

Posted
  • Author
Posted (edited)

 

$aLinks will be an array containing all the links (if the SRE works properly).

Insert

#include <Array.au3>

at the beginning of the script, then after

$aLinks = StringRegExp($oReceived, '<\s*?a href=".*?profile/\w+;".*?>', 3)

insert

_ArrayDisplay($aLinks)

Then you can loop through each value of the array and do what you need with each link.  (FileWriteLine, perhaps?)

Edit: Perhaps you can try this, as I'm unable to test it:

#include <Array.au3>
#include <File.au3>

Func Main()
    ; The data to be sent
    $sPD = 'username=Admin&password=password&remember=on'
    $search = 'age%5Bmin%5D=18&age%5Bmax%5D=30&age_step=1&by=distance&country=&region=&distance%5Bmin%5D=&distance%5Bmax%5D=50&distance_step=10&pseudo=&sex=1&size%5Bmin%5D=120&size%5Bmax%5D=220&size_step=5&weight%5Bmin%5D=30&weight%5Bmax%5D=115&weight_step=5&mandatory%5B%5D=shape&mandatory%5B%5D=eyes_color&mandatory%5B%5D=origins&mandatory%5B%5D=hair_color&mandatory%5B%5D=hair_size&mandatory%5B%5D=hair_style&mandatory%5B%5D=styles&mandatory%5B%5D=character&mandatory%5B%5D=features&mandatory%5B%5D=diet&mandatory%5B%5D=favourite_food&mandatory%5B%5D=tobacco&mandatory%5B%5D=alcohol'

    ; Creating the object
    $oHTTP = ObjCreate("winhttp.winhttprequest.5.1")
    $oHTTP.Open("POST", "http://127.0.0.1/auth/login", False)
    $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
    $oHTTP.Send($sPD)
    Sleep(1000)
    $oHTTP.Open("POST", "http://127.0.0.1/mySearch", False)
    $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
    $oHTTP.Send($search)

    $oHTTP.Open("GET", "http://127.0.0.1/mySearch", False)
    $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
    $oHTTP.Send($search)

    ; Download the body response if any, and get the server status response code.
    $oReceived = $oHTTP.ResponseText
    $oStatusCode = $oHTTP.Status
    $aLinks = StringRegExp($oReceived, '<\s*?a href="(.*?profile/\w+ ;)".*?>', 3)
    _ArrayDisplay($aLinks, "Links")

    If $oStatusCode <> 200 Then MsgBox(4096, "Response code", $oStatusCode)

    ; Saves the body response regardless of the Response code
    $file = FileOpen("Received.html", 2) ; The value of 2 overwrites the file if it already exists
    FileWrite($file, $oReceived)
    FileClose($file)
    _FileWriteFromArray("IDs.txt", $aLinks)
EndFunc

 

Here is my code:

; Download the body response if any, and get the server status response code.
$oReceived = $oHTTP.ResponseText
$oStatusCode = $oHTTP.Status
$aLinks = StringRegExp($oReceived, '<a title="View profile" href="http://127.0.0.1/profile/(.*?)" >', 3)
_ArrayDisplay($aLinks)

If $oStatusCode <> 200 then
 MsgBox(4096, "Response code", $oStatusCode)
EndIf

; Saves the body response regardless of the Response code 
 $file = FileOpen("Received.html", 2) ; The value of 2 overwrites the file if it already exists
 $IDs = FileOpen("IDs.txt", 2) ;
 FileWrite($file, $oReceived)
 FileClose($file)
 _FileWriteFromArray($IDs, $aLinks)
  FileClose($IDs)
 EndFunc

When I run it, it won't even show the arrays and the txt file is empty.

Edited by xanathos
xanathos Seeker

xanathos

Posted
  • Author
Posted

Tried it this way:

$aLinks = StringRegExp($oReceived, '.?profile/(\d+)', 3)
_ArrayDisplay($aLinks, "test")

Now, problem is that it will return only one occurrence of the first ID found.

Shouldn't I convert the html file into a txt or source file first?

xanathos Seeker

xanathos

Posted
  • Author
Posted (edited)

Ok, managed to make it work, problem was that since it's ajax, it won't return the profile ID as it should be but in raw so it was a var:

$alinks = _stringbetween($oReceived, '"url":"\/index.php\/profile\/', '\')
_ArrayDisplay($aLinks, "test")

Now I'm coming across another problem:

$oHTTP.Open("GET", "http://127.0.0.1/profile/" & $aLinks[0], False)
$oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
$oHTTP.Send()
sleep(250)

$oReceived = $oHTTP.ResponseText
$oStatusCode = $oHTTP.Status

If $oStatusCode = 200 then
 MsgBox(0, "Response code", "Profile " & $aLinks[0] & " visited!")
EndIf

Each page returns several (usually 14 but it can change) IDs, the above example does work, I wanna visit all the IDs links with a 3 seconds delay between each visit, and move to next page, but do I really have to do that for each array, or is there a simpler way?

Edited by xanathos
hiho Wayfarer

hiho

Posted
Posted

I'm curious why do you need such personal information like age, sex, location etc...

$search = 'age%5Bmin%5D=18&age%5Bmax%5D=30&age_step=1&by=distance&country=&region=&distance%5Bmin%5D=&distance%5Bmax%5D=50&distance_step=10&pseudo=&sex=1&size%5Bmin%5D=120&size%5Bmax%5D=220&size_step=5&weight%5Bmin%5D=30&weight%5Bmax%5D=115&weight_step=5&mandatory%5B%5D=shape&mandatory%5B%5D=eyes_color&mandatory%5B%5D=origins&mandatory%5B%5D=hair_color&mandatory%5B%5D=hair_size&mandatory%5B%5D=hair_style&mandatory%5B%5D=styles&mandatory%5B%5D=character&mandatory%5B%5D=features&mandatory%5B%5D=diet&mandatory%5B%5D=favourite_food&mandatory%5B%5D=tobacco&mandatory%5B%5D=alcohol'
xanathos Seeker

xanathos

Posted
  • Author
Posted

 

I'm curious why do you need such personal information like age, sex, location etc...

$search = 'age%5Bmin%5D=18&age%5Bmax%5D=30&age_step=1&by=distance&country=&region=&distance%5Bmin%5D=&distance%5Bmax%5D=50&distance_step=10&pseudo=&sex=1&size%5Bmin%5D=120&size%5Bmax%5D=220&size_step=5&weight%5Bmin%5D=30&weight%5Bmax%5D=115&weight_step=5&mandatory%5B%5D=shape&mandatory%5B%5D=eyes_color&mandatory%5B%5D=origins&mandatory%5B%5D=hair_color&mandatory%5B%5D=hair_size&mandatory%5B%5D=hair_style&mandatory%5B%5D=styles&mandatory%5B%5D=character&mandatory%5B%5D=features&mandatory%5B%5D=diet&mandatory%5B%5D=favourite_food&mandatory%5B%5D=tobacco&mandatory%5B%5D=alcohol'

 

I don't need them, that's just to the post data for the search fields...

GMK Universalist

GMK

Posted
Posted (edited)

Something like this?

For $iLink = 0 To UBound($aLinks) - 1
    $oHTTP.Open("GET", "http://127.0.0.1/profile/" & $aLinks[$iLink], False)
    $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
    $oHTTP.Send()
    Sleep(250)

    $oReceived = $oHTTP.ResponseText
    $oStatusCode = $oHTTP.Status

    If $oStatusCode = 200 Then
        MsgBox(0, "Response code", "Profile " & $aLinks[$iLink] & " visited!")
    EndIf
 
    Sleep(3000) ; Three-second delay
Next

Now I'm coming across another problem:

$oHTTP.Open("GET", "http://127.0.0.1/profile/" & $aLinks[0], False)
$oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
$oHTTP.Send()
sleep(250)

$oReceived = $oHTTP.ResponseText
$oStatusCode = $oHTTP.Status

If $oStatusCode = 200 then
 MsgBox(0, "Response code", "Profile " & $aLinks[0] & " visited!")
EndIf

Each page returns several (usually 14 but it can change) IDs, the above example does work, I wanna visit all the IDs links with a 3 seconds delay between each visit, and move to next page, but do I really have to do that for each array, or is there a simpler way?

Edited by GMK
xanathos Seeker

xanathos

Posted
  • Author
Posted (edited)

@GMK, you're using $iLink when starting the loop, not $i (look at the msgbox line).

 

Good notice!

Here is the code of my main function so far:

Func Main()
; The data to be sent
$sPD = 'username=admin&password=password&remember=on'
$search = 'age%5Bmin%5D=18&age%5Bmax%5D=30&age_step=1&by=distance&country=&region=&distance%5Bmin%5D=&distance%5Bmax%5D=50&distance_step=10&pseudo=&sex=1&size%5Bmin%5D=120&size%5Bmax%5D=220&size_step=5&weight%5Bmin%5D=30&weight%5Bmax%5D=115&weight_step=5&mandatory%5B%5D=shape&mandatory%5B%5D=eyes_color&mandatory%5B%5D=origins&mandatory%5B%5D=hair_color&mandatory%5B%5D=hair_size&mandatory%5B%5D=hair_style&mandatory%5B%5D=styles&mandatory%5B%5D=character&mandatory%5B%5D=features&mandatory%5B%5D=diet&mandatory%5B%5D=favourite_food&mandatory%5B%5D=tobacco&mandatory%5B%5D=alcohol'

; Creating the object
$oHTTP = ObjCreate("winhttp.winhttprequest.5.1")
$oHTTP.Open("POST", "http://127.0.0.1//auth/login", False)
$oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
$oHTTP.Send($sPD)

$oHTTP.Open("POST", "http://127.0.0.1/mySearch", False)
$oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
$oHTTP.Send($search)

$oHTTP.Open("GET", "http://127.0.0.1/mySearch", False)
$oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
$oHTTP.Send()


; Download the body response if any, and get the server status response code.
$oReceived = $oHTTP.ResponseText
$oStatusCode = $oHTTP.Status


$aLinks = _stringbetween($oReceived, '"url":"\/index.php\/profile\/', '\')
_ArrayDisplay($aLinks, "test")
sleep(250)
   
For $iLink = 0 To UBound($aLinks) - 1
    $oHTTP.Open("GET", "http://127.0.0.1/profile/" & $aLinks[$iLink], False)
    $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
    $oHTTP.Send()
    Sleep(250)

    $oReceived = $oHTTP.ResponseText
    $oStatusCode = $oHTTP.Status

    If $oStatusCode = 200 Then
        GUICtrlCreateListViewItem(_NowTime(5) & "|" & $aLinks[$iLink] & "| Visited", $log)
    EndIf
 
    Sleep(3000) ; Three-second delay
 Next

_GUICtrlListView_DeleteAllItems($log) 
sleep(250)
$oHTTP.Open("GET", "http://127.0.0.1/mySearch/ajax_loadPages/?page=2", False)
$oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
$oHTTP.Send()
 
$oReceived = $oHTTP.ResponseText
$oStatusCode = $oHTTP.Status


$aLinks = _stringbetween($oReceived, '"url":"\/index.php\/profile\/', '\')
_ArrayDisplay($aLinks, "test2")
sleep(250)

$lastp = FileOpen("Dernier.html", 2)
FileWrite($lastp, $oReceived)
FileClose($lastp)
;~  _FileWriteFromArray($IDs, $aLinks)
;~   FileClose($IDs)
 EndFunc

2 problems here:

1. As you can see, once all IDs have been visited I would like to move to next page and repeat the loop above, and do so, until last page is reached.

2. BUT there is no way to know the last page until the last page is reached. (For example if there are 16 pages of results, I don't know it until page 16 is reached, but if I try to access to page 17 manually, I can check for a string on the page that says "no results" and then we'd know there are no more pages, OR actually, from the first page, we know how many IDs the search has returned, and max IDs is 15 per page, so we can search for the IDs returned and divide that, by 15 to obtain the number of pages?)

Edited by xanathos
GMK Universalist

GMK

Posted
Posted

Thanks for that catch, hiho.

Perhaps you could declare a variable for your page number, then use a While loop and increment the variable and search the page for "no results" and if found, exit the loop.

xanathos Seeker

xanathos

Posted
  • Author
Posted

Thanks for that catch, hiho.

Perhaps you could declare a variable for your page number, then use a While loop and increment the variable and search the page for "no results" and if found, exit the loop.

 

That would make a loop (pages one) inside another loop (IDs one), right?

GMK Universalist

GMK

Posted
Posted

Something like this (though I'm not sure how Dernier.html should fit--inside or outside the While loop):

Func Main()
    ; The data to be sent
    $sPD = 'username=admin&password=password&remember=on'
    $search = 'age%5Bmin%5D=18&age%5Bmax%5D=30&age_step=1&by=distance&country=&region=&distance%5Bmin%5D=&distance%5Bmax%5D=50&distance_step=10&pseudo=&sex=1&size%5Bmin%5D=120&size%5Bmax%5D=220&size_step=5&weight%5Bmin%5D=30&weight%5Bmax%5D=115&weight_step=5&mandatory%5B%5D=shape&mandatory%5B%5D=eyes_color&mandatory%5B%5D=origins&mandatory%5B%5D=hair_color&mandatory%5B%5D=hair_size&mandatory%5B%5D=hair_style&mandatory%5B%5D=styles&mandatory%5B%5D=character&mandatory%5B%5D=features&mandatory%5B%5D=diet&mandatory%5B%5D=favourite_food&mandatory%5B%5D=tobacco&mandatory%5B%5D=alcohol'

    ; Creating the object
    $oHTTP = ObjCreate("winhttp.winhttprequest.5.1")
    $oHTTP.Open("POST", "http://127.0.0.1//auth/login", False)
    $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
    $oHTTP.Send($sPD)

    $oHTTP.Open("POST", "http://127.0.0.1/mySearch", False)
    $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
    $oHTTP.Send($search)

    $iPage = 1
    While 1
        $oHTTP.Open("GET", "http://127.0.0.1/mySearch/ajax_loadPages/?page=" & $iPage, False)
        $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
        $oHTTP.Send()

        ; Download the body response if any, and get the server status response code.
        $oReceived = $oHTTP.ResponseText
        If StringInStr($oReceived, "no results") Then
            ExitLoop
        EndIf
        $oStatusCode = $oHTTP.Status

        $aLinks = _StringBetween($oReceived, '"url":"\/index.php\/profile\/', '\')
        _ArrayDisplay($aLinks, "test")
        Sleep(250)

        For $iLink = 0 To UBound($aLinks) - 1
            $oHTTP.Open("GET", "http://127.0.0.1/profile/" & $aLinks[$iLink], False)
            $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
            $oHTTP.Send()
            Sleep(250)

            $oReceived = $oHTTP.ResponseText
            $oStatusCode = $oHTTP.Status

            If $oStatusCode = 200 Then
                GUICtrlCreateListViewItem(_NowTime(5) & "|" & $aLinks[$iLink] & "| Visited", $log)
            EndIf

            Sleep(3000) ; Three-second delay
        Next

        _GUICtrlListView_DeleteAllItems($log)
        Sleep(250)
        $iPage += 1
    WEnd
    
    $lastp = FileOpen("Dernier.html", 2)
    FileWrite($lastp, $oReceived)
    FileClose($lastp)
;~  _FileWriteFromArray($IDs, $aLinks)
;~   FileClose($IDs)
EndFunc   ;==>Main
xanathos Seeker

xanathos

Posted
  • Author
Posted (edited)

 

Something like this (though I'm not sure how Dernier.html should fit--inside or outside the While loop):

Func Main()
    ; The data to be sent
    $sPD = 'username=admin&password=password&remember=on'
    $search = 'age%5Bmin%5D=18&age%5Bmax%5D=30&age_step=1&by=distance&country=&region=&distance%5Bmin%5D=&distance%5Bmax%5D=50&distance_step=10&pseudo=&sex=1&size%5Bmin%5D=120&size%5Bmax%5D=220&size_step=5&weight%5Bmin%5D=30&weight%5Bmax%5D=115&weight_step=5&mandatory%5B%5D=shape&mandatory%5B%5D=eyes_color&mandatory%5B%5D=origins&mandatory%5B%5D=hair_color&mandatory%5B%5D=hair_size&mandatory%5B%5D=hair_style&mandatory%5B%5D=styles&mandatory%5B%5D=character&mandatory%5B%5D=features&mandatory%5B%5D=diet&mandatory%5B%5D=favourite_food&mandatory%5B%5D=tobacco&mandatory%5B%5D=alcohol'

    ; Creating the object
    $oHTTP = ObjCreate("winhttp.winhttprequest.5.1")
    $oHTTP.Open("POST", "http://127.0.0.1//auth/login", False)
    $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
    $oHTTP.Send($sPD)

    $oHTTP.Open("POST", "http://127.0.0.1/mySearch", False)
    $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
    $oHTTP.Send($search)

    $iPage = 1
    While 1
        $oHTTP.Open("GET", "http://127.0.0.1/mySearch/ajax_loadPages/?page=" & $iPage, False)
        $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
        $oHTTP.Send()

        ; Download the body response if any, and get the server status response code.
        $oReceived = $oHTTP.ResponseText
        If StringInStr($oReceived, "no results") Then
            ExitLoop
        EndIf
        $oStatusCode = $oHTTP.Status

        $aLinks = _StringBetween($oReceived, '"url":"\/index.php\/profile\/', '\')
        _ArrayDisplay($aLinks, "test")
        Sleep(250)

        For $iLink = 0 To UBound($aLinks) - 1
            $oHTTP.Open("GET", "http://127.0.0.1/profile/" & $aLinks[$iLink], False)
            $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
            $oHTTP.Send()
            Sleep(250)

            $oReceived = $oHTTP.ResponseText
            $oStatusCode = $oHTTP.Status

            If $oStatusCode = 200 Then
                GUICtrlCreateListViewItem(_NowTime(5) & "|" & $aLinks[$iLink] & "| Visited", $log)
            EndIf

            Sleep(3000) ; Three-second delay
        Next

        _GUICtrlListView_DeleteAllItems($log)
        Sleep(250)
        $iPage += 1
    WEnd
    
    $lastp = FileOpen("Dernier.html", 2)
    FileWrite($lastp, $oReceived)
    FileClose($lastp)
;~  _FileWriteFromArray($IDs, $aLinks)
;~   FileClose($IDs)
EndFunc   ;==>Main

 

the html file is just some kind of check, I sometimes check manually, just to make sure the body response is the good one, so yeah it's not needed, I will keep it outside of the loop. Gonna try that, after hitting the gym thanks. :P

Edited by xanathos
xanathos Seeker

xanathos

Posted
  • Author
Posted (edited)

 

Something like this (though I'm not sure how Dernier.html should fit--inside or outside the While loop):

Func Main()
    ; The data to be sent
    $sPD = 'username=admin&password=password&remember=on'
    $search = 'age%5Bmin%5D=18&age%5Bmax%5D=30&age_step=1&by=distance&country=&region=&distance%5Bmin%5D=&distance%5Bmax%5D=50&distance_step=10&pseudo=&sex=1&size%5Bmin%5D=120&size%5Bmax%5D=220&size_step=5&weight%5Bmin%5D=30&weight%5Bmax%5D=115&weight_step=5&mandatory%5B%5D=shape&mandatory%5B%5D=eyes_color&mandatory%5B%5D=origins&mandatory%5B%5D=hair_color&mandatory%5B%5D=hair_size&mandatory%5B%5D=hair_style&mandatory%5B%5D=styles&mandatory%5B%5D=character&mandatory%5B%5D=features&mandatory%5B%5D=diet&mandatory%5B%5D=favourite_food&mandatory%5B%5D=tobacco&mandatory%5B%5D=alcohol'

    ; Creating the object
    $oHTTP = ObjCreate("winhttp.winhttprequest.5.1")
    $oHTTP.Open("POST", "http://127.0.0.1//auth/login", False)
    $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
    $oHTTP.Send($sPD)

    $oHTTP.Open("POST", "http://127.0.0.1/mySearch", False)
    $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
    $oHTTP.Send($search)

    $iPage = 1
    While 1
        $oHTTP.Open("GET", "http://127.0.0.1/mySearch/ajax_loadPages/?page=" & $iPage, False)
        $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
        $oHTTP.Send()

        ; Download the body response if any, and get the server status response code.
        $oReceived = $oHTTP.ResponseText
        If StringInStr($oReceived, "no results") Then
            ExitLoop
        EndIf
        $oStatusCode = $oHTTP.Status

        $aLinks = _StringBetween($oReceived, '"url":"\/index.php\/profile\/', '\')
        _ArrayDisplay($aLinks, "test")
        Sleep(250)

        For $iLink = 0 To UBound($aLinks) - 1
            $oHTTP.Open("GET", "http://127.0.0.1/profile/" & $aLinks[$iLink], False)
            $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
            $oHTTP.Send()
            Sleep(250)

            $oReceived = $oHTTP.ResponseText
            $oStatusCode = $oHTTP.Status

            If $oStatusCode = 200 Then
                GUICtrlCreateListViewItem(_NowTime(5) & "|" & $aLinks[$iLink] & "| Visited", $log)
            EndIf

            Sleep(3000) ; Three-second delay
        Next

        _GUICtrlListView_DeleteAllItems($log)
        Sleep(250)
        $iPage += 1
    WEnd
    
    $lastp = FileOpen("Dernier.html", 2)
    FileWrite($lastp, $oReceived)
    FileClose($lastp)
;~  _FileWriteFromArray($IDs, $aLinks)
;~   FileClose($IDs)
EndFunc   ;==>Main

 

Worked great.

Next step where I'm stuck: I'm basically storing each IDs in a txt file with the timestamp they've been visited at, once (for example) 24 hours passed, that ID would be deleted from the txt file, this is not the problem yet, I'm just trying to be clear in my explanation; So the problem is that the file has to be read to check if the ID's string is there or not, if yes, it moves to the next ID link, if not, it will visit that ID, there's a conflict somewhere, but I'm not too sure where, I would guess it comes from the loops while ID gets written, it will be checked, so sometimes it will randomly appear as visited, although it was not, and vice-versa.

Here is a part of the code where I have implemented the string search and condition:

For $iLink = 0 To UBound($aLinks) - 1
   If StringInStr(FileRead("IDs.txt",1), $aLinks[$iLink]) Then
      MsgBox(0,"Test",$aLinks[$iLink] & " already visited!")
      Else
    $oHTTP.Open("GET", "http://127.0.0.1/profile/" & $aLinks[$iLink], False)
    $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded")
    $oHTTP.Send()
    Sleep(250)
    EndIf

    $oReceived = $oHTTP.ResponseText
    $oStatusCode = $oHTTP.Status

    If $oStatusCode = 200 Then
       GUICtrlSetState($log, $GUI_ENABLE)
    EndIf

$aLog = _GUICtrlListView_CreateArray($log, Default)
$hFile = FileOpen("IDs.txt", 1) ; 1 = append
If StringInStr($oReceived,'ID.'& $aLinks[$iLink]) Then
        GUICtrlCreateListViewItem(_NowTime(5) & "|" & $aLinks[$iLink] & "| YES", $log)
        _GUICtrlStatusBar_SetText($StatusBar, @TAB & _GUICtrlListView_GetItemCount($log) & " visited profiles",2)
_FileWriteFromArray("IDs.txt", $aLog & @CRLF,1)
FileClose($hFile)
     Else
        _GUICtrlEdit_AppendText($log, _NowTime(5) & ": " & $aLinks[$iLink] & " - NO" & @CRLF)
        _FileWriteFromArray("IDs.txt", $aLog & @CRLF,1)
        FileClose($hFile)
     EndIf

    Sleep(3000) ; Three-second delay

 Next

 sleep(250)
         $iPage += 1
    WEnd


  EndFunc

Any idea where the conflict could come from?

Edited by xanathos
xanathos Seeker

xanathos

Posted
  • Author
Posted

I have partially fixed the problem, but now it will write the whole content of the array over and over (every 3 seconds I supposed since its in the loop?), so the ending txt file is kinda big, I'm not too sure how to take it out of the loop, since I want it to write "on the fly" everytime ID gets visited...

GMK Universalist

GMK

Posted
Posted

Well, one thing I notice is that you're using _FileWriteFromArray, which, in spite of using FileOpen and FileClose as you have here, will overwrite IDs.txt with $aLog in every loop iteration.  You may be better off writing IDs.txt to an array and using _ArraySearch to check for the visited profile.  Also, you may want to loop through $aLog and add each line using FileWriteLine and add to the IDs array.  Then at the end of the function, write the IDs array back to IDs.txt.  Finally, you may notice a significant speed difference if you keep all FileOpen and FileClose (and _FileReadToArray) functions outside of the loops.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing
×
×
  • Create New...