xanathos

Not really, its not the main purpose of the script, but to make it run in optimal way, it does require to switch between proxies, yes.

Gotta ask the IT guy from our school, which is pretty stupid considering the server is running local only...

Our students database have a blacklisting for few hours, and being able to process through the large database without being blacklisted is a criteria for the project.

Hello there! First here is my code: Func _OpenLink() Local $OkChk Local $iOk = 0 Local $array = FileReadToArray($Source) Global Const $sUserName = "", $sPassword = "" ; set correct $aListView = _GuiCtrlListView_CreateArray($hListView, Default) $hSession = _WinHttpOpen("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.135 Safari/537.36 Edge/12.10240", $WINHTTP_ACCESS_TYPE_NAMED_PROXY, Guictrlread($h_sProxy) & ":" & GuiCtrlRead($h_iPort)) $sHost = '127.0.0.1' $sReq = _WinHttpConnect($hSession, $sHost, $INTERNET_DEFAULT_HTTPS_PORT) For $i = 0 To UBound($array) - 1 if $i > 0 Then $sResponse = _WinHttpSimpleSSLRequest($sReq, 'GET', 'check=' & $array[$i] & @CRLF) $ExpDate = _StringBetween($sResponse,'"exp":"', '"}') $Type = _StringBetween($sResponse,'"type":"', '"') If StringInStr($sResponse, 'true') Then $iOk += 1 $GetIt = GUICtrlCreateListViewItem($array[$i] & "| Yes" & "|" & $ExpDate[0] & "|" & $Type[0], $hListView) FileWrite($Dest, $array[$i] & " | " & $ExpDate[0] & " | " & $Type[0] & @CRLF) SoundPlay("s3.wav") EndIf If StringInStr($sResponse, 'used') Then GUICtrlCreateListViewItem($array[$i] & "| Used" & "|" & $ExpDate[0] & "|" & $Type[0], $hListView) EndIf If StringInStr($sResponse, 'not available') Then GUICtrlCreateListViewItem($array[$i] & "| Non available" & "|" & $ExpDate[0] & "|" & $Type[0], $hListView) EndIf If StringInStr($sResponse, 'unknown') Then GUICtrlCreateListViewItem($array[$i] & "| Unknown", $hListView) EndIf EndIf GuiCtrlSetData($hLabel, $iOk & " ok on " & _GUICtrlListView_GetItemCount($hListView) & " on " & _FileCountLines($Source) & " - ETA: " & (((_FileCountLines($Source))-(_GUICtrlListView_GetItemCount($hListView)))*45)/60 & " minutes" ) Sleep(45000) Next EndFunc What I want to do is switch between 3 proxies every 30 arrays, first proxy for first 30 arrays, second for the next 30, and 3rd for the next (Proxy 1: 0 to 30; proxy 2: 31-60; proxy 3: 61-90, then loop that). I tried with if $i > 30 then change the hSession variable to read $h_sProxy2 and $h_iPort2 (these are inputs for proxy 2), and if $i > 60 read $h_sProxy3, and so on, then if $i > 90, set $i to 0, but no luck, there must be somewhere where its overriden. If someone could help, hope its not confusing.

In the case of FileWriteLine, I could write directly to IDs.txt file, no?

I have partially fixed the problem, but now it will write the whole content of the array over and over (every 3 seconds I supposed since its in the loop?), so the ending txt file is kinda big, I'm not too sure how to take it out of the loop, since I want it to write "on the fly" everytime ID gets visited...

Worked great. Next step where I'm stuck: I'm basically storing each IDs in a txt file with the timestamp they've been visited at, once (for example) 24 hours passed, that ID would be deleted from the txt file, this is not the problem yet, I'm just trying to be clear in my explanation; So the problem is that the file has to be read to check if the ID's string is there or not, if yes, it moves to the next ID link, if not, it will visit that ID, there's a conflict somewhere, but I'm not too sure where, I would guess it comes from the loops while ID gets written, it will be checked, so sometimes it will randomly appear as visited, although it was not, and vice-versa. Here is a part of the code where I have implemented the string search and condition: For $iLink = 0 To UBound($aLinks) - 1 If StringInStr(FileRead("IDs.txt",1), $aLinks[$iLink]) Then MsgBox(0,"Test",$aLinks[$iLink] & " already visited!") Else $oHTTP.Open("GET", "http://127.0.0.1/profile/" & $aLinks[$iLink], False) $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded") $oHTTP.Send() Sleep(250) EndIf $oReceived = $oHTTP.ResponseText $oStatusCode = $oHTTP.Status If $oStatusCode = 200 Then GUICtrlSetState($log, $GUI_ENABLE) EndIf $aLog = _GUICtrlListView_CreateArray($log, Default) $hFile = FileOpen("IDs.txt", 1) ; 1 = append If StringInStr($oReceived,'ID.'& $aLinks[$iLink]) Then GUICtrlCreateListViewItem(_NowTime(5) & "|" & $aLinks[$iLink] & "| YES", $log) _GUICtrlStatusBar_SetText($StatusBar, @TAB & _GUICtrlListView_GetItemCount($log) & " visited profiles",2) _FileWriteFromArray("IDs.txt", $aLog & @CRLF,1) FileClose($hFile) Else _GUICtrlEdit_AppendText($log, _NowTime(5) & ": " & $aLinks[$iLink] & " - NO" & @CRLF) _FileWriteFromArray("IDs.txt", $aLog & @CRLF,1) FileClose($hFile) EndIf Sleep(3000) ; Three-second delay Next sleep(250) $iPage += 1 WEnd EndFunc Any idea where the conflict could come from?

the html file is just some kind of check, I sometimes check manually, just to make sure the body response is the good one, so yeah it's not needed, I will keep it outside of the loop. Gonna try that, after hitting the gym thanks.

That would make a loop (pages one) inside another loop (IDs one), right?

Good notice! Here is the code of my main function so far: Func Main() ; The data to be sent $sPD = 'username=admin&password=password&remember=on' $search = 'age%5Bmin%5D=18&age%5Bmax%5D=30&age_step=1&by=distance&country=&region=&distance%5Bmin%5D=&distance%5Bmax%5D=50&distance_step=10&pseudo=&sex=1&size%5Bmin%5D=120&size%5Bmax%5D=220&size_step=5&weight%5Bmin%5D=30&weight%5Bmax%5D=115&weight_step=5&mandatory%5B%5D=shape&mandatory%5B%5D=eyes_color&mandatory%5B%5D=origins&mandatory%5B%5D=hair_color&mandatory%5B%5D=hair_size&mandatory%5B%5D=hair_style&mandatory%5B%5D=styles&mandatory%5B%5D=character&mandatory%5B%5D=features&mandatory%5B%5D=diet&mandatory%5B%5D=favourite_food&mandatory%5B%5D=tobacco&mandatory%5B%5D=alcohol' ; Creating the object $oHTTP = ObjCreate("winhttp.winhttprequest.5.1") $oHTTP.Open("POST", "http://127.0.0.1//auth/login", False) $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded") $oHTTP.Send($sPD) $oHTTP.Open("POST", "http://127.0.0.1/mySearch", False) $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded") $oHTTP.Send($search) $oHTTP.Open("GET", "http://127.0.0.1/mySearch", False) $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded") $oHTTP.Send() ; Download the body response if any, and get the server status response code. $oReceived = $oHTTP.ResponseText $oStatusCode = $oHTTP.Status $aLinks = _stringbetween($oReceived, '"url":"\/index.php\/profile\/', '\') _ArrayDisplay($aLinks, "test") sleep(250) For $iLink = 0 To UBound($aLinks) - 1 $oHTTP.Open("GET", "http://127.0.0.1/profile/" & $aLinks[$iLink], False) $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded") $oHTTP.Send() Sleep(250) $oReceived = $oHTTP.ResponseText $oStatusCode = $oHTTP.Status If $oStatusCode = 200 Then GUICtrlCreateListViewItem(_NowTime(5) & "|" & $aLinks[$iLink] & "| Visited", $log) EndIf Sleep(3000) ; Three-second delay Next _GUICtrlListView_DeleteAllItems($log) sleep(250) $oHTTP.Open("GET", "http://127.0.0.1/mySearch/ajax_loadPages/?page=2", False) $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded") $oHTTP.Send() $oReceived = $oHTTP.ResponseText $oStatusCode = $oHTTP.Status $aLinks = _stringbetween($oReceived, '"url":"\/index.php\/profile\/', '\') _ArrayDisplay($aLinks, "test2") sleep(250) $lastp = FileOpen("Dernier.html", 2) FileWrite($lastp, $oReceived) FileClose($lastp) ;~ _FileWriteFromArray($IDs, $aLinks) ;~ FileClose($IDs) EndFunc 2 problems here: 1. As you can see, once all IDs have been visited I would like to move to next page and repeat the loop above, and do so, until last page is reached. 2. BUT there is no way to know the last page until the last page is reached. (For example if there are 16 pages of results, I don't know it until page 16 is reached, but if I try to access to page 17 manually, I can check for a string on the page that says "no results" and then we'd know there are no more pages, OR actually, from the first page, we know how many IDs the search has returned, and max IDs is 15 per page, so we can search for the IDs returned and divide that, by 15 to obtain the number of pages?)

I don't need them, that's just to the post data for the search fields...

Bump

Ok, managed to make it work, problem was that since it's ajax, it won't return the profile ID as it should be but in raw so it was a var: $alinks = _stringbetween($oReceived, '"url":"\/index.php\/profile\/', '\') _ArrayDisplay($aLinks, "test") Now I'm coming across another problem: $oHTTP.Open("GET", "http://127.0.0.1/profile/" & $aLinks[0], False) $oHTTP.SetRequestHeader("Content-Type", "application/x-www-form-urlencoded") $oHTTP.Send() sleep(250) $oReceived = $oHTTP.ResponseText $oStatusCode = $oHTTP.Status If $oStatusCode = 200 then MsgBox(0, "Response code", "Profile " & $aLinks[0] & " visited!") EndIf Each page returns several (usually 14 but it can change) IDs, the above example does work, I wanna visit all the IDs links with a 3 seconds delay between each visit, and move to next page, but do I really have to do that for each array, or is there a simpler way?

Tried it this way: $aLinks = StringRegExp($oReceived, '.?profile/(\d+)', 3) _ArrayDisplay($aLinks, "test") Now, problem is that it will return only one occurrence of the first ID found. Shouldn't I convert the html file into a txt or source file first?

Here is my code: ; Download the body response if any, and get the server status response code. $oReceived = $oHTTP.ResponseText $oStatusCode = $oHTTP.Status $aLinks = StringRegExp($oReceived, '<a title="View profile" href="http://127.0.0.1/profile/(.*?)" >', 3) _ArrayDisplay($aLinks) If $oStatusCode <> 200 then MsgBox(4096, "Response code", $oStatusCode) EndIf ; Saves the body response regardless of the Response code $file = FileOpen("Received.html", 2) ; The value of 2 overwrites the file if it already exists $IDs = FileOpen("IDs.txt", 2) ; FileWrite($file, $oReceived) FileClose($file) _FileWriteFromArray($IDs, $aLinks) FileClose($IDs) EndFunc When I run it, it won't even show the arrays and the txt file is empty.