Jump to content

The way to remove the links that are duplicates?

Recommended Posts


yo, i need some help.

Im training with _StringsBetween, on big site! Alot fun, My script turned 8 times created a text file with a weight of 1GB. I saw that in the file was a lot of repetition. Is it possible to somehow set the string to save the file, only one URL without repetition?

#include <ButtonConstants.au3>
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <WindowsConstants.au3>
#include <INet.au3>
#include <StringConstants.au3>
#include <File.au3>

;I'm coming for blood, no code of conduct, no law.
;I'm coming for blood, no code of conduct, no law.
#Region ### START Koda GUI section ### Form=
$Form1 = GUICreate("Form1", 1211, 812, 43, 110)
$Button1 = GUICtrlCreateButton("Button1", 0, 8, 249, 81)
$Button2 = GUICtrlCreateButton("Button2", 350, 8, 249, 81)
$Edit1 = GUICtrlCreateEdit("0", 8, 128, 609, 257, BitOR($ES_CENTER,$ES_AUTOHSCROLL,$ES_READONLY,$ES_WANTRETURN))
$Edit2 = GUICtrlCreateEdit("2", 632, 0, 577, 809)
#EndRegion ### END Koda
local $iFileSize = FileGetSize('')

Func VisitFrontPage()
    local $Liczba = _FileCountLines(@ScriptDir&'\data\links.txt')
    local $liczba2 = GUICtrlSetData($Edit1,Random(1,$Liczba,1))
    local $Liczba3 = GUICtrlRead($Edit1)
    local $Liczba4 = FileReadLine(@ScriptDir&'\data\links.txt',$Liczba3)
$data = _INetGetSource ( FileReadLine(@ScriptDir&'\data\links.txt',Random(1,$Liczba,1)))
$linki = StringRegExp($data, '<a href="http://www.wykop.pl/link/(.*?)/" title=""',3)
For $q = 0 To UBound($linki) -1
$linki = StringRegExp($data, 'href="http://www.wykop.pl/ludzie/(.*?)/">',3) ; pobieranie ludzi co dodali znaleziska
For $w = 0 To UBound($linki) -1
$linki = StringRegExp($data, '<a href="http://www.wykop.pl/ludzie/(.*?)/" title="',3) ; pobieranie ludzi co sa na stronie z mikro
For $e = 0 To UBound($linki) -1
$linki = StringRegExp($data, '<a class="tag create" href="http://www.wykop.pl/tag/(.*?)/"><em>',3) ; pobieranie ludzi co sa na stronie z mikro
For $r = 0 To UBound($linki) -1
$linki = StringRegExp($data, '<class="showTagSummary" href="http://www.wykop.pl/tag/(.*?)">',3) ; pobieranie ludzi co sa na stronie z mikro
For $t = 0 To UBound($linki) -1
 $linki = StringRegExp($data, 'href="http://www.wykop.pl/ludzie/(.*?)/" title=""',3) ; pobieranie ludzi z znaleziska, komentarze
For $a = 0 To UBound($linki) -1
 $linki = StringRegExp($data, 'href="http://www.wykop.pl/tag/index/(.*?)/"',3) ; pobierane tagi ze znaleziska,
For $s = 0 To UBound($linki) -1
 $linki = StringRegExp($data, '<a class="clearfix" href="http://www.wykop.pl/link/(.*?)/?utm_source',3) ; pobierane znaleziska z prawego menu
For $d = 0 To UBound($linki) -1
GuiCtrlSetData($Edit2, GuiCtrlRead($Edit2)+1)

While 1
    $nMsg = GUIGetMsg()
    Switch $nMsg
        Case $Button1
        Case $Button2
        Case $GUI_EVENT_CLOSE



Share this post

Link to post
Share on other sites

Why do you make the function VisitFrontPage recursive and what's the purpose of

local $iFileSize = FileGetSize('')

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post

Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • QaisBsharat
      By QaisBsharat
      May i know how can i automation qt application?
      Using controlclick not mouse and positions
    • nooneclose
      By nooneclose
      Good afternoon, I am in need of some help. I am sure this is a stupid question requiring only one or two lines of code. However, I would greatly appreciate the help I cannot figure this out. I also tried searching for the answer on the internet but no one except me apparently seems to be having a hard time figuring this out and or is asking about it. 
      I simply want to change the "status" of an email from unread to read once I have processed it. My code is over 500 lines and I would like not to clutter this post with it. Assume I have all my includes and connections properly defined and stuff. 
      Here is the bit of code where I am trying to change the email that was used from unread to read:
      Func ChangeEmailStatus() ;******************************************************************************* ; changes the status of an email from unread to Read ;******************************************************************************* Local $iRows = UBound($aItems, $UBOUND_ROWS) MsgBox("", "Number of Unread emails (Before Change)", $iRows) _OL_ItemModify($oOutlook,$aItems[$i][0], Default, "Read=True") MsgBox("", "Array Display 1", $aItems[1][0]) MsgBox("", "Array Display 2", $aItems[2][0]) Local $iRows = UBound($aItems, $UBOUND_ROWS) MsgBox("", "Number of Unread emails (After Change)", $iRows) EndFunc  
    • SharpDressedMan
      By SharpDressedMan
      Is there any way to check for validity of a pointer to a DllStruct ?
      In the above code, a DllStruct is created from an invalid pointer. How to prevent from doing this ?
      local $tStruct = DllStructCreate("int i") DllStructSetData($tStruct, "i", 123) local $pStruct = DllStructGetPtr($tStruct) $tStruct = 0 ; destroy DllStruct ==> $pStruct becomes invalid ; how to check here for validity of $pStruct and prevent from doing the following ? $tStruct = DllStructCreate("int i", $pStruct) ; create DllStruct from invalid pointer... MsgBox(0, @ScriptName, @error) ; displays '0' : invalid pointer is not detected by DllStructCreate() MsgBox(0, @ScriptName, IsDllStruct($tStruct)) ; displays '1' : tStruct is assumed as a valid DllStruct object, which is not Thanks for help
    • nooneclose
      By nooneclose
      I need to send a string of text to this popup and click on the ok button to save it.
      Here is the code I have so far:
      ;Start IE Sleep(7000) $oIE = _IECreate("http://www.google.com") Sleep(500) _IELoadWait($oIE) $hIE = _IEPropertyGet($oIE, "hwnd") ; Get Handle of the IE window Sleep(500) WinSetState($hIE, "", @SW_MAXIMIZE) ;Wait for a browser page load to complete Sleep(3000) _IENavigate($oIE, "https://properURL.com") Sleep(8000) _IELoadWait($oIE) ;Attach to a browser control embedded in another window $oIE = _IEAttach("https://"properURL.com", url") ConsoleWrite('@@ Debug(' & @ScriptLineNumber & ') : $oIE = ' & $oIE & @CRLF & '>Error code: ' & @error & ' Extended code: 0x' & Hex(@extended) & @CRLF) ;### Debug Console Sleep(2000) ;Get the title of the webpage ;Local $wTitle = _IEPropertyGet($oIE, "title") ;MsgBox($MB_SYSTEMMODAL, "Webpage title:", $wTitle) ;Clicks the new button Sleep(3000) _IEAction($nWorkOrderB, "focus") _IEAction($nWorkOrderB, "click") Sleep(5000) ;Store the Element names where the important data will be sent ;Store the long description button Local $wLongDButton = _IEGetObjById($oIE, "m65d795a4-img") ;Store the long Description field id Local $wComments = _IEGetObjById($oIE, "ma6499a9c-rte_iframe") ;Store the ok button id that is in the long description Local $wCommOk = _IEGetObjById($oIE, "m74031266-pb") ;******************************************************************************* ; Send the stored data to the proper field ;******************************************************************************* ;Click the long description button Sleep(300) _IEAction($wLongDButton, "focus") _IEAction($wLongDButton, "click") Sleep(300) ;Sends the Comments Sleep(500) _IEAction($wComments, "focus") _IEAction($wComments, "click") Sleep(500) _IEFormElementSetValue($wComments, "hello darkness my old friend") ;Click the ok button Sleep(500) _IEAction($wCommOk, "focus") _IEAction($wCommOk, "click") Sleep(500)  
      Here is the popup:

    • yasha
      By yasha
      i want am trying to select a nimber to run a program and then select where to save the excel result at before hand
      the problem is that it does not save in the folder i want but the folder before any solutions
      $sFolder = ""
          ; Create a constant variable in Local scope of the message to display in FileSelectFolder.
          Local Const $sMessage = "Select a folder"
          ; Display an open dialog to select a file.
          $sFileSelectFolder = FileSelectFolder($sMessage, $sFolder)
          If @error Then
              ; Display the error message.
              MsgBox($MB_SYSTEMMODAL, "", "No folder was selected.")
              ; Display the selected folder.
              MsgBox($MB_SYSTEMMODAL, "", "You chose the following folder:" & @CRLF & $sFileSelectFolder)
      $oExcel = ObjCreate("Excel.Application")                   ; Create an Excel Object
      $oExcel.Visible = 1                                        ; Let Excel show itself
      $oExcel.Workbooks.Open("J:\OPS\OPS_Share\Planners\2 - Weekly Reports\Auto download\"& $YY & $MM & $DD & " ORDER.xls",0)
      $oExcel.ActiveWorkbook.Saveas ( $sFileSelectFolder,""& $YY & $MM & $DD & " ORDER.xlsx", 1)
      i only want to save it as ddmmyy order inside documents but it saves in libraries as documents ddmmyy order.