Trying to extract text from a website

[lostandconfused]

Hello,

I have been asked to convert a few hundred web pages to text and get certain text out. Here is the process that I envision:

1. Save a website to a text file

2. Extract the data between two set HTML tags ( Between this  ->>> <div class="arrow"  role="button" tabindex="0" ></div></div><div class="thisentry"><p class="title"><a class="title may-blank " href=" <<<---and this--->>></a>&#32;<span class="domain">)

3. Remove text from the newly extracted data (always the same -> " tabindex="1" >)

4. Run though the original text file with steps 2 and 3 until no more text is found

5. Save all output to another text file

 

Here is what I have so far:
 

#include <INet.au3> ; needed for get source (hmtl)
#include <String.au3> ; needed for stringbetween
#include <MsgBoxConstants.au3>; needed for msgbox

$source = _INetGetSource("https://www.url.com") ;get html
FileWrite(@ScriptDir & "\url.html",$source); save file
Local $read = FileRead(@ScriptDir & "\url.html") ;read file
Local $readtitle = _StringBetween($read,'<div class="arrow"  role="button" tabindex="0" ></div></div><div class="thisentry"><p class="title"><a class="title may-blank " href="','</a> <span class="domain">') ;read text from file
Local $newtitle = StringReplace($readtitle, '" tabindex="1" >', " ")
MsgBox(64, "Finished", "Show" & $newtitle) ;display first found string

At this point I am just outputting to a MsgBox to see the results of my strings.

Not really sure why I cant even see the one proper result. Just seeing what you see in the image attached.

Any help with this issue would be appreciated. Thank you in advance!

 

P.S. Also looking for any tips on where to start with AutoIT. Just have basic knowledge of creating batch files and super basic to HTML and CSS, so anything helps. Specifically looking for help with dealing with text files (input, manipulation, output), intro programming (loops, variables, etc.) and anything else that might be able to help someone like me. Thanks!

[Danyfirex]

_StringBetween return an array.

 

Saludos

[lostandconfused]

17 hours ago, Danyfirex said:

_StringBetween return an array.

 

Saludos

Reading though the wiki page now. https://www.autoitscript.com/wiki/Arrays

How do I apply StringReplace to the output of _StringBetween properly? Any help is appreciated! Thank you.

[AutoBert]

Try this:

$source = _INetGetSource("https://www.url.com") ;get html
FileWrite(@ScriptDir & "\url.html", $source); save file
Local $read = FileRead(@ScriptDir & "\url.html") ;read file
Local $readtitle = _StringBetween($read, '<div class="arrow"  role="button" tabindex="0" ></div></div><div class="thisentry"><p class="title"><a class="title may-blank " href="', '</a> <span class="domain">') ;read text from file
If IsArray $readtitle Then
    Local $newtitle = StringReplace($readtitle[0], '" tabindex="1" >', " ")
    MsgBox(64, "Finished", "Show" & $newtitle) ;display first found string
Else
    MsgBox(64, "Finished", "Nothing found") ;display hint
EndIf

 

[lostandconfused]

Thank you AutoBert!!!!

Based on this page I was able to make it work with just one minor adjustment - putting parenthesis around the array:

$source = _INetGetSource("https://www.url.com") ;get html
FileWrite(@ScriptDir & "\url.html", $source); save file
Local $read = FileRead(@ScriptDir & "\url.html") ;read file
Local $readtitle = _StringBetween($read, '<div class="arrow"  role="button" tabindex="0" ></div></div><div class="thisentry"><p class="title"><a class="title may-blank " href="', '</a> <span class="domain">') ;read text from file
If IsArray($readtitle) Then
    Local $newtitle = StringReplace($readtitle[0], '" tabindex="1" >', " ")
    MsgBox(64, "Finished", "Show" & $newtitle) ;display first found string
Else
    MsgBox(64, "Finished", "Nothing found") ;display hint
EndIf

This looks great so far.

One last question. How can I make the program write the output (after searching for all matching words) to a separate text file? I have been banging my head against the wall on that one all day today...

[AutoBert]

At top of scropt insert:

#include <File.au3>

and this:

$PathToYourFileInclusiveName=@ScriptDir&'\CatchedTitles'
_FileWriteFromArray($PathToYourFileInclusiveName,$readtitle)

after MsgBox 'Finished'.

24 minutes ago, lostandconfused said:

Based on this page I was able to make it work with just one minor adjustment - putting parenthesis around the array:

Sorry, for this syntax error on forgotten parenthesis.

Edited January 28, 2016 by AutoBert

[lostandconfused]

AutoBert, please don't be sorry. I just wanted to make sure that in the future someone stumbling onto this post can just copy/paste like I did from many other threads. Your advice and posts have been more help then reading documentation/other info for a week. I really appreciate it!!!

Your advice was spot on.

#include <INet.au3> ; needed for get source (hmtl)
#include <String.au3> ; needed for stringbetween
#include <MsgBoxConstants.au3>
#include <File.au3>


$source = _INetGetSource("https://www.reddit.com/r/nfl") ;get html
FileWrite(@ScriptDir & "\reddit-r-nfl.html",$source); save file
Local $read = FileRead(@ScriptDir & "\reddit-r-nfl.html") ;read file
Local $readtitle = _StringBetween($read, '<div class="arrow down login-required access-required" data-event-action="downvote" role="button" aria-label="downvote" tabindex="0" ></div></div><div class="entry unvoted"><p class="title"><a class="title may-blank " href="','</a>&#32;<span class="domain">') ;read URL and title from file
If IsArray($readtitle) Then
    Local $newtitle = StringReplace($readtitle[0], '" tabindex="1" >', " ")
    $PathToYourFileInclusiveName=@ScriptDir&'\CatchedTitles.txt'
    _FileWriteFromArray($PathToYourFileInclusiveName,$readtitle)
Else
    MsgBox(64, "Finished", "Nothing found") ;display hint
EndIf

This works perfectly!!

[AutoBert]

39 minutes ago, lostandconfused said:

Your advice and posts have been more help then reading documentation/other info for a week. I really appreciate it!!!

So you can change your nick, maybe hopefullSeeker.

[mLipok]

Since I do not want to make the new forum member even more confused, so right away I will point to the correct post:

 

 

Edited January 29, 2016 by mLipok
wording

[mLipok]

and to make @lostandconfused more happy, here is my example:

expandcollapse popup

#include <INet.au3>; needed for get source (hmtl)
#include <String.au3>; needed for StringBetween
#include <MsgBoxConstants.au3>; needed for MsgBox
#include <File.au3>; needed for _FileWriteFromArray

;~ _Example()
_Example2()

Func _Example()
    $source = _INetGetSource("https://www.reddit.com/r/nfl") ;get html
    FileWrite(@ScriptDir & "\reddit-r-nfl.html", $source); save file
    Local $read = FileRead(@ScriptDir & "\reddit-r-nfl.html") ;read file
    Local $readtitle = _StringBetween($read, '<div class="arrow down login-required access-required" data-event-action="downvote" role="button" aria-label="downvote" tabindex="0" ></div></div><div class="entry unvoted"><p class="title"><a class="title may-blank " href="', '</a> <span class="domain">') ;read URL and title from file
    If IsArray($readtitle) Then
        Local $newtitle = StringReplace($readtitle[0], '" tabindex="1" >', " ")
        $PathToYourFileInclusiveName = @ScriptDir & '\CatchedTitles.txt'
        _FileWriteFromArray($PathToYourFileInclusiveName, $readtitle)
    Else
        MsgBox(64, "Finished", "Nothing found") ;display hint
    EndIf

EndFunc   ;==>_Example


Func _Example2()
    ; get html
    Local $sHTML_Content = _INetGetSource("https://www.reddit.com/r/nfl")
    Local $sReport_FileFullPath = @ScriptDir & '\CatchedTitles.txt'
    FileDelete($sReport_FileFullPath)

    ; read URL and title from file
    Local $aReadTitle = StringRegExp($sHTML_Content, '<div class="entry unvoted">.*?href="(.*?)".*?>(.*?)<.*?', $STR_REGEXPARRAYGLOBALMATCH)
    If Not @error Then
        For $iURL_idx = 0 To UBound($aReadTitle) - 1 Step 2
            FileWrite($sReport_FileFullPath, $aReadTitle[$iURL_idx] & ' ' & $aReadTitle[$iURL_idx + 1] & @CRLF)
        Next
    Else
        MsgBox($MB_ICONINFORMATION, "Finished", "Nothing found") ;display hint
    EndIf
EndFunc   ;==>_Example2

 

Edited January 29, 2016 by mLipok
magic number >> $STR_REGEXPARRAYGLOBALMATCH