lostandconfused

Trying to extract text from a website

10 posts in this topic

Hello,

I have been asked to convert a few hundred web pages to text and get certain text out. Here is the process that I envision:

1. Save a website to a text file

2. Extract the data between two set HTML tags ( Between this  ->>> <div class="arrow"  role="button" tabindex="0" ></div></div><div class="thisentry"><p class="title"><a class="title may-blank " href=" <<<---and this--->>></a>&#32;<span class="domain">)

3. Remove text from the newly extracted data (always the same -> " tabindex="1" >)

4. Run though the original text file with steps 2 and 3 until no more text is found

5. Save all output to another text file

 

Here is what I have so far:
 

#include <INet.au3> ; needed for get source (hmtl)
#include <String.au3> ; needed for stringbetween
#include <MsgBoxConstants.au3>; needed for msgbox

$source = _INetGetSource("https://www.url.com") ;get html
FileWrite(@ScriptDir & "\url.html",$source); save file
Local $read = FileRead(@ScriptDir & "\url.html") ;read file
Local $readtitle = _StringBetween($read,'<div class="arrow"  role="button" tabindex="0" ></div></div><div class="thisentry"><p class="title"><a class="title may-blank " href="','</a> <span class="domain">') ;read text from file
Local $newtitle = StringReplace($readtitle, '" tabindex="1" >', " ")
MsgBox(64, "Finished", "Show" & $newtitle) ;display first found string

At this point I am just outputting to a MsgBox to see the results of my strings.

Not really sure why I cant even see the one proper result. Just seeing what you see in the image attached.

Any help with this issue would be appreciated. Thank you in advance!

 

P.S. Also looking for any tips on where to start with AutoIT. Just have basic knowledge of creating batch files and super basic to HTML and CSS, so anything helps. Specifically looking for help with dealing with text files (input, manipulation, output), intro programming (loops, variables, etc.) and anything else that might be able to help someone like me. Thanks!

error.JPG

Share this post


Link to post
Share on other sites



_StringBetween return an array.

 

Saludos

Share this post


Link to post
Share on other sites
17 hours ago, Danyfirex said:

_StringBetween return an array.

 

Saludos

Reading though the wiki page now. https://www.autoitscript.com/wiki/Arrays

How do I apply StringReplace to the output of _StringBetween properly? Any help is appreciated! Thank you.

Share this post


Link to post
Share on other sites

Try this:

$source = _INetGetSource("https://www.url.com") ;get html
FileWrite(@ScriptDir & "\url.html", $source); save file
Local $read = FileRead(@ScriptDir & "\url.html") ;read file
Local $readtitle = _StringBetween($read, '<div class="arrow"  role="button" tabindex="0" ></div></div><div class="thisentry"><p class="title"><a class="title may-blank " href="', '</a> <span class="domain">') ;read text from file
If IsArray $readtitle Then
    Local $newtitle = StringReplace($readtitle[0], '" tabindex="1" >', " ")
    MsgBox(64, "Finished", "Show" & $newtitle) ;display first found string
Else
    MsgBox(64, "Finished", "Nothing found") ;display hint
EndIf

 

Share this post


Link to post
Share on other sites

Thank you AutoBert!!!!

Based on this page I was able to make it work with just one minor adjustment - putting parenthesis around the array:

$source = _INetGetSource("https://www.url.com") ;get html
FileWrite(@ScriptDir & "\url.html", $source); save file
Local $read = FileRead(@ScriptDir & "\url.html") ;read file
Local $readtitle = _StringBetween($read, '<div class="arrow"  role="button" tabindex="0" ></div></div><div class="thisentry"><p class="title"><a class="title may-blank " href="', '</a> <span class="domain">') ;read text from file
If IsArray($readtitle) Then
    Local $newtitle = StringReplace($readtitle[0], '" tabindex="1" >', " ")
    MsgBox(64, "Finished", "Show" & $newtitle) ;display first found string
Else
    MsgBox(64, "Finished", "Nothing found") ;display hint
EndIf

This looks great so far.

One last question. How can I make the program write the output (after searching for all matching words) to a separate text file? I have been banging my head against the wall on that one all day today...

Share this post


Link to post
Share on other sites

#6 ·  Posted (edited)

At top of scropt insert:

#include <File.au3>

and this:

$PathToYourFileInclusiveName=@ScriptDir&'\CatchedTitles'
_FileWriteFromArray($PathToYourFileInclusiveName,$readtitle)

after MsgBox 'Finished'.

24 minutes ago, lostandconfused said:

Based on this page I was able to make it work with just one minor adjustment - putting parenthesis around the array:

Sorry, for this syntax error on forgotten parenthesis.

Edited by AutoBert

Share this post


Link to post
Share on other sites

AutoBert, please don't be sorry. I just wanted to make sure that in the future someone stumbling onto this post can just copy/paste like I did from many other threads. Your advice and posts have been more help then reading documentation/other info for a week. I really appreciate it!!!

Your advice was spot on.

#include <INet.au3> ; needed for get source (hmtl)
#include <String.au3> ; needed for stringbetween
#include <MsgBoxConstants.au3>
#include <File.au3>


$source = _INetGetSource("https://www.reddit.com/r/nfl") ;get html
FileWrite(@ScriptDir & "\reddit-r-nfl.html",$source); save file
Local $read = FileRead(@ScriptDir & "\reddit-r-nfl.html") ;read file
Local $readtitle = _StringBetween($read, '<div class="arrow down login-required access-required" data-event-action="downvote" role="button" aria-label="downvote" tabindex="0" ></div></div><div class="entry unvoted"><p class="title"><a class="title may-blank " href="','</a>&#32;<span class="domain">') ;read URL and title from file
If IsArray($readtitle) Then
    Local $newtitle = StringReplace($readtitle[0], '" tabindex="1" >', " ")
    $PathToYourFileInclusiveName=@ScriptDir&'\CatchedTitles.txt'
    _FileWriteFromArray($PathToYourFileInclusiveName,$readtitle)
Else
    MsgBox(64, "Finished", "Nothing found") ;display hint
EndIf

This works perfectly!!

Share this post


Link to post
Share on other sites
39 minutes ago, lostandconfused said:

Your advice and posts have been more help then reading documentation/other info for a week. I really appreciate it!!!

So you can change your nick, maybe hopefullSeeker.

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

Since I do not want to make the new forum member even more confused, so right away I will point to the correct post:

 

 

Edited by mLipok
wording

Signature beginning:   Wondering who uses AutoIT and what it can be used for ?
* GHAPI UDF - modest begining - comunication with GitHub REST API *
ADO.au3 UDF     POP3.au3 UDF     XML.au3 UDF    How to use IE.au3  UDF with  AutoIt v3.3.14.x  for other useful stuff click the following button

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST API *

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 - BETA * ADO.au3 UDF SMTP Mailer UDF *

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Best coding practices * 

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * 

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2017-06-04

Share this post


Link to post
Share on other sites

#10 ·  Posted (edited)

and to make @lostandconfused more happy, here is my example:

#include <INet.au3>; needed for get source (hmtl)
#include <String.au3>; needed for StringBetween
#include <MsgBoxConstants.au3>; needed for MsgBox
#include <File.au3>; needed for _FileWriteFromArray

;~ _Example()
_Example2()

Func _Example()
    $source = _INetGetSource("https://www.reddit.com/r/nfl") ;get html
    FileWrite(@ScriptDir & "\reddit-r-nfl.html", $source); save file
    Local $read = FileRead(@ScriptDir & "\reddit-r-nfl.html") ;read file
    Local $readtitle = _StringBetween($read, '<div class="arrow down login-required access-required" data-event-action="downvote" role="button" aria-label="downvote" tabindex="0" ></div></div><div class="entry unvoted"><p class="title"><a class="title may-blank " href="', '</a> <span class="domain">') ;read URL and title from file
    If IsArray($readtitle) Then
        Local $newtitle = StringReplace($readtitle[0], '" tabindex="1" >', " ")
        $PathToYourFileInclusiveName = @ScriptDir & '\CatchedTitles.txt'
        _FileWriteFromArray($PathToYourFileInclusiveName, $readtitle)
    Else
        MsgBox(64, "Finished", "Nothing found") ;display hint
    EndIf

EndFunc   ;==>_Example


Func _Example2()
    ; get html
    Local $sHTML_Content = _INetGetSource("https://www.reddit.com/r/nfl")
    Local $sReport_FileFullPath = @ScriptDir & '\CatchedTitles.txt'
    FileDelete($sReport_FileFullPath)

    ; read URL and title from file
    Local $aReadTitle = StringRegExp($sHTML_Content, '<div class="entry unvoted">.*?href="(.*?)".*?>(.*?)<.*?', $STR_REGEXPARRAYGLOBALMATCH)
    If Not @error Then
        For $iURL_idx = 0 To UBound($aReadTitle) - 1 Step 2
            FileWrite($sReport_FileFullPath, $aReadTitle[$iURL_idx] & ' ' & $aReadTitle[$iURL_idx + 1] & @CRLF)
        Next
    Else
        MsgBox($MB_ICONINFORMATION, "Finished", "Nothing found") ;display hint
    EndIf
EndFunc   ;==>_Example2

 

Edited by mLipok
magic number >> $STR_REGEXPARRAYGLOBALMATCH

Signature beginning:   Wondering who uses AutoIT and what it can be used for ?
* GHAPI UDF - modest begining - comunication with GitHub REST API *
ADO.au3 UDF     POP3.au3 UDF     XML.au3 UDF    How to use IE.au3  UDF with  AutoIt v3.3.14.x  for other useful stuff click the following button

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST API *

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 - BETA * ADO.au3 UDF SMTP Mailer UDF *

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Best coding practices * 

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * 

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

Signature last update: 2017-06-04

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now