lostandconfused Posted January 28, 2016 Posted January 28, 2016 Hello, I have been asked to convert a few hundred web pages to text and get certain text out. Here is the process that I envision: 1. Save a website to a text file 2. Extract the data between two set HTML tags ( Between this ->>> <div class="arrow" role="button" tabindex="0" ></div></div><div class="thisentry"><p class="title"><a class="title may-blank " href=" <<<---and this--->>></a> <span class="domain">) 3. Remove text from the newly extracted data (always the same -> " tabindex="1" >) 4. Run though the original text file with steps 2 and 3 until no more text is found 5. Save all output to another text file Here is what I have so far: #include <INet.au3> ; needed for get source (hmtl) #include <String.au3> ; needed for stringbetween #include <MsgBoxConstants.au3>; needed for msgbox $source = _INetGetSource("https://www.url.com") ;get html FileWrite(@ScriptDir & "\url.html",$source); save file Local $read = FileRead(@ScriptDir & "\url.html") ;read file Local $readtitle = _StringBetween($read,'<div class="arrow" role="button" tabindex="0" ></div></div><div class="thisentry"><p class="title"><a class="title may-blank " href="','</a> <span class="domain">') ;read text from file Local $newtitle = StringReplace($readtitle, '" tabindex="1" >', " ") MsgBox(64, "Finished", "Show" & $newtitle) ;display first found string At this point I am just outputting to a MsgBox to see the results of my strings. Not really sure why I cant even see the one proper result. Just seeing what you see in the image attached. Any help with this issue would be appreciated. Thank you in advance! P.S. Also looking for any tips on where to start with AutoIT. Just have basic knowledge of creating batch files and super basic to HTML and CSS, so anything helps. Specifically looking for help with dealing with text files (input, manipulation, output), intro programming (loops, variables, etc.) and anything else that might be able to help someone like me. Thanks!
Danyfirex Posted January 28, 2016 Posted January 28, 2016 _StringBetween return an array. Saludos Danysys.com AutoIt... UDFs: VirusTotal API 2.0 UDF - libZPlay UDF - Apps: Guitar Tab Tester - VirusTotal Hash Checker Examples: Text-to-Speech ISpVoice Interface - Get installed applications - Enable/Disable Network connection PrintHookProc - WINTRUST - Mute Microphone Level - Get Connected NetWorks - Create NetWork Connection ShortCut
lostandconfused Posted January 28, 2016 Author Posted January 28, 2016 17 hours ago, Danyfirex said: _StringBetween return an array. Saludos Reading though the wiki page now. https://www.autoitscript.com/wiki/Arrays How do I apply StringReplace to the output of _StringBetween properly? Any help is appreciated! Thank you.
AutoBert Posted January 28, 2016 Posted January 28, 2016 Try this: $source = _INetGetSource("https://www.url.com") ;get html FileWrite(@ScriptDir & "\url.html", $source); save file Local $read = FileRead(@ScriptDir & "\url.html") ;read file Local $readtitle = _StringBetween($read, '<div class="arrow" role="button" tabindex="0" ></div></div><div class="thisentry"><p class="title"><a class="title may-blank " href="', '</a> <span class="domain">') ;read text from file If IsArray $readtitle Then Local $newtitle = StringReplace($readtitle[0], '" tabindex="1" >', " ") MsgBox(64, "Finished", "Show" & $newtitle) ;display first found string Else MsgBox(64, "Finished", "Nothing found") ;display hint EndIf
lostandconfused Posted January 28, 2016 Author Posted January 28, 2016 Thank you AutoBert!!!! Based on this page I was able to make it work with just one minor adjustment - putting parenthesis around the array: $source = _INetGetSource("https://www.url.com") ;get html FileWrite(@ScriptDir & "\url.html", $source); save file Local $read = FileRead(@ScriptDir & "\url.html") ;read file Local $readtitle = _StringBetween($read, '<div class="arrow" role="button" tabindex="0" ></div></div><div class="thisentry"><p class="title"><a class="title may-blank " href="', '</a> <span class="domain">') ;read text from file If IsArray($readtitle) Then Local $newtitle = StringReplace($readtitle[0], '" tabindex="1" >', " ") MsgBox(64, "Finished", "Show" & $newtitle) ;display first found string Else MsgBox(64, "Finished", "Nothing found") ;display hint EndIf This looks great so far. One last question. How can I make the program write the output (after searching for all matching words) to a separate text file? I have been banging my head against the wall on that one all day today...
AutoBert Posted January 28, 2016 Posted January 28, 2016 (edited) At top of scropt insert: #include <File.au3> and this: $PathToYourFileInclusiveName=@ScriptDir&'\CatchedTitles' _FileWriteFromArray($PathToYourFileInclusiveName,$readtitle) after MsgBox 'Finished'. 24 minutes ago, lostandconfused said: Based on this page I was able to make it work with just one minor adjustment - putting parenthesis around the array: Sorry, for this syntax error on forgotten parenthesis. Edited January 28, 2016 by AutoBert
lostandconfused Posted January 28, 2016 Author Posted January 28, 2016 AutoBert, please don't be sorry. I just wanted to make sure that in the future someone stumbling onto this post can just copy/paste like I did from many other threads. Your advice and posts have been more help then reading documentation/other info for a week. I really appreciate it!!! Your advice was spot on. #include <INet.au3> ; needed for get source (hmtl) #include <String.au3> ; needed for stringbetween #include <MsgBoxConstants.au3> #include <File.au3> $source = _INetGetSource("https://www.reddit.com/r/nfl") ;get html FileWrite(@ScriptDir & "\reddit-r-nfl.html",$source); save file Local $read = FileRead(@ScriptDir & "\reddit-r-nfl.html") ;read file Local $readtitle = _StringBetween($read, '<div class="arrow down login-required access-required" data-event-action="downvote" role="button" aria-label="downvote" tabindex="0" ></div></div><div class="entry unvoted"><p class="title"><a class="title may-blank " href="','</a> <span class="domain">') ;read URL and title from file If IsArray($readtitle) Then Local $newtitle = StringReplace($readtitle[0], '" tabindex="1" >', " ") $PathToYourFileInclusiveName=@ScriptDir&'\CatchedTitles.txt' _FileWriteFromArray($PathToYourFileInclusiveName,$readtitle) Else MsgBox(64, "Finished", "Nothing found") ;display hint EndIf This works perfectly!!
AutoBert Posted January 29, 2016 Posted January 29, 2016 39 minutes ago, lostandconfused said: Your advice and posts have been more help then reading documentation/other info for a week. I really appreciate it!!! So you can change your nick, maybe hopefullSeeker.
mLipok Posted January 29, 2016 Posted January 29, 2016 (edited) Since I do not want to make the new forum member even more confused, so right away I will point to the correct post: Edited January 29, 2016 by mLipok wording Signature beginning:* Please remember: "AutoIt"..... * Wondering who uses AutoIt and what it can be used for ? * Forum Rules ** ADO.au3 UDF * POP3.au3 UDF * XML.au3 UDF * IE on Windows 11 * How to ask ChatGPT for AutoIt Code * for other useful stuff click the following button: Spoiler Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST API * ErrorLog.au3 UDF - A logging Library * Include Dependency Tree (Tool for analyzing script relations) * Show_Macro_Values.au3 * My contribution to others projects or UDF based on others projects: * _sql.au3 UDF * POP3.au3 UDF * RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF * SMTP Mailer UDF * Dual Monitor resolution detection * * 2GUI on Dual Monitor System * _SciLexer.au3 UDF * SciTE - Lexer for console pane * Useful links: * Forum Rules * Forum etiquette * Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * Wiki: * Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * OpenOffice/LibreOffice/XLS Related: WriterDemo.au3 * XLS/MDB from scratch with ADOX IE Related: * How to use IE.au3 UDF with AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * IE in TaskScheduler * IE Embedded Control Versioning (use IE9+ and HTML5 in a GUI) * PDF Related: * How to get reference to PDF object embeded in IE * IE on Windows 11 * I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions * EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *I also encourage you to check awesome @trancexx code: * Create COM objects from modules without any demand on user to register anything. * Another COM object registering stuff * OnHungApp handler * Avoid "AutoIt Error" message box in unknown errors * HTML editor * winhttp.au3 related : * https://www.autoitscript.com/forum/topic/206771-winhttpau3-download-problem-youre-speaking-plain-http-to-an-ssl-enabled-server-port/ "Homo sum; humani nil a me alienum puto" - Publius Terentius Afer"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming" , be and \\//_. Anticipating Errors : "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty." Signature last update: 2023-04-24
mLipok Posted January 29, 2016 Posted January 29, 2016 (edited) and to make @lostandconfused more happy, here is my example: expandcollapse popup#include <INet.au3>; needed for get source (hmtl) #include <String.au3>; needed for StringBetween #include <MsgBoxConstants.au3>; needed for MsgBox #include <File.au3>; needed for _FileWriteFromArray ;~ _Example() _Example2() Func _Example() $source = _INetGetSource("https://www.reddit.com/r/nfl") ;get html FileWrite(@ScriptDir & "\reddit-r-nfl.html", $source); save file Local $read = FileRead(@ScriptDir & "\reddit-r-nfl.html") ;read file Local $readtitle = _StringBetween($read, '<div class="arrow down login-required access-required" data-event-action="downvote" role="button" aria-label="downvote" tabindex="0" ></div></div><div class="entry unvoted"><p class="title"><a class="title may-blank " href="', '</a> <span class="domain">') ;read URL and title from file If IsArray($readtitle) Then Local $newtitle = StringReplace($readtitle[0], '" tabindex="1" >', " ") $PathToYourFileInclusiveName = @ScriptDir & '\CatchedTitles.txt' _FileWriteFromArray($PathToYourFileInclusiveName, $readtitle) Else MsgBox(64, "Finished", "Nothing found") ;display hint EndIf EndFunc ;==>_Example Func _Example2() ; get html Local $sHTML_Content = _INetGetSource("https://www.reddit.com/r/nfl") Local $sReport_FileFullPath = @ScriptDir & '\CatchedTitles.txt' FileDelete($sReport_FileFullPath) ; read URL and title from file Local $aReadTitle = StringRegExp($sHTML_Content, '<div class="entry unvoted">.*?href="(.*?)".*?>(.*?)<.*?', $STR_REGEXPARRAYGLOBALMATCH) If Not @error Then For $iURL_idx = 0 To UBound($aReadTitle) - 1 Step 2 FileWrite($sReport_FileFullPath, $aReadTitle[$iURL_idx] & ' ' & $aReadTitle[$iURL_idx + 1] & @CRLF) Next Else MsgBox($MB_ICONINFORMATION, "Finished", "Nothing found") ;display hint EndIf EndFunc ;==>_Example2 Edited January 29, 2016 by mLipok magic number >> $STR_REGEXPARRAYGLOBALMATCH Signature beginning:* Please remember: "AutoIt"..... * Wondering who uses AutoIt and what it can be used for ? * Forum Rules ** ADO.au3 UDF * POP3.au3 UDF * XML.au3 UDF * IE on Windows 11 * How to ask ChatGPT for AutoIt Code * for other useful stuff click the following button: Spoiler Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST API * ErrorLog.au3 UDF - A logging Library * Include Dependency Tree (Tool for analyzing script relations) * Show_Macro_Values.au3 * My contribution to others projects or UDF based on others projects: * _sql.au3 UDF * POP3.au3 UDF * RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF * SMTP Mailer UDF * Dual Monitor resolution detection * * 2GUI on Dual Monitor System * _SciLexer.au3 UDF * SciTE - Lexer for console pane * Useful links: * Forum Rules * Forum etiquette * Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * Wiki: * Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * OpenOffice/LibreOffice/XLS Related: WriterDemo.au3 * XLS/MDB from scratch with ADOX IE Related: * How to use IE.au3 UDF with AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * IE in TaskScheduler * IE Embedded Control Versioning (use IE9+ and HTML5 in a GUI) * PDF Related: * How to get reference to PDF object embeded in IE * IE on Windows 11 * I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions * EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *I also encourage you to check awesome @trancexx code: * Create COM objects from modules without any demand on user to register anything. * Another COM object registering stuff * OnHungApp handler * Avoid "AutoIt Error" message box in unknown errors * HTML editor * winhttp.au3 related : * https://www.autoitscript.com/forum/topic/206771-winhttpau3-download-problem-youre-speaking-plain-http-to-an-ssl-enabled-server-port/ "Homo sum; humani nil a me alienum puto" - Publius Terentius Afer"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming" , be and \\//_. Anticipating Errors : "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty." Signature last update: 2023-04-24
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now