Sign in to follow this  
Followers 0
iahngy

Extract a big block of text out of a big file

10 posts in this topic

Hi ,

The file I hve is about 6MB ..I 'd like to extract out like 3000 lines starting from a certain keyword text in that file and end by another keyword text. If I use traditional way like a while loop with filereadline and search for that keyword using regex, it takes abit long . I wonder if it is possible to use regex to extract out such big block of text using the string from fileread?

I tried the patterns I learnt so far from Malky, Mikell , Melba and some others but it doesn't work for a big

block .

Attached the example file ..maybe I 'd like to start take out the data block  from test1000 to test4000 or wherever you can help with 3000 line block :)

 

Share this post


Link to post
Share on other sites



What about trying StringBetween? (That uses regex)

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

_StringBetween() isn't always great...

#include <Array.au3>
#include <Constants.au3>

Local $sData =  FileRead('big.txt')
Local $aSRE = StringRegExp($sData, '(?s)^test1\R(.*)\Rtest1000\R', $STR_REGEXPARRAYGLOBALMATCH)
If Not @error Then
    ConsoleWrite($aSRE[0])
EndIf
Edited by guinness

_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

Try this:

$sResult = StringRegExpReplace(FileRead(@ScriptDir & "\big.txt"), "(?si)(?:.*)test999\s+(.*?)test4001\s+(.*)", "$1")

ConsoleWrite($sResult & @CRLF)

Br,

UEZ


Please don't send me any personal message and ask for support! I will not reply!

Selection of finest graphical examples at Codepen.io

The own fart smells best!
Her 'sikim hıyar' diyene bir avuç tuz alıp koşma!
¯\_(ツ)_/¯

Share this post


Link to post
Share on other sites

#5 ·  Posted (edited)

$sResult = _Extract("big.txt", "test1000", "test4000")
FileWrite("1.txt", $sResult)

Func _Extract($file, $from, $to)
  Return StringRegExpReplace(FileRead($file), '(?is).*?(' & $from & '\W.*?' & $to & ')\W?.*', "$1")
EndFunc

non included, the way to do if keywords contain special characters  :)

And how will you manage this if there are several occurences of the keyword(s) in the text ?

Edited by mikell

Share this post


Link to post
Share on other sites

Thank you all..most of your code works. I found out stringregex only works with max of about 4 million of chars ( or around 4500000) ( I use stringlen to count the string from fileread , my file has len:19129246 chars.

$s = FileRead($p2)
$l = StringLen($s)
ConsoleWrite('len:' &$l &@crlf)

if I use stringleft to take out  about 4 million chars, the regex works..but more than that it wont find it. it dump out the whole main file.

$sl = StringLeft($s, 4500000) ;min is 150000 for one unit
$sResult = StringRegExpReplace($sl, "(?si)(?:.*)(d_unit123s+.*?)d_unit456w+s+(.*)", "$1")

So I guess I have to cut that main string into some parts to search for certain unit.

Mikell,

the file I parse it has keyword of the unit IDs ...it has some keyword like d_id_unit123 ..something like that ...

Share this post


Link to post
Share on other sites

#7 ·  Posted (edited)

Beware that d in a pattern represents a digit. If the bounding "keywords" actually contain d, then escape those backslash d or enclose the keywords in Q ... E  as in Qd_id_unit123E wich instruct the engine to treat verbatim whatever is inside.

You can try placing (*LIMIT_MATCH=9999999) at the very start of your pattern, with 9999999 being a large enough value.

Lastly if you still hit a brick wall, I suggest to use StringRegExp (not *Replace).

Edited by jchd

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Share this post


Link to post
Share on other sites

Lastly if you still hit a brick wall, I suggest to use StringRegExp (not *Replace).

Yes, I was on the right track!
1 person likes this

_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

#9 ·  Posted (edited)

Hi JHCD, I tried Guiness's  stringregex code , it works w/o even putting any limit in it.

as for other stringregrexreplace of Mikell and Malky, it doesn't do anything with the set limit to 9999999 or any  number.

I tried to work on more stringregex to see if it can repeat of search for the same unit the rest of file...but doenst work so far...and I dun know how to set the end keyword for a unit block that is  at the end of the file..would be cool if I  can get help on that :) .

Edited by iahngy

Share this post


Link to post
Share on other sites

so I changed the end keyword for the block (test result )which always included in instead of another unit id .

it works great ...it able to search for again for the same unit ..I added the trick from Mikell the question mark ? for middle and end of pattern.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0