Jump to content

Finding combinations and remove line


Read
 Share

Recommended Posts

I'm glad a working and fast solution has beend found.

As the big input file is still read in one go I would suggest to run this script after rebooting the PC so memory isn't too fragmented.

My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2022-02-19 - Version 1.6.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2017-07-21 - Version 0.4.0.1) - Download - General Help & Support - Example Scripts
OutlookEX (2021-11-16 - Version 1.7.0.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX_GUI (2021-04-13 - Version 1.4.0.0) - Download
Outlook Tools (2019-07-22 - Version 0.6.0.0) - Download - General Help & Support - Wiki
PowerPoint (2021-08-31 - Version 1.5.0.0) - Download - General Help & Support - Example Scripts - Wiki
Task Scheduler (NEW 2022-07-28 - Version 1.6.0.1) - Download - General Help & Support - Wiki

Standard UDFs:
Excel - Example Scripts - Wiki
Word - Wiki

Tutorials:
ADO - Wiki
WebDriver - Wiki

 

Link to comment
Share on other sites

  • Replies 91
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

spiff59, water, et al,

Slick idea! It would be pretty cool if one could process the entire file with a single SRER call per delete mask.

You seem to be the first to report no memory error using Fileread(). I'm guessing a 64-bit machine with 8GB RAM?

The regular expression does need some more work though, as it only partially deletes a line, leaving behind the 6th unmatched number pair and the @CRLF.

Edit: On closer look, the SRER always leaves the @CRLF behind, but only leaves the 6th unmatched number behind when it's the first or last pair on the data line.

Edited by Spiff59
Link to comment
Share on other sites

spiff59,

Yes, 64 bit 8GB RAM however the major difference is that I am not trying to split the file into an array, as you pointed out. I know that my regex's need work ( see my sig). That is something that the origionator of this thread can work out.

kylomas

Forum Rules         Procedure for posting code

"I like pigs.  Dogs look up to us.  Cats look down on us.  Pigs treat us as equals."

- Sir Winston Churchill

Link to comment
Share on other sites

This memory thing was discussed a few years ago, probably in the developers forum (can't find the thread). I remember Valik and I did some shouting at each other there.

@Spiff59

Take a look at Task Manager while running your script

Posted Image

You will notice some things.

Your ToolTip() only shows half the real value. Strings in AutoIt are Unicode so they take two bytes.

Peak Working Set (similar (but not the same) as the Peak Memory Usage in XP) is quite a bit higher than the current, as extra memory is needed for the copying.

I hope this sheds some of your doubts, as you are using (almost) three times the memory (albeit temporary) of what you write in the ToolTip()!!

My results on x64:

Posted Image

x64 on a fresh boot goes one step higher (I only have 6 GB RAM):

Posted Image

Link to comment
Share on other sites

Thanks for explaining!

BTW: Funy avatar :)

My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2022-02-19 - Version 1.6.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2017-07-21 - Version 0.4.0.1) - Download - General Help & Support - Example Scripts
OutlookEX (2021-11-16 - Version 1.7.0.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX_GUI (2021-04-13 - Version 1.4.0.0) - Download
Outlook Tools (2019-07-22 - Version 0.6.0.0) - Download - General Help & Support - Wiki
PowerPoint (2021-08-31 - Version 1.5.0.0) - Download - General Help & Support - Example Scripts - Wiki
Task Scheduler (NEW 2022-07-28 - Version 1.6.0.1) - Download - General Help & Support - Wiki

Standard UDFs:
Excel - Example Scripts - Wiki
Word - Wiki

Tutorials:
ADO - Wiki
WebDriver - Wiki

 

Link to comment
Share on other sites

OK guys, I just woke up so please excuse any major fail which may follow.

1,583,704K?????????????????????????????????

The Fvck?

Yes? K = kilo = 1000. Bytes. So that's 1,5 GB.
Link to comment
Share on other sites

The example from kylomas, scanning the entire input file with a single SRER statement, has so much of a speed advantage over single-record processing that I thought I'd see if I could work the bugs out and get it working properly. The following modified script is the result:

#include <File.au3>
Global $aKeys, $count

$timer    = TimerInit()
_FileReadToArray(@ScriptDir & "DeletePatterns.txt", $aKeys)
If $aKeys[$aKeys[0]] = "" Then $aKeys[0] -= 1
For $i = 1 To $aKeys[0]
    $aKeys[$i] = "(?m)^.*(" & StringReplace($aKeys[$i], " ", ").+(") & ").*rn"
Next

$sFile = FileRead(@ScriptDir & "DataFile.txt")
For $i = 1 To $aKeys[0]
    $sFile = StringRegExpReplace($sFile, $aKeys[$i], "")
    $count += @extended
Next
FileDelete(@ScriptDir & "DataFile_new.txt")
FileWrite(@ScriptDir & "DataFile_new.txt",$sFile)
MsgBox(0,"", $count & " deletions" & @CRLF & Round(TimerDiff($timer)/1000,2) & ' seconds')

I think I may have to scan through my batch scripts and see if any of those can take advantage of using StringRegExpReplace() :)

Link to comment
Share on other sites

spiff59,

Very nice! Stopped working on it as the thread originator has not responded in some days, nor did I see any attempt at code from him/her...

Just ran this with several instances of IE open (forum and streaming a radio broadcast), Live Mail open, and Scite running with no memory problems.

I suspect that because this is valid

$x = stringsplit($x,@crlf,3)
memory usage is multiplied to allow interim copies of the target variable/array.

kylomas

Edited by kylomas

Forum Rules         Procedure for posting code

"I like pigs.  Dogs look up to us.  Cats look down on us.  Pigs treat us as equals."

- Sir Winston Churchill

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...