Remove duplicate line entries from huge files

[Simucal]

Hmm, is there a way we could remove the dupes if the lines were not adjacent using the RegExpReplace?

I'm not 100% postive they will never come in the form:

<SourceFile>TreeLock 1.24.cs</SourceFile>
<SourceFile>Pogue 2.33.cs</SourceFile>
<SourceFile>TreeLock 1.24.cs</SourceFile>
<SourceFile>Pogue 2.33.cs</SourceFile>
<SourceFile>TreeLock 1.24.cs</SourceFile>
<SourceFile>Pogue 2.33.cs</SourceFile>
<SourceFile>TreeLock 1.24.cs</SourceFile>
<SourceFile>Pogue 2.33.cs</SourceFile>

as well as adjacent to eachother (and there can be varying amounts from 1-10ish..)

Edited December 20, 2007 by Simucal

[PsaltyDS]

Hmm, is there a way we could remove the dupes if the lines were not adjacent using the RegExpReplace?

I'm not 100% postive they will never come in the form:

<SourceFile>TreeLock 1.24.cs</SourceFile>
<SourceFile>Pogue 2.33.cs</SourceFile>
<SourceFile>TreeLock 1.24.cs</SourceFile>
<SourceFile>Pogue 2.33.cs</SourceFile>
<SourceFile>TreeLock 1.24.cs</SourceFile>
<SourceFile>Pogue 2.33.cs</SourceFile>
<SourceFile>TreeLock 1.24.cs</SourceFile>
<SourceFile>Pogue 2.33.cs</SourceFile>

as well as adjacent to eachother (and there can be varying amounts from 1-10ish..)

I don't know about getting the whole file in one shot, but in your original example they were grouped between <LoadClasses> and </LoadClasses> tags. Is that consistent throughout the file? Does the same .cs file name ever show up in more than one <LoadClasses> grouping? Details like that will determine how hard it is, I would think.

[randallc]

Hmm, is there a way we could remove the dupes if the lines were not adjacent using the RegExpReplace?

as well as adjacent to eachother (and there can be varying amounts from 1-10ish..)

Hi,

1-10 different files? (or 1-10 duplications of 2 files?)

Maybe you need to post a segment of your files with all possible permutations?

Best, Randall

[should be easy if there's only 1-10 pairs, but only 2 files each time!] -

(?m)^(.*\r\n.*\r\n)\1+

Edited December 20, 2007 by randallc

[Simucal]

Lets just say that all <SourceFiles> can be duplicated any number of times in any order.. wether that be in a pattern, adjacent, or not. There could possibly be up to 10 different <SourceFiles> such as "TreeLock.cs" or "Pogue.cs" or "Warrior.cs". I need the clear the dupes of them as quickly as possibly because these files could be filled to the brim with dupes.

The RegExpReplace seems like it is by far the fastest method but I dont know if a pattern exists that will allow for this kind of removal.

expandcollapse popup

<?xml version="1.0"?>
<GliderConfig>
  <WaypointCloseness>5.0</WaypointCloseness>
  <WebNotifyCredentials>
  </WebNotifyCredentials>
  <WebNotifyEnabled>False</WebNotifyEnabled>
  <WebNotifyURL>
  </WebNotifyURL>
  <WindowPos>-1244,346</WindowPos>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>Pogue 2.33.cs</SourceFile>
    <SourceFile>Pogue 2.33.cs</SourceFile>
    <SourceFile>Warrior.cs</SourceFile>
    <SourceFile>Pogue 2.33.cs</SourceFile>
    <SourceFile>Pogue 2.33.cs</SourceFile>
    <SourceFile>Pogue 2.33.cs</SourceFile>
    <SourceFile>Pogue 2.33.cs</SourceFile>
    <SourceFile>Warrior.cs</SourceFile>
    <SourceFile>Pogue 2.33.cs</SourceFile>
    <SourceFile>Pogue 2.33.cs</SourceFile>
    <SourceFile>Pogue 2.33.cs</SourceFile>
    <SourceFile>Warrior.cs</SourceFile>
    <SourceFile>Warrior.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
  </LoadClasses>
</GliderConfig>

Edited December 20, 2007 by Simucal

[randallc]

Hi,

Looks like it is possible after all.

I posted a query at a Regexp forum, and there are some answers;

Regexp Remove duplicate lines

Along the lines of "(?m)^(.*+)$(?=(?:\r?\n.*+)*?\r?\n\1$)" and "^(.*+)\r?\n(?=(?:.*+\r?\n)*?\1$)"

I haven't got them working in AutoIt yet; maybe psaltyDS has time to get it going?

I will have time later..

Best, randall

[Simucal]

Great find randallc! I just read that regexp forum thread you posted. I've attempted to convert "(?m)^(.*+)$(?=(?:\r?\n.*+)*?\r?\n\1$)" and "^(.*+)\r?\n(?=(?:.*+\r?\n)*?\1$)" to work with autoit's regular expression as well but have had little success. I'll keep tinkering with it.

PsaltyDS, maybe if you get a chance take a look at that pattern.

[randallc]

Hi,

I do have it wrking, but depends on the original fi;le i make!

'I need a more "real" file to test, I think;

But it returns only one set of everything, but no "Sourcefile" is missing; depends what you want

StringRegExpReplace($sFileData, "(?m)^(.*+)\r?\n(?=(?:.*+\r?\n)*?\1$)", ""); ; errors out if multipke "1st line XML version in file;oÝ÷ Ù«­¢+ØìIµÕÁÍIáÀ̹ÔÌ(íøÕѽ%ÐÍ]ÉÁÁÉ}U͹ͤõäì¡ÑÑÀè¼½ÉáÙ¥¹½´½½Éյ̼ļÌÜàäܽM¡½ÝQ¡É¹ÍÁà(íø¥¹±Õ±ÐíÉÉä¹ÔÌÐìì¡ÑÑÀè¼½ÝÝܹÉձȵáÁÉÍÍ¥½¹Ì¹¥¹¼½ÕÁ±¥Ñ±¥¹Ì¹¡Ñµ°íx ¸©lÀäÈí¸ÀäÈíÉt¬¤ÀäÈìĬ¼ÀÌØìÄ°ÅÕ½Ðíx ¸¨¤ ÀäÈíÈüÀäÈí¸ÀäÈìĤ¬ÀÌØìÅÕ½Ðì°)1½°ÀÌØíÍMÉ¥±È°ÀÌØíÍMÉ¥±0ôMÉ¥ÁѥȵÀìÅÕ½ÐìÀäÈíQÍйÑáÐÅÕ½Ðì°ÀÌØíÍMÉ¥±ôMÉ¥ÁѥȵÀìÅÕ½ÐìÀäÈíQÍÑIMµ±°¹ÑáÐÅÕ½Ðì°ÀÌØíÍMÉ¥±4ôMÉ¥ÁѥȵÀìÅÕ½ÐìÀäÈíQÍÑI5¥Õ´¹ÑáÐÅÕ½Ðì)
½¹Í½±]É¥Ñ ÅÕ½Ðí¥±ÑM¥é ÀÌØíÍMÉ¥±¤ôÅÕ½ÐìµÀ쥱ÑM¥é ÀÌØíÍMÉ¥±¤µÀì1¤(ÀÌØíÍIÍձѥ±ôMÉ¥ÁѥȵÀìÅÕ½ÐìÀäÈíIÍÕ±Ñ̹ÑáÐÅÕ½Ðì)¥±±Ñ ÀÌØíÍIÍձѥ±¤(ÀÌØíÑ¥µÉÍѵÀÄôQ¥µÉ%¹¥Ð ¤)1½°ÀÌØíÍ¥±Ñô¥±I ÀÌØíÍMÉ¥±¤°ÀÌØíÍ¥±ÑÈôÀÌØíÍ¥±ÑìÉ¥¹ÁÕÐ¥±ÌÍ¥¹±ÍÑÉ¥¹)
½¹Í½±]É¥Ñ ÅÕ½ÐíIQ¥µôÅÕ½ÐìµÀìI½Õ¹¡Q¥µÉ¥ ÀÌØíÑ¥µÉÍѵÀĤ¤µÀìÅÕ½ÐìÅÕ½ÐìµÀìQµÀìÅÕ½ÐìµÍÅÕ½ÐìµÀì1¤(ÀÌØíÑ¥µÉÍѵÀÌôQ¥µÉ%¹¥Ð ¤(ìôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôô(ÀÌØíÍ¥±ÑÄôMÑÉ¥¹IáÁIÁ± ÀÌØíÍ¥±Ñ°ÅÕ½Ðì ý´¥x ¸¨¬¤ÀäÈíÈüÀäÈí¸ üô ü踨¬ÀäÈíÈüÀäÈí¸¤¨üÀäÈìÄÀÌØì¤ÅÕ½Ðì°ÅÕ½ÐìÅÕ½Ðì¤ììÉɽÉ̽ÕÐ¥µÕ±Ñ¥Á­ÅÕ½ÐìÅÍб¥¹a50ÙÉÍ¥½¸¥¸¥±ìáÑÉÅÕ½Ðì½±¥É
½¹¥ÅÕ½Ðìнѽ¼(ìôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôôô)
½¹Í½±]É¥Ñ ÅÕ½ÐíIáÀQ¥µôÅÕ½ÐìµÀìI½Õ¹¡Q¥µÉ¥ ÀÌØíÑ¥µÉÍѵÀ̤¤µÀìÅÕ½ÐìÅÕ½ÐìµÀìQµÀìÅÕ½ÐìµÍÅÕ½ÐìµÀì1¤(ÀÌØíÑ¥µÉÍѵÀÈôQ¥µÉ%¹¥Ð ¤)¥±]É¥Ñ ÀÌØíÍIÍձѥ±°ÀÌØíÍ¥±ÑĤ)
½¹Í½±]É¥Ñ ÅÕ½Ðí]É¥ÑQ¥µôÅÕ½ÐìµÀìI½Õ¹¡Q¥µÉ¥ ÀÌØíÑ¥µÉÍѵÀȤ¤µÀìÅÕ½ÐìÅÕ½ÐìµÀìQµÀìÅÕ½ÐìµÍÅÕ½ÐìµÀì1¤)
½¹Í½±]É¥Ñ ÅÕ½ÐíQ½Ñ°Q¥µôÅÕ½ÐìµÀìI½Õ¹¡Q¥µÉ¥ ÀÌØíÑ¥µÉÍѵÀĤ¤µÀìÅÕ½ÐìÅÕ½ÐìµÀìQµÀìÅÕ½ÐìµÍÅÕ½ÐìµÀì1¤(íø
½¹Í½±]É¥Ñ ÅÕ½ÐíMÑÉ¥¹5¥ ÀÌØíÍ¥±ÑÄ°ÈÈÄ°ÜÀÀ¤ôÅÕ½ÐìµÀìMÑÉ¥¹5¥ ÀÌØíÍ¥±ÑÄ°ÈÈÄ°ÜÀÀ¤µÀì1¤)
½¹Í½±]É¥Ñ ÅÕ½ÐìÀÌØíÍ¥±ÑÄôÅÕ½ÐìµÀìÀÌØíÍ¥±ÑĵÀì1¤

Best, randall

[randallc]

Hi @Simucal

I can't edit above post;

Do you want to join that forum and ask the questions directly?; add to my thread if you like..

Best, randall