Jump to content

Remove duplicate line entries from huge files

Simucal

By Simucal
in AutoIt General Help and Support

 Share

Recommended Posts

Simucal Universalist

Simucal

Posted
  • Author
Posted (edited)

Hmm, is there a way we could remove the dupes if the lines were not adjacent using the RegExpReplace?

I'm not 100% postive they will never come in the form:

<SourceFile>TreeLock 1.24.cs</SourceFile>
<SourceFile>Pogue 2.33.cs</SourceFile>
<SourceFile>TreeLock 1.24.cs</SourceFile>
<SourceFile>Pogue 2.33.cs</SourceFile>
<SourceFile>TreeLock 1.24.cs</SourceFile>
<SourceFile>Pogue 2.33.cs</SourceFile>
<SourceFile>TreeLock 1.24.cs</SourceFile>
<SourceFile>Pogue 2.33.cs</SourceFile>

as well as adjacent to eachother (and there can be varying amounts from 1-10ish..)

Edited by Simucal
AutoIt Scripts:Aimbot: Proof of Concept - PixelSearching Aimbot with several search/autoshoot/lock-on techniques.Sliding Toolbar - Add a nice Sliding Toolbar to your next script. Click the link to see an animation of it in action!FontInfo UDF - Get list of system fonts, or search to see if a particular font is installed.Get Extended Property UDF - Retrieve a files extended properties (e.g., video/image dimensions, file version, bitrate of song/video, etc)
PsaltyDS Universalist

PsaltyDS

Posted
Posted

Hmm, is there a way we could remove the dupes if the lines were not adjacent using the RegExpReplace?

I'm not 100% postive they will never come in the form:

<SourceFile>TreeLock 1.24.cs</SourceFile>
<SourceFile>Pogue 2.33.cs</SourceFile>
<SourceFile>TreeLock 1.24.cs</SourceFile>
<SourceFile>Pogue 2.33.cs</SourceFile>
<SourceFile>TreeLock 1.24.cs</SourceFile>
<SourceFile>Pogue 2.33.cs</SourceFile>
<SourceFile>TreeLock 1.24.cs</SourceFile>
<SourceFile>Pogue 2.33.cs</SourceFile>

as well as adjacent to eachother (and there can be varying amounts from 1-10ish..)

I don't know about getting the whole file in one shot, but in your original example they were grouped between <LoadClasses> and </LoadClasses> tags. Is that consistent throughout the file? Does the same .cs file name ever show up in more than one <LoadClasses> grouping? Details like that will determine how hard it is, I would think.

:)

Valuater's AutoIt 1-2-3, Class... Is now in Session!For those who want somebody to write the script for them: RentACoder"Any technology distinguishable from magic is insufficiently advanced." -- Geek's corollary to Clarke's law
randallc Universalist

randallc

Posted
Posted (edited)

Hmm, is there a way we could remove the dupes if the lines were not adjacent using the RegExpReplace?

as well as adjacent to eachother (and there can be varying amounts from 1-10ish..)

Hi,

1-10 different files? (or 1-10 duplications of 2 files?)

Maybe you need to post a segment of your files with all possible permutations?

Best, Randall

[should be easy if there's only 1-10 pairs, but only 2 files each time!] - 

(?m)^(.*\r\n.*\r\n)\1+
Edited by randallc
ExcelCOM... AccessCom.. Word2... FileListToArrayNew...SearchMiner... Regexps...SQL...Explorer...Array2D.. _GUIListView...array problem...APITailRW
Simucal Universalist

Simucal

Posted
  • Author
Posted (edited)

Lets just say that all <SourceFiles> can be duplicated any number of times in any order.. wether that be in a pattern, adjacent, or not. There could possibly be up to 10 different <SourceFiles> such as "TreeLock.cs" or "Pogue.cs" or "Warrior.cs". I need the clear the dupes of them as quickly as possibly because these files could be filled to the brim with dupes.

The RegExpReplace seems like it is by far the fastest method but I dont know if a pattern exists that will allow for this kind of removal.

<?xml version="1.0"?>
<GliderConfig>
  <WaypointCloseness>5.0</WaypointCloseness>
  <WebNotifyCredentials>
  </WebNotifyCredentials>
  <WebNotifyEnabled>False</WebNotifyEnabled>
  <WebNotifyURL>
  </WebNotifyURL>
  <WindowPos>-1244,346</WindowPos>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>Pogue 2.33.cs</SourceFile>
    <SourceFile>Pogue 2.33.cs</SourceFile>
    <SourceFile>Warrior.cs</SourceFile>
    <SourceFile>Pogue 2.33.cs</SourceFile>
    <SourceFile>Pogue 2.33.cs</SourceFile>
    <SourceFile>Pogue 2.33.cs</SourceFile>
    <SourceFile>Pogue 2.33.cs</SourceFile>
    <SourceFile>Warrior.cs</SourceFile>
    <SourceFile>Pogue 2.33.cs</SourceFile>
    <SourceFile>Pogue 2.33.cs</SourceFile>
    <SourceFile>Pogue 2.33.cs</SourceFile>
    <SourceFile>Warrior.cs</SourceFile>
    <SourceFile>Warrior.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
    <SourceFile>TreeLock 1.24.cs</SourceFile>
  </LoadClasses>
</GliderConfig>
Edited by Simucal
AutoIt Scripts:Aimbot: Proof of Concept - PixelSearching Aimbot with several search/autoshoot/lock-on techniques.Sliding Toolbar - Add a nice Sliding Toolbar to your next script. Click the link to see an animation of it in action!FontInfo UDF - Get list of system fonts, or search to see if a particular font is installed.Get Extended Property UDF - Retrieve a files extended properties (e.g., video/image dimensions, file version, bitrate of song/video, etc)
randallc Universalist

randallc

Posted
Posted

Hi,

Looks like it is possible after all.

I posted a query at a Regexp forum, and there are some answers;

Regexp Remove duplicate lines

Along the lines of "(?m)^(.*+)$(?=(?:\r?\n.*+)*?\r?\n\1$)" and "^(.*+)\r?\n(?=(?:.*+\r?\n)*?\1$)"

I haven't got them working in AutoIt yet; maybe psaltyDS has time to get it going?

I will have time later..

Best, randall

ExcelCOM... AccessCom.. Word2... FileListToArrayNew...SearchMiner... Regexps...SQL...Explorer...Array2D.. _GUIListView...array problem...APITailRW
Simucal Universalist

Simucal

Posted
  • Author
Posted

Great find randallc! I just read that regexp forum thread you posted. I've attempted to convert "(?m)^(.*+)$(?=(?:\r?\n.*+)*?\r?\n\1$)" and "^(.*+)\r?\n(?=(?:.*+\r?\n)*?\1$)" to work with autoit's regular expression as well but have had little success. I'll keep tinkering with it.

PsaltyDS, maybe if you get a chance take a look at that pattern.

AutoIt Scripts:Aimbot: Proof of Concept - PixelSearching Aimbot with several search/autoshoot/lock-on techniques.Sliding Toolbar - Add a nice Sliding Toolbar to your next script. Click the link to see an animation of it in action!FontInfo UDF - Get list of system fonts, or search to see if a particular font is installed.Get Extended Property UDF - Retrieve a files extended properties (e.g., video/image dimensions, file version, bitrate of song/video, etc)
randallc Universalist

randallc

Posted
Posted

Hi,

I do have it wrking, but depends on the original fi;le i make!

'I need a more "real" file to test, I think;

But it returns only one set of everything, but no "Sourcefile" is missing; depends what you want

StringRegExpReplace($sFileData, "(?m)^(.*+)\r?\n(?=(?:.*+\r?\n)*?\1$)", ""); ; errors out if multipke "1st line XML version in file;o脻梅 脵芦颅垄+脴矛I碌脮脕脥I谩脌脤鹿脭脤(铆酶脮脩陆%脨脥]脡脕脕脡}U脥鹿脥陇玫盲矛隆脩脩脌猫录陆脡谩脵楼鹿陆麓陆陆脡脮碌脤录脛录脤脺脿盲脺陆M隆陆脻Q隆脡鹿脥脕脿(铆酶楼鹿卤脮卤脨铆脡脡盲鹿脭脤脨矛矛隆脩脩脌猫录陆脻脻脺鹿脡脮卤脠碌谩脕脡脥脥楼陆鹿脤鹿楼鹿录陆脮脕卤楼脩卤楼鹿脤鹿隆脩碌掳铆x 赂漏l脌盲脠铆赂脌盲脠铆脡t卢陇脌盲脠矛脛卢录脌脤脴矛脛掳脜脮陆脨铆x 赂篓陇 脌盲脠铆脠眉脌盲脠铆赂脌盲脠矛脛陇卢脌脤脴矛脜脮陆脨矛掳)1陆掳脌脤脴铆脥M脡楼卤脠掳脌脤脴铆脥M脡楼卤0么M脡楼脕脩楼脠碌脌矛脜脮陆脨矛脌盲脠铆Q脥脨鹿脩谩脨脜脮陆脨矛掳脌脤脴铆脥M脡楼卤么M脡楼脕脩楼脠碌脌矛脜脮陆脨矛脌盲脠铆Q脥脩IM碌卤掳鹿脩谩脨脜脮陆脨矛掳脌脤脴铆脥M脡楼卤4么M脡楼脕脩楼脠碌脌矛脜脮陆脨矛脌盲脠铆Q脥脩I5楼脮麓鹿脩谩脨脜脮陆脨矛)
陆鹿脥陆卤]脡楼脩 脜脮陆脨铆楼卤脩M楼茅 脌脤脴铆脥M脡楼卤陇么脜脮陆脨矛碌脌矛楼卤脩M楼茅 脌脤脴铆脥M脡楼卤陇碌脌矛1陇(脌脤脴铆脥I脥脮卤脩楼卤么M脡楼脕脩楼脠碌脌矛脜脮陆脨矛脌盲脠铆I脥脮卤脩脤鹿脩谩脨脜脮陆脨矛)楼卤卤脩 脌脤脴铆脥I脥脮卤脩楼卤陇(脌脤脴铆脩楼碌脡脥脩碌脌脛么Q楼碌脡%鹿楼脨 陇)1陆掳脌脤脴铆脥楼卤脩么楼卤I 脌脤脴铆脥M脡楼卤陇掳脌脤脴铆脥楼卤脩脠么脌脤脴铆脥楼卤脩矛脡楼鹿脕脮脨楼卤脤脥楼鹿卤脥脩脡楼鹿)
陆鹿脥陆卤]脡楼脩 脜脮陆脨铆IQ楼碌么脜脮陆脨矛碌脌矛I陆脮鹿隆Q楼碌脡楼 脌脤脴铆脩楼碌脡脥脩碌脌脛陇陇碌脌矛脜脮陆脨矛脜脮陆脨矛碌脌矛Q碌脌矛脜脮陆脨矛碌脥脜脮陆脨矛碌脌矛1陇(脌脤脴铆脩楼碌脡脥脩碌脌脤么Q楼碌脡%鹿楼脨 陇(矛么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么(脌脤脴铆脥楼卤脩脛么M脩脡楼鹿I谩脕I脕卤 脌脤脴铆脥楼卤脩掳脜脮陆脨矛 媒麓楼x 赂篓卢陇脌盲脠铆脠眉脌盲脠铆赂 眉么 眉猫赂篓卢脌盲脠铆脠眉脌盲脠铆赂陇篓眉脌盲脠矛脛脌脤脴矛陇脜脮陆脨矛掳脜脮陆脨矛脜脮陆脨矛陇矛矛脡脡陆脡脤陆脮脨楼碌脮卤脩楼脕颅脜脮陆脨矛脜脥脨卤楼鹿a50脵脡脥楼陆赂楼赂楼卤矛谩脩脡脜脮陆脨矛陆卤楼脡
陆鹿楼脜脮陆脨矛脨陆脩陆录(矛么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么么)
陆鹿脥陆卤]脡楼脩 脜脮陆脨铆I谩脌Q楼碌么脜脮陆脨矛碌脌矛I陆脮鹿隆Q楼碌脡楼 脌脤脴铆脩楼碌脡脥脩碌脌脤陇陇碌脌矛脜脮陆脨矛脜脮陆脨矛碌脌矛Q碌脌矛脜脮陆脨矛碌脥脜脮陆脨矛碌脌矛1陇(脌脤脴铆脩楼碌脡脥脩碌脌脠么Q楼碌脡%鹿楼脨 陇)楼卤]脡楼脩 脌脤脴铆脥I脥脮卤脩楼卤掳脌脤脴铆脥楼卤脩脛陇)
陆鹿脥陆卤]脡楼脩 脜脮陆脨铆]脡楼脩Q楼碌么脜脮陆脨矛碌脌矛I陆脮鹿隆Q楼碌脡楼 脌脤脴铆脩楼碌脡脥脩碌脌脠陇陇碌脌矛脜脮陆脨矛脜脮陆脨矛碌脌矛Q碌脌矛脜脮陆脨矛碌脥脜脮陆脨矛碌脌矛1陇)
陆鹿脥陆卤]脡楼脩 脜脮陆脨铆Q陆脩掳Q楼碌么脜脮陆脨矛碌脌矛I陆脮鹿隆Q楼碌脡楼 脌脤脴铆脩楼碌脡脥脩碌脌脛陇陇碌脌矛脜脮陆脨矛脜脮陆脨矛碌脌矛Q碌脌矛脜脮陆脨矛碌脥脜脮陆脨矛碌脌矛1陇(铆酶
陆鹿脥陆卤]脡楼脩 脜脮陆脨铆M脩脡楼鹿5楼 脌脤脴铆脥楼卤脩脛掳脠脠脛掳脺脌脌陇么脜脮陆脨矛碌脌矛M脩脡楼鹿5楼 脌脤脴铆脥楼卤脩脛掳脠脠脛掳脺脌脌陇碌脌矛1陇)
陆鹿脥陆卤]脡楼脩 脜脮陆脨矛脌脤脴铆脥楼卤脩脛么脜脮陆脨矛碌脌矛脌脤脴铆脥楼卤脩脛碌脌矛1陇
Best, randall
ExcelCOM... AccessCom.. Word2... FileListToArrayNew...SearchMiner... Regexps...SQL...Explorer...Array2D.. _GUIListView...array problem...APITailRW

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing
×
×
  • Create New...