Comparing LARGE files

[Delta]

I was hoping someone could help me i have two LARGE files each is a list of about 10000 blacklisted sites. I want to combine this list but i don't want to make duplicate entries. Is it possible to compare each list in an autoit script and have it pull out the duplicate entries? or is there an easier way to remove them?

[Das Ami]

Read the files into 2 arrays and if-loop them, that's the only idea I have right now.

[Vindicator209]

Somewhere along the lines of

$line=1
Do
$file1 = FileReadLine("file1.txt",$line)
$file2 = FileReadLine("file2.txt",$line)
$line = $line + 1
If $file1 = $file2 Then
FileWriteLine("NewFile.txt",$file1)
Else
FileWriteLine("NewFile.txt",$file1)
FileWriteLine("NewFile.txt",$file2)
EndIf
Until <===Some State Here

Then you can choose to delete the first 2 files..

havent tested that yet, but its something along those lines

[Delta]

That wouldn't work because its only comparing the same line with each text file and both text files aren't in the same order.

Also i just found out there are comments in the second file. They look like this: [some comment here]. How would i mass remove the comments?

Edited June 15, 2007 by Deltaforce229

[Das Ami]

That wouldn't work because its only comparing the same line with each text file and both text files aren't in the same order.

Also i just found out there are comments in the second file. They look like this: [some comment here]. How would i mass remove the comments?

Stringinstr and Stringreplace.

http://www.autoitscript.com/forum/index.ph...ite=%2Bcomments

[Delta]

Stringinstr and Stringreplace.

http://www.autoitscript.com/forum/index.ph...ite=%2Bcomments

Did you even look at that link? all those topic have to do with comments in autoit I.E. #cs, #ce, ;

[Jos]

One time job?

Download PSPad editor, copy the 2 files into one and do option Edit/Sort and tick remove duplicates...

[Delta]

JdeB you rule!

*bows before the almighty JdeB*