Problem find line duplicate from log

[anhchangtk]

I'm want create project find line duplicate, last save with file name defaul, the project get info form 1 text file, last project will find line duplicate from log file save all line duplicate and line had tested

Me had use code

#include <Array.au3>
#include<File.au3>
#include <ButtonConstants.au3>
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>

$filename="testduplication.txt"
$file = FileOpen($filename)
$log = "Duplication.log"
While 1
$line = FileReadLine($file,1)
FileWirteLine($log,$line)
WEnd
Fileclose($file)
FileClose($log)

Me can make project write line from testduplication.txt to Duplication.log . But i'm want write line done, the app will find duplication in Duplication.log

Can you help me ?

Thanks you very much

[foster74]

Let's say testduplication.txt is:

Dog

Cat

Dog

You want Duplication.log to be:

Dog

Cat

Or

Dog?

[anhchangtk]

testduplication.txt will save

Dog

Cat

Duplication.log will save

Dog

Cat

Dog

Please hlep me Thanks you

[foster74]

Like this?

#Include <Array.au3>
#Include <File.au3>
$file1 = FileOpenDialog("Input", @scriptdir, "Text Documents (*.txt)", 1)
$file2 = FileOpenDialog("Output", @scriptdir, "Text Documents (*.txt)", 1)
Dim $Darray
Dim $Darray2[1]
_FileReadToArray($file1,$Darray)
_ArrayDelete($Darray,0)
$Before_uni = ubound($Darray)
$Darray = _ArrayUnique($Darray)
_ArrayDelete($Darray,0)
$After_uni = ubound($Darray)
_FileWriteFromArray($file2,$Darray)
$Results = $Before_uni - $After_uni
$s = 's'
If $Results = 1 then $s = ''
msgbox(0,"Deleted",$Results & ' Duplicate line'&$s&' deleted.')

[anhchangtk]

me run the script, been report error

$Darray = _ArrayUnique($Darray)
$Darray = ^ ERROR

[foster74]

Did you put line "#Include <Array.au3>"? If you did tell me what error you get when running:

#Include <Array.au3>
#Include <File.au3>
$file1 = FileOpenDialog("Input", @scriptdir, "Text Documents (*.txt)", 1)
$file2 = FileOpenDialog("Output", @scriptdir, "Text Documents (*.txt)", 1)
Dim $Darray
Dim $Darray2[1]
_FileReadToArray($file1,$Darray)
_ArrayDelete($Darray,0)
$Before_uni = ubound($Darray)
$Darray = _ArrayUnique($Darray)
msgbox(0,"Error",'ArrayUnique @Error = ' & @Error)
_ArrayDelete($Darray,0)
$After_uni = ubound($Darray)
_FileWriteFromArray($file2,$Darray)
$Results = $Before_uni - $After_uni
$s = 's'
If $Results = 1 then $s = ''
msgbox(0,"Deleted",$Results & ' Duplicate line'&$s&' deleted.')

[anhchangtk]

now me had you array + for but script not run

#Include <Array.au3>
#Include <File.au3>

$filename = "test.txt"
$filelog = "log.log"
$filedone = "done.txt"
$gd = "|"
$vt = "1"
$log = fileopen($filelog,0)
$file = FileOpen($filename, 0)
$done = fileopen($filedone, 1)
If $file = -1 Then
    msgbox(16,"Error","Unable to open file.")
Exit
EndIf

For 1 to $line
$linef = FileReadLine($file)
$linel = filereadline($log)
$arrayf = StringSplit($linef,$gd)
$array1 = $arrayf[$vt]
$arrayl = StringSplit($linel,$gd)
$array2 = $arrayl[$vt]
if $array1 == $array2 Then
filewriteline($log,$array1 & ">> Had saved database, check log")
Else
FileWriteLine($done,$array1)
EndIf
Next
FileClose($file)
FileClose($log)
FileClose($done)

This is terms test.txt

| Dog | 1 | 22.000 | 19.03.2009 |
| Cat | 1 | 32.000 | 19.03.2009 |
| Pig | 1 | 50.000 | 19.03.2009 |

This is terms log.log

| Dog | 1 | 22.000 | 19.03.2009 |
| Cat | 1 | 32.000 | 19.3.2009 |
| Dog | 1 | 22.000 | 19.03.2009 |
| Cat | 1 | 32.000 | 19.3.2009 |
| Dog | 1 | 22.000 | 19.03.2009 |
| Cat | 1 | 32.000 | 19.3.2009 |
| Dog | 1 | 22.000 | 19.03.2009 |
| Cat | 1 | 32.000 | 19.3.2009 |
| Dog | 1 | 22.000 | 19.03.2009 |
| Cat | 1 | 32.000 | 19.3.2009 |
| Dog | 1 | 22.000 | 19.03.2009 |
| Cat | 1 | 32.000 | 19.3.2009 |
| Dog | 1 | 22.000 | 19.03.2009 |
| Cat | 1 | 32.000 | 19.3.2009 |

If the script run it will save to file done.txt

| Pig | 1 | 50.000 | 19.03.2009 |

Please help me

[Authenticity]

expandcollapse popup

#include <Array.au3>
#include <File.au3>

Dim $sText = '| Dog | 1 | 22.000 | 19.03.2009 |' & @CRLF & _
             '| Cat | 1 | 32.000 | 19.3.2009 |' & @CRLF & _
             '| Dog | 1 | 22.000 | 19.03.2009 |' & @CRLF & _
             '| Cat | 1 | 32.000 | 19.3.2009 |' & @CRLF & _
             '| Dog | 1 | 22.000 | 19.03.2009 |' & @CRLF & _
             '| Cat | 1 | 32.000 | 19.3.2009 |' & @CRLF & _
             '| Dog | 1 | 22.000 | 19.03.2009 |' & @CRLF & _
             '| Cat | 1 | 32.000 | 19.3.2009 |' & @CRLF & _
             '| Dog | 1 | 22.000 | 19.03.2009 |' & @CRLF & _
             '| Cat | 1 | 32.000 | 19.3.2009 |' & @CRLF & _
             '| Dog | 1 | 22.000 | 19.03.2009 |' & @CRLF & _
             '| Cat | 1 | 32.000 | 19.3.2009 |' & @CRLF & _
             '| Dog | 1 | 22.000 | 19.03.2009 |' & @CRLF & _
             '| Cat | 1 | 32.000 | 19.3.2009 |'
             
FileWrite(@ScriptDir & '\testduplications.txt', $sText) ; ... just to demonstrate.

Dim $sFileText = FileRead(@ScriptDir & '\testduplications.txt')

Dim $aLines = StringSplit($sFileText, @CRLF, 3)
_ArrayDisplay($aLines, Default, -1, Default, Chr(0xFF))

Dim $i = 0, $j = 1
Dim $sTmp

While $i < UBound($aLines)-1
    $j = _ArraySearch($aLines, $aLines[$i], $j)
    
    If $j <> -1 Then
        _ArrayDelete($aLines, $j)
        ;$j =+ 1
    Else
        $i += 1
        $j = $i + 1
    EndIf
WEnd

_ArrayDisplay($aLines, Default, -1, Default, Chr(0xFF))
_FileWriteFromArray(@ScriptDir & '\Duplicates.log', $aLines)

What is the purpose of the terms file? You search for duplicates of lines on the current text aren't you?

[anhchangtk]

Result display is correct but i'm want export to file *.txt

And the script will find line dup in the testduplications.txt and Duplicates.log

Your script only check dup in the testduplications.txt and save Duplicates.log but check on 2 file

[Authenticity]

Use this loop then:

While $i < UBound($aLines)-1
    $j = _ArraySearch($aLines, $aLines[$i], $j)
   
    If $j <> -1 Then
        FileWriteLine($hFileDuplicated, $aLines[$j]) 
        _ArrayDelete($aLines, $j)
        ;$j =+ 1
    Else
        $i += 1
        $j = $i + 1
    EndIf
WEnd

[anhchangtk]

@foster74: Thanks you, but your script not run good, sript been error me had add #include <Array.au3> but not run

@Authenticity: Can you make to me complete script ??? Complete by code you and this code me or you can make a code complete by you give me

#include <Array.au3>
#include<File.au3>
#include <ButtonConstants.au3>
#include <EditConstants.au3>
#include <GUIConstantsEx.au3>
#include <StaticConstants.au3>
#include <WindowsConstants.au3>

$filename="testduplication.txt"
$file = FileOpen($filename)
$log = "Duplication.log"
While 1
$line = FileReadLine($file,1)
FileWirteLine($log,$line)
WEnd
Fileclose($file)
FileClose($log)

Me had apply _arraysearch to my script but the script run done not save to file

Please help me

Thanks you

Edited March 19, 2009 by anhchangtk

[Authenticity]

How villain would I be if I steal the learning treasure from you? ;]

Anyway, try the loop as an example, it may not suite your needs but it's the way to go, or in much clever way which I don't know ;[.