Jump to content

Help Finding duplicates in a file


KeeWay
 Share

Recommended Posts

I am still a newbe even though I have been using auto it for about a year doing simple ftp scripts and process watching, I am not a programmer but have managed so far by searching the forums gettng examples and using them etc.. this time I find myself in need some help and appreciate any assistance you can give...

what i am try to do is from my ftp log that I generate while sending the files up to our host servers i need to compare the size of the file to the bytes sent, i have gotten it down to creating a temp file that has the file name and size/bytes sent but i need to evaluate the file and remove all lines that the size and bytes sent are equal. here is the test code i have so far but i am kind of stumped on how to proceed to the next step. i have looked at the many posts about finding the unique or duplicate lines but i only want to leave the entries that do not match or have any of the sent bytes blank..

#include <Date.au3>
#include <file.au3>
#include <array.au3>
AutoItSetOption("TrayIconHide", 1);hides icon
$varFP = IniRead("Settings.ini", "Settings", "FilePath", "NotFound")
$varBP = IniRead("Settings.ini", "Settings", "BackupPath", "NotFound")
$varOP = IniRead("Settings.ini", "Settings", "OutgoingPath", "NotFound")
$varSC = IniRead("Settings.ini", "Settings", "ScriptsPath", "NotFound")
$varWK = IniRead("Settings.ini", "Settings", "WorkPath", "NotFound")
$varER = IniRead("Settings.ini", "Settings", "Error", "NotFound")
$varSubject = IniRead("Settings.ini", "Settings", "Subject", "NotFound")
$varBody = IniRead("Settings.ini", "Settings", "Body", "NotFound")
$varTO = IniRead("Settings.ini", "Settings", "TO", "NotFound")

Dim $aRecords, $yRecords, $s_TempFile, $fsize, $member, $barray

If Not _FileReadToArray($varWK & "FTPLOG_10112006-073426.log", $aRecords) Then
;MsgBox(4096, "Error", " Error reading log to Array  error:" & @error)
    Exit
EndIf
$s_TempFile = _TempFile()

For $x = 1 To $aRecords[0]
    If StringInStr($aRecords[$x], "Sending file to member") Then
        $mtemp = StringTrimLeft($aRecords[$x], 27)
        $member = StringLeft($mtemp, 9)
        $fsize = FileGetSize($varOP & $member)
                        
        FileWriteLine($s_TempFile, $member & "=" & $fsize)
        
    ;MsgBox(4096, "Name suitable for new temporary file", $s_TempFile)
        
    ElseIf StringInStr($aRecords[$x], "bytes sent") Then
        $btemp = StringTrimLeft($aRecords[$x], 5)
        $barray = StringSplit($btemp, " ")
        
        FileWriteLine($s_TempFile, $member & "=" & $barray[1])
        
    EndIf
Next

here is the results of reading the log file and writing to the temp file. The ftp file size of D07335569 shows that the first entry is 687 bytes and the second entry is the bytes sent of 685 and i only want to either leave that one in this file or another one so that i can re-send the file back up to the host server.

D07335569=687
D07335569=687
S00727981=641
S00727981=641
D07335569=687
D07335569=685
S00727981=641
S00727981=641

i can post the log file if you need....

thanks

james

Link to comment
Share on other sites

The nonautoit solution is to use sort and uniq from gnuwin32 coreutils

run(@comspec & " /C sort " & $filename & " |uniq > result.txt"

And then read the resulting file.

The AutoIt solution is a bit mor complex but not to much. You need some sort of "associative array" (search the forums and you will find samples).

Assosiate each data item you want to be unique with a count or placeholder...

Or just add the item to a string if it does not allready exists.

If NOT StringInStr($data, $test) Then
    $data = $data & "|" & $test
EndIf
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...