MattHiggs

Duplicate file script

5 posts in this topic

#1 ·  Posted (edited)

Hello auto it community.  I wrote a script in order assist me with cleaning up some of my duplicate files.  I host a file server in my home environment, on which...lets just say I'm a "hoarder" of software I think might be useful, and I sometimes mistakenly download the same software I have already downloaded.  So, I wrote this little utility to accept two directories as input, destination (folder you want to merge into) and source (folder you ultimately want removed after all relevant files are moved to destination).  It will then scan both directories, then loop through the source directory to see if a particular file is present in the destination directory (the relevant paths in each directory will have to match as well, which is done on purpose for scenarios like a directory with extracted ISO contents).  If the file is not present, it, along with its relative directory structure, is moved to the destination directory.  If it is present, the tool uses (script downloads and extracts to app data directory at beginning) Microsoft File Checksum Integrity Verifier tool to check and see if the two files are the same (have the same hash value).  If they are the same, the tool will automatically determine which file is the most recent, and will perform whatever action is required to make it so that the destination folder has the most recent version of the file.  If the hash values are not the same, the tool will then prompt user on what they would like done, user inputs answer into console, and action is performed.  Had a lot of fun with this one.  Enjoy!  **Edit  Sorry.   Forgot to mention you would need the attached UDF to compile.  My bad.

#Region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_UseX64=y
#AutoIt3Wrapper_Res_SaveSource=y
#AutoIt3Wrapper_Res_Language=1033
#AutoIt3Wrapper_Res_requestedExecutionLevel=highestAvailable
#EndRegion ;**** Directives created by AutoIt3Wrapper_GUI ****
; *** Start added by AutoIt3Wrapper ***
#include <FileConstants.au3>
; *** End added by AutoIt3Wrapper ***
#cs ----------------------------------------------------------------------------

 AutoIt Version: 3.3.15.0 (Beta)
 Author:         myName

 Script Function:
    Template AutoIt script.

#ce ----------------------------------------------------------------------------

; Script Start - Add your code below here
#include <File.au3>
#include <Array.au3>
#include <Date.au3>
#include <Console.au3>
#include <Constants.au3>
$checksumpath = @AppDataDir & "\checksumverify"
If Not FileExists ( $checksumpath & "\fciv.exe" ) Then
    If Not FileExists ( $checksumpath ) Then
        DirCreate ( $checksumpath )
    EndIf
    $URL = InetGet ( "https://download.microsoft.com/download/c/f/4/cf454ae0-a4bb-4123-8333-a1b6737712f7/Windows-KB841290-x86-ENU.exe", @AppDataDir & "\checksumpackage.exe", 3 )
    RunWait ( @ComSpec & ' /c checksumpackage.exe /Q /T:"' & $checksumpath & '" /C', @AppDataDir, @SW_HIDE )
    InetClose ( $URL )
EndIf

Cout ("")
$a = FileSelectFolder ( "Select destination (the folder you want to merge the files into)", "" )
$b = FileSelectFolder ( "Select source (the folder you want to remove files from and delete)", "" )
$dest = _FileListToArrayRec ( $a, "*", $FLTA_FILESFOLDERS, $FLTAR_RECUR )
$source = _FileListToArrayRec ( $b, "*", $FLTA_FILESFOLDERS, $FLTAR_RECUR )
For $i = 1 To $source[0] Step 1
    $index = _ArraySearch ( $dest, $source[$i] )
    If @error Then
        SetError (0)
        Cout ( "Copying " & $source[$i] & @CRLF )
        If IsFile ( $b & "\" & $source[$i] ) = 0 Then
            DirMove ( $b & "\" & $source[$i], $a & "\" & $source[$i], 1 )
        Else
            FileMove ( $b & "\" & $source[$i], $a & "\" & $source[$i], 9 )
        EndIf
    Else
        If IsFile ( $b & "\" & $source[$i] ) = 0 Then
            ContinueLoop
        Else
            $hash = Run ( @ComSpec & " /c fciv -add " & '"' & $a & "\" & $dest[$index] & '"' & " -sha1", $checksumpath, @SW_HIDE, $STDOUT_CHILD )
            ProcessWaitClose ( $hash )
            $hash2 = Run ( @ComSpec & " /c fciv -add " & '"' & $b & "\" & $source[$i] & '"' & " -sha1", $checksumpath, @SW_HIDE, $STDOUT_CHILD )
            ProcessWaitClose ( $hash2 )
            $hashstring = StringTrimLeft ( StringTrimRight ( StdoutRead ( $hash ), 2 ), 58 )
            $hashstring2 = StringTrimLeft ( StringTrimRight ( StdoutRead ( $hash2 ), 2 ), 58 )
            $firstsplit = StringSplit ( $hashstring, " " )
            $secondsplit = StringSplit ( $hashstring2, " " )
            $num = 0

            If StringCompare ( $firstsplit[1], $secondsplit[1] ) <> 0 Then
                Do
                    Cout ( "The file to be copied from the source (" & $source[$i] & ") has the same file name as a file in the destination, but the hash values are different.  What do you want to do?" & @CRLF & "[R]ename   [D]elete   [M]ove(Replace)" & @CRLF )
                    $user = Getch ()
                Until $user == "r" Or $user == "R" Or $user == "d" Or $user == "D" Or $user == "m" Or $user == "M"
                Cout (@CRLF)
                Select
                    Case $user == "r" Or $user == "R"
                        $holdstring = ""
                        $filepathstring = StringSplit ( $source[$i], "\" )
                        If @error Then
                            SetError (0)
                            $justfile = $source[$i]
                        Else
                            $justfile = $filepathstring[$filepathstring[0]]
                            If $filepathstring[0] = 2 Then
                                $holdstring = $filepathstring[1]
                            ElseIf $filepathstring[0] = 3 Then
                                $holdstring = $filepathstring[1] & "\" & $filepathstring[2]
                            Else
                                For $hh = 1 To $filepathstring[0] - 1 Step 1
                                    $holdstring = $holdstring & $filepathstring[$hh] & "\"
                                Next
                                $holdstring = StringTrimRight ( $holdstring, 1 )
                            EndIf
                        EndIf


                        $rename = StringSplit ( $justfile, "." )
                        $newname = ""
                        $num = 0
                            Do
                                $num += 1
                                If $rename[0] = 2 Then
                                    $newname = $rename[1] & "(" & $num & ")." & $rename[2]
                                ElseIf $rename[0] = 3 Then
                                    $newname = $rename[1] & "." & $rename[2] & "(" & $num & ")." & $rename[3]
                                Else
                                    For $f = 1 To $rename[0] Step 1
                                        If $f = $rename[0] - 1 Then
                                            $newname = $newname & $rename[$f] & "(" & $num & ")."
                                        Else
                                            $newname = $newname & $rename[$f] & "."
                                        EndIf
                                    Next
                                    $newname = StringTrimRight ( $newname, 1 )
                                EndIf
                                $foldhold = ""
                                If $holdstring == "" Then
                                    $foldhold = $newname
                                Else
                                    $foldhold = $holdstring & "\" & $newname
                                EndIf


                            Until Not FileExists ( $a & "\" & $foldhold )
                            Cout ( "Copying " & $foldhold & @CRLF )
                            Do
                                FileMove ( $b & "\" & $source[$i], $a & "\" & $foldhold, 9 )
                            Until Not FileExists ( $b & "\" & $source[$i] ) And FileExists ( $a & "\" & $foldhold )


                    Case $user == "d" Or $user == "D"
                        Cout ( "Deleting " & $source[$i] & @CRLF )
                        Do
                            FileDelete ( $b & "\" & $source[$i] )
                        Until Not FileExists ( $b & "\" & $source[$i] )
                    Case $user == "m" Or $user == "M"
                        Cout ( "Copying " & $source[$i] & @CRLF )
                        Do
                            FileMove ( $b & "\" & $source[$i], $a & "\" & $source[$i], 9 )
                        Until Not FileExists ( $b & "\" & $source[$i] )
                EndSelect
            Else
                $desttime = _Date_Time_GetFileTime ( $a & "\" & $dest[$index] )
                $sourcetime = _Date_Time_GetFileTime ( $b & "\" & $source[$i] )
                $compare = _Date_Time_CompareFileTime ( $desttime[2], $sourcetime[2] )
                If $compare = 0 Or $compare = 1 Then
                    Cout ( "Deleting " & $source[$i] & @CRLF )
                    Do
                        FileDelete ( $b & "\" & $source[$i] )
                    Until Not FileExists ( $b & "\" & $source[$i] )
                ElseIf $compare = -1 Then
                    Cout ( "Copying " & $source[$i] & @CRLF )
                    Do
                        FileMove ( $b & "\" & $source[$i], $a & "\" & $source[$i], 9 )
                    Until Not FileExists ( $b & "\" & $source[$i] )
                Else
                    Cout ( "????????????????????" & @CRLF )
                EndIf
            EndIf

        EndIf
    EndIf
Next
DirRemove ( $b, $DIR_REMOVE )


Func IsFile($sFilePath)
    Return Number(FileExists($sFilePath) And StringInStr(FileGetAttrib($sFilePath), "D", 2, 1) = 0)
EndFunc   ;==>IsFile

 

Console.au3

Edited by MattHiggs

Share this post


Link to post
Share on other sites



Interesting util. This may answer a problem that I have been working. Thanks for posting. Cheers.

Share this post


Link to post
Share on other sites
21 hours ago, stealthmsgr said:

Interesting util. This may answer a problem that I have been working. Thanks for posting. Cheers.

No problem man.  Its the small things (word of thanks) that make it all worth it and then some.

Share this post


Link to post
Share on other sites

Looks pretty good.  Just curious, is there a reason you opted for the MS tool instead of the hashing functions in the Crypt UDF?  
 

Share this post


Link to post
Share on other sites

Not really.  I had previous experience with the ms tool, so it was as simple as copying and pasting code, thats all.  The path of least resistance.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now