Find and delete duplicate file names

StevenHook · June 22, 2010

Hi,

I've searched around a bit and I can't seem to find any samples of what I'm trying to achieve.

I need to create a script (as a patch to distribute to customers who experience problems) that searched a pre-defined folder structure (ie. c:\mycontent\cliparts\) and all it's subfolders for duplicate file names, and delete one of the files.

Any pointers would be greatly appreciated.

Thanks

Steven

Xenobiologist · June 22, 2010

#include <Array.au3>

Global $StartPfad = 'c:\Downloads\'
Global $FileTyp = '*'
$aFiles = _GetFilesFolder_Rekursiv($StartPfad, $FileTyp, 0)

_ArrayDisplay($aFiles)

;==================================================================================================
; Function Name: _GetFilesFolder_Rekursiv($sPath [, $sExt='*' [, $iDir=-1 [, $iRetType=0 ,[$sDelim='0']]]])
; Description:  Rekursive Auflistung von Dateien und/oder Ordnern
; Parameter(s): $sPath  der Basispfad für die Auflistung ('.' -aktueller Pfad, '..' -Parentpfad)
;   $sExt   Erweiterung für Dateiauswahl '*' oder -1 für alle (Standard)
;   $iDir   -1 Dateien+Ordner(Standard), 0 nur Dateien, 1 nur Ordner
;   optional: $iRetType 0 gibt Array, 1 gibt String zurück
;   optional: $sDelim   legt Trennzeichen für Stringrückgabe fest
;   0 -@CRLF (Standard); 1 -@CR; 2 -@LF; oder beliebiges Zeichen
; Return Value(s): Array (Standard) od. String mit den gefundenen Pfaden der Dateien und/oder Ordner
;   Array[0] enthält die Anzahl der gefundenen Dateien/Ordner
; Author(s):    BugFix (bugfix@autoit.de)
;==================================================================================================
Func _GetFilesFolder_Rekursiv($sPath, $sExt = '*', $iDir = -1, $iRetType = 0, $sDelim = '0')
    Global $oFSO = ObjCreate('Scripting.FileSystemObject')
    Global $strFiles = ''
    Switch $sDelim
        Case '1'
            $sDelim = @CR
        Case '2'
            $sDelim = @LF
        Case Else
            $sDelim = @CRLF
    EndSwitch
    If ($iRetType < 0) Or ($iRetType > 1) Then $iRetType = 0
    If $sExt = -1 Then $sExt = '*'
    If ($iDir < -1) Or ($iDir > 1) Then $iDir = -1
    _ShowSubFolders($oFSO.GetFolder($sPath), $sExt, $iDir, $sDelim)
    If $iRetType = 0 Then
        Local $aOut
;~      $aOut = StringSplit(StringTrimRight($strFiles, 1), $sDelim, 1)
        $aOut = StringSplit(StringStripWS($strFiles, 2), $sDelim, 1)
        If $aOut[1] = '' Then
            ReDim $aOut[1]
            $aOut[0] = 0
        EndIf
        Return $aOut
    Else
;~      Return StringTrimRight($strFiles, 1)
        Return StringStripWS($strFiles, 2)
    EndIf
EndFunc ;==>_GetFilesFolder_Rekursiv

Func _ShowSubFolders($Folder, $Ext = '*', $Dir = -1, $Delim = @CRLF)
    If Not IsDeclared("strFiles") Then Global $strFiles = ''
    If ($Dir = -1) Or ($Dir = 0) Then
        For $file In $Folder.Files
            If $Ext <> '*' Then
                If StringRight($file.Name, StringLen($Ext)) = $Ext Then _
                        $strFiles &= $file.Path & $Delim
            Else
                $strFiles &= $file.Path & $Delim
            EndIf
        Next
    EndIf
    For $Subfolder In $Folder.SubFolders
        If ($Dir = -1) Or ($Dir = 1) Then $strFiles &= $Subfolder.Path & '\' & $Delim
        _ShowSubFolders($Subfolder, $Ext, $Dir, $Delim)
    Next
EndFunc ;==>_ShowSubFolders

Spiff59 · June 22, 2010

Another offering (a bit more than a "pointer" I suppose):

#include <array.au3>
#include <file.au3>
$Target_Folder = "C:\Test"

$Array = _FileListToArray_Recursive($Target_Folder, "", "*", "", 1, 2, True)
_ArrayDisplay($Array) ; for testing

; expand to 2-dimension array with filename, fullpath
Dim $Array2[$Array[0] + 1][2]
$Array2[0][0] = $Array[0]
For $x = 1 to $Array[0]
    $Array2[$x][0] = StringTrimLeft($Array[$x],StringInStr($Array[$x], "\", 0, -1))
    $Array2[$x][1] = $Array[$x]
Next
_ArrayDisplay($Array2) ; for testing

; sort by filename
_ArraySort($Array2, 0, 1)
_ArrayDisplay($Array2) ; for testing

; delete duplicates
$x = 2
While $x <= $Array2[0][0]
    If $Array2[$x][0] = $Array2[$x -1][0] Then
;       If FileDelete($Array2[$x][1]) Then
            ToolTip("Deleting file: " & $Array2[$x][1]) ; for testing
            _ArrayDelete($Array2, $x)
            $Array2[0][0] -= 1
;       Else
;           MsgBox(1,"", "Unable to delete file: " & $Array2[$x][1]
;           ExitLoop
;       EndIf
    Else
        $x += 1
    EndIf
WEnd
_ArrayDisplay($Array2) ; for testing
Exit

;===============================================================================
; Recursive version of _FileListToArray() from thread #96952
Func _FileListToArray_Recursive($sPath, $sExcludeFolderList = "", $sIncludeList = "*", $sExcludeList = "", $iReturnType = 0, $iReturnFormat = 0, $bRecursive = False)
    Local $sRet = "", $sReturnFormat = ""

    ; Edit include path (strip trailing slashes, and append single slash)
    $sPath = StringRegExpReplace($sPath, "[\\/]+\z", "") & "\"
    If Not FileExists($sPath) Then Return SetError(1, 1, "")

    ; Edit exclude folders list
    If $sExcludeFolderList Then
        ; Strip leading/trailing spaces and semi-colons, any adjacent semi-colons, and spaces surrounding semi-colons
        $sExcludeFolderList = StringRegExpReplace(StringRegExpReplace($sExcludeFolderList, "(\s*;\s*)+", ";"), "\A;|;\z", "")
        ; Convert to Regular Expression, step 1: Wrap brackets around . and $ (what other characters needed?)
        $sExcludeFolderList = StringRegExpReplace($sExcludeFolderList, '[.$]', '\[\0\]')
        ; Convert to Regular Expression, step 2: Convert '?' to '.', and '*' to '.*?'
        $sExcludeFolderList = StringReplace(StringReplace($sExcludeFolderList, "?", "."), "*", ".*?")
        ; Convert to Regular Expression, step 3; case-insensitive, convert ';' to '|', match from first char, terminate strings
        $sExcludeFolderList = "(?i)\A(?!" & StringReplace($sExcludeFolderList, ";", "$|")  & "$)"
    EndIf

    ; Edit include files list
    If $sIncludeList ="*" Then
        $sIncludeList = ""
    Else
        If StringRegExp($sIncludeList, "[\\/ :> <\|]|(?s)\A\s*\z") Then Return SetError(2, 2, "")
        ; Strip leading/trailing spaces and semi-colons, any adjacent semi-colons, and spaces surrounding semi-colons
        $sIncludeList = StringRegExpReplace(StringRegExpReplace($sIncludeList, "(\s*;\s*)+", ";"), "\A;|;\z", "")
        ; Convert to Regular Expression, step 1: Wrap brackets around . and $ (what other characters needed?)
        $sIncludeList = StringRegExpReplace($sIncludeList, '[.$]', '\[\0\]')
        ; Convert to Regular Expression, step 2: Convert '?' to '.', and '*' to '.*?'
        $sIncludeList = StringReplace(StringReplace($sIncludeList, "?", "."), "*", ".*?")
        ; Convert to Regular Expression, step 3; case-insensitive, convert ';' to '|', match from first char, terminate strings
        $sIncludeList = "(?i)\A(" & StringReplace($sIncludeList, ";", "$|")  & "$)"
    EndIf

    ; Edit exclude files list
    If $sExcludeList Then
        ; Strip leading/trailing spaces and semi-colons, any adjacent semi-colons, and spaces surrounding semi-colons
        $sExcludeList = StringRegExpReplace(StringRegExpReplace($sExcludeList, "(\s*;\s*)+", ";"), "\A;|;\z", "")
        ; Convert to Regular Expression, step 1: Wrap brackets around . and $ (what other characters needed?)
        $sExcludeList = StringRegExpReplace($sExcludeList, '[.$]', '\[\0\]')
        ; Convert to Regular Expression, step 2: Convert '?' to '.', and '*' to '.*?'
        $sExcludeList = StringReplace(StringReplace($sExcludeList, "?", "."), "*", ".*?")
        ; Convert to Regular Expression, step 3; case-insensitive, convert ';' to '|', match from first char, terminate strings
        $sExcludeList = "(?i)\A(?!" & StringReplace($sExcludeList, ";", "$|")  & "$)"
    EndIf

;   MsgBox(1,"Masks","File include: " & $sIncludeList & @CRLF & "File exclude: " & $ExcludeList & @CRLF _
;           & "Dir include : " & $FolderInclude & @CRLF & "Dir exclude : " & $ExcludeFolderList)

    If Not ($iReturnType = 0 Or $iReturnType = 1 Or $iReturnType = 2) Then Return SetError(3, 3, "")

    Local $sOrigPathLen = StringLen($sPath), $aQueue[64] = [1,$sPath], $iQMax = 63
    While $aQueue[0]
        $WorkFolder = $aQueue[$aQueue[0]]
        $aQueue[0] -= 1
        $search = FileFindFirstFile($WorkFolder & "*")
        If @error Then ContinueLoop
        Switch $iReturnFormat
            Case 1 ; relative path
                $sReturnFormat = StringTrimLeft($WorkFolder, $sOrigPathLen)
            Case 2 ; full path
                $sReturnFormat = $WorkFolder
        EndSwitch
        While 1
            $file = FileFindNextFile($search)
            If @error Then ExitLoop
            If @extended Then ; Folder
                If $sExcludeFolderList And Not StringRegExp($file, $sExcludeFolderList) Then ContinueLoop
                If $bRecursive Then
                    If $aQueue[0] = $iQMax Then
                        $iQMax += 128
                        ReDim $aQueue[$iQMax + 1]
                    EndIf
                    $aQueue[0] += 1
                    $aQueue[$aQueue[0]] = $WorkFolder & $file & "\"
                EndIf
                If $iReturnType = 1 Then ContinueLoop
            Else ; File
                If $iReturnType = 2 Then ContinueLoop
            EndIf
            If $sIncludeList And Not StringRegExp($file, $sIncludeList) Then ContinueLoop
            If $sExcludeList And Not StringRegExp($file, $sExcludeList) Then ContinueLoop
            $sRet &= $sReturnFormat & $file & "|"
        WEnd
        FileClose($search)
    WEnd
    If Not $sRet Then Return SetError(4, 4, "")
    Return StringSplit(StringTrimRight($sRet, 1), "|")
EndFunc

The dangerous parts (filedelete) are presently commented-out.

Edited June 22, 2010 by Spiff59

Sign In

Find and delete duplicate file names

Recommended Posts

StevenHook

Xenobiologist

Spiff59

Create an account or sign in to comment

Create an account

Sign in

Recently Browsing 0 members

Browse

AutoIt Resources

Release

Beta