Find and delete duplicate file names

[StevenHook]

Hi,

I've searched around a bit and I can't seem to find any samples of what I'm trying to achieve.

I need to create a script (as a patch to distribute to customers who experience problems) that searched a pre-defined folder structure (ie. c:\mycontent\cliparts\) and all it's subfolders for duplicate file names, and delete one of the files.

Any pointers would be greatly appreciated.

Thanks

Steven

[Xenobiologist]

expandcollapse popup

#include <Array.au3>

Global $StartPfad = 'c:\Downloads\'
Global $FileTyp = '*'
$aFiles = _GetFilesFolder_Rekursiv($StartPfad, $FileTyp, 0)

_ArrayDisplay($aFiles)

;==================================================================================================
; Function Name: _GetFilesFolder_Rekursiv($sPath [, $sExt='*' [, $iDir=-1 [, $iRetType=0 ,[$sDelim='0']]]])
; Description:  Rekursive Auflistung von Dateien und/oder Ordnern
; Parameter(s): $sPath  der Basispfad für die Auflistung ('.' -aktueller Pfad, '..' -Parentpfad)
;   $sExt   Erweiterung für Dateiauswahl '*' oder -1 für alle (Standard)
;   $iDir   -1 Dateien+Ordner(Standard), 0 nur Dateien, 1 nur Ordner
;   optional: $iRetType 0 gibt Array, 1 gibt String zurück
;   optional: $sDelim   legt Trennzeichen für Stringrückgabe fest
;   0 -@CRLF (Standard); 1 -@CR; 2 -@LF; oder beliebiges Zeichen
; Return Value(s): Array (Standard) od. String mit den gefundenen Pfaden der Dateien und/oder Ordner
;   Array[0] enthält die Anzahl der gefundenen Dateien/Ordner
; Author(s):    BugFix (bugfix@autoit.de)
;==================================================================================================
Func _GetFilesFolder_Rekursiv($sPath, $sExt = '*', $iDir = -1, $iRetType = 0, $sDelim = '0')
    Global $oFSO = ObjCreate('Scripting.FileSystemObject')
    Global $strFiles = ''
    Switch $sDelim
        Case '1'
            $sDelim = @CR
        Case '2'
            $sDelim = @LF
        Case Else
            $sDelim = @CRLF
    EndSwitch
    If ($iRetType < 0) Or ($iRetType > 1) Then $iRetType = 0
    If $sExt = -1 Then $sExt = '*'
    If ($iDir < -1) Or ($iDir > 1) Then $iDir = -1
    _ShowSubFolders($oFSO.GetFolder($sPath), $sExt, $iDir, $sDelim)
    If $iRetType = 0 Then
        Local $aOut
;~      $aOut = StringSplit(StringTrimRight($strFiles, 1), $sDelim, 1)
        $aOut = StringSplit(StringStripWS($strFiles, 2), $sDelim, 1)
        If $aOut[1] = '' Then
            ReDim $aOut[1]
            $aOut[0] = 0
        EndIf
        Return $aOut
    Else
;~      Return StringTrimRight($strFiles, 1)
        Return StringStripWS($strFiles, 2)
    EndIf
EndFunc ;==>_GetFilesFolder_Rekursiv

Func _ShowSubFolders($Folder, $Ext = '*', $Dir = -1, $Delim = @CRLF)
    If Not IsDeclared("strFiles") Then Global $strFiles = ''
    If ($Dir = -1) Or ($Dir = 0) Then
        For $file In $Folder.Files
            If $Ext <> '*' Then
                If StringRight($file.Name, StringLen($Ext)) = $Ext Then _
                        $strFiles &= $file.Path & $Delim
            Else
                $strFiles &= $file.Path & $Delim
            EndIf
        Next
    EndIf
    For $Subfolder In $Folder.SubFolders
        If ($Dir = -1) Or ($Dir = 1) Then $strFiles &= $Subfolder.Path & '\' & $Delim
        _ShowSubFolders($Subfolder, $Ext, $Dir, $Delim)
    Next
EndFunc ;==>_ShowSubFolders

[Spiff59]

Another offering (a bit more than a "pointer" I suppose):

expandcollapse popup

#include <array.au3>
#include <file.au3>
$Target_Folder = "C:\Test"

$Array = _FileListToArray_Recursive($Target_Folder, "", "*", "", 1, 2, True)
_ArrayDisplay($Array) ; for testing

; expand to 2-dimension array with filename, fullpath
Dim $Array2[$Array[0] + 1][2]
$Array2[0][0] = $Array[0]
For $x = 1 to $Array[0]
    $Array2[$x][0] = StringTrimLeft($Array[$x],StringInStr($Array[$x], "\", 0, -1))
    $Array2[$x][1] = $Array[$x]
Next
_ArrayDisplay($Array2) ; for testing

; sort by filename
_ArraySort($Array2, 0, 1)
_ArrayDisplay($Array2) ; for testing

; delete duplicates
$x = 2
While $x <= $Array2[0][0]
    If $Array2[$x][0] = $Array2[$x -1][0] Then
;       If FileDelete($Array2[$x][1]) Then
            ToolTip("Deleting file: " & $Array2[$x][1]) ; for testing
            _ArrayDelete($Array2, $x)
            $Array2[0][0] -= 1
;       Else
;           MsgBox(1,"", "Unable to delete file: " & $Array2[$x][1]
;           ExitLoop
;       EndIf
    Else
        $x += 1
    EndIf
WEnd
_ArrayDisplay($Array2) ; for testing
Exit

;===============================================================================
; Recursive version of _FileListToArray() from thread #96952
Func _FileListToArray_Recursive($sPath, $sExcludeFolderList = "", $sIncludeList = "*", $sExcludeList = "", $iReturnType = 0, $iReturnFormat = 0, $bRecursive = False)
    Local $sRet = "", $sReturnFormat = ""

    ; Edit include path (strip trailing slashes, and append single slash)
    $sPath = StringRegExpReplace($sPath, "[\\/]+\z", "") & "\"
    If Not FileExists($sPath) Then Return SetError(1, 1, "")

    ; Edit exclude folders list
    If $sExcludeFolderList Then
        ; Strip leading/trailing spaces and semi-colons, any adjacent semi-colons, and spaces surrounding semi-colons
        $sExcludeFolderList = StringRegExpReplace(StringRegExpReplace($sExcludeFolderList, "(\s*;\s*)+", ";"), "\A;|;\z", "")
        ; Convert to Regular Expression, step 1: Wrap brackets around . and $ (what other characters needed?)
        $sExcludeFolderList = StringRegExpReplace($sExcludeFolderList, '[.$]', '\[\0\]')
        ; Convert to Regular Expression, step 2: Convert '?' to '.', and '*' to '.*?'
        $sExcludeFolderList = StringReplace(StringReplace($sExcludeFolderList, "?", "."), "*", ".*?")
        ; Convert to Regular Expression, step 3; case-insensitive, convert ';' to '|', match from first char, terminate strings
        $sExcludeFolderList = "(?i)\A(?!" & StringReplace($sExcludeFolderList, ";", "$|")  & "$)"
    EndIf

    ; Edit include files list
    If $sIncludeList ="*" Then
        $sIncludeList = ""
    Else
        If StringRegExp($sIncludeList, "[\\/ :> <\|]|(?s)\A\s*\z") Then Return SetError(2, 2, "")
        ; Strip leading/trailing spaces and semi-colons, any adjacent semi-colons, and spaces surrounding semi-colons
        $sIncludeList = StringRegExpReplace(StringRegExpReplace($sIncludeList, "(\s*;\s*)+", ";"), "\A;|;\z", "")
        ; Convert to Regular Expression, step 1: Wrap brackets around . and $ (what other characters needed?)
        $sIncludeList = StringRegExpReplace($sIncludeList, '[.$]', '\[\0\]')
        ; Convert to Regular Expression, step 2: Convert '?' to '.', and '*' to '.*?'
        $sIncludeList = StringReplace(StringReplace($sIncludeList, "?", "."), "*", ".*?")
        ; Convert to Regular Expression, step 3; case-insensitive, convert ';' to '|', match from first char, terminate strings
        $sIncludeList = "(?i)\A(" & StringReplace($sIncludeList, ";", "$|")  & "$)"
    EndIf

    ; Edit exclude files list
    If $sExcludeList Then
        ; Strip leading/trailing spaces and semi-colons, any adjacent semi-colons, and spaces surrounding semi-colons
        $sExcludeList = StringRegExpReplace(StringRegExpReplace($sExcludeList, "(\s*;\s*)+", ";"), "\A;|;\z", "")
        ; Convert to Regular Expression, step 1: Wrap brackets around . and $ (what other characters needed?)
        $sExcludeList = StringRegExpReplace($sExcludeList, '[.$]', '\[\0\]')
        ; Convert to Regular Expression, step 2: Convert '?' to '.', and '*' to '.*?'
        $sExcludeList = StringReplace(StringReplace($sExcludeList, "?", "."), "*", ".*?")
        ; Convert to Regular Expression, step 3; case-insensitive, convert ';' to '|', match from first char, terminate strings
        $sExcludeList = "(?i)\A(?!" & StringReplace($sExcludeList, ";", "$|")  & "$)"
    EndIf

;   MsgBox(1,"Masks","File include: " & $sIncludeList & @CRLF & "File exclude: " & $ExcludeList & @CRLF _
;           & "Dir include : " & $FolderInclude & @CRLF & "Dir exclude : " & $ExcludeFolderList)

    If Not ($iReturnType = 0 Or $iReturnType = 1 Or $iReturnType = 2) Then Return SetError(3, 3, "")

    Local $sOrigPathLen = StringLen($sPath), $aQueue[64] = [1,$sPath], $iQMax = 63
    While $aQueue[0]
        $WorkFolder = $aQueue[$aQueue[0]]
        $aQueue[0] -= 1
        $search = FileFindFirstFile($WorkFolder & "*")
        If @error Then ContinueLoop
        Switch $iReturnFormat
            Case 1 ; relative path
                $sReturnFormat = StringTrimLeft($WorkFolder, $sOrigPathLen)
            Case 2 ; full path
                $sReturnFormat = $WorkFolder
        EndSwitch
        While 1
            $file = FileFindNextFile($search)
            If @error Then ExitLoop
            If @extended Then ; Folder
                If $sExcludeFolderList And Not StringRegExp($file, $sExcludeFolderList) Then ContinueLoop
                If $bRecursive Then
                    If $aQueue[0] = $iQMax Then
                        $iQMax += 128
                        ReDim $aQueue[$iQMax + 1]
                    EndIf
                    $aQueue[0] += 1
                    $aQueue[$aQueue[0]] = $WorkFolder & $file & "\"
                EndIf
                If $iReturnType = 1 Then ContinueLoop
            Else ; File
                If $iReturnType = 2 Then ContinueLoop
            EndIf
            If $sIncludeList And Not StringRegExp($file, $sIncludeList) Then ContinueLoop
            If $sExcludeList And Not StringRegExp($file, $sExcludeList) Then ContinueLoop
            $sRet &= $sReturnFormat & $file & "|"
        WEnd
        FileClose($search)
    WEnd
    If Not $sRet Then Return SetError(4, 4, "")
    Return StringSplit(StringTrimRight($sRet, 1), "|")
EndFunc

The dangerous parts (filedelete) are presently commented-out.

Edited June 22, 2010 by Spiff59