Sign in to follow this  
Followers 0
StevenHook

Find and delete duplicate file names

3 posts in this topic

Hi,

I've searched around a bit and I can't seem to find any samples of what I'm trying to achieve.

I need to create a script (as a patch to distribute to customers who experience problems) that searched a pre-defined folder structure (ie. c:\mycontent\cliparts\) and all it's subfolders for duplicate file names, and delete one of the files.

Any pointers would be greatly appreciated.

Thanks

Steven

Share this post


Link to post
Share on other sites



#include <Array.au3>

Global $StartPfad = 'c:\Downloads\'
Global $FileTyp = '*'
$aFiles = _GetFilesFolder_Rekursiv($StartPfad, $FileTyp, 0)

_ArrayDisplay($aFiles)

;==================================================================================================
; Function Name: _GetFilesFolder_Rekursiv($sPath [, $sExt='*' [, $iDir=-1 [, $iRetType=0 ,[$sDelim='0']]]])
; Description:  Rekursive Auflistung von Dateien und/oder Ordnern
; Parameter(s): $sPath  der Basispfad für die Auflistung ('.' -aktueller Pfad, '..' -Parentpfad)
;   $sExt   Erweiterung für Dateiauswahl '*' oder -1 für alle (Standard)
;   $iDir   -1 Dateien+Ordner(Standard), 0 nur Dateien, 1 nur Ordner
;   optional: $iRetType 0 gibt Array, 1 gibt String zurück
;   optional: $sDelim   legt Trennzeichen für Stringrückgabe fest
;   0 -@CRLF (Standard); 1 -@CR; 2 -@LF; oder beliebiges Zeichen
; Return Value(s): Array (Standard) od. String mit den gefundenen Pfaden der Dateien und/oder Ordner
;   Array[0] enthält die Anzahl der gefundenen Dateien/Ordner
; Author(s):    BugFix (bugfix@autoit.de)
;==================================================================================================
Func _GetFilesFolder_Rekursiv($sPath, $sExt = '*', $iDir = -1, $iRetType = 0, $sDelim = '0')
    Global $oFSO = ObjCreate('Scripting.FileSystemObject')
    Global $strFiles = ''
    Switch $sDelim
        Case '1'
            $sDelim = @CR
        Case '2'
            $sDelim = @LF
        Case Else
            $sDelim = @CRLF
    EndSwitch
    If ($iRetType < 0) Or ($iRetType > 1) Then $iRetType = 0
    If $sExt = -1 Then $sExt = '*'
    If ($iDir < -1) Or ($iDir > 1) Then $iDir = -1
    _ShowSubFolders($oFSO.GetFolder($sPath), $sExt, $iDir, $sDelim)
    If $iRetType = 0 Then
        Local $aOut
;~      $aOut = StringSplit(StringTrimRight($strFiles, 1), $sDelim, 1)
        $aOut = StringSplit(StringStripWS($strFiles, 2), $sDelim, 1)
        If $aOut[1] = '' Then
            ReDim $aOut[1]
            $aOut[0] = 0
        EndIf
        Return $aOut
    Else
;~      Return StringTrimRight($strFiles, 1)
        Return StringStripWS($strFiles, 2)
    EndIf
EndFunc ;==>_GetFilesFolder_Rekursiv

Func _ShowSubFolders($Folder, $Ext = '*', $Dir = -1, $Delim = @CRLF)
    If Not IsDeclared("strFiles") Then Global $strFiles = ''
    If ($Dir = -1) Or ($Dir = 0) Then
        For $file In $Folder.Files
            If $Ext <> '*' Then
                If StringRight($file.Name, StringLen($Ext)) = $Ext Then _
                        $strFiles &= $file.Path & $Delim
            Else
                $strFiles &= $file.Path & $Delim
            EndIf
        Next
    EndIf
    For $Subfolder In $Folder.SubFolders
        If ($Dir = -1) Or ($Dir = 1) Then $strFiles &= $Subfolder.Path & '\' & $Delim
        _ShowSubFolders($Subfolder, $Ext, $Dir, $Delim)
    Next
EndFunc ;==>_ShowSubFolders


Scripts & functions Organize Includes Let Scite organize the include files

Yahtzee The game "Yahtzee" (Kniffel, DiceLion)

LoginWrapper Secure scripts by adding a query (authentication)

_RunOnlyOnThis UDF Make sure that a script can only be executed on ... (Windows / HD / ...)

Internet-Café Server/Client Application Open CD, Start Browser, Lock remote client, etc.

MultipleFuncsWithOneHotkey Start different funcs by hitting one hotkey different times

Share this post


Link to post
Share on other sites

#3 ·  Posted (edited)

Another offering (a bit more than a "pointer" I suppose):

#include <array.au3>
#include <file.au3>
$Target_Folder = "C:\Test"

$Array = _FileListToArray_Recursive($Target_Folder, "", "*", "", 1, 2, True)
_ArrayDisplay($Array) ; for testing

; expand to 2-dimension array with filename, fullpath
Dim $Array2[$Array[0] + 1][2]
$Array2[0][0] = $Array[0]
For $x = 1 to $Array[0]
    $Array2[$x][0] = StringTrimLeft($Array[$x],StringInStr($Array[$x], "\", 0, -1))
    $Array2[$x][1] = $Array[$x]
Next
_ArrayDisplay($Array2) ; for testing

; sort by filename
_ArraySort($Array2, 0, 1)
_ArrayDisplay($Array2) ; for testing

; delete duplicates
$x = 2
While $x <= $Array2[0][0]
    If $Array2[$x][0] = $Array2[$x -1][0] Then
;       If FileDelete($Array2[$x][1]) Then
            ToolTip("Deleting file: " & $Array2[$x][1]) ; for testing
            _ArrayDelete($Array2, $x)
            $Array2[0][0] -= 1
;       Else
;           MsgBox(1,"", "Unable to delete file: " & $Array2[$x][1]
;           ExitLoop
;       EndIf
    Else
        $x += 1
    EndIf
WEnd
_ArrayDisplay($Array2) ; for testing
Exit

;===============================================================================
; Recursive version of _FileListToArray() from thread #96952
Func _FileListToArray_Recursive($sPath, $sExcludeFolderList = "", $sIncludeList = "*", $sExcludeList = "", $iReturnType = 0, $iReturnFormat = 0, $bRecursive = False)
    Local $sRet = "", $sReturnFormat = ""

    ; Edit include path (strip trailing slashes, and append single slash)
    $sPath = StringRegExpReplace($sPath, "[\\/]+\z", "") & "\"
    If Not FileExists($sPath) Then Return SetError(1, 1, "")

    ; Edit exclude folders list
    If $sExcludeFolderList Then
        ; Strip leading/trailing spaces and semi-colons, any adjacent semi-colons, and spaces surrounding semi-colons
        $sExcludeFolderList = StringRegExpReplace(StringRegExpReplace($sExcludeFolderList, "(\s*;\s*)+", ";"), "\A;|;\z", "")
        ; Convert to Regular Expression, step 1: Wrap brackets around . and $ (what other characters needed?)
        $sExcludeFolderList = StringRegExpReplace($sExcludeFolderList, '[.$]', '\[\0\]')
        ; Convert to Regular Expression, step 2: Convert '?' to '.', and '*' to '.*?'
        $sExcludeFolderList = StringReplace(StringReplace($sExcludeFolderList, "?", "."), "*", ".*?")
        ; Convert to Regular Expression, step 3; case-insensitive, convert ';' to '|', match from first char, terminate strings
        $sExcludeFolderList = "(?i)\A(?!" & StringReplace($sExcludeFolderList, ";", "$|")  & "$)"
    EndIf

    ; Edit include files list
    If $sIncludeList ="*" Then
        $sIncludeList = ""
    Else
        If StringRegExp($sIncludeList, "[\\/ :> <\|]|(?s)\A\s*\z") Then Return SetError(2, 2, "")
        ; Strip leading/trailing spaces and semi-colons, any adjacent semi-colons, and spaces surrounding semi-colons
        $sIncludeList = StringRegExpReplace(StringRegExpReplace($sIncludeList, "(\s*;\s*)+", ";"), "\A;|;\z", "")
        ; Convert to Regular Expression, step 1: Wrap brackets around . and $ (what other characters needed?)
        $sIncludeList = StringRegExpReplace($sIncludeList, '[.$]', '\[\0\]')
        ; Convert to Regular Expression, step 2: Convert '?' to '.', and '*' to '.*?'
        $sIncludeList = StringReplace(StringReplace($sIncludeList, "?", "."), "*", ".*?")
        ; Convert to Regular Expression, step 3; case-insensitive, convert ';' to '|', match from first char, terminate strings
        $sIncludeList = "(?i)\A(" & StringReplace($sIncludeList, ";", "$|")  & "$)"
    EndIf

    ; Edit exclude files list
    If $sExcludeList Then
        ; Strip leading/trailing spaces and semi-colons, any adjacent semi-colons, and spaces surrounding semi-colons
        $sExcludeList = StringRegExpReplace(StringRegExpReplace($sExcludeList, "(\s*;\s*)+", ";"), "\A;|;\z", "")
        ; Convert to Regular Expression, step 1: Wrap brackets around . and $ (what other characters needed?)
        $sExcludeList = StringRegExpReplace($sExcludeList, '[.$]', '\[\0\]')
        ; Convert to Regular Expression, step 2: Convert '?' to '.', and '*' to '.*?'
        $sExcludeList = StringReplace(StringReplace($sExcludeList, "?", "."), "*", ".*?")
        ; Convert to Regular Expression, step 3; case-insensitive, convert ';' to '|', match from first char, terminate strings
        $sExcludeList = "(?i)\A(?!" & StringReplace($sExcludeList, ";", "$|")  & "$)"
    EndIf

;   MsgBox(1,"Masks","File include: " & $sIncludeList & @CRLF & "File exclude: " & $ExcludeList & @CRLF _
;           & "Dir include : " & $FolderInclude & @CRLF & "Dir exclude : " & $ExcludeFolderList)

    If Not ($iReturnType = 0 Or $iReturnType = 1 Or $iReturnType = 2) Then Return SetError(3, 3, "")

    Local $sOrigPathLen = StringLen($sPath), $aQueue[64] = [1,$sPath], $iQMax = 63
    While $aQueue[0]
        $WorkFolder = $aQueue[$aQueue[0]]
        $aQueue[0] -= 1
        $search = FileFindFirstFile($WorkFolder & "*")
        If @error Then ContinueLoop
        Switch $iReturnFormat
            Case 1 ; relative path
                $sReturnFormat = StringTrimLeft($WorkFolder, $sOrigPathLen)
            Case 2 ; full path
                $sReturnFormat = $WorkFolder
        EndSwitch
        While 1
            $file = FileFindNextFile($search)
            If @error Then ExitLoop
            If @extended Then ; Folder
                If $sExcludeFolderList And Not StringRegExp($file, $sExcludeFolderList) Then ContinueLoop
                If $bRecursive Then
                    If $aQueue[0] = $iQMax Then
                        $iQMax += 128
                        ReDim $aQueue[$iQMax + 1]
                    EndIf
                    $aQueue[0] += 1
                    $aQueue[$aQueue[0]] = $WorkFolder & $file & "\"
                EndIf
                If $iReturnType = 1 Then ContinueLoop
            Else ; File
                If $iReturnType = 2 Then ContinueLoop
            EndIf
            If $sIncludeList And Not StringRegExp($file, $sIncludeList) Then ContinueLoop
            If $sExcludeList And Not StringRegExp($file, $sExcludeList) Then ContinueLoop
            $sRet &= $sReturnFormat & $file & "|"
        WEnd
        FileClose($search)
    WEnd
    If Not $sRet Then Return SetError(4, 4, "")
    Return StringSplit(StringTrimRight($sRet, 1), "|")
EndFunc

The dangerous parts (filedelete) are presently commented-out.

Edited by Spiff59

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0