_LargeFileCopy UDF

[wraithdu]

Hmm. Try to create a small reproducer script with all code req'd and I'll take a look. Maybe you just need a second pair of eyes on it. I've looked through all the path manipulation code in my UDF and tested the regular expressions, and nothing is mangling UNC style paths.

[llewxam]

OK, good idea. I'll work on that tomorrow night, and who knows, maybe that will show me the way to correcting it....

Thanks

Ian

[llewxam]

Thought I should report back. I did start small and had good results, then today I folded all of the tweaks in to my Sync tool and everything was working great - until I tried to exceed the normal 260 character limit using UNC paths - those transfers failed.

Here is a stripped-down tester for you:

expandcollapse popup

#region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_UseUpx=n
#AutoIt3Wrapper_Run_Tidy=y
#endregion ;**** Directives created by AutoIt3Wrapper_GUI ****

#include <WinAPI.au3>
#include <Array.au3>

$iToRead = 8 * 1024 * 1024

$Source = "c:testunc"
$Target = "c:testunc2"

If StringRight($Source, 1) <> "" Then $Source &= ""
If StringRight($Target, 1) <> "" Then $Target &= ""
$SourceLength = StringLen($Source)
If $SourceLength == 3 Then $SourceLength -= 1; gotta have the array elements start with a backslash...

$FileList = _FileListToArrayRec($Source, "", "*", "", 1, 2, True)



If StringLeft($Source, 2) == "" Then
$SourceUNCPrefix = "?UNC"
$Source = StringTrimLeft($Source, 2)
Else
$SourceUNCPrefix = "?"
EndIf
If StringLeft($Target, 2) == "" Then
$TargetUNCPrefix = "?UNC"
$Target = StringTrimLeft($Target, 2)
Else
$TargetUNCPrefix = "?"
EndIf
_ArrayTrim($FileList, $SourceLength, 0, 1)

For $a = 1 To $FileList[0]
$CopySource = $SourceUNCPrefix & $Source & $FileList[$a]
$CopyTarget = $TargetUNCPrefix & $Target & $FileList[$a]
$Copy = _LargeFileCopy($CopySource, $CopyTarget, BitOR(1, 2))
$Error = @error
If $Copy == 1 Then
$Message = "SUCCESS"
Else
$Message = $Error
EndIf
ConsoleWrite($CopySource & " -> " & $CopyTarget & " -> " & $Message & @CR)
Next



; #FUNCTION# ====================================================================================================
; Name...........:  _LargeFileCopy
; Description....:  Copy large files in such a way as to keep AutoIt GUIs responsive
; Syntax.........:  _LargeFileCopy($sSrc, $sDest, $iFlags)
; Parameters.....:  $sSrc       - Source file name
;                   $sDest      - Destination: may be a file name or directory
;                   $iFlags     - [Optional] Combine flags with BitOR
;                               | 1 - Overwrite existing file
;                               | 2 - Create destination directory structure
;                               | 4 - Flush the destination file buffer before returning
;                               | 8 - Verify source and destination are identical via bit by bit comparison
;                               |16 - Verify source and destination are identical via MD5 hash
;                               |32 - Verify source and destination file size only
;                               + If more than one verify flag is set, the smallest flag will take precedence
;
; Return values..:  Success     - 1
;                   Failure     - 0 and sets @error
;                               | 1 - Failed to open source file, or source was a directory
;                               | 2 - Destination file exists and overwrite flag not set
;                               | 3 - Failed to create destination file
;                               | 4 - Read error during copy
;                               | 5 - Write error during copy
;                               | 6 - Verify failed
; Author.........:  Erik Pilsits
; Modified.......:
; Remarks........: MODIFIED BY IAN MAXWELL FOR THIS SPECIFIC SCRIPT, DO NOT COPY THIS FOR RE-USE!!!!
; Related........:
; Link...........:
; Example........:
; ===============================================================================================================
Func _LargeFileCopy($sSrc, $sDest, $iFlags = 0)
; check / fix source and dest path syntax
_LFC_FixPaths($sSrc, $sDest)

; open file for reading, fail if it doesn't exist or directory
Local $hSrc = _LFC_CreateFile($sSrc, $GENERIC_READ, $FILE_SHARE_READ, $OPEN_EXISTING, 0)
If Not $hSrc Then Return SetError(1, 0, 0)

; set option flags
Local $fOverwrite = (BitAND($iFlags, 1) = 1)
Local $fCreate = (BitAND($iFlags, 2) = 2)

; check destination
_LFC_CheckDestination($sSrc, $sDest, $fOverwrite, $fCreate)
If @error Then
_WinAPI_CloseHandle($hSrc)
Return SetError(2, 0, 0)
EndIf

; create new file for writing, overwrite
Local $hDest = _LFC_CreateFile($sDest, BitOR($GENERIC_READ, $GENERIC_WRITE), 0, $CREATE_ALWAYS, 0)
If Not $hDest Then
_WinAPI_CloseHandle($hSrc)
Return SetError(3, 0, 0)
EndIf

; check for 0 byte source file
Local $iSize = _WinAPI_GetFileSizeEx($hSrc)
If $iSize = 0 Then
; done, close handles and return success
_WinAPI_CloseHandle($hDest)
_WinAPI_CloseHandle($hSrc)
Return 1
EndIf

; perform copy
Local $iRead, $iWritten, $iTotal = 0, $mSrc = 0, $iReadError = 0, $iWriteError = 0, $iVerifyError = 0
; allocate buffers
Local $hBuffer = DllStructCreate("byte[" & $iToRead & "]")
Local $pBuffer = DllStructGetPtr($hBuffer)

Do
If Not _WinAPI_ReadFile($hSrc, $pBuffer, $iToRead, $iRead) Then
$iReadError = 1
ExitLoop
EndIf
If $iRead = 0 Then ExitLoop ; end of file, edge case if file is an exact multiple of the buffer size
If Not _WinAPI_WriteFile($hDest, $pBuffer, $iRead, $iWritten) Or ($iRead <> $iWritten) Then
$iWriteError = 1
ExitLoop
EndIf
Until $iRead < $iToRead

_WinAPI_CloseHandle($hDest)
_WinAPI_CloseHandle($hSrc)

If $iReadError Then
Return SetError(4, 0, 0)
ElseIf $iWriteError Then
Return SetError(5, 0, 0)
Else
Return 1
EndIf
EndFunc   ;==>_LargeFileCopy
#region INTERNAL FUNCTIONS
Func _LFC_CheckDestination($sSrc, ByRef $sDest, $fOverwrite, $fCreate)
If (StringRight($sDest, 1) = "") Or StringInStr(FileGetAttrib($sDest), "D") Then
; assume directory
If $sSrc = "" Then
; raw copy, must provide a file name
Return SetError(2)
Else
; use source file name
If StringRight($sDest, 1) <> "" Then $sDest &= "" ; add trailing 
$sDest &= StringRegExpReplace($sSrc, ".*", "")
EndIf
EndIf
; check overwrite
If (Not $fOverwrite) And FileExists($sDest) Then Return SetError(1)
; create destination parent directory
; if create flag not set and destination doesn't exist, CreateFile will fail
If $fCreate Then DirCreate(StringRegExpReplace($sDest, "^(.*).*?$", "${1}"))
;~  DirCreate(StringRegExpReplace($sDest, "^(.*).*?$", "${1}"))
EndFunc   ;==>_LFC_CheckDestination
Func _LFC_CreateFile($sPath, $iAccess, $iShareMode, $iCreation, $iFlags)
; open the file with existing HIDDEN or SYSTEM attributes to avoid failure when using CREATE_ALWAYS
If $iCreation = $CREATE_ALWAYS Then
Local $sAttrib = FileGetAttrib($sPath)
If StringInStr($sAttrib, "H") Then $iFlags = BitOR($iFlags, $FILE_ATTRIBUTE_HIDDEN)
If StringInStr($sAttrib, "S") Then $iFlags = BitOR($iFlags, $FILE_ATTRIBUTE_SYSTEM)
EndIf
Local $hFile = DllCall("kernel32.dll", "handle", "CreateFileW", "wstr", $sPath, "dword", $iAccess, "dword", $iShareMode, "ptr", 0, _
"dword", $iCreation, "dword", $iFlags, "ptr", 0)
If @error Or ($hFile[0] = Ptr(-1)) Then Return SetError(1, 0, 0)
Return $hFile[0]
EndFunc   ;==>_LFC_CreateFile
Func _LFC_FixPath(ByRef $sPath)
; fix possible forward /
$sPath = StringRegExpReplace($sPath, "/", "")
If Not StringInStr($sPath, ":") Then
; not a full path
; check for bare, relative, or UNC paths
If StringLeft($sPath, 1) = "" Then
; check for UNC path
If StringLeft($sPath, 2) <> "" Then
; not a UNC path
$sPath = "." & $sPath
; else, probably a UNC path, leave it alone
EndIf
Else
; check for bare or relative path
If StringLeft($sPath, 1) <> "." Then
; bare path
$sPath = "." & $sPath
; else, probably relative path, leave it alone
EndIf
EndIf
EndIf
EndFunc   ;==>_LFC_FixPath
Func _LFC_FixPaths(ByRef $sSrc, ByRef $sDest)
; remove trailing 's from source only
$sSrc = StringRegExpReplace($sSrc, "+$", "")
; check for relative paths in source and dest
_LFC_FixPath($sSrc)
_LFC_FixPath($sDest)
EndFunc   ;==>_LFC_FixPaths
#endregion INTERNAL FUNCTIONS


Func _FileListToArrayRec($sPath, $sExcludeFolderList = "", $sIncludeList = "*", $sExcludeList = "", $iReturnType = 0, $iReturnFormat = 0, $bRecursive = False)
Local $sRet = "", $sReturnFormat = ""
$sPath = StringRegExpReplace($sPath, "[/]+z", "") & ""
If Not FileExists($sPath) Then Return SetError(1, 1, "")
If $sIncludeList = "*" Then
$sIncludeList = ""
Else
If StringRegExp($sIncludeList, "[/ :> <|]|(?s)As*z") Then Return SetError(2, 2, "")
$sIncludeList = StringRegExpReplace(StringRegExpReplace($sIncludeList, "(s*;s*)+", ";"), "A;|;z", "")
$sIncludeList = StringRegExpReplace($sIncludeList, '[.$]', '[0]')
$sIncludeList = StringReplace(StringReplace($sIncludeList, "?", "."), "*", ".*?")
$sIncludeList = "(?i)A(" & StringReplace($sIncludeList, ";", "$|") & "$)"
EndIf
If Not ($iReturnType = 0 Or $iReturnType = 1 Or $iReturnType = 2) Then Return SetError(3, 3, "")
Local $sOrigPathLen = StringLen($sPath), $aQueue[64] = [1, $sPath], $iQMax = 63
While $aQueue[0]
$WorkFolder = $aQueue[$aQueue[0]]
$aQueue[0] -= 1
$search = FileFindFirstFile($WorkFolder & "*") ; get full list of current folder
If @error Then ContinueLoop
Switch $iReturnFormat
Case 1 ; relative path
$sReturnFormat = StringTrimLeft($WorkFolder, $sOrigPathLen)
Case 2 ; full path
$sReturnFormat = $WorkFolder
EndSwitch
While 1 ; process current folder
$file = FileFindNextFile($search)
If @error Then ExitLoop
If @extended Then ; Folder
If $bRecursive Then ; push folder onto queue
If $aQueue[0] = $iQMax Then
$iQMax += 128
ReDim $aQueue[$iQMax + 1]
EndIf
$aQueue[0] += 1
$aQueue[$aQueue[0]] = $WorkFolder & $file & ""
EndIf
If $iReturnType = 1 Then ContinueLoop
Else ; File
If $iReturnType = 2 Then ContinueLoop
EndIf
If $sIncludeList And Not StringRegExp($file, $sIncludeList) Then ContinueLoop
$sRet &= $sReturnFormat & $file & "|" ; append to output
WEnd
FileClose($search)
WEnd
If Not $sRet Then Return SetError(4, 4, "")
Return StringSplit(StringTrimRight($sRet, 1), "|")
EndFunc   ;==>_FileListToArrayRec

To test, create the $Source folder and put some stuff in there, couple files and small folders, and let it run, and everything will be fine. Then start to add some subfolders to the $Target and it will fail when it gets too long with @error 3.

Any further thoughts on how this can be overcome?

Thanks!

Ian

[wraithdu]

I'll probably get around to testing this Monday.

[llewxam]

That would be great! I have been poking at this for the last 24 hours and haven't got a clue. All I have found is that newer versions of AutoIt are Unicode compliant, and the CreateFileW you call is the Unicode version, sooooooo......why won't a Unicode path work?????? gggrrrrrrrrrr!!!!!

Maybe something unrelated? A SciTE wrapper or compiler option to force Unicode that I haven't been finding? I even tried using #RequireAdmin out of exacerbation.....

Thanks a million

Ian

[wraithdu]

Well that didn't take long to figure out. I should probably open a ticket for this too. AutoIt's internal DirCreate is reporting success but is truncating the new directory to 255 characters. CreateFile obviously fails if the parent directory does not exist, so there ya go.

I'll need to replace the call(s) to DirCreate with the CreateDirectory API which works with UNC style extra long paths.

[wraithdu]

Hmm, gonna take a little while longer. CreateDirectory does not work recursively, so I have to design that, including a function to crack all the possible path types.

[llewxam]

Sounds like a PITA, but yours will be the only routine out there that can do it, I tested Yashied's UDF and it bonked as well.

Thought - maybe your revised UDF should internally convert to UNC and just get it out of the way, Seems like you going through all the effort of making it CAPABLE of handling a Long UNC path doesn't make a lot of sense if it doesn't actually do it.... If you do or don't add that feature, it's still awesome that you are making progress.

Ian

[wraithdu]

Thought - maybe your revised UDF should internally convert to UNC and just get it out of the way

I'm deciding if I want to do that or not. My current thought is that Windows doesn't do that internally, so I shouldn't either. It should be the scripter's responsibility to check path length. Also, if I do it for directory creation, then I should do it for CreateFile as well, and that seems out of scope. Edited November 21, 2011 by wraithdu

[wraithdu]

Ok, try this updated version of your example. Please review the new functions to see if anything jumps out at you as being incorrect.

expandcollapse popup

#include <WinAPI.au3>
#include <Array.au3>

$iToRead = 8 * 1024 * 1024

$Source = @ScriptDir & "testdir"
$Target = @ScriptDir & "asiodfoiwoiawoiefioawejfaawoeifoafjjfejiwaoaiwefjoawiefaoiawoefjofijijewoaoawejfjaw"
DirRemove($Target, 1)

If StringRight($Source, 1) <> "" Then $Source &= ""
If StringRight($Target, 1) <> "" Then $Target &= ""
$SourceLength = StringLen($Source)
If $SourceLength == 3 Then $SourceLength -= 1; gotta have the array elements start with a backslash...

$FileList = _FileListToArrayRec($Source, "", "*", "", 1, 2, True)

If StringLeft($Source, 2) == "" Then
    $SourceUNCPrefix = "?UNC"
    $Source = StringTrimLeft($Source, 2)
Else
    $SourceUNCPrefix = "?"
EndIf

If StringLeft($Target, 2) == "" Then
    $TargetUNCPrefix = "?UNC"
    $Target = StringTrimLeft($Target, 2)
Else
    $TargetUNCPrefix = "?"
EndIf

_ArrayTrim($FileList, $SourceLength, 0, 1)

For $a = 1 To $FileList[0]
    $CopySource = $SourceUNCPrefix & $Source & $FileList[$a]
    $CopyTarget = $TargetUNCPrefix & $Target & $FileList[$a]

    ConsoleWrite(StringLen($CopySource) & " : " & StringLen($CopyTarget) & @CRLF)

    $Copy = _LargeFileCopy($CopySource, $CopyTarget, BitOR(1, 2))
    $Error = @error
    If $Copy == 1 Then
        $Message = "SUCCESS"
    Else
        $Message = $Error
    EndIf
    ConsoleWrite($CopySource & " -> " & $CopyTarget & " -> " & $Message & @CR)
Next

; #FUNCTION# ====================================================================================================
; Name...........:  _LargeFileCopy
; Description....:  Copy large files in such a way as to keep AutoIt GUIs responsive
; Syntax.........:  _LargeFileCopy($sSrc, $sDest, $iFlags)
; Parameters.....:  $sSrc      - Source file name
;                  $sDest     - Destination: may be a file name or directory
;                  $iFlags   - [Optional] Combine flags with BitOR
;                              | 1 - Overwrite existing file
;                              | 2 - Create destination directory structure
;                              | 4 - Flush the destination file buffer before returning
;                              | 8 - Verify source and destination are identical via bit by bit comparison
;                              |16 - Verify source and destination are identical via MD5 hash
;                              |32 - Verify source and destination file size only
;                              + If more than one verify flag is set, the smallest flag will take precedence
;
; Return values..:  Success  - 1
;                  Failure   - 0 and sets @error
;                              | 1 - Failed to open source file, or source was a directory
;                              | 2 - Destination file exists and overwrite flag not set
;                              | 3 - Failed to create destination file
;                              | 4 - Read error during copy
;                              | 5 - Write error during copy
;                              | 6 - Verify failed
; Author.........:  Erik Pilsits
; Modified.......:
; Remarks........: MODIFIED BY IAN MAXWELL FOR THIS SPECIFIC SCRIPT, DO NOT COPY THIS FOR RE-USE!!!!
; Related........:
; Link...........:
; Example........:
; ===============================================================================================================
Func _LargeFileCopy($sSrc, $sDest, $iFlags = 0)
    ; check / fix source and dest path syntax
    _LFC_FixPaths($sSrc, $sDest)

    ; open file for reading, fail if it doesn't exist or directory
    Local $hSrc = _LFC_CreateFile($sSrc, $GENERIC_READ, $FILE_SHARE_READ, $OPEN_EXISTING, 0)
    If Not $hSrc Then Return SetError(1, 0, 0)

    ; set option flags
    Local $fOverwrite = (BitAND($iFlags, 1) = 1)
    Local $fCreate = (BitAND($iFlags, 2) = 2)

    ; check destination
    _LFC_CheckDestination($sSrc, $sDest, $fOverwrite, $fCreate)
    If @error Then
        _WinAPI_CloseHandle($hSrc)
        Return SetError(2, 0, 0)
    EndIf

    ; create new file for writing, overwrite
    Local $hDest = _LFC_CreateFile($sDest, BitOR($GENERIC_READ, $GENERIC_WRITE), 0, $CREATE_ALWAYS, 0)
    If Not $hDest Then
        _WinAPI_CloseHandle($hSrc)
        Return SetError(3, 0, 0)
    EndIf

    ; check for 0 byte source file
    Local $iSize = _WinAPI_GetFileSizeEx($hSrc)
    If $iSize = 0 Then
        ; done, close handles and return success
        _WinAPI_CloseHandle($hDest)
        _WinAPI_CloseHandle($hSrc)
        Return 1
    EndIf

    ; perform copy
    Local $iRead, $iWritten, $iTotal = 0, $mSrc = 0, $iReadError = 0, $iWriteError = 0, $iVerifyError = 0
    ; allocate buffers
    Local $hBuffer = DllStructCreate("byte[" & $iToRead & "]")
    Local $pBuffer = DllStructGetPtr($hBuffer)

    Do
        If Not _WinAPI_ReadFile($hSrc, $pBuffer, $iToRead, $iRead) Then
            $iReadError = 1
            ExitLoop
        EndIf
        If $iRead = 0 Then ExitLoop ; end of file, edge case if file is an exact multiple of the buffer size
        If Not _WinAPI_WriteFile($hDest, $pBuffer, $iRead, $iWritten) Or ($iRead <> $iWritten) Then
            $iWriteError = 1
            ExitLoop
        EndIf
    Until $iRead < $iToRead

    _WinAPI_CloseHandle($hDest)
    _WinAPI_CloseHandle($hSrc)

    If $iReadError Then
        Return SetError(4, 0, 0)
    ElseIf $iWriteError Then
        Return SetError(5, 0, 0)
    Else
        Return 1
    EndIf
EndFunc   ;==>_LargeFileCopy

#region INTERNAL FUNCTIONS
Func _LFC_CheckDestination($sSrc, ByRef $sDest, $fOverwrite, $fCreate)
    If (StringRight($sDest, 1) = "") Or StringInStr(FileGetAttrib($sDest), "D") Then
        ; assume directory
        If $sSrc = "" Then
            ; raw copy, must provide a file name
            Return SetError(2)
        Else
            ; use source file name
            If StringRight($sDest, 1) <> "" Then $sDest &= "" ; add trailing 
            $sDest &= StringRegExpReplace($sSrc, ".*", "")
        EndIf
    EndIf
    ; check overwrite
    If (Not $fOverwrite) And FileExists($sDest) Then Return SetError(1)
    ; create destination parent directory
    ; if create flag not set and destination doesn't exist, CreateFile will fail
    If $fCreate Then _LFC_CreateDirectory(StringRegExpReplace($sDest, "^(.*).*?$", "${1}"))
EndFunc   ;==>_LFC_CheckDestination

Func _LFC_CrackPath($sPath)
    ; make sure path is terminated
    If StringRight($sPath, 1) <> "" Then $sPath &= ""
    ; get prefix
    Local $aRet = StringRegExp($sPath, "(?i)^?unc|?|", 1)
    If Not IsArray($aRet) Then
        $aRet = ""
    Else
        $aRet = $aRet[0]
    EndIf
    ; capture and remove the root
    If ($aRet = "") Or ($aRet = "?UNC") Then
        ; UNC network path
        $aRet = StringRegExp($sPath, "(?i)^(" & StringReplace(StringReplace($aRet, "", ""), "?", "?") & ".*?.*?)", 1)
    Else
        ; $aRet = "" or ? => local path
        Local $iTrim = StringLen($aRet) + 3
        Local $aRet[1] = [StringLeft($sPath, $iTrim)]
    EndIf
    Local $aPath = StringTrimLeft($sPath, StringLen($aRet[0]))
    ; check if path given was just a root
    If $aPath <> "" Then
        ; crack path, prepend  to get first segment
        $aPath = StringRegExp("" & $aPath, "(.*?)(?=)", 3)
        ReDim $aRet[UBound($aPath) + 1]
        For $i = 0 To UBound($aPath) - 1
            $aRet[$i + 1] = $aPath[$i]
        Next
    EndIf
    ;
    Return $aRet
EndFunc

Func _LFC_CreateDirectory($sPath)
    ; check path already exists
    If FileExists($sPath) Then
        ; check if it is a direcotry
        If StringInStr(FileGetAttrib($sPath), "D") Then
            Return 1
        Else
            Return SetError(1, 0, 0)
        EndIf
    EndIf
    ;
    Local $aPath = _LFC_CrackPath($sPath)
    ; make sure root exists
    Local $sSegment = $aPath[0]
    If Not FileExists($sSegment) Then Return SetError(2, 0, 0)
    Local $ret
    For $i = 1 To UBound($aPath) - 1
        $sSegment &= $aPath[$i] & ""
        If Not FileExists($sSegment) Then
            ; create sub-path
            $ret = DllCall("kernel32.dll", "bool", "CreateDirectoryW", "wstr", $sSegment, "ptr", 0)
            If @error Or Not $ret[0] Then Return SetError(3, 0, 0)
        EndIf
    Next
    Return 1
EndFunc

Func _LFC_CreateFile($sPath, $iAccess, $iShareMode, $iCreation, $iFlags)
    ; open the file with existing HIDDEN or SYSTEM attributes to avoid failure when using CREATE_ALWAYS
    If $iCreation = $CREATE_ALWAYS Then
        Local $sAttrib = FileGetAttrib($sPath)
        If StringInStr($sAttrib, "H") Then $iFlags = BitOR($iFlags, $FILE_ATTRIBUTE_HIDDEN)
        If StringInStr($sAttrib, "S") Then $iFlags = BitOR($iFlags, $FILE_ATTRIBUTE_SYSTEM)
    EndIf
    Local $hFile = DllCall("kernel32.dll", "handle", "CreateFileW", "wstr", $sPath, "dword", $iAccess, "dword", $iShareMode, "ptr", 0, _
            "dword", $iCreation, "dword", $iFlags, "ptr", 0)
    If @error Or ($hFile[0] = Ptr(-1)) Then
        ConsoleWrite("ERR: " & $hFile[0] & " : " & _WinAPI_GetLastErrorMessage())
        ConsoleWrite($sPath & @CRLF)
        Return SetError(1, 0, 0)
    EndIf
    Return $hFile[0]
EndFunc   ;==>_LFC_CreateFile

Func _LFC_FixPath(ByRef $sPath)
    ; fix possible forward /
    $sPath = StringRegExpReplace($sPath, "/", "")
    If Not StringInStr($sPath, ":") Then
        ; not a full path
        ; check for bare, relative, or UNC paths
        If StringLeft($sPath, 1) = "" Then
            ; check for UNC path
            If StringLeft($sPath, 2) <> "" Then
                ; not a UNC path
                $sPath = "." & $sPath
                ; else, probably a UNC path, leave it alone
            EndIf
        Else
            ; check for bare or relative path
            If StringLeft($sPath, 1) <> "." Then
                ; bare path
                $sPath = "." & $sPath
                ; else, probably relative path, leave it alone
            EndIf
        EndIf
    EndIf
EndFunc   ;==>_LFC_FixPath

Func _LFC_FixPaths(ByRef $sSrc, ByRef $sDest)
    ; remove trailing 's from source only
    $sSrc = StringRegExpReplace($sSrc, "+$", "")
    ; check for relative paths in source and dest
    _LFC_FixPath($sSrc)
    _LFC_FixPath($sDest)
EndFunc   ;==>_LFC_FixPaths
#endregion INTERNAL FUNCTIONS

Func _FileListToArrayRec($sPath, $sExcludeFolderList = "", $sIncludeList = "*", $sExcludeList = "", $iReturnType = 0, $iReturnFormat = 0, $bRecursive = False)
    Local $sRet = "", $sReturnFormat = ""
    $sPath = StringRegExpReplace($sPath, "[/]+z", "") & ""
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If $sIncludeList = "*" Then
        $sIncludeList = ""
    Else
        If StringRegExp($sIncludeList, "[/ :> <|]|(?s)As*z") Then Return SetError(2, 2, "")
        $sIncludeList = StringRegExpReplace(StringRegExpReplace($sIncludeList, "(s*;s*)+", ";"), "A;|;z", "")
        $sIncludeList = StringRegExpReplace($sIncludeList, '[.$]', '[0]')
        $sIncludeList = StringReplace(StringReplace($sIncludeList, "?", "."), "*", ".*?")
        $sIncludeList = "(?i)A(" & StringReplace($sIncludeList, ";", "$|") & "$)"
    EndIf
    If Not ($iReturnType = 0 Or $iReturnType = 1 Or $iReturnType = 2) Then Return SetError(3, 3, "")
    Local $sOrigPathLen = StringLen($sPath), $aQueue[64] = [1, $sPath], $iQMax = 63
    While $aQueue[0]
        $WorkFolder = $aQueue[$aQueue[0]]
        $aQueue[0] -= 1
        $search = FileFindFirstFile($WorkFolder & "*") ; get full list of current folder
        If @error Then ContinueLoop
        Switch $iReturnFormat
            Case 1 ; relative path
                $sReturnFormat = StringTrimLeft($WorkFolder, $sOrigPathLen)
            Case 2 ; full path
                $sReturnFormat = $WorkFolder
        EndSwitch
        While 1 ; process current folder
            $file = FileFindNextFile($search)
            If @error Then ExitLoop
            If @extended Then ; Folder
                If $bRecursive Then ; push folder onto queue
                    If $aQueue[0] = $iQMax Then
                        $iQMax += 128
                        ReDim $aQueue[$iQMax + 1]
                    EndIf
                    $aQueue[0] += 1
                    $aQueue[$aQueue[0]] = $WorkFolder & $file & ""
                EndIf
                If $iReturnType = 1 Then ContinueLoop
            Else ; File
                If $iReturnType = 2 Then ContinueLoop
            EndIf
            If $sIncludeList And Not StringRegExp($file, $sIncludeList) Then ContinueLoop
            $sRet &= $sReturnFormat & $file & "|" ; append to output
        WEnd
        FileClose($search)
    WEnd
    If Not $sRet Then Return SetError(4, 4, "")
    Return StringSplit(StringTrimRight($sRet, 1), "|")
EndFunc   ;==>_FileListToArrayRec

Edit:

I've created ticket #2054 for this issue.

Edited November 21, 2011 by wraithdu

[llewxam]

NICE MAN!!

I just ran a test and got a path length of 452 and it was all good! haha, right on! And the new FileListToArrayRec that I have been messing around with did NOT find any files in that huge path, but adding the UNC prefix made it able to find the files, so no horrible mess appears to be needed for that to be happy!

GOOD JOB!

Ian

[wraithdu]

Cool Please let me know if you find any problems or edge cases. I'll update the main UDF soon if you do not.

[islandspapand]

thanks for a great UDF.

a quick noob question how do i make a progressbar-copy.. i have tried a few thing but i do not totaly under stand the function paramter calls

that i have tried eksample:

#include <Process.au3>

#include <_Largefilecopy.au3>

#include <Array.au3>

Local $Test, $FileCopy, $1, $2,$iWritten, $iTotal,$vVar

; $iTotal=FileGetSize("E:ZS7-168ASM4-KP0_wo3.1.0.0.WIM")

;$iWritten=FileGetSize("E:MSIZS7-168ASM4-KP0_wo3.1.0.0.WIM")

$FileCopy=_LargeFileCopy("E:ZS7-168ASM4-KP0_wo3.1.0.0.WIM", "E:MSIZS7-168ASM4-KP0_wo3.1.0.0.WIM", 1,60000,"",test($iWritten, $iTotal))

ConsoleWrite("1 : "&$iWritten&@CR)

ConsoleWrite("2 : "&$iTotal&@CR)

If $FileCopy=0 Then MsgBox(0,"Copy","Fail to Copy")

If $FileCopy=1 Then MsgBox(0,"Copy","Copy'ed Your file")

Func test($iWritten, $iTotal)

$iTotal=FileGetSize("E:ZS7-168ASM4-KP0_wo3.1.0.0.WIM")

$iWritten=FileGetSize("E:MSIZS7-168ASM4-KP0_wo3.1.0.0.WIM")

ConsoleWrite("1 : "&$iWritten&@CR)

ConsoleWrite("2 : "&$iTotal&@CR)

EndFunc

hope somebody can help. Thanks in advance

[wraithdu]

The progress function is automatically called with the 3 parameters described in the function header. You need to change your test function to take all three parameters, even if you don't intend to use them. Then in your call to _LargeFileCopy pass only the name of your progress function as a string, ie "test".

[Topher]

Just what I needed. Thank you.

[Syed23]

Thanks for this Extradinary UDF... i have a quick question. is there a way to copy the entire folder by using this function? i am following the below task where it fails everytime.. any reason?

#include<Array.au3>
#include <_LargeFileCopy.au3>

Global $coppst[2]
$coppst[0] = "c:exchange"
$coppst[1] = "C:UsersQ03200AppDataLocalMicrosoftOutlook"

For $s = 0 To UBound($coppst)-1  Step +1
;$m = DirCopy($coppst[$s], "C:backup"&StringTrimLeft($coppst[$s],2),1)

; MsgBox(0, "", $coppst[$s]&"*.*       "& "C:backup" & StringTrimLeft($coppst[$s], 2)&"*.psst")
$m = _LargeFileCopy($coppst[$s]&"*.*", "C:backup" & StringTrimLeft($coppst[$s], 2)&"*.*", BitOR(2,8))
MsgBox(0,"",$m)
Next

[wraithdu]

Quick answer... no, this UDF does not natively support recursive copying, and it's not going to.

It also does not support use of wildcards. You'll have to implement your own search/loop/copy algorithms.

Edited June 25, 2012 by wraithdu

[Syed23]

Quick answer... no, this UDF does not natively support recursive copying, and it's not going to.

It also does not support use of wildcards. You'll have to implement your own search/loop/copy algorithms.

Thanks Wraithdu! may i know what is the methodology we are using for the copy process? because what i could feel is when i use this function to copy file this is bit faster than normal process. can you give me an overview of the technique you have used in this function? how it is copying in this speed?

[llewxam]

Syed23

If you look over the code in this UDF you will see that wraithdu is using WinAPI calls. There are more lines of code in using this method since wraithdu's UDF has to specify the position in the file to open, as well as position to write in the destination, additional error checking, handling UNC paths, etc. but his code is proof that well-written and optimized code can be far faster than fewer lines of poor code.

hmm, maybe it is just easier to say that using Windows internal API calls will result in faster code

Ian

[Syed23]

Just curious, I have used this function in my GUI and it works perfect, but in some cases when copy bit huge size of file the GUI shows not responding and the marquee progress bar freezes.. could you please help me to over come this? if you have any suggestion how to trouble shoot could you please tell me that so that i can give you the exact scenario where i am facing that problem. I am using this function in Windows 7 64 bit machines.

Thanks in advance!