Jump to content
Sign in to follow this  
wraithdu

_LargeFileCopy UDF

Recommended Posts

Hmm. Try to create a small reproducer script with all code req'd and I'll take a look. Maybe you just need a second pair of eyes on it. I've looked through all the path manipulation code in my UDF and tested the regular expressions, and nothing is mangling UNC style paths.

Share this post


Link to post
Share on other sites

OK, good idea. I'll work on that tomorrow night, and who knows, maybe that will show me the way to correcting it....

Thanks

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

Thought I should report back. I did start small and had good results, then today I folded all of the tweaks in to my Sync tool and everything was working great - until I tried to exceed the normal 260 character limit using UNC paths - those transfers failed. :D

Here is a stripped-down tester for you:

#region ;**** Directives created by AutoIt3Wrapper_GUI ****
#AutoIt3Wrapper_UseUpx=n
#AutoIt3Wrapper_Run_Tidy=y
#endregion ;**** Directives created by AutoIt3Wrapper_GUI ****

#include <WinAPI.au3>
#include <Array.au3>

$iToRead = 8 * 1024 * 1024

$Source = "c:testunc"
$Target = "c:testunc2"

If StringRight($Source, 1) <> "" Then $Source &= ""
If StringRight($Target, 1) <> "" Then $Target &= ""
$SourceLength = StringLen($Source)
If $SourceLength == 3 Then $SourceLength -= 1; gotta have the array elements start with a backslash...

$FileList = _FileListToArrayRec($Source, "", "*", "", 1, 2, True)



If StringLeft($Source, 2) == "" Then
$SourceUNCPrefix = "?UNC"
$Source = StringTrimLeft($Source, 2)
Else
$SourceUNCPrefix = "?"
EndIf
If StringLeft($Target, 2) == "" Then
$TargetUNCPrefix = "?UNC"
$Target = StringTrimLeft($Target, 2)
Else
$TargetUNCPrefix = "?"
EndIf
_ArrayTrim($FileList, $SourceLength, 0, 1)

For $a = 1 To $FileList[0]
$CopySource = $SourceUNCPrefix & $Source & $FileList[$a]
$CopyTarget = $TargetUNCPrefix & $Target & $FileList[$a]
$Copy = _LargeFileCopy($CopySource, $CopyTarget, BitOR(1, 2))
$Error = @error
If $Copy == 1 Then
$Message = "SUCCESS"
Else
$Message = $Error
EndIf
ConsoleWrite($CopySource & " -> " & $CopyTarget & " -> " & $Message & @CR)
Next



; #FUNCTION# ====================================================================================================
; Name...........:  _LargeFileCopy
; Description....:  Copy large files in such a way as to keep AutoIt GUIs responsive
; Syntax.........:  _LargeFileCopy($sSrc, $sDest, $iFlags)
; Parameters.....:  $sSrc       - Source file name
;                   $sDest      - Destination: may be a file name or directory
;                   $iFlags     - [Optional] Combine flags with BitOR
;                               | 1 - Overwrite existing file
;                               | 2 - Create destination directory structure
;                               | 4 - Flush the destination file buffer before returning
;                               | 8 - Verify source and destination are identical via bit by bit comparison
;                               |16 - Verify source and destination are identical via MD5 hash
;                               |32 - Verify source and destination file size only
;                               + If more than one verify flag is set, the smallest flag will take precedence
;
; Return values..:  Success     - 1
;                   Failure     - 0 and sets @error
;                               | 1 - Failed to open source file, or source was a directory
;                               | 2 - Destination file exists and overwrite flag not set
;                               | 3 - Failed to create destination file
;                               | 4 - Read error during copy
;                               | 5 - Write error during copy
;                               | 6 - Verify failed
; Author.........:  Erik Pilsits
; Modified.......:
; Remarks........: MODIFIED BY IAN MAXWELL FOR THIS SPECIFIC SCRIPT, DO NOT COPY THIS FOR RE-USE!!!!
; Related........:
; Link...........:
; Example........:
; ===============================================================================================================
Func _LargeFileCopy($sSrc, $sDest, $iFlags = 0)
; check / fix source and dest path syntax
_LFC_FixPaths($sSrc, $sDest)

; open file for reading, fail if it doesn't exist or directory
Local $hSrc = _LFC_CreateFile($sSrc, $GENERIC_READ, $FILE_SHARE_READ, $OPEN_EXISTING, 0)
If Not $hSrc Then Return SetError(1, 0, 0)

; set option flags
Local $fOverwrite = (BitAND($iFlags, 1) = 1)
Local $fCreate = (BitAND($iFlags, 2) = 2)

; check destination
_LFC_CheckDestination($sSrc, $sDest, $fOverwrite, $fCreate)
If @error Then
_WinAPI_CloseHandle($hSrc)
Return SetError(2, 0, 0)
EndIf

; create new file for writing, overwrite
Local $hDest = _LFC_CreateFile($sDest, BitOR($GENERIC_READ, $GENERIC_WRITE), 0, $CREATE_ALWAYS, 0)
If Not $hDest Then
_WinAPI_CloseHandle($hSrc)
Return SetError(3, 0, 0)
EndIf

; check for 0 byte source file
Local $iSize = _WinAPI_GetFileSizeEx($hSrc)
If $iSize = 0 Then
; done, close handles and return success
_WinAPI_CloseHandle($hDest)
_WinAPI_CloseHandle($hSrc)
Return 1
EndIf

; perform copy
Local $iRead, $iWritten, $iTotal = 0, $mSrc = 0, $iReadError = 0, $iWriteError = 0, $iVerifyError = 0
; allocate buffers
Local $hBuffer = DllStructCreate("byte[" & $iToRead & "]")
Local $pBuffer = DllStructGetPtr($hBuffer)

Do
If Not _WinAPI_ReadFile($hSrc, $pBuffer, $iToRead, $iRead) Then
$iReadError = 1
ExitLoop
EndIf
If $iRead = 0 Then ExitLoop ; end of file, edge case if file is an exact multiple of the buffer size
If Not _WinAPI_WriteFile($hDest, $pBuffer, $iRead, $iWritten) Or ($iRead <> $iWritten) Then
$iWriteError = 1
ExitLoop
EndIf
Until $iRead < $iToRead

_WinAPI_CloseHandle($hDest)
_WinAPI_CloseHandle($hSrc)

If $iReadError Then
Return SetError(4, 0, 0)
ElseIf $iWriteError Then
Return SetError(5, 0, 0)
Else
Return 1
EndIf
EndFunc   ;==>_LargeFileCopy
#region INTERNAL FUNCTIONS
Func _LFC_CheckDestination($sSrc, ByRef $sDest, $fOverwrite, $fCreate)
If (StringRight($sDest, 1) = "") Or StringInStr(FileGetAttrib($sDest), "D") Then
; assume directory
If $sSrc = "" Then
; raw copy, must provide a file name
Return SetError(2)
Else
; use source file name
If StringRight($sDest, 1) <> "" Then $sDest &= "" ; add trailing 
$sDest &= StringRegExpReplace($sSrc, ".*", "")
EndIf
EndIf
; check overwrite
If (Not $fOverwrite) And FileExists($sDest) Then Return SetError(1)
; create destination parent directory
; if create flag not set and destination doesn't exist, CreateFile will fail
If $fCreate Then DirCreate(StringRegExpReplace($sDest, "^(.*).*?$", "${1}"))
;~  DirCreate(StringRegExpReplace($sDest, "^(.*).*?$", "${1}"))
EndFunc   ;==>_LFC_CheckDestination
Func _LFC_CreateFile($sPath, $iAccess, $iShareMode, $iCreation, $iFlags)
; open the file with existing HIDDEN or SYSTEM attributes to avoid failure when using CREATE_ALWAYS
If $iCreation = $CREATE_ALWAYS Then
Local $sAttrib = FileGetAttrib($sPath)
If StringInStr($sAttrib, "H") Then $iFlags = BitOR($iFlags, $FILE_ATTRIBUTE_HIDDEN)
If StringInStr($sAttrib, "S") Then $iFlags = BitOR($iFlags, $FILE_ATTRIBUTE_SYSTEM)
EndIf
Local $hFile = DllCall("kernel32.dll", "handle", "CreateFileW", "wstr", $sPath, "dword", $iAccess, "dword", $iShareMode, "ptr", 0, _
"dword", $iCreation, "dword", $iFlags, "ptr", 0)
If @error Or ($hFile[0] = Ptr(-1)) Then Return SetError(1, 0, 0)
Return $hFile[0]
EndFunc   ;==>_LFC_CreateFile
Func _LFC_FixPath(ByRef $sPath)
; fix possible forward /
$sPath = StringRegExpReplace($sPath, "/", "")
If Not StringInStr($sPath, ":") Then
; not a full path
; check for bare, relative, or UNC paths
If StringLeft($sPath, 1) = "" Then
; check for UNC path
If StringLeft($sPath, 2) <> "" Then
; not a UNC path
$sPath = "." & $sPath
; else, probably a UNC path, leave it alone
EndIf
Else
; check for bare or relative path
If StringLeft($sPath, 1) <> "." Then
; bare path
$sPath = "." & $sPath
; else, probably relative path, leave it alone
EndIf
EndIf
EndIf
EndFunc   ;==>_LFC_FixPath
Func _LFC_FixPaths(ByRef $sSrc, ByRef $sDest)
; remove trailing 's from source only
$sSrc = StringRegExpReplace($sSrc, "+$", "")
; check for relative paths in source and dest
_LFC_FixPath($sSrc)
_LFC_FixPath($sDest)
EndFunc   ;==>_LFC_FixPaths
#endregion INTERNAL FUNCTIONS


Func _FileListToArrayRec($sPath, $sExcludeFolderList = "", $sIncludeList = "*", $sExcludeList = "", $iReturnType = 0, $iReturnFormat = 0, $bRecursive = False)
Local $sRet = "", $sReturnFormat = ""
$sPath = StringRegExpReplace($sPath, "[/]+z", "") & ""
If Not FileExists($sPath) Then Return SetError(1, 1, "")
If $sIncludeList = "*" Then
$sIncludeList = ""
Else
If StringRegExp($sIncludeList, "[/ :> <|]|(?s)As*z") Then Return SetError(2, 2, "")
$sIncludeList = StringRegExpReplace(StringRegExpReplace($sIncludeList, "(s*;s*)+", ";"), "A;|;z", "")
$sIncludeList = StringRegExpReplace($sIncludeList, '[.$]', '[0]')
$sIncludeList = StringReplace(StringReplace($sIncludeList, "?", "."), "*", ".*?")
$sIncludeList = "(?i)A(" & StringReplace($sIncludeList, ";", "$|") & "$)"
EndIf
If Not ($iReturnType = 0 Or $iReturnType = 1 Or $iReturnType = 2) Then Return SetError(3, 3, "")
Local $sOrigPathLen = StringLen($sPath), $aQueue[64] = [1, $sPath], $iQMax = 63
While $aQueue[0]
$WorkFolder = $aQueue[$aQueue[0]]
$aQueue[0] -= 1
$search = FileFindFirstFile($WorkFolder & "*") ; get full list of current folder
If @error Then ContinueLoop
Switch $iReturnFormat
Case 1 ; relative path
$sReturnFormat = StringTrimLeft($WorkFolder, $sOrigPathLen)
Case 2 ; full path
$sReturnFormat = $WorkFolder
EndSwitch
While 1 ; process current folder
$file = FileFindNextFile($search)
If @error Then ExitLoop
If @extended Then ; Folder
If $bRecursive Then ; push folder onto queue
If $aQueue[0] = $iQMax Then
$iQMax += 128
ReDim $aQueue[$iQMax + 1]
EndIf
$aQueue[0] += 1
$aQueue[$aQueue[0]] = $WorkFolder & $file & ""
EndIf
If $iReturnType = 1 Then ContinueLoop
Else ; File
If $iReturnType = 2 Then ContinueLoop
EndIf
If $sIncludeList And Not StringRegExp($file, $sIncludeList) Then ContinueLoop
$sRet &= $sReturnFormat & $file & "|" ; append to output
WEnd
FileClose($search)
WEnd
If Not $sRet Then Return SetError(4, 4, "")
Return StringSplit(StringTrimRight($sRet, 1), "|")
EndFunc   ;==>_FileListToArrayRec

To test, create the $Source folder and put some stuff in there, couple files and small folders, and let it run, and everything will be fine. Then start to add some subfolders to the $Target and it will fail when it gets too long with @error 3.

Any further thoughts on how this can be overcome?

Thanks!

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

That would be great! I have been poking at this for the last 24 hours and haven't got a clue. All I have found is that newer versions of AutoIt are Unicode compliant, and the CreateFileW you call is the Unicode version, sooooooo......why won't a Unicode path work?????? gggrrrrrrrrrr!!!!! :D

Maybe something unrelated? A SciTE wrapper or compiler option to force Unicode that I haven't been finding? I even tried using #RequireAdmin out of exacerbation.....

Thanks a million

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

Well that didn't take long to figure out. I should probably open a ticket for this too. AutoIt's internal DirCreate is reporting success but is truncating the new directory to 255 characters. CreateFile obviously fails if the parent directory does not exist, so there ya go.

I'll need to replace the call(s) to DirCreate with the CreateDirectory API which works with UNC style extra long paths.

Share this post


Link to post
Share on other sites

Hmm, gonna take a little while longer. CreateDirectory does not work recursively, so I have to design that, including a function to crack all the possible path types.

Share this post


Link to post
Share on other sites

Sounds like a PITA, but yours will be the only routine out there that can do it, I tested Yashied's UDF and it bonked as well.

Thought - maybe your revised UDF should internally convert to UNC and just get it out of the way, Seems like you going through all the effort of making it CAPABLE of handling a Long UNC path doesn't make a lot of sense if it doesn't actually do it.... If you do or don't add that feature, it's still awesome that you are making progress.

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

Thought - maybe your revised UDF should internally convert to UNC and just get it out of the way

I'm deciding if I want to do that or not. My current thought is that Windows doesn't do that internally, so I shouldn't either. It should be the scripter's responsibility to check path length. Also, if I do it for directory creation, then I should do it for CreateFile as well, and that seems out of scope. Edited by wraithdu

Share this post


Link to post
Share on other sites

Ok, try this updated version of your example. Please review the new functions to see if anything jumps out at you as being incorrect.

#include <WinAPI.au3>
#include <Array.au3>

$iToRead = 8 * 1024 * 1024

$Source = @ScriptDir & "testdir"
$Target = @ScriptDir & "asiodfoiwoiawoiefioawejfaawoeifoafjjfejiwaoaiwefjoawiefaoiawoefjofijijewoaoawejfjaw"
DirRemove($Target, 1)

If StringRight($Source, 1) <> "" Then $Source &= ""
If StringRight($Target, 1) <> "" Then $Target &= ""
$SourceLength = StringLen($Source)
If $SourceLength == 3 Then $SourceLength -= 1; gotta have the array elements start with a backslash...

$FileList = _FileListToArrayRec($Source, "", "*", "", 1, 2, True)

If StringLeft($Source, 2) == "" Then
    $SourceUNCPrefix = "?UNC"
    $Source = StringTrimLeft($Source, 2)
Else
    $SourceUNCPrefix = "?"
EndIf

If StringLeft($Target, 2) == "" Then
    $TargetUNCPrefix = "?UNC"
    $Target = StringTrimLeft($Target, 2)
Else
    $TargetUNCPrefix = "?"
EndIf

_ArrayTrim($FileList, $SourceLength, 0, 1)

For $a = 1 To $FileList[0]
    $CopySource = $SourceUNCPrefix & $Source & $FileList[$a]
    $CopyTarget = $TargetUNCPrefix & $Target & $FileList[$a]

    ConsoleWrite(StringLen($CopySource) & " : " & StringLen($CopyTarget) & @CRLF)

    $Copy = _LargeFileCopy($CopySource, $CopyTarget, BitOR(1, 2))
    $Error = @error
    If $Copy == 1 Then
        $Message = "SUCCESS"
    Else
        $Message = $Error
    EndIf
    ConsoleWrite($CopySource & " -> " & $CopyTarget & " -> " & $Message & @CR)
Next

; #FUNCTION# ====================================================================================================
; Name...........:  _LargeFileCopy
; Description....:  Copy large files in such a way as to keep AutoIt GUIs responsive
; Syntax.........:  _LargeFileCopy($sSrc, $sDest, $iFlags)
; Parameters.....:  $sSrc      - Source file name
;                  $sDest     - Destination: may be a file name or directory
;                  $iFlags   - [Optional] Combine flags with BitOR
;                              | 1 - Overwrite existing file
;                              | 2 - Create destination directory structure
;                              | 4 - Flush the destination file buffer before returning
;                              | 8 - Verify source and destination are identical via bit by bit comparison
;                              |16 - Verify source and destination are identical via MD5 hash
;                              |32 - Verify source and destination file size only
;                              + If more than one verify flag is set, the smallest flag will take precedence
;
; Return values..:  Success  - 1
;                  Failure   - 0 and sets @error
;                              | 1 - Failed to open source file, or source was a directory
;                              | 2 - Destination file exists and overwrite flag not set
;                              | 3 - Failed to create destination file
;                              | 4 - Read error during copy
;                              | 5 - Write error during copy
;                              | 6 - Verify failed
; Author.........:  Erik Pilsits
; Modified.......:
; Remarks........: MODIFIED BY IAN MAXWELL FOR THIS SPECIFIC SCRIPT, DO NOT COPY THIS FOR RE-USE!!!!
; Related........:
; Link...........:
; Example........:
; ===============================================================================================================
Func _LargeFileCopy($sSrc, $sDest, $iFlags = 0)
    ; check / fix source and dest path syntax
    _LFC_FixPaths($sSrc, $sDest)

    ; open file for reading, fail if it doesn't exist or directory
    Local $hSrc = _LFC_CreateFile($sSrc, $GENERIC_READ, $FILE_SHARE_READ, $OPEN_EXISTING, 0)
    If Not $hSrc Then Return SetError(1, 0, 0)

    ; set option flags
    Local $fOverwrite = (BitAND($iFlags, 1) = 1)
    Local $fCreate = (BitAND($iFlags, 2) = 2)

    ; check destination
    _LFC_CheckDestination($sSrc, $sDest, $fOverwrite, $fCreate)
    If @error Then
        _WinAPI_CloseHandle($hSrc)
        Return SetError(2, 0, 0)
    EndIf

    ; create new file for writing, overwrite
    Local $hDest = _LFC_CreateFile($sDest, BitOR($GENERIC_READ, $GENERIC_WRITE), 0, $CREATE_ALWAYS, 0)
    If Not $hDest Then
        _WinAPI_CloseHandle($hSrc)
        Return SetError(3, 0, 0)
    EndIf

    ; check for 0 byte source file
    Local $iSize = _WinAPI_GetFileSizeEx($hSrc)
    If $iSize = 0 Then
        ; done, close handles and return success
        _WinAPI_CloseHandle($hDest)
        _WinAPI_CloseHandle($hSrc)
        Return 1
    EndIf

    ; perform copy
    Local $iRead, $iWritten, $iTotal = 0, $mSrc = 0, $iReadError = 0, $iWriteError = 0, $iVerifyError = 0
    ; allocate buffers
    Local $hBuffer = DllStructCreate("byte[" & $iToRead & "]")
    Local $pBuffer = DllStructGetPtr($hBuffer)

    Do
        If Not _WinAPI_ReadFile($hSrc, $pBuffer, $iToRead, $iRead) Then
            $iReadError = 1
            ExitLoop
        EndIf
        If $iRead = 0 Then ExitLoop ; end of file, edge case if file is an exact multiple of the buffer size
        If Not _WinAPI_WriteFile($hDest, $pBuffer, $iRead, $iWritten) Or ($iRead <> $iWritten) Then
            $iWriteError = 1
            ExitLoop
        EndIf
    Until $iRead < $iToRead

    _WinAPI_CloseHandle($hDest)
    _WinAPI_CloseHandle($hSrc)

    If $iReadError Then
        Return SetError(4, 0, 0)
    ElseIf $iWriteError Then
        Return SetError(5, 0, 0)
    Else
        Return 1
    EndIf
EndFunc   ;==>_LargeFileCopy

#region INTERNAL FUNCTIONS
Func _LFC_CheckDestination($sSrc, ByRef $sDest, $fOverwrite, $fCreate)
    If (StringRight($sDest, 1) = "") Or StringInStr(FileGetAttrib($sDest), "D") Then
        ; assume directory
        If $sSrc = "" Then
            ; raw copy, must provide a file name
            Return SetError(2)
        Else
            ; use source file name
            If StringRight($sDest, 1) <> "" Then $sDest &= "" ; add trailing 
            $sDest &= StringRegExpReplace($sSrc, ".*", "")
        EndIf
    EndIf
    ; check overwrite
    If (Not $fOverwrite) And FileExists($sDest) Then Return SetError(1)
    ; create destination parent directory
    ; if create flag not set and destination doesn't exist, CreateFile will fail
    If $fCreate Then _LFC_CreateDirectory(StringRegExpReplace($sDest, "^(.*).*?$", "${1}"))
EndFunc   ;==>_LFC_CheckDestination

Func _LFC_CrackPath($sPath)
    ; make sure path is terminated
    If StringRight($sPath, 1) <> "" Then $sPath &= ""
    ; get prefix
    Local $aRet = StringRegExp($sPath, "(?i)^?unc|?|", 1)
    If Not IsArray($aRet) Then
        $aRet = ""
    Else
        $aRet = $aRet[0]
    EndIf
    ; capture and remove the root
    If ($aRet = "") Or ($aRet = "?UNC") Then
        ; UNC network path
        $aRet = StringRegExp($sPath, "(?i)^(" & StringReplace(StringReplace($aRet, "", ""), "?", "?") & ".*?.*?)", 1)
    Else
        ; $aRet = "" or ? => local path
        Local $iTrim = StringLen($aRet) + 3
        Local $aRet[1] = [StringLeft($sPath, $iTrim)]
    EndIf
    Local $aPath = StringTrimLeft($sPath, StringLen($aRet[0]))
    ; check if path given was just a root
    If $aPath <> "" Then
        ; crack path, prepend  to get first segment
        $aPath = StringRegExp("" & $aPath, "(.*?)(?=)", 3)
        ReDim $aRet[UBound($aPath) + 1]
        For $i = 0 To UBound($aPath) - 1
            $aRet[$i + 1] = $aPath[$i]
        Next
    EndIf
    ;
    Return $aRet
EndFunc

Func _LFC_CreateDirectory($sPath)
    ; check path already exists
    If FileExists($sPath) Then
        ; check if it is a direcotry
        If StringInStr(FileGetAttrib($sPath), "D") Then
            Return 1
        Else
            Return SetError(1, 0, 0)
        EndIf
    EndIf
    ;
    Local $aPath = _LFC_CrackPath($sPath)
    ; make sure root exists
    Local $sSegment = $aPath[0]
    If Not FileExists($sSegment) Then Return SetError(2, 0, 0)
    Local $ret
    For $i = 1 To UBound($aPath) - 1
        $sSegment &= $aPath[$i] & ""
        If Not FileExists($sSegment) Then
            ; create sub-path
            $ret = DllCall("kernel32.dll", "bool", "CreateDirectoryW", "wstr", $sSegment, "ptr", 0)
            If @error Or Not $ret[0] Then Return SetError(3, 0, 0)
        EndIf
    Next
    Return 1
EndFunc

Func _LFC_CreateFile($sPath, $iAccess, $iShareMode, $iCreation, $iFlags)
    ; open the file with existing HIDDEN or SYSTEM attributes to avoid failure when using CREATE_ALWAYS
    If $iCreation = $CREATE_ALWAYS Then
        Local $sAttrib = FileGetAttrib($sPath)
        If StringInStr($sAttrib, "H") Then $iFlags = BitOR($iFlags, $FILE_ATTRIBUTE_HIDDEN)
        If StringInStr($sAttrib, "S") Then $iFlags = BitOR($iFlags, $FILE_ATTRIBUTE_SYSTEM)
    EndIf
    Local $hFile = DllCall("kernel32.dll", "handle", "CreateFileW", "wstr", $sPath, "dword", $iAccess, "dword", $iShareMode, "ptr", 0, _
            "dword", $iCreation, "dword", $iFlags, "ptr", 0)
    If @error Or ($hFile[0] = Ptr(-1)) Then
        ConsoleWrite("ERR: " & $hFile[0] & " : " & _WinAPI_GetLastErrorMessage())
        ConsoleWrite($sPath & @CRLF)
        Return SetError(1, 0, 0)
    EndIf
    Return $hFile[0]
EndFunc   ;==>_LFC_CreateFile

Func _LFC_FixPath(ByRef $sPath)
    ; fix possible forward /
    $sPath = StringRegExpReplace($sPath, "/", "")
    If Not StringInStr($sPath, ":") Then
        ; not a full path
        ; check for bare, relative, or UNC paths
        If StringLeft($sPath, 1) = "" Then
            ; check for UNC path
            If StringLeft($sPath, 2) <> "" Then
                ; not a UNC path
                $sPath = "." & $sPath
                ; else, probably a UNC path, leave it alone
            EndIf
        Else
            ; check for bare or relative path
            If StringLeft($sPath, 1) <> "." Then
                ; bare path
                $sPath = "." & $sPath
                ; else, probably relative path, leave it alone
            EndIf
        EndIf
    EndIf
EndFunc   ;==>_LFC_FixPath

Func _LFC_FixPaths(ByRef $sSrc, ByRef $sDest)
    ; remove trailing 's from source only
    $sSrc = StringRegExpReplace($sSrc, "+$", "")
    ; check for relative paths in source and dest
    _LFC_FixPath($sSrc)
    _LFC_FixPath($sDest)
EndFunc   ;==>_LFC_FixPaths
#endregion INTERNAL FUNCTIONS

Func _FileListToArrayRec($sPath, $sExcludeFolderList = "", $sIncludeList = "*", $sExcludeList = "", $iReturnType = 0, $iReturnFormat = 0, $bRecursive = False)
    Local $sRet = "", $sReturnFormat = ""
    $sPath = StringRegExpReplace($sPath, "[/]+z", "") & ""
    If Not FileExists($sPath) Then Return SetError(1, 1, "")
    If $sIncludeList = "*" Then
        $sIncludeList = ""
    Else
        If StringRegExp($sIncludeList, "[/ :> <|]|(?s)As*z") Then Return SetError(2, 2, "")
        $sIncludeList = StringRegExpReplace(StringRegExpReplace($sIncludeList, "(s*;s*)+", ";"), "A;|;z", "")
        $sIncludeList = StringRegExpReplace($sIncludeList, '[.$]', '[0]')
        $sIncludeList = StringReplace(StringReplace($sIncludeList, "?", "."), "*", ".*?")
        $sIncludeList = "(?i)A(" & StringReplace($sIncludeList, ";", "$|") & "$)"
    EndIf
    If Not ($iReturnType = 0 Or $iReturnType = 1 Or $iReturnType = 2) Then Return SetError(3, 3, "")
    Local $sOrigPathLen = StringLen($sPath), $aQueue[64] = [1, $sPath], $iQMax = 63
    While $aQueue[0]
        $WorkFolder = $aQueue[$aQueue[0]]
        $aQueue[0] -= 1
        $search = FileFindFirstFile($WorkFolder & "*") ; get full list of current folder
        If @error Then ContinueLoop
        Switch $iReturnFormat
            Case 1 ; relative path
                $sReturnFormat = StringTrimLeft($WorkFolder, $sOrigPathLen)
            Case 2 ; full path
                $sReturnFormat = $WorkFolder
        EndSwitch
        While 1 ; process current folder
            $file = FileFindNextFile($search)
            If @error Then ExitLoop
            If @extended Then ; Folder
                If $bRecursive Then ; push folder onto queue
                    If $aQueue[0] = $iQMax Then
                        $iQMax += 128
                        ReDim $aQueue[$iQMax + 1]
                    EndIf
                    $aQueue[0] += 1
                    $aQueue[$aQueue[0]] = $WorkFolder & $file & ""
                EndIf
                If $iReturnType = 1 Then ContinueLoop
            Else ; File
                If $iReturnType = 2 Then ContinueLoop
            EndIf
            If $sIncludeList And Not StringRegExp($file, $sIncludeList) Then ContinueLoop
            $sRet &= $sReturnFormat & $file & "|" ; append to output
        WEnd
        FileClose($search)
    WEnd
    If Not $sRet Then Return SetError(4, 4, "")
    Return StringSplit(StringTrimRight($sRet, 1), "|")
EndFunc   ;==>_FileListToArrayRec

Edit:

I've created ticket #2054 for this issue.

Edited by wraithdu

Share this post


Link to post
Share on other sites

NICE MAN!! :D

I just ran a test and got a path length of 452 and it was all good! haha, right on! And the new FileListToArrayRec that I have been messing around with did NOT find any files in that huge path, but adding the UNC prefix made it able to find the files, so no horrible mess appears to be needed for that to be happy!

GOOD JOB! :oops:

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

thanks for a great UDF.

a quick noob question how do i make a progressbar-copy.. i have tried a few thing but i do not totaly under stand the function paramter calls

that i have tried eksample:

#include <Process.au3>

#include <_Largefilecopy.au3>

#include <Array.au3>

Local $Test, $FileCopy, $1, $2,$iWritten, $iTotal,$vVar

; $iTotal=FileGetSize("E:ZS7-168ASM4-KP0_wo3.1.0.0.WIM")

;$iWritten=FileGetSize("E:MSIZS7-168ASM4-KP0_wo3.1.0.0.WIM")

$FileCopy=_LargeFileCopy("E:ZS7-168ASM4-KP0_wo3.1.0.0.WIM", "E:MSIZS7-168ASM4-KP0_wo3.1.0.0.WIM", 1,60000,"",test($iWritten, $iTotal))

ConsoleWrite("1 : "&$iWritten&@CR)

ConsoleWrite("2 : "&$iTotal&@CR)

If $FileCopy=0 Then MsgBox(0,"Copy","Fail to Copy")

If $FileCopy=1 Then MsgBox(0,"Copy","Copy'ed Your file")

Func test($iWritten, $iTotal)

$iTotal=FileGetSize("E:ZS7-168ASM4-KP0_wo3.1.0.0.WIM")

$iWritten=FileGetSize("E:MSIZS7-168ASM4-KP0_wo3.1.0.0.WIM")

ConsoleWrite("1 : "&$iWritten&@CR)

ConsoleWrite("2 : "&$iTotal&@CR)

EndFunc

hope somebody can help. Thanks in advance

Share this post


Link to post
Share on other sites

The progress function is automatically called with the 3 parameters described in the function header. You need to change your test function to take all three parameters, even if you don't intend to use them. Then in your call to _LargeFileCopy pass only the name of your progress function as a string, ie "test".

Share this post


Link to post
Share on other sites

Thanks for this Extradinary UDF... ;) i have a quick question. is there a way to copy the entire folder by using this function? i am following the below task where it fails everytime.. any reason?

#include<Array.au3>
#include <_LargeFileCopy.au3>

Global $coppst[2]
$coppst[0] = "c:exchange"
$coppst[1] = "C:UsersQ03200AppDataLocalMicrosoftOutlook"

For $s = 0 To UBound($coppst)-1  Step +1
;$m = DirCopy($coppst[$s], "C:backup"&StringTrimLeft($coppst[$s],2),1)

; MsgBox(0, "", $coppst[$s]&"*.*       "& "C:backup" & StringTrimLeft($coppst[$s], 2)&"*.psst")
$m = _LargeFileCopy($coppst[$s]&"*.*", "C:backup" & StringTrimLeft($coppst[$s], 2)&"*.*", BitOR(2,8))
MsgBox(0,"",$m)
Next

Thank you,Regards,[font="Garamond"][size="4"]K.Syed Ibrahim.[/size][/font]

Share this post


Link to post
Share on other sites

Quick answer... no, this UDF does not natively support recursive copying, and it's not going to.

It also does not support use of wildcards. You'll have to implement your own search/loop/copy algorithms.

Edited by wraithdu

Share this post


Link to post
Share on other sites

Quick answer... no, this UDF does not natively support recursive copying, and it's not going to.

It also does not support use of wildcards. You'll have to implement your own search/loop/copy algorithms.

Thanks Wraithdu! may i know what is the methodology we are using for the copy process? because what i could feel is when i use this function to copy file this is bit faster than normal process. can you give me an overview of the technique you have used in this function? how it is copying in this speed?


Thank you,Regards,[font="Garamond"][size="4"]K.Syed Ibrahim.[/size][/font]

Share this post


Link to post
Share on other sites

Syed23

If you look over the code in this UDF you will see that wraithdu is using WinAPI calls. There are more lines of code in using this method since wraithdu's UDF has to specify the position in the file to open, as well as position to write in the destination, additional error checking, handling UNC paths, etc. but his code is proof that well-written and optimized code can be far faster than fewer lines of poor code.

hmm, maybe it is just easier to say that using Windows internal API calls will result in faster code :)

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

Just curious, I have used this function in my GUI and it works perfect, but in some cases when copy bit huge size of file the GUI shows not responding and the marquee progress bar freezes.. could you please help me to over come this? if you have any suggestion how to trouble shoot could you please tell me that so that i can give you the exact scenario where i am facing that problem. I am using this function in Windows 7 64 bit machines.

Thanks in advance!


Thank you,Regards,[font="Garamond"][size="4"]K.Syed Ibrahim.[/size][/font]

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By wraithdu
      The project grew out of my _LargeFileCopy UDF. It is an XCOPY replacement that improves in a few areas that were important to me. First, file verification can be done via bit-by-bit comparison, file hash, or simple size comparison. Second, it supports copying NTFS file/directory compression, encryption, and ACLs. It also has an optional simple progress display. All source files are in the archive. You can ignore the compilation warnings if you use Obfuscator.

      This is a commandline utility, and provides help as such. All you need is 'a3copy /?' and you should be good to go. Enjoy!

      Update: 10/8/12
      - Some fixes in _LargeFileCopy (see LFC thread)
      - Use DllCall versions of FindFirst/NextFile

      Update: 10/4/12
      - Added ability to specify multiple sources to be consolidated into the destination
      - Added flag to flatten the output, ie during a directory copy, all files are copied to the top level of the destination
      - Added check for cyclic copy
      - Built with new and improved _LargeFileCopy, including code cleanup and many optimizations

      Update: 9/22/10
      - Added check for destination same as source

      Update: 9/20/10
      - Added --version switch
      - Changes to _LargeFileCopy UDF, updated hashing functions
      - Updated user input function

      Update: 8/26/10
      - Added Aliases: custom INI defined long options
      - Added return codes, useful for scripting
      - Fix for verification of 0 byte files (fix is in _LargeFileCopy.au3)

      Update: 8/22/10
      - Added support for EFS encryption
      - Small bug fixes and code cleanup

      Help:

      A3COPY 1.0.2.3 Copies file and directory trees. A3COPY source [source [source] ...] destination [options] source Specifies the file(s) to copy. destination Specifies the location and/or name of new files. /C Copy file/directory compression state (NTFS only). /E Copy also empty directories. /F Flush file buffer after each copy. /G Show copy progress of current file. /H Copy hidden or system files. /J Copy using unbuffered I/O. /K Copy file/directory attributes. /L Flatten the output into one destination directory. /N Copy file/directory encryption state (NTFS only). /O Copy file/directory ownership and ACL information (NTFS only). Requires admin rights. /P Print full paths of source and destination. /Q Don't print file names. /R Allow operations on read-only files/directories. /S Recurse into subdirectories. /T Set destination filetimes same as source. NOTE: Only one verify method will be used. The final option flag will take precedence. /V or /V1 Verify the integrity of the copied files using a bit by bit comparison. /V2 Verify the integrity of the copied files using a MD5 hash. /V3 Verify the integrity of the copied files using file size only. Not valid for unbuffered copy operations. /W Apply wildcard to folder search when using /S. /Y Overwrite destination without prompt. /Z Never overwrite destination, suppress prompt. /? This usage information. Long options: --filespec Show source/destination filespec help. --copy Same as /EHRSTVY. --backup Same as /EHKRSTVY. --clone Same as /CEHKNORSTVY. --version Print version information. --help Same as /?. Aliases: Aliases may be set up in 'a3copy.ini' in the same folder as A3COPY. The aliases are invoked in the same way as the built-in long options. Example: a3copy src dest --mycopy [aliases] mycopy=EJSV cpyroot=CKNOV Return codes: 0: no errors 1: source not found 2: invalid file name / wildcard pattern 3: one or more errors copying files
      Filespec help:

      A3COPY 1.0.2.3 Copies file and directory trees. Filespec for source/destination: If source is a single file, then destination may be a file or directory. If destination ends in a '' then a directory is assumed. If not, then a single destination file is assumed unless a direcotry of the same name already exists. If destination is a directory, then the source file name will be used for the destination file. If source is a directory or wildcard filespec, then destination is assumed to be a directory. Wildcard filespec follows standard Windows usage of '*' and '?'. When the source is a directory, if filespec does not include a wildcard then it is assumed the directory is to be copied, and the top level directory attributes will be duplicated if the appropriate switches are set (/C, /K, /N, or /O). If the source filespec includes a wildcard, then it is assumed directory contents are to be copied, and the top level directory attributes will not be duplicated. a3copy.zip
×
×
  • Create New...