Jump to content
Sign in to follow this  
wraithdu

_LargeFileCopy UDF

Recommended Posts

Not a stupid question, but your script is not right. Your parameters to GUICtrlCreateAvi are wrong (see the help file). And you need to use BitOR to combine styles, not BitAND. I did notice though that AutoIt seems to have repainting issues on the AVI if you minimize the window, the AVI no longer seems to play until it loops and starts over again (but this has nothing to do with my UDF). Here's a fixed version of your script with the addition of a filename label and a few fixes, to show that it's working.

#AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6
#include <_LargeFileCopy.au3>
#include "RecFileListToArray.au3"
#include"Array.au3"
#include <GUIConstantsEx.au3>
#include <AVIConstants.au3>

Global $msg, $timer, $ret
Global $src, $SrcBut, $SrcLab, $Srcres
Global $Dest, $DestLab, $DestBut, $DestRes, $GetList, $go, $ani1, $label
_Main()

Func _Main()

GUICreate("File Copy", 550, 300)
$SrcLab = GUICtrlCreateLabel("Select Source Folder", 5, 10, 110, 40)
$src = GUICtrlCreateInput("", 115, 5, 400, 20)
$SrcBut = GUICtrlCreateButton("...", 520, 5, 25, 20)
$DestLab = GUICtrlCreateLabel("Select Destination Folder", 5, 60, 110, 40)
$Dest = GUICtrlCreateInput("", 115, 55, 400, 20)
$DestBut = GUICtrlCreateButton("...", 520, 55, 25, 20)
$go = GUICtrlCreateButton("Copy Large File", 200, 150, 150, 50)
; ACS_AUTOPLAY is not doing anything, playback stops as soon as you hide the control
$ani1 = GUICtrlCreateAvi(@SystemDir & "shell32.dll", 165, 10, 230, 275, 75, BitOR($ACS_CENTER, $ACS_TRANSPARENT))
GUICtrlSetState(-1, $GUI_HIDE)
$label = GUICtrlCreateLabel("", 10, 210, 540, 20)

GUISetState()

While 1
$msg = GUIGetMsg()

If $msg = $SrcBut Then
$Srcres = FileSelectFolder("Select Folder", @DesktopDir, 1)
GUICtrlSetData($src, $Srcres)

EndIf

If $msg = $GUI_EVENT_CLOSE Then ExitLoop

If $msg = $DestBut Then
$DestRes = FileSelectFolder("Select Folder", @DesktopDir, 1)
GUICtrlSetData($Dest, $DestRes)

EndIf

If $msg = $go Then
If GUICtrlRead($src) = "" Then
MsgBox(48, "Source Path - Warning", "Source path can not be blank")
ElseIf GUICtrlRead($Dest) = "" Then
MsgBox(48, "Destination Path - Warning", "Destination path can not be blank")
ElseIf GUICtrlRead($src) <> "" And GUICtrlRead($Dest) <> "" Then
GUICtrlSetState($ani1, $GUI_SHOW)
GUICtrlSetState($ani1, 1) ; apparently this is needed, as $GUI_HIDE stops playback
$timer = TimerInit()
$GetList = _RecFileListToArray(GUICtrlRead($src) & "", "*", 0, 1, 1)

;Exit
If IsArray($GetList) Then
For $i = 0 To UBound($GetList) - 1
GUICtrlSetData($label, $GetList[$i])
$ret = _LargeFileCopy(GUICtrlRead($src) & "" & $GetList[$i], GUICtrlRead($Dest) & "" & $GetList[$i], BitOR(1, 2, 8))
Next
EndIf
GUICtrlSetData($label, "")
GUICtrlSetState($ani1, $GUI_HIDE)
EndIf
EndIf
WEnd
EndFunc ;==>_Main

I am sorry for bothering you.. still that hangs when the copy process goes in background :(

Thank you,Regards,[font="Garamond"][size="4"]K.Syed Ibrahim.[/size][/font]

Share this post


Link to post
Share on other sites

You need to be more specific.

"still that hangs" - What hangs? The AVI issue I mentioned is an AutoIt problem and not related to the UDF.

"goes in background" - What goes where, how and when?

Unless you get the classic 'this application is not responding' dialog with the offer to forcefully kill the process, then whatever 'hanging' you're experiencing is not related to the UDF. It could even be your AV program actively scanning a large file which will hang up even my UDF if it's aggressive enough.

Share this post


Link to post
Share on other sites

First off- thanks for this great UDF.

I've tested and used it only with file copy to USB sticks, here are some results you may find interesting.

Legend:

LFC 0.25, XP , 379.41

LFC- LargeFileCopy buffered

0.25- buffer used for LFC- _LargeFileCopy($src, $dest, BitOR(1, 2, 8, 128, 512), $buffer * 1024 * 1024, $CALG_MD5)

379.41- seconds to finish

XP- I386 folder from XP SP3 Setup CD, being copied from the internal hard disk, 5878 small files, 375MB total.

ISO- two ISO files, 414MB and 495MB

FileCopy- AutoIt internal FileCopy function

WFileCopy- Windows

xcopy - Windows xcopy with /q and /h switches. Xcopy and LFC unbuffered took a lot more time so weren't tested at all.

YCopy- Yashied's

Tested in FAT16, FAT32, NTFS with a fast USB 2.0 stick- A and quite slow one-B, under XP SP3 and 7 x64, tests were repeated several times with format before each test, default cluster size for 4GB partition, AV turned off.

A, under XP

===============NTFS=============

LFC 0.25, XP , 124.96

LFC 0.25, ISO , 57.11

LFC 0.5, XP , 154.68

LFC 0.5, ISO , 173.81

LFC 1, XP , 143.39

LFC 1, ISO , 120.71

LFC 2, XP , 133.38

LFC 2, ISO , 87.61

LFC 4, XP , 133.78

LFC 4, ISO , 71.54

LFC 8, XP , 129.07

LFC 8, ISO , 60.94

LFC 16, XP , 127.24

LFC 16, ISO , 56.5

xcopy, XP , 135.18

xcopy,ISO , 46.31

FileCopy , XP , 128.31

FileCopy , ISO , 46.33

WFileCopy , XP , 0.32

WFileCopy , ISO , 46.32

YCopy , XP , 154.1

YCopy , ISO , 44.28

===============Fat 16============

LFC 0.25, XP , 160.53

LFC 0.25, ISO , 58.06

LFC 0.5, XP , 159.38

LFC 0.5, ISO , 55.52

LFC 1, XP , 159.9

LFC 1, ISO , 54.06

LFC 2, XP , 157.06

LFC 2, ISO , 52.34

LFC 4, XP , 154.45

LFC 4, ISO , 52.76

LFC 8, XP , 159.14

LFC 8, ISO , 52.79

LFC 16, XP , 150.89

LFC 16, ISO , 52.88

xcopy, XP , 185.33

xcopy,ISO , 51.21

FileCopy , XP , 155.35

FileCopy , ISO , 50.03

WFileCopy , XP , 0.05

WFileCopy , ISO , 51.33

YCopy , XP , 178.05

YCopy , ISO , 50.26

===============Fat 32============

LFC 0.25, XP , 177.03

LFC 0.25, ISO , 59.61

LFC 0.5, XP , 173.95

LFC 0.5, ISO , 55.82

LFC 1, XP , 167.27

LFC 1, ISO , 54.53

LFC 2, XP , 168.64

LFC 2, ISO , 54.25

LFC 4, XP , 172.99

LFC 4, ISO , 51.37

LFC 8, XP , 152.86

LFC 8, ISO , 52.84

LFC 16, XP , 170.97

LFC 16, ISO , 53.84

xcopy, XP , 206.25

xcopy,ISO , 50.92

FileCopy , XP , 169.8

FileCopy , ISO , 51.15

WFileCopy , XP , 0.04

WFileCopy , ISO , 51.62

YCopy , XP , 181.01

YCopy , ISO , 50.57

A, under Windows 7

===============NTFS=============

LFC 0.25, XP , 379.41

LFC 0.25, ISO , 49.57

LFC 0.5, XP , 401.51

LFC 0.5, ISO , 48.83

LFC 1, XP , 405.99

LFC 1, ISO , 48.59

LFC 2, XP , 400.09

LFC 2, ISO , 47.65

LFC 4, XP , 399.18

LFC 4, ISO , 49.7

LFC 8, XP , 397.1

LFC 8, ISO , 48.07

LFC 16, XP , 397.45

LFC 16, ISO , 48.28

xcopy, XP , 569.33

xcopy,ISO , 49.7

FileCopy , XP , 377.08

FileCopy , ISO , 49.91

WFileCopy , XP , 404.86

WFileCopy , ISO , 50.28

YCopy , XP , 406.69

YCopy , ISO , 50.18

===============Fat 16============

LFC 0.25, XP , 117.95

LFC 0.25, ISO , 44.34

LFC 0.5, XP , 156.83

LFC 0.5, ISO , 42.33

LFC 1, XP , 156.99

LFC 1, ISO , 41.47

LFC 2, XP , 158.42

LFC 2, ISO , 41.35

LFC 4, XP , 157

LFC 4, ISO , 41.21

LFC 8, XP , 157.85

LFC 8, ISO , 41.25

LFC 16, XP , 160.97

LFC 16, ISO , 61.21

xcopy, XP , 178.43

xcopy,ISO , 41.35

FileCopy , XP , 149.71

FileCopy , ISO , 40.32

WFileCopy , XP , 163.25

WFileCopy , ISO , 40.59

YCopy , XP , 148.88

YCopy , ISO , 40.5

===============Fat 32============

LFC 0.25, XP , 114.1

LFC 0.25, ISO , 45.84

LFC 0.5, XP , 152.42

LFC 0.5, ISO , 42.75

LFC 1, XP , 153.22

LFC 1, ISO , 42.01

LFC 2, XP , 169.16

LFC 2, ISO , 41.85

LFC 4, XP , 153.72

LFC 4, ISO , 41.74

LFC 8, XP , 152.91

LFC 8, ISO , 41.83

LFC 16, XP , 156.06

LFC 16, ISO , 60.91

xcopy, XP , 173.69

xcopy,ISO , 43.07

FileCopy , XP , 142.32

FileCopy , ISO , 40.94

WFileCopy , XP , 115.86

WFileCopy , ISO , 41.3

YCopy , XP , 143.08

YCopy , ISO , 41.03

B, under Windows 7

===============NTFS=============

LFC 0.25, XP , 747.8

LFC 0.25, ISO , 165.11

LFC 0.5, XP , 705.83

LFC 0.5, ISO , 163.25

LFC 1, XP , 724.65

LFC 1, ISO , 163.6

LFC 2, XP , 668.09

LFC 2, ISO , 164.18

LFC 4, XP , 685.93

LFC 4, ISO , 163.48

LFC 8, XP , 657.66

LFC 8, ISO , 164.7

LFC 16, XP , 686.28

LFC 16, ISO , 162.81

xcopy, XP , 715.94

xcopy,ISO , 172.56

FileCopy , XP , 626.53

FileCopy , ISO , 162.05

WFileCopy , XP , 712.84

WFileCopy , ISO , 159.22

YCopy , XP , 666.32

YCopy , ISO , 160.87

===============Fat 16============

LFC 0.25, XP , 742.78

LFC 0.25, ISO , 170.15

LFC 0.5, XP , 791.88

LFC 0.5, ISO , 160.17

LFC 1, XP , 754.53

LFC 1, ISO , 156.01

LFC 2, XP , 778.32

LFC 2, ISO , 154.28

LFC 4, XP , 784.73

LFC 4, ISO , 153.31

LFC 8, XP , 778.35

LFC 8, ISO , 153.1

LFC 16, XP , 784.75

LFC 16, ISO , 168.92

xcopy, XP , 832.92

xcopy,ISO , 152.01

FileCopy , XP , 778.39

FileCopy , ISO , 151.43

WFileCopy , XP , 838.63

WFileCopy , ISO , 152.23

YCopy , XP , 798.12

YCopy , ISO , 151.12

===============Fat 32============

LFC 0.25, XP , 814.16

LFC 0.25, ISO , 178.34

LFC 0.5, XP , 862.87

LFC 0.5, ISO , 170.46

LFC 1, XP , 846.75

LFC 1, ISO , 167.16

LFC 2, XP , 852.09

LFC 2, ISO , 164.55

LFC 4, XP , 866.86

LFC 4, ISO , 166.16

LFC 8, XP , 862.91

LFC 8, ISO , 164.63

LFC 16, XP , 833.4

LFC 16, ISO , 186.34

xcopy, XP , 917.72

xcopy,ISO , 158.97

FileCopy , XP , 851.13

FileCopy , ISO , 162.71

WFileCopy , XP , 859.61

WFileCopy , ISO , 162.83

YCopy , XP , 890.42

YCopy , ISO , 163

Did not have patience to repeat the test with the slow stick under XP, but resluts were quite informative for me- LFC flies with small files when the proper buffer is used.

For large files, above say 100-200MB I couldn't get it not the freeze the GUI- it does not freeze during copying, but once finished, until windows file system buffer/cache is emptied, it gets frozen. For files above, say 500MB it's quite noticable, can't recall if optimize for performace/quick removal policy helped or not, but wasn't a path to take anyway. Ended up using copy.exe and ProcessGetStats, like

As a side note- _LFC_GetDiskClusterSize function seems to need a correction, return should be $aRet[3]*$aRet[2], BytesPerSector*SectorsPerCluster, unless you had other intentions.

Edited by ilko

Share this post


Link to post
Share on other sites

Hi ,

Thank you so much for this wonderful script. with help of this tool i created a tool which made my job so easy. But i have a doubt when i copy the file from source location to destination location date is getting changed. could you please help me how to avoid changing the date? sorry for bothering you.


Thank you,Regards,[font="Garamond"][size="4"]K.Syed Ibrahim.[/size][/font]

Share this post


Link to post
Share on other sites

@Sayed23

Use the appropriate flag(s):

; #FUNCTION# ====================================================================================================
; Name...........: _LargeFileCopy
; Description....: Copy large files in such a way as to keep AutoIt GUIs responsive
; Syntax.........: _LargeFileCopy($sSrc, $sDest[, $iFlags = 0[, $iToRead = 2097152[, $iAlg = $CALG_MD5[, $sFunction = ""[, $vUserVar = Default]]]]])
; Parameters.....: $sSrc     - Source file name
;                $sDest  - Destination: may be a file name or directory
;                $iFlags     - [Optional] Combine flags with BitOR
;                            | 1 - Overwrite existing file
;                            | 2 - Create destination directory structure
;                            | 4 - Flush the destination file buffer before returning
;                            | 8 - Verify source and destination are identical via bit by bit comparison
;                            | 16 - Verify source and destination are identical via MD5 hash
;                            | 32 - Verify source and destination file size only
;                            | 64 - Copy source file attributes (NOT including Compression or Encryption)
;                            | 128 - Copy source file Creation time
;                            | 256 - Copy source file Last Accessed time
;                            | 512 - Copy source file Modified time
;                            |1024 - Copy source file Security Descriptors and Ownership
;                            |2048 - Copy source compression state
;                            |4096 - Copy source encryption state

Share this post


Link to post
Share on other sites

Thank you @iLko.. will be the below syntax correct?

$mds = _LargeFileCopy(GUICtrlRead($DrivePath) & $Docs & "\" & $DocsAr[$i], "C:\Users\" & @UserName & "\Documents" & "\" & $DocsAr[$i], BitAND(BitOR(1, 2, 8),(128,256,512)))

Thank you,Regards,[font="Garamond"][size="4"]K.Syed Ibrahim.[/size][/font]

Share this post


Link to post
Share on other sites

As a side note- _LFC_GetDiskClusterSize function seems to need a correction, return should be $aRet[3]*$aRet[2], BytesPerSector*SectorsPerCluster, unless you had other intentions.

I actually caught this bug, and I thought I uploaded a fixed version. My local sources are fixed, I had moved some more functions into FileEx, so not sure what happened. Anyway, I'll update shortly.

Share this post


Link to post
Share on other sites

I have used _LargeFileCopy before and it worked great! Thanks for the script! However with a new script I'm having problems.

Wraithdu, (or anyone!) do you know why FileCopy and _LargeFileCopy fails? DirCopy does work.

I get a Zero (0) returned when I run my code using either FileCopy or _LargeFileCopy

_LargeFileCopy.au3 is in the C:\Program Files (x86)\AutoIt3\Include directory.

My actual script does a lot more than this, so I toned it down to only do the copy. I used _LargeFileCopy in a script last year and it worked perfectly.

#include <MsgBoxConstants.au3>
#include <FileConstants.au3>
#include <_LargeFileCopy.au3>
#include <File.au3>


Local $sSourceTestDir = "C:\Validation\214" ;Source is 3 Dirs Deep, over 12,000 files, 400MB
Local $sDestTestDir = "C:\FileCopyTest\214" ;Empty Dir

MsgBox($MB_OK,"","Source Dir is "&$sSourceTestDir&" and Dest Dir is "&$sDestTestDir&". Copy Files Next!",5)
;$iRes = FileCopy($sSourceTestDir,$sDestTestDir,$FC_OVERWRITE + $FC_CREATEPATH)
;$iRes = _LargeFileCopy($sSourceTestDir,$sDestTestDir,1)
$iRes = DirCopy($sSourceTestDir, $sDestTestDir,$FC_OVERWRITE)
MsgBox($MB_OK,"","$iRes is "&$iRes,10)

I have created the Sink directories C:\FileCopyTest\214 while trying to get _LargeFileCopy working. However DirCopy creates these if they are not there.

I have tried adding backslashes ( C:\Validation\214\ ), asterisks, ( C:\Validation\214\* ) & ( C:\Validation\214\*.* ) to the code.

I have made both the Source and Sink directories Shared, Full Control for Everyone and turned off Read Only attributes.

I would prefer to use the _LargeFileCopy because of the progress bar showing the darned script is actually doing something instead of waiting, not knowing if the script is stuck or working.

I have tried this on multiple systems; Windows 10, Windows 8.1 and get the same results. I'm pulling out my hair trying to figure out what I'm doing wrong!

Any help is certainly appreciated!

JibsMan

Share this post


Link to post
Share on other sites

Back from the deads ...

I've the same issue here, but it's something I think I can manage.

I've only two regrets :

- it miss a post copy control to see if it is about to rewrite the exact same file

- it miss lot of return (for example, the MD5 key, or the detailled output when there's an error while copying, etc.)

wraithdu , have you ever considered to extend the functionalities of this tool ? 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By wraithdu
      The project grew out of my _LargeFileCopy UDF. It is an XCOPY replacement that improves in a few areas that were important to me. First, file verification can be done via bit-by-bit comparison, file hash, or simple size comparison. Second, it supports copying NTFS file/directory compression, encryption, and ACLs. It also has an optional simple progress display. All source files are in the archive. You can ignore the compilation warnings if you use Obfuscator.

      This is a commandline utility, and provides help as such. All you need is 'a3copy /?' and you should be good to go. Enjoy!

      Update: 10/8/12
      - Some fixes in _LargeFileCopy (see LFC thread)
      - Use DllCall versions of FindFirst/NextFile

      Update: 10/4/12
      - Added ability to specify multiple sources to be consolidated into the destination
      - Added flag to flatten the output, ie during a directory copy, all files are copied to the top level of the destination
      - Added check for cyclic copy
      - Built with new and improved _LargeFileCopy, including code cleanup and many optimizations

      Update: 9/22/10
      - Added check for destination same as source

      Update: 9/20/10
      - Added --version switch
      - Changes to _LargeFileCopy UDF, updated hashing functions
      - Updated user input function

      Update: 8/26/10
      - Added Aliases: custom INI defined long options
      - Added return codes, useful for scripting
      - Fix for verification of 0 byte files (fix is in _LargeFileCopy.au3)

      Update: 8/22/10
      - Added support for EFS encryption
      - Small bug fixes and code cleanup

      Help:

      A3COPY 1.0.2.3 Copies file and directory trees. A3COPY source [source [source] ...] destination [options] source Specifies the file(s) to copy. destination Specifies the location and/or name of new files. /C Copy file/directory compression state (NTFS only). /E Copy also empty directories. /F Flush file buffer after each copy. /G Show copy progress of current file. /H Copy hidden or system files. /J Copy using unbuffered I/O. /K Copy file/directory attributes. /L Flatten the output into one destination directory. /N Copy file/directory encryption state (NTFS only). /O Copy file/directory ownership and ACL information (NTFS only). Requires admin rights. /P Print full paths of source and destination. /Q Don't print file names. /R Allow operations on read-only files/directories. /S Recurse into subdirectories. /T Set destination filetimes same as source. NOTE: Only one verify method will be used. The final option flag will take precedence. /V or /V1 Verify the integrity of the copied files using a bit by bit comparison. /V2 Verify the integrity of the copied files using a MD5 hash. /V3 Verify the integrity of the copied files using file size only. Not valid for unbuffered copy operations. /W Apply wildcard to folder search when using /S. /Y Overwrite destination without prompt. /Z Never overwrite destination, suppress prompt. /? This usage information. Long options: --filespec Show source/destination filespec help. --copy Same as /EHRSTVY. --backup Same as /EHKRSTVY. --clone Same as /CEHKNORSTVY. --version Print version information. --help Same as /?. Aliases: Aliases may be set up in 'a3copy.ini' in the same folder as A3COPY. The aliases are invoked in the same way as the built-in long options. Example: a3copy src dest --mycopy [aliases] mycopy=EJSV cpyroot=CKNOV Return codes: 0: no errors 1: source not found 2: invalid file name / wildcard pattern 3: one or more errors copying files
      Filespec help:

      A3COPY 1.0.2.3 Copies file and directory trees. Filespec for source/destination: If source is a single file, then destination may be a file or directory. If destination ends in a '' then a directory is assumed. If not, then a single destination file is assumed unless a direcotry of the same name already exists. If destination is a directory, then the source file name will be used for the destination file. If source is a directory or wildcard filespec, then destination is assumed to be a directory. Wildcard filespec follows standard Windows usage of '*' and '?'. When the source is a directory, if filespec does not include a wildcard then it is assumed the directory is to be copied, and the top level directory attributes will be duplicated if the appropriate switches are set (/C, /K, /N, or /O). If the source filespec includes a wildcard, then it is assumed directory contents are to be copied, and the top level directory attributes will not be duplicated. a3copy.zip
×
×
  • Create New...