Jump to content
wraithdu

_LargeFileCopy UDF

Recommended Posts

wraithdu

Several people have had problems with their GUIs freezing when they try to copy large files using FileCopy(). This is because FileCopy() is a blocking function, and the GUI stops responding to messages from the OS during the operation. If that goes on long enough, the OS thinks it is hung and you get the (Not Responding) thing going on.

Here's a function that copies large files using Windows API calls in 8MB 2 MB chunks. This keeps the GUI alive and responsive. I've added additional copy verification functionality as well (disabled by default).

Take a look at my A3COPY project for a great usage example of this UDF.

Update: 1/31/13

- Moved a few more functions into _FileEx.

- Fixed a bug in GetDiskClusterSize function.

Update: 10/8/12

- Fixed bug due to missing RtlCopyMemory and RtlCompareMemory exports in x86 kernel32.dll. Now uses RtlMoveMemory and RtlCompareMemory from ntdll.dll (sorry).

Update: 10/4/12

Read NOTES at top of UDF

- Reduced default buffer size to 2 MB

- Added flags for copying source file attributes, all three file times individually, ACL security descriptors and ownership, and NTFS specific compression and encryption

- Optimized buffer creation

- Enhanced path handling to fully support UNC and network conventions ( C:... , ?C:... , srvshare... , ?UNCsrvshare... )

- Removed C runtime functions (msvcrt.dll) in favor of native kernel32 functions

- Code cleanup and many optimizations in LFC and the accompanying _FileEx UDF

Update: 4/15/11

- Added additional verification methods - bit by bit, and size only

- The default verification method (ie iFlag 8) is now bit by bit - it is faster and less computationally intensive than MD5 hashing, it also allows the function to bail out early in the case of a failed copy operation

Update: 3/9/11

SCRIPT BREAKING CHANGES

- Combined some parameters into one iFlags parameter

- Re-ordered some parameters based on priority

- Added a parameter to specify a different read buffer size

- Destination file is now opened with GENERIC_READ | GENERIC_WRITE per MSDN for better compatibility copying files across a network

- No longer create destination directory structure unless specified in iFlags parameter

Update: 9/20/10

- Removed tracexxx's hashing functions (sorry!) in favor of a different method which is not RAM limited

- Hashing algorithm can be specified (those supported by Crypt.au3: CALG_MD2, CALG_MD4, CALG_MD5, CALG_SHA1)

Update: 8/26/10

- Fix for failed verification of 0 byte files

- Added check for 0 byte source in RAW functions

Update: 8/22/10

- Small change to CreateFile function to avoid errors when opening SYSTEM or HIDDEN files when specifying CREATE_ALWAYS.

Update: 8/10/10

- Added _LargeFileCopyUnbuffered and _LargeRawCopyUnbuffered which will copy files using unbuffered I/O.

- MD5 hashing of the source file / memory block is now done on the fly from the memory buffer. This avoids an additional read of the source file later on and saves time.

- Destination file MD5 hashing is now done with trancexxx's nice hashing UDF. It uses the Windows API and is really fast.

Update: 7/9/2010

- Added _LargeRawCopy which will copy large memory blocks (usually from an embedded resource) to a file in the same manner as _LargeFileCopy. This function also has the verify functionality via MD5 hash.

Update: 7/8/2010

- Added optional user function and variable parameters. This function, if given, will be called after each write operation with the total bytes written, total file size in bytes, and the user given variable. This is perfect for a copy-with-progess type of function call.

Example

#NoTrayIcon
#AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6
#include <_LargeFileCopy.au3>

_Main()

Func _Main()
    Local $msg, $timer, $ret
    Local $src = "ubuntu-10.04-desktop-i386.iso" ; <---- path to a really large file goes here, recommended 600MB+
    Local $destpath = @DesktopDir & "testdir" ; <---- destination path goes here
    GUICreate("File Copy Test", 250, 150)
    GUICtrlCreateLabel("Try to interact with the GUI during file copy...", 20, 15)
    Local $go = GUICtrlCreateButton("Copy Large File", 50, 50, 150, 50)
    Local $doing = GUICtrlCreateLabel("Ready...", 20, 120, 200)
    GUISetState()

    Do
        $msg = GUIGetMsg()
        If $msg = $go Then
            ; create destination path
            DirCreate($destpath)
            ; internal FileCopy() function
            GUICtrlSetData($doing, "FileCopy()...")
            $timer = TimerInit()
            ConsoleWrite("FileCopy:" & @CRLF)
            $ret = FileCopy($src, $destpath, 1)
            ConsoleWrite("return: " & $ret & @CRLF & "error: " & @error & @CRLF & "time: " & TimerDiff($timer) & " ms" & @CRLF)
            ConsoleWrite("====================" & @CRLF)
            ; _LargeFileCopy() UDF function
            GUICtrlSetData($doing, "_LargeFileCopy()...")
            $timer = TimerInit()
            ConsoleWrite("_LargeFileCopy:" & @CRLF)
            $ret = _LargeFileCopy($src, $destpath, 1)
            ConsoleWrite("return: " & $ret & @CRLF & "error: " & @error & @CRLF & "time: " & TimerDiff($timer) & " ms" & @CRLF)
            ConsoleWrite("====================" & @CRLF)
            GUICtrlSetData($doing, "Ready...")
        EndIf
    Until $msg = -3
EndFunc

LargeFileCopy.zip

Edited by wraithdu
  • Like 2

Share this post


Link to post
Share on other sites
KaFu
wraithdu

Small update in 1st post.

Share this post


Link to post
Share on other sites
Fran

Looks good, will for sure someday come in handy

Came in handy today! ;)

Thanx wraithdu!!

:blink:

Share this post


Link to post
Share on other sites
Fran

Just one more question...

How would I apply this to FileInstall()?

I'm creating a script that will run an update for a program on the user's computer. And copying the update file to a temp directory (using FileInstall), is what causes my script to run.

(I've already applied the _LargeFileCopy function in another gui and it works 100%)

F

Share this post


Link to post
Share on other sites
wraithdu

Hmm, this is just like FileCopy, in that a physical source file must exist. FileInstall decompresses an included resource from the EXE into a physical file. If that is what's hanging your GUI, then you need an alternative to FileInstall. Is there a reason you can't include the update file as a separate file with your updater? Or setup a network download location or something like that? Once you have a physical file you can use my function to copy it around if you need to.

A more intensive option would be to include the update as a resource, as James suggested, then read the binary data in chunks and pass it to _WinAPI_WriteFile. It's just as I've done, except the source data would be directly from the resource section. I suppose UPX could compress that data somewhat to save space as well.

Edited by wraithdu

Share this post


Link to post
Share on other sites
wraithdu

See the latest update. I wrote a _LargeRawCopy to copy large memory blocks in the same way. So you should be able to embed your update as a resource, and use Zedna's UDF to get the resource pointer and size.

Be careful with this function. It checks for read access to the memory range pointed to by $pSrc and $iSrcSize, but it obviously cannot guarantee what that memory actually contains.

Share this post


Link to post
Share on other sites
gcue

ive been SumTingWong's script for copying

http://www.autoitscript.com/forum/index.php?showtopic=11888&view=findpost&p=82020

but as you mentioned, I get the freezing gui. so your UDF caught my interest-the verifying is really cool too!

I do wish the process was cancelable though - possible?

Share this post


Link to post
Share on other sites
wraithdu

It can be. You'd have to add a global flag in the copy loop and a cancel button in your GUI. Something like an 'If $flag Then close handles and return an error' in the 'Do... Until' loop. That's not a feature I want to add to the UDF, but it should be very easy for you to modify.

Share this post


Link to post
Share on other sites
gcue

so if i specify a directory it doesnt copy subdirs?

i tried both

$src = @DesktopDir

$src = @DesktopDir & "\*.*"

Share this post


Link to post
Share on other sites
wraithdu

No. If you bother to read the headers, the function takes a single source file. You may specify a file name or existing directory for the destination. If you specify a path that doesn't exist for the destination, it will assume it's a file name.

I don't see the need to implement a recursive copy at this point. It's easy enough for you to design whatever kind of loop you need and plug in my function where appropriate.

Share this post


Link to post
Share on other sites
Fran

Wraithdu, Zedna, James,

Thank you all for your comments and suggestions.

It's all a bit greek to me at this stage, but I'm sure it will start making sense as soon as I begin applying it.

I hope I can push on your buttons again if I get stuck somewhere :blink:

Fran

Share this post


Link to post
Share on other sites
jfcby

Whew, this UDF save me time while copying my large files!

Edited by jfcby

Determined -- Devoted -- Delivered Make your mind up -- to seriously apply yourself -- accomplishing the desired results. **** A soft answer turneth away wrath: but grievous words stir up anger. Proverbs 15:1 KJB ****

Share this post


Link to post
Share on other sites
supersonic

wraithdu,

thank you for sharing! :blink:

Your UDF works pretty well...

Greets,

-supersonic.

Edited by supersonic

Share this post


Link to post
Share on other sites
wraithdu

Update in first post with a link to my A3COPY project.

Share this post


Link to post
Share on other sites
wraithdu

Small update, see first post.

Share this post


Link to post
Share on other sites
wraithdu

Updated, see first post.

Share this post


Link to post
Share on other sites
llewxam

Just a quick observation, this UDF is FAST FAST FAST with dealing with copying lots of little files, as well as dealing with larger files without causing the dreaded GUI stutter!! My previous method was to use

$PID = Run(@ComSpec & " /c copy /y " & Chr(34) & $copySource & Chr(34) & " " & Chr(34) & $copyDestination & Chr(34), @ScriptDir, @SW_HIDE)

With a Do....Until loop to check the status with ProcessGetStats which I thought was doing pretty well. I included a files copied per second function in an app I have been working on, and just for giggles switched to this UDF, when dealing with lots of little files it shot up from 12-17 per second to 50-60! Very low overhead!!

I also tinkered with the buffer size, wondering if I would get better performance in those smaller files with a smaller buffer, no difference.

So thanks a lot for posting wraithdu!!!

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites
wraithdu

Updated, see first post.

The reason I removed trancexxx's hashing functions and rewrote them myself, is that the CreateFileMapping / MapViewOfFile method is RAM limited. Meaning if you try to hash a file larger than the available RAM, the MapViewOfFile function fails. This is, unfortunately, unacceptable.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Similar Content

    • wraithdu
      By wraithdu
      The project grew out of my _LargeFileCopy UDF. It is an XCOPY replacement that improves in a few areas that were important to me. First, file verification can be done via bit-by-bit comparison, file hash, or simple size comparison. Second, it supports copying NTFS file/directory compression, encryption, and ACLs. It also has an optional simple progress display. All source files are in the archive. You can ignore the compilation warnings if you use Obfuscator.

      This is a commandline utility, and provides help as such. All you need is 'a3copy /?' and you should be good to go. Enjoy!

      Update: 10/8/12
      - Some fixes in _LargeFileCopy (see LFC thread)
      - Use DllCall versions of FindFirst/NextFile

      Update: 10/4/12
      - Added ability to specify multiple sources to be consolidated into the destination
      - Added flag to flatten the output, ie during a directory copy, all files are copied to the top level of the destination
      - Added check for cyclic copy
      - Built with new and improved _LargeFileCopy, including code cleanup and many optimizations

      Update: 9/22/10
      - Added check for destination same as source

      Update: 9/20/10
      - Added --version switch
      - Changes to _LargeFileCopy UDF, updated hashing functions
      - Updated user input function

      Update: 8/26/10
      - Added Aliases: custom INI defined long options
      - Added return codes, useful for scripting
      - Fix for verification of 0 byte files (fix is in _LargeFileCopy.au3)

      Update: 8/22/10
      - Added support for EFS encryption
      - Small bug fixes and code cleanup

      Help:

      A3COPY 1.0.2.3 Copies file and directory trees. A3COPY source [source [source] ...] destination [options] source Specifies the file(s) to copy. destination Specifies the location and/or name of new files. /C Copy file/directory compression state (NTFS only). /E Copy also empty directories. /F Flush file buffer after each copy. /G Show copy progress of current file. /H Copy hidden or system files. /J Copy using unbuffered I/O. /K Copy file/directory attributes. /L Flatten the output into one destination directory. /N Copy file/directory encryption state (NTFS only). /O Copy file/directory ownership and ACL information (NTFS only). Requires admin rights. /P Print full paths of source and destination. /Q Don't print file names. /R Allow operations on read-only files/directories. /S Recurse into subdirectories. /T Set destination filetimes same as source. NOTE: Only one verify method will be used. The final option flag will take precedence. /V or /V1 Verify the integrity of the copied files using a bit by bit comparison. /V2 Verify the integrity of the copied files using a MD5 hash. /V3 Verify the integrity of the copied files using file size only. Not valid for unbuffered copy operations. /W Apply wildcard to folder search when using /S. /Y Overwrite destination without prompt. /Z Never overwrite destination, suppress prompt. /? This usage information. Long options: --filespec Show source/destination filespec help. --copy Same as /EHRSTVY. --backup Same as /EHKRSTVY. --clone Same as /CEHKNORSTVY. --version Print version information. --help Same as /?. Aliases: Aliases may be set up in 'a3copy.ini' in the same folder as A3COPY. The aliases are invoked in the same way as the built-in long options. Example: a3copy src dest --mycopy [aliases] mycopy=EJSV cpyroot=CKNOV Return codes: 0: no errors 1: source not found 2: invalid file name / wildcard pattern 3: one or more errors copying files
      Filespec help:

      A3COPY 1.0.2.3 Copies file and directory trees. Filespec for source/destination: If source is a single file, then destination may be a file or directory. If destination ends in a '' then a directory is assumed. If not, then a single destination file is assumed unless a direcotry of the same name already exists. If destination is a directory, then the source file name will be used for the destination file. If source is a directory or wildcard filespec, then destination is assumed to be a directory. Wildcard filespec follows standard Windows usage of '*' and '?'. When the source is a directory, if filespec does not include a wildcard then it is assumed the directory is to be copied, and the top level directory attributes will be duplicated if the appropriate switches are set (/C, /K, /N, or /O). If the source filespec includes a wildcard, then it is assumed directory contents are to be copied, and the top level directory attributes will not be duplicated. a3copy.zip
×

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.