Jump to content
Sign in to follow this  
wraithdu

_LargeFileCopy UDF

Recommended Posts

Good results! And of course it will be a drop-in replacement for most folks, though I had to make some modifications since I don't use your internal $sFunction, I prefer an AdlibRegister to track speed, progress etc.

Anyway, what I did was found 2 folders on my NAS with roughly the same GB of data but with very different fie sizes to get a good metric on very large VS very small files. The files were transferred to my PC, mid-range performing NAS, gigabit connection all the way through.

Code folder, 6,934 files, 4.80GB, vast majority under 100KB

LFC November 2011: date preservation on -> date preservation off, speed loss

7:58 -> 6:15, 22%

LFC September 2012: date preservation on -> date preservation off, speed loss

5:08 -> 5:07, margin of error

TV show folder, 28 files, 4.87GB, all over 100MB

LFC November 2011: date preservation on -> date preservation off, speed loss

5:38 -> 5:31, 3%

LFC September 2012: date preservation on -> date preservation off, speed loss

4:33 -> 4:35, margin of error

I was frankly thinking that your date preservation wasn't working since it took such little time, but it is working fine. VERY nice! And speed without updating the timestamp has improved a lot as well showing lots of optimizations!!

Now for the questions. I dug through the code a bit but still have a few grey areas:

  • Are there any file attributes a file could have that would cause the copy to fail?
  • Can the timestamps be applied to the creation of a folder, or just the files?
  • Is it moot to include the "accessed" timestamp, since it will always be at the moment it is copied? (The source's "accessed" is changed as well)
  • The code has a comment "disable NTFS specific features" and goes in to _LFC_PathGetDrive, but I don't see anything I recognize as disabling an NTFS feature, what is that referring to?
  • _LFC_CheckDestination looks to be specifically removing UNC path prefixes via StringRegExpReplace functions, is that going to cause a problem for long path lengths? Or am I reading that wrong, I still haven't gotten to the point of using regular expressions.......

Great stuff, looking forward to more testing, I didn't have that much time tonight. I only had time to test the date functions, MD5 at the least will happen soon, I use Ward's so I expect that to be a good deal faster.

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

Will hopefully get time to play with the one you just posted tomorrow some time........ Unless you update it! :P


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

Awesome results :) I'm glad to see the speed improvements. So to answer some questions.

  • There aren't any attributes to my knowledge that would fail a copy operation. If a file exists with the System or Hidden attribute, then calling CreateFile with CREATE_ALWAYS will fail unless those attributes are specified in the call. However LFC pre-deletes existing files if $fOverwrite is true, after first setting FILE_ATTRIBUTE_NORMAL to prevent any read-only failures.
  • Yes timestamps can be applied to folders with the same functions (FILE_FLAG_BACKUP_SEMANTICS is supplied to get folder handles). LFC does not handle folders internally however.
  • Accessed times are weird (do some googling). They are actually disabled by default in Vista+. Even so they are updated somehow, you can look and see that fact, but I don't know exactly what triggers it. And even then there is a 1 hour resolution on NTFS, and a 1 day resolution on FAT. Honestly I wouldn't worry either way. If you're setting one timestamp, you might as well do them all.
  • If either the source or destination is not NTFS, then checking / setting Encryption, Compression, and ACLs are all disabled, since they are NTFS specific features. This is controlled with the $IsNTFS and $fACL flags.
  • Paths are manipulated in two functions. _LFC_FixPath only operates on paths without a : (colon). In that case, it ensures a path is in one of the following formats: .. (relative), . (rooted in working dir), or (unc). _LFC_CheckDestination checks whether the specified destination is a directory or a file (it is assumed to be a directory if it does not exist and ends in a ). If it is a directory then the source file name is appended and used as the destination file name. If the destination file exists, $fOverwrite is checked and the file is deleted if true, otherwise the copy fails. Otherwise $fCreate is checked and the destination parent directory is created if true and it doesn't exist. If the destination parent does not exist and $fCreate is false, the copy will fail.

And one last note on file verification. I highly recommend using the bit by bit comparison. If you think about it, hashing MUST read all the bits anyway, and has the extra expense of the hashing operation. So unless you are storing a hash for checking later on, in this case it is much more efficient to do the bit comparison.

Edited by wraithdu

Share this post


Link to post
Share on other sites

I spent some more time crushing this UDF today, what I did differently this time was I didn't add it to my Sync tool, I just put together a very simple script to do a 3-lap run of one of my folders and took the average. I decided to bite the bullet and do this for all 3 builds of LFC I had, as well as with and without date preservation. That amounted to copying my code folder from my NAS to my hard drive 18 times!

Below, I refer to LFC1, LFC2, and LFC3. 1 was from November 2011, 2 was from September 26 2012, and 3 was September 28 2012. The folder has 6,947 files 4.80GB.

LFC1 with date preservation off, average time 370.466 seconds

LFC1 with date preservation on, average time 525.550 seconds (yeah, my method sucked!! haha)

LFC2 with date preservation off, average time 344.128 seconds

LFC2 with date preservation on, average time 344.574 seconds

LFC3 with date preservation off, average time 329.003 seconds

LFC3 with date preservation on, average time 342.791 seconds

I was surprised at LFC3 with date preservation on. LFC2 with and without barely changed, LFC3 had a huge penalty. The without times were a lot better though.

I hope to start beating on some of the other abilities of the UDF tomorrow a bit, I want to look at penalties for attribute preservation as I can see that being useful as well as testing removal of all attributes.

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

Nice testing! I appreciate all the HDD cycles ;) However LFC3-1 seems like a fluke to me. There weren't any changes between your builds 2 and 3 that should have made a difference unless you were using hash verification or copying ACLs. Odd.

I realize that I need to make a few more changes for UNC paths in my PathGetDrive function. It won't affect your testing though, so carry on. It would be interesting to see at this point how LFC stacks up against a few other methods like AutoIt's FileCopy, cmd's copy, or even robocopy. I'm not suggesting you have to do that though ;)

Share this post


Link to post
Share on other sites

I already had planned on throwing AutoIt's FileCopy in there for comparison, as well as just a good-ole Windows copy and paste with a stopwatch for kicks. And your UDF was already faster than RoboCopy I think!

As for the LFC3-1, I know it seems "flukey", but here are the raw numbers from the 3 runs I had it do:

LFC3 with time preservation off

339.427585039488 seconds

LFC3 with time preservation off

323.300680584697 seconds

LFC3 with time preservation off

324.281295648437 seconds

average time 329.003187090874

Don't read in to the first run being the slowest, in many tests I ran the first was the fastest, there was no sort of caching going on. I was also not using any verification, ACL, or attribute copying, just the files.

LFC1, LFC2&3 with no date preservation "$CopyTheFile = _LargeFileCopy($FileList[$a], $CopyDestination, BitOR(1, 2))"

LFC2&3 with date preservation "$CopyTheFile = _LargeFileCopy($FileList[$a], $CopyDestination, BitOR(1, 2, 128, 256, 512))"

Ian

PS - Don't worry about the HDD cycles, it's an SSD, no cycles to speak of!! :D

Edited by llewxam

My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

Next update, few changes in this one. I've updated some of the path routines to fully support all the UNC type paths that were introduced for long path support. This was mainly for the cluster size calculation (which was still using an old function that only supported drives with letters) for unbuffered copying. I also swapped out the CRT functions for the Rtl* functions (the Rtl* functions use kernel32.dll instead of msvcrt.dll). These changes affect the bit by bit comparison, and the _LargeRawCopyUnbuffered function. Lastly I stripped out the functions that I had originally copied from my _FileEx UDF and just included the FileEx UDF. I suppose at the start I wasn't using much of it, which is why I just copied them in, but now I'm using practically the whole thing, so I'd rather just maintain one copy of those functions. None of these changes should have any impact on performance.

I think I might actually be done messing with this thing now, unless you find any bugs or have any more additions.

*snip*

Edited by wraithdu

Share this post


Link to post
Share on other sites

Only thing I'd really like to ask is that the created folder timestamps match up to the original timestamps (I guess yet another flag...), then I'd say it's pretty much perfect! I haven't gotten around to testing the unbuffered copy yet, what do you see comparing the speed of buffered VS unbuffered?

I just did one of the silly tests I was planning on doing over the weekend - your UDF took my code folder from my NAS in 5:55 seconds, FileCopy took 8:10 :)

Also, I am currently building a new version of my Sync tool around the new UDF and it's going to be sweeeeeet :D

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

According to xcopy, unbuffered copying is supposed to be faster for very large files. I don't know if I buy it though.

So regarding folder timestamps... how do you envision LFC handling these? LFC only gets a source file and a destination (folder or file). The folders it creates are the parent folders to the destination file, but it knows nothing about how these parents relate to the source. So, what do you expect the logic to be, and what destination folder(s) do you expect LFC to time stamp?

Programmatically it is trivial, the function calls to read a set timestamps are exactly the same as the files, and you could use the functions from _FileEx to do so. But I'm not sure how to sanely implement this given the fact LFC has no way to relate source to destination since it only operates file by file.

Share this post


Link to post
Share on other sites

Gotcha. Without parsing the code to get the answer to this myself, I would imagine that if the UDF is sending a file to "C:testdirnew foldercodesync" and the only folder in that path that exists is "testdir", then the UDF would know that it has to create "new folder", "code", and "sync" individually, not all in one line. Therefore, any time the UDF has to create a folder, grab the timestamp from the folder in $sSrc.

I suppose the logic would be:

1) Realize a folder has to be created

2) Parse $sSrc up to the location of the missing folder in $sDest

3) Grab timestamp from the parsed $sSrc

4) Apply timestamp to created folder

5) Repeat until all folders have been created

If that is oversimplifying the issue sorry, just thinking of the steps..........

Also, I will find time to run some speed comparisons of buffered VS unbuffered, it would be easy to divert a file copy according to size to the appropriate routine.

Ian

Edited by llewxam

My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

That's what I thought you meant... but realize that in terms of individual file copies and LFC, the source folder structure has nothing to do with the destination folder structure and final location of the copied file. Indiscriminately copying source folder time stamps with no frame of reference is irresponsible.

This is really the place where a tool built around LFC comes into play, like my A3COPY project (which needs some major modification based on the new LFC UDF). A3COPY works just like xcopy, and will preserve folder as well as file timestamps where appropriate and instructed to do so. That only works, of course, when the source is a folder. Now you can reasonably say, OK, we're cloning a whole directory tree and I know where the time stamps are coming from.

So, I guess the answer is no to your request, folder time stamping is additional functionality outside the scope of LFC. I loathe the idea of building a full directory copy into LFC, as that is practically the entire scope of a full project like A3COPY(all the handling for attributes, especially compression and encryption is really laborious). I think you should be able to put something more targeted together for your sync tool using the file time functions in _FileEx fairly easily.

Share this post


Link to post
Share on other sites

Agreed about making LFC's job dealing with whole folders being outside the scope of the UDF. No prob, I'll shoehorn that in to my app as it gets closer to completion.

Thanks!

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

Major sweet awesome updates in OP. Run don't walk!

@llewxam

Take note, additional flags for Encryption and Compression have been separated from the Attributes flag. This will be script breaking for you.

I've also optimized some more. This could be valuable for your copy tool, but you'll see more of an improvement when copying to a network destination versus local file system. Based on the changes, the fastest code path is copying to a file name which does not exist with the creation flag off. This of course assumes you create your destination files' parent directory first.

Edited by wraithdu

Share this post


Link to post
Share on other sites

The new flags don't hurt what I have been testing and building SyncV4 with, BitOR(1, 2, 128, 256, 512). I probably won't have time to integrate it in to my current build to test it out tonight, but am slightly confused by your last sentence. Are you suggesting it would be better for me to script in the logic for folder creation rather than using your 2 flag? Overall I mean, would I see a net gain by using the built-in DirCreate as needed and leaving the 2 off? I could do that, and it would give me a different way of handling the timestamp preservation of the newly created folders than I was thinking of...

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

I was testing today on the work network for an app I wrote. The reason the code path I mentioned is fastest, is because otherwise the destination existence (file or folder) is checked on every file copy. It is unavoidable to do this at least once. I use the _FileEx_GetAttributes function to both check for existence and get the required info to check if the destination is a directory. However this function returns much faster when the destination doesn't exist. Additionally, if you pass flag 2, then _FileEx_CreateDirectory first checks the full destination path, then each segment from the root to determine what needs to be created.

As I mentioned, on a local file system this is negligible at best. However over a network, it dropped copy time for 25 jpgs ~35 K in size from 9 seconds to 7. You can extrapolate from there for a huge number of files.

One thing to note on setting directory times (and I had to account for this in A3COPY), if you copy any new files into the folder then those times are updated. So you have to set the times as the very last thing you do to the folder.

Share this post


Link to post
Share on other sites

Makes sense. It almost makes me think that I should build an array of all created directories as the script runs, and at the end of the script have it set the timestamps for those folders. And since my script would know the relative paths (from and to), it would not be a big deal to track the source, get it's timestamp, and the created folder path to apply those timestamps at the end.

On the other hand, it makes me wonder if it is worth the effort. I am possibly being more picky about this than it is worth! Relatively small overhead, but really necessary?? Perhaps not.

And thanks for the thoughts on the path creation. It's a challenging concept to overcome - I would hate to have my script parse the entire list of files up front to find folders needing creation and create them prior to the files being copies to avoid the 2 flag, because what if the copy was stopped before completion, because then there would be a ton of empty folders in the destination. I'll have to stew on that.

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

It depends on how your tool parses the source. A3COPY for example uses an iterative search. For each source directory, it pre-creates the destination and calls _LargeFileCopy without the 2 flag. If more folders are found, it stores the source and destination for time stamps, attributes, etc for later, and adds the new folder to the search array. Rinse and repeat.

Really you're only going to see a difference if you are copying a lot of files into one directory. If you have 1 file in 1000 directories, not so much.

If you're interested in keeping a running list of folders and such, take a look at the A3COPY source. I implemented an algorithm to process stored directories as soon as possible, thereby keeping the storage array as small as possible, instead of storing everything until the end.

As far as being picky... depends on your backup / sync scenario I suppose. I like to keep file times and attributes, including for folders. I would provide it as an option to your users.

Edited by wraithdu

Share this post


Link to post
Share on other sites

I can't help without an example of the problem.

Hi Wraithdu,

Here i have an example which will help me to create a directory and copy 58MB file. but the copy AVI progress hangs, is that wrong using avi on GUI during copy? :( sorry if i have some stupid question.

#AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6
#include <_LargeFileCopy.au3>
#include "RecFileListToArray.au3"
#include"Array.au3"
#include <GUIConstantsEx.au3>
#include <AVIConstants.au3>

Global $msg, $timer, $ret
Global $src, $SrcBut, $SrcLab, $Srcres
Global $Dest, $DestLab, $DestBut, $DestRes, $GetList, $go,$ani1
_Main()

Func _Main()

GUICreate("File Copy", 550, 300)
$SrcLab = GUICtrlCreateLabel("Select Source Folder", 5, 10, 110, 40)
$src = GUICtrlCreateInput("", 115, 5, 400, 20)
$SrcBut = GUICtrlCreateButton("...", 520, 5, 25, 20)
$DestLab = GUICtrlCreateLabel("Select Destination Folder", 5, 60, 110, 40)
$Dest = GUICtrlCreateInput("", 115, 55, 400, 20)
$DestBut = GUICtrlCreateButton("...", 520, 55, 25, 20)
$go = GUICtrlCreateButton("Copy Large File", 200, 150, 150, 50)
$ani1 = GUICtrlCreateAvi(@SystemDir & "shell32.dll", 165, 10, 230,BitAND($ACS_AUTOPLAY,$ACS_CENTER))
GUICtrlSetState(-1,$GUI_HIDE)


GUISetState()



While 1
$msg = GUIGetMsg()

If $msg = $SrcBut Then
$Srcres = FileSelectFolder("Select Folder", @DesktopDir, 1)
GUICtrlSetData($src, $Srcres)

EndIf

If $msg = $GUI_EVENT_CLOSE Then ExitLoop

If $msg = $DestBut Then
$DestRes = FileSelectFolder("Select Folder", @DesktopDir, 1)
GUICtrlSetData($Dest, $DestRes)

EndIf

If $msg = $go Then
If GUICtrlRead($src) = "" Then
MsgBox(48, "Source Path - Warning", "Source path can not be blank")
ElseIf GUICtrlRead($Dest) = "" Then
MsgBox(48, "Destination Path - Warning", "Destination path can not be blank")
ElseIf GUICtrlRead($src) <> "" And GUICtrlRead($Dest) <> "" Then
GUICtrlSetState($ani1,$GUI_SHOW)
GUICtrlSetState($ani1, 1)
$timer = TimerInit()
$GetList = _RecFileListToArray(GUICtrlRead($src) & "", "*", 0, 1, 1)

;Exit
If IsArray($GetList) Then
For $i = 0 To UBound($GetList) -1
$ret = _LargeFileCopy(GUICtrlRead($src) & "" & $GetList[$i], GUICtrlRead($Dest) & "" & $GetList[$i], BitOR(1, 2, 8))
Next
EndIf
EndIf
EndIf
WEnd
EndFunc   ;==>_Main

Thank you,Regards,[font="Garamond"][size="4"]K.Syed Ibrahim.[/size][/font]

Share this post


Link to post
Share on other sites

Not a stupid question, but your script is not right. Your parameters to GUICtrlCreateAvi are wrong (see the help file). And you need to use BitOR to combine styles, not BitAND. I did notice though that AutoIt seems to have repainting issues on the AVI if you minimize the window, the AVI no longer seems to play until it loops and starts over again (but this has nothing to do with my UDF). Here's a fixed version of your script with the addition of a filename label and a few fixes, to show that it's working.

#AutoIt3Wrapper_Au3Check_Parameters=-q -d -w 1 -w 2 -w 3 -w 4 -w 5 -w 6
#include <_LargeFileCopy.au3>
#include "RecFileListToArray.au3"
#include"Array.au3"
#include <GUIConstantsEx.au3>
#include <AVIConstants.au3>

Global $msg, $timer, $ret
Global $src, $SrcBut, $SrcLab, $Srcres
Global $Dest, $DestLab, $DestBut, $DestRes, $GetList, $go, $ani1, $label
_Main()

Func _Main()

    GUICreate("File Copy", 550, 300)
    $SrcLab = GUICtrlCreateLabel("Select Source Folder", 5, 10, 110, 40)
    $src = GUICtrlCreateInput("", 115, 5, 400, 20)
    $SrcBut = GUICtrlCreateButton("...", 520, 5, 25, 20)
    $DestLab = GUICtrlCreateLabel("Select Destination Folder", 5, 60, 110, 40)
    $Dest = GUICtrlCreateInput("", 115, 55, 400, 20)
    $DestBut = GUICtrlCreateButton("...", 520, 55, 25, 20)
    $go = GUICtrlCreateButton("Copy Large File", 200, 150, 150, 50)
    ; ACS_AUTOPLAY is not doing anything, playback stops as soon as you hide the control
    $ani1 = GUICtrlCreateAvi(@SystemDir & "shell32.dll", 165, 10, 230, 275, 75, BitOR($ACS_CENTER, $ACS_TRANSPARENT))
    GUICtrlSetState(-1, $GUI_HIDE)
    $label = GUICtrlCreateLabel("", 10, 210, 540, 20)

    GUISetState()

    While 1
        $msg = GUIGetMsg()

        If $msg = $SrcBut Then
            $Srcres = FileSelectFolder("Select Folder", @DesktopDir, 1)
            GUICtrlSetData($src, $Srcres)

        EndIf

        If $msg = $GUI_EVENT_CLOSE Then ExitLoop

        If $msg = $DestBut Then
            $DestRes = FileSelectFolder("Select Folder", @DesktopDir, 1)
            GUICtrlSetData($Dest, $DestRes)

        EndIf

        If $msg = $go Then
            If GUICtrlRead($src) = "" Then
                MsgBox(48, "Source Path - Warning", "Source path can not be blank")
            ElseIf GUICtrlRead($Dest) = "" Then
                MsgBox(48, "Destination Path - Warning", "Destination path can not be blank")
            ElseIf GUICtrlRead($src) <> "" And GUICtrlRead($Dest) <> "" Then
                GUICtrlSetState($ani1, $GUI_SHOW)
                GUICtrlSetState($ani1, 1) ; apparently this is needed, as $GUI_HIDE stops playback
                $timer = TimerInit()
                $GetList = _RecFileListToArray(GUICtrlRead($src) & "", "*", 0, 1, 1)

                ;Exit
                If IsArray($GetList) Then
                    For $i = 0 To UBound($GetList) - 1
                        GUICtrlSetData($label, $GetList[$i])
                        $ret = _LargeFileCopy(GUICtrlRead($src) & "" & $GetList[$i], GUICtrlRead($Dest) & "" & $GetList[$i], BitOR(1, 2, 8))
                    Next
                EndIf
                GUICtrlSetData($label, "")
                GUICtrlSetState($ani1, $GUI_HIDE)
            EndIf
        EndIf
    WEnd
EndFunc   ;==>_Main
Edited by wraithdu

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

  • Similar Content

    • By wraithdu
      The project grew out of my _LargeFileCopy UDF. It is an XCOPY replacement that improves in a few areas that were important to me. First, file verification can be done via bit-by-bit comparison, file hash, or simple size comparison. Second, it supports copying NTFS file/directory compression, encryption, and ACLs. It also has an optional simple progress display. All source files are in the archive. You can ignore the compilation warnings if you use Obfuscator.

      This is a commandline utility, and provides help as such. All you need is 'a3copy /?' and you should be good to go. Enjoy!

      Update: 10/8/12
      - Some fixes in _LargeFileCopy (see LFC thread)
      - Use DllCall versions of FindFirst/NextFile

      Update: 10/4/12
      - Added ability to specify multiple sources to be consolidated into the destination
      - Added flag to flatten the output, ie during a directory copy, all files are copied to the top level of the destination
      - Added check for cyclic copy
      - Built with new and improved _LargeFileCopy, including code cleanup and many optimizations

      Update: 9/22/10
      - Added check for destination same as source

      Update: 9/20/10
      - Added --version switch
      - Changes to _LargeFileCopy UDF, updated hashing functions
      - Updated user input function

      Update: 8/26/10
      - Added Aliases: custom INI defined long options
      - Added return codes, useful for scripting
      - Fix for verification of 0 byte files (fix is in _LargeFileCopy.au3)

      Update: 8/22/10
      - Added support for EFS encryption
      - Small bug fixes and code cleanup

      Help:

      A3COPY 1.0.2.3 Copies file and directory trees. A3COPY source [source [source] ...] destination [options] source Specifies the file(s) to copy. destination Specifies the location and/or name of new files. /C Copy file/directory compression state (NTFS only). /E Copy also empty directories. /F Flush file buffer after each copy. /G Show copy progress of current file. /H Copy hidden or system files. /J Copy using unbuffered I/O. /K Copy file/directory attributes. /L Flatten the output into one destination directory. /N Copy file/directory encryption state (NTFS only). /O Copy file/directory ownership and ACL information (NTFS only). Requires admin rights. /P Print full paths of source and destination. /Q Don't print file names. /R Allow operations on read-only files/directories. /S Recurse into subdirectories. /T Set destination filetimes same as source. NOTE: Only one verify method will be used. The final option flag will take precedence. /V or /V1 Verify the integrity of the copied files using a bit by bit comparison. /V2 Verify the integrity of the copied files using a MD5 hash. /V3 Verify the integrity of the copied files using file size only. Not valid for unbuffered copy operations. /W Apply wildcard to folder search when using /S. /Y Overwrite destination without prompt. /Z Never overwrite destination, suppress prompt. /? This usage information. Long options: --filespec Show source/destination filespec help. --copy Same as /EHRSTVY. --backup Same as /EHKRSTVY. --clone Same as /CEHKNORSTVY. --version Print version information. --help Same as /?. Aliases: Aliases may be set up in 'a3copy.ini' in the same folder as A3COPY. The aliases are invoked in the same way as the built-in long options. Example: a3copy src dest --mycopy [aliases] mycopy=EJSV cpyroot=CKNOV Return codes: 0: no errors 1: source not found 2: invalid file name / wildcard pattern 3: one or more errors copying files
      Filespec help:

      A3COPY 1.0.2.3 Copies file and directory trees. Filespec for source/destination: If source is a single file, then destination may be a file or directory. If destination ends in a '' then a directory is assumed. If not, then a single destination file is assumed unless a direcotry of the same name already exists. If destination is a directory, then the source file name will be used for the destination file. If source is a directory or wildcard filespec, then destination is assumed to be a directory. Wildcard filespec follows standard Windows usage of '*' and '?'. When the source is a directory, if filespec does not include a wildcard then it is assumed the directory is to be copied, and the top level directory attributes will be duplicated if the appropriate switches are set (/C, /K, /N, or /O). If the source filespec includes a wildcard, then it is assumed directory contents are to be copied, and the top level directory attributes will not be duplicated. a3copy.zip
×
×
  • Create New...