FAT Suite - FAT12, FAT16, FAT32 filesystem support

[RTFC 281]

Being sufficiently ancient to recall CP/M, monochrome CRTs, and woolly mammoths, I thought I'd share something from the bad old days: FAT filesystems (FAT12, FAT16, FAT32).

Still widely used in USB flashRAM, SD cards, RAMdisks, and microcontroller-based embedded systems, (F)ile (A)llocation (T)able filesystems are simple and robust, providing a great environment for learning how filesystems work. Moreover, many operating systems other than Windows can handle them, enabling data exchange between different platforms. That's the good part.
I've often wondered why nobody had yet written a FAT UDF library for AutoIt. Now I know why. When a normally bone-dry tech page introduces FAT with the words "Welcome to Hell" you know you're in for a rough ride. It took me almost a year of grief and many, many wrecked (virtual) volumes to get to this first beta release. That's the bad part.

Why was it so painful to develop, you rhetorically ask? The I/O itself is fairly straightforward, but backward compatibility, poor design choices and some blatant mistakes have produced a convoluted mess of rules with exceptions, exceptions to the exceptions, and exceptions to the exceptions to the exceptions (and in case you're wondering, no, I'm not talking about my own scripts here). It's like the Gregorian calendar, which has 365 days per year, unless the year is divisible by 4 (366), unless the year is also divisible by 100 (365), unless the year is also divisible by 400 (366). Add to that non-standard address-widths (paired 12-bit in FAT12, 28-bit in FAT32), file fragmentation, lost clusters, cross-, open-, and self-linked chaining errors, and the train wreck that is VFAT Long File Names support , and you've got yourself a challenge.

The result is a main library called the FAT Suite (FATsuite.au3), to which I've added a large demo GUI called FAT Manager (FATman.au3, not to be confused with the famous superhero), plus my trusty MBR handler (BuildPartitionTable.au3), and a few sample image files with different types of FAT. If running older versions of AutoIt, you'll additionally need Yashied's outstanding WinAPIEx.au3. I've also gratefully incorporated two excellent UDFs by Ascend4nt and trancexx respectively (with acknowledgement, of course). Many thanks to you all!

When you start FATman and load a volume, you'll see a pretty lame directory/file tree above a small log-window (ripped from my CodeScanner). Bog-standard menus allow you to copy, rename, remove files and dirs, etcetera. No big deal, right? You're not impressed.

But please consider this:
When FATman loads a volume (either from an image file or a physical drive) it is unmounted. As far as Windows is concerned, the volume is inaccessible to you. But your actions are not handled by the OS, but by the FATsuite UDFs. If you don't appreciate the significance of unmounted full access, then this library probably isn't for you. For my fellow codehounds, however, check out the main features below. And yes, it can be slow for large filled volumes with a small cluster size (especially the demo GUI tree refresh). But it also offers you full access to your files and other data, that is, more than Windows usually offers you.

Main Features:

• FAT volume creation and (re)formatting (18 KB - 2 TB; slow for large volumes)
• visualising FAT architecture, with interactive tool (manipulate total size, cluster size, number of FAT copies)
• two-way file transfer between FAT volume and memory (UDF only, no GUI support)
• two-way file transfer between FAT volume and other, mounted (FAT/NTFS) volumes
• internal volume-, directory-, and file I/O (create/copy/move/truncate/remove/rename/filefind/edit filespecs)
• *physical* directory sorting by any criterion (unlike MicroSoft's "sorted" view of physically unsorted data)
• wiping files and physical removal of "deleted" files/subdirs in dir table and zeroing DataRegion clusters
• retrieving lost/deleted/damaged data clusters (with or without FAT reference)
• detecting/patching FAT chaining errors (cross-linked, open-ended, and circular chains)
• raw FAT I/O, including flipping CleanShutdown and Hard R/W-error bits
• manipulating bad clusters (single creation/single removal/bulk removal) with file chain rerouting
• determining cluster ownership for any defined FAT entry
• extracting all filesystem parameters, and editing some
• rawcopy image-to-volume and volume-to-image (imagefile or physical device)
• optional comparison/synchronisation of multiple FAT copies (the table, not the entire volume)
• File Find utility (UDF only, no GUI support)
• heavily annotated code for learning about FAT filesystem mechanics

But beware, you can do enormous damage to your FAT volumes.   For example, there's no soft-delete with recycle bin retrieval, and confirmation checks are few. Once you remove/wipe/truncate a file, it's gone forever, period. So please, please, please, read the FATsuite Remarks section before calling its UDFs; also read the annotations in the UDFs; study FATman's many examples; always work with imagefiles of which you keep multiple backup copies. Remember, it's a beta release ( glitches possible!); you may accidentally parse a wrong parameter at some point; and Murphy's Law always applies.

 

FATsuite.v1.1.7z (AutoIt 3.3.12 compliant version 1.1; likely won't run in legacy AutoIt environments)

FATimageExamples.7z Sample FAT image files (reduced in size)

 

I hope this contribution helps to clarify how FAT filesystems work, and that you have fun playing with it, or find some other use(s) for it.
RT

PS Okay, I admit the mammoth was already slightly dead when I saw it, and had been for quite a while.
 

Edited January 30, 2016 by RTFC

[RTFC 281]

Beta version 0.9 released.

Bug fixed: drive letter assignment for primary volume on external device when preceded by a logical volume on the same device (you're not likely to encounter this in the wild, but hey, a bug is a bug).

Added functionality: enabled two-way rawcopy for physical devices (not perfect)

Caveats: You will likely experience difficulties accessing multiple volumes on removable devices. Windows recognises only the first primary partition on a removable device, or if a primary partition is absent, only the first logical partition. Attempting to write to any other volume on the device will produce an access-denied error (5). You may have to use a dedicated rawcopy utility to upload your edited volume imagefile back to the device in this case (take great care to identify the correct partition to write to). If that doesn't work, you'll need a partition manager to temporarily set your selected volume as primary partition.

So what this means is, you can take a snapshot of a volume on a physical device by loading that volume in FATman and selecting <Volume to Imagefile> in the Volume menu, then load the imagefile, analyse/edit it at your leisure, then reload the physical volume and upload the imagefile to it. However, removable drives can cause all kinds of problems preventing this final step to work, in which case an external rawcopy utility may be required. Note also that this scenario will only work for removable drive volumes that were previously mounted (i,e, had an associated drive letter), due to the Windows limitation of a single primary partition outlined above.

There is also a potential issue in attempting to write to an unmounted volume on physicaldrive# when any other volume on that same drive is still mounted, which will cause the drive lock to fail, which in turn causes the write to fail. To avoid this, FATman now (temporarily) unmounts all mounted volumes on a physical device (relevant only for fixed drives, I guess), restoring the mountings upon exit/reloading a different volume.

Finally, there is another potential issue in writes to volumes whose filesystem area as defined by original formatting tool does not adhere to cluster granularity (extending sector-sized sections to the next cluster boundary for cluster-aligned I/O (relevant for paging, for example)). To get around this, I've added an optional $useSectors boolean (default: False) to _CopyImageFile2Volume that when enabled will write out the entire volume sector-by-sector (slow as mud, of course ).

Hope it helps.

[RTFC 281]

Version 1.0 released.

No fancy new features, I'm afraid , nor bug fixes, just attempting to make the code Autoit.3.3.12 compliant (keeping up with the Joneses, progress, evolution and all...).

For legacy reasons, the first version is still available for now, but may disappear in future.

Edited July 16, 2014 by RTFC

[RTFC 281]

Version 1.1 released.

Bug fix related to AutoIt 3.3.12 upgrade:

_ArrayInsert now no longer supports "inserting" at the top of the array, causing FATsuite's free cluster management to utterly fail, so no image could ever be loaded.

[careca 243]

No use for me, but good work.

[TheSaint 1,897]

Thanks for sharing.

An excellent find, for my first foray into Examples for many many months.

[RTFC 281]

Thanks careca & TheSaint!

@careca: Much appreciated, especially that you took the trouble to comment even if it's of no use to you.

RT

 

Edit: oops, Yashied's #include  WinAPIEx.au3 was still enabled, instead of the new <WinAPIFiles.au3>. Makes no difference to functionality, just looks cleaner now.

Edited July 16, 2014 by RTFC