Jump to content
Sign in to follow this  
joakim

NTFS Data Recovery

Recommended Posts

joakim

Developmen of this project has recently been re-activated.

DDan at forensicfocus made huge contribution!

It supports recovery of all files off an NTFS volume, when the $MFT record entry is still intact. That means regular deleted files, resident, non-resident, compressed, sparse, alternate data streams etc.

It supports images files (dd style) of both disk and partition. For disk images, both MBR and GPT style is supported.

Reparse points is now solved internally without depending on external tools.

It has a simpel gui and is not that hard to figure out how to use.

Anyways, here's the code, and I'm sure some people find it interesting.

It has been tested ok on XP SP3 x86 and x64, Vista x86, Windows 7 x86 and x64, and Windows 8 x64.

Documentation: http://code.google.com/p/mft2csv/wiki/NTFS_File_Extracter

Latest version can usually be found at; http://code.google.com/p/mft2csv/downloads/list

Todo:

  • Add native support for PhysicalDrive to remove the requirement that the volume must be mounted (though the code is already there in the support for image files).
Edited by joakim

Share this post


Link to post
Share on other sites
joakim

We continued development recently and now there is a much more stable and mature version available. The crash on Windows 7 x64 has been fixed, just to name one. From the readme:

This tool's main purpose is to extract files off NTFS formatted volumes. In this version it is not possible to specify specific files to extract. It will extract all. When starting up the program, you will be asked to select an output directory where to dump all extracted files. This can be reconfigured later anyway. On program startup, it will scan attached disks for any mounted NTFS volumes. Found volumes are put into the upper dropdown box. Select target volume and click button "Extract All Files".

It is also possible to select a disk image file to extract from. Found NTFS volumes will be put into lower dropdown box. Select target volume and click button "Extract from image".

Extracted files will be put into specified output directory with the original directory structure. $MFT records with files marked as deleted, will get a prefix in the output similar to [DEL+refnumber]originalname.ext. It supports extraction of all files with its MFT record intact.

A logfile is generated for each extraction, where issues and the relevant record are written.

Has been tested OK on XP 32-bit and 64-bit, Vista 32-bit, Windows 7 32-bit and 64-bit.

.

NTFSDataRecovery_v2000.zip

Share this post


Link to post
Share on other sites
llewxam

Niiiiiiiiiiice, makes me motivated to complete the forensic imaging software I started forever ago! Are there any plans at this point to parse the bitmap file in order to allow selection of specific files for extraction?

Thanks for the continued efforts!

Ian

Edited by llewxam

My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites
llewxam

Quick first impressions:

Disk Access error until I included #RequireAdmin (Win7x64)

Line 63, "output" written as "utput", only shows when you change the source after reaching the GUI

This looks like another script where using an AdlibRegister to update the GUI during extraction phase could increase performance a lot

Also possible improvement would be progress bar for overall progress and per-file for when it hits a big one

Possibly consider having the GUI not stay on top

AND IT IS TOTALLY AWESOME!!!!!!! :D I am highly impressed!

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites
JFX

That's really great :thumbsup:

To speedup use case sensitivity for hex-string comparison

Also dllopen should help a bit.

Share this post


Link to post
Share on other sites
joakim

@Ilewxam

I thought about different things regarding $bitmap. For the $MFT attribute bitmap, it is possible to parse it too, but since we scan the complete $MFT anyway and generate an array of the filesystem, it is probably easier to expand on that like populating a second array for deleted files. Regarding the $bitmap metafile, that's an other story, but could be used to dump unallocated. Also related is "slack" in all its forms and definitions, which I've thought about as an option to handle. However it will not happen anytime soon. Regarding the AdlibRegister we only update the gui every 10th record so I'm not sure the gains are that much. However if it turns out anything makes it run faster, it is still worth implementing.

@JFX

Did you mean setting something like

Global $hKernel32dll = DllOpen("kernel32.dll")

and modifying all winapi functions accordingly to use the global handle?

Just tried both those fixes (if I got them right) and it runs a little faster..

Share this post


Link to post
Share on other sites
llewxam

Only other request I could think of at the moment is handling the _WinAPI_ReadFile and WriteFile - it makes sense how you have it, making the buffer size the same as the file size, so one read, one write. However, that makes it impossible to have a smooth progressbar on transferring those large files. I started working on breaking it up in to smaller bits but my eyes can't handle any more coding right now so reverted in order to post this hack, which is my idea of the AdlibRegister.

Again though, I am really loving what I am seeing!

Ian

NTFSDataRecovery.zip


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites
JFX

@joakim

Yes, passing the dllopen handle to the dllcalls should avoid loading and closing the dll on every call.

Speed gain is very small, just 3 seconds here, but maybe useful for a drive with very much files.

Share this post


Link to post
Share on other sites
joakim

So a new version is up with the fixes suggested so far. Also now is added an option to only extract deleted files.

All suggestions for improvements are much appreciated :)

NTFSDataRecovery.zip

Share this post


Link to post
Share on other sites
wraithdu

Couple things from recent comments:

I'm not sure how exactly you're using ReadFile / WriteFile, but check out my _LargeFileCopy UDF for buffered copying with callbacks for progress.

DllCalls to system loaded DLLs like kernel32 have miniscule (at best) speedups from using DllOpen / DllClose and passing the handle to the call. Those DLLs are always loaded so there is no load / unload overhead. And the lookup from passing the string name "kernel32.dll" versus a handle makes virtually no difference. So I wouldn't bother wasting time converting code or losing sleep over it.

Edited by wraithdu

Share this post


Link to post
Share on other sites
Ddan

Hi,

I'm new to the forum, so if I'm doing something wrong please tell me kindly.

Joakim and I are working independently on this project. My main focus is getting the MFT analysis working properly. At this stage, I'm less concerned with minor speed-ups or smoothing progress bars as I think that can all come later. I'd love to hear any comments in regard to incorrectly extracted files. There is one bug fix that wasn't included in the latest version. If you search for "$Type > 128" (without the inverted commas) and change the value to 256, those No Data Attribute toss-outs should stop.

I'd like to take up some of the comments which have been posted and which don't seem to have been addressed. So in no particular order:

@llewxam (and later wraithdu)

<Only other request I could think of at the moment is handling the _WinAPI_ReadFile and WriteFile - it makes sense how you have it, making the buffer size the same as the file size, so one read, one write.>

I'm not sure where you get this from, the files are handled in 16 cluster chunks. You'll both probably recognise that 16 clusters is a compression unit. A little experiment will show you why we went down this track. For simplicity try this on a freshly formatted usb stick. Copy a single file of 500+ mb to the stick and then look at the mft record and in particular the data run. What you will see is a simple data run of x clusters at location y. Now mark the file for compression and look at the new mft record. It will now have a non-resident attribute list. If you follow the list to the mft records that now contain the data run, you will see the longest data run you have ever seen all neatly broken up into compression units. The problem with handling the run is that you need to look at each compression unit to determine whether or not any compression has been done.

@llewxam

< Are there any plans at this point to parse the bitmap file in order to allow selection of specific files for extraction?>

No plans currently for single file extraction. Should be very easy to implement though as all the required data is in the FileTree array. It's held in mft record number order and contains all files and folders resolved back to the root folder (as far as possible anyway). Just really needs to be sorted, collected and presented for choosing. I'm tempted to add "Go for it!".

@llewxam

< This looks like another script where using an AdlibRegister to update the GUI during extraction phase could increase performance a lot>

Did you take any timings on this? I find running the same extraction multiple times often shows variations up to 2m (in 1h 50m). I guess it depends on what the system idling in the background consumes.

Ddan

Edited by Ddan

Share this post


Link to post
Share on other sites
llewxam

Hi,

I'm new to the forum, so if I'm doing something wrong please tell me kindly.

HI! :) Nice to have you come by!

@llewxam (and later wraithdu)

<Only other request I could think of at the moment is handling the _WinAPI_ReadFile and WriteFile - it makes sense how you have it, making the buffer size the same as the file size, so one read, one write.>

I'm not sure where you get this from, the files are handled in 16 cluster chunks.

I can only take your word for it, as nothing ever comes right out and says "for.... to 16" or "Do...Until X == 16", I have never tried to find the values of items such as $RUN_Clusters[$r] or tried to understand the workings of "If (($RUN_VCN[$r+1]=0) And ($i+$RUN_Clusters[$r+1]=16) And $IsCompressed) Then" :D

@llewxam

Did you take any timings on this? I find running the same extraction multiple times often shows variations up to 2m (in 1h 50m). I guess it depends on what the system idling in the background consumes.

I didn't get very scientific with this script, but when I put the same AdlibRegister method to work on Joakim's MFT2CSV performance jumped 20% for me, from 200 records per second top 240. And the way I figure the performance improvement could be even better for lower-end machines, as the GUI updating would take more time on a slower computer, so the less time you have to do it the more time can be spent on number crunching. But I could try to find the time to run a comparison to see what happens on this script.

Glad to see this is still a work in progress, and I wish I were able to contribute more to it than just some little GUI speed tweaks, but you guys are pretty far above my head as it is! :)

Thanks, and welcome.

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites
Ddan

I can only take your word for it, as nothing ever comes right out and says "for.... to 16" or "Do...Until X == 16", I have never tried to find the values of items such as $RUN_Clusters[$r] or tried to understand the workings of "If (($RUN_VCN[$r+1]=0) And ($i+$RUN_Clusters[$r+1]=16) And $IsCompressed) Then" :D

True, but on the other hand:

_WinAPI_ReadFile1($hDisk, DllStructGetPtr($cBuffer), $BytesPerCluster * 16, $nBytes)

and

_WinAPI_WriteFile1($hFile, DllStructGetPtr($cBuffer), $BytesPerCluster * 16, $nBytes)

sort of says it all. (Haven't worked out the smiley bit yet, so assume a bigger grin than your's)

Ddan

Share this post


Link to post
Share on other sites
llewxam

(Haven't worked out the smiley bit yet, so assume a bigger grin than your's)

HAHAHA!!!!

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites
llewxam

Just a quick message that I am trying this tool in a real-world situation and it is showing good results. A client's hard drive crashed, I pulled the files I needed with R-Studio and got her back up, did not make an image of the drive and decided to try this tool on her failing drive to see how it would go. It has been running a very long time since the drive is in very bad shape, but the files it has pulled off look good.

Another hugely useful feature would be to select from the available files which you want it to grab, no doubt that will be in the works at some point.

So, A+ for effectiveness with the current build on a dying drive!

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites
Ddan

NTFSDataRecovery.zip

Just a quick message that I am trying this tool in a real-world situation and it is showing good results. A client's hard drive crashed, I pulled the files I needed with R-Studio and got her back up, did not make an image of the drive and decided to try this tool on her failing drive to see how it would go. It has been running a very long time since the drive is in very bad shape, but the files it has pulled off look good.

Just a few quick comments.

First off, use this attached version as earlier one had a few bugs, which I think have now been corrected.

In regard to recovering a failing drive, always take an image first (I use WinHex). That way you don't need the drive anymore but more importantly, the access time is a million times quicker. Well...maybe not a million.

Share this post


Link to post
Share on other sites
llewxam

hehe, agreed on imaging first, I usually do but just felt like giving it a good torture test! :D

I started coding a drive cloning program of my own which won't have nearly all of the features of WinHex which I use as well, but it will do the things I care about - clone to or from an image, disk-to-disk clone, sector integrity test, and zero-fill the drive (secure wipe). It mostly works but without having a way to gain exclusive access to the target drive when restoring an image it fails most of the time, and I have not found a solution to that.

I have sent the as-is version to a few people to check out, if you would like to look it over I would be happy to send it to you in a PM. Just let me know.

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites
joakim

$Ddan

I will take a look at it when I'm done moving into the new house.

$llewxam

I bet it's nt.6x that's screwing you. Read up on this; http://msdn.microsoft.com/en-us/library/windows/desktop/aa364575(v=vs.85).aspx I wrote a tiny app a few years ago called LockVolume, which was posted as reboot.pro, but the site is currently down. There you will see an example of how I solved it. But I could be wrong, as it might be something different. Hard to say without having seen it.

Share this post


Link to post
Share on other sites
llewxam

Thanks joakim, I will have a look. I found some code on the forum here yesterday that I thought would fix me up, it helped but not 100%, when writing an image to the disk it gets the first 1 or 2 8MB chunks written then has 5-5 fails in a row, then works again. Frustrating!

Thanks again, will check it out later tonight.

Ian


My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites
joakim

Then it's a different thing as I thought the issue where related to writing directly to filesystem sectors (for instance restoring image). If it was for image creation, ignore my comment completely.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

×