Jump to content
Sign in to follow this  
KaFu

MFT-Access - Reading & parsing the Master File Table on NTFS Filesystems

Recommended Posts

I've got a personal project I've been working on to update a secure erase UDF of mine. It accomplishes some of the same things as your UDF, but using the defrag API. Is that at all related to what you're doing and would you be interested in seeing it? It works in at least one direction to find virtual and physical clusters belonging to any file, including sparse and EFS encrypted files. It cannot decrypt these files however, and extraction only yields undecryptable data. But it is useful for my purposes.

Edited by wraithdu

Share this post


Link to post
Share on other sites

Here's my current UDF and an example.

Now that looks really nice :), I definitely have to dive deeper into this one. One quick finding I've got, replace the $GENERIC_READ in the _FM_GetDiskSectorSize() function call to _FM_CreateFile() with 0 otherwise on Vista+ elevated Access is required.

Edit: Oh, that's not enough. In the function _FM_ReadMapToFile() you access the disk directly again, and at that point the $GENERIC_READ is definitely required. When UAC is acitvated the script definitely needs elevated access, add this line to the top of the example and compile it.

#AutoIt3Wrapper_Res_requestedExecutionLevel=requireAdministrator

Maybe add a cross check if UAC is activated to the top of the include, and if so a second check if the script runs elevated.

But this looks really, really good ;)!

Edited by KaFu

Share this post


Link to post
Share on other sites

Here's my current UDF and an example.

Thanks for those. You have had many examples I've had great benefit from. I'm currently not sure if these ones can be used in "my" solution since it's purpose is to be completely FS api independant. But I'm very thankful for the sources anyway.

In the last version of the extracter (now at v1.6) I've added experimental support for compressed files. And thanks to trancexx for the examples of RtlDecompressBuffer usage that now came useful.

The biggest and most important challenge, the way I see it, now is solving file paths / B+ trees. When that's done a whole new world of interesting possibilities suddenly became real..

Share this post


Link to post
Share on other sites

Just back from vacation :)

I'm glad you found it at least interesting, I know it's a different approach than your work. To comment on one thing, admin access has always been required for disk defragmentation. Since my UDF uses that same API it's no surprise the same access is needed.

Share this post


Link to post
Share on other sites

Added an option to the file extracter for browsing and doing a low level file copy/extraction based on the file name. It was added for faster extraction and detection of regular files (ie non system files), and uses NtQueryInformationFile and FileInternalInformation to get to the index number. (wraithdu maybe you recognize parts of that code)

Share this post


Link to post
Share on other sites

There has been som great improvements to these tools lately:

  • A new console app to decode records for individual files has been introduced.
  • Compressed and sparse files are now fully working with extraction.
  • Extraction for extremely fragmented files are also now supported, as $ATTRIBUTE_LIST is resolved and implemented.
  • Full extraction of all ADS's tied to a file.

Source: http://code.google.com/p/mft2csv/downloads/list

Share this post


Link to post
Share on other sites

Looking great! One suggestion though - too much GUI updating. I've found that throwing in an AdlibRegister for things like GUI updates can massively increase performance. I commented out the ProgressSet of your For/Next loop, placed AdlibRegister("_DisplayProgress") immediately above the For/Next loop, and added the function:

Func _DisplayProgress()
ProgressSet(Round((($i / $ExpectedRecords) * 100), 2), Round(($i / $ExpectedRecords) * 100, 2) & " % finished with exporting $MFT data to csv", Round($i / (TimerDiff($begin) / 1000), 1) & " Records per second.")
EndFunc ;==>_DisplayProgress

That increased performance from 200 records per second to 240!!

But it is looking pretty damn sweet man! I would assume that some correlation would need to be made with the $Bitmap file and some pretty handy data recovery software could be built from this!!!

Great stuff

Ian

EDIT - Sorry, should have specified that this change was made to MFT2CSV :D

Edited by llewxam

My projects:

  • IP Scanner - Multi-threaded ping tool to scan your available networks for used and available IP addresses, shows ping times, resolves IPs in to host names, and allows individual IPs to be pinged.
  • INFSniff - Great technicians tool - a tool which scans DriverPacks archives for INF files and parses out the HWIDs to a database file, and rapidly scans the local machine's HWIDs, searches the database for matches, and installs them.
  • PPK3 (Persistent Process Killer V3) - Another for the techs - suppress running processes that you need to keep away, helpful when fighting spyware/viruses.
  • Sync Tool - Folder sync tool with lots of real time information and several checking methods.
  • USMT Front End - Front End for Microsoft's User State Migration Tool, including all files needed for USMT 3.01 and 4.01, 32 bit and 64 bit versions.
  • Audit Tool - Computer audit tool to gather vital hardware, Windows, and Office information for IT managers and field techs. Capabilities include creating a customized site agent.
  • CSV Viewer - Displays CSV files with automatic column sizing and font selection. Lines can also be copied to the clipboard for data extraction.
  • MyDirStat - Lists number and size of files on a drive or specified path, allows for deletion within the app.
  • 2048 Game - My version of 2048, fun tile game.
  • Juice Lab - Ecigarette liquid making calculator.
  • Data Protector - Secure notes to save sensitive information.
  • VHD Footer - Add a footer to a forensic hard drive image to allow it to be mounted or used as a virtual machine hard drive.
  • Find in File - Searches files containing a specified phrase.

Share this post


Link to post
Share on other sites

I think that's the ultimate goal, to create a data recovery solution for NTFS. But it's still a way to go from here to there.. Looking more promising though :D

Thank you very much for the tip :huh:

Share this post


Link to post
Share on other sites

Reading is not a problem. Have a look at the newest code in MFTRCRD how $INDEX_ROOT and $INDEX_ALLOCATION is decoded. Writing is slightly more tricky though, and completely blocked on the systemdrive on nt6.x (unless you implement a kernel driver with it). If you look at SetMACE, a timestamp manipulating utility, writing is actually implemented in the latest version. Of course you need to know for sure what, where and how you are writing to disk, to prevent filesystem corruption. And actually you gave me an idea there, for how to implement timestamp modification in INDX records..

Share this post


Link to post
Share on other sites

Hi joakim,

1: modifying INDX records do not require kernel level driver.

2: for past 2 days I have been searching all over the net for step 1 : modifying $I30 and ultimately decided to ask you - cause yours was the only utility which does 100% justice to reading/parsing MFT.

3: INDX records are used for Forensics - hence you will have to modify this inorder to make SetMACE a perfect utility to bypass even the best of the best Forensic softwares.

regards

DeltaRocked

Share this post


Link to post
Share on other sites

The kernel driver requirement is only necessary when working on nt6.x's systemdrive (inside the boundaries of the volume/filesystem). All else works just fine without such a driver.

With the last code I added yesterday, $I30 is almost fully decoded. Implementing modification of it can be done with a little more work.

Regarding forensic implications, there's also other things to handle to make it perfect. The journal is one place I already have thought of. Another is the $LogFile, but that one is more tricky. When that's done, it is approaching perfection.

What exactly is your goal?

Share this post


Link to post
Share on other sites

Goal : to add a file in MFT and add an INDX record. that is for anti-malware purpose.

As of this moment , I am creating a file through CreateFile, then opening up MFT reaching that location and reading 1024 bytes and then writing '0's . after that I am relocating the same file to a different record within MFT.

Due this INDX contains the file record (wyhich has been created by Windows :guitar: ) but physically the file is located in some other location - hence another file with the same name cannot be created. I cannot say more but its for anti-malware.

regards

Deltarocked.

[uPDATE]

I am unable to print $INDEX_ROOT when using MFTRCRD.

MFTRCRD.exe g:autorun.inf -d attriblist=on indxdump=on

Starting MFTRCRD by Joakim Schicht

Version 1.0.0.12

Target is a File

Filesystem on g: is NTFS

NtQueryInformationFile: Success

File IndexNumber: 27

NtQueryInformationFile: Success

BytesPerSector: 512

SectorsPerCluster: 8

ReservedSectors: 0

SectorsPerTrack: 63

NumberOfHeads: 255

HiddenSectors: 8064

TotalSectors: 7827583

LogicalClusterNumberforthefileMFT: 262144

LogicalClusterNumberforthefileMFTMirr: 489223

Target 1B000000 found

Found attributes:

$STANDARD_INFORMATION (1)

$FILE_NAME (1)

$OBJECT_ID (1)

$DATA (1)

Record header info:

Offst to update sequence number: 48

Update sequence array size (words): 3

$LogFile sequence number (LSN): 0000000000801A64

Sequence number: 1

Hard link count: 1

Offset to first Attribute: 115

Flags: FILE

Real size of the FILE record: 344

Allocated size of the FILE record: 1024

File reference to the base FILE record: 0000000000000000

Next Attribute Id: 0005

Number of this MFT Record: 27

Update Sequence Number (a): 0400

Update Sequence Array (a): 00000000

$STANDARD_INFORMATION 1:

HEADER_Flags:

CreationTime (CTime): 2012-07-27 10:49:11:370:5000

LastWriteTime (ATime): 2012-07-27 11:21:29:058:0000

ChangeTime (MTime): 2012-07-27 11:21:29:058:0000

LastAccessTime (RTime): 2012-07-27 10:49:11:370:5000

DOS File Permissions: archive

Max Versions: 0

Version Number: 0

Class ID: 0

Owner ID: 0

Security ID: 258

USN: 0000000000000000

$FILE_NAME 1:

ParentSequenceNo: 5

CreationTime (CTime): 2012-07-27 10:49:11:370:5000

LastWriteTime (ATime): 2012-07-27 10:49:11:370:5000

ChangeTime (MTime): 2012-07-27 10:49:11:370:5000

LastAccessTime (RTime): 2012-07-27 10:49:11:370:5000

AllocSize: 0

RealSize: 0

Flags: archive

NameLength: 11

NameType: POSIX

NameSpace: 10

FileName: autorun.inf

ParentReferenceNo: 5

$OBJECT_ID 1:

GUID Object Id: EFAC75C8-B6D7-E111-A7A3-002618C2EA56

GUID Birth Volume Id: NOT PRESENT

GUID Birth Object Id: NOT PRESENT

GUID Domain Id: NOT PRESENT

$DATA 1:

Length: 32

Non-resident flag: 00

Name length: 0

Offset to the Name: 24

Flags: 0000

Attribute Id: 0001

Resident - Length of the Attribute: 3

Resident - Offset to the Attribute: 24

Resident - Indexed flag: 0

Resident - Padding: 00

Non-Resident - Starting VCN:

Non-Resident - Last VCN:

Non-Resident - Offset to the Data Runs:

Non-Resident - Compression Unit Size:

Non-Resident - Padding:

Non-Resident - Allocated size of the attribute:

Non-Resident - Real size of the attribute:

Non-Resident - Initialized data size of the stream:

The Attribute's Name:

FALSE------------->

FINISHED!!

Edited by deltarocked

Share this post


Link to post
Share on other sites

I think I need some more details in order to understand the excersise. Consider communication channel alternatives.

Share this post


Link to post
Share on other sites

Are you sure that the file autorun.inf really has an $INDEX_ROOT attribute?

Try again with;

MFTRCRD G:5 -a attriblist=off indxdump=on

Note attriblist=off since I'm 100% a small text file never will be that fragmented.. (speeds up processing). The "G:5" will then dump INDX's for the root directory.

Share this post


Link to post
Share on other sites

Also, you should use v1.0.0.13, the latest version which has fixes for a lot of the relevant things you care about :)

Share this post


Link to post
Share on other sites

Hi joakim,

the G:5 has worked perfectly. Sorry, couldn't answer earlier (Spondylosis) kept me away from the comp.

Within next few hours chk your mail, I will be sending you an interesting log of $MFT and $I30.

You can replicate it and have fun.

Regards

Deltarocked.

Share this post


Link to post
Share on other sites

I had wondered, if it would be possible to use this script to find all the data on the disk that is "available for writing" (files that have been deleted by the OS, but which remain on the HDD), and overwrite it with blank data (basically removing the ability to recover any files). [sorry if the answer is obvious, I am at work and cannot read through all of the code/comments yet]

It could be a good security/privacy tool, I tried something similar a few years back, but it was sooo slow.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...