Sign in to follow this  
Followers 0
wraithdu

Securely overwrite files and free space

38 posts in this topic

#1 ·  Posted (edited)

Update: 11/14/2012

- Fixed bug in Freespace erase - it was using sector size by mistake (did not cause failure, but slowed the wipe)

- Replaced dir move/remove functions with long file name compatible API functions (thanks KaFu)

- Added long path and UNC support to the _FileMapping UDF (this change affected portions of the UDF only used to wipe sparse / encrypted / compressed files on NTFS)

- Wiping of these special files is now restricted to local volumes - you cannot get the volume handle required from remote volumes

- NOTE: When running under UAC full admin permissions are required to use the defrag API to wipe these special files

- Various updates to included _FileEx UDF

Update: 10/1/12

- _SecureDirectoryDelete is now an iterative function, instead of recursive.

- Added ability to overwrite with random data. To activate this feature, you must use the user array format with the array element corresponding to the random pass set to -1. Note this *will* incur a performance penalty since the buffer is filled with new random data each time it is used.

- Registered _SD_FreeBuffers() as an on-exit function.

- Changed the way buffer recreation is handled - new buffers are automatically created each time you pass a different value or array for $aPatterns.

- Some updates to the included _FileEx UDF.

Update: 9/11/12

- _SecureDirectoryDelete returns an array of errors for all errors encountered during recursion. This includes the error number, path at which the error was encountered, and the type of path item (file or folder).

Update: 9/10/12

- More optimizations, read new header for important info

  • Optimized buffer creation - done once on first call to any function, survives for life of script.
  • $aPatterns parameter now accepts a single integer 0 <= i <= 255 to indicate a single pass using that byte pattern (-1 for default 3-pass DoD_E)
  • Changed final file / dir removal to restore original file / dir name in case of removal failure.
- SCRIPT BREAKING - Changed the order of the $fDelete and $aPatterns parameters, moving the more commonly used $aPatterns parameter to a higher priority.

Update: 9/7/12

- Code cleanup and some optimizations

- Fixed a bug that could slow down wiping of compressed / encrypted / sparse files if using more than one pass (this bug did not affect the actual wiping of these files)

- Added callback functions to monitor the progress of the wipe

  • File: Callback receives bytes written and total bytes.
  • Dir: Callback is passed to file delete and receives bytes written and total bytes per file. Total directory size and bytes is not calculated, as the penalty would be too high for these basic functions. If you need total directory size, do this calculation yourself before calling this function, and keep track of the data written.
  • Freespace: Callback is passed bytes written and total free bytes. Under certain circumstances, total bytes written may not equal total free bytes on the last call.
Update: 9/5/12

- Major rewrite

  • Rewrote delete function, includes a special method to securely erase NTFS compressed, sparse, and EFS encrypted files.
  • Simplified the parameters. Only need to specify if the file should be deleted after being overwritten, and can specify custom delete patterns.
  • Added a directory deletion function to recursively wipe directories
  • Added a freespace wipe function, based on the Sysinternals SDelete utility
  • Reparse points: symlinks and junctions are now handled, they are simply deleted, NOT followed or overwritten.
- The archive includes my _FileMapping UDF and _FileEx UDF for support functions.

I tested this A LOT, which also accounts for the delay in releasing. I've tested on NTFS and FAT32 drives, and it functions as it should. That said, I take no responsibility for data loss.

If there are testers familiar with data recovery, I'd love tests done on the thoroughness of the wiping, especially the free space wipe. NTFS has some real gotchas that normal apps can't deal with safely, so some stuff may slip through. If so, I'd like to know about it.

I plan on working on some sort of cluster tip / slack space wipe to add to the freespace function, but I have to figure out how best to handle that.

Update: 8/20/10

- Uses unbuffered I/O to overwrite the file

- Added fFileTime and fDelete parameters

- Code cleanup

EDIT: New version. The buffer creation has been optimized. This one optionally renames the file 10 times to a random name, resets the file timestamps to Jan. 1, 1980, 12:01am, and optionally takes a user array of integer values from 0 to 255 (decimal or hex) to use as the erasing patterns. This way the user can control the patterns used to erase the data and the number of passes. If no array is passed, then the default DoD short method (DoD_E) is used. The last parameter lets the user specify the block size to be used, default is 32768 bytes (32 Kb).

Just something I felt like whipping up. This script does a secure erase of any file. It uses the DoD short method as described in the script. Currently it writes in 32Kb blocks which seems to give pretty good performance. Depending on the file size, it may increase the erased file's size by up to 32 Kb. You can play with the size if you feel like it. On my computer, it erased a 5.5 MB file (16.5 MB in total written data) in 0.08 seconds. I verified the results with a Hex editor to be sure. Then it will rename the file 10 times using a randomly generated name. This *should* make the file names unrecoverable as well as the data.

Enjoy!

Example

$r = _SecureFileDelete(@ScriptDir & "test.bin", -1, False) ; use default patterns, do not delete
ConsoleWrite($r & " : " & @error & @CRLF)

Local $array[3] = [0x00,0xFF,0x1C]
$r = _SecureFileDelete(@ScriptDir & "test.bin", $array, False) ; use custom patterns, no delete
ConsoleWrite($r & " : " & @error & @CRLF)

Dim $array[2] = [0,-1]
$r = _SecureFileDelete(@ScriptDir & "test.bin", $array, False) ; two passes, last pass random data, no delete
ConsoleWrite($r & " : " & @error & @CRLF)

SecureDelete.zip

Edited by wraithdu
1 person likes this

Share this post


Link to post
Share on other sites



Nice to see this, You should add more methods :P

Would randomize the creation/modified/accessed dates add any security benefits?


Don't bother, It's inside your monitor!------GUISetOnEvent should behave more like HotKeySet()

Share this post


Link to post
Share on other sites

It's good to know how it's done.

Nice


♡♡♡

.

eMyvnE

Share this post


Link to post
Share on other sites

I suppose it could. Probably just easier to set it to something static like 19800101000001.

Re: methods...there's really no advantage to any method over another, and I think most experts agree that on modern drives one random pass is good enough. 3 to 7 passes is probably overkill, while Gutmann is just a waste of HDD cycles.

I have some thoughts about user options. Let me see what I like and I'll post an update.

Share this post


Link to post
Share on other sites

#5 ·  Posted (edited)

See first post Edited by wraithdu

Share this post


Link to post
Share on other sites

I've used DoD a thousand times before. Its safe to say that anything destroyed by that method is gone for good.

So cool to see the internals of that beast! Thanks for that! :P

Share this post


Link to post
Share on other sites

#7 ·  Posted (edited)

Updated the first post. I tried to optimize the buffer creation. This was the part that was taking time, not the actual data writing. Ironically, converting the patterns to strings then filling the buffer all in one shot was way faster than keeping all the data binary and filling the buffer by byte index -

For $i = 0 to 2
    For $j = 1 to 32768
        DllStructSetData($buff, 1, $aPatterns[$i], $j)
    Next
Next
Edited by wraithdu

Share this post


Link to post
Share on other sites

#8 ·  Posted (edited)

ah..beta

Edited by Creator

Share this post


Link to post
Share on other sites

Wha, beta??

Share this post


Link to post
Share on other sites

Update, see first post.

Share this post


Link to post
Share on other sites

Sorry for necropost, but I'd like some information; if I would use dod method 7 steps or gutman 35 steps??????

Share this post


Link to post
Share on other sites

"Studies have shown that most of today’s media can be effectively cleared by one overwrite..."

http://en.wikipedia.org/wiki/Data_remanence

Share this post


Link to post
Share on other sites

Updated, see OP.

Share this post


Link to post
Share on other sites

Share this post


Link to post
Share on other sites

One more update, hopefully the last for a while unless bugs are found.

Share this post


Link to post
Share on other sites

Updated. See OP and read UDF header for important info. WARNING - this version has a script breaking change!!

Share this post


Link to post
Share on other sites

This UDF has come on leaps and bounds. Thanks wraithdu.


_AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_ConvertLHWebColor()/_ConvertSHWebColor()_DesktopDimensions()_DisplayPassword()_DotNet_Load()/_DotNet_Unload()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource()_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUIRegisterMsg()_GUISetIcon()_Icon_Clear()/_Icon_Set()_IdleTime()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsSystemDrive()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringCompact()_StringIsValid()_StringRegExpMetaCharacters()_StringReplaceWholeWord()_StringStripChars()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_Create_CTL_CODE()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()Au3 script parsingAutoIt SearchAutoIt3 PortableAutoIt3WrapperToPragmaAutoItWinGetTitle()/AutoItWinSetTitle()CodingDirToHTML5FileInstallrFileReadLastChars()GeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIEventsGUIGetBkColor()Int_Parse() & Int_TryParse()IsISBN()LockFile()Mapping CtrlIDsOOP in AutoItParseHeadersToSciTE()PasswordValidPasteBinPosts Per DayPreExpandProtect GlobalsQueue()Resource UpdateResourcesExSciTE JumpSettings INISHELLHOOKShunting-YardSignature CreatorStack()Stopwatch()StringAddLF()/StringStripLF()StringEOLToCRLF()VSCROLLWM_COPYDATAMore Examples...

Updated: 04/09/2015

Share this post


Link to post
Share on other sites

Thanks. I usually dislike frequent updates, and especially script breaking ones, so I'm breaking my own rules. But I've been working on a script, the basis of which is this UDF, so I keep tweaking and refining based on the greater body of work. A lot of the optimization has grown out of the recursive directory function.... which is another broken rule. I usually don't write recursive functions for my UDFs, as I believe it should be left up to implementation. But this one has enough nuance that I thought it was worth including.

Share this post


Link to post
Share on other sites

Hello,

i just tried your script but cannot make it to work.

Here is the test .au , i used on a file test.bin inside the script directory.

<code>

#include <_SecureDelete.au3>

$r = _SecureFileDelete(@ScriptDir & "test.bin", True) ; use default patterns, do not delete

ConsoleWrite($r & " : " & @error & @CRLF)

Local $array[3] = [0x00,0xFF,0x1C]

$r = _SecureFileDelete(@ScriptDir & "test.bin", False, $array) ; use custom patterns, no delete

ConsoleWrite($r & " : " & @error & @CRLF)

</code>

I get

0 : 3

0 : 3

both times ,

file is not destroyed (monitored via a hex-editor)

i even tried True for deleting this file , neither deletes the file.

Also tried the same for "D:test.bin" with the corresponding changes on the .au script , neither saw anything changes.

Any ideas?

Share this post


Link to post
Share on other sites

Reread the information in the first post, he has modified the _SecureFileDelete function so that the parameters are not in the same place any longer. The example script in the first post has not been updated to reflect the changes made. You need to put the patterns array first, and then the option to delete the file or not.


If I posted any code, assume that code was written using the latest release version unless stated otherwise. Also, if it doesn't work on XP I can't help with that because I don't have access to XP, and I'm not going to.
Give a programmer the correct code and he can do his work for a day. Teach a programmer to debug and he can do his work for a lifetime - by Chirag Gude
How to ask questions the smart way!

I hereby grant any person the right to use any code I post, that I am the original author of, on the autoitscript.com forums, unless I've specifically stated otherwise in the code or the thread post. If you do use my code all I ask, as a courtesy, is to make note of where you got it from.

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.  -  ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.  -  Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.  -  _FileGetProperty - Retrieve the properties of a file  -  SciTE Toolbar - A toolbar demo for use with the SciTE editor  -  GUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.  -   Latin Square password generator

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0