Jump to content



Photo

FileWipe - very fast (using memory mapped files + memset)

file wipe fast secure delete

  • Please log in to reply
20 replies to this topic

#1 Zedna

Zedna

    AutoIt rulez!

  • MVPs
  • 8,409 posts

Posted 03 May 2012 - 05:16 PM

This function rewrite whole content of given file by defined (implicitly 0x0) character and finally delete it, it's called as secure deleting.
This should be VERY fast!!

Just note that there is no error checking inside FileWipe()
Maybe I will post later also version with error checking.
You can comment line FileDelete($sFileName) to see how it's rewritten before deletion.

EDIT: There can be problems with too big files (>2GB or > amount of RAM) see post #8 by Kafu

AutoIt         
#include <WinAPI.au3> ; prepare testing file FileDelete('1.txt') FileWrite('1.txt','123abc') FileWipe('1.txt') Func FileWipe($sFileName, $nByte = 0x0)     If Not FileExists($sFileName) Then Return     $iSize = FileGetSize($sFileName)     $hFile = _WinAPI_CreateFile($sFileName, 2, 6)     $hMapping = _WinAPI_CreateFileMapping($hFile)     $pAddress = _WinAPI_MapViewOfFile($hMapping)     MemSet($pAddress, $nByte, $iSize)     _WinAPI_UnmapViewOfFile($pAddress)     _WinAPI_CloseHandle($hMapping)     _WinAPI_CloseHandle($hFile)      FileDelete($sFileName) EndFunc Func MemSet($pDest, $nChar, $nCount)     DllCall("msvcrt.dll", "ptr:cdecl", "memset", "ptr", $pDest, "int", $nChar, "int", $nCount)     If @error Then Return SetError(1,0,False)     Return True EndFunc ; from WinAPIEx - just simplified for this purpose Func _WinAPI_CreateFileMapping($hFile)     Local $Ret = DllCall('kernel32.dll', 'ptr', 'CreateFileMappingW', 'ptr', $hFile, 'ptr', 0, 'dword', 0x4, 'dword', 0, 'dword', 0, 'ptr', 0)     If (@error) Or (Not $Ret[0]) Then Return SetError(1, 0, 0)     Return $Ret[0] EndFunc Func _WinAPI_MapViewOfFile($hMapping)     Local $Ret = DllCall('kernel32.dll', 'ptr', 'MapViewOfFile', 'ptr', $hMapping, 'dword', 0x6, 'dword', 0, 'dword', 0, 'dword', 0)     If (@error) Or (Not $Ret[0]) Then Return SetError(1, 0, 0)     Return $Ret[0] EndFunc Func _WinAPI_UnmapViewOfFile($pAddress)     DllCall('kernel32.dll', 'int', 'UnmapViewOfFile', 'ptr', $pAddress)     If @error Then Return SetError(1, 0, 0)     Return 1 EndFunc


Here is my first test version without memory mapped files:
Func FileWipe($sFileName, $nByte = 0x0)     If Not FileExists($sFileName) Then Return     $iSize = FileGetSize($sFileName)     $tBuffer = DLLStructCreate("byte[" & $iSize & "]")     MemSet(DLLStructGetPtr($tBuffer), $nByte, $iSize)     $hFile = _WinAPI_CreateFile($sFileName, 2, 6)     _WinAPI_WriteFile($hFile, DLLStructGetPtr($tBuffer), $iSize, $iSize)     _WinAPI_CloseHandle($hFile)     FileDelete($sFileName) EndFunc

Edited by Zedna, 07 May 2012 - 07:27 PM.








#2 Zedna

Zedna

    AutoIt rulez!

  • MVPs
  • 8,409 posts

Posted 03 May 2012 - 07:04 PM

I could swear I read somewhere that overwriting a file with Null characters does absolutely nothing, but I can't find anything to support that statement. I did find this though. http://www.planetsourcecode.com/vb/scripts/ShowCode.asp?txtCodeId=29245


1) absolut nonsense :-)
There are several wipe methods, some of them rewrite file several times with different and/or random data, this my method is basic one.
You can easily accomodate it to your needs.

2) your link is obsolete, http://www.planetsourcecode.com/vb/scrip...ts/ShowCode.asp?txtCodeId=2924 --> anyway this code does nothing as I see in first quick look

Please post only if you know something about it. Thanks.

EDIT:
some links
http://en.wikipedia.org/wiki/Data_remanence
http://en.wikipedia.org/wiki/Data_erasure

Edited by Zedna, 03 May 2012 - 07:11 PM.


#3 knightz93

knightz93

    Seeker

  • Active Members
  • 23 posts

Posted 03 May 2012 - 07:16 PM

nice!! i search for secure deleting method but i didn't find it in Autoit.. Thanks for Zedna . so, file will be null before deleting and we can't recover this file again, right??
I will do my best in this forum my code : WindowsSwitcher3d()

#4 MvGulik

MvGulik

    Universalist

  • Active Members
  • PipPipPipPipPipPip
  • 2,795 posts

Posted 03 May 2012 - 08:19 PM

? ... Skeptic here! ... Its this code writing faster to disk than AutoIt itself will do?
(That is at least what it sounds like to me.)

#5 Yashied

Yashied

    Happy in Moscow

  • MVPs
  • 2,513 posts

Posted 04 May 2012 - 09:43 PM

@Zedna

You can use _WinAPI_FillMemory() or _WinAPI_ZeroMemory() instead of MemSet().

#6 ricky03

ricky03

    Polymath

  • Active Members
  • PipPipPipPip
  • 221 posts

Posted 07 May 2012 - 01:25 PM

Hello,

thanks for you code, very useful.

In the code without mapped files, there is a little mistake in the code :
$hFile = _WinAPI_CreateFile($sFileName, 2, 6)

What do you think about my code :
Func _FileWipeSecure($sFileName, $times = 4)   Local $iSize, $tBuffer, $hFile     Local $nByte[3] = [1,0x0,0x1]   if $times < 1 then $times = 1   If Not FileExists($sFileName) Then SetError(1,0,0)     $iSize = FileGetSize($sFileName)     $tBuffer = DLLStructCreate("byte[" & $iSize & "]") For $i = 1 to $times   MemSet(DLLStructGetPtr($tBuffer), $nByte[$nByte[0]], $iSize)   $hFile = _WinAPI_CreateFile($sFileName, 2, 6)   _WinAPI_WriteFile($hFile, DLLStructGetPtr($tBuffer), $iSize, $iSize)   _WinAPI_CloseHandle($hFile)   if $nByte[0] = 1 Then    $nByte[0] += 1   Else    $nByte[0] = 1   Endif Next     Return FileDelete($sFileName) EndFunc

Edited by ricky03, 07 May 2012 - 01:42 PM.


#7 Zedna

Zedna

    AutoIt rulez!

  • MVPs
  • 8,409 posts

Posted 07 May 2012 - 02:45 PM

@ricky03
Thanks for correction in variable name, fixed.
I don't like multiple overwriting, so I didn't do that. Anyway feel free to make it in your modifications.
I think memory mapped files will be much faster, maybe I will do some test scripts later.

Edited by Zedna, 07 May 2012 - 07:39 PM.


#8 KaFu

KaFu

    Hey, it's just me, KhaFoo...

  • MVPs
  • 3,194 posts

Posted 07 May 2012 - 03:04 PM

Hi Zedna,

I really like this one ;), but some points came to my mind. Better use _Winapi_FileGetSizeEx to pass the 2 GB limit. Also I would recommend to process larger files in a loop, as the files are mapped to the process address space and thus large files are likly to consume too much RAM (up to crashing the script). My post on "Using the CreateFileMapping function with an Offset" should contain all required details for that.

Regards

Edited by KaFu, 07 May 2012 - 03:04 PM.


#9 MvGulik

MvGulik

    Universalist

  • Active Members
  • PipPipPipPipPipPip
  • 2,795 posts

Posted 07 May 2012 - 03:41 PM

Other version. (although for all I know it might actually NOT overwrite the original file data. ... Disk cluster and such.)
Func WipeFile_Zeros($sFilespec)     If FileExists($sFilespec) Then         Local Const $FILE_EREASE = 2         Local Const $FILE_BEGIN = 0 ;; <constants.au3>         Local $iLenght = FileGetSize($sFilespec)         Local $hFile = FileOpen($sFilespec, $FILE_EREASE) ;; Binary(+16) don't seems to matter.)         FileSetPos($hFile, $iLenght - 1, $FILE_BEGIN)         FileWrite($hFile, Chr(0))         FileClose($hFile)     EndIf EndFunc


Err ... think I just got scooped.
Whats wrong with this code. ? (... fixed)

---

Note to self: forum (update) message might be a little older than you think.</constants.au3>

Edited by MvGulik, 07 May 2012 - 04:09 PM.


#10 MvGulik

MvGulik

    Universalist

  • Active Members
  • PipPipPipPipPipPip
  • 2,795 posts

Posted 07 May 2012 - 07:03 PM

(although for all I know it might actually NOT overwrite the original file data. ... Disk cluster and such.)

After a little testing. This seems to be exactly the case. ('seems': as in, although the test I did is leaving little room for any other conclusion, its also not 100% foolproof. And to boot its was only done on one windows version.)

... Bummer! ...

#11 Zedna

Zedna

    AutoIt rulez!

  • MVPs
  • 8,409 posts

Posted 07 May 2012 - 07:21 PM

@Yashied,Kafu

Thanks for the tips.
I will play with it maybe later.

Edited by Zedna, 07 May 2012 - 07:22 PM.


#12 Zedna

Zedna

    AutoIt rulez!

  • MVPs
  • 8,409 posts

Posted 07 May 2012 - 07:34 PM

nice!! i search for secure deleting method but i didn't find it in Autoit.. Thanks for Zedna . so, file will be null before deleting and we can't recover this file again, right??


Yes.

#13 wraithdu

wraithdu

    I am less fun than a twisted ankle.

  • MVPs
  • 2,140 posts

Posted 08 May 2012 - 02:17 PM

I wasn't going to pipe in, but here's my older implementation: http://www.autoitscript.com/forum/topic/82954-securely-overwrite-files/

I have a bunch of unpublished updates that use the work I did with the windows defrag api that can be used to really test this. That code can pull raw data directly from the disk sectors to verify overwriting of the file. I also use some neat tricks with the defrag api to overwrite EFS encrypted and OS compressed and sparse files. The thing with those is using CreateFile or mapped files does not overwrite the physical disk where the files actually reside. The new data is simply written somewhere else and the old data marked deallocated and lost to posterity.

At some point soon I hope to finalize what I've done and post an update. I really had meant to go through the SDelete source code to compare to my implementation... maybe I'll get to that as well.

#14 Zedna

Zedna

    AutoIt rulez!

  • MVPs
  • 8,409 posts

Posted 08 May 2012 - 03:35 PM

@wraithdu
Great!! I'm amazed.

#15 wraithdu

wraithdu

    I am less fun than a twisted ankle.

  • MVPs
  • 2,140 posts

Posted 08 May 2012 - 05:18 PM

I'll pop these in here for now for some preliminary review. There's still some debugging code in them, and the free space wipe is mostly untested - it's the main reason I wanted to review the SDelete source code. But it works, so any feedback is appreciated. If you have questions on usage, let me know. Check the utility functions in the _FileMapping UDF for functions to print the file map and read it back to a file - used to extract files directly from disk both before and after wiping for verification purposes (you'll need a hex editor for after ;) ).

Using the _FileMapping functions on compressed and EFS encrypted files is actually pretty neat, as you get to see the actual stored compressed/encrypted content which is usually transparent to the user.

Attached Files


Edited by wraithdu, 08 May 2012 - 05:19 PM.


#16 michaelslamet

michaelslamet

    Universalist

  • Active Members
  • PipPipPipPipPip
  • 294 posts

Posted 03 October 2012 - 08:13 AM

Hi Zedna,
Is this FileWipe function accept wildcard like *.txt for a paramter?
If no, is there any similiar function that accept wildcard as a paramter?

Thanks :-)

#17 guinness

guinness

    guinness

  • MVPs
  • 11,050 posts

Posted 03 October 2012 - 08:27 AM

Hi Zedna,
Is this FileWipe function accept wildcard like *.txt for a paramter?
If no, is there any similiar function that accept wildcard as a paramter?

Thanks :-)

Looking at the code I would say no, but why not try it first before asking.

Example List: _AdapterConnections()_AlwaysRun()_AppMon()_AppMonEx()_BinaryBin()_CheckMsgBox()_CmdLineRaw()_ContextMenu()_DesktopDimensions()_DisplayPassword()_Fibonacci()_FileCompare()_FileCompareContents()_FileNameByHandle()_FilePrefix/SRE()_FindInFile()_GetBackgroundColor()/_SetBackgroundColor()_GetConrolID()_GetCtrlClass()_GetDirectoryFormat()_GetDriveMediaType()_GetFilename()/_GetFilenameExt()_GetHardwareID()_GetIP()_GetIP_Country()_GetOSLanguage()_GetSavedSource_GetStringSize()_GetSystemPaths()_GetURLImage()_GIFImage()_GoogleWeather()_GUICtrlCreateGroup()_GUICtrlListBox_CreateArray()_GUICtrlListView_CreateArray()_GUICtrlListView_SaveCSV()_GUICtrlListView_SaveHTML()_GUICtrlListView_SaveTxt()_GUICtrlListView_SaveXML()_GUICtrlMenu_Recent()_GUICtrlMenu_SetItemImage()_GUICtrlTreeView_CreateArray()_GUIDisable()_GUIImageList_SetIconFromHandle()_GUISetIcon()_Icon_Clear()/_Icon_Set()_InetGet()_InetGetGUI()_InetGetProgress()_IPDetails()_IsFileOlder()_IsGUID()_IsHex()_IsPalindrome()_IsRegKey()_IsStringRegExp()_IsUPX()_IsValidType()_IsWebColor()_Language()_Log()_MicrosoftInternetConnectivity()_MSDNDataType()_PathFull/GetRelative/Split()_PathSplitEx()_PrintFromArray()_ProgressSetMarquee()_ReDim()_RockPaperScissors()/_RockPaperScissorsLizardSpock()_ScrollingCredits_SelfDelete()_SelfRename()_SelfUpdate()_SendTo()_ShellAll()_ShellFile()_ShellFolder()_SingletonHWID()_SingletonPID()_Startup()_StringIsValid()_StringReplaceWholeWord()_StringStripChar()_Temperature()_TrialPeriod()_UKToUSDate()/_USToUKDate()_WinAPI_CreateGUID()_WMIDateStringToDate()/_DateToWMIDateString()AutoIt SearchAutoIt3 PortableAutoItWinGetTitle()/AutoItWinSetTitle()CodingFileInstallrGeoIP databaseGUI - Only Close ButtonGUI ExamplesGUICtrlDeleteImage()GUICtrlGetBkColor()GUICtrlGetStyle()GUIGetBkColor()LockFile()PasteBinSciTE JumpSignature CreatorWM_COPYDATAMore Examples...Updated: 11/04/2013


#18 michaelslamet

michaelslamet

    Universalist

  • Active Members
  • PipPipPipPipPip
  • 294 posts

Posted 03 October 2012 - 11:58 PM

Looking at the code I would say no, but why not try it first before asking.


You're right, sorry, my bad :( thanks for remind :)

#19 nullschritt

nullschritt

    Adventurer

  • Active Members
  • PipPip
  • 129 posts

Posted 27 November 2012 - 10:56 PM

Is there a way to get this to write files over 2gb or ram?

My understanding is that any time you "map" the file, the whole file goes into memory, so any file over the maximum ram size would cause an error?

Is there a way to add data without loading the already existing content into memory?

I think this would be a great utility to use as a disk cleaner (map over all "free" space with empty bytes to perm. erase data)

If there is not a way, I suppose I could use a loop to make multiple 2gb files and delete them, but I think that may take longer.

#20 BrewManNH

BrewManNH

    באָבקעס מיט קודוצ׳ה

  • MVPs
  • 7,053 posts

Posted 27 November 2012 - 11:17 PM

I think this would be a great utility to use as a disk cleaner (map over all "free" space with empty bytes to perm. erase data)

http://www.dban.org/

How to ask questions the smart way!

Back up and restore Windows user files _Array.au3 - Modified array functions that include support for 2D arrays.ColorChooser - An add-on for SciTE that pops up a color dialog so you can select and paste a color code into a script.

Customizable Splashscreen GUI w/Progress Bar - Create a custom "splash screen" GUI with a progress bar and custom label.

_FileGetProperty - Retrieve the properties of a file SciTE Toolbar - A toolbar demo for use with the SciTE editorGUIRegisterMsg demo - Demo script to show how to use the Windows messages to interact with controls and your GUI.

GUIToolTip UDF Demo - Demo script to show how to use the GUIToolTip UDF to create and use customized tooltips.

Posted Image






Also tagged with one or more of these keywords: file, wipe, fast, secure, delete

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users