|Home » Learning Curve
The Secure Delete Hoax: The Ins and Outs of Privacy
Mac users are becoming more and more sensitised to privacy - and in that vein, a number of rainmaker products have entered the market. So beware.
Why Deleting Doesn't Do It
Deleting doesn't do it because nothing is deleted. The disk space is sent back to the system, but the actual bytes on disk haven't changed. The file system has to waste time writing your file to disk, but it doesn't have to waste time 'unwriting' it. It simply makes it available to the next best caller.
Any intelligent program can look through disk free space and find and reconstruct your files. It's as simple as that.
Think White Board
The best way to visualize this is to think in terms of a white board - with an eraser that just isn't good (clean) enough. You can get most of the old stuff off there, but it will smear; and even though you can read what someone writes on it, you can also see what's behind it - what used to be there.
|The recovery of at least one or two layers of overwritten data isn't too hard to perform by reading the signal from the analog head electronics with a high-quality digital sampling oscilloscope, downloading the sampled waveform to a PC, and analysing it in software to recover the previously recorded signal. What the software does is generate an 'ideal' read signal and subtract it from what was actually read, leaving as the difference the remnant of the previous signal.|
- Dr Peter Gutmann PhD, University of Auckland
Disks aren't white boards, but the analogy is perfect: Writing new data on a hard drive minimizes the 'old data' to the point where your file system will ignore it. But forensic hardware used against you will be able to read through.
Why Overwriting Doesn't Do It
It's hardware - and not software - that is used by the serious players to reveal secrets on your hard drives. Software stuff like EnCase is for kiddies - for cops who like to pretend they're J Edgar Hoover. The 'software' tools are easily defeated - it's the hardware tools you have to watch out for. They're merciless.
They work by constructing a signal map of your entire hard drive, and then comparing this map, byte for byte, with what should be there. From the discrepancies they can backtrack several overwrites to find out what you had before you overwrote your files - several overwrites ago. Yes, it's that scary.
The Hoax With DOD/NSA 'Secure' Delete
There is no such thing as 'DOD' or 'NSA' secure delete for hard drives. There never has been and there never will be. On this the DOD and the NSA are unequivocal: The only secure delete for permanent media such as computer hard drives is incineration. Companies boasting use of 'DOD-approved' or 'NSA-approved' methods are citing standards for something entirely different - volatile memory - a subtlety they don't bother telling you about - naturally.
What the DOD and NSA sometimes recommend is that memory - RAM chips - be overwritten several times with Mickey Mouse algorithms as an extra precaution. They are not so dumb as to recommend something so simple for hard drives. (For that matter, neither the DOD nor the NSA are so dumb as to tell you what secure delete algorithms they really use - think about it.)
The only reason you find the DOD and the NSA quoted is because
- it sounds good; and
- the programmers can't do any better.
Otherwise, it's a complete hoax. The only approved method for secure delete is the Gutmann method - a method so difficult to understand, and even more difficult to implement, that most companies don't dare try. But if you really need secure delete, it's the only way to go.
Dr Peter Gutmann of the University of Auckland (NZ) made an exhaustive study of hard drive media a few years back. His analysis of their encoding techniques became the basis of his approach to secure deletion. His research is today the 'industry standard' work on the subject.
Dr Gutmann is also quick to understand that 'MILITARY', 'NISPOM', 'DOD' and 'NSA' algorithms are at best just so much smoke-screening, and for precisely the reason cited above, and therefore focused his research on what he knew the DOD and the NSA had to focus on: actual physical recording technology characteristics.
How Hard Drives Work
Every hard drive type uses a different electronic method to store and fetch data. Also, there are tons of discrepancies in this rather 'antiquated' technology: Drives can store things between tracks as well as on them, or at the end of tracks where normally nothing is ever stored, and so forth. And expert forensic hardware can get at this data.
The Gutmann method combines data about all the weaknesses of all hard drive types, noting their cyclical tendencies, the algorithms they use to economically store data, etc., and from this builds a 35-step plan to make it nigh on impossible to extrapolate back.
The Gutmann method is not easy. Its preliminary steps alone represent more shredding than 'DOD-approved' or 'NSA-approved' secure delete will ever give you - and at that stage, the Gutmann method has hardly begun.
The Gutmann method starts with four random overwrites - this just to 'shuffle the cards' a bit. After these follow twenty seven (27) calculated steps which target all the hard drive types in the world - and here Peter has combined algorithms, 'killing two or more birds with one stone' so to speak, to keep the number of steps down to twenty-seven.
The agenda for what needs to be done within these twenty-seven steps is fairly straightforward, but to keep one step ahead of the oppostion, their exact order must be randomised and fully obfuscated - and the method used to randomise their order must also be obfuscated.
After these twenty-seven steps are completed, four final random overwrites follow.
At this point, after your file has been wiped thirty-five times, you are pretty secure. (You would not be if you had relied on 'DOD' or 'NSA' - you'd be toast instead.) But one thing remains: The actual file name. The file name itself can be incriminating, and so it must be obfuscated as well.
And as an added precaution, the file should be truncated to 'zero size', so no data about its previous whereabouts remains in the system. All told, these last precautions bring the grand total up to thirty-nine.
The Good Doctor Speaks
In his afterthoughts, Dr Gutmann correctly points out that no hard drive needs all twenty-seven middle steps; the problem is finding an algorithm for just your hard drive. As your hard drive will fall into one of the existing categories, and as software must be able to address all categories, the extra effort will ensure that no matter what recording technology is used by your hard drive, the data will be securely deleted.
Flushing To Disk
A final problem arises through the modern method of caching intermediate values before storing on physical media - a so-called 'lazy write'. This method is used to save the file system undue trouble and wear and tear. Stores to disk are only written when necessary, and can remain in an intermediate 'cache' area long after the file write is completed.
In other words, any attempt at secure delete is useless if the overwrite data cannot be forced out of cache and onto disk. If it is not forced to disk, all that will be written will be the final step - and then once again you will be 'toast' if your drive is investigated with forensic hardware.
The techniques used to override the 'lazy write' settings differ from file system to file system. But whatever they are, they must be implemented, or else all of the above is for naught.
Combing The Market - The PC
Most PC products claiming to offer 'secure delete' do nothing of the sort. Many products claim to offer 'government approved' or 'DOD approved' or 'NSA approved' shredding, but do nothing of the sort. The PC market has long been a haven for amateurs and charlatans out to make a quick - and dishonest - buck.
It's not enough to have secure delete on a PC anyway. Windows has its Registry with its countless nooks and crannies, and many products - notably Eraser - don't do anything about that part of privacy at all.
Combing The Market - The Mac
The Mac platform has remained gratefully sheltered, and yet the same issues will soon pertain: Is there someone in the vicinity of your computer capable of accessing proprietary information? Do you have any software on disk that might 'phone home' with this information, in much the same way as the notorious Windows Update facility?
A few products for the Mac claim to offer secure deletion - the same applies to them as above. None go the whole route; none attempt to implement 'Gutmann'; most cite the acronyms 'DOD' and 'NSA' and 'MILITARY'.
For an enlightening look at just how crooked this market is, check out the screenshot below - it's for Windows & Internet Cleaner Pro, a typical 'shredder'.
It looks cute - for Windows. Let's take a closer look at what is there.
A Closer Look
From the Windows & Internet Cleaner Pro preferences:
|Overwrite all bytes of file with character zeros||Overwrite times:|
|Overwrite all bytes of file with character ones||Reset file to zero bytes|
|Overwrite all bytes of file with random numbers|
|Note: Each overwrite represents one pass of erasing.|
It's cute. It looks cute. But wait a minute here - 'character ones'? What are 'character ones'? Are they the character '1' - in which case they have a value of 31h - or are they binary ones (ouch) - in which case they have a value of 01h?
What the program author(s) seem to be assuming is that, inasmuch as everything is supposedly binary in the world of computers, that 'character ones' and 'character zeros' are opposites - when of course they are not.
Domain name- neoimagic.com
Start of registration- 06/08/02 13:11:24
Registered through- 06/08/05 13:11:24
HuiQiang Huang (firstname.lastname@example.org)
No.502 room, 2 unit, 606 building, ShengLiXiaoQu
Windows & Internet Cleaner Pro is only one glaring example. Most are in this class or even worse.
Remember: No 'overwrites' of any sort - so-called 'military grade', DOD, NISPOM, NSA - are going to do it. Those standards are for memory chips, not hard drives - a subtle little point the vendors of these products don't want to tell you about.
SafeShred Xtreme is a product for the Mac, written along the same lines as Windows & Internet Cleaner Pro: it offers stylish doodads to click on, but not enough sophisticated shredding to thwart even the Keystone Cops. (In fact, the SafeShred Xtreme website has a disclaimer about how Norton UnErase can recover files shredded with the product.)
The following image explains just what SafeShred Xtreme does and does not do - all you need is to keep your wits about you, and a single grain of salt.
There are five mutually exclusive shredding options in this program:
- SafeShred Default (1010..., then 0101...)
- DoD 5220.22M (Use a character, its complement, then a random character)
- Random (Use random characters)
- All 0's (0000...)
- All 1's (1111....)
The program can be configured to run the chosen shredding option up to twenty times.
The black on white description of what is supposed to be a US Department of Defence shredding standard has to take the cake. Anyone believing that the most powerful, most security-paranoid country on the planet really feels secure with shredding as described here needs to do some shopping at clueBay.com.
And - sorry, but 'twenty' iterations? Why stop at twenty? Is '20' some kind of magical number? The code only has to iterate the loop - there can be no additional overhead - so why not set the top limit at 1,000? Or 10,000? Or - aw heck - why not a cool million?
But it wouldn't matter. Foregoing the obvious first step in design of such a program, namely to research the subject matter, the authors of SafeShred Xtreme just 'wing it' - and leave you more vulnerable after using their program than before.
SPX is a shredding engine. It is a brutal, merciless shredding engine, and surprisingly is one of the few reliable products available on the market today on any platform. SPX is fully Gutmann-compliant.
SPX was written in direct response to the rainmakers flooding the market with their trademark tactic: Find someone who can write a few lines of Visual Basic, Delphi, REALbasic, or AppleScript, slap together a few pretty windows, then spam, spam, spam, scare and exploit. Clearly someone had to speak out against this.
SPX was originally written for the Windows platform and was featured in the October 2000 edition of Windows magazine. It has now been successfully ported to Mac OS X, where additional functionality has been added.
Do not rely on 'military grade', DOD, NISPOM, or NSA. If you really need to shred files, rely only on the best.
(Click here to visit the SPX page in the AppleCore Project.)
(And if you run a PC, look into SPX and the accompanying E3 Security Kit for Windows.)