On File Management (3)

Part three.

Get It Try It

File management and file systems have never been Apple's strong suit. Apple have been good or even excellent at a number of things through their corporate history, but file management isn't one of them.

Their first file system for the original Mac - created to support a hierarchical paradigm - was flat. 'MFS' ('Macintosh File System') had to be abandoned. HFS ('Hierarchical File System') was born.

There have been any number of further file management scandals at Apple over the years - this in contrast to an industry where such things almost never occur. There's Tom Karpik's discovery of the 'massive data loss' bug as one example and there's the infamous Oompa Loompa as another.

The 'massive data loss' bug exposed something very wrong in Apple file management. The initial assumption was that the 'bad code' was inside Apple's Finder, but that's never been proven. It's a case of 'you're damned if you did and you're damned if you didn't'. If the flawed code was truly inside Finder.app, then Apple engineers were guilty of putting file system code in the worst place possible; and if it wasn't in Finder.app, then the entire file system was in danger. And if that fix (there are those who claim there was one) only took care of Finder.app, then the system's still in trouble.

Oompa Loompa exposed a number of weaknesses in Apple file system design. It got through to systems by using a design flaw and it propagated and kept out of sight by using another.

No one would have ever suggested running a nontrivial network with old Macintosh computers. Today, as Apple move towards more and more simplicity with their iPhones and iPads, the software in the world around, run by server parks which truly make the world around turn, grows more and more complex. Apple are incapable and seem unwilling to address that fact.

It took several years for Apple to get around to stopping their Finder from dropping .DS_Store files in networks - something that drove network administrators to distraction. The move to 10.6 came with a flawed interface for Windows file systems which got network admins in trouble with their bosses and in several reported cases left without their usual productivity bonuses.

The idea of administering a computer network with Apple's Finder simply doesn't occur. In the world of the corporation that made the graphical user interface a household mainstream commodity, professional IT staff are - barring access to alternative non-Apple tools - forced to go to the command line, something rather bizarre and not particularly welcome or efficient.

A good file system protects itself. A good file system is an entity unto itself and answers only to itself. It's a computer within the computer, so to speak. No application - and not even the OS kernel itself - can get at it directly. Operations are carried out through requests. Neither unprivileged nor privileged code can get at the actual file system code. But such is not the case with Apple's OS X. The Apple engineers and Apple fanboys might rise in indignation against such a claim but that changes nothing. For that's the way it is and that's the way it's always been.

For years the official Apple documentation for the patchwork HFS APIs was riddled with dire warnings to be careful, as flawed calls could potentially hose the entire file system. A self-respecting file system would never allow such a thing.

The first-ever release of Safari hosed file systems. Apple caught the bug in Safari within hours, fixed it, and re-released Safari with the same version number. They've never owned up to what they've done, but posters on the Apple discussion forums were able to compare message digests from downloads at different times and discovered the Safari binary had indeed been fixed.

But a coding flaw in Safari should never jeopardise a file system. A file system should toss out flawed requests and refuse to deal with them. Not so Apple's file system.

November 2007

November 2007: the OS X version of the day is 10.5 Leopard, and Tom Karpik has been privy to a massive data loss 'bug'.

http://tomkarpik.com/articles/massive-data-loss-bug-in-leopard/

The 'update' at the top of the article states:

'Update: The bug occurs regardless of the type of destination being moved to (whether it's local USB, local Firewire, SMB, etc). Also, I have been informed that this bug goes back all the way to Panther.'

Robert Rodgers adds:

'I accidently chmod'd a directory while a massive (40+ GB) file move was being copied into it to be -w by me, files went poof.'

These are the types of things file systems are supposed to protect against. They're supposed to be designed to protect not only against user error but also against programming error. Neither Finder.app nor any other application is supposed to have any means to interfere.

The general situation is exacerbated by virtue of the fact that Apple and Apple alone have chosen all along to use a 'replace' paradigm on file operations, this in contrast to the industry standard today called the 'merge' paradigm.

File operations aren't supposed to destroy existing system objects unless specifically instructed by the user. And this holds recursively down the entire hierarchy chain. Both copying and moving are supposed to respect existing directories, users are supposed to be prompted to replace existing files (but not directories) and source files with no counterpart at the destination are supposed to be copied/moved in as the case may be. The user is supposed to be kept in the loop throughout the complete traversal. Not so with Apple: the topmost node at the source is removed or not and its counterpart at the destination is copied or moved in.

This is clearly explained in the article at the link below. Apple's 'replace' paradigm forces applications into the types of scenarios where the Safari bug can occur (and did occur). Further exacerbating the situation is how application code - and not the file system - has to watch out for file type incompatibility: one can't go replacing an entire directory hive with potentially thousands of files with a single download file (which is what happened to entire user home directories with the Safari bug).

http://rixstep.com/2/2/20071106,00.shtml

Back to the Karpik narrative.

'Leopard's Finder has a glaring bug in its directory-moving code, leading to horrendous data loss if a destination volume disappears while a move operation is in action. I first came across it when Samba crashed while I was moving a directory from my desktop over to a Samba mount on my FreeBSD server.'

There's a way to perform file moves and the above isn't it. File moves aren't really moves at all. The files themselves don't move - the references to them do. A move is simply a variant of the file rename. Moving file 'a' to 'b' means the reference to 'b' gets replaced by the old reference to 'a', the file 'b' is removed if it exists, and the old reference to 'a' is removed as well.

The above holds for cases when both the source and destination are on the same volume. This is necessary because each volume has its own volume control block (the generic term) which is called the ilist in Unix. (HFS implements things differently but supports the Unix paradigm as much as it can with its catalogue file.)

The 'VCB/ilist/catalogue file' contains information on all files resident on the volume. It has no knowledge of other volumes or the files on them.

The actual physical locations of the chunks of the file (extents - devolving down to disk drive locations expressed as head, cylinder, and sector) don't need to change because a file is being 'moved' from one directory to another - all that needs is as outlined above: the place where the file is referenced.

But things change when moving files across volume boundaries. Unix has a special way of dealing with this in the code to the command line tool mv: when the destination is found to reside on another volume, the source data must be first copied to the destination volume, and if an integrity check indicates the copy was successful, then and only then can the source be removed.

It's of course imperative the above steps are carried out in the precise order given and that the removal of the source takes place if and only if the destination is found to be intact and integral and byte for byte identical with the source. (The NeXTSTEP code contains excellent methods for performing such comparisons.)

This is what should have happened (but did not happen) on Tom Karpik's system.

All good code is always ready for exceptions as well. A structured exception handling system is vital to the survival of a file system. Anything can happen during a file operation. Use your imagination.

1. A copy is being performed between a server in Tulsa Oklahoma and another in Vladivostok Russia and a rodent eats through a cable or a satellite shuts down during the transmission.
2. There's a blackout (loss of power) at either end during transmission. Someone trips over a power cable. One of the two cities loses a power grid.
3. There's a brownout (temporary interruption of power) at either end during transmission.
4. There's a computer hardware failure at either end.
5. Somebody pulls out the destination USB drive.
6. Another process goes in and makes the destination unwritable.
7. Another process goes in and makes the source unreadable.

File systems are created to detect such unexpected occurrences or exceptions. A structured method of handling exception lets code anywhere up the call chain inspect the data to see what went wrong and if possible correct it.

Norwegian 'Henrik' explains in a comment posted 27 May 2010 that the same bug seems to exist in Snow Leopard as well.

'I have the newest update of snowleopard and this problem is still not fixed. The reason why it doesn't work is the following: When mac OSX are moving/coping files it starts with making an 'placekeeper/repository' for the file before it actually moves/copies the file into this 'placekeeper'. The mounted folder which you are trying to copy has settings activated in a way that gives user no right to edit/delete files after they are placed there. As you can understand the problem is then that when mac OSX is first making this 'placekeeper' for the file, before starting the acutal datamoving, the 'placekepper' file is allready locked, and the real data cannot be transferred into the 'placekeeper'. All you have to do is to change the right of the files in the serverdirectory in a way that makes everybody able to edit/change and delete files. Then it will work i think. Sry for my bad english, I am Norwegian.'

'Pipo' adds the following.

'I strongly do NOT recommend to use Mac OS X Leopard in a productive environment! I was moving 31 GB of image data (over 19,000 JPEG images) from a Windows Vista SMB-share to an external USB-drive connected to my Mac Mini with Leopard 10.5.1 on it. It's a Samsung MP0804H 80GB disk inside an external Revoltec-case, formatted with HFS+ (case-sensitive & journaled). It looks like all data has been successfully moved without any vanished files. But when I look at those fotos, 5% of them were garbled. I could look at those pictures in Quick Look, Finder, Photoshop CS3, Illustrator CS3, iPhoto - no matter in which application I was looking at them, they were really messed up. Part of the picture was covered by pink stripes. Also, while importing those 19,000 images to iPhoto '08 7.1.1 it reported over 500 defective images.'



What does this mean? It means that no OS X code can be trusted to perform moves across volume boundaries. The risk is proportional to the size of the data being moved - the longer the operation takes, the more opportunity for something bad to happen, something from which Apple code might not be prepared or designed to extricate you.

It's important to understand even legacy 'Unix' code (in command line tools such as cp and mv) can be compromised. A good way to demonstrate this is to copy a file with known extended attributes: if the extended attributes follow along, then Apple's diligent engineers have of necessity altered the original Unix code. And therefore that code can't be trusted either.

Code for the Unix command line tool cp has been capable of copying both ordinary file data and Apple extended attributes since at least version 10.4; therefore the code must be suspect. Today Apple use an even lower level API to do the bidding of Unix: copyfile(). Comments from the source code to copyfile() indicate there have been difficulties getting everything up to snuff.

/*
 * copytree -- recursively copy a hierarchy.
 *
 * Unlike normal copyfile(), copytree() can copy an entire hierarchy.
 * Care is taken to keep the ACLs set up correctly, in addition to the
 * normal copying that is done. (When copying a hierarchy, we can't
 * get rid of the 'allow-all-writes' ACE on a directory until we're done
 * copying the *contents* of the directory.)
 *
 * The other big difference from copyfile (for the moment) is that copytree()
 * will use a call-back function to pass along information about what is
 * about to be copied, and whether or not it succeeded.
 *
 * copytree() is called from copyfile() -- but copytree() itself then calls
 * copyfile() to copy each individual object.
 *
 * XXX - no effort is made to handle overlapping hierarchies at the moment.
 *
 */

#ifdef NOTYET
    // This doesn't handle filesystem crossing and case sensitivity
    // So there's got to be a better way

    /*
     * To work on as well:
     * We have a few cases when copying a hierarchy:
     * 1) src is a non-directory, dst is a directory;
     * 2) src is a non-directory, dst is a non-directory;
     * 3) src is a non-directory, dst does not exist;
     * 4) src is a directory, dst is a directory;
     * 5) src is a directory, dst is a non-directory;
     * 6) src is a directory, dst does not exist
     *
     * (1) copies src to dst/basename(src).
     * (2) fails if COPYFILE_EXCLUSIVE is set, otherwise copies src to dst.
     * (3) and (6) copy src to the name dst.
     * (4) copies the contents of src to the contents of dst.
     * (5) is an error.
     */
/*
 * the original copyfile() routine; this copies a source file to a destination
 * file. Note that because we need to set the names in the state variable, this
 * is not just the same as opening the two files, and then calling fcopyfile().
 * Oh, if only life were that simple!
 */

    /*
     * COPYFILE_PACK causes us to create an Apple Double version of the
     * source file, and puts it into the destination file. See
     * copyfile_pack() below for all the gory details.
     */

    /*
     * COPYFILE_UNPACK is the undoing of COPYFILE_PACK, obviously.
     * The goal there is to take an Apple Double file, and turn it
     * into a normal file (with data fork, resource fork, modes,
     * extended attributes, ACLs, etc.).
     */

    /*
     * If we have quarantine info set, we attempt
     * to apply it to dst_fd. We don't care if
     * it fails, not yet anyway.
     */

    /*
     * COPYFILE_XATTR tells us to copy the extended attributes;
     * this is seperate from the extended security (aka ACLs),
     * however. If we succeed in this, we continue to the next
     * stage; if we fail, we return with an error value. Note
     * that we fail if the errno is ENOTSUP, but we don't print
     * a warning in that case.
     */

    /*
     * Simialr to above, this tells us whether or not to copy
     * the non-meta data portion of the file. We attempt to
     * remove (via unlink) the destination file if we fail.
     */

    /*
     * COPYFILE_SECURITY requests that we copy the security, both
     * extended and mundane (that is, ACLs and POSIX).
     */

copyfile() is an extremely complex piece of code that takes into account all Apple's file system 'enhancements': extended attributes, Finder info, resource forks, file flags, access control entries. The logic is propagated upwards into cp and mv, into CoreFoundation and legacy Carbon file operations, and NeXTSTEP's Cocoa. No code on an OS X box can be trusted.

Is there a way out of this? Yes, but it's makeshift and a bit uncomfortable. A number of people in Tom Karpik's thread suggest doing precisely what the Unix mv would do - manually. And it's more or less foolproof.

Remember to not perform nontrivial file moves across volume boundaries. When required to do so, then do the following.

1. Manually perform the file operation first as a file copy.
2. Use a tool afterwards to compare source and destination recursively byte for byte.
3. Remove the source if and only if source and destination completely match.

The greater the amount of data, the greater your risk, the greater your potential loss.

See Also
Learning Curve: On File Management (1)
Learning Curve: On File Management (2)