Home » Learning Curve
OS X File System VoodooWhat was once a brilliantly simple system...
Unix at its inception is a brilliantly simple system. File management is divided into read, write, and execute permissions for all files. And directories are files too - everything in a Unix system is a file.
Using this RWX system was perfectly adequate for thirty years. But people watching Microsoft struggling with their own archaic system model (a Multics model which predates Unix) were too tempted to introduce Redmond plaster solutions for issues largely nonexistent on Unix.
There is strength in simplicity. There is elegance too. Complexity corrupts. Microsoft Windows/NT grows all the more complex and shows it. Even Unix systems across the board - Apple's is not the only example - are growing more complex, thus weakening the system as a whole and undermining the original philosophy of Bell Labs Unix creators Ken Thompson and Dennis Ritchie.
It's important as well to realise that complexity not only weakens a system, but also makes it more vulnerable to attack. Finding nooks and crannies that system engineers can't keep track of means interlopers get in.
Unwarranted complexity also makes administration more difficult or even fruitless. And as personal computer users are ipso facto their own administrators, it makes their lives more difficult as well.
The following is a short review of how Unix file systems work.
The Beginning
The Unix file system was designed to afford a modicum of privacy and protection for both users and the system itself. Each file and directory is assigned an access mode. This is known as mandatory access control - file system objects (files and directories) always have associated permission bits.
Files coming over from Windows (or disc distributions from corporations such as Adobe) often have their files marked '777' - full access for everyone, something very frowned on in the multiuser enterprise world of Unix.
Unix assigns read, write, and execute rights to the file's owner, the user group the owner assigns, and everyone else. These rights are normally expressed as a three-digit octal number where rights are added bitwise.
Default rights are often '644', meaning the owner can read and write but the group and everyone else can only read.
A read right is 4, a write right is 2, and an execute right is 1. So full access for everyone is 4 + 2 + 1 = 7 for owner/user, group, and other. As Windows doesn't have any such system by default, files come across to Unix systems as '777'.
Directories are files too. In fact, one of the corner precepts of Unix is that directories are files and no more. They're files used by the system for a specific purpose, but they're still files - 'streams of bytes'.
The execute bit is a bit puzzling for Unix neophytes. The file's owner can namely specify if the file can be 'executed' - if it can be run, as an application file, or a script, or whatever. One of the first tutorials in Brian Kernighan's excellent The Unix Programming Environment touches precisely on this subject. It's easy to learn how to write shell scripts, but it takes another step to actually run them.
Let's take a simple example. Let's create a script which simply lists the content of one's home directory, no frills.
Start a new text file - and be sure to save as text and nothing else - and put this and only this in it:
ls ~
Now save the file as 'list_home' on your desktop or in another directory of your choosing.
Now fire up Terminal.app and navigate to the directory you chose above. Then invoke the following command and see what happens. (Be sure to add './' to the beginning of the filename - it's for your security.)
$ ./list_home
-bash: ./list_home: Permission denied
The command interpreter - the shell - refuses to run your script, claiming permission was denied. Why? Because list_home does not have execute rights. This is easy to remedy for you the file owner.
$ chmod 744 list_home
chmod changes the mode (the permissions) of a file. Giving list_home a '7' means the file now has the execute bit (at least for you). Now try the command again.
$ ./list_home
Applications Downloads Music Sites
Desktop Library Pictures
Documents Movies Public
The execute right also applies to directories: it means the user can enter a directory. Note it might be possible to still list and even alter the contents of a directory, but setting the directory as one's 'working directory' will not be possible. Run the following command to see where you are at the moment.
$ pwd
Read and write rights also apply to directories. To understand their importance, it's necessary to first understand what those 'streams of bytes' actually contain. For essentially all they contain are filenames and system indexes to corresponding full data for each file.
That's right: directories contain no file system data save for filenames. All else is located elsewhere.
What does this mean in terms of read and write?
√ You have to have read permission to list the contents of a directory.
√ You have to have write permission to change the contents of a directory.
How does this - especially with regard to write permission - apply in everyday scenarios?
√ You can't delete a file if you don't have write permission to its directory.
√ You can't rename a file if you don't have write permission to its directory.
√ You can't add a file to a directory if you don't have write permission to the directory.
This applies further to ordinary copy and move operations.
√ You can't copy or move a file into a directory if you don't have write permission for the directory.
√ Neither can you move a file from one directory to another if you don't have write permission for both directories.
$ mkdir test // creates the directory 'test'
$ mv list_home test // moves 'list_home' to the directory 'test'
$ ls -l test // shows the 'long' listing of the contents of 'test'
total 8
-rwxr--r-- 1 osxuser staff 4 May 21 14:39 list_home // list_home is there
$ chmod 444 test // changes the permissions of 'test' to read only
$ cd test // attempt to change working directory to 'test'
-bash: cd: test: Permission denied
// you can't because 'test' lacks the execute bit
$ chmod 544 test // add the execute bit to 'test' for you the file's owner
$ cd test // now try again to change working directory to 'test'
// this time it works - 'no news' is 'good news' in Unix
$ ls // list the contents of 'test'
list_home // 'list_home' is the only file there
$ ls -l . // try the long listing
total 8
-rwxr--r-- 1 osxuser staff 4 May 21 14:39 list_home
$ ls -dl . // try the long listing for the directory 'test' itself
dr-xr--r-- 3 osxuser staff 102 May 21 15:14 .
$ mv list_home .. // now try to move 'list_home' back to its previous directory
mv: rename list_home to ../list_home: Permission denied
// doesn't work - 'test' doesn't have write permission
The directory 'test' was stripped of its write permission. Thus the attempt to move a file in that directory failed - the system would have to write to the directory itself, and the system was denied. The requisite execute permission was missing.
Note: this endemic propensity in Unix file systems plays a dirty trick on some of Apple's less than brilliant enhancements in recent years. Someone close to the Apple coders (perhaps their 'user experience engineers') complained of frustration trying to save files that lacked write permission. Given that Apple coders do all in their power to keep users away from system areas, the location of those files was most certainly the user home area, which meant that the complainers had themselves removed write permission, meaning they were complaining about something they themselves had done.
So Apple coders came up with one of the dumbest ideas ever in the world of Unix.
1. Start referring to files as 'locked' and 'unlocked'. (There is no such thing in Unix file systems.)
2. Don't tell the dimwit that the file lacks write permission - say instead the file is 'locked', then offer to 'unlock' it. It's all a lie.
3. At the user's behest, move the new file into its actual directory rather than writing directly to it as Unix users have done for more than forty years.
4. The file's permissions aren't changed. The next time the user wants to save, it's the same silly procedure all over again.
That's a cheat and a bad one. But it also kicks Apple upside the head.
Save a file to a directory, make sure the file is writeable, but remove write permission from its directory. Now try to save the file again.
'list_home' still has write permission, but because the Apple coders created a clown's solution to a nonexistent issue, nothing works as it should anymore. And the ultimate irony is that the dimwit who started it all is back to the same diagnostic message that was so infuriating in the first place.
And if that doesn't convince you that Apple system architects and 'engineers' are a bunch of idiots, then you need to read the above again.
But back on track. A Unix file system is incredibly powerful, mostly to the extent it remains eminently simple.
Now let's look at where feature creep starts ruining things.
Flags
Unix file flags didn't originate at Apple, but they were further perverted at Apple. The Rixstep application Xattrib is the best way to view what's going on.
There are currently seven (7) user flags and five (5) system flags. A few points:
- The user flag 'hidden' is an addition by Apple to signal their Finder to not list a file. (Why anyone would want to conceal things on OS X is another matter - it's ridiculous.)
- The 'immutable' flags prevent files from being modified.
- The user flag 'compressed' means the file has been compressed. (This flag cannot be modified through a user interface.)
- The 'no unlink' flags are important: they prevent a file from being deleted.
But how about that 'no unlink' flag? Is there no way to prevent deletion without it? Of course there is. Simply remove write permission from the file's directory and from that directory's parent directory. And so forth. It's a bit unwieldy but it works.
The 'immutable' flag is similar to removing write permission. Apple's brilliant 'lock/unlock' scheme will work on files so marked, but they can't be deleted. What the logic is here is anyone's guess.
The one thing these flags offer that ordinary Unix cannot offer is the single-user reset. Some of the system commands can only be reset in single-user mode: that is, before anyone at all has logged in. A claim that the feature is still interesting remains up for debate; otherwise there's little available to warrant further complexity.
Apple used to use the flags to help protect system files. Now they've switched to access control lists which effectively offer them no more than the flags. Access control lists (ACLs) are merely the flavour of the month.
Access Control Lists
Access control lists come to OS X via FreeBSD, but that doesn't certify their intelligence. Access control lists add yet another layer of complexity to a system that was pretty much fine beforehand, thank you.
ACLs may perhaps be best explained via the Rixstep application ACL.
The above screenshot shows ACL viewing the directory 'test'. We can see the filename, the owner, the group, the mode, and the flags. We see as well that one of the entries ('..' or the parent directory) is listed in an alternate colour. That means that selecting it in the upper pane will reveal more information in the lower pane The file namely has an access control list.
The lower pane tells us that 'test' has an access control entry that's specified in terms of a group, and that group is 'everyone', and that means exactly what it seems to mean.
The entry is a 'deny' rather than an 'allow' entry: it's going to limit access to the file system object. And what is denied? Deleting the file ('del').
Access control entries (ACEs) can be inherited. This is going to happen on directories rather than files. If the 'inherited' ACE is present on a directory, then subdirectories created in that directory will also adapt the same ACE.
Here's the data sheet for ACL. It shows all the possibilities for setting further gunk on objects in your file system.
An access control list contains one or more access control entries. There's no guaranteed method to the madness: ACEs can be added at any index, and there's no provision for removing duplicates. But the order of the ACEs is crucial: the system guarantees that it will only inspect the list (starting with index zero) until a resolution is found. The pertinent ACE is compared to the access mode the user requested. The system will return a 'yes' or a 'no' as soon as possible and won't worry what's found later in the list.
What This All Means
What this all means is we have three separate yet overlapping methods of access control. This makes administration of one's file system increasingly impossible.
Most of these new methods are unavailable to the end user. Software engineers will have tools such as these (or the equivalent) but users will not, thereby making it possible for the system to lay booby traps which make users perplexed and helpless.
The same pitfalls can victimise seasoned administrators as well. Consider the following situation.
$ cat list_home // output the contents of 'list_home'
ls ~ // the contents of 'list_home'
Now apply an ACE to 'list_home'.
Now try to output the file again.
$ cat list_home
cat: list_home: Permission denied
Check the file permissions. Note the new '+' which seems to mean 'there's more than meets the eye'.
$ ls -l list_home
-rwxr--r--+ 1 osxuser staff 5 May 21 17:55 list_home
Add a flag to see closer which other rules may apply.
$ ls -el list_home
-rwxr--r--+ 1 osxuser staff 5 May 21 17:55 list_home
0: group:everyone deny read
And yet the file is still marked as readable.
Now try the converse: remove all permissions from 'list_home' so that no one - not even you - can so much as read the file (much less modify it or execute it).
$ chmod 0 list_home
Check the permissions.
$ ls -l list_home
---------- 1 osxuser staff 5 May 21 17:55 list_home
No permissions whatsoever.
Try reading the file.
$ cat list_home
cat: list_home: Permission denied
Now apply some file system voodoo and try again.
$ cat list_home
ls ~
The file can be read despite it having no file permissions whatsoever.
There are myriad ramifications involved, all exacerbated by the fact that ACLs aren't normally visible to the end user. Couple this with the ability to set ACEs that are inherited down the line - without any indication where they come from in the first place - and you have a situation ripe for chaos.
Standard file attributes say a file can't be read, yet it can. Standard file attributes say a file can be read, yet it can't. And so forth.
What does an end user do if a file simply can't be deleted? Would this perhaps be a system file flag that can only be reset in single-user mode? What does the end user know about single-user mode? Would the typical end user even know how to get things into single-user mode?
What opportunities open to the black hats?
Remedies
There are clear issues here which need to be remedied.
√ Design the system with a single layer of access control. Make sure they don't overlap. To the extent multiple layers are needed, make sure the system is well documented so the user (administrator) knows which layers override others. It makes no sense (and is potentially dangerous) to allow access to objects for which access is denied through standard file permissions.
√ Give users a fighting chance. The black hats and skiddies will have all the tools, end users won't. If anyone or anything can pervert your file system, demand your vendor (eg Apple) provide you with the tools to both see and remedy what's going on.
√ Demand your OS vendor (eg Apple) be less secretive and fully document how their system works.
√ Take the time to learn yourself how your system really works.
The UNIX and the Echo
There dwelt in the land of New Jersey the UNIX, a fair maid whom savants travelled far to admire. Dazzled by her purity, all sought to espouse her, one for her virginal grace, another for her polished civility, yet another for her agility in performing exacting tasks seldom accomplished even in much richer lands. So large of heart and accommodating of nature was she that the UNIX adopted all but the most insufferably rich of her suitors. Soon, many offspring grew and prospered and spread to the ends of the earth.
Nature herself smiled and answered to the UNIX more eagerly than to other mortal beings. Humbler folk, who knew little of more courtly manners, delighted in her echo, so precise and crystal clear they scarce believed she could be answered by the same rocks and woods that so garbled their own shouts into the wilderness. And the compliant UNIX obliged with perfect echoes of whatever she was asked.
When one impatient swain asked the UNIX, 'Echo nothing', the UNIX obligingly opened her mouth, echoed nothing, and closed it again.
'Whatever do you mean', the youth demanded, 'opening your mouth like that? Henceforth never open your mouth when you are supposed to echo nothing!' And the UNIX obliged.
'But I want a perfect performance, even when you echo nothing', pleaded a sensitive youth, 'and no perfect echoes can come from a closed mouth.' Not wishing to offend either one, the UNIX agreed to say different nothings for the impatient youth and the insensitive youth. She called the sensitive nothing '\n'.
Yet now when she said '\n', she was not really saying nothing so she had to open her mouth twice, once to say '\n' and once to say nothing, and so she did not please the sensitive youth, who said forthwith, 'The \n sounds like a perfect nothing to me, but the second one ruins it. I want you to take back one of them.' So the UNIX, who could not abide offending, agreed to undo some echoes, and called that '\c'. Now the sensitive youth could hear a perfect echo of nothing by asking for '\n' and '\c' together. But they say that he died of a surfeit of notation before he ever heard one.
|