|Home » Learning Curve
File Management with a ♥
Get to know what you didn't know you don't know. Tap tap tap.
Tap tap tap. Things are getting easier. Sort of. Tap tap tap. But somebody has to do real file management. The pros have to do it all the time. And even the lowliest of the low have to do it on occasion.
Where will you be when that time comes? Because you can't do it on a Mac. At least not with your standard Mac equipment.
Or are you supposed to take your Mac to your genius bar and let someone you recognise from the drive-thru at Taco Bell perform hazardous operations and potentially screw things up more than you ever could?
You had to work harder for your driving licence. You had to learn the road signs and the brake times and you had to display a modicum of coordination and dexterity behind the wheel. Traffic authorities need you to be skilled and educated so you don't hurt others.
But when it comes to personal computers, the vendors want you as dumb as you can be. They love stupid. And if they can do anything about it, they'll make you even dumber.
The world of Apple is no exception. Tap tap tap.
There are so many nooks and crannies and secret hooks on your simple OS X Unix file system it's not funny. Actually there are fewer complications than on other systems. And the Unix file system is, if anything, consistent. It makes sense. It's not flippant or fetid.
The Unix file system - the file system you use on your Mac - is straightforward and eminently logical. To get an appreciation for how the Unix file system works, you have to go back to the concept days when it was just an idea.
Nobody made file systems like Unix back before Unix. No one. The most prevalent file systems in the world were those made by IBM for their mainframes. And you wouldn't want to navigate around them. Unix turned everything upside down, turned it all on its head.
Stream of Bytes
The first and most important characteristic of the Unix file system is that everything is a stream of bytes. The operative word is 'stream'. Prior to Unix, files could be random access or be made up of 'records' of a predetermined size. There was no such thing as opening a file and scooting to a specified offset. Not always. Unix changed all that. Starting with Unix, you opened any file the same way. Even directories.
OS X with HFS has a problem with directories but compatibility is still pretty good. It's namely possible to open directories as if they're ordinary files. See the ACP application GDE as an example.
GDE delves down to low level in the file system to see what's really there. What it gets back are records of data - not records as determined by the file system but records as determined by the software accessing the directory.
Each record in a directory has the following elements.
|d_ino||The inode number. We'll get back to what inodes are in a minute.|
|d_reclen||The length of the record. This helps the software calculate where the next record begins. Records are not all of the same size.|
|d_type||The type of entry. Directories have only file names and pointers to data. They don't have the data itself. And Unix has a number of file types. These aren't document types or in any way indicate how different files are opened. These are file types to be treated accordingly by the file system software itself. The screenshot shows two types: DT_DIR and DT_REG, standing for directory and 'regular' respectively. 'Regular' files are just what they seem to be - regular files.|
|d_namelen||The length in bytes of the filename. HFS uses UTF-8 for its filenames. The UTF-8 system allows all possible characters from Chinese to Thai and back to English, thanks to the founder of Unix, Ken Thompson, who invented the system for IBM back in the old days.|
|d_name||The filename itself.|
All a Unix file system needs to help you is the filename and the inode. You try to access files by name; the file system looks up the name and finds the corresponding inode. And the inode tells the system where to find the actual data - an eminently simple system. And if you don't believe that, then you need to try another file system sometime. Or even Apple's HFS at a low level. Unix spoils you.
Fortunately most file systems take after Unix today. Unix caused a revolution in computing, and that's a good thing. Even Microsoft took after Unix, although they insisted on twisting one character around and kept to their 'drive letters'.
Older systems such as those from DEC and MS-DOS and even Windows NT still use 'drive letters' corresponding to the actual physical storage devices connected to the machine. Unix works in a different way. Under Unix, all physical storage devices are connected into the same file system hierarchy. Look under /Volumes, if your Finder will take you there.
Files are stored on disk wherever the file system wants to store them. Disks are organised into heads and tracks and sectors, something that's of no relevance for users simply opening files.
But users have to have a way to organise their files, just as file systems do. That's where directories come in. The bonus with Unix file systems is that directories can be hierarchical. This was another novelty largely unheard of before Unix. So directories can be stored inside other directories. This is what makes the Unix file system so flexible, so dynamic.
For example, all Apple's system files are located in /System/Library. You'll mostly find subdirectories there - child directories of Library. Library is in turn a subdirectory of System. And System, strange as it may sound, is a subdirectory of /.
/ is the top-level directory on a Unix file system. It's often called 'root' as it's very much the root. See the file system as a tree's root structure. There's one directory at the top - the root. Then things can branch out from there. And every subdirectory of root (/) can have subdirectories of its own, and so on and so forth.
/ is not only used to denote the top-level root directory - it's also the path component separator. That's right: at the top level the '/' signifies the system's root directory, whilst otherwise it separates subdirectories and files from each other.
/System/Library has three path components. The file system can determine that Library is a directory by checking the file type: DT_DIR. That System is also a directory can be inferred or also checked the same way.
Check out this system file:
Glass.aiff is sound file. The Unix type is 'regular'. But it'll be opened by an appropriate application. So how would the application (and the system) find the file to open it?
- Find the root directory.
- Search inside root for System.
- Scoot to System and find Library.
- Scoot to Library and find Sounds.
- Scoot to Sounds and find Glass.aiff.
- Find an application that can open Glass.aiff.
- Invoke the application to open Glass.aiff.
Sounds complicated? Your Mac processes billions of instructions per second. A few steps to open a file is nothing.
Here's a system 'open' sheet for Glass.aiff. The system's determined what application is best suited to open the file. The system's already determined where the file is located. It's gone from root and through the other directories and found the file's inode. Now the file can be opened. Just click OK.
Now that you know what your Mac does to give you the files you want, it's time to look at how it determines what you can do with them. And now it's important to remember the environment the creators of Unix worked in - an environment not much different from any other environment, back then or today.
- Multiple users. Can there be multiple users on the same physical computer? Not back in the days of the beige Mac or the original IBM PC, no. But all professional systems have it, and they manage multiple users by requiring login credentials. You tell the system who you are and give some authentication such as a password and you get in. And the system remembers who you are for the duration of your stay.
- Shared resources. Multiple users means system resources - such as files - are possibly going to be shared by all those users. So your file TOPSECRET.PDF might be viewable for everyone logging in. Unless of course you can keep intruders out.
This connotes that the owner of a file must be able to set the permissions for all users for that file. And that's what owners do. They set file permissions in terms of reading, writing, and executing files.
Without read permission, you can't even see a file. Without write permission, you can't modify it. Without execute permission, you can't run a file if it's an application or a script file. These permissions have to be granted to you by the file's owner.
This capability, coupled with the capability to create user-specific areas of the file system, means all users on a Mac get their own home area (home directory) and can effectively lock the door and keep everyone else out. And there's no way short of hacking the entire system that anyone else can get in.
Here's full system info for Glass.aiff. We'll skip the fields that aren't pertinent.
|Device||We'll skip it.|
|Device Type||We'll skip it.|
|Inode||The system uses that number to access the file data.|
|Size||That's the size of the file in bytes.|
|Mode||Tote that this field can perhaps be modified. We'll be right back to Mode in a second.|
|Block||Another indicator of the actual storage required by the file. We skip.|
|Links||We skip (for now).|
|Optimal Block Size||We skip.|
|Owner||root is the top administrator on the system. (That's usually not you, even though you bought the machine.)|
|System/User Flags||We skip.|
|Group||All users fall into different groups. root's group wheel is very exclusive as is root itself.|
|File Generation #||We skip.|
Below that we see four time stamps. For the time the file was first created, when its system metadata was last changed, when the file was last modified, and when someone last opened it for any reason.
Unix mode fields have two values, usually concatenated together. They're stored as a single value.
The digits in the mode field are usually given in the octal radix, but if you don't know what that means, then it's no biggie at this point.
The leftmost two digits of the mode are clipped off in the screenshot. They indicate the system file type. Left are four digits indicating file permissions, what we want to concentrate on.
Each of the three rightmost digits specifies complete permissions for one category of user. Starting from the third digit from the right in the above screenshot:
|User||The owner. The user who created the file and thereby owns it.|
|Group||The group that owns the file, normally the owner's default group.|
The three file permissions are again:
So you add up the permissions you want. Their sum is the digit you give for permissions for any user category.
The above example in the screenshot tells us that the owner can both read and modify the file, users in the group wheel can only read it, and everyone else can only read it as well.
Glass.aiff cannot be 'executed'. The eXecute bit is very important on Unix. Brian Kernighan's seminal work The Unix Programming Environment goes through the creation of a script and how it's enabled for execution. Brian uses the command line to do that. Unfortunately, despite Apple's glimmering graphical user interface, it's impossible to do with their system file manager Finder. Earlier versions of Finder, still paraplegic in so many ways, were at least capable of this simple operation.
Today it's all tap tap tap.
Finder's Keepers: Losers & Weepers?
But that's not the end of the shortcomings of Apple's opinion of what you should know about your computer and what you should be able to do with your own data. Here's a short list of the important things Apple's Finder anno 2013 will not help you with.
- The eXecute bit. What they've tried to do at Apple is boil it all down to the Windows paradigms where it's not possible to prevent read access and there's no equivalent of the eXecute bit.
- No Write-Only. Apple have things like 'Read' and 'Read & Write'. Note that the file system permits write-only access - something that's eminently useful - but Apple's file manager won't.
- The System/User Flags. These were skipped on the first run-through but they're very important. Do you have quirky things happening to your files and your Finder won't tell you what's wrong? The answer could be here. Look at the screenshot and try to digest what's there.
- The file type. These at last are the Unix file system types.
Next you have the special bits - that first digit in the Info sheet right before the octal permissions. This bit can take on one or more of three values.
- Pipe/FIFO. A communications channel. FIFO stands for 'first in, first out'. It's a type of buffer different from the cafeteria tray paradigm where it's 'last in, first out' (as used on a process stack).
- Character. Used to describe driver-level devices that stream rather than work in blocks.
- Directory. The file is a directory.
- Block. The opposite of Character. Input/output is by block rather than byte. You'll see both Character and Block in /dev, a directory on your Mac that your Finder can't find.
- Regular. Glass.aiff is such a file as it's a regular file.
- Symlink. A symlink file contains only a path to another file, so that attempting to open it will instead open the file it points to.
- Socket. Sockets are the mainstay of the Internet but also used locally. They're communication endpoints.
- Whiteout. Used infrequently, they appear when one file system is mounted atop another and the underlying system disappears for a while.
None of the above is available in your Finder. It's not visible and it's not manageable in your Finder.
- Sticky bit. Originally used to tell the system kernel to not remove a program from memory as it's used so often. Today the bit is used in protecting files in a directory as well.
- Set GID. Setting the bit for the 'group ID' gives the user special privileges.
- Set UID. Setting the bit for the 'user ID' gives the user special privileges.
- Access. The three octal digits with values of 1 and/or 2 and/or 4 that set the actual access permissions for the file. The mode 0644 for Glass.aiff therefore means 'no special bits, read and write for the owner, read for everyone else'.
- User flags. These can be set by the user. They can be used to prevent certain actions such as dumping the file, modifying the file, removing the file, appending the file - and a special case for Apple - telling your Finder to not list the file.
- System flags. These are often managed by the top level administrator. Some require you reboot your computer to remove.
No one can change the system file type. Users should be able to manipulate the special bits, see and work with the full granularity of the permissions bits, and work with the system and user flags. But your poor Finder will do none of that for you. Should anything awry happen and should you be at a loss as to why things are so screwed up, nothing in the factory-approved tools shipped with your Mac will help.
Access Control Lists
And we've just begun to look. A new thing in vogue on BSD systems (including Apple's) is the access control list. Access control lists or ACLs seem to make more trouble than they're worth. But despite no clear difference in performance, Apple have begun to use ACLs to protect system files instead of special flags.
How do you as a user see these ACLs? You don't. Not with Apple's tools, not with your Finder. If something doesn't work right, you'll either find an obscure post in a techie web forum or you'll be told to shut up and not worry about such things. Or both.
What are access control lists? Simple. They're lists of access controls for files. Every file can have a list of them.
An access control list contains a list of access control entries. Once again, nothing in your factory-default Mac will help you. Nothing. And these tiny bugs can be used to cripple your system and drive you crazy.
Here's the same file, Glass.aiff, seen through an ACL editor. The ACEs would be seen in the lower pane. There are none. But given the right permissions, they could be created.
Here's the ACE edit sheet for Glass.aiff. ACEs can be used to allow or deny special types of access, by user or by group, and the ACEs can be inherited in different ways.
Note that the possibilities far exceed the 'rwx' of basic Unix.
A look at the system root shows that many files placed there by Apple have ACLs. Except your Finder won't tell you. Or help you deal with ACLs at all.
The /Applications folder has ACL glue stuck to it. /Applications has been marked with an ACE so that everyone, no matter who they are, is denied the ability to remove it. Note as well that this ACE is not inherited, so if you create subdirectories of your own under /Applications, you're able to remove them without problem.
Note as well that there are system APIs and arcane command line tools so you can set these ACEs. But Apple don't want you to know that. They're without heart: they think you're stupid, they want to make you stupid if you're not, and they want to keep you that way.
There are myriad facets to your Mac file system that you will ordinarily not have to touch. But that's a far cry from a system intentionally geared to not give you a clue or a fighting chance when you need it.
Finder today hides your own ~/Library folder. A healthy ~/Library contains well over one thousand files. And they're all your files. But Apple don't want you to get at them. Or touch them. Or know anything about them.
If those files were system files where it's best that even the most advanced users tread carefully, then that's one thing. But these are your files, in your particular proprietary area of the disk. And Apple are trying to keep you out of your own back yard.
That's not a nice thing to do. Neither is hiding important system stuff from you. It's one thing to keep a tidy work desk and keep the unusual stuff out of the way. It's quite another to deliberately make it difficult to access when you need to.
Tap tap tap.