Rixstep
 About | ACP | Buy | Industry Watch | Learning Curve | News | Products | Search | Substack
Home » Learning Curve » Developers Workshop

Red Pill

Certain things should never be seen and certain things should never be released.


Get It

Try It

There's a special version of Xfile that's never going to be released. It's called Red Pill.

Xfile is a Unix file manager. In fact it's probably the only one for the platform. It deliberately eschews any form of API that's not vanilla Unix - that won't essentially work on any distro. It deliberately avoids delving down into 'beige box'/toolbox/Carbon APIs to deal with file system stuff. It gains by that.

When it comes to file management on Unix there are two basic API levels. And the one feeds the other so to speak - much as close, open, read, and write feed fclose, fopen, fread, and fwrite.

As everything on Unix is a file directories are files too. And they can be opened by ordinary open calls. They can no longer be read by read and they certainly can't be written to with write but the open call will return a valid file descriptor for a directory. And once you have a file descriptor you get the directory entries with getdirentries.

if ((fd = open(path, O_RDONLY, 0)) >= 0) {
    while (getdirentries(fd, buf, bufsize, basep) > 0) {
        /* * */
    }
    close(fd);
}

But getdirentries actually feeds the higher level API.

if (d = opendir(path)) {
    while (readdir(d)) {
        /* * */
    }
    closedir(d);
}

Now there are certain things readdir is not supposed to report. Or preferably will skip over. Such as files with zero inodes - structures marked for deletion by the system. readdir is supposed to report meaningful information in a palatable manner.

But things can easily get out of hand. And there's a bit of latitude here depending on exactly what file system you're supporting. Such as HFS.

Apple have to hide a lot of stuff. HFS is not a perfect fit but Apple engineers have done their best to take the impossible task given them and come up with something that almost works.

They have to muck with things in the layer between getdirentries and readdir. To paraphrase Darryl F Zanuck: they have to sugarcoat the truth. It's not so much they want to hide things as they understand you'd regularly rather not see them.

Between getdirentries (which they have to be true to) and readdir they (have to) modify reality to make it more palatable.

␀␀␀␀HFS+ Private Data

One of the painful areas of incompatibility is with hard links - additional 'names' for the same physical file. The 'Mac OS' way sees file system IDs leading always to a unique path; the Unix way does not. Hard links have to be supported; what should be done with them?

There's a special directory in root called '␀␀␀␀HFS+ Private Data'. It's mostly invisible and it has absolutely obscene Finder flags in case it's not so it most likely won't show up in a Finder folder anyway.

And it's amazing what can be found in there.

Buried in ␀␀␀␀HFS+ Private Data are all the files that have been 'hard linked' in the system - files with more than one path. For most practical purposes the user doesn't notice a difference. Attempts to write to these files from the GUI will bork things but this won't happen from the 'Unix' command line level.

But if you want your files to behave as ordinary files again - too bad. Once they're banished to ␀␀␀␀HFS+ Private Data they're gone forever. They don't even come back if you remove all links but one. They're just gone. Bye.

Elvis Never Left

And you don't really want to see Elvis either. 'Elvis' is the code name for the journaling in use with HFS since Panther 10.3. It's there on your hard drive too but you'd be hard put to find it. But it's there alright.

There are other things you ordinarily don't want to see - and Apple hide a lot of them from you for your own good - but they're still there. And even for the inquisitive using Xfile there can be limits.

Xfile already shows you more than any other file manager will be capable of. It shows you /.vol and /dev for example; it lets you get inside both and look and play around; and sometimes this is essential.

But there are things even Xfile would balk at displaying. That's where Red Pill comes in.

Blue Pill

Red Pill operates in two distinct modes: Blue Pill (the default) and Red Pill. The default Blue Pill is depicted below.

.fseventd, .Spotlight-V100, .Trashes, .vol, and dev can all be seen as directories on the left; the empty sentinel file .com.apple.timemachine.supported and .hotfiles.btree can be seen along with the cited directories on the right.

But that's it.

Red Pill

Flipping Red Pill's mode produces another picture entirely.

Suddenly several new files hidden from readdir are visible on both sides: the hard link repository ␀␀␀␀HFS+ Private Data; and the 'Elvis' files .journal and .journal_info_block.

The 'Elvis' files are a chapter unto themselves - and a very interesting one too: officially they don't exist. At least not for long. They have zero inodes - and that normally means the system will remove them in a coming cleanup operation.

But it as a correlate means one other thing: the file system will leave these 'files' alone. So Elvis stays right where he is even though everyone thinks he's on his way out of the building.

Opening the Door

␀␀␀␀HFS+ Private Data too is handled in a low level way: its permissions are set to 000. No one has any access to it. But that's a door that can be opened.

sudo chmod 777 '␀␀␀␀HFS+ Private Data'

And when you step inside things look like this.

It's pretty ugly. Each of these 1200+ entries represents what appears to be a real file on the outside: strip away the 'iNode' prefix and look for the files with that inode and you'll know what the reference is about. The files exist in here even though they appear to exist elsewhere.

But it's particularly ugly - not to say annoying - to see this directory 'dangle' below 'Volumes' on the left. Suffice it to say if you saw that every day you'd soon tire of the HFS file system and perhaps its operating system too.

.HFS+ Private Directory Data\r

The directory '.HFS+ Private Directory Data\r' is for Time Machine. Note the cute 'carriage return' on the end, presumably to thwart attempts to access it from the command line. And the system shell doesn't want to send back a folder icon any more than it wants to send one back for 'dev'. But in the latter case it's a different file system; with the former you're still on the root system.

iNode103100

The sharp eye will have noticed that one file in particular stands out in the hard link repository: 'iNode103100' with 87 links. Find out what files share that inode from the command line.

$ sudo ls -iR | grep 103100
Password:
 103100 alias.1
 103100 alloc.1
 103100 bg.1
 103100 bind.1
 103100 bindkey.1
 103100 break.1
 103100 breaksw.1
 103100 builtin.1
 103100 builtins.1
 103100 case.1
 103100 cd.1
 103100 chdir.1
 103100 command.1
 103100 complete.1
 103100 continue.1
 103100 default.1
 103100 dirs.1
 103100 do.1
 103100 done.1
 103100 echotc.1
 103100 elif.1
 103100 else.1
 103100 end.1
 103100 endif.1
 103100 endsw.1
 103100 esac.1
 103100 eval.1
 103100 exec.1
 103100 exit.1
 103100 export.1
 103100 fc.1
 103100 fg.1
 103100 fi.1
 103100 filetest.1
 103100 for.1
 103100 foreach.1
 103100 getopts.1
 103100 glob.1
 103100 goto.1
 103100 hash.1
 103100 hashstat.1
 103100 history.1
 103100 hup.1
 103100 if.1
 103100 jobid.1
 103100 jobs.1
 103100 limit.1
 103100 log.1
 103100 logout.1
 103100 ls-F.1
 103100 notify.1
 103100 onintr.1
 103100 popd.1
 103100 pushd.1
 103100 read.1
 103100 readonly.1
 103100 rehash.1
 103100 repeat.1
 103100 return.1
 103100 sched.1
 103100 set.1
 103100 setenv.1
 103100 settc.1
 103100 setty.1
 103100 setvar.1
 103100 shift.1
 103100 source.1
 103100 stop.1
 103100 suspend.1
 103100 switch.1
 103100 telltc.1
 103100 then.1
 103100 times.1
 103100 trap.1
 103100 type.1
 103100 ulimit.1
 103100 umask.1
 103100 unalias.1
 103100 uncomplete.1
 103100 unhash.1
 103100 unlimit.1
 103100 unset.1
 103100 unsetenv.1
 103100 until.1
 103100 wait.1
 103100 where.1
 103100 while.1

They're all in '/usr/share/man1' and they have 87 bedfellows who share another inode.

$ sudo ls -iR | grep 680003
 680003 alias.1.gz
 680003 alloc.1.gz
 680003 bg.1.gz
 680003 bind.1.gz
 680003 bindkey.1.gz
 680003 break.1.gz
 680003 breaksw.1.gz
 680003 builtin.1.gz
 680003 builtins.1.gz
 680003 case.1.gz
 680003 cd.1.gz
 680003 chdir.1.gz
 680003 command.1.gz
 680003 complete.1.gz
 680003 continue.1.gz
 680003 default.1.gz
 680003 dirs.1.gz
 680003 do.1.gz
 680003 done.1.gz
 680003 echotc.1.gz
 680003 elif.1.gz
 680003 else.1.gz
 680003 end.1.gz
 680003 endif.1.gz
 680003 endsw.1.gz
 680003 esac.1.gz
 680003 eval.1.gz
 680003 exec.1.gz
 680003 exit.1.gz
 680003 export.1.gz
 680003 fc.1.gz
 680003 fg.1.gz
 680003 fi.1.gz
 680003 filetest.1.gz
 680003 for.1.gz
 680003 foreach.1.gz
 680003 getopts.1.gz
 680003 glob.1.gz
 680003 goto.1.gz
 680003 hash.1.gz
 680003 hashstat.1.gz
 680003 history.1.gz
 680003 hup.1.gz
 680003 if.1.gz
 680003 jobid.1.gz
 680003 jobs.1.gz
 680003 limit.1.gz
 680003 log.1.gz
 680003 logout.1.gz
 680003 ls-F.1.gz
 680003 notify.1.gz
 680003 onintr.1.gz
 680003 popd.1.gz
 680003 pushd.1.gz
 680003 read.1.gz
 680003 readonly.1.gz
 680003 rehash.1.gz
 680003 repeat.1.gz
 680003 return.1.gz
 680003 sched.1.gz
 680003 set.1.gz
 680003 setenv.1.gz
 680003 settc.1.gz
 680003 setty.1.gz
 680003 setvar.1.gz
 680003 shift.1.gz
 680003 source.1.gz
 680003 stop.1.gz
 680003 suspend.1.gz
 680003 switch.1.gz
 680003 telltc.1.gz
 680003 then.1.gz
 680003 times.1.gz
 680003 trap.1.gz
 680003 type.1.gz
 680003 ulimit.1.gz
 680003 umask.1.gz
 680003 unalias.1.gz
 680003 uncomplete.1.gz
 680003 unhash.1.gz
 680003 unlimit.1.gz
 680003 unset.1.gz
 680003 unsetenv.1.gz
 680003 until.1.gz
 680003 wait.1.gz
 680003 where.1.gz
 680003 while.1.gz

Each of the files in each group point to the same single physical file. There appear to be 174 different files here when in reality there are but two. And this is managed by the invention '␀␀␀␀HFS+ Private Data'.

The Other Pill

But after a while it's time to switch pills. It's good to be able to see this stuff; after all one should be able to see everything on one's hard drive if one wants; but it's not very practical - or pretty - to see all the time.

See Also
Learning Curve: Red Pill, Blue Pill
Industry Watch: ACP 2.0.1 Rollout Complete

About | ACP | Buy | Industry Watch | Learning Curve | News | Products | Search | Substack
Copyright © Rixstep. All rights reserved.