About | ACP | Buy | Industry Watch | Learning Curve | News | Products | Search | Substack
Home » Learning Curve

On File Management (8)

Part eight. Dealing with Apple idiosyncrasies - 'hidden' files.

Get It

Try It

Apple may be touting a Unix on their OS X but their HFS file systems aren't a good fit.

Apple engineers spent considerable time, blood, sweat, and tears trying to get their own HFS file systems as Unix-conformant as possible - and what they did was quite the feat - but it's not a perfect fit by a long shot and several key features of Unix are conspicuous in their absence.

Using another file system such as UFS will avoid these shortcomings but Apple's UFS leaves a lot to be desired (a lot) and most users just settle in with the default HFS* anyway.

  • Hidden files. Unix doesn't hide files. It can keep files out of the way (for your convenience) but it neither can nor would attempt to actually hide anything - it goes against Unix philosophy. Apple on the other hand have had numerous (financial) reasons to be able to hide things.
  • Extended attributes. Several flavours of Unix support extended attributes but Apple's aren't a standard.
  • Access control lists. ACLs can have wandered over from Microsoft and Dave Cutler's NTFS (where they're the only file system security Windows has) and they're implemented today on FreeBSD, but they're not widespread and they present new headaches for administrators who might just be wondering why they need them.

Hidden Files

The only user land way for Unix to 'hide' a file is by giving it a 'dot' name.

$ pwd
$ ls
$ ls -a
.    ..
$ ls -ilo
total 0
2836202 drwx------    2 macuser  staff  -   68 Dec 14 09:26 501
$ ls -ailo
total 0
     20 drwxrwx---    3 macuser  admin  -  102 Dec 14 09:26 .
      2 drwxrwxr-t   31 root     admin  - 1156 Dec 14 09:41 ..
2836202 drwx------    2 macuser  staff  -   68 Dec 14 09:26 501

Not using the -a switch with ls keeps certain files out of the way.

Apple have always been more ambitious.

  • .hidden - a text file planted in directories with one file name per line. Used by Finder to know which files to not list.
  • Special file flag 'Hidden' - a crossover from old HFS into Unix territory (and proprietary to Apple). Also instructs Finder.

The .hidden file isn't in use today anymore; the special file flag 'Hidden' is.

Bottom line: if you're using Finder then you're not going to see everything. Apple have a special property list key that you can use to expose stuff but that requires rebooting Finder to see things take effect. (Rixstep have a nifty little free tool to do this for you if you absolutely insist. Rixstep also offer a more fun variant as well for general lolz.)

Why hide files? Well Unix doesn't - it simply helps you keep things out of the way. But the poor Finder user can be SOL. A console (Terminal.app) will show you what's missing but Apple bet the Cupertino campus that not many people will try that.

Apple's deal with the record companies hinged on them being able to demonstrate that file sharing would effectively be thwarted. This wouldn't have been possible without an exaggerated dependence on 'TFF'. But even for 'TFF' diehards there's a way around most anything.

The Zero Inode

But Apple take things a step further. They have elements of their file system they don't want anyone to see - stuff on your computer they're making it as difficult as possible for you to know about and find.

One of the techniques they use is the zero inode.

'Inodes' are indexes into a volume control block - a contiguous otherwise unmapped section of your hard drive (one per volume) that contains all the data on the files on that volume: where the file is located (actual 'extents'); permissions, special flags, ownership, etc.

The inodes start at index 1 - this because index 0 (the zero inode) has a special meaning. The zero inode is a signal to a file system's 'cleanup' routines: it signals a file that's to be removed from the file system.

Files aren't immediately cleaned away when completely unlinked: the file system incorporates lazy write to save on wear and tear on the hard drive. The file system waits until the drive controller is in the vicinity of a pending write operation.

Apple 'use' ('abuse') this facility to keep even more files hidden from view. They know - their code knows - these files aren't to be cleared away: they're just hidden, really hidden. Ordinary file system APIs won't pick them up.

But as with everything else, there's ways around it all.

The 'd_ino' field at the far left is the inode field. The items '.HFS+ Private Directory Data' (used by Time Machine); '.journal' and '.journal_info_block' (used by the journaling system); and '␀␀␀␀HFS+ Private Data' (used for banished multi-linked files) all have a zero inode. They're not taken up by higher level file system APIs. Only lower level ones.

Two of the items are considered 'cloaked' - they're not mentioned at all at higher levels. (The number of cloaked items is arrived at by a simple subtraction.)

[Unix directories aren't accessible with standard APIs anymore. Dedicated APIs are needed instead. And they're a bit more complex as well. The main reason for this is Unix file names are no longer limited to 14 bytes (and haven't been for a long long time). The four fields in the above screenshot are for the file's inode, the length of the record, the item type such as 'regular', directory, or symbolic link, the length of the file name, and finally the file name itself. It could be argued that one of the two fields 'd_reclen' and 'd_namelen' is redundant.]

True Privacy

Any number of amateurish third party applications have popped up over the years, all promising to hide files away (and thereby somehow protect them). None of them work. What's worse: they give users a false sense of privacy.

People can trust in these 'snake oil' applications and then end up getting hacked good and proper by someone only remotely familiar with Unix file system intrinsics. Instead of taking proper measures to protect files, users pay $20-30 for an application that leaves them wide open.

Three such applications reviewed over the years are found here, here, and here. Software listings contain myriad more.

But you can still protect your files if you go about it the right way. For Unix was built as a multi-user system from the get-go. The creators of Unix understood that users need privacy - it's just that this method isn't what you find online for $20-30. And you don't need to spend any money to get true privacy anyway.

Bury your sensitive files in an inaccessible directory: it's as simple as that. Mark the directory 0700 owned by you and no one can see what's inside. Mark your files as accessible only by you and no one can touch them.

Here's what happens when ordinary (non-root) users try to get inside the home directory of the Unix root account. See for yourself: they can't get at the files and they can't even see what's there. That's true privacy.

$ cd /private/var                                           # cd to parent of root's home directory
$ pwd                                                       # print working directory
/private/var                                                # checks out OK
$ ls -dilo root                                             # list the directory itself
3452 drwxr-x---   6 root  wheel  - 204 Sep 14  2009 root    # accessible only by root and wheel
$ ls -ilo root                                              # list the contents of the directory
ls: root: Permission denied                                 # nope - only root and wheel can read it

Use Unix to protect your files and safeguard your privacy. Nothing else works.

See Also
Learning Curve: On File Management (1)
Learning Curve: On File Management (2)
Learning Curve: On File Management (3)
Learning Curve: On File Management (4)
Learning Curve: On File Management (5)
Learning Curve: On File Management (6)
Learning Curve: On File Management (7)

Developers Workshop: It Wasn't Good Then, It's No Better Now

About | ACP | Buy | Industry Watch | Learning Curve | News | Products | Search | Substack
Copyright © Rixstep. All rights reserved.