About | ACP | Get Stuff | Industry Watch | Learning Curve | Newsletter | Search | Test Drive
Home » Learning Curve

.DS_Store Redux

Arno speaks.

Get It

Try It

One of the most sought after OS X terms in search engines across the planet is and remains .DS_Store. People discover them on their systems, can't understand what they are, and feel powerless to stop their proliferation - so they go searching. And quite a few of them end up at this site.

And now, four years after the original article on the subject, the person responsible for .DS_Store comes forward.

He's Arno Gourdol and today he's an 'engineering manager' at Adobe. But back in 1999 when .DS_Store first saw the light of day he was the man in charge at Apple.

Arno Gourdol created .DS_Store.

Arno recently wrote a blog post about the origin of the name (stands for 'desktop services store') and in a roundabout way snuck in an apology for how he and his team 'screwed up' - all the while not really admitting to himself how badly they did in fact screw up.

'Back in 1999 I was the technical lead for the Mac OS X Finder at Apple', writes Gourdol. 'At that time the Finder code base was some 8 years old and had reached the end of its useful life. Making any changes to it required huge engineering effort and any changes usually broke two or three seemingly unrelated features. For Mac OS X we decided to rewrite the Finder from scratch.'

[Editor's note: a code base that's 8 years old in 1999 is from 1991. In 1991 NeXTSTEP was still six years away from Cupertino. Apple were still running a standalone 'operating system' with 'cooperative multitasking' and little or no network connectivity. Sir Tim unveiled his 'World Wide Web' in 1991 and the 'web revolution' was still four years away.]

Gourdol goes on.

'Since I had previously been responsible for naming Icon Services and Navigation Services, we decided to go with Desktop Services. Hence the name .DS_Store for 'desktop services store'. We added a '.' in front of it so that it would be considered as an invisible file by Unix OS, including Mac OS.'

It's time to drop the bomb.

'There is also an unfortunate bug that is not fixed to this day that results in an excessive creation of .DS_Store files. Those files should only be created if the user actually makes adjustments to the view settings or sets a manual location for icons in a folder. That's unfortunately not what happens and visiting a folder pretty much guarantees a .DS_Store file will get created.'

'Not fixed to this day' implies the bug is eight years old.


Comments at the blog post have been noted elsewhere but a few are worth citing again.

'That's the REAL problem with .DS_Store', writes 'Anonymous'. 'It is based on a set of poor assumptions. For example, in a network environment you're assuming that everybody wants the same view settings.'

Gourdol comes back to comment on the comments.

'I've been biting my tongue for a few years now, and while John Siracusa had been a thoughtful critic of the Mac OS X Finder, I must say I agree with almost everything he has to say about the Finder. In fact, he was one of the few people who was right on the money (I think) regarding what the Finder should have been/should be some day.

'We actually used printouts of John's columns to try to influence the decision makers at the time, as sometimes a voice from the outside is given more weight than a chorus on the inside. Unfortunately, there were powerful forces at work. My only consolation is that I know that it could have been worse. One day, I shall tell the tale.'

[Editor's note: one wonders what these 'powerful forces' were and what they were working at. Perhaps that's the point.]

The Basics!

'.DS_Store is an idea implemented by someone aware of the potential for waste and chaos on user machines, who is aware of the concept 'sloppy programming', but who really doesn't give a damn.' So goes the original article. Eight years to (not) fix that 'bug' is of course is a long time. Too long. But it goes farther than that. Way farther.

The basics here: if you're sitting at Apple in 1999 with two years of NeXTSTEP and eight years of networking under your belt and you still think .DS_Store is a good idea you're either batshit insane or you picked the wrong career.

.DS_Store is an attempt to preserve that John Siracusa wet dream of 'spatiality' - to always have folders and icons in folders come up in the same place. [The Siracusa dream is more morbid than that but the rest of the insanity has nothing to do with .DS_Store.]

One of the similes Siracusa used to use was driving an automobile. You have the gear shift in the same place, the steering wheel in the same place, the ignition in the same place, the accelerator in the same place, and the brake and shift pedals in the same place.

[What happens when someone else drives your car and pushes the seat backward or forward is of course something Siracusa doesn't want to get into.]

Gear shift, steering wheel, ignition, accelerator, brake, shift pedal - heck, throw in the horn and the turning signal lever too: that's eight devices. Which can always appear in the same location. So you don't have to really look at any of them - you know where they are.

And Siracusa's pet computer had perhaps twenty folders all told. Not much to worry about there.

But now compare that with a NeXTSTEP based OS X system of today. When working only alone a user is likely to encounter twenty thousand folders or more. With hundreds of thousand of files.

And things get even hairier when you remember most users are going to be browsing their corporate networks.

'It is unfortunate that these files pop up all over the place. I always do a search for them before burning CDs or making archives', comments 'Anonymous' at the Gourdol post. And only recently have Apple added a flag to Finder to stop it littering networks with the junk.

The Rough Maths!

.DS_Store files take a minimum of two clusters on disk. Considering HFS wants 4 KB per cluster, that's 8 KB wasted for each and every .DS_Store file.

As .DS_Store is specific to directories and as an OS X system can have 20,000 or more directories, it's easy to see how much disk space is needed to accommodate the system. [At the bare minimum - .DS_Store files can easily grow to double or treble their original size.]

Two clusters for 8 KB for 20,000 directories is 160,000 KB. That's over 150 MB disk space to accommodate the .DS_Store idea on a reasonably used OS X system at a bare minimum.

For one user.

But OS X systems - as opposed to their distant MacOS cousins - are multiuser systems. So the disk space allocated for .DS_Store must be extended by another 150 MB for each user who can even theoretically be found logging in to the system.

Where should one set the limit? Are Apple to allow only ten users per system - and no more? Twenty? Five? .DS_Store for so few as seven users easily surpasses the gigabyte mark. And the model still hasn't ventured into the network.

Or for that matter has the model made accommodations for the system being multiuser. For .DS_Store works only one way - it assumes there's only one user on the system. No one can ever push that car seat backward or forward.

No, to even consider this idea a second longer the entire .DS_Store idea must be revamped to accommodate the prospect of multiple users. The file format must change. The system must be able to store this persistent data on a per-user basis - with identification of each user with entries within.

And on entering any folder the system must first, before rendering the folder contents, read through this possibly extensive file, correlate the current user with the data within, and pluck out that data.

And as soon as the user moves a folder or an icon the system must do the same thing in reverse: dig back into the .DS_Store, correlate with the current user, update the settings, and put it all back for use next time around.

And the system would have to do this in the network as well. Where perhaps hundreds of thousands of new folders wait.

It's Not Only .DS_Store!

And it's not only .DS_Store implementing this 'spatiality': the screen coordinates for files and folders are stored in your hard drive's volume control block - a storage that's supposed to be used to manage the file system and nothing else. [DiskWarrior addicts take note.]

struct FileInfo {
  OSType              fileType;               /* The type of the file */
  OSType              fileCreator;            /* The file's creator */
  UInt16              finderFlags;            /* ex: kHasBundle, kIsInvisible... */
  Point               location;               /* File's location in the folder */
                                              /* If set to {0, 0}, the Finder will place the item automatically */
  UInt16              reservedField;          /* (set to 0) */
typedef struct FileInfo                 FileInfo;
struct FolderInfo {
  Rect                windowBounds;           /* The position and dimension of the folder's window */
  UInt16              finderFlags;            /* ex. kIsInvisible, kNameLocked, etc.*/
  Point               location;               /* Folder's location in the parent folder */
                                              /* If set to {0, 0}, the Finder will place the item automatically */
  UInt16              reservedField;          /* (set to 0) */
typedef struct FolderInfo               FolderInfo;

The basic unit is called 'FSCatalogInfo' and it's 144 bytes in length. Within is another structure or union of structures with data for files and folders.

For all files and folders there are coordinates for where the item is to be displayed in its parent folder; for folders there are also (absolute screen) coordinates for the upper left and lower right.

This data cannot be stored on a per user basis without totally gutting the HFS file system.

And So What?

Assume for a fleeting second that gigabytes of disk storage were available for per user .DS_Store files both on the local machine and in the network. Assume for another fleeting second that it was possible to redesign the .DS_Store system so it accommodated all these millions of settings. Assume for yet another fleeing second that the vendor were willing to completely scrap the HFS file system and build a new file system that accommodated per user file and folder screen coordinates.

Would it help?

Computer screens of today can have widths of 1,500 pixels and heights of 1,000 pixels. Imagine now a user trying to recognise one of twenty thousand folders by its on screen placement alone.

'Oh yeah! Of course! That's my blah-blah folder! It always comes up at an X offset of 142 and a Y offset of 268! I recognised it immediately! And I know it's not the NONSENSE folder because that one comes up at an X offset of 145 and a Y offset of 286!'

There's no way the idea is of any benefit even if it could be implemented, and there is no software engineer who's even going to entertain the implementation.

Microsoft Tried Too!

Microsoft have tried the same thing. They use their Registry to store 'MRU streams' which are the equivalent and which build up over time with no way to have them cleaned, pruned, or removed.

Security Risks!

Gourdol's team have even gone so far as to put 'too much data' into .DS_Store - data that had no business being there. Users discovered this data was being inadvertently uploaded to websites and exposing confidential information.

But unlike Gourdol, the official policy of Apple is that .DS_Store cannot hurt because they can't be seen. But if they can't be seen, why are so many people to this day searching for information about them and help to get them removed?

Urban legend has it NeXT and Apple chief software engineer Avie Tevanian wrote technical note 2034 which admonished Apple computer users to be 'good netizens' in what Eric S Raymond calls 'the Internetted world'. Yet despite years of exposure to this world it seems certain people just don't want to get it.

They're not personal anymore. Personal computers that is. They share everything. Everything they come into contact with.

Unless of course you're irretrievably antisocial.

Further Reading
Industry Watch: .DS_Store
Industry Watch: .DS_Insecure
Learning Curve: Zeroes Are Nice
The Technological: Desktop Services Store

About | ACP | Get Stuff | Industry Watch | Learning Curve | Newsletter | Search | Test Drive
Copyright © Rixstep. All rights reserved.