About | ACP | Buy | Industry Watch | Learning Curve | News | Products | Search | Substack
Home » Learning Curve

The Lost Story

What's coming after Mojave?

Get It

Try It

KØBENHAVN (Rixstep) — There's a story out there somewhere, on one of our servers, we suspect. We did all the research. We thought we'd published it, but now we can't find it. But it's important, so we'll try again.

It has to do with filesystems. Not filesystems like APFS, but filesystems in terms of file management. File management systems.

Good file management systems are hermetically sealed. You can't get in or get out without asking permission, if they're designed right. You can't interfere with ongoing operations unless asked (by the system). You can't do anything without passing through the system's checkpoints.

Back in the day, file management systems for personal computers were of course of another colour. Those computer systems weren't even called operating systems - they were called disk operating systems. Microsoft started integrating Unix into MS-DOS ('Microsoft Disk Operating System') with version 2.11, but Unix doesn't have much of a file management system, to tell the truth.

The first Macintosh didn't have much in the way of a file management system either. And it had something called MFS - Macintosh File System. The original Mac was held together by strings and glue. Steve Jobs admitted this mistake years later, in so many words. His NeXTSTEP wasn't that much better either, to tell the truth, but at least it had a hierarchical filesystem, something the original Mac definitely didn't have.

The people at Rixstep and Radsoft have been building file and disk management tools for donkeys years, most likely because they find it fascinating. They find filesystems and secondary storage matters interesting. Others may like database management systems, for example, or something else, but not them. They like file management and disk systems - and as low down and as bare metal as you can get. To them, Peter Norton was a wuss.

And around about when Apple first released APFS, Apple also came out with an Xcode 10 beta. This was before 10.14 Mojave. So both APFS and Xcode 10 beta got put on a dedicated test machine.

There wasn't a lot to say about APFS. It seemed to be working OK. The Xcode beta was another matter.

Things like this - with Xcode - recur regularly. Someone misses something in Cupertino, project files don't work as they should, people find workarounds, another version of Xcode comes out that finally fixes the bugs in the previous version but simultaneously breaks the project files, because of the workarounds that were necessary at the time. Looking at the nature of these bloopers, and keeping in mind that they can take so long to notice and to fix, one really has to wonder.

Xcode 10, already in beta, had such a blooper. All versions of Xcode, possibly excluding a few point updates for version 9, used 'tilded' paths in project settings. Which seems reasonable. Xcode 10 suddenly did not. But we didn't immediately see the obvious workaround, because Xcode had had so many other related errors over the years. There were, for example, several versions that simply couldn't find external frameworks, no matter what one tried. (Yes there was ultimately a workaround, but this ended up breaking project files in later versions of Xcode of course.)

Whatever. We had Xcode 10 beta on disk with APFS. And the final Xcode 10 had come out in the meantime. So we're sitting there with a test machine with both Xcode 10 and Xcode 10 beta sitting under Applications under root. It was time to remove the one that we were never going to use.

It's hard to say after the fact, but both versions of Xcode should have approximately the same number of files. The most recent version - Xcode 10.1 (10B61) at time of writing - has 356,717 files under /Applications. That's a lot of files. (Most of the bulk comes from iPhone configurations, it would seem, and yes, one has to wonder why each and every iPhone has to have a new and unique development environment.) The Xcode 10 beta had presumably about the same number of files - 350,000 or thereabouts. And now it was time to remove them and recover that disk space.

Now a few words about NeXTSTEP's file management APIs. You may have heard that a lot of NeXTSTEP code goes through something called Core Foundation. Whether this is inherited from NeXT or is an Apple Carbon concession is moot. A lot of that Core Foundation code must presumably bottom out in Unix file management APIs (to the extent there are any). And from there it's down to the drivers, and after that it's no one's concern. Presumably.

NeXSTEP was meticulously designed. Classes were divided into two big categories time and again. Distinction was made between mutable and immutable classes. (See Andy Lee's pivotal AppKiDo to discover this architecture.)

Distinction was also made between abstract classes and user interface classes. Code and controls that were visible onscreen were put in one framework, code and controls that never interacted were in another. The abstract classes did the dirty work, the visible classes took all the credit.

The guys who put this together - the team leader was Avie Tevanian - pulled off the greatest single achievement in computer science since the CSRC team in Murray Hill.

But all was not perfect. Their roots were Unix and the Mac. Unix doesn't have a file management system per se. You want to copy a file? Then you open it, read it, and write what you get to the target file. (Starting with Unix, everything was a 'stream of bytes'.) You want to copy files en masse from the command line? Use cp. It's well tested. The same with moving files - you use mv.

But what happens if you have a conflict of file type at the destination? What happens if a target item has an ACE? Or a system flag? What happens if a hard drive fails? Can you roll back? Or if the network goes down? Or if there's a blackout? Or a brownout? File management intrinsics aren't driver code, but they should be regarded as such. Their code should be tested just as rigorously, as strenuously, as meticulously, and as thoroughly, by as many people as possible, on as many machines as possible. This is core code, and it should reside in the system, and as part of public libraries (frameworks). Only an amateur would hide code like this away in a private application maintained by the OS vendor. That'd be highly unprofessional.

Unix doesn't have a GUI. It's all command-line. You'll get some feedback from command line programs, but not a lot. The code in those programs always bottoms out in Unix intrinsics.

We all know what happens with Apple file management. Unfortunately. To Apple, file management has always been an afterthought, not an overriding and crucial concern. And it shows. Steve Jobs didn't understand OO when he visited Alan Kay, and his people in the Mac group didn't understand filesystems and file management. But Steve was man enough to admit his mistakes. Over twenty years since the NeXT 'bailout', it looks like Apple's designers never will.

There's no reason why Apple's OS X 'macOS' can't be best of breed. They have some of the best names in the industry. Yet...

Take a look at the declared APIs for NSFileManager and NSWorkspace. Do you see the correlations there? It's possible to build a 'no-code web browser' using Apple's WebKit. And their Safari is but the 'pretty face' on WebKit. As it should be. But can the same be said of their file management? When key functionality is found in NSWorkspace but not in NSFileManager and vice-versa? And no, copyfile() is not the answer. Clients that rely on that function can break at any time. And copyfile() has no counterpart in NSFileManager, much less in NSWorkspace. And NSWorkspace isn't even visual or user-interactive anymore. It's a good question what it's doing, what it's good for, but Arno was brought in to appease the haters, and...

Too many important files have been lost, too many systems hosed. Microsoft Windows is the undisputed King of Vulnerabilities, but Apple OSes are the Dukes of Data Loss. Data - secondary storage - is the most important thing in your computer. It's where your system resides, it's where your applications reside, it's where you keep your important files, it's where you keep everything. Backup routines and Time Machine are one thing, but they're not a replacement for a satisfactory file management engine.

Then there's the matter of dealing with collisions - with all the world on one side of the fence and Apple on the other. On Apple's platforms, they call traditional operations 'merge', but on the other side of the fence, there is no such term. There's only one way to do it.

Truth be told, the 'Apple' way is a copout. It's a lot easier to implement from the OS side. Otherwise it's a royal pain for all others involved. And this one copout yields really bad karma down the line. Karma that had Apple's glorious Finder tripping over its own feet on several occasions, where it's been sheer luck that it tripped - otherwise there'd have been a major catastrophe if it hadn't - like the 'Massive Data Loss' scandal.

We also see this ugly head in the infamous Safari Build 48, which Apple of course never owned up to. We see it in ridiculous quirks where entire hives of files can be wiped out with a single click, yes even today.

Neither copy nor move operations should remove things. They may, at the most, replace individual files when so advised by a user, but they should never replace directories. Never. Yet, because Apple APIs will do this, considerable code must be placed in client applications to prevent disastrous scenarios that any self-respecting system would categorically disallow.

The GNUstep code for file management is riddled with repetitive (and redundant) sanity checks to prevent cyclical and similar operations that could otherwise hose an entire machine. But that's good - that's at system level. Not at client application level, where apps running Apple's OS have to themselves do the system's dirty work.

But back on topic again. We had some 350,000 files to remove. From an Xcode 10 beta. So all you do is select the application in your file manager and click delete, right?

Right. If you want to wait half an hour.

All file management functions with the Xfile System used the NeXT 'workspace' API. This was the 'visual' part of NeXT file management. The abstract part, doing the dirty work, was NSFileManager. But NSFileManager had functions and methods all over the place, and NSWorkspace had one method to cover it all. And now an ordinary file delete was taking 28 minutes...

We found a couple of APFS researchers in Scandinavia and wrote to them. Could this be an issue with APFS, we asked. They'd never tested quite such an operation themselves, but they were willing to try to duplicate the scenario. They used command line stuff and generated about one million files for the test, then removed them all - again, from the command line.

No issues whatsoever. So we started looking into NSWorkspace. And, as a first experiment, replaced NSWorkspace methods with NSFileManager methods. And, lo and behold, discovered two things.

  1. Yes, file removals again worked as they should. Almost. See below.
  2. Removing files with NSFileManager works differently. Really.

The main difference can be seen when removing a hive - that's to say a hierarchy of directories and files. Try this.

  1. Create the directory 'a' somewhere.
  2. Create the directory 'b' inside of 'a'.
  3. Place a file - any file - inside 'b'. Name it 'file', for example.
  4. Mark 'b' 0500 - so it can't be written to.

Now try to remove 'a'. You should get this.

rm -fr a
rm: a/b/file: Permission denied
rm: a/b: Directory not empty
rm: a: Directory not empty

But if you were to use NSWorkspace, you'd never see that happen. Everything would be removed without a hitch.

(Yesssss, we sort of remember scratching our heads about this when first putting together Xfile, as it didn't make sense. Such an operation shouldn't succeed! And yet it did succeed!)

Implementing the same thing with NSFileManager means you have to first drill down in the hive you want gone, and 'loosen' up directory permissions. Fortunately, NSFileManager has a handy method for serving up an array of all your subpaths.

But even so. This requires 'taking a moment'. For NSWorkspace is clearly not just the 'pretty face' on NSFileManager. There are things going on which are not documented. And there are other things as well. Such as the half hour it takes to delete files with NSWorkspace, where it takes but a few seconds with NSFileManager.

Clearly something isn't right. And the most serious thing is the fact that these two code bases are not congruent. NSWorkspace should be the pretty face on NSFileManager - that and no more. Yet a quick look at both their pages at the Apple site shows that neither class is complete, both classes have glaring inconsistencies, and there is no overall design anymore, as there once was.

Certain important file operations require use of NSFileManager. Others require NSWorkspace, which is the exclusive repository for such functionality. And it should never be like that.

And now, just to add more hot spice to the mix, check those documentation pages for real this time. And note how much core functionality has been deprecated.

Deprecate: there can be times when such a drastic measure is unavoidable. But it's not often. Apple coders suffer from Deprecation Mania™. Watch Ali 'TextEdit' Ozer talk about deprecating MACRO DEFINITIONS. WTF is wrong with them? Are they trying to undermine the ISV sector?

And if you start slashing functionality, you're supposed to refer to a replacement method and explain why you've done that. And yes, you should only do it rarely.

But how much key functionality has been slashed at Apple since the release of OS X?

  • They slashed the menu item class, the class that made it possible to tear off submenus and move those submenus anywhere you wanted on your desktop. According to an insider, this was done because the Carbon programmers were bitching because they couldn't get at the same technology. (They could have if they'd been real programmers and taken the half day to learn Objective-C.)

  • They slashed the third title bar appearance. Apple's HI group said it'd be too confusing for their typical users, who they regard as not being especially intelligent. For windows can actually have one of three modes or states, not only two, as the interface will hint today. You can be a window in an inactive application, or a window in the active application but not ready for keyboard input, or you can be the window taking keyboard input in the active application. Three.

    NeXTSTEP distinguished between these three conditions. Inactive got a light grey title bar, active but not key got dark grey, and active and key got black. Apple's HI revision blurred the lines - the HI people thought that what was bleating obvious for NeXTSTEP users would be too confusing for their own.

  • They took out the possibility to yourself decide whether you wanted vertical scroll bars on the right or on the left. (NeXTSTEP had them by default on the left.)

  • They yanked out the mail delivery API. Presumably because they were afraid someone was going to hack their Rock Solid Foundation™.

The list is long...

So today Apple's file APIs are divided into two chief categories, but few of of them work the same way as their counterparts, the differences aren't documented, one group is supposed to be visual but isn't, both groups are incomplete and inconsistent - and the deprecations continue.

And there's still no robust encapsulated file management object. And 'merge' is still only an option.

What's the version coming after Mojave going to be called?

PS. A good friend found the original file. Thanks, Tom!

See Also
Apple's File System APIs
Developers Workshop: Y.G.B.K.
Hotspots: Leopard: OS Xhumation
Developers Workshop: performFileOperation:
Hosing OS X with Apple's Idea of 'Expected Behaviour'
Learning Curve: Rebel Scum: More Attacks on 'Expected Behaviour'

Apple: NSWorkspace Reference
Apple: NSFileManager Reference

Learning Curve: 4893378 FAQ
Industry Watch: Sanity Checks at Apple
Learning Curve: A Sanity Check for Apple
Learning Curve: 4893378: 'Expected Behaviour'

Industry Watch: The QuickBooks Disaster
Apple Support Discussions: Desktop GONE

Slashdot: Data Loss Bug In OS X 10.5 Leopard
Tom Karpik: Massive Data Loss Bug in Leopard

MacInTouch: Mac OS X 10.5 Leopard: Finder Data-Loss Bug
Learning Curve: Rebel Scum: More Attacks on 'Expected Behaviour'

About Rixstep

Stockholm/London-based Rixstep are a constellation of programmers and support staff from Radsoft Laboratories who tired of Windows vulnerabilities, Linux driver issues, and cursing x86 hardware all day long. Rixstep have many years of experience behind their efforts, with teaching and consulting credentials from the likes of British Aerospace, General Electric, Lockheed Martin, Lloyds TSB, SAAB Defence Systems, British Broadcasting Corporation, Barclays Bank, IBM, Microsoft, and Sony/Ericsson.

Rixstep and Radsoft products are or have been in use by Sweden's Royal Mail, Sony/Ericsson, the US Department of Defense, the offices of the US Supreme Court, the Government of Western Australia, the German Federal Police, Verizon Wireless, Los Alamos National Laboratory, Microsoft Corporation, the New York Times, Apple Inc, Oxford University, and hundreds of research institutes around the globe. See here.

All Content and Software Copyright © Rixstep. All Rights Reserved.

John Cattelin
Media Contact
ACP/Xfile licences
About | ACP | Buy | Industry Watch | Learning Curve | News | Products | Search | Substack
Copyright © Rixstep. All rights reserved.