Rixstep
 About | ACP | Buy Stuff | Industry Watch | Learning Curve | Newsletter | Search | Test Drive
Home » Learning Curve

XNU

Pick your team wisely and carefully.


Get It

Try It

Once upon a time, back in the day, one of our bosses had a special request. A special assignment. A secret assignment. There were several divisions in our organisation, and this boss was worried about another division, run by a handsome woman originally from IBM, for whom he may have had a romantic interest.

When the woman - let's call her 'Anna' - took over this division, she immediately saw that things weren't good, and decided she'd have to sack a lot of people, primarily but not exclusively in their IT admin department. Doing this in Sweden isn't easy, as employment is protected by law, and you probably have to go to court to sack anyone.

But Anna found a way around that. She was going to reorganise the division from top to bottom, make old managerial positions and corresponding duties obsolete, then - ahem - 'welcome' all managers to apply for the new positions - and summarily sack the ones she didn't like.

Our boss explained that the job was to go to work within their IT department, to 'follow the head honcho around like a rucksack', so one got to know what tricks he was up to. As Anna suspected he and the others had something up their sleeves - a way to extort or to put pressure on her when push came to shove.

The assignment, in other words, was to be on hand to recover the system from deliberate sabotage.

The official explanation for the placement was that the department needed a backup, and that the head honcho hadn't provided for one. This chap was self-educated and knew very little about Unix system administration in general, having taken only an introductory course in Pascal programming (!) in the north of the country before making the trek down to the nation's capital. From that point on, everything he did was based on books he bought with company money. He copied out interesting scripts (without understanding the code) because he thought they seemed 'cool'. And he was a control freak - a total control freak. And he was in thick with the people who were destroying the division - people Anna knew had to go. It was this situation Anna was most worried about.

It took some time for the savvy honcho to acquiesce to Anna's wishes in even the slightest. She had to visit him time and again and remind him, scold him.

'I TOLD YOU THAT YOU ARE TO TAKE HIM EVERYWHERE! TO LET HIM FOLLOW YOU AROUND LIKE A RUCKSACK! HE'S TO LEARN EVERYTHING YOU'RE DOING!'

The honcho would back off a bit, but he kept buying more and more books and putting in more and more shell scripts. He also messed with the system's on-disk architecture.

At one point, there was a glitch in the system. The system was used for payroll, with tens of thousands of employees depending on it, and it was also used by the corporate legal eagles. There couldn't be a glitch in the system - it wasn't tolerated.

But there it was.

The situation seemed pretty serious, so in comes a techie from the reseller. He was an amiable chap, sat himself down at the network console, and started typing away in a flurry. Two hundred fifty character command lines, and he'd get one character wrong at about the one hundred fifty mark. There was no 'arrow up' feature, so he had to type it all again. And he typed it just as fast. WHOOSH! And he'd finally get it right. And he took some long minutes to inspect the network. After which he leaned back in his chair. Troubled.

'What is it? What's wrong? Aside from the obvious, that is.'

He shook his head. 'I don't know. I'm not sure what it is. One thing's certain: this is not Unix!'

The overwhelming impression that the system was no longer what it was set out to be.

That's what happened to Apple.

They even openly declare it in their choice of names for their kernel.

XNU - 'X is Not Unix'.

Why they want to distinguish themselves from Unix isn't known. Chief honcho at that point was Avie Tevanian, and he was a true Unix guy, but he's also the one who, after looking at the disaster that was Apple's old 'MacOS', said: 'why don't we just build a new OS from the ground up?'

NeXT relied on FreeBSD. A good move. How much they changed in FreeBSD isn't known, but of course it's best to change as little as possible. There's already a team in place to take care of FreeBSD. Messing with vetted code takes time and resources and easily leads to bugs and vulnerabilities.

Not so at Apple. Time and again they pick up the latest open source offerings, then alter them to make them more 'Mac-like', whatever that is. Those makeovers can take ages. Infamously they held off on important updates to OpenBSD's sudo that the rest of the world got almost two years earlier. Famous hacker Charlie Miller has revealed the secret to hacking a Mac.

/Library/Documentation/Acknowledgements.rtf

That's a list of all the open source modules on the Mac.

All Charlie does is find the current version number of each component on his Mac, then find the website of the team responsible for that component, and check what their latest release version is. If their version number is higher than Apple's, he then checks their change log and sees what vulnerabilities they've patched. It's easy going.

But it's not just down to bugs and vulnerabilities with Apple's Unix. All these various flavours of Unix differ a bit, but Apple's Unix is way out in left field. So much so that one can be expected to utter at some point:

'This is not Unix.'

Let's start with the most basic of basics: file management.

Unix files admit of three basic permissions: read, write, execute. This is reflected in the APIs as well.

NAME
     open, openat -- open or create a file for reading or writing

SYNOPSIS
     #include <fcntl.h>

     int
     open(const char *path, int oflag, ...);

DESCRIPTION
     The flags specified for the oflag argument are formed by or'ing the fol-
     lowing values:

           O_RDONLY        open for reading only
           O_WRONLY        open for writing only
           O_RDWR          open for reading and writing

You can open a Unix file for reading and only reading. You have to explicitly ask to open a file for writing if you want to write to it.

All the information on how the file is stored in secondary storage is found in the file's iblock. Direct access for a few clusters perhaps, then pointers to clusters of additional clusters, then pointers to pointers to clusters, and so forth. The file itself is not moved because of a write operation. All the above details are hidden by the kernel. The thing that's most important is that the file itself doesn't move - it still has the same inode after the write.

So worked NeXTSTEP too. So did OS X too. Of course. Nothing could be more self-evident. Then the hedonists of Cupertino got in the picture.

You no longer write to existing files on Apple's OS. Nowhere save at the command line, where Unix remains Unix. But from Apple's own GUI? No. You never write to the files you think you're writing to. This is one reason the all-important hard link can't be preserved across file writes.

[Yes, this is true: you can't write to multi-linked files on an Apple OS without destroying everything. There are actually several reasons for this, the above is one. And no, Apple will never fix this. Ed.]

To say that this leads to unbelievably chaotic situations is to put it very lightly. There are so many things that Mac users don't even know they can't do. It's a crying shame. Apple will never own up. Apple will never tell their users. Oh look at the pretty screen!

So what do they do? Why can't they just open files for writing like everyone else on the planet?

'X is not Unix.'

First, they create a temporary location. Actually, they didn't come up with this themselves - it was already in place in the NeXT code. They do it in Windows too.

The purpose of this temporary location is to safeguard against mishaps - blackouts, brownouts, whatever. The application may also intermittently save to this location as the session proceeds.

But after this, when it's time to really save the file, according to the user's express command, the contents at the temporary location are WRITTEN to the actual file.

This affords maximum safety, as the temporary source need not be deleted until it's known the copy to the final location was successful. But no more. Sorry.

The inode doesn't change on a copy. Unnecessary and egregious disk operations are avoided. A copy is a simple write. Easy peasy. Piece of cake. A procedure as old as Unix. Except at Apple.

What they do at Apple is obliterate the original file, thus causing additional bookkeeping on the part of the kernel. If there were multiple links (hard links) then those links are mercilessly destroyed. At that point, when the system is finally ready, the temporary file and all its bookkeeping data are moved, not copied, to what was once the actual location (path, filename, etc) which for a fraction of a second didn't exist.

This of course leads to countless 'Three Stooges' complications.

Unix doesn't require a directory to have write permissions only to write to a file. A Unix directory contains only two data elements: the filename (sans path) and the inode. So if you're writing to a file, this information doesn't have to change.

But in Apple's strange world, the directory has to be written.

First, they have to remove the old entry for the file. Both the filename and the inode are removed.

Second, they have to insert the new entry for the (new) file. The filename will be the same, but the inode will not.

And to do that, they have to write to the directory. Not the file, but the directory.

But what happens if the directory doesn't give you write permissions?

Then nothing. But, and for the longest time, the diagnostic given Mac users was that something was wrong with their file. Not their directory, but a file. Which of course led them nowhere. They finally changed the wording after about - get ready - ten years. But that cockamamie system is still in place.

X is not Unix.

Unix is a system that was honed and perfected over decades. Unix wasn't made with security as its primary priority. But Unix of course observed the most basic and essential rules of the game.

Unix systems were made for a lot of people, including the twenty-five or so gurus working at Mountain Road in New Jersey. Those guys did a lot of wild and wacky things with code. Unix couldn't let their wayward code interfere with the system in general. And people might appreciate a modicum of privacy. So users could block off their own home areas (under /usr) so others couldn't get in and poke around. Many of them had experience on mainframe systems such as IBM's MVS and VM, and many worked on the General Electric Multics clones. They were familiar with the concepts.

Apple built a circuit board. They built a single-user system without a monitor or keyboard. There was no Time Sharing Option, no Interactive Structured Programming Facility, no round-robin task switching, none of that. They built a circuit board.

To this day, almost all Apple apps mislead their users and hide the fact that the systems are actually multiuser. The architects and the marketing droids at Apple deliberately reinforce this illusion. Almost all apps come with a half megabyte image file with a BIG ARROW that tells the user where to DRAG the file to make it 'install'. And this is often the wrong location.

Following are the allowed paths for applications on a Mac. These are paths that are automatically recognised by the system. You don't have to type in a full path - just type in the name - when you want to invoke an application along one of these paths. The system will automatically find it. In any of the locations given below. Any of them.

This data is resident in the system. This part of the system comes from NeXT.

Paths
-----
NSAllDomainsMask, NSAllApplicationsDirectory
(
    "~/Applications",
    "~/Applications/Utilities",
    "~/Developer/Applications",
    "~/Applications/Demos",
    "/Applications",
    "/Applications/Utilities",
    "/Developer/Applications",
    "/Applications/Demos",
    "/Network/Applications",
    "/Network/Applications/Utilities",
    "/Network/Developer/Applications",
    "/Network/Applications/Demos"
)

Note that several of these paths have been in use but been subsequently discarded by Apple. The 'Developer' paths were used for Project Builder, Interface Builder, Icon Composer, and all the other NeXT tools. And, for a while, by Apple's Xcode. But no more.

Disregarding the 'Developer' ones, you still have quite a lot of opportunities, most notably ~/Applications and ~/Applications/Utilities. These directly correspond to directories found at root. ~/Applications/Demos too, although you might find it unnecessary.

The important things:

- You don't have to worry about creating conflicts at root.
- Your software remains your software. No one else needs to have access.

Speaking of easy drags, Apple's Unix once had the Mother of All Exploits. It was fantastic. It didn't need a Charlie Miller or a Patrick Wardle. It didn't require hacking. It didn't require much in the skills department either. All you had to do was carry off a simple drag-drop. Only once.

Then the system was yours. You owned it.

No privilege escalation required. A simple configuration file could be picked up anywhere online.

The system was yours. You owned it. A simple-drag-drop.

No real Unix would allow something so stupid. Maybe it's good that they keep saying 'X is not Unix'.

There was namely a directory wide open for anyone to write to, but, on reboot, the system read this directory and followed its instructions to the letter.

That was it.

So OK, maybe this was a Big Blooper™? And the conscientious Apple wizards found out about the hole and patched it in record time?

No. They knew about it. All along. And they intentionally left it open.

For years.

Why, you ask? Good question. The consensus is that this was for some third party software that needed the hole to be able to work properly (!) and that Apple gambled that the hole would never be exploited, would never turn into an epidemic.

But it wasn't just a hole, to quote one of the researchers - it was a CRATER.

A crater so vast, so embarrassing, that it remains unparalleled in the world of computer science. To this day. No one else has ever done anything that stupid. Only Apple.

Filesystems are the Atlas of any system. Applications can screw up, anything can happen, but if your filesystem is not integral, you're screwed. You simply won't make it.

The Unix filesystem was robust, but even dmr and ken couldn't foresee what would come. Unix file management primitives are primitive. There are basic APIs opening, reading, and writing files, but not much beyond that. The command line programs such as cp and mv are robust. Yet there's no concept of the filesystem itself needing to be inviolate.

David Neal Cutler came upon that concept.

A filesystem was, in effect, a device. A device that had to be inviolate. You couldn't get at its innards to screw things up as you could do so infamously with 'MacOS'. You didn't need BIG LETTERS IN RED throughout the documentation WARNING people to be careful as the slightest screwup could HOSE THE ENTIRE SYSTEM.

No, the filesystem took care of that. The filesystem performed 'sanity checks' on incoming variables to make sure everything was OK. You didn't accept commands that essentially asked the filesystem to copy itself onto itself. And so forth.

File operations also need to be guarded by exception handlers. The handlers must be on the lookout for anything and everything - running out of disk space, permissions conflicts, blackouts, brownouts, hardware glitches - ANYTHING. And a good file system will meticulously craft a robust 'rollback' system, so that, if at any stage of the operation, things go south, then the filesystem knows how to revert it all.

Cutler implemented such a system on VMS, and he later adapted it to his 'NT' for Microsoft.

From the moment an application, on behalf of the user, requests a file operation, the filesystem takes over. And, if need be, it interacts directly with the user, not the application. It controls everything. The application is not involved.

Are there conflicts? Is there a reason to consider a rollback? The file system - and not the application - interacts with the user. The application is left out of it. As should be.

Filesystem code is as meticulous as driver code. Driver writers are very meticulous. Right down to their source code formatting. Things are laid out clearly and plainly. As they must be.

Filesystem code is tested as vigorously as driver code. Perhaps more so. All possible 'monkey tests' are performed in realtime, in real world scenarios. It's this code that becomes the 'Atlas' of the system, the foundation on which all else builds.

Apple never had such a system, and likely never will. The concept of the inviolate filesystem is rare in the world of Unix, but Apple can often take things one step farther - one step deeper.

Sometimes it's a simple slip-up in the code. Perpetrated by a newbie who's injudiciously been given access to code that shouldn't be accessible to newbies. A code change built on typically amateurish assumptions. Such as not needing exception handling to recover data in the event of a blackout. For this happens on Apple systems.

And Apple's responses? They sound like they're crafted by Redmond droids. 'Be more careful' and the like. Care is supposed to be taken when writing the code so that users are protected. Red ink is a sign that something's wrong. Back in the day when Apple had only DISK operating systems: that's one thing. That doesn't count anymore in the Major Leagues.

You do not, for example, enable a 'move' operation across physical disk boundaries, and begin by destroying the source, confident in your trust in Mac 'technology' that nothing can possibly go wrong. Yet that's exactly what's happened. And the vicious responses to those reporting this catastrophe were completely in line with what could be expected.

Unix systems don't have specially crafted user-level applications that hide things from user. It's one thing to keep irrelevant data out of the way, and another thing to act as though that data doesn't even exist. Telling programmers 'we don't want you going there' is much of the same. As is giving the in-folder coordinates of directories something like (16384, 16384). That's just childish. Like putting limericks in your kernel.

Your karma check for today:
There once was was a user that whined
his existing OS was so blind,
he'd do better to pirate
an OS that ran great
but found his hardware declined.
Please don't steal Mac OS!
Really, that's way uncool.
(C) Apple Computer, Inc.

Copyright (c) 2006 Apple Computer, Inc. All rights reserved. The purpose of this Apple software is to protect Apple copyrighted materials from unauthorized copying and use. You may not copy, modify, reverse engineer, publicly display, publicly perform, sublicense, transfer or redistribute this file, in whole or in part. If you have obtained a copy of this Apple software and do not have a valid license from Apple Computer to use it, please immediately destroy or delete it from your computer. Dont_Steal_Mac_OS_X

And yes, that's actually in Apple's OS as a kernel extension, located here.

/System/Library/Extensions/Dont Steal Mac OS X.kext

142 0 0xffffff7f83819000 0x5000 0x5000 com.apple.Dont_Steal_Mac_OS_X (7.0.0) 770F366C-A027-39FD-A618-BFAF66C67202 <14 9 8 5 3 1>

https://discussions.apple.com/thread/2616017

well, i was wandering around in system/library/extensions to find a solution to my unsolved problem (http://discussions.apple.com/thread.jspa?threadID=2513839) so i thought deleting old soundcard drivers might do it better. while that i saw DONT STEAL MAC OS X.kext file. i was curious. so i dragged a copy of that in my desktop then deleted the original file. could not boot again into mac os x

now that i erased my bootcamp partition to install a fresh snow leopard so that i could modify files. i installed snow leopard, then moved into other partition n dragged dont steal mac os x.kext to system/library/extensions back. but still cant boot the ex-snow leopard partition. is there anything that can be done without formatting the drive. cuz i really i mean really need to use that system. formatting is so lame u know. then loading everything back, i dont have that much of time tho i need to boot it up back again so that i can work . gimme a way outta this plz?

is there anything that can be done? it shud have been booting correctly by now but it aint booting.

Fascinating? It's time to stop being fascinated. It's time to start being disgusted.

Some Unix systems are still proprietary. But the most popular, most widespread, are not. Dennis Ritchie released several of his inventions, including the C programming language, into the public domain. Tim Berners-Lee released the 'web' itself into the public domain. But not Apple.

The Unix systems out there range from distros with the Linux kernel, such as Ubuntu, to the formidable Red Hat Enterprise Linux, which IBM just bought for billions upon billions, to the bulletproof OpenBSD. They're all open source. As they should be.

The world's in one corner, Microsoft and Apple are in the other.

Pick your team wisely and carefully.

Further Reading
The Technological: com.apple.Dont_Steal_Mac_OS_X

About Rixstep

Stockholm/London-based Rixstep are a constellation of programmers and support staff from Radsoft Laboratories who tired of Windows vulnerabilities, Linux driver issues, and cursing x86 hardware all day long. Rixstep have many years of experience behind their efforts, with teaching and consulting credentials from the likes of British Aerospace, General Electric, Lockheed Martin, Lloyds TSB, SAAB Defence Systems, British Broadcasting Corporation, Barclays Bank, IBM, Microsoft, and Sony/Ericsson.

Rixstep and Radsoft products are or have been in use by Sweden's Royal Mail, Sony/Ericsson, the US Department of Defense, the offices of the US Supreme Court, the Government of Western Australia, the German Federal Police, Verizon Wireless, Los Alamos National Laboratory, Microsoft Corporation, the New York Times, Apple Inc, Oxford University, and hundreds of research institutes around the globe. See here.

All Content and Software Copyright © Rixstep. All Rights Reserved.

CONTACT INFO:
John Cattelin
Media Contact
contact@rixstep.com
PURCHASE INFO:
ACP/Xfile licences
User/Family/Business
http://rixstep.com/buy
About | ACP | Buy Stuff | Industry Watch | Learning Curve | Newsletter | Search | Test Drive
Copyright © Rixstep. All rights reserved.