Rixstep
 About | ACP | Buy Stuff | Industry Watch | Learning Curve | Newsletter | Search | Test Drive
Home » Industry Watch

The Rixstep XA Utilities and Tools

It's quite the collection!


Get It

Try It

We've spent quite a lot of time developing applications that specifically target macOS extended attributes and even combat Apple's abuse of that technology. They're quite the collection.

The Backbone

The backbone for this technology is the xattr API. It's presented as part of the 'BSD General Commands Manual' but there aren't many BSDs that use it. This is Apple's own. The API was introduced in the mid-2000s.

Programmatically the interface is controlled by four function calls.

  • listxattr lists the XAs for a given path.
  • getxattr gets a given XA for a given path.
  • setxattr sets an XA for a given path.
  • removexattr removes an XA at a given path.

Those four calls can be combined to achieve mostly anything at all.

Our First

Our first attempt to grapple with this technology was our own command-line tool xattr, using the above function calls. xattr wasn't used much but was created as a first step in the overall research effort.

xattr was followed by Xattr.app, which is the same thing but in a graphical environment.

This is the way Xattr looked back in 2006 when it was combatting the Oompa Loompa worm.



This is the way Xattr looks today.



Xattr.app lists all XAs per file, displays the contents of each XA in either hex or text format, exports XAs, imports XAs too, and removes XAs, either individually or all at once.

xabatch

This was an attempt to create a general-purpose XA tool. And it worked - and is still in wide use today, particularly in admin scripts. xabatch can be used to both add and delete XAs - in batch. This is accomplished with a proprietary DSV script file format, elegant and to the point. An xabatch file tells the tool which XAs are to be removed and which, if any, XAs are to be added.

xabatch <command file> <file list>

xabatch expects a path to the script (command file) as argv[1]. Thereafter follow paths to the files that are to be adjusted.

XaBatch

The GUI version of the above tool. This one uses low-level directory enumeration. Like the tool, it's very fast. And it's also easy to use. An XaBatch document contains a list of commands, the starting path, and the option to recurse, all of which is edited within the application's graphical interface.



This is XaBatch in action, modifying XAs on some 100 thousand files in a minute.

Why is all this important?

Why is all this important? A good question. Extended attributes were heralded when they arrived, as they replaced the useless and cumbersome resource forks idea from Apple's best-forgotten 'Frog' (beige box) days. They gave application developers the opportunity to add 'enhancements' to applications (as long as they weren't misused - see TN2034).

But Apple soon discovered another use for XAs in the immediate wake of the advent of the iPhone. They began developing what can only be described as a diabolical system for total control of the user, with their 'Quarantine' XA used as the entry port to it all.

Huge chunks of legacy code were rewritten. Basic tools like 'Unarchiver.app' were modded to take note of the key XA and act accordingly. The system's launch services were modded as well. And so forth. And, on top of this, Apple built an elaborate superstructure to control every aspect of computer use - to suppress user freedom.

The reason? Apple claimed all along it was for 'user security', but research in the field proved this to be false. Other systems derivative of BSD - and regarded as much more secure - don't even consider use of the technology. And they're still miles more secure than Apple will ever be.

Advocates of the super-secure OpenBSD or any number of popular Linux distros don't have to ask permission to save files. They're not told that downloads are evil. They don't find their own operating system continually getting in the way. And yet they're super-secure.

The reason had nothing to do with protecting the users, making life better for them.

It was all about money. (It always is.)

The reason was to corral all independent software development into the Apple App Store, Apple's ginormous cash cow, much more formidable than most people imagine. Exacting a breathtaking 30% commission on all sales - three times the industry norm - plus a nonrefundable $100 fee just to get the chance to be featured, Apple in essence made a gutted labour force pay for the privilege of being rejected. Placing the icon for their App Store on the desktop, they almost guaranteed they'd corner and control the entire market - and they exploited this ruthlessly. This in turn led to their being denounced by programming legend and venture capitalist Paul Graham, the New York Times, and Forbes, amongst others. Mark Pilgrim wrote his famous essay 'Tinkerer's Sunset' as a reaction to these developments at Apple.

Yet, for all Apple's nefarious machinations, they cannot exert 100% control. Not on the Mac. iPhone users can of course 'jailbreak' their devices to get their freedom, but Mac users need none of that - the system has to allow users to do what they want with their own property. So Apple had to find ways to push people into the corral anyway.

Thus began our game of 'cat and mouse' with Apple, where it's difficult to determine who's the cat and who's the mouse. We're used to that over here - we've been fighting evil corporate interests since Day One, starting with Microsoft Windows. But few people suspected that Apple, of all companies, would try a similar dishonest trick. But they did. And so we dug and dug and dug. It wasn't easy, or much fun, but we ended up coming out on top.

That's why it's important.

appleclean

Things looked bleak with the approach of Catalina. Admins wrote to us with their forecasts of doom. Some of them manage major networks with lots of Macs, and the 'new Apple Inc' was making life difficult for them in many ways. (One other way, aside from the above one with XAs, was their total trashing of the once-superlative defaults system, to what end no one could fathom (and still can't - it's shaky, reminiscent of the thinking behind their 'undo move to trash' scheme).

Another of course was the junkyard of legacy NeXT (Cocoa) APIs and frameworks. Methods disappeared and were not replaced. No clue was given as to what was going on. One drag-drop method was changed three times in a single year (with perfectly good versions deprecated of course - meaning people had to back through all their code that had been working fine all those years to grapple with it) only to be scheduled for scratching in the future - drag-drop is no longer 'safe', according to Apple).

The first ray of hope came when it was found that Apple's new system couldn't quarantine command-line tools. Apple put seals on their own binaries but do not offer third-party a way to do it too. Anything run at the command line will run - regardless.

We created appleclean in that spirit.

appleclean is actually a very simple tool - the equivalent of 'xattr -crsv [target]', but it does that in its own code. This gives us the option of adding more code later if needed - such as code to purge the launch service caches if needed (but it's not needed yet).

We also enhanced two earlier applications - CandS and Lightman - to be aware of activity in Safari and other web utilities. When they see something happening, they jump in and clean things up - in realtime. We called this 'Seahaven', as the analogy to the movie 'Truman Show' became immediately and eminently apparent: it feels like a prison, and the idea is to make you think it is, but it's actually possible to get out anyway.



CandS ('Clean and Seal') was made to take care of downloads - you cleanse them manually (drag-drop) as soon as you get them. Seahaven was added so Safari and general web activity was detectable. Lightman already reports on system activity, so the refit was easy.



Once the picture became clear, we came out with both six-shooters blazing. We wanted people to understand what we'd discovered - we wanted them to stop blindly trusting their favourite company. A company that whacks them with a 40% profit margin on their products - sometimes almost half the price is pure air. (And they seem willing to contribute anyway, as if Tim Cook were their Queen Bee. There's a great video by Louis Grossman where he tries to get a female customer to just consider a ThinkPad, as her MBP is broken again, but she literally won't listen - she just doesn't hear him.)

And still we weren't done. It was time to delve into the innards of Spotlight.

Spotlight is powered by the file system events daemon FSEvents. FSEvents churns out a lot of data, used by Spotlight and any application that's interested. FSEvents can be considered Apple's answer to the Microsoft change notification - and it goes further, much much further. It was obvious that we needed a new 'Software Tool' - to borrow the analogy of Unix legend Brian Kernighan - and that 'tool' (actually a Cocoa application) became Changes.



What you get out of Changes is what you put into it - just as it is with the FSEvents daemon. No holds are barred. Changes makes it possible to study the activity of FSEvents in realtime. And boy, is there a lot of data available - and a lot of activity going on in your system! All the time. It's ferocious!

The outcome of a week or two studying FSEvents with Changes resulted in the first version of what today is a staple that's always running on our desktops: Keymaster.

This first version of Keymaster was a singleton, offering to guard standard directories under user home and reporting on total XA 'infections'. This turned into a second (now final) version of Keymaster, which is document-based, so users can configure whatever directories they want. (The first version of Keymaster was renamed Keymaster Solo and released for free distribution, at no cost.)

Rollcall

So we have quite the collection here.

First, created in all naivety and trust, are the Cocoa/command-line duo Xattr and xattr. They do the job, but they don't address the issues Apple introduced.

XaBatch and xabatch are another duo, but these can in fact be used to cleanse a system asynchronously. Even creating an XaBatch document with a single entry for the Apple quarantine XA would do it, especially if the root is set to ~/Downloads. A second document could guard ~/Desktop, for example.

appleclean is included with all Rixstep downloads. The idea is to drop down to a command line immediately you unzip your download and from the download directory run:

% ./appleclean

And it is fast - very fast.

(Alternatively run xattr -crsv <download directory> before unpacking.)

appleclean makes things simpler for those unfamiliar with the command line and it provides for future updates should Apple decide to up the ante and get even nastier.

CandS, used (with drag-drop) to manually 'clean and seal' downloads (or other files such as image files edited by Apple's Preview.app), now adds the 'Seahaven' monitor for web ingress. Lightman, an OS diagnostic tool, does the same.

Changes can at the very least be very educational, as it exposes the level of and type of activity behind the scenes in your computer.

And, finally, Keymaster provides complete synchronous protection and control.

This is about as good as you can get. What happens at this impasse is up to Apple. As they're likely to continue along this same path, at least for a while, it might be prudent to take a closer look at alternative platforms such as OpenBSD and Ubuntu.

One thing is certain: no system deserves to survive with Apple's bad attitude.

Further Reading
The Catalina Files
COVID19 Resource Page

About Rixstep

Stockholm/London-based Rixstep are a constellation of programmers and support staff from Radsoft Laboratories who tired of Windows vulnerabilities, Linux driver issues, and cursing x86 hardware all day long. Rixstep have many years of experience behind their efforts, with teaching and consulting credentials from the likes of British Aerospace, General Electric, Lockheed Martin, Lloyds TSB, SAAB Defence Systems, British Broadcasting Corporation, Barclays Bank, IBM, Microsoft, and Sony/Ericsson.

Rixstep and Radsoft products are or have been in use by Sweden's Royal Mail, Sony/Ericsson, the US Department of Defense, the offices of the US Supreme Court, the Government of Western Australia, the German Federal Police, Verizon Wireless, Los Alamos National Laboratory, Microsoft Corporation, the New York Times, Apple Inc, Oxford University, and hundreds of research institutes around the globe. See here.

All Content and Software Copyright © Rixstep. All Rights Reserved.

CONTACT INFO:
John Cattelin
Media Contact
contact@rixstep.com
PURCHASE INFO:
ACP/Xfile licences
User/Family/Business
http://rixstep.com/buy
About | ACP | Buy Stuff | Industry Watch | Learning Curve | Newsletter | Search | Test Drive
Copyright © Rixstep. All rights reserved.