|Home » Learning Curve » Developers Workshop
A Walk on the Wild Side
'This little piece of geniality...'
ALISO VIEJO (Rixstep) — There's quite a story behind this video. (To save time, speed it up to 2x so it takes only half an hour. Odds are you still won't last through the whole thing.)
[What are you looking at? The above application is Xfile, Rixstep's file manager for Apple's 'macOS' (OS X, built on NeXT's NeXTSTEP and OPENSTEP) operating system. Xfile hones in on the considerable 'hive' (directory tree) of Xcode.app, Apple's application development system, with over 350,000 files in close to 100,000 directories. You can see the directory tree on the left in Xfile, and a directory listing for each selected directory on the right. Neither Apple's default file manager 'Finder' nor the third-party 'PathFinder' (née SNAX) are capable of expanding that hive, much less enumerating it, as Xfile is seen to do. After speeding through all ~100,000 directories and ~350,000 files, Xfile makes a quick stopover at /dev, the crucial macOS system directory that Apple's Finder, now more than twenty years after its introduction, still can't 'find', much less display. Finally, Xfile returns to its starting point, at Xcode. That's what you're looking at. Xfile traverses and lists the contents of more than 1,600 directories per minute, or more than 27 directories per second. Ed.]
But, to begin this story, one must of needs go back even further and begin with an even earlier story. A story about this.
'CIP': it stands for 'Chris (Pirillo's) IP Probe'. Chris, of Lockergnome fame, put us on the map back in 1998 and later wrote to us with a request.
This was of course back in the dialup days. 28.8 modems and wonderful stuff like that. Speeding up site access was a big thing back then. Using the system hosts file to cut out the DNS middleman meant a tremendous speed boost. So ideally one resolved all one's URLs before one used them.
Chris had a utility that did these DNS resolutions alright - but it was written in Visual Basic and could only resolve one URL at a time. All the time between query and response was 'wait time' - wasted time. S-L-O-W.
Could we do something better, he asked.
Based in Cambridge at the time and teaching multitasking and synchronisation to the likes of Lloyds, the BBC, and the IBM think tank, we knew it could be done and how it could be done. We knew how we could really make the app sizzle.
The result was CIP.
CIP incorporates just about every whiz-bang trick in the Cutler playbook. All possible dedicated and undedicated synchronisation objects. So yes, it's fast. Really fast.
The graphic above also hints at how how spam and other unwanted secondary URLs could be blocked, in conjunction with a utility from the Bloatbusters. So that stuff you didn't want was looped back.
And everything was then fine and jim-dandy - until one day we got a letter from a chap in Singapore. He had an issue with CIP: it was when going to 'File->New'. Then he ran into a snag. Because it took a long time to clear the display. We'd never experienced anything like that.
'How many items do you have?' we asked. 'We got about 2,000 here.'
'Oh I got about 20,000, 25,000', he told us.
Windows (and Linux GUIs and just about every GUI out there, save for Apple's macOS) has menus on windows. Document windows don't exist. There is no separate application menu. If you want to start a new file, you use the window you already have, you don't (you can't) create a new one.
Therefore the need to deallocate the memory you're already using. And traditional methods worked fine - but not for datasets with tens of thousands of records.
So traditional memory management wasn't going to do it. But crafty Dave had something called the 'heap' API. You could programmatically create a 'heap' in free memory, then allocate like mad on that heap, and then, when the time came, you didn't need to free each and every allocation you'd made on that heap, you just obliterated the heap itself. All in one system call.
But NeXT's API had nothing like that, and Unix didn't either. There were third-party solutions available - but on licence for hundreds upon hundreds of dollars. So what to do? For the plan had been to port CIP to OS X.
Thus a question was sent to a Cocoa dev mailing list. Bad Move™.
One of the first comments was that no listing, anywhere, ever, could have as much as 4,000 records.
You can't make that shit up.
A certain Swiss programmer, more versed in old 'MacOS' than the new NeXT technology, who'd written to us earlier, boasting about his front end to an open source chess machine, wrote that we couldn't blame Apple for our own crappy code. Whereupon they found out that the sample code that we'd posted came directly from Aaron Hillegass. Panic ensued, with one of the 'Landed Gentry' yelping: 'Aaron! Get in here! We know you're in here, lurking!'
But the cowboy of course never appeared.
At issue was the prospect of having to delete thousands of object instantiations.
A quick tutorial.
NeXT's NSTableView is a purist take on object orientation. Microsoft's counterpart ListView makes a few compromises. It's possible to populate a ListView with static data. It's also possible to populate it dynamically.
The only way to populate an NSTableView is dynamically. The dynamic approach works like this.
- The client app tells the tableview to start reloading things.
- This initiates a 'callback' 'Q&A' from the tableview to the client.
- The tableview figures out how many (and which) rows are visible, and asks the client how many rows all told. the client responds.
- The tableview then begins a long series of queries for data per row and column. Each of these cells can contain an instantiation of a sovereign object.
To add: NeXT frameworks have never been satisfactorily explicit on 'selected' and 'focused' states. Microsoft's ListView is. (You get a dashed line around the row.) NSTableView has no method of visual feedback for a row that is focused but not selected, and, in fact, no way to indicate which row (of all the selected rows) is in fact in focus. The NeXT APIs can help at programmatic level, but the user is left without a clue.
Some of the above is due to the fact that Microsoft's version can retain state data, but the Apple version cannot. This gets dicey when refreshing a listing, as NSTableView can't remember what's been selected and focused, leading to hilarious bloopers with many macOS well known software titles.
Part of the above anomaly is circumvented by the fact that Apple's and NeXT's OSes and GUIs - in contrast to every other platform out there - admits of multiple documents. But the question remains: can one successfully and satisfactorily maintain an undimensioned record listing? What happens if, despite the opportunity to create additional documents with thousands more records, with possibly dozens or more columns of data cells, one needs to use the same NSTableView over and over again?
And what happens when you have to delete it all, row by row and column by column?
You're seeing the solution in the above video.
What's not seen - but what works wonderfully - is that selections and the NeXT equivalent of 'focus' are retained on refreshes.
A humorous aside: despite having every opportunity to accomplish this themselves, Microsoft, when attempting to deal with a ListView refresh in their Explorer, do the following.
- Create a new but as yet invisible ListView behind the current visible one.
- Populate this invisible ListView.
- Make the invisible ListView visible and bring it to the foreground.
- Obliterate the old ListView.
None of that would be necessary if Microsoft 'did it right' but, unfortunately, they don't.
The solution used in Xfile deals with the above impossibility, and you can see the results in the video. Similar solutions are used by Xscan and macTag. So yes, it's possible to build a robust file manager, even if everybody else is still tripping up and resorting to lame excuses. No other file manager for the platform can even expand a directory hive like that, much less speed through it as Xfile can be seen doing above.
A similar experiment was performed previously. Yes, the competition failed miserably that time as well.
The hardware's fast enough, the underlying frameworks and rendering code are good enough - but isn't it amazing that, with so much brilliant technology, today's application developers can do so little?
For what is programmatically prohibited by other file managers is, to Xfile, merely a walk in the park.
What To Do
- Fix NSTableView. It's been broken for over ten years. Those who were strategically left behind when Scott Forstall recruited for the iPhone project (yes that long ago) messed up the original NeXT logic, quite obviously never understood it, tried feebly a few times to fix it, but no, never got it right again. So fix it, dammit. (And while you're at it, please fix the .DS_Store bug and the kerning nonsense. But don't wear yourselves out.)
[No exaggeration: NeXT's NSTableView was best in the world - until Forstall's rejects got their hands on it. But the tip of the iceberg. Ed.]
- Add visual feedback for 'focused but not selected'. Or at least for 'focused and selected', as the former may cause the current code owners to frazzle. Let NSTableView have custody of row refs, not by index but by unique ID. (Too many ISVs actually use row indexes to preserve selections - just how dumb this is can't be described in sufficiently poetic language.)
[For those who still don't get it: it's where your keyboard focus is. You'll see it after the fact if you hit arrow up or arrow down - but not before. Only your program will know. Is that fair? Ed.]
'I have this little piece of geniality running on a 56k and WIN98 SE. I have no clue on how it works but it does the job. Websites are found faster and show up on your screen substantially quicker. The developers of this little program deserve one hell of a pat on the back. I recommend to read the web page, what I'm going to do right now to set up at best this little miracle. Great!'
Radsoft Product Gallery: CIP
Industry Watch: Sanity Checks at Apple
Industry Watch: Xfile Unleashed!
Industry Watch: Xfile: 'Every Other Day'
Industry Watch: The QuickBooks Disaster
Industry Watch: 'When I'm 64'
Industry Watch: 9,737 files (97,370 data cells) in 0.143 seconds
Industry Watch: Xfile: A Walk in the Park
The Technological: Apple's File System APIs
Coldspots: Snow Leopard: Under the Hood 1
Learning Curve: iPod Therefore iPay
Learning Curve: A Sanity Check for Apple
Learning Curve: 4893378 FAQ
Learning Curve: 4893378: 'Expected Behaviour'
Learning Curve: Hosing OS X with Apple's Idea of 'Expected Behaviour'
Learning Curve: Rebel Scum: More Attacks on 'Expected Behaviour'
Learning Curve: Take a Stroll with Xfile Part One
Learning Curve: Take a Stroll with Xfile Part Two
Learning Curve: Take a Stroll with Xfile Part Three
Learning Curve: One Thousand Two Hundred?
Learning Curve: PRISM: Staying Under the Radar
Learning Curve: Taps for the Micro
Learning Curve: The Lost Story
Developers Workshop: HFS: The Good & The Bad
Developers Workshop: Oh Where Oh Where Did My Resource Fork Go?
Developers Workshop: /home & /net
Developers Workshop: Designers Gone Wild
Developers Workshop: Apple's UI Hall of Shame
Developers Workshop: NSTableView Revisited
Developers Workshop: The Longest Screed
Developers Workshop: It Wasn't Good Then, It's No Better Now
Developers Workshop: Bug Reports & Core Rot
Developers Workshop: _getdirentries_is_not_available...
Hotspots: Double Nuts
Hotspots: A Litter of Nitwits
Stockholm/London-based Rixstep are a constellation of programmers and support staff from Radsoft Laboratories who tired of Windows vulnerabilities, Linux driver issues, and cursing x86 hardware all day long. Rixstep have many years of experience behind their efforts, with teaching and consulting credentials from the likes of British Aerospace, General Electric, Lockheed Martin, Lloyds TSB, SAAB Defence Systems, British Broadcasting Corporation, Barclays Bank, IBM, Microsoft, and Sony/Ericsson.
Rixstep and Radsoft products are or have been in use by Sweden's Royal Mail, Sony/Ericsson, the US Department of Defense, the offices of the US Supreme Court, the Government of Western Australia, the German Federal Police, Verizon Wireless, Los Alamos National Laboratory, Microsoft Corporation, the New York Times, Apple Inc, Oxford University, and hundreds of research institutes around the globe. See here.
All Content and Software Copyright © Rixstep. All Rights Reserved.