About | ACP | Buy | Industry Watch | Learning Curve | News | Products | Search | Substack
Home » Learning Curve

An Object Lesson in Object-Based

One way in, one way out. Three would be a crowd.

Get It

Try It

ZÜRICH (Rixstep) — To steal words from a Dave: a filesystem - its management, its caretaker - must be 'impervious', independent, sovereign, and 'totally encapsulated'.

A filesystem must be rigorous. It must not perform operations that can damage its integrity, or the integrity of the data of its user. It must necessarily perform extensive 'sanity checks', as they're called by the industry, to make sure the client application isn't trying to throw a curve ball, inadvertently or otherwise, even deliberately. Mistakes can happen, and so too for that matter can malfeasance.

Device driver writers worry about stuff like this all the time. So too do writers of filesystems. At most software houses at any rate. One can't simply let loose a number of random APIs, forget the sanity checking, and hope for the best. Next thing you know, you'd have a massive data loss scandal on your hands.

And no one wants that to happen (again).

When asked to describe his old boss Dave Cutler, a former employee of his used one word: 'Marine'. He went on to tell of how they used to watch Dave debugging his burgeoning kernel. 'He was racing Formula One.'

It's not known if Dave actually wrote the following API, much less designed it. His blueprint for Prism/Emerald didn't include a GUI. Real men don't use GUIs, was his creed. GUis are as bad as Unix, and that's written by a committee of PhDs. An operating system must be the conception of a single system architect.

(Like Ken Thompson?)

But Dave was finally told, about two years into his project, that he wasn't just building a file server, but also a client, and both (not just the client but also the server - 'zero administration') needed a graphical user interface. And Dave's system design is all over it.

Dave wrote VMS for Digital Equipment. Colleagues at IBM knew it well. It was DEC's cash cow, and it was bulletproof. Yes, bulletproof, they said. And his followup to VMS, with an acronym created by doing a 'HAL' on the original name, would be even more so: all important functionality would be encapsulated. The system would be 'object based'.

(Not object oriented, as some misunderstood, but object based. Objects that didn't allow access - any access at all - to their internals. You simply sent requests to their 'portals', and you got a response. Perhaps the answer to a question. Or you could request an operation, in which case you got both results and a report back. But you never got to see the internals.)

And you could send any wacky request you wanted, but the object wouldn't go down. It was programmed that way. No need for red ink in a programming manual. For as such matters were of an ultimately crucial nature, and as the object writers knew best about all happening inside the object, they should and must take responsibility for protecting it.

Then the protective code need be written only once, not tens of thousands of times by neophyte programmers who don't really know what they're doing, jeopardising the system and user data.

The protective code was written only once, not thousands of times. It was written correctly and tested rigorously. The object protected itself. No mishaps: it even compensated for bad client code. The object stood, the system stood, the user none the wiser. It worked, it just worked. It was insanely great.

There may be many ways to corrupt a Windows system. Dave didn't know initially his Prism/Emerald would be a Windows system, only client-server. But the API he used for Windows, SHFileOperation, is an object lesson in object-based.

SHFileOperation takes but one argument: a pointer to a structure. The client fills in a structure, like filling in an application form, and submits it. Fuzzing can be used to overwhelm the system with random input. Will the system stand? The fuzzing will tell you. And you'll fix the code until it does stand.

And there can be only one way in and one way out. One entry point and one exit. Like in all good program design.


The structure that SHFileOperation sends along is a SHFILEOPSTRUCT. It comes in two flavours: 'A' ('ANSI') and 'W' ('wide'). (Apple filesystems use UTF-8. As it should be.)

typedef struct _SHFILEOPSTRUCTA {
  HWND         hwnd;
  UINT         wFunc;
  PCZZSTR      pFrom;
  PCZZSTR      pTo;
  BOOL         fAnyOperationsAborted;
  LPVOID       hNameMappings;
  PCSTR        lpszProgressTitle;

First the preliminaries.

hwnd is of type HWND. Both stand for 'handle to window', a unique (but opaque) integer identifier for the calling window (or the main window of the caller).

wFunc is a good example of Hungarian notation, and Windows programming is full of it. The 'w' initially stood for 'word', a 16-bit value, but today it's a UINT (unsigned int) instead.

PCZZSTR is a mouthful, stands for 'pointer to constant double-zero terminated string'. Yes, the string will have a double zero at the end, because it can contain more than one zero-terminated string, because Windows doesn't have fancy stuff like NSArray.

fFlags, with prefix 'f' which can be assumed to stand for 'flags', is of type FILEOP_FLAGS, which, it can be further assumed, is big enough to hold several bitwise values.

fAnyOperationsAborted (note the Hungarian 'f' again) is simply a BOOL, a YES or a NO. Note that this is in addition to the function's return value.

hNameMappings ('h' for 'handle' again) is of type LPVOID (long pointer to void) and will point opaquely to data.

lpszProgressTitle (long pointer to zero-terminated string) has an alternate window title used in certain circumstances.

So we request a file operation, specify what files are involved, how the operation and user interaction are to proceed, and where our application window is (for cosmetic and other purposes - the system can disable us during the operation).

That's pretty much all. So what operations are available?

The Ops

SHFileOperation performs four types of operations. The wFunc field is one of the following.

FO_COPYCopy the files specified in the pFrom member to the location specified in the pTo member.
FO_DELETEDelete the files specified in pFrom.
FO_MOVEMove the files specified in pFrom to the location specified in pTo.
FO_RENAME  Rename the file specified in pFrom. You cannot use this flag to rename multiple files with a single function call. Use FO_MOVE instead.

That's it. FO_RENAME can seem a bit redundant, mostly because it is. (Some Unix concepts can be hard to grasp for some.)

The Flags

This provides further insight into what's going on.

FOF_ALLOWUNDOIn later versions, the scope of an undo is a user session.
FOF_FILESONLYPerform the operation only on files (not on folders) if a wildcard file name (.) is specified.
FOF_MULTIDESTFILESThe pTo member specifies multiple destination files (one for each source file in pFrom) rather than one directory where all source files are to be deposited.
FOF_NOCONFIRMATIONRespond with Yes to All for any dialog box that is displayed.
FOF_NOCONFIRMMKDIRDo not ask the user to confirm the creation of a new directory if the operation requires one to be created.
FOF_NOCOPYSECURITYATTRIBS  Do not copy the security attributes of the file. The destination file receives the security attributes of its new folder.
FOF_NOERRORUIDo not display a dialog to the user if an error occurs.
FOF_NORECURSIONOnly perform the operation in the local directory. Do not operate recursively into subdirectories, which is the default behaviour.
FOF_NO_UIPerform the operation silently, presenting no UI to the user.
FOF_RENAMEONCOLLISIONGive the file being operated on a new name in a move, copy, or rename operation if a file with the target name already exists at the destination.
FOF_SILENTDo not display a progress dialog box.
FOF_SIMPLEPROGRESSDisplay a progress dialog box but do not show individual file names as they are operated on.
FOF_WANTMAPPINGHANDLEIf FOF_RENAMEONCOLLISION is specified and any files were renamed, assign a name mapping object that contains their old and new names to the hNameMappings member. This object must be freed using SHFreeNameMappings when it is no longer needed.
FOF_WANTNUKEWARNINGSend a warning if a file is being permanently destroyed during a delete operation rather than recycled.

A number of things can be immediately inferred.

√ That there's the flag FOF_SILENT means that SHFileOperation normally is not silent. This is yet another reminder that the client application is not in control - the 'object' behind SHFileOperation is.

FOF_NO_UI and FOF_SIMPLEPROGRESS are more indications of the above. Once that call is sent off to SHFileOperation, the system takes over. The system creates its own threads in your process and positions its own GUI relative to yours (the hwnd).

The application isn't performing file operations - all the application does is send a request to the filesystem. Then the application butts out. At that point, it's a conversation between the user and the filesystem. Three would be a crowd.

The Error Codes

These too can give a glimpse into what's going on.

DE_SAMEFILEThe source and destination files are the same file.
DE_MANYSRC1DESTMultiple file paths were specified in the source buffer, but only one destination file path.
DE_DIFFDIRRename operation was specified but the destination path is a different directory. Use the move operation instead.
DE_ROOTDIRThe source is a root directory, which cannot be moved or renamed.
DE_OPCANCELLEDThe operation was canceled by the user, or silently canceled if the appropriate flags were supplied to SHFileOperation.
DE_DESTSUBTREEThe destination is a subtree of the source.
DE_ACCESSDENIEDSRCSecurity settings denied access to the source.
DE_PATHTOODEEPThe source or destination path exceeded or would exceed MAX_PATH.
DE_MANYDESTThe operation involved multiple destination paths, which can fail in the case of a move operation.
DE_INVALIDFILESThe path in the source or destination or both was invalid.
DE_DESTSAMETREEThe source and destination have the same parent folder.
DE_FLDDESTISFILEThe destination path is an existing file.
DE_FILEDESTISFLDThe destination path is an existing folder.
DE_FILENAMETOOLONGThe name of the file exceeds MAX_PATH.
DE_DEST_IS_CDROMThe destination is a read-only CD-ROM, possibly unformatted.
DE_DEST_IS_DVDThe destination is a read-only DVD, possibly unformatted.
DE_DEST_IS_CDRECORD  The destination is a writable CD-ROM, possibly unformatted.
DE_FILE_TOO_LARGEThe file involved in the operation is too large for the destination media or file system.
DE_SRC_IS_CDROMThe source is a read-only CD-ROM, possibly unformatted.
DE_SRC_IS_DVDThe source is a read-only DVD, possibly unformatted.
DE_SRC_IS_CDRECORDThe source is a writable CD-ROM, possibly unformatted.
DE_ERROR_MAXMAX_PATH was exceeded during the operation.

DE_SAMEFILE: the Unix command line won't let you try to copy a file onto itself, but would Apple's APIs?

cp: XXXX.XXXX and XXXX.XXXX are identical (not copied).

DE_ROOTDIR, DE_DESTSAMETREE, DE_FILEDESTISFLD, and DE_DESTSUBTREE would be interesting to test. 'TFF' escaped some of those traps by sheer luck in the past.

√ Several of the above codes refer to 'curve balls' tossed by the client (application). They can be typos, they can be malicious in nature, they can be anything.

√ Summing up: the client application has full control, with fine granularity. But more; the system, not the client application, has full control for the entire operation. Keep your hands off the drivers, Ken said.

Back to the Beige Box

This isn't quite back to the beige box, but it's close: this is from Apple's early documentation for their 'Carbon' API for OS X. (The Carbon API represents Apple's 'old' or 'beige box' API in use before the transition to OS X.)

As of 18 February 2002, Apple had a 'Carbon' API replacing the old 'beige box' API, for use on OS X. (10.2 Jaguar would be released seven months later, 24 August.) The documentation at that time covered OS X versions 10.0 and 10.1.

Caveats that were mentioned in February 2002:

'You should not access File Manager structures directly if accessor functions for the structure exist. For example, you should call PBGetFCBInfo rather than calling LMGetFCBSPtr and walking the FCB table.'

'Similarly, you should create file system specification (FSSpec) records using the functions PBMakeFSSpec or FSMakeFSSpec instead of filling in your own structures. File system specification records must contain a volume reference number, a parent directory, and a name. Substituting a working directory for the volume reference number or a full or partial pathname for the name is not supported.'

'Functions that cannot be called by PowerPC applications (such as PBGetAltAccessSync and PBGetAltAccessAsync) are not supported.'

As of 18 February 2002, Carbon had 19 functions to get or set catalog info:

FSGetCatalogInfo, FSSetCatalogInfo, FSpSetFInfo, HGetFInfo, HSetFInfo, PBGetCatInfoAsync, PBGetCatInfoSync, PBGetCatalogInfoAsync, PBGetCatalogInfoSync, PBGetXCatInfoAsync, PBGetXCatInfoSync, PBHGetFInfoAsync, PBHGetFInfoSync, PBHSetFInfoAsync, PBHSetFInfoSync, PBSetCatInfoAsync, PBSetCatInfoSync, PBSetCatalogInfoAsync, PBSetCatalogInfoSyncv

4 functions for copying files:

PBHCopyFileAsync, PBHCopyFileSync, PBHMoveRenameAsync, PBHMoveRenameSync

7 functions for creating directories:

DirCreate, FSCreateDirectoryUnicode, FSpDirCreate, PBCreateDirectoryUnicodeAsync, PBCreateDirectoryUnicodeSync, PBDirCreateAsync, PBDirCreateSync

7 functions for creating files (sic):

FSpCreate, FSCreateFileUnicode, HCreate, PBHCreateAsync, PBHCreateSync, PBCreateFileUnicodeAsync, PBCreateFileUnicodeSync

7 functions for deleting directories and files:

FSpDelete, FSDeleteObject, HDelete, PBHDeleteAsync, PBDeleteObjectAsync, PBDeleteObjectSync, PBHDeleteSync

6 functions for exchanging files (CNID 16):

FSExchangeObjects, FSpExchangeFiles, PBExchangeFilesAsync, PBExchangeFilesSync, PBExchangeObjectsAsync, PBExchangeObjectsSync

14 functions for moving and renaming directories and files:

CatMove, FSMoveObject, FSpCatMove, FSpRename, FSRenameUnicode, HRename, PBCatMoveAsync, PBCatMoveSync, PBHRenameAsync, PBHRenameSync, PBMoveObjectAsync, PBMoveObjectSync, PBRenameUnicodeAsync, PBRenameUnicodeSync

And, all told, some 363 functions. Three hundred sixty-three.

From 900 to 1

NeXT brought its own file management system in its AppKit and Foundation, later named Cocoa. It's NSFileManager at the lower abstract level (in Foundation) and NSWorkspace in the higher 'visible' level (AppKit). NSWorkspace (AppKit) interacts with the user and refers file operations down to NSFileManager (Foundation).

NSWorkspace's performFileOperation is the system's workhorse, and should be capable of doing anything (via NSFileManager). The requested file operation is not expressed as an integer but as a character string.

NSString *NSWorkspaceCompressOperation   // @"compress"
NSString *NSWorkspaceCopyOperation       // @"copy"
NSString *NSWorkspaceDecompressOperation // @"decompress"
NSString *NSWorkspaceDecryptOperation    // @"decrypt"
NSString *NSWorkspaceDestroyOperation    // @"destroy"
NSString *NSWorkspaceDuplicateOperation  // @"duplicate"
NSString *NSWorkspaceEncryptOperation    // @"encrypt"
NSString *NSWorkspaceLinkOperation       // @"link"
NSString *NSWorkspaceMoveOperation       // @"move"
NSString *NSWorkspaceRecycleOperation    // @"recycle"

Copy, destroy, duplicate, move, and recycle work. The others were never migrated/developed, perhaps as a result of the about-face after demonstrating NeXT's File Viewer.

NSWorkspace has a number of other 'functions' as well, and much of the functionality of performFileOperation is duplicated in separate APIs.

But it's to NSFileManager everything's supposed to devolve, as NSWorkspace is the 'visible' class (in AppKit) and NSFileManager is the invisible one, resident in the abstract Foundation.

NSFileManager does much the same as NSWorkspace, but without the ability to interact with the user. There's no 'one size fits all' method for handling all (or close to all) file operations. Worse still: there seems to have been a 'semantics' war inside One Infinite, as methods (and even macros) get renamed (and deprecated) for no apparent reason.

There's the matter of coding principles at One Infinite.

Good code isn't supposed to reinvent the wheel, good code has to strive for 'one way in, one way out' for both stability and security, yet recent tests show that this is not the case at Apple anymore: changes have been made to both NSFileManager and NSWorkspace that are not congruent, functionality in the one class performing not at all as in the other. Boners like that should get you a special appointment with your career counselor. That they happen under the aegis of Apple management...

Then there's the matter of who's coding what at One Infinite.

NeXT brought engineers to Cupertino, the survivors in Cupertino had to go back to school (and yes, even learn C). Then Scott Forstall came along and decorated the iPhone project with what he felt were the best programmers in the corporation - leaving precisely what?

Things started to fall apart with 'Tiger' even as work was ramped up on iPhone: thermal grease, screen remnants, mooing noises as Apple tried to simultaneously switch to Intel.

Those Scott Forstall left behind started poking their noses into and tinkering with legacy tried-and-true NeXT code and coming up with extraordinary conclusions. (No one was very familiar with ObjC, much less AppKit and Foundation at that time.) Some of the 'bloopers' were corrected (but not for Tiger) but most were not.

No one at Apple cares about 'macOS' today: it's a 'necessary evil'. Apple will remain, by choice, a 'one-trick pony'. Cook says he doesn't understand why people even have computers anymore, Apple adverts ask cheekily 'what's a computer', and all would be fine and good if indeed the mobile device had truly replaced the computer.

But it hasn't. The latest Xcode - for macOS, not for iOS - has half a million files, many of which are for mobile configuration, yet you can't develop mobile on mobile. You - and Apple - still need their 'computer OS'.

And when the world's largest corporation still can't find the rationale for keeping a dedicated group for maintaining (fixing) their 'computer OS'...

Taking the word 'computer' out of their corporate name takes on a whole new meaning.

User security has to be the most important concern in computing, and, in that regard, Microsoft will always be an 'epic fail'. And that's really a shame when one sees the kind of hardware Microsoft (yes Microsoft) can offer today, some of it lacklustre, but lots of it nothing short of brilliant. But who can use such an environment, with its obstinate backslash replacing the forward slash for no adult reason, and its incurable propensity for malware accumulation, given that it's still a standalone system?

But second to user security (or some would say at the same level) is user data security, and Apple, despite being able to tout Rock Solid Foundation™ today, have a unique record when it comes to filesystem scandals.

Or why is it that admins, who are regularly blown away by sparkling Apple hardware, when asking about Apple's file management tool and are then shown Finder, walk away?

Your homework assignment for tonight: try to figure out why NSWorkspace is in AppKit (the framework for 'visible' code) but is never seen onscreen. Bonus points: figure out why .DS_Store still exists, or ever existed at all.

You can't beat the pyramids!
 - King Tut

See Also
Rixstep: The NeXTonian
Microsoft: SHFileOperation
Apple: NSWorkspace
Apple: NSFileManager
The Technological: Apple's File System APIs
Learning Curve: Rebel Scum: More Attacks on 'Expected Behaviour'
MacInTouch: Mac OS X 10.5 Leopard: Finder Data-Loss Bug
Slashdot: Data Loss Bug In OS X 10.5 Leopard

About | ACP | Buy | Industry Watch | Learning Curve | News | Products | Search | Substack
Copyright © Rixstep. All rights reserved.