Virtual memory and what it does to you

Nothing is real and nothing to get hung about.

Get It Try It

So you're cruising along with your Mac and everything is fine and then suddenly things start slowing down. It might be two hours or two days since you last booted. You hit cmd-Q to close a few apps you're not using and suddenly you see that dreaded spinning beach ball. And the system gets stuck.

You have to be clever to get out of a mess like that. Both the dock and the menu bar are geared to the application you have in the foreground. And if that app won't budge, then you need to find the window corner of some other app and click on it.

There's really no point in starting something (like Activity Monitor) to see what's going on. You're hitting close to 100% CPU (or sometimes more according to Activity Monitor). And what you're probably hearing, aside from your own heavy breathing, is your hard drive going nuts.

Those 'sleeper' apps got pushed out of memory as other things you've run came in and hogged it all. The RAM. The VM. Now your system has to write all their crap back out to disk to make room for the apps you want to exit - just long enough for them to accept that cmd-Q and be outta there. But it can take time.

RAM & VM

RAM is random access memory. An old term. Think of 'memory chips'. They're on your motherboard (mobo) sometimes. Sometimes you can install more if you want. They're a bit expensive but they're gobs faster than hard drives.

Things have to be in RAM to run. This is the way all our machines work. Programs are loaded at an address in RAM, the system gets told where to go to start running the program, the program gets its data from the RAM it took. And so forth.

Hard drives are a lot cheaper than RAM. So if you don't have enough RAM to hold all your programs at once - along with their data - then you have to think of something. And that something is virtual memory.

It's called virtual because it's virtually like real memory. You increase the capacity of your computer without spending a fortune on more RAM chips. But everything has to be in RAM to run, so how do you go about getting virtual memory to work?

You swap. When things start getting full in RAM, when you have to load another app except you don't have any RAM for it to run in, you look around for an app that hasn't been used in a while, write it out to disk, then you bring in the new app.

That's the simplest scenario. And maybe it's not all that simple either. But that's the basic concept: write things to disk when needed, and read them back from disk into memory when they need to run. And keep the disk I/O to a minimum because hard drives are slow compared to RAM.

Unix has a file attribute bit that used to be used in this context. It's called the sticky bit. The sticky bit is still used - but for something else today. Back in the first years of Unix, it told the OS to not take an image (an app) out of RAM. Apps that used the sticky bit were invoked frequently and keeping them in RAM meant cutting down on incessant swapping or thrashing.

For it's possible to get in a situation where there's a hysterical level of disk activity but nothing goes forward. All your system does is write things out to disk, only to have to write other things out, and so forth. That's a bad situation to say the least.

Swapping securely

But virtual memory is more than extended RAM. Microsoft had that in the early years of Windows and screens still locked up. Apple users had to manually configure how much memory their apps were going to use. Simply not right.

The advent of 32-bit systems meant personal computers could for the first time run in a secure environment.

What's a secure environment? Many things.

• No userland apps can touch the hardware. The apps may not realise they're not touching the hardware but they can't touch it. In fact, for the system to work right, the apps have to believe they are touching it.

• The rules aren't the same anymore. Memory addresses aren't real. Apps can't directly address hard drive heads, cylinders, sectors. They can't see into the peripheral ports. They're cut off. Even screen real estate is usually something they can't play with unless the system lets them.

Most of the unwashed don't think about this, but maybe some have: what exactly stops the page you're reading now from running off the right end of the window and continuing to the edge of your computer screen?

CPUs must now of needs have different modes (rings in Intel parlance) to run in. You need at least a good old ordinary mode and then you need something often called kernel mode or privileged mode. Because the rule about not touching the hardware? That's a rule meant to be broken. Nothing is real.

Ordinary apps like your browser can't get at the hardware but obviously some things can. And your CPU has to make sure those who want to get backstage have valid passes, so to speak.

Why go through all that trouble to make things so unreal? One reason is virtual memory - you still want to be able to extend things. And get more into your Mac than you paid for. But the other and perhaps far more important reason is security.

Secure systems can't let intruders and rogue code steal secrets. The memory used by an application must not be available to any other application. Yes it's ultimately all in RAM at the end of the day. But that RAM is being used and reused all day long - and you can't let one app walk into memory left behind by another app and just go reading what was left behind. The system has to clear things away each time before reuse - it has to 'zero out' the memory. Secure systems do that.

But they need to do more.

Just like the invisible daycare teacher who stops the text in your browser window from escaping your browser window, you need some way to make sure applications don't overwrite each other's RAM. Small hope for that in the early Silicon Valley pirate days. Now things are different and it's big money calling the shots. You can't have one app go berserk and start overwriting (and corrupting) another app.

The fault that's not a fault

Page faults: this is how you stop it. Remember that the apps themselves fully believe (they must believe) they're really accessing RAM after all. Except they're not. The addresses they use - they look like and behave like real addresses - are actually addresses in virtual memory.

How much memory do your applications think they have access to? The classic exercise for programmers using 'bare metal' languages like C is to write a simple routine that figures out how many bits there are in a CPU register - a CPU register that's going to be used to specify memory addresses. Naturally there are 32 bits on a 32-bit machine. Just like there were 16 bits on a 16-bit machine before that. And 8 bits before that.

But even a primitive IBM PC could address more memory than its 16 bits would indicate. Processors always have ways around things. The original PC was a 16/8 machine - it used the 8088 processor, which was a bit of a comedown from the 8086 which was true 16-bit all the way.

What does '16/8' mean? Or what does '32/16' found on early Macs mean? The first number is the number of bits in a CPU register - the register length - and the second is the width of the buses. The buses are what carry things in and out of the CPU, to RAM to other destinations.

16 bits can address only 65,536 bytes of memory. But the original PC could address a full megabyte. (And Bill Gates was rumoured to say they only needed 640 KB of that.) How did they do it?

They divided up RAM addresses into two parts: a segment and an offset. Segment addresses were always on paragraph boundaries where a paragraph is an address divisible by 16. So you in effect sent out a segment address (which was multiplied by 16 or shifted left 4 bits) and then an offset address that assumed the starting point was the segment address you just sent out.

Shift 65,535 left four bits and see what you get. You can use Calculator.app to see. Put it in hex mode and fill it with 20 'F's. Then switch back to decimal and see what you got. It's 1,048,575 - one megabyte - 1. There are ways around everything.

How and why does this apply to 32-bit machines with virtual memory? 32-bit machines don't use just 32 bits to take care of memory - they most often use 48. Those extra 16 bits are for internal virtual memory bookkeeping.

The application starts running and begins accessing memory. It thinks it's got access to 4,294,967,296 bytes (4 GB) but you most often didn't have that much RAM in your Mac. The application refers to its addresses with 32 bits. Except those addresses aren't real. They're handled by the virtual memory manager.

The virtual memory manager takes the requests from your application and tries to find regions in real memory to place it all. And it uses those extra bits to keep tabs on where things go. If you're thinking this is incredibly complex, you're right - but it's actually quite simple, and a lot more useful than the original conceptual model.

The interrupt

CPUs are always doing things you don't know about. You're probably aware of the fact that no single CPU (no single core) can do two things at once. And yet your Mac seems to be doing all sorts of things at once. This is time slicing - processes such as your application are given small slices of time (quanta) to run in. They get control of the CPU and run for a while, and then the referee blows the whistle, copies out their current state somewhere, ushers the next process (app) in, and so forth.

CPUs are also keeping track of the time. They do this through things known as interrupts. An interrupt interrupts the CPU as the name implies. Interrupts come in all sorts of flavours and levels of priority. Sometimes a CPU can mask (ignore) an interrupt for a while, sometimes not. A system timer interrupt is a non-maskable interrupt - it can't be ignored. The time value has to be accurate.

And CPUs are also doing things like throwing exceptions. CPUs throw exceptions when exceptional things happen - when things are going wrong. And things are always going wrong in your Mac. And one of the coolest and grooviest things that can go wrong is the page fault.

Memory both in RAM and in your 'virtual memory' on disk is divided up into pages. A page of memory is customarily 4,096 bytes or 4 KB. That's the smallest I/O (input/output) value - the smallest amount that can be read or written.

So what happens when your app wants to access a memory address it thinks is real (but which actually connects to another address which is real) but that real address is taken by another application? Swap time? Sort of. What happens is that the CPU discovers the 'page' of memory your application is looking for isn't in real memory where it's supposed to be and so it issues a page fault, a type of exception. 'Something's wrong, that page is faulty.' Whereby the virtual memory manager kicks in, figures out what's at fault, what's supposed to be where, sees the page is read in from disk, and then tells the CPU to try again.

This happens thousands of times per minute on your Mac. If you were to check the rightmost tab on the ACP utility Lightman, you'd see.



That's the 'swap' tab. (The code that runs the library functions used for that tab was written by Avie Tevanian.)

Look at the 8th row. Hit cmd-R a few times to get more numbers in there. The 8th row is the number of page faults in your system. You should be up in at least eight digits - tens of millions of page faults. So page faults are a good thing. They're also behind the brilliant design idea that you don't really have to load a program into memory to make it run - you simply set up its page tables and then let the CPU do all the hard work, issuing page faults all over the place.

So virtual memory on a 32-bit machine gives you access to more memory than you have - but it also makes way for an effective and efficient housekeeping system so those apps of yours run safer as well. But that's only 32-bit. What happens when you move to 64-bit?

18,446,744,073,709,551,616 bytes

64-bit applications can use address ranges so big it's not funny. Bill Joy said it's not about databases anymore - it's about data structures, all kept in memory, with rarely a need to go to disk.

The address range of 64-bit applications is so huge your calculator can't print it correctly in the readout. It's too long. It's 18,446,744,073,709,551,616 bytes. And that's a big number. Not even Microsoft Word can find a way to waste all that.

But if everything is so hunky-dory, then how do things get so messed up on a Mac? Aye there's the rub. For all systems need to go to disk. The key to speedy and smooth VM is to not go to disk more than you need to. And one of the worst things is to get yourself in a situation where you have to start copying things between swap files. Your hard drive is slow; it starts stealing time slices from your apps; you see that dreaded spinning beach ball.

Disk access on that level is often optimised far beyond ordinary I/O routines. Systems use file mapping to take advantage of speed perks. The apps using file mapping don't actually ask to read from and write to disk. They can specify filenames or not as needs be, but they need to be careful to allocate memory on a page boundary. For the routines supporting this assume it's all ready to go in easily accessible page (4 KB) chunks. Something like this is used for your VM routines. Except they run a lot faster still.

David Cutler had an easy time when he came to Microsoft. He designed a file format (COFF) where everything from code and data to resources like menus and dialogs and graphic stuff is all baked into the same file. Loading an application on his OSes (creating a process) is simply a matter of adding a new entry to the process table, fixing up the VM tables, and a few hundred other things. The app doesn't need to be loaded into memory. The CPU will take care of that.

Nothing starts in RAM in David's system - it all starts on disk. The actual disk image of an application is its own virtual memory backup. It doesn't need to be copied to a system swap file - it already is a system swap file.

Look again at that last Lightman tab. Look at the second row. It's called 'CopyOnWrites'. There are parts of a process that change in memory over time. The resources - menus, dialogs, graphics - don't change. The application code doesn't change. But the application data will change. All the time. Incessantly. But if the program file is its own virtual memory backup and the data needs to be written to disk, you can't write to the actual program file, can you?

No of course not. So those segments are marked with a special 'copy on write' flag so that the first time the app tries to change something in there to 'write' to it, the VM manager moves the segment's VM backup to its own swap file - to 'copy' it there instead. The VM manager was told to 'copy on write' and that's exactly what it did.

But back to those 18,446,744,073,709,551,616 bytes of potential RAM. Is there any way your Mac will ever have that much memory? Probably not. What's painfully noticeable already with 10.6 and even with 10.5 is that swap file usage goes through the roof where 10.4 had no problem at all.

Swapwatch

Swapwatch is a new applet from Rixstep that simply watches your swap. It reckons things are getting difficult when you close in on that 1 GB threshold. Swapwatch can't cure what ails your system - nothing can save Apple - but it can tell you when it's a good idea to reboot. Unfortunately, Mac users are doing a lot of that lately.

You must never try to do anything about your swap files. They might be part of an ideal solution but they are in use by the system and shouldn't be tampered with. The system would be fine if you started removing them, even though the disk space you free won't go back to your quota immediately. But removing them solves nothing: the system will have to make new files as needed.

David Cutler's solution seems better in comparison. His talent might have been wasted at a company like Microsoft, but he knew what he was doing. He didn't start you with a measly swap file of 64 MB - he fastened down a single file equal in size to the size of your installed RAM plus a few more meg for bookkeeping. And that's mostly all you ever needed.

Applications loaded by processes being created. Pages were swapped into memory as needed, as indicated by the page faults the CPU raised. Data sections were copied to the system swap file when write operations were pending.

Applications could be marked as being 'aggressively' configured so the system knew they didn't care much if they stayed in memory between invocations. Applications could be marked in the opposite manner too. The concept of 'working space' came about but was used in an entirely different way from the rest of the industry.

A Microsoft jurist confronted Dave about two years after he'd moved from DEC into Microsoft's Building 8, telling him he'd noticed that the acronym for his new system 'WNT' was related to Dave's old created 'VMS' the same way 'IBM' was related to 'HAL'. Dave asked him: 'two years we've been here - it took you two years to realise that?'

• Getting more RAM will solve your issues temporarily - until the engineer dweebs who work with Macs with ten times the capacity of yours start writing new code and telling their bosses 'oh no it runs fast enough and it's frugal with memory'.

• It's more important to get rid of the guzzlers on your system. The Billy the Kid in this context is Adobe Flash (or almost anything from that company).

• You visit YouTube a lot? Then visit this link first. Turn on HTML5. Clips in HTML5 are of better quality, load much faster, and are no burden on your Mac.

• Don't forget to reboot when you have to - it clears out the swap and gives you a fresh start. Back to 64 MB again.

See Also
ACP: Swapwatch