IanG on Tap

This is a public service announcement.

I've recently had the good fortune to be reading The Best of Verity Stob. For those of you who have not enjoyed her work, Stob has for many years been writing satirical articles in various computing press magazines, including the dear departed .EXE, not to mention DDJ. They're great, and I highly recommend them. Particularly if you are British. Especially if you used computers in the late 1980s and early 1990s.

That's not what the public service announcement is about though, so please bear with me.

I've particularly enjoyed being reminded by Ms. Stob of my earliest computing experiences. The very first article was written in 1988, and while I was only 14 at the time, I first started playing with computer at around the age of 6. So Stob's references to the daily minor tragedies of D-type connectors and DIP switches warm my heart, reminding me of a time when the idea of a hard disk with a 1GB capacity was inconceivable, much less a laptop computer with twice that much RAM. (Actually come to think of it, the idea of owning any kind of hard disk was at the distinctly exotic end of the spectrum.)

Interestingly, one article from around that era happened to strike a surprisingly contemporary chord. In a glossary of terms I found the following:

Virtual Memory. Describes a new RAM card that, when fitted in your PC, remains unrecognised through all possible permutations of DIPswitch settings.

Plus ca change... This kind of thing is still happening today. It might not have affected you yet, hence this Public Service Announcement.

In this day and age, memory is pretty cheap. The cost of 4GB of memory is sufficiently low now that, what with Virtual PC and VMWare being all the rage, it seems positively foolish not to load up a new PC with as much RAM as it will hold. But if you do this, there's a good chance you'll hit a problem. It's a new variant of a very old nemesis.

Those of us who used PCs in the days so exquisitely chronicled by Verity Stob will remember that 640K is a number of some significance. Indeed, even if you didn't get to experience it in person, there's every chance you'll be familiar with the figure simply because it has been immortalised in computing folklore. There is a widely held but incorrect belief that Bill Gates once said something rather short sighted on the subject. In fact he never said 640K should be enough for anyone.

Due to design decisions made by IBM (not Microsoft) the IBM PC could only use 640 kilobytes of its 1 megabyte memory map to address main memory. The rest was taken up with video memory, BIOS ROM, and various devices. So a full 37.5% of the memory map was out of bounds, imposing the infamous 640K limit on how much RAM you could fit, before having to resort to all sorts of hacks that never really worked very satisfactorily.

Then 32-bit systems came along. A few years later, 32-bit the mainstream OS market caught up. The 640K limit was relegated to a painful footnote in the history of the PC. (Yes, I know I just ignored about half a decade of the horrors of not-quite-32-bit operating systems, where the ghost of the 16-bit machine could still haunt you. But I for one ran Linux on the first PC I owned back in 1993, and then installed Windows NT in 1994 and haven't looked back, so I was able to avoid most of that carnage.)

However, history is now repeating itself. Moore's law has doomed all successful computer architectures to die a death due to running out of address bits. And while Moore's law has been on the wane for a while now, it hasn't ground to a halt yet, and we've already gone past the point where 32 bits are enough.

Hence the problem which this blog entry is very gradually lumbering towards describing.

To address 4GB of memory you need 32 bits of address bus. (Assuming individual bytes are addressable.) This gives us a problem - the same problem that IBM faced when designing the original PC. You tend to want to have more than just memory in a computer - you need things like graphics cards and hard disks to be accessible to the computer in order for it to be able to use them. So just as the original PC had to carve up the 8086's 1MB addressing range into memory (640K) and 'other' (384K), the same problem exists today if you want to fit memory and devices into a 32-bit address range: not all of the available 4GB of address space can be given over to memory.

(Yes, I know the x86 architecture supports a separate IO address space, which should in theory avoid this problem. However, for reasons I won't go into, there are good reasons to map IO devices into memory address, so a lot of devices end up not living in IO space.)

This shouldn't be a problem. Contrary to popular belief, Intel's 32-bit CPUs have more than 32 address pins. They actually have 36, giving them the ability to address a 64GB physical address space. This means that you should in theory be able to fit 4GB of RAM, and still have plenty of space for memory mapped devices. In fact you can fit a whole lot more than 4GB. Of course programs won't be able to see it all at once. But that's not necessarily a problem - you could have 10 processes each using 2GB of memory each. That's a total memory usage of 20GB, but any individual process is still using only half of the virtual address map, leaving 2GB over for the OS and any memory mapped devices.

Unfortunately it's not always so simple.

Historically, the PCI bus in most ordinary PCs has shared the same physical address space as the CPU. This means that if a device on the PCI bus looks at physical address 0x12345678, it will see exactly the same thing as the CPU would if it looked at the same physical address. It doesn't have to be this way of course - the CPU bus and the PCI bus are physically separate, and are connected by a bridging chip of some kind. This bridge could in theory perform some kind of mapping between addresses on the two buses. But for a long time, this didn't happen in PCs.

Windows NT/2000/XP/2003 has always supported mapping between PCI buses and the main system bus. This was necessary because back in the day when NT supported RISC systems, most of those did actually perform such a mapping. I used to write device drivers for DEC Alpha systems, and with those, the physical address the CPU used to access a particular bit of memory was usually different from the address used on the PCI bus. So if you write a device driver and wish to access a memory-mapped device, or get the device to perform some DMA, you are required to use system calls to find out what addresses the device will use on the PCI bus, and what physical address this corresponds to on the system bus. (Which will of course be different again from the virtual address of the memory in question.)

Presumably the reason PC architecture machines didn't do this is that this mapping layer increases costs, and makes it harder to achieve performance goals. So for years, the address map of the PCI bus and the address map of the system bus have been one and the same thing in the PC. (So the mapping APIs ended up passing you back the same address you passed in. Doubtless there are loads of device drivers out there that take advantage of this, omitting this call as an 'optimization'.) For a long time this wasn't a problem, because there was a whole 4GB of address space, so devices typically lurk up in the top 1GB of physical address space, leaving the bottom 3GB for memory.

And 3GB should be enough for anyone right?

So what actually happens if you go out and buy 4GB of memory for your PC? Well, it's just like the DOS days - there's a hole in your memory map for the IO. (Now it's only 25% of the total address space, but it's still a big hole.) So the bottom 3GB of your memory will be available, but there's an issue with that last 1GB.

If you're lucky, your PC will be able to do something clever. The usual solution is to make that last gigabyte appear further up in the memory map. So your memory map looks like this:

So your memory's all still available, it just happens not to be contiguous. The first 3GB are present where you'd expect to find them, then there's a 1GB hole, reserved for devices, and the 4th GB of memory appears in the 5th GB of address space. Since Pentiums can address 64GB of physical address space, there's no problem with this 32 bit processor addressing stuff that requires 33 bits to reach. (Unless you have a Pentium M of a certain age. Most of these only have 32 bits of physical address pins, in order to save power. However, the latest greatest ones appear to support extended address ranges. At least the Computer Properties panel on my new laptop claims to support it, and that's a Pentium M.)

(There is one tiny snag. The top 1GB may now be invisible to PCI devices, making it impossible to DMA into that memory. I say 'may' because I'm not sure if the latest PCs are able to do PCI/system bus address mapping like the old RISC systems could. If they can, that will solve the problem. So long as your device drivers are written properly. Do you feel lucky?)

So if you chose your motherboard wisely/flukily, all 4GB will be available to you. However, if you're unlucky you'll find your memory map looks like this:

Notice that only 3GB of the 4GB that you purchased made it into the memory map. So that last 1GB is unavailable to you. And don't think you can dust off that old copy of QEMM - this isn't memory which is hiding somewhere, and which you can coerce the OS into using, like in the good old days. Well...the old days. No, this memory is quite simply invisible to the CPU. If your motherboard is in this category, there is no physical way the CPU can get access to that memory. (In fact you'd probably find that the top 4 address bits of the processor aren't actually connected up to anything!)

What's particularly galling is that certain motherboards are advertised as accepting up to 4GB of memory, and yet won't let you use the last 1GB! ("Oh, you were expecting to use all 4GB. Sorry, we only guarantee that the motherboard will work when you put 4GB in.")

(NOTE: To all the people who are going to email me to tell me about the /3GB switch in Windows, please don't. It has absolutely nothing to do with this. The /3GB switch is all about how much virtual address space a program can have. The problems described in this article are all about physical address space, which is not the same thing at all. This is not a Windows problem - running Linux won't help. The fundamental problem is that the memory is inaccessible to the CPU. It's a hardware problem.)

How can you know which kind of motherboard to buy in order to avoid this? Right now it's not all that easy to tell actually, short of reading the datasheets for the chipset... If the mobo can take more than 4GB of memory, then you might be OK. Although they might just take the attitude of "If you can afford to fit 32GB of memory, you can afford for 1GB to go missing"...

In short, caveat emptor.