IanG on Tap

I've just read Chris Anderson's reply to Miguel de Icaza's comments on the Longhorn changes.

One aspect of the discussion particularly interests me: the complexity of APIs and of interfaces in general. (Actually it was all interesting. It's just that I wanted to say something about this topic...)

Finding the right model for any interface is often a hard and interesting problem. And it's not just of programming interfaces - user interfaces are also hard, even with non-computer-based systems. Donald Norman comes up with an excellent example of a confusing interface to a refrigerator in his excellent book The Design of Everyday Things. In that example the front panel completely misrepresents the underlying model employed by the fridge's 'user interface' making it incredibly hard to get it to do what you want.

Of course with computers we have a great deal more flexibility than with most mechanical things, because we are not constrained to choose a model that has a direct physical implementation as we are with, say, a thermostat. We get to build the model we want. And in theory this means that we have the potential to produce the perfect interface (whether UI or API) to any piece of software. In practice, this flexibility often results in arbitrary or overly-complex interfaces instead.

But what is the right level of complexity? Suppose there is some underlying physical reality (such as the graphics hardware). What's the right balance between simplicity and flexibility for an API wrapping this underlying hardware?

I think there can sometimes be an unfortunate tendency to assume that simpler is better. Let me pick one of my favourite examples from the mechanical world.

Manual vs. Automatic Transmission

Most cars offer two different user interfaces. You can choose either manual or automatic transmission. (I gather in the US, manual transmission is often referred to as a 'stick shift'. But we don't call it that over here. And we don't usually use the term 'gas pedal' either, in case you're wondering what the 'accelerator' is.)

Manual transmission is the lower level interface. It doesn't really bother to abstract anything - it just gives you a bunch of levers to control the oily bits. You have a pedal to control the engine throttle, a lever for selecting gear ratios, a pedal enabling you to disconnect the engine from the transmission system, and a pedal to vary the amount of friction retarding the rotation of the wheels. Oh and a wheel to change the direction the front wheels point. (Some cars (e.g. mine) introduce a level of indirection in between the accelerator pedal and the throttle to enable features like traction control and a choice of pedal position to throttle position mapping function. But once you've disabled traction control, and chosen your throttle map, the relationship between pedal position and throttle setting is direct.)

Automatic transmission instead attempts to present a simpler abstraction. In place of the gear selector is a stick that basically offers forwards, backwards and stationary. (OK, some automatic gearboxes also provide a mechanism to select a specific gear ratio, but that's effective a backdoor allowing you to select a lower level abstraction; you've changed to a different API at that point...) The accelerator pedal is recast as a 'go faster' pedal, because it not only controls the engine throttle, it also influences the system's choice of gear ratio - there is no longer a straightforward physical mapping between what you do with your right foot, and what the power train does, but the conceptual mapping is arguably more straightforward - if you want to go faster you simply put your foot to the floor; if a gear change is required to get a level of acceleration commensurate with the degree of lead-footedness, the automatic transmission will do that for you. (Brakes and steering are the same as with the manual model though.)

So automatic transmission supposedly presents a simpler interface. Two pedals rather than three - push one to speed up, the other to slow down. And you simply choose between forwards, backwards, and stop, rather than having to decide on the most appropriate gear ratio. And this 'simpler' interface is commonly presented as being better - automatic transmissions usually cost more than manual ones.

So why do I find cars with manual transmission easier to drive?

You might think that I'm just a cheapskate, and that this is a post-hoc rationalisation to justify my refusal to pay for an automatic gearbox. But as anyone familiar with my taste in cars will tell you, cost isn't the overriding factor in my choice of vehicle... I'd take manual transmission even if it cost more than automatic. The simple fact is that I find manual transmission easier to drive.

Define 'Simple'

I don't think that automatic transmission is in fact simpler. It's certainly more complex to implement - you need actuators to select gear ratios, you need some kind of control system to choose ratios, and most automatics have a torque converter in place of a clutch. But that's not what I mean - I don't care how difficult it was for the car manufacturer to make the gearbox, but I do care about how well they encapsulated the complexity.

And that's where every automatic transmission I've ever driven has fallen short. (And I've driven loads, on a very wide range of different cars in various countries.) They don't really encapsulate the complexity. They fail to deliver on the promised simple model of "push this harder to go faster". In short, they are a leaky abstraction.

Sure, to a first approximation they get it right. For simple stuff like driving in a straight line, or going down to the shops, they're just fine. But I'm not really interested in getting from A to B. A former colleague of mine once summed it up perfectly, I thought, saying that a car is a device for getting from A to A, while putting as big a smile on your face as possible. I like to drive down roads that don't actually go anywhere, but which have lots of corners. (And, ideally, run-off areas, or failing that, crash barriers...)

Consider driving on a race track. And I mean an interesting race track like Spa Franchorchamps, with corners rather than a big oval. When you enter a complex of curves in such a track, you need to plan ahead - it's not sufficient just to be thinking about the corner you're about to go round. You need to think about the corner after that - when you leave this corner, will you be in the right place on the track to head into the next corner? In fact you often need to think three corners ahead.

Changing corner mid-gear is tricky stuff and if you can possibly avoid it you should. It unsettles the car, so all other things being equal you'll be able to carry more speed round a corner if you don't need to change gear half way round. (This is even the case with modern Formula 1 cars, which have hydraulically operated manual gearboxes with automatic clutches. These can change gear in something ridiculous like 1/100th of a second. But even so, you'll often hear drivers selecting what seems like a strangely tall gear for entry to a corner in order not to have to change gear for the next corner.) This means you tend to need to pick your gear on the approach to a corner or a complex of corners. Since you know exactly what's coming up, you can make an informed choice about which gear to select. But an automatic gearbox cannot - it won't know what's coming up. All it knows is what's already happened, and what you're doing with your feet right now. If it manages to choose the best gear for the corner you're about to take while you're braking, then it'll be through sheer luck.

And it's not just race tracks where this is an issue. Consider the situation where you're stuck behind a slow moving vehicle on a country lane. You might know that in a quarter of a mile, there's a short stretch of straight road with good visibility, and that this will be the next safe opportunity to overtake. If it's all clear when you get there, you want to complete the overtaking manoeuvre as fast as possible - the sooner you're back on your own side of the road, the safer it is for everyone. This means that you want to be in the right gear at the point where you start the manoeuvre. Since you know it's coming up, and since you'll be doing all your observation for a few seconds before attempting the manoeuvre, you get plenty of warning, and with a manual gearbox, you can select a suitable gear in advance. But with an automatic box, the first warning it gets is when you put your foot down after pulling out. This gives you one of those heart-stopping moments where you've pulled out onto the wrong side of the road, and have to sit and wait for a second while the gearbox gets its act together. This makes overtaking less safe than it can be, and I don't like that.

And also, there are plenty of twisty turny public roads in the UK where being in the right gear is as important as it is on track. It can even be important for some of the more interesting road junctions in the UK come to that. (And in case you're wondering, yes, that IanG is me, but the Sam probably isn't who you're thinking of.)

So the big problem is that in attempting to provide a simple abstraction, automatic transmission reduces the drivability of a car. I've found that the only way to get the best out of a car with automatic transmission is to learn what its behaviours are so you can anticipate the bad choices that it will make, and adapt your driving style to make it do the right thing. The irony is that this means you not only need to understand the underlying reality of the characteristics of the engine and gear ratios, you also need to understand how the abstraction placed on top of this works in order to be able to work around it.

Seen in this light, the manual transmission looks a whole lot simpler by comparison. It might have more levels, pedals and selectable options, but at least you know exactly what each control will do. There is no need to try and outsmart the abstraction simply to get it to do what you want.

This is exactly the problem that I believe Joel Spolsky is referring to with his term 'leaky abstraction'.

When Are You Going To Stop Going On About Cars?

Not in this life I suspect... But I'll give it a rest for now. So how does this apply to software?

I think it's very hard to judge the true complexity of an API from its surface area; I think Miguel's analysis of Avalon is therefore rather simple-minded. If a simpler-looking API turns out to be an excessively leaky abstraction, you might well be better of with one that looks more complex. If the bigger API has a more straightforward mapping to the underlying reality, it might turn out to be simpler once you've learnt how to use it, despite having more pedals and levers.

I think the key is identifying abstractions that can be implemented convincingly. Some abstractions work so well you don't ever really notice any leaks. Virtual memory is a pretty good abstraction. The instruction set and programming model of any modern CPU is a great abstraction. So is binary - how many of you noticed when PC motherboards switched over from using 5v to around 3v as the standard way of representing '1'? (Most people weren't even aware that this happened. It occurred some time in the mid 1990s in case you're wondering.) Managed object references in the CLR are another pretty leak-resistant abstraction. VB6 on the other hand is, to my mind, the automatic transmission of the computing world - it works really well for straightforward stuff, and some people love it, but as soon as you start pushing the limits, the leakiness of the abstraction gets in your way far more than it helps. For a great many tasks it's just fine, and a lot less effort than the lower level alternatives, but the disconnect between the abstract model and the underlying reality will bite you from time to time.

Is there any recipe for a successful abstraction? I'm not really sure. I think conceptual integrity and completeness are probably important. I suspect that simplicity, while desirable, is less important - the x86 programming model is not a simple thing, but it has been successfully implemented in many very different ways.