IanG on Tap

I’ve just been watching the recording of the excellent language panel talk from the PDC 2010 conference. The player uses Silverlight Smooth Streaming to adapt the video quality to the available bandwidth. The source material was all in high definition, and in situations where you can get a sustained 3Mb between Microsoft’s servers and your computer, you get a rather good image. But if you have less bandwidth, you’re not out of luck—you still get to watch the stream. Smooth Streaming can drop down to lower quality video to adapt to slower bitrates.

Now I’ve noticed that not everyone has the same obsession with detail in video and graphics as me, so trying to demonstrate smooth streaming in action can be frustrating. There will be a massively obvious change in quality when it switches bitrate that I would have thought anyone watching couldn’t possibly miss, but it turns out that most people find it hard to tell the difference. So I was delighted by Anders Hejlsberg’s choice of shirt for this panel. He made it far easier to see the changes in bitrate.

Here is his shirt, in glorious HD. (If this doesn’t look big enough to you to be an HD image, bear in mind that I’ve cropped that out of a wide shot that was showing all four panel members.)

Figure 1: Anders tries to decide just how much he likes his shirt

The video player frequently decided that there was no possible way a 3Mb stream could ever fit down my 8Mb internet connection. (Congestion elsewhere in the network I assume—I don’t usually have trouble getting all 8Mb when necessary, thanks to a 5:1 contention ratio.) The drop in resolution brought with it an interesting visual artefact: Anders’ shirt suddenly turned into a mass of stripes:

Figure 2: Anders is amused by his shirt suddenly going all stripy

This is aliasing in action, by the way. (It’s the same fundamental phenomenon that antialiasing tries to eliminate in fonts.) Aliasing occurs whenever you have some sort of discrete sampling process (e.g., taking a photo with a digital camera or converting font outlines into pixels) in which the sampling resolution is too low for the level of detail in the source. It so happens that with any regular sampling system (e.g. a grid of pixels), if you try to represent a repeating pattern whose scale is too close to the sampling pattern, it ends up looking like a much coarser pattern.

Here, switching to this medium resolution has created a situation where the spacing of the pattern on Anders’ shirt is too close to the spacing of the pixels. The limitations of the sampling resolution end up causing what should be fine detail to show up as a coarser pattern—the horizontal stripes you see here. In signal processing terms, we’d call that an alias—a low-frequency manifestation of a high-frequency component in the original signal.

This looks to me like a shortcoming in the encoding process—whenever you down-scale an image, you should always run it through a low-pass filter first. This removes the detail that you’re unable to represent, in order to stop that detail causing low-frequency aliases. In other words, the original high-definition image should have had the detail in the shirt blurred away before being scaled down, because that would have prevented those stripes from appearing.

Sometimes you can’t avoid it—it might be that the aliasing occurs at the point of acquisition. (E.g., if your camera is perfectly in focus, pointing at an object with too much detail—at that point, the stripes would be there in the original image acquired by the camera. The only way to avoid that is to deliberately defocus the camera slightly, which sort of acts like a low-pass filter.)

But in this case, there’s no aliasing on the original high-definition picture, so that aliasing has crept in as part of the downscaling. That’s usually entirely avoidable, so it’s a bit disappointing that it has happened, although it’s convenient if your goal is to show when things switch between different quality levels.

I was curious to see how the lower bitrates looked, so I deliberately started overloading my internet connection with other work to push Smooth Streaming down to some lower bitrates. (Incidentally, this experiment verified that I definitely had all 8Mb of bandwidth available to me, so the streaming bottleneck definitely wasn’t at my end.) Here’s the next notch down:

Figure 3: Anders is saddened by the increasingly fuzzy image

It’s noticeably more pixelated in the high contrast boundaries, e.g. the top of his head, the edges of his trousers. The aliasing is pretty much the same as in the previous image.

And here’s the lowest bitrate available:

Figure 4: Anders cheers up when he realises that the decreased resolution means many more people will be able to see the video

The aliasing hasn’t got any worse at lower resolutions, but it hasn’t gone away either. You’d typically expect it to look different for different scales, but it’s difficult to compare images here—the camera was at different levels of zoom, and Anders was leaning at different angles in each shot, so it’s hard to draw any further conclusions from these last two images. Anyway, if you’ve ever been watching a smooth streaming video source, and have noticed an ordinary looking pattern suddenly go stripy, this is why.