IanG on Tap

I've been struggling with a problem with the Windows CryptoAPI over the last few days. I thought I'd write about it in the hope that the next person to hit this problem has a slightly easier time googling for the solution than I did...

I had some data encrypted on a Windows 2003 box using the RC2 symmetric encryption algorithm. This data had to be decrypted by various other boxes. The machines running Windows 2003 were all able to decrypt the data without problems, but on any Windows 2000 box, the CryptDecrypt API returned the NTE_BAD_DATA error. As it happens, in my particular case the encryption in Windows 2003 was being done using the .NET RC2CryptoServiceProvider, but the problem can also be reproduced by encrypting the data on Windows 2003 with CryptEncrypt. (As you'd hope, regardless of whether you use the .NET classes or the Win32 crypto API to encrypt data, if you use the same algorithm, key, and cleartext, you get the same ciphertext.)

A little googling revealed that I was far from alone - this appears to be a common problem. Sadly, the solution was much less in evidence than the complaint... Both on the web and on newsgroup archives, unanswered pleas for help with this issue were the norm. And in many cases, well-meaning but unhelpful advice was given. (Most often, those trying to help had merely read the documentation for NTE_BAD_DATA and declared that the problem was almost certainly to do with padding. While that's what the documentation suggests, it's not actually the problem here.)

I did eventually find the solution. Daniel Sie at Microsoft described the root cause of the problem on the microsoft.public.platformsdk.security newsgroup here, back in 2002. But seeing how many times people seem to have had this problem since without being directed to that explanation, I guess I'm not the only one who had trouble finding it.

There were two problems with my code. The first didn't take long to discover. It was the second that is a whole lot less obvious.

Both problems were an upshot of the fact that the defaults chosen by the Crypto API change from one version of Windows to the next. For example, you don't have to specify which particular crypto service provider you require when you call CryptAcquireContext at startup, but if you don't, the provider you get is OS-dependent, and may even be user-dependent. (It is possible to configure user-specific settings for default providers.) Currently, the "Microsoft Strong Provider" is the default, but the documentation suggests that it hasn't always been. (Given the history of export regulations surrounding cryptography, it would be surprising if the 'Strong' provider was always the default in all locales - here in the UK, you couldn't get versions of Windows with 128-bit encryption a few years ago.)

As it happens, the provider name wasn't the problem in my case - I explicitly set the provider, and even checked that I was getting the one I asked for by calling CryptGetProvParam, passing in PP_NAME.

But the provider isn't the only default that can change. Even when you specify the provider explicitly, the defaults within that provider change from one OS to the next.

For example, the first problem in my code was that the default key length for RC2 keys is different in Windows 2000 and Windows 2003. When you create a key with CryptGenKey, or derive one from a password with CryptDeriveKey, you can specify a key length in the upper 16 bits of the dwFlags parameter. If you pass 0, it chooses a default key length. For RC2, Windows 2003 will give you a 128 bit key, but you'll only get 40 bits on Windows 2000. Even if you pass CryptDeriveKey a 128-bit HCRYPTHASH it'll only create a 40-bit key unless you tell it otherwise. I hadn't realised this. In fact I failed to spot the documentation that points out where to set the key length - I thought, wrongly, that passing in a 128 bit hash to CryptDeriveKey would be sufficient to get a 128-bit key. On Windows 2003, that's what I was getting, but on Windows 2000, I was only getting a 40-bit key. (Fortunately, I had some pre-generated ciphertext in my unit tests, rather than generating the ciphertext on the fly in the test harness, so the problem was evident from unit test failures on Windows 2000. Hoorah for unit tests!)

The second problem was similar, but not quite so obvious. At least not to me. It hadn't occurred to me to ask the following question:

How long is a 128-bit key?

On Windows 2000, it turns out that the default answer for RC2 keys is a rather surprising 40 bits.

More specifically, the default 'effective key length' of a 128-bit key is 40 bits. I'm not a cryptography expert, so I don't fully understand what that means, but according to the documentation, just because you have a 128-bit key, the RC2 implementation won't necessarily use all 128 bits! So having creating a 128-bit key, you have to set the effective key length. (Call CryptSetKeyParam passing KP_EFFECTIVE_KEYLEN.) At least you have to do this on some versions of Windows. On Windows 2003, and Windows XP sp1, if you ask for a 128-bit key you'll get one, but on older versions, you need to do this 'No, I really do mean 128 bits' call after creating the key.