IanG on Tap

Nulls can complicate code. One of the lesser known C# language features, the ?? operator, can sometimes help. This operator lets you provide an alternate value to be used in the event that an expression evaluates to null. For example:

string displayName = elem.Name ?? "[Unknown]";

If elem.Name evaluates to anything other than null, then displayName will be elem.Name. But if elem.Name is null, displayName will become "[Unknown]". To get the same result in C# 1 requires the cryptic but useful ternary operator (aka conditional operator) (‘Confusing C family language newbies since 1978’). The result is longer, and requires you to write out the expression twice:

string displayName = elem.Name == null ? "[Unknown]" : elem.Name;

In my experience, most C# developers don’t seem to know about ??. In my Applied WPF course, one of the labs uses it in passing, and I invariably get people either asking me what it does, or occasionally telling me it’s a typo!

People usually like it once they get it, because it’s a clear improvement for this sort of example. So why isn’t it better known? I have a theory: I think it’s less useful than it seems. It only solves one specific example of the ‘nulls are a pain’ class of problems. Most of the time, I want something slightly more powerful than ??.

What I’d Like

While I frequently have to deal with values that might be null, I’ve found that in the non-null case I usually want to work with a member of the object, rather than the object itself. For example, consider this very simple singly-linked list node:

public class Node
{
    public Node Next;
    public string Value;
}

Suppose you have a variable of type Node which may be null, and that you want to return the Value if the variable’s non-null, and return null otherwise. You can’t use the ?? operator here – you have to revert to the good old ternary operator:

Node n = GetNode();
string value = n == null ? null : n.Value;

This is tolerable, but it’s a case that comes up often enough that I can see the value in having a special syntax to handle it. Sadly, the ?? operator is not that syntax – it only seems to cover a small minority of the cases.

A different solution suggests itself when you look at another feature added in C# 2: lifting for nullable types.

Lifting

Lifting in C# is a convenient way of working with values that might be null: rather than crashing when you try and use a null value, the nullness just ends up propagating. For example, C# supports lifting when adding numbers together:

public static int? Add(int? x, int? y)
{
    return x + y;
}

If you call this function with integers, it will add them as expected. But you can pass in null for either argument, in which case the result will be null. So we say that the + operator is lifted. (See Eric Lippert’s description of lifting in C# for more information.)

Not all operators are lifted in C#. The arithmetic ones are, so you can perform numeric calculations with nullable types. But many are not. Member access is not lifted, for example. If it were, I’d be able to replace this:

string value = n == null ? null : n.Value;

with this:

string value = n.Value;

If ‘.’ were lifted, value would end up as null if either n itself were null, or the Value member of the object referred to by n were null.

This seems like a much more succinct way to deal with the problem, but of course that’s not what ‘.’ does in practice. Morever, we cannot overload ‘.’ in C#, so on the face of it, it seems that only the C# compiler team would be able to make this work. However, we can achieve something close by getting the relevant code into expression tree form, and then modifying the tree.

Expression Tree Rewriting

C# can convert an expression into tree of objects. This means we can change how an expression works by simply changing the tree. We could use that to modify the behaviour of ‘.’ to add lifting.

(This idea is pretty similar to LISP’s macro system – code rewriting expressions or generating new ones. There’s a significant difference though: LISP macros run at compile time. As far as I can tell, we don’t have that option with expression trees in C# 3. However, even though we are obliged to wait until runtime to perform the tree rewriting, we do get to compile the result into IL. So after the initial overhead, the performance should be identical to normally compiled code.)

One slight snag is that expression trees are immutable. So you can’t just change them – you end up having to build a new tree. Fortunately, Microsoft provides an example class in the documentation that helps you do this: ExpressionVisitor. This walks every node in an expression tree, and lets you build an edited copy. And it’s smart about unchanged subtrees – it just reuses those rather than copying them. (That’s one of the advantages of immutability – it’s safe to reuse objects like that because you know they’re not going to change.)

The following class derives from the ExpressionVisitor example, and overrides VisitMemberAccess to change the way member access works:

public class NullLiftModifier : ExpressionVisitor
{
  // The method that kicks off the tree walk is
  // protected, so we need to define a public
  // method that provides access to it.
  public Expression Modify(Expression expression)
  {
    return Visit(expression);
  }


  // Change how '.' performs member access.
  protected override Expression VisitMemberAccess(
               MemberExpression originalExpression)
  {
    MemberExpression memberAccessExpression =
     (MemberExpression) base.VisitMemberAccess(originalExpression);


    Expression nullTest = Expression.Equal(
        memberAccessExpression.Expression,
        Expression.Constant(null,
          memberAccessExpression.Expression.Type));
    return Expression.Condition(
        nullTest,
        Expression.Constant(null, memberAccessExpression.Type),
        memberAccessExpression);
  }
}

The call to the base class builds the expression we’re about to rewrite (possibly recursively rewriting some of its sub expressions.) We then generate some extra expressions to test the target object reference for null, and then to return either a null constant or the original expression, depending on whether the target reference is null. Or to put it in code, it changes this:

obj.Member

to this:

obj == null ? null : obj.Member

For ease of use, we can wrap this rewriter in a helper function that takes an expression, rewrites it, and then compiles the result:

static Func<T, TResult> LiftMemberAccessToNull<T, TResult>(
    Expression<Func<T, TResult>> expr)
{
    NullLiftModifier modifier = new NullLiftModifier();
    var x = (Expression<Func<T, TResult>>) modifier.Modify(expr);
    return x.Compile();
}

The compilation here will happen at runtime – Compile turns the expression tree into IL in a dynamically generated method, and returns a delegate pointing to that method.

To test this, I wrote the following, which uses the Node class introduced earlier:

var get0 = LiftMemberAccessToNull((Node f) => f.Value);
var get1 = LiftMemberAccessToNull((Node f) => f.Next.Value);
var get2 = LiftMemberAccessToNull((Node f) => f.Next.Next.Value);
var get3 = LiftMemberAccessToNull((Node f) => f.Next.Next.Next.Value);

So we have four methods, which pick out the value of the current node, the next node, the node after that, and finally the node after that. I’m using the ordinary member access syntax, but I’m calling the code I wrote earlier to add lifting to the ‘.’ operator. To test these methods, I wrote the following (which, just for fun/added confusion, uses the ?? operator to help print out the results):

Node f1 = new Node { Value = "One" };
Node f2 = new Node { Value = "Two" };
Node f3 = new Node { Value = "Three" };
f1.Next = f2; f2.Next = f3;

Node[] foos = { f1, f2, f3, null };
foreach (Node f in foos)
{
  Console.WriteLine("Value: "                + (get0(f) ?? "null"));
  Console.WriteLine("Next.Value: "           + (get1(f) ?? "null"));
  Console.WriteLine("Next.Next.Value: "      + (get2(f) ?? "null"));
  Console.WriteLine("Next.Next.Next.Value: " + (get3(f) ?? "null"));

  Console.WriteLine();
}

This builds a list with three nodes, and iterates through this list, trying all four get methods on each of the nodes.

Normally, you’d expect this to throw a null reference exception. If you have a reference ‘f’ to the first node, the expression f.Next.Next.Next.Value is looking for the fourth node (i.e. three along from the first). There are only three nodes, so that final Next will evaluate to null. Normally you’d expect the attempt to access Value to throw an exception. But we’ve run these expressions through my lifting rewriter, so instead, we see this:

Value: One
Next.Value: Two
Next.Next.Value: Three
Next.Next.Next.Value: null

Value: Two
Next.Value: Three
Next.Next.Value: null
Next.Next.Next.Value: null

Value: Three
Next.Value: null
Next.Next.Value: null
Next.Next.Next.Value: null

Value: null
Next.Value: null
Next.Next.Value: null
Next.Next.Next.Value: null

This illustrates that the lifting is working as desired. This is consistent with the + operator: adding null to a number gives you null; likewise here, when I try to access a member via a null reference I get back null.

Limitations

One big problem with this is that it’s pretty clunky. I have to wrap up the member access in a lambda, and then pass it to my function to get it rewritten and compiled. If I want to make sure I perform that step just the first time my code runs, and yet I still want my lambdas to have access to local variables in scope, the code’s going to get even uglier. So while it might be a neat trick to illustrate the idea, I don’t see myself using this in production code.

This is a shame, because the idea of rewriting expressions is a pretty powerful one. Lifting member access is only one example of what you can do with this technique. It would be great if there were a clean syntax for this sort of idea.

(Also, beware that I’ve not tested this in any serious way – it’s only intended to illustrate an idea. It’s probably full of subtle flaws as well as the more glaring issues. I suspect it doesn’t play nicely with Nullable<T>, for example. Please don’t use it as is in your production code! That said, I offer this code under the MIT license, so feel free to do what you will with it within the terms of the license. Just consider yourself warned. You can grab the source from here: http://www.interact-sw.co.uk/downloads/LiftMemberAccess.zip)

A Monadic Footnote

When I first started to look at this, I had been considering a slightly different approach. The idea of a nullable value (whether it’s a Nullable<T> or just an ordinary reference) is pretty similar to Haskell’s Maybe type. I can use Maybe to write a Haskell equivalent to the Add function I wrote in C# above:

addMaybe :: Maybe Integer -> Maybe Integer -> Maybe Integer
addMaybe x y =
  do
    xVal <- x
    yVal <- y
    return (xVal + yVal)

This is slightly long-winded, because in Haskell, the addition operator is not suitably lifted – it doesn’t recognize Maybe types. (I could arrange for this lifting to happen in Haskell by plugging the code above into the right place and doing some other extra work, incidentally. But this entry’s long enough already.) So I had to write code to explicitly extract the values. A slightly closer C# version might look like this:

// Note: doesn't work properly!
public static int? AddNullable(int? x, int? y)
{
    int xVal = x.Value;
    int yVal = y.Value;
    return xVal + yVal;
}

However, this C# doesn’t work, because we’ve lost the lifting – it’ll throw an exception if you pass in null for either argument. But the Haskell version works fine, as we can see from this interactive GHCi session:

*Main> addMaybe (Just 18) (Just 24)
Just 42
*Main> addMaybe (Just 123) Nothing
Nothing
*Main> addMaybe Nothing (Just 345)
Nothing
*Main> addMaybe Nothing Nothing
Nothing

(Note, ‘Just’ is a constructor for a Maybe instance. So (Just 18) is roughly similar to new Nullable<int>(18) in C#. And Nothing constructs a Maybe that contains no value – it’s more or less like null.)

This works because Maybe belongs to Haskell’s Monad type class. The syntax involving ‘do’ and ‘<-’ is available on monadic types, and in the case of the Maybe monad, the <- operator provides lifted access to the value inside the Maybe – if either x or y turn out to contain Nothing then that whole ‘do’ block will evaluate to Nothing. (See this allegedly ‘gentle’ introduction to monads for more information on monads in Haskell.)

I had been thinking of going down a similar path as a result of reading Wes Dyer’s excellent explanation of monads. He points out that LINQ uses monadic constructs – the standard SelectMany query operator is essentially the monadic ‘bind’ operator by another name. He even implements a C# version of Maybe, and I could perhaps use that to rewrite my C# version roughly like this:

var r = from xVal in x.ToMaybe()
        from yVal in y.ToMaybe()
        select x + y;

Not only does this follow a fairly similar pattern to the Haskell version, it would also offer the same lifting behaviour – it would work correctly in the face of Nothing (null) values.

So it occurred to me that I could use this technique to perform the member access lifting I was looking for. Here’s a class that tries to do this by providing some extension methods that effectively define the monadic ‘bind’ operator for references:

public static class NullableExtensions
{
  public static V Select<T, V>(this T maybeT, Func<T, V> s)
  {
    if (maybeT == null) { return default(V); }
    return s(maybeT);
  }

  public static V SelectMany<T, U, V>(this T maybeT,
                          Func<T, U> k, Func<T, U, V> s)
  {
    if (maybeT == null) { return default(V); }

    U maybeBoundResult = k(maybeT);
    if (maybeBoundResult == null) { return default(V); }
    return s(maybeT, maybeBoundResult);
  }
}

The SelectMany implementation here just bails out early if it hits a null, and returns null without attempting to evaluate anything that follows. (And Select does much the same.) Note: I’ve used the phrase ‘maybe’ here to indicate potentially empty values – I’m not actually using Wes’s Maybe implementation. This particular code snippet stands alone.

Again, I would recommend NOT using this in production code – it effectively adds standard LINQ query operators to System.Object, which is probably not a sane thing to do in a real program. But with just that code (I don’t need any of the code from earlier) I can now write this sort of thing:

int? two = 2;
int? one = 1;
int? none = null;

var twoPlusTwo = from x in two
                 from y in two
                 select x + y;
Console.WriteLine(twoPlusTwo);

var onePlusNothing = from x in one
                     from y in none
                     select x + y;
Console.WriteLine(onePlusNothing);

var nothingPlusTwo = from x in none
                     from y in two
                     select x + y;
Console.WriteLine(nothingPlusTwo);

var onePlusTwoPlusOne = from x in one
                        from y in two
                        from z in one
                        select x + y + z;
Console.WriteLine(onePlusTwoPlusOne);

This prints out results for the calculations that have non-null results, and blank lines for the others. (Console.WriteLine just prints an empty line when given a null.) So lifted use of nullable types is working. But we can also use this to deal with member access through potentially null references.

The simplest example of working with member access wouldn’t work if I had only provided the SelectMany method. I added Select as well was so it would work even if there were only one from clause. That lets me write this sort of thing:

foreach (var item in doc.SelectNodes("//item").Cast<XmlNode>())
{
    var title = from attr in item.Attributes["title"]
                select attr.Value;

    Console.WriteLine(title);
}

This provides a solution of sorts to the member access problem we started with. I’ve run into this particular example on many occasions – I’ve used an XPath query against an XmlDocument and need to deal with the fact that some attributes may be missing on some of the elements that came back. (And yes, I could have written a more selective XPath query in this particular case. But when you’re dealing with lots of optional attributes, that gets tedious.) If the ‘title’ attribute is missing on a particular item, the code won’t hit an exception on the attr.Value expression, because my implementation of Select chooses not to evaluate the select clause at all if attr is null. (And the presence of SelectMany means the same will be true if I use multiple from clauses.)

So it’s a similar result to the expression-tree-based lifting of the ‘.’ operator: I get to write code that attempts to access members via references that may be null without needing an explicit null check. However, the resulting code looks pretty arcane, so again, I don’t think I’d use this in production code. In practice, the problem in this particular example is better solved in C# with an ordinary helper function.

Conclusion

I’ve used some tricks here to implement a couple of variations on my original theme: trying to avoid writing explicit tests for null references when accessing member variables, by lifting member access over null. Unfortunately, neither solution is pretty, so I consider this experiment an interesting failure. And perhaps that’s to be expected – I’m trying to subvert a fundamental feature of the language that was never designed to be modified. If C# had wanted to make it easy, either ‘.’ would have been overloadable, or the compiler would support compile-time hygienic macros.

Would I want to see either of those things in C#? I’m not sure I would – I’ve suffered too much ‘clever’ C++ that tries to redefine language behaviours. To be fair, a lot of the horror there was down to how the clunky macro system in C++ is entirely unrelated to the rest of the language. However, some of the horror was with better integrated features, such as the way you can in fact overload member access in C++.

There’s a lot to be said for a language where code does what it appears to do. It’s normal to spend more time reading code than writing it, and if you need to question fundamentals such as “is that thing that looks like field access really accessing a field?” every time you look at a line of code, that’s going to slow you down massively, and increase the chances of misunderstanding the code.

It reminds me of a troubleshooting consulting engagement I had years ago. A memory leak turned out to be the result of a couple of ‘clever’ behaviours interacting badly. I still remember the growing expression of disbelief on the developer’s face as I stepped into what looked like a single, straightforward statement, but which turned out to involve no fewer than seven implicit function calls! He had no idea it was doing all that, and he’d written it!

If I were going to have a syntax that means “get this thing from that object, unless we have a null reference, in which case return null”, I think I’d prefer a new, specialized syntax over a new behaviour for an established syntax.

So I think I’ll file these under “neat trick; DO NOT USE!”