Is using sentinel pointer values near the maximum underlying integer value safe?

Question

I was going through some code, that in addition to null pointers is using some special values like (T*)-1, generally as return values if some "create" function fails.

Where the type being pointed to is a type large enough such that ((T*)-n) + sizeof(T) will overflow, meaning that address could never actually be allocated for an instance of type T, is this OK? Could a compiler see something like if (ptr == (T*)-1), decide that is impossible and optimise it out?

Answer 1

TL;DR: (T*)-1 will likely work as intended in practice, but for safety, portability, and future-proofing, you should use null pointers as sentinels instead.

I was going through some code, that in addition to null pointers is using some special values like (T*)-1, generally as return values if some "create" function fails.

In fact, some POSIX interfaces, such as shmat(), behave similarly, returning (void *)-1 to indicate an error. For them, this is the equivalent of many other standard functions returning the int value -1. It is a value that is intended never to be a valid return value for a successful call. That must therefore work on every POSIX-conforming implementation, and I think other POSIX requirements have the combined effect of requiring the same to hold for pointer types other than void *, too.

More generally, C explicitly permits integers to be converted to pointers without restriction, but with the caveat that

Except [for null pointer constants], the result is implementation-defined, might not be correctly aligned, might not point to an entity of the referenced type, and might be a trap representation.

(C2011, 6.3.2.3/5). The main concerns with such a conversion, then, are

• that the result of (T*)-1 is a trap representation, in which case the scheme you describe produces undefined behavior.
• that the result of (T*)-1 could be a valid pointer to a T, in which case using it as a sentinel is unsafe.

To the best of my knowledge, the first of those is not an issue for any C implementation you're likely to meet. I think the second is unlikely to be an issue for you in practice, either, but if you are targeting non-POSIX systems then I am less confident about that one.

You go on to ask,

Where the type being pointed to is a type large enough such that ((T*)-n) + sizeof(T) will overflow, meaning that address could never actually be allocated for an instance of type T, is this OK? Could a compiler see something like if (ptr == (T*)-1), decide that is impossible and optimise it out?

This is an interesting question. Supposing that (T*)-1 does not produce a trap representation, this provision applies:

Two pointers compare equal if and only if both are null pointers, both are pointers to the same object (including a pointer to an object and a subobject at its beginning) or function, both are pointers to one past the last element of the same array object, or one is a pointer to one past the end of one array object and the other is a pointer to the start of a different array object that happens to immediately follow the first array object in the address space.

(C2011, 6.5.9/6)

Unfortunately, however, this is a bit of a mess.

Although the standard places constraints on the types of pointer operands of an == expression, it does not require their values to be valid pointers. Lest there be any doubt about this, it is required for internal consistency with the provisions of section 6.3.2.3, which specify results of equality comparisons involving null pointers (not limited to null pointer constants).

If at least one of the operands of x == y is an invalid pointer other than a null pointer, such as, we may presume, (T *)-1, then none of the alternatives given by 6.5.9/6 holds, so the expression should evaluate to 0. A compiler might use that to justify optimizing out the test and branch.

In practice, however, implementations often fail to conform in this regard. Instead, they take their cue from historic behavior, perhaps justifying themselves by the fleeting reference to the address space in 6.5.9/6, or maybe taking a liberal view of what an object is. For implementations that afford a flat view of the address space, this manifests as == being evaluated in terms of whether the addresses to which the pointer values correspond are the same, regardless of those addresses' relationships to any object. An implementation such as that must not optimize out the == test, because it cannot safely assume that it will always fail.

The bottom line, then, is that although the compiler is unlikely to optimize away the test, you cannot rely on the standard for assurance that it will not do so. You are on safer ground if you use null pointers as your sentinels, for notwithstanding the inconsistency I called out, in practice, null pointers of the same type do compare equal in all implementations, pursuant to 6.3.2.3/4.