Questions on libc's pointer encryption

Question

glibc and eglibc have a PTR_MANGLE, which encrypts pointers in writable memory (more correctly, 'XOR' rather than 'encrypt').

I'm not finding much reading on the feature. man -k PTR_MANGLE returns no hits, and Google is returning some superficial chatter. One of the few definitive articles is Drepper's Pointer Encryption on Live Journal.

Is there any in-depth documentation on it? Can it be extended into the user space process, or is it limited to the runtime library? If so, what is the compiler switch or option to enable the feature? Can the feature be disabled in the runtime?

Answer 1

PTR_MANGLE is an internal feature in glibc, built on macros. It's not automated by the compiler in any way. You could duplicate the same thing in your applications, but you'd also have to do it manually; it works something like:

uintptr_t xor_key; // needs to be initialized with random "key" before use
#define PTR_MANGLE(p) (1 ? (void *)((uintptr_t)(p) ^ xor_key) : p)

This may be completely different from the glibc implementation; I haven't looked at it in a long time and just wrote this off the top of my head. The seemingly useless use of the conditional operator is to force the resulting expression to have the same type as the original pointer so it can be used directly.

Note that the operation is its own inverse, so PTR_MANGLE can be used for both "encryption" and "decryption".