Is it possible to store pointers in shared memory without using offsets?

Question

When using shared memory, each process may mmap the shared region into a different area of its respective address space. This means that when storing pointers within the shared region, you need to store them as offsets of the start of the shared region. Unfortunately, this complicates use of atomic instructions (e.g. if you're trying to write a lock free algorithm). For example, say you have a bunch of reference counted nodes in shared memory, created by a single writer. The writer periodically atomically updates a pointer 'p' to point to a valid node with positive reference count. Readers want to atomically write to 'p' because it points to the beginning of a node (a struct) whose first element is a reference count. Since p always points to a valid node, incrementing the ref count is safe, and makes it safe to dereference 'p' and access other members. However, this all only works when everything is in the same address space. If the nodes and the 'p' pointer are stored in shared memory, then clients suffer a race condition:

1. x = read p
2. y = x + offset
3. Increment refcount at y

During step 2, p may change and x may no longer point to a valid node. The only workaround I can think of is somehow forcing all processes to agree on where to map the shared memory, so that real pointers rather than offsets can be stored in the mmap'd region. Is there any way to do that? I see MAP_FIXED in the mmap documentation, but I don't know how I could pick an address that would be safe.

Edit: Using inline assembly and the 'lock' prefix on x86 maybe it's possible to build a "increment ptr X with offset Y by value Z"? Equivalent options on other architectures? Haven't written a lot of assembly, don't know if the needed instructions exist.

Answer 1

On low level the x86 atomic inctruction can do all this tree steps at once:

1. x = read p
2. y = x + offset Increment
3. refcount at y

//
      mov  edi, Destination
      mov  edx, DataOffset
      mov  ecx, NewData
 @Repeat:
      mov  eax, [edi + edx]    //load OldData
//Here you can also increment eax and save to [edi + edx]          
      lock cmpxchg dword ptr [edi + edx], ecx
      jnz  @Repeat
//