Posix shared memory resize with mremap when several processes are using the segment

Question

I’m storing some data in a shared memory array used by several processes. At some point I want to grow the array.

Assume there’s already a synchronisation mechanism between the processes

Initially Process 1 will create the segment and process 2 will open it.

Process 1
shm_open() O_CREAT
ftruncate()
mmap() MAP_SHARED

Process 2
shm_open()
mmap()

At some point one process wants to grow the array and resize the shared segment.

Process 1 calls
ftruncate()
mremap() MREMAP_MAYMOVE

Shall Process 2 be notified of the resize and call mremap() to update it’s own virtual address too ?

If Process 2 has to be notified I’m thinking of opening a second shared memory segment with some metadata e.g the table’s capacity and a mutex. Each process stores the table’s capacity initially from shared memory and on each operation checks the local value against the shared memory metadata value. If the value has changed it will call mremap()

Is this a proper way to do this if mremap() has to be called on each process after resizing?

Answer 1

Shall Process 2 be notified of the resize and call mremap() to update it’s own virtual address too ?

Process 2 has mapped a specific region of the shared memory segment. Another process increasing the size of the segment does not invalidate that mapping. Nor does that change the region of the shared memory segment that is mapped in Process 2 -- even if process 2 originally mapped the whole segment, the part beyond the original end of the segment is not automatically mapped in process 2.

Therefore, as long as process 2 does not require access to additional pages of the segment, it has no need to update its mapping at all. If it wants to access that additional shared memory, however, then it does need to update its mapping. It can attempt to do that via Linux-specific mremap(), or by munmap() followed by mmap(). The latter requires that it still have an open file descriptor for the segment. Either way, it might not be possible to map a larger space starting at the same base address. For that matter, it might not be possible to map a larger space at all, regardless of whether any other process was able to do so.

About the mapping's base address

By default, mremap() will attempt to modify the mapped region without changing its base address, and munmap() + mmap() can be asked to do the same by requesting the original base address from mmap() and passing the MAP_FIXED flag. These will fail if the mapping cannot be expanded at that address. In the latter case, that will also leave the segment altogether unmapped.

You can allow for a new base address to be chosen by specifying the MREMAP_MAYMOVE among the mremap() flags, or by avoiding specifying MAP_FIXED to mmap(). Such attempts may succeed when expanding the mapping at the same address fails, but note well that a change of base address invalidates all that process's pointers into the original mapping, wherever stored.

Therefore, if you're going to provide for a change of base address, then you're probably best off not storing any pointers into the mapping at all, except for a single one to the base address. In that case, access the contents via that pointer or otherwise relative to it.

If Process 2 has to be notified I’m thinking of opening a second shared memory segment with some metadata e.g the table’s capacity and a mutex. Each process stores the table’s capacity initially from shared memory and on each operation checks the local value against the shared memory metadata value. If the value has changed it will call mremap()

Is this a proper way to do this if mremap() has to be called on each process after resizing?

Provided that you properly synchronize access to the metadata, your scheme sounds feasible. In fact, although there may be other reasons to do so, nothing presented in the question suggests that it should even be necessary to put the metadata in a separate shared memory segment. The expansion should not affect the content of the original pages of the segment or their mapping in process 2, so if the metadata are stored there then process 2 should still be able to access them after the expansion.

Note, however, that if the mutex, semaphore, or similar used to synchronize access to the segment is inside that segment, then you need to account for the fact that mremap() may move it, and munmap() / mmap() will leave you in need of a more complicated recovery scheme for the case that the munmap() succeeds but the subsequent mmap() fails.

---

In understanding all this, it may be useful to keep in mind that

• the size of the shared memory segment is a property of the segment, maintained by the kernel.

• the characteristics of each mapping of the segment, including the region mapped (a specific range of pages) and the base address to which it is mapped, are properties of a specific process, independent of all other mappings of the segment in the same or other processes.

• the characteristics of each mapping of the segment are also largely independent of the segment itself. In particular, the the offset and size of the mapped region does not change in response to changes to the size of the segment.