How do I implement dynamic shared memory resizing?

Question

Currently I use shm_open to get a file descriptor and then use ftruncate and mmap whenever I want to add a new buffer to the shared memory. Each buffer is used individually for its own purposes.

Now what I need to do is arbitrarily resize buffers. And also munmap buffers and reuse the free space again later.

The only solution I can come up with for the first problem is: ftuncate(file_size + old_buffer_size + extra_size), mmap, copy data accross into the new buffer and then munmap the original data. This looks very expensive to me and there is probably a better way. It also entails removing the original buffer every time.

For the second problem I don't even have a bad solution, I clearly can't move memory around everytime a buffer is removed. And if I keep track of free memory and use it whenever possible it will slow down allocation process as well as leave me with bits and pieces in between that are unused.

I hope this is not too confusing. Thanks

Answer 1

As best as I understand you need to grow (or shrink) the existing memory mapping. Under linux shared memory implemented as a file, located in /dev/shm memory filesystem. All operations in this file is the same as on the regular files (and file descriptors).

if you want to grow the existing mapping first expand the file size with ftruncate (as you wrote) then use mremap to expand the mapping to the requested size.

If you store pointers points to this region you maybe have to update these, but first try to call with 0 flag. In this case the system tries to grow the existing mapping to the requested size (if there is no collision with other preserved memory region) and pointers remains valid.

If previous option not available use MREMAP_MAYMOVE flag. In this case the system remaps to another locations, but mostly it's done effectively (no copy applied by the system.) then update the pointers.

Shrinking is the same but the reverse order.

Answer 2

I wrote an open source library for just this purpose: rszshm - resizable pointer-safe shared memory

To quote from the description page:

To accommodate resizing, rszshm first maps a large, private, noreserve map. This serves to claim a span of addresses. The shared file mapping then overlays the beginning of the span. Later calls to extend the mapping overlay more of the span. Attempts to extend beyond the end of the span return an error.

I extend a mapping by calling mmap with MAP_FIXED at the original address, and with the new size.