Is mmap atomic?

Question

Are mmap calls atomic in their effect?

That is, does a mapping change made by mmap appear atomically to other threads accessing the affected region?

As a litmus test, consider the case you do a mmap in a file of all zeros (from thread T1 which is at this point the only thread), then start a second thread T2 reading from the region. Then, again on T1 (the original thread) do a second mmap call for the same region, replacing the mapping with a new one against a file of all ones.

Is it possible for the reader thread to read a one from some page (i.e., see the second mmap in effect) and then subsequently read a zero from some page (i.e., see the first mapping in effect)?

You may assume that the reads on the reader thread are properly fenced, i.e., that the effect above does not occur solely due to CPU/coherency level memory access reordering.

Answer 1

Mmap(2) is atomic with respect to the mappings across all threads; in part, at least, because unmap(2) also is. To break it down, the scenario described looks something like:

MapRegion(from, to, obj) {
     Lock(&CurProc->map)
     while MapIntersect(&CurProc->map, from, to, &range) {
            MapUnMap(&CurProc->map, range.from, range.to)
            MapObjectRemove(&CurProc->map, range.from, range.to)
     }
     MapInsert(&CurProcc->map, from, to, obj)
     UnLock(&CurProc->map)
}

Following this, map_unmap has to ensure that while it is removing the mappings, no thread can access them. Notice the Lock(&thisproc->map).

MapUnMap(map, from, to) {
    foreach page in map.mmu[from .. to] {
         update page structure to invalidate mapping
    }
    foreach cpu in map.HasUsed {
         cause cpu to invoke tlb cache invalidation for (map, from, to)
    }
}

The first phase is to re-write the processor specific page tables to invalidate the area(s).

The second phase is to force every cpu that has ever loaded this map into its translation cache to invalidate that cache. This bit is highly architecture dependent. On an older x86, rewriting cr3 is typically enough, so the HasUsed is really CurrentlyUsing; whereas a newer amd64 might be able to cache multiple address space identifiers, so would be HasUsed. On an ARM, local tlb invalidation is broadcast to the local cluster; so HasUsed would refer to cluster ids rather than cpu ones. For more detail, search for tlb shootdown, as this is colloquially known as.

Once these two phases are complete, no thread can access this address range. Any attempt to do so will cause a fault, which will cause the faulting thread to Lock its mapping structure, which is already locked by the mapping thread, so it will wait until the mapping is complete. When the mapping is complete, all of the old mappings have been removed and replaced by new mappings, so there is no way to retrieve a previous mapping after this point.

What if another thread references the address range during the update? It will either continue with stale data or fault. In this respect stale data isn't an inconsistency, it is as if it had been referenced just before the mapping thread had entered mmap(2). The faulting case is the same as for faulting thread above.

In summary, update to the mappings is implemented using a series of transactions which ensure a consistent view of the address space. The cost of these transactions is architecture specific. The code to implement this can be quite intricate as it needs to guard against implicit operations, such as speculative fetching, as well as explicit ones.