Java mmap fails on Android with "mmap failed: ENOMEM (Out of memory)"

Question

Memory mapping a large file on Android in Java works good. But when mapping more than ~1.5GB in total even with multiple mapping calls it fails with:

mmap failed: ENOMEM (Out of memory)

See the full discussion here. Note: It does not fail on a server Linux. The android:largeHeap="true" is enabled for the application.

The following Java code is called a few hundred times requesting ~1MB per call:

ByteBuffer buf = raFile.getChannel().map(allowWrites ? FileChannel.MapMode.READ_WRITE : FileChannel.MapMode.READ_ONLY, offset, byteCount);

to avoid requesting one large contiguous memory chunk which is often harder to be found. See the full code here. Keep in mind that doubling the 'segment size' (i.e. the size of a single map call) has no effect, which means it stops at the similar memory position. Also it is important to note that 2 apps with both slightly under the limit are executing fine (hinting to a per process limit).

Related questions are here, here, here, here, here, here, here and here.

Will using multiple files instead of one file with multiple mappings help?

I've read that this could be a per process limit for the virtual address space. Where can I find more about this? Can I change this setting with NDK e.g. how to call ulimit? Could madvise help me a bit here?

Update

See my answer here for a mmap tool usable in Java

Answer 1

Your problem is surely caused by virtual address space exhausting. Probably your problem reproduces on 32-bit Android devices, where available to user address space is physically limited to 2GB and cannot be bumped. (Although it may be 3GB (unlikely so) and it is configured during OS build process). Probably ~500 MB is used for system libraries, JVM and its heap. And ~1.5 GB is available for you.

The only way in this situation IMO - keep being mapped only file sections that are really used now and unmap unused ones as soon as possible. You can utilize some kind of sliding window where only small part of file will be mapped to memory, and when you finish - unmap that part, advance your window position and map that updated window, so on.

Also when you map whole large file - your process becomes an attractive victim for system's Out-Of-Memory killer. Because when you read such mapped file - consumption of physical memory raises and at some moment process will be killed.

Answer 2

As we cannot increase the virtual address limit on Android via an API (that does not need root access), also I've not yet seen this in the Android source code. The only possible solution I see is to implement kind of a cache, which mmaps segments on access and releases older segments if a certain number of segments is already mmapped. This means we are doing the work the OS normally does for us automatically, which is a bit ugly.

To make it working under Android one can use this answer / util-mmap. Hopefully someone can implement such a mmap cache for Android at some point, maybe even us :)