MappedByteBuffer sliding window

Question

Are there any ways of getting a MappedByteBuffer to have a sliding window on a file. I have a very large file (20GB) but I only want to make 100MBs at a time. I have tried this just discarding the old buffer and creating a new one from the chanel, but this eats memory as the old buffers do not seem to be reused.

Any ideas?

Answer 1

You can force the old buffer to release it memory immediately with

((DirectBuffer) buffer).cleaner().clean();

Disclaimer: I have only used this with Sun/Oracle/OpenJDK Java 6 update 18 and later. It might not be available or work correctly with older versions, or other platforms. Thank you @EJP.

---

Unless you have a 32-bit OS, I would just map the entire file into memory (using multiple mappings) This will be much more efficient and simpler to manage. In thsi situation I only clean up the ByteBuffers as a part of closing the file (in unit tests)

You can use TB of virtual memory and use very little physical or even disk space. In this example, I map 8 TB of virtual memory on a machine with 24 GB of memory and a 120 GB drive.

http://vanillajava.blogspot.com/2011/12/using-memory-mapped-file-for-huge.html

In summary: on a 64-bit machine virtual memory is amazingly cheap, it not something you need to worry about.

BTW: Most 64-bit machines are actually limited to 48-bits of virtual memory. This is a limit of 256 TB rather than the 16 EB (18,000,000 TB) they can address in theory.

---

You might find this library interesting. I didn't mention it before as it may not be suitable for you, but you may find the approach and some of the techniques I use in code interesting.

https://github.com/peter-lawrey/Java-Chronicle