Are there any ways of getting a MappedByteBuffer to have a sliding window on a file. I have a very large file (20GB) but I only want to make 100MBs at a time. I have tried this just discarding the old buffer and creating a new one from the chanel, but this eats memory as the old buffers do not seem to be reused.
Any ideas?
You can force the old buffer to release it memory immediately with
((DirectBuffer) buffer).cleaner().clean();
Disclaimer: I have only used this with Sun/Oracle/OpenJDK Java 6 update 18 and later. It might not be available or work correctly with older versions, or other platforms. Thank you @EJP.
Unless you have a 32-bit OS, I would just map the entire file into memory (using multiple mappings) This will be much more efficient and simpler to manage. In thsi situation I only clean up the ByteBuffers as a part of closing the file (in unit tests)
You can use TB of virtual memory and use very little physical or even disk space. In this example, I map 8 TB of virtual memory on a machine with 24 GB of memory and a 120 GB drive.
http://vanillajava.blogspot.com/2011/12/using-memory-mapped-file-for-huge.html
In summary: on a 64-bit machine virtual memory is amazingly cheap, it not something you need to worry about.
BTW: Most 64-bit machines are actually limited to 48-bits of virtual memory. This is a limit of 256 TB rather than the 16 EB (18,000,000 TB) they can address in theory.
You might find this library interesting. I didn't mention it before as it may not be suitable for you, but you may find the approach and some of the techniques I use in code interesting.
https://github.com/peter-lawrey/Java-Chronicle
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With