Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

MappedByteBuffer sliding window

Tags:

java

file

nio

Are there any ways of getting a MappedByteBuffer to have a sliding window on a file. I have a very large file (20GB) but I only want to make 100MBs at a time. I have tried this just discarding the old buffer and creating a new one from the chanel, but this eats memory as the old buffers do not seem to be reused.

Any ideas?

like image 774
Mike Streeton Avatar asked Mar 05 '12 16:03

Mike Streeton


1 Answers

You can force the old buffer to release it memory immediately with

((DirectBuffer) buffer).cleaner().clean();

Disclaimer: I have only used this with Sun/Oracle/OpenJDK Java 6 update 18 and later. It might not be available or work correctly with older versions, or other platforms. Thank you @EJP.


Unless you have a 32-bit OS, I would just map the entire file into memory (using multiple mappings) This will be much more efficient and simpler to manage. In thsi situation I only clean up the ByteBuffers as a part of closing the file (in unit tests)

You can use TB of virtual memory and use very little physical or even disk space. In this example, I map 8 TB of virtual memory on a machine with 24 GB of memory and a 120 GB drive.

http://vanillajava.blogspot.com/2011/12/using-memory-mapped-file-for-huge.html

In summary: on a 64-bit machine virtual memory is amazingly cheap, it not something you need to worry about.

BTW: Most 64-bit machines are actually limited to 48-bits of virtual memory. This is a limit of 256 TB rather than the 16 EB (18,000,000 TB) they can address in theory.


You might find this library interesting. I didn't mention it before as it may not be suitable for you, but you may find the approach and some of the techniques I use in code interesting.

https://github.com/peter-lawrey/Java-Chronicle

like image 58
Peter Lawrey Avatar answered Nov 05 '22 05:11

Peter Lawrey