Sensible buffer size when downloading files in Java

Question

What is the sensible buffer size to use when transferring (irrespective of UP/DOWN) large files (3-4 gigs) with Java?

byte buf[] = new byte[1024]

or

byte buf[] = new byte[5 * 1024 * 1024]

Sometimes even if you use a large buffer and pass to a read(byte array[]) method this doesn't guarantee you that you will get a full 5 me buffer. In my tests I have observed that the average size is usually 1.5kb per read() invocation. Does this make sense performance wise? I'd be glad if someone could point me to a resource discussing the issue in greater details.

Answer 1

It sounds like you're reading from a network connection (TCP?)

1500 bytes is the default Ethernet MTU, which explains why you're typically getting 1.5KB per read. The MTU can often be increased to 9KB by configuring the network stacks to use jumbo frames.

With this in mind, there is almost certainly no point in making buf larger than 9KB. Using a smaller buffer (say over 1KB) may or may not negatively impact performance.

In any case, the only way to get a definitive answer is through benchmarking various buffer sizes.

Answer 2

In my research & testing, 8K is the optimal buffer size when reading from a socket on Linux with Java 6. If you allocate a buffer that is bigger than 8K it will simply be a waste of space. I've read that the native call Java makes uses an 8K buffer and this is why 8K is optimal but I've lose my reference. There is a bug pointing to this fact but it is not conclusive evidence: http://bugs.sun.com/view_bug.do?bug_id=6444633

That said, try experimenting on the platform you're interested in deploying on and you will find the optimal buffer size. If you're lazy, 8K is a great default.