Java creating byte array whose size is represented by a long

Question

I'm trying to create a byte array whose size is of type long. For example, think of it as:

long x = _________; byte[] b = new byte[x];  

Apparently you can only specify an int for the size of a byte array.

Before anyone asks why I would need a byte array so large, I'll say I need to encapsulate data of message formats that I am not writing, and one of these message types has a length of an unsigned int (long in Java).

Is there a way to create this byte array?

I am thinking if there's no way around it, I can create a byte array output stream and keep feeding it bytes, but I don't know if there's any restriction on a size of a byte array...

Answer 1

(It is probably a bit late for the OP, but it might still be useful for others)

Unfortunately Java does not support arrays with more than 2³¹−1 elements. The maximum consumption is 2 GiB of space for a byte[] array, or 16 GiB of space for a long[] array.

While it is probably not applicable in this case, if the array is going to be sparse, you might be able to get away with using an associative data structure like a Map to match each used offset to the appropriate value. In addition, Trove provides an more memory-efficient implementation for storing primitive values than standard Java collections.

If the array is not sparse and you really, really do need the whole blob in memory, you will probably have to use a two-dimensional structure, e.g. with a Map matching offsets modulo 1024 to the proper 1024-byte array. This approach might be be more memory efficient even for sparse arrays, since adjacent filled cells can share the same Map entry.