Writing out using a Bytebuffer returns differently from simple write out

Question

I'm trying to write byte data to a file and part of optimizing it, I want to reduce the number of times I write out to the file.

Currently, I'm using:

try (RandomAccessFile out = new RandomAccessFile(file, "rw")) {

    for(int i = 0; i < totalPkts; i++) {
        out.writeInt(data.get(i));
    }
}

This works fine, but is very slow since there are many out.write calls.

So instead I'm trying to use a ByteBuffer to write out instead:

try (RandomAccessFile out = new RandomAccessFile(file, "rw")) {
    ByteBuffer buffer = ByteBuffer.allocate(totalPkts*4); // prevent BufferOverflow

    for(int i = 0; i < totalPkts; i++) {
        buffer.putInt(data.get(i));
    }
    out.write(buffer.array());
}

The files generated by the two methods have different sizes. The ByteBuffer file is almost a whole mb bigger than the normal out file. which would explain why the diff would say they are not the same.

I tried using a DataOutputStream and the results are the same. The file written using ByteBuffer is still about 1mb larger than the one not using a ByteBuffer.

Answer 1

The reason why your file when written using the ByteBuffer is bigger is because of how you initialize the ByteBuffer. When you simply write using writeInt(int), you are writing only the data that you need to write. However, when you create your ByteBuffer, you create it larger than it needs to be to prevent overflow which I understand. However, that also means that the underlying array, the one returned from the array() method, is also larger than it needs to be. Since arrays don't understand positions and limits, when you write the array, all of the data and the empty space after that is written, causing the large file size. What you need to do is after putting all of your data inside the buffer, you need to flip the buffer and then create a new array with length the same as the limit() of the buffer (once flipped), and then get(byte[]) all of the data from the buffer. Once you have that array, you need to write it.

This is what I'm talking about:

try (RandomAccessFile out = new RandomAccessFile(file, "rw")) {
    ByteBuffer buffer = ByteBuffer.allocate(totalPkts*4);

    for(int i = 0; i < totalPkts; i++) {
        buffer.putInt(data.get(i));
    }

    buffer.flip();
    byte[] data = new byte[buffer.limit()];
    buffer.get(data);
    out.write(data);
}