How to write large binary data fast in Java? [duplicate]

Question

I am writing an STL file which consists of an 80 byte header, a 4 byte integer, and then records of 50 bytes each consisting of floats and a short integer.

Using a RandomAccessFile I can easily write the data, but it's horribly slow. This uses the same interface as DataOutputStream. If there is an easy way to buffer the Data Output Stream, I could use that, but the annoying part is the need to write out all the records, and at the end count the number of triangles output and write that integer to bytes 81-84.

The straightforward, but slow way, just focusing on the majority of the work, which is writing each facet:

public static void writeBinarySTL(Triangle t, RandomAccessFile d) throws IOException {
    d.writeFloat((float)t.normal.x);
    d.writeFloat((float)t.normal.y);
    d.writeFloat((float)t.normal.z);
    d.writeFloat((float)t.p1.x);
    d.writeFloat((float)t.p1.y);
    d.writeFloat((float)t.p1.z);
    d.writeFloat((float)t.p2.x);
    d.writeFloat((float)t.p2.y);
    d.writeFloat((float)t.p2.z);
    d.writeFloat((float)t.p3.x);
    d.writeFloat((float)t.p3.y);
    d.writeFloat((float)t.p3.z);
    d.writeShort(0);
}

Is there any elegant way to write this kind of binary data into a blockwise, fast I/O class?

It also occurs to me that the STL file format is supposed to be low byte first, and Java is probably high byte first. So perhaps all my writeFloats are in vain, and I will have to find a manual conversion so that it comes out in little-endian form?

If I have to, I would be willing to close the file, reopen at the end in a randomaccessfile, seek to byte 81 and write the count.

So this edit is in response to the two answers to the question that should work. The first is adding a BufferedWriter. The result is amazingly fast. I know this laptop is high end with an SSD, but I wasn't expecting this kind of performance, let alone from Java. Just by buffering output, 96Mb file written in 0.5 seconds, and 196Mb in 1.5 seconds.

To see whether nio would offer even higher performance, I attempted to implement the solution by @sturcotte06. The code does not attempt to write the header, I just focused on the 50 byte records for each triangle.

public static void writeBinarySTL2(Shape3d s, String filename) {
    java.nio.file.Path filePath = Paths.get(filename);

    // Open a channel in write mode on your file.
    try (WritableByteChannel channel = Files.newByteChannel(filePath, StandardOpenOption.CREATE)) {
        // Allocate a new buffer.
        ByteBuffer buf = ByteBuffer.allocate(50 * 1024);
        ArrayList<Triangle> triangles = s.triangles;
        // Write your triangle data to the buffer.
        for (int i = 0; i < triangles.size(); i += 1024) {
            for (int j = i; j < i + 1024; ++j)
                writeBinarySTL(triangles.get(j), buf);
            buf.flip(); // stop modifying buffer so it can be written to disk
            channel.write(buf);  // Write your buffer's data.
        }
        channel.close();
    } catch (IOException e) {
        e.printStackTrace();
    }
}

I tried both WRITE (which the documentation says requires an existing file) and CREATE which the documentation claims will either write to an existing file or make a new one.

Both options fail to create the file Sphere902_bin2.stl

java.nio.file.NoSuchFileException: Sphere902_bin2.stl
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79)
        at 
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
        at 
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
        at 

sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:230) at java.nio.file.Files.newByteChannel(Files.java:361) at java.nio.file.Files.newByteChannel(Files.java:407) at edu.stevens.scad.STL.writeBinarySTL2(STL.java:105)

I don't believe the code that writes the bytes is relevant, but this is what I came up with:

public static void writeBinarySTL(Triangle t, ByteBuffer buf) {
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.normal.x)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.normal.y)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.normal.y)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p1.x)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p1.y)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p1.z)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p2.x)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p2.y)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p2.z)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p3.x)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p3.y)));
    buf.putInt(Integer.reverseBytes(Float.floatToIntBits((float)t.p3.z)));
    buf.putShort((short)0);
}

Here is an MWE showing the code not working when writing beyond the size of the buffer:

package language;
import java.io.*;
import java.nio.*;
import java.nio.file.*;
import java.nio.channels.*;

public class FastWritenio {
    public static void writeUsingPrintWriter() throws IOException {
        PrintWriter pw = new PrintWriter(new FileWriter("test.txt"));
        pw.print("testing");
        pw.close();
    }
    public static void writeUsingnio(int numTrials, int bufferSize, int putsPer) throws IOException {
        String filename = "d:/test.dat";
        java.nio.file.Path filePath = Paths.get(filename);
        WritableByteChannel channel = Files.newByteChannel(filePath, StandardOpenOption.CREATE, StandardOpenOption.WRITE, StandardOpenOption.READ);
        ByteBuffer buf = ByteBuffer.allocate(bufferSize);
        for (int t = 0; t < numTrials; ++t) {
            for (int i = 0; i < putsPer; i ++) {
                buf.putInt(i);
            }
            buf.flip(); // stop modifying buffer so it can be written to disk
            channel.write(buf);  // Write your buffer's data.
            // Without this line, it crashes:  buf.flip();
            // but with it this code is very slow.
        }
        channel.close();
    }
    public static void main(String[] args) throws IOException {
        writeUsingPrintWriter();
        long t0 = System.nanoTime();
        writeUsingnio(1024*256, 8*1024, 2048);
        System.out.println((System.nanoTime()-t0)*1e-9);

    }
}

Answer 1

Use nio's ByteBuffer class:

public static void writeBinarySTL(Triangle t, ByteBuffer buf) {
    buf.putFloat((float)t.normal.x);
    // ...
}

// Create a new path to your file on the default file system.
Path filePath = Paths.get("file.txt");

// Open a channel in write mode on your file.
try (WritableByteChannel channel = Files.newByteChannel(filePath, StandardOpenOption.WRITE)) {
    // Allocate a new buffer.
    ByteBuffer buf = ByteBuffer.allocate(8192);

    // Write your triangle data to the buffer.
    writeBinarySTL(t, buf);

    // Flip from write mode to read mode.
    buf.flip();

    // Write your buffer's data.
    channel.write(buf);
}

ByteBuffer tutorial

ByteBuffer javadoc

WriteableByteChannel javadoc

Files javadoc

EDIT:

public static boolean tryWriteBinarySTL(Triangle t, ByteBuffer buf) {
    final int triangleBytes = 50; // set this.
    if (buf.remaining() < triangleBytes) {
       return false;
    }

    buf.putFloat((float)t.normal.x);
    // ...
    return true;
}

// Create a new path to your file on the default file system.
Path filePath = Paths.get("file.txt");

// Open a channel in write mode on your file.
try (WritableByteChannel channel = Files.newByteChannel(filePath, StandardOpenOption.WRITE)) {
    // Allocate a new buffer.
    ByteBuffer buf = ByteBuffer.allocate(8192);

    // Write your triangle data to the buffer.
    for (Triangle triangle : triangles) {
        while (!tryWriteBinarySTL(triangle, buf) ) {
            // Flush buffer.
            buf.flip();
            while (buf.hasRemaining()) {
                channel.write(buf);
            }

            buf.flip();
        }
    }

    // Write remaining.
    buf.flip();
    channel.write(buf);
}