How to append/write huge data file text in Java

Question

I have a database with 150k records. I want to write this to file as fast as possible. I've tried many approaches, but all seem slow. How do I make this faster?

I read these records in blocks of 40k. So first I read 40k then another 40k and so on.

After reading the records, this process returns a StringBuilder which contains 40k lines. Then we write this StringBuilder to a file.

private static void write(StringBuilder sb, Boolean append) throws Exception {
    File file = File.createTempFile("foo", ".txt");

    FileWriter writer = new FileWriter(file.getAbsoluteFile(), append);
    PrintWriter out = new PrintWriter(writer);
    try {
        out.print(sb);           
        out.flush();
        writer.flush();
    } finally {
        writer.close();
        out.close();
    }
}

I read this other example but it is equally slow: Fastest way to write huge data in text file Java

I also tried it with NIO api:

private static void write(StringBuilder sb, Boolean append)) throws Exception {
    FileChannel rwChannel = new FileOutputStream("textfile.txt", true).getChannel();
    ByteBuffer bb = ByteBuffer.wrap(sb.toString().getBytes("UTF-8"));
    rwChannel.write(bb);
    rwChannel.close();
}

Which is the best method to write/append huge data into file?

Answer 1

You don’t need a PrintWriter here. If you have whatever kind of Writer (e.g. a FileWriter) you can simply invoke append(sb) on it. And you don’t need to flush, close implies flushing.

private static void write(StringBuilder sb, Boolean append) throws Exception {
  File file = File.createTempFile("foo", ".txt");

  try(FileWriter writer = new FileWriter(file.getAbsoluteFile(), append)) {
      writer.append(sb);
  }
}

On my system I encountered a small performance improvement using a Channel rather than an OutputStream:

private static void write0a(StringBuilder sb, Boolean append) throws Exception {
  File file = File.createTempFile("foo", ".txt");

  try(Writer writer = Channels.newWriter(new FileOutputStream(
      file.getAbsoluteFile(), append).getChannel(), "UTF-8")) {
    writer.append(sb);
  }
}

However these are only slight improvements. I don’t see much possibilities here as all the code ends up calling the same routines. What could really improve your performance is keeping the Writer alive during the invocations and not flushing every record.

Answer 2

If you have a huge amount of data, it's better that you don't store it to StringBuilder and then write it to file at once.

This is the best scenario:

1) Before you start process on the data create FileInputStream

FileOutputStream fos = new FileOutputStream("/path/of/your/file");

2) Create and OutputStreamWriter from this file

OutputStreamWriter w = new OutputStreamWriter(fos, "UTF-8");

3) Create BufferedWriter (Improve file writing performance)

BufferedWriter bw = new BufferedWriter(w);

4) Pass bw to your process function and then flush/close

bw.flush();
bw.close();

The functionality of StringBuilder and BufferedWriter is almost same, So you do not need to change your code so much. The only negative point of this scenario is that, your process will involve all the time that the data are writing to file, but if you don't process the data in different thread, it is not an issue.

In this way, it doesn't matter how large data is it