Fastest way to write huge data in text file Java

Question

I have to write huge data in text[csv] file. I used BufferedWriter to write the data and it took around 40 secs to write 174 mb of data. Is this the fastest speed java can offer?

bufferedWriter = new BufferedWriter ( new FileWriter ( "fileName.csv" ) ); 

Note: These 40 secs include the time of iterating and fetching the records from resultset as well. :) . 174 mb is for 400000 rows in resultset.

Answer 1

You might try removing the BufferedWriter and just using the FileWriter directly. On a modern system there's a good chance you're just writing to the drive's cache memory anyway.

It takes me in the range of 4-5 seconds to write 175MB (4 million strings) -- this is on a dual-core 2.4GHz Dell running Windows XP with an 80GB, 7200-RPM Hitachi disk.

Can you isolate how much of the time is record retrieval and how much is file writing?

import java.io.BufferedWriter; import java.io.File; import java.io.FileWriter; import java.io.IOException; import java.io.Writer; import java.util.ArrayList; import java.util.List;  public class FileWritingPerfTest {       private static final int ITERATIONS = 5; private static final double MEG = (Math.pow(1024, 2)); private static final int RECORD_COUNT = 4000000; private static final String RECORD = "Help I am trapped in a fortune cookie factory\n"; private static final int RECSIZE = RECORD.getBytes().length;  public static void main(String[] args) throws Exception {     List<String> records = new ArrayList<String>(RECORD_COUNT);     int size = 0;     for (int i = 0; i < RECORD_COUNT; i++) {         records.add(RECORD);         size += RECSIZE;     }     System.out.println(records.size() + " 'records'");     System.out.println(size / MEG + " MB");          for (int i = 0; i < ITERATIONS; i++) {         System.out.println("\nIteration " + i);                  writeRaw(records);         writeBuffered(records, 8192);         writeBuffered(records, (int) MEG);         writeBuffered(records, 4 * (int) MEG);     } }  private static void writeRaw(List<String> records) throws IOException {     File file = File.createTempFile("foo", ".txt");     try {         FileWriter writer = new FileWriter(file);         System.out.print("Writing raw... ");         write(records, writer);     } finally {         // comment this out if you want to inspect the files afterward         file.delete();     } }  private static void writeBuffered(List<String> records, int bufSize) throws IOException {     File file = File.createTempFile("foo", ".txt");     try {         FileWriter writer = new FileWriter(file);         BufferedWriter bufferedWriter = new BufferedWriter(writer, bufSize);              System.out.print("Writing buffered (buffer size: " + bufSize + ")... ");         write(records, bufferedWriter);     } finally {         // comment this out if you want to inspect the files afterward         file.delete();     } }  private static void write(List<String> records, Writer writer) throws IOException {     long start = System.currentTimeMillis();     for (String record: records) {         writer.write(record);     }     // writer.flush(); // close() should take care of this     writer.close();      long end = System.currentTimeMillis();     System.out.println((end - start) / 1000f + " seconds"); } } 

Answer 2

try memory mapped files (takes 300 m/s to write 174MB in my m/c, core 2 duo, 2.5GB RAM) :

byte[] buffer = "Help I am trapped in a fortune cookie factory\n".getBytes(); int number_of_lines = 400000;  FileChannel rwChannel = new RandomAccessFile("textfile.txt", "rw").getChannel(); ByteBuffer wrBuf = rwChannel.map(FileChannel.MapMode.READ_WRITE, 0, buffer.length * number_of_lines); for (int i = 0; i < number_of_lines; i++) {     wrBuf.put(buffer); } rwChannel.close();