Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Fastest way to write to file?

Tags:

java

file-io

People also ask

What is the best method to write large amount of data to a file?

The best solution would be implement own Writer which directly uses write(byte[]) method of FileOutputStream which used underlying native writeBytes method . like @DavidMoles said source format of data is also very important in this scenario. If data is already available in bytes write directly to FileOutputSteam.

What is the difference between BufferedWriter and FileWriter?

FileWriter writes directly into Files and should be used only when the number of writes is less. BufferedWriter: BufferedWriter is almost similar to FileWriter but it uses internal buffer to write data into File. So if the number of write operations is more, the actual IO operations are less and performance is better.

How do you write multiple data from a text file in Java?

boolean append = true; String filename = "/path/to/file"; BufferedWriter writer = new BufferedWriter(new FileWriter(filename, append)); // OR: BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(new FileOutputStream(filename, append))); writer. write(line1); writer. newLine(); writer.


Make sure you allocate a large enough buffer:

BufferedWriter out = new BufferedWriter(new FileWriter(file), 32768);

What sort of OS are you running on? That can make a big difference too. However, taking a minute to write out a file of less-than-enormous size sounds like a system problem. On Linux or other *ix systems, you can use things like strace to see if the JVM is making lots of unnecessary system calls. (A very long time ago, Java I/O was pretty dumb and would make insane numbers of low-level write() system calls if you weren't careful, but when I say "a long time ago" I mean 1998 or so.)

edit — note that the situation of a Java program writing a simple file in a simple way, and yet being really slow, is an inherently odd one. Can you tell if the CPU is heavily loaded while the file is being written? It shouldn't be; there should be almost no CPU load from such a thing.


A simple test for you

char[] chars = new char[100*1024*1024];
Arrays.fill(chars, 'A');
String text = new String(chars);
long start = System.nanoTime();
BufferedWriter bw = new BufferedWriter(new FileWriter("/tmp/a.txt"));
bw.write(text);
bw.close();
long time = System.nanoTime() - start;
System.out.println("Wrote " + chars.length*1000L/time+" MB/s.");

Prints

Wrote 135 MB/s.

You could look into Java's NIO capabilities. It may support what you want to do.

Java NIO FileChannel versus FileOutputstream performance / usefulness


Try using memory mapped files:

FileChannel rwChannel = new RandomAccessFile("textfile.txt", "rw").getChannel();
ByteBuffer wrBuf = rwChannel.map(FileChannel.MapMode.READ_WRITE, 0, textToSave.length());

wrBuf.put(textToSave.getBytes());

rwChannel.close();