What is the fastest way to write a large amount of data from memory to a file?

Question

I have a program that generates a lot of data and puts it in a queue to write but the problem is its generating data faster than I'm currently writing(causing it to max memory and start to slow down). Order does not matter as I plan to parse the file later.

I looked around a bit and found a few questions that helped me design my current process(but I still find it slow). Here's my code so far:

//...background multi-threaded process keeps building the queue..
FileWriter writer = new FileWriter("foo.txt",true);
        BufferedWriter bufferWritter = new BufferedWriter(writer);
        while(!queue_of_stuff_to_write.isEmpty()) {
            String data = solutions.poll().data;
            bufferWritter.newLine();
            bufferWritter.write(data);
        }
        bufferWritter.close();

I'm pretty new to programming so I maybe assessing this wrong(maybe a hardware issue as I'm using EC2), but is there a to very quickly dump the queue results into a file or if my approach is okay can I improve it somehow? As order does not matter, does it make more sense to write to multiple files on multiple drives? Will threading make it faster?,etc..I'm not exactly sure the best approach and any suggestions would be great. My goal is to save the results of the queue(sorry no outputting to /dev/null :-) and keep memory consumption as low as possible for my app(I'm not 100% sure but the queue fills up 15gig, so I'm assuming it'll be a 15gig+ file).

Fastest way to write huge data in text file Java (realized I should use buffered writer) Concurrent file write in Java on Windows (made me see that maybe multi-threading writes wasn't a great idea)

Answer 1

Looking at that code, one thing that springs to mind is character encoding. You're writing strings, but ultimately, it's bytes that go to the streams. A writer character-to-byte encoding under the hood, and it's doing it in the same thread that is handling writing. That may mean that there is time being spent encoding that is delaying writes, which could reduce the rate at which data is written.

A simple change would be to use a queue of byte[] instead of String, do the encoding in the threads which push onto the queue, and have the IO code use a BufferedOutputStream rather than a BufferedWriter.

This may also reduce memory consumption, if the encoded text takes up less than two bytes per character on average. For latin text and UTF-8 encoding, this will usually be true.

However, i suspect it's likely that you're simply generating data faster than your IO subsystem can handle it. You will need to make your IO subsystem faster - either by using a faster one (if you're on EC2, perhaps renting a faster instance, or writing to a different backend - SQS vs EBS vs local disk, etc), or by ganging several IO subsystems together in parallel somehow.