Similarities and differences between PipedOutputStream / PipedInputStream (or Reader/Writer) and BlockingQueue for async file writing in Java [closed]

Question

I'm trying to modify a serial program that reads from a database, and writes results to a file, this is done in a blocking way and I think we can get performance boost of there is a memory buffer and have the file being written in the "background" asynchronously

I can think of the "job interview" solution, using Threads, shared resource, syncronized blocks etc... but I'm sure there is a better way (is there a nice little "delayed write" library out there that will do it for me?)

Does any of the java.util.concurrent package offer any help? java.nio? or perhaps JMS/ActiveMQ?

What about PipedOutputStream / PipedInputStream as a basis for my buffer?

How do I implement a delayed / background / buffered / non-blocking / asynchronous file writter in Java?

Edit:

Based on the suggestions, and to avoid having this question closed, (as I think it's still relevant based on the answers comments and votes) here is an attempt to make it more focused. (I'm keeping the original question above so the answers will still remain in context, as there are some very good ones)

• What are the practical differences between PipedOutputStream / PipedInputStream (or PipedReader / PipedWriter) and BlockingQueue
• and is it preferable to use the latter for asynchronous file writing? (Or is this apples to oranges comparison and if so, I'd like to know why?)

Answer 1

You probably want to use a bounded blocking queue between the producer (the database reader) and the consumer (the file writer).

Java's ArrayBlockingQueue does the job quite nicely. The producer blocks if the buffer is full, avoiding any issues with consuming too much memory.

Doing the producing and consuming in concurrent threads is best achieved using Java's Executors framework.

Answer 2

I can think of the "job interview" solution, using Threads, shared resource, syncronized blocks etc... but I'm sure there is a better way (is there a nice little "delayed write" library out there that will do it for me?)

I've never come across a "delayed write" library. But I suppose that this really just an output stream / writer that writes to a queue or circular buffer with a private thread that reads from the queue / buffer and writes to the blocking output. This should work, but it might be difficult to avoid copying the cost of double-copying data.

Does any of the java.util.concurrent package offer any help?

Possibly.

java.nio?

Possibly.

or perhaps JMS/ActiveMQ?

I doubt it ... if your goal is to write to a local file.

What about PipedOutputStream / PipedInputStream as a basis for my buffer?

That could help. But you still need to implement the threads that read / write the streams.

---

Ignoring the mechanics of how you might implement this, I suspect that you won't get a significant speed-up by doing asynchronous I/O in this case. If you profile the application in its current form, you will probably find that the primary bottleneck is getting data from the database. Writing to the file is likely to be orders of magnitude faster. If that is the case, then overlapping database and file I/O is unlikely to give a significant speed-up.

And if file output does turn out to be a bottleneck, then a simpler way to speed it up would be to increase the output stream buffer size. This is a simple to do - just add an extra buffer-size parameter to the BufferedOutputStream constructor. You should try that before embarking on a major rewrite.

Answer 3

Java 7 has asynchronous I/O in 'NIO 2'.

If you can't use Java 7 I wouldn't bother trying to implement it at all frankly. You could do something horrible involving Futures but the benefit obtained would probably be zero if not negative.