Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Writing a file using multiple threads

Tags:

I am trying to write a single huge file in Java using multiple threads.

I have tried both FileWriter and bufferedWriter classes in Java.

The content being written is actually an entire table (Postgres) being read using CopyManager and written. Each line in the file is a single tuple from the table and I am writing 100s of lines at a time.

Approach to write:

The single to-be-written file is opened by multiple threads in append mode. Each thread thereafter tries writing to the file file.

Following are the issues I face:

  • Once a while, the contents of the file gets overwritten i.e: One line remains incomplete and the next line starts from there itself. My assumption here is that the buffers for writer are getting full. This forces the writer to immediately write the data onto the file. The data written may not be a complete line and before it can write the remainder, the next thread writes its content onto the file.
  • While using Filewriter, once a while I see a single black line in the file.

Any suggestions, how to avoid this data integrity issue?