Does MultiThreading in Java takes much time for task completion?

Question

I have to search for a string in 10 large size files (in zip format 70 MB) and have to print the lines with the search string to corresponding 10 output files.(i.e. file 1 output should be in output_file1...file2---> output_file2). The same program takes 15 mins for a single file. But if use 10 threads to read 10 files and to write in 10 different files it should complete in 15 mins but its taking 40 mins.

How can I solve this. Or multithreading will take this much time only?

Answer 1

Accessing files concurrently typically goes slower after 2-3 threads since the hard disk ends up thrashing about trying to read from all files simultaneously, similar to reading a defragmented file.

To avoid this, split the work into file readers and file parsers. The file readers bring in the data from the file (also decompressing), and the file parsers parse the data. You could use PipedInputStream/PipedOutputStream to forward data from your file readers to file parsers.

Because your files are zipped, reading involves both I/O and cpu, which can be interleaved nicely across 2-4 threads reading all files. For parsing the files, it's easiest to have just one thread reading from the PipedInputStream, so you will have one parser thread per file. Using multiple threads per file requires splitting up the stream and handling seaching at block boundaries, which complicates the process, and is not necessary here, since you probably have sufficient parallelism with 10 parser threads and 2-4 reader threads.