Dividing a large file in multithreaded application?

Question

I was asked this basic question in an interview which I couldn't answer:

How can we divide a large file between multi threads to process it? If we are running a multi threaded application and input is a large file and we have to provide each thread a part of the file to make it fast. How we can achieve it in Java?

Can anyone explain me how we will do this? Any example will also be appreciated.

Answer 1

Get 3 starting positions

    File f = new File("xxx");
    long size = f.length();
    long p1 = 0;
    long p2 = f.length() / 3;
    long p3 = p2 + f.length() / 3;

Pass position and length to read to thread, move file to position and read n bytes

    FileInputStream is = new FileInputStream("xxx");
    FileChannel c= is.getChannel();
    c.position(position);
    // read ...

Answer 2

In an interview the response should be that this is a consumer-producer problem. So you should have 1 producer thread reading the file and put the lines in a syncronized collection like a Vector, and you could have n consumer threads taking the lines from the collection and process them.

If it helps to prepare another interview questions could be: Hyper-threading, deadlocks, busy waiting.

There are some books for interviews like cracking the code interview, or Java Programming Interviews Exposed.