How to split a CSV file into multiple chunks and read those chunks in parallel in Java code

Question

I have a very big CSV file (1GB+), it has 100,000 line.

I need to write a Java program to parse each line from the CSV file to create a body for a HTTP request to send out.

In other words, I need send out 100,000 HTTP requests which are corresponding to the lines in the CSV file. It will be very long if I do these in a single thread.

I'd like to create 1,000 threads to do i) read a line from the CSV file, ii) create a HTTP request whose body contains the read line's content, and iii) send the HTTP request out and receive response.

In this way, I need to split the CSV file into 1,000 chunks, and those chunks should have no overlapped lines in each other.

What's the best way to such a splitting procedure?

Answer 1

If you're looking to unzip and parse in the same operation, have a look at https://github.com/skjolber/unzip-csv.