Split large csv file and keep header in each part

Question

How to split a large csv file (~100GB) and preserve the header in each part ?

For instance

h1 h2
a  aa
b  bb

into

h1 h2
a  aa

and

h1 h2
b  bb

Answer 1

First you need to separate the header and the content :

header=$(head -1 $file)
data=$(tail -n +2 $file)

Then you want to split the data

echo $data | split [options...] -

In the options you have to specify the size of the chunks and the pattern for the name of the resulting files. The trailing - must not be removed as it specifies split to read data from stdin.

Then you can insert the header at the top of each file

sed -i "1i$header" $splitOutputFile

You should obviously do that last part in a for loop, but its exact code will depend on the prefix chosen for the split operation.