Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Unix, split file into chunks of max N bytes, keeping complete lines

Tags:

bash

unix

split

I'd like to split a file into chunks of maximum N bytes, while keeping complete lines.

Something like the following breaks up first and last lines of each chunk on exact byte boundries.

split -b 100m -d data.tsv data.tsv.
like image 635
user213397 Avatar asked Sep 06 '13 16:09

user213397


People also ask

How do you split a large file into multiple smaller pieces in Unix?

If you use the -l (a lowercase L) option, replace linenumber with the number of lines you'd like in each of the smaller files (the default is 1,000). If you use the -b option, replace bytes with the number of bytes you'd like in each of the smaller files.

How do you split a file into equal parts in Unix?

To split a file equally into two files, we use the '-n' option. By specifying '-n 2' the file is split equally into two files.

How do you split a large file in Unix?

For files over the 2GB size limit, the standard Unix/Linux command "split" can be used to break up the file. For the large size product example, "split -b 1024m BIG-File-6_plus-GB-in-total-size. tgz BIG-File-6_plus-GB-in-total-size". The "-b 1024m" splits the original file into 1GB chunks.

How do you break a file into parts?

Open the Zip file. Open the Tools tab. Click the Split Size dropdown button and select the appropriate size for each of the parts of the split Zip file. If you choose Custom Size in the Split Size dropdown list, another small window will open and allow you to enter in a custom size specified in megabytes.


1 Answers

Sounds like a job for split -C:

split -C 100m -d data.tsv data.tsv.
like image 150
svante Avatar answered Oct 10 '22 22:10

svante