Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Getting one line in a huge file with bash

Tags:

How can I get a particular line in a 3 gig text file. All the lines have:

  • the same length, and
  • are delimited by \n.

And I need to be able to get any line on demand.

How can this be done? Only one line need be returned.

like image 304
JavaRocky Avatar asked May 08 '10 12:05

JavaRocky


People also ask

How to read from a file line by line in Bash?

We make use of the read and cat commands, for loops, while loops, etc to read from the file and iterate over the file line by line with a few lines of script in BASH. We can use the read command to read the contents of a file line by line. We use the -r argument to the read command to avoid any backslash-escaped characters.

How to join multiple lines into one single line in Linux?

The tr command is also ideal for joining multiple lines without delimiters, with spacing, and with a single character delimiter. Pipe the output to a sed command to get rid of the extra commas. We have successfully learned how to join multiple lines into one single line in a file in Linux.

How do I read the contents of a file line by line?

We can use the read command to read the contents of a file line by line. We use the -r argument to the read command to avoid any backslash-escaped characters. #!usr/bin/env bash file="temp.txt" while read -r line; do echo -e "$line " done <$file

How do I read a list of months in Bash?

In Bash, you can use a while loop on the command line to read each line of text from a file and do something with it. Our text file is called “data.txt.” It holds a list of the months of the year. January February March . . October November December


1 Answers

If all the lines have the same length, the best way by far will be to use dd(1) and give it a skip parameter.

Let the block size be the length of each line (including the newline), then you can do:

$ dd if=filename bs=<line-length> skip=<line_no - 1> count=1 2>/dev/null

The idea is to seek past all the previous lines (skip=<line_no - 1>) and read a single line (count=1). Because the block size is set to the line length (bs=<line-length>), each block is effectively a single line. Redirect stderr so you don't get the annoying stats at the end.

That should be much more efficient than streaming the lines before the one you want through a program to read all the lines and then throw them away, as dd will seek to the position you want in the file and read only one line of data from the file.

like image 178
camh Avatar answered Sep 21 '22 19:09

camh