Count the number of lines in a file without reading entire file into memory?

Question

I'm processing huge data files (millions of lines each).

Before I start processing I'd like to get a count of the number of lines in the file, so I can then indicate how far along the processing is.

Because of the size of the files, it would not be practical to read the entire file into memory, just to count how many lines there are. Does anyone have a good suggestion on how to do this?

Answer 1

Reading the file a line at a time:

count = File.foreach(filename).inject(0) {|c, line| c+1} 

or the Perl-ish

File.foreach(filename) {} count = $. 

or

count = 0 File.open(filename) {|f| count = f.read.count("\n")} 

Will be slower than

count = %x{wc -l #{filename}}.split.first.to_i 

Answer 2

If you are in a Unix environment, you can just let wc -l do the work.

It will not load the whole file into memory; since it is optimized for streaming file and count word/line the performance is good enough rather then streaming the file yourself in Ruby.

SSCCE:

filename = 'a_file/somewhere.txt' line_count = `wc -l "#{filename}"`.strip.split(' ')[0].to_i p line_count 

Or if you want a collection of files passed on the command line:

wc_output = `wc -l "#{ARGV.join('" "')}"` line_count = wc_output.match(/^ *([0-9]+) +total$/).captures[0].to_i p line_count