Get the number of lines in a text file using R

Question

Is there a way to get the number of lines in a file without importing it?

So far this is what I am doing

myfiles <- list.files(pattern="*.dat") myfilesContent <- lapply(myfiles, read.delim, header=F, quote="\"") for (i in 1:length(myfiles)){   test[[i]] <- length(myfilesContent[[i]]$V1) } 

but is too time consuming since each file is quite big.

Answer 1

You can count the number of newline characters (\n, will also work for \r\n on Windows) in a file. This will give you a correct answer iff:

1. There is a newline char at the end of last line (BTW, read.csv gives a warning if this doesn't hold)
2. The table does not contain a newline character in the data (e.g. within quotes)

I'll suffice to read the file in parts. Below I set chunk (tmp buf) size of 65536 bytes:

f <- file("filename.csv", open="rb") nlines <- 0L while (length(chunk <- readBin(f, "raw", 65536)) > 0) {    nlines <- nlines + sum(chunk == as.raw(10L)) } print(nlines) close(f) 

Benchmarks on a ca. 512 MB ASCII text file, 12101000 text lines, Linux:

• readBin: ca. 2.4 s.

• @luis_js's wc-based solution: 0.1 s.

• read.delim: 39.6 s.

• EDIT: reading a file line by line with readLines (f <- file("/tmp/test.txt", open="r"); nlines <- 0L; while (length(l <- readLines(f, 128)) > 0) nlines <- nlines + length(l); close(f)): 32.0 s.