Reading large files into R

Question

I am a newbie to R, but I am aware that it chokes on "big" files. I am trying to read a 200MB data file. I have tried it in csv format and also converting it to tab delimited txt but in both cases I use up my 4GB of RAM before the file loads.

Is it normal that R would use 4GB or memory to load a 200MB file, or could there be something wrong with the file and it is causing R to keep reading a bunch of nothingness in addition to the data?

Answer 1

From ?read.table

Less memory will be used if colClasses is specified as one of the six atomic vector classes.

...

Using nrows, even as a mild over-estimate, will help memory usage.

Use both of these arguments.

Ensure that you properly specify numeric for your numeric data. See here: Specifying colClasses in the read.csv

And do not under-estimate nrows.

If you're running 64-bit R, you might try the 32-bit version. It will use less memory to hold the same data.

See here also: Extend memory size limit in R