Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Where does R store temporary files

Tags:

memory

r

I am running some basic data manipulation on a Macbook Air (4GB Memory, 120GB HD with 8GB available). My input file is about 40 MB, and I don't write anything to the disk until end of the process. However, in the middle of my process, my Mac says there's no memory to run. I checked hard drive and found there's about 500MB left. So here are my questions:

  1. How is it possible that R filled up my disk so quickly? My understanding is that R store everything in memory (unless I explicitly write something out to disk).
  2. If R does write temporary files on the disk, how can I find these files to delete them?

Thanks a lot.

Update 1: error message I got:

Force Quit Applications: Your Mac OS X startup disk has no more space available for 
application memory

Update 2: I checked tempdir() and it shows "var/folders/k_xxxxxxx/T//Rtmpdp9GCo". But I can't locate this director from my Finder

Update 3: After unlink(tempdir(),recursive=TRUE) in R and restarting my computer, I got my disk space back. I still would like to know if R write on my hard drive to avoid similar situations in the future.

Update 4: My main object is about 1GB. I use Activity Monitor to track process, and while Memory usage is about 2GB, Disk activity is extremely high: Data read: 14GB, data write, 44GB. I have no idea what R is writing.

like image 486
AdamNYC Avatar asked Nov 26 '12 15:11

AdamNYC


2 Answers

R writes to a temporary per-session directory which it also cleans up at exit.

It follows convention and respects TMP and related environment variables.

like image 69
Dirk Eddelbuettel Avatar answered Sep 29 '22 10:09

Dirk Eddelbuettel


What makes you think that disk space has anything to do with this? R needs all objects held in memory, not off disk (by default; there are add-on packages that allow a subset of operations on on-disk stored files too big to fit into RAM).

One of the steps in the "process" is causing R to request a chunk of RAM from the OS to enable it to continue. The OS could not comply and thus R terminated the "process" that you were running with the error message you failed to give us. [Hint, it would help if you showed the actual error not your paraphrasing thereof. Some inkling of the code you were running would also help. 40MB on-disk sounds like a reasonably large file; how many rows/columns etc.? How big is the object within R; object.size()?

like image 35
Gavin Simpson Avatar answered Sep 29 '22 08:09

Gavin Simpson