Forcing R (and Rstudio) to use the virtual memory on Windows

Question

I'm working with large datasets and quite often R produces an error telling it can't allocate a vector of that size or it doesn't have enough memory.

My computer has 16GB RAM (Windows 10) and I'm working with datasets of around 4GB but some operations need a lot of memory, for example converting the dataset from wide format to long. In some situations I can use gc() to realease some memory but many times it's not enough.

Sometimes I can break the dataset on smaller chunks but sometimes I need to work with all the table at once.

I've read that Linux users don't have this problem, but what about Windows?

I've tried setting a large pagefile on a SSD (200GB) but I've found that R doesn't use it at all.

I can see the task manager and when the memory consumption reaches 16GB R stops working. The size of the pagefile doesn't seem to make any difference.

How can I force R to use the pagefile? Do I need to compile it myself with some special flags?

PD: My experience is that deleting an object rm() and later using gc() doesn't recover all the memory. As I perform operations with large datasets my computer has less and less free memory at every step, no matter if I use gc().

PD2: I expect not to hear trivial solutions like "you need more RAM memory"

PD3: I've been testing and the problem only happens in Rstudio. If I use directly R it works well. Does anybody know how to do it in RStudio.

Answer 1

In order to get it working automatically every time you start RStudio the solution with R_MAX_MEM_SIZE is ignored, both if created as an environment variable or if created inside the .Rprofile.

Writing memory.limit(64000) is ignored too.

The proper way is adding the following line in the file .Rprofile

invisible(utils::memory.limit(64000))

or whatever number you want.

Of course you need to have a pagefile big enough. That number includes free RAM and free pagefile space.

Using the pagefile is slower but it's going to be used only when needed.

Something strange I've found is that it only let's you increase the maximum memory to use but it doesn't allow you to decrease it.