Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How can I limit the cache used by copying so there is still memory available for other caches?

Tags:

Basic situation:

I am copying some NTFS disks in openSUSE. Each one is 2 TB. When I do this, the system runs slow.

My guesses:

I believe it is likely due to caching. Linux decides to discard useful caches (for example, KDE 4 bloat, virtual machine disks, LibreOffice binaries, Thunderbird binaries, etc.) and instead fill all available memory (24 GB total) with stuff from the copying disks, which will be read only once, then written and never used again. So then any time I use these applications (or KDE 4), the disk needs to be read again, and reading the bloat off the disk again makes things freeze/hiccup.

Due to the cache being gone and the fact that these bloated applications need lots of cache, this makes the system horribly slow.

Since it is USB, the disk and disk controller are not the bottleneck, so using ionice does not make it faster.

I believe it is the cache rather than just the motherboard going too slow, because if I stop everything copying, it still runs choppy for a while until it recaches everything.

And if I restart the copying, it takes a minute before it is choppy again. But also, I can limit it to around 40 MB/s, and it runs faster again (not because it has the right things cached, but because the motherboard busses have lots of extra bandwidth for the system disks). I can fully accept a performance loss from my motherboard's I/O capability being completely consumed (which is 100% used, meaning 0% wasted power which makes me happy), but I can't accept that this caching mechanism performs so terribly in this specific use case.

# free              total       used       free     shared    buffers     cached Mem:      24731556   24531876     199680          0    8834056   12998916 -/+ buffers/cache:    2698904   22032652 Swap:      4194300      24764    4169536 

I also tried the same thing on Ubuntu, which causes a total system hang instead. ;)

And to clarify, I am not asking how to leave memory free for the "system", but for "cache". I know that cache memory is automatically given back to the system when needed, but my problem is that it is not reserved for caching of specific things.

Is there some way to tell these copy operations to limit memory usage so some important things remain cached, and therefore any slowdowns are a result of normal disk usage and not rereading the same commonly used files? For example, is there a setting of max memory per process/user/file system allowed to be used as cache/buffers?

like image 978
Peter Avatar asked Apr 11 '12 11:04

Peter


People also ask

What is the limit of cache memory?

The maximum theoretical cache size is 2 GB. The size of cache you can specify is limited by the amount of physical memory and paging space available to the system. The shared class cache consists of memory mapped files that are created on disk and remain when the operating system is restarted.

What happens when cache memory is full?

This begs the question of what happens if the cache memory is already full. The answer is that some of the contents of the cache memory has to be “evicted” to make room for the new information that needs to be written there.

How can I tell how much cache is being used?

Right-click on the Start button and click on Task Manager. 2. On the Task Manager screen, click on the Performance tab > click on CPU in the left pane. In the right-pane, you will see L1, L2 and L3 Cache sizes listed under “Virtualization” section.


2 Answers

The nocache command is the general answer to this problem! It is also in Debian and Ubuntu 13.10 (Saucy Salamander).

Thanks, Peter, for alerting us to the --drop-cache" option in rsync. But that was rejected upstream (Bug 9560 – drop-cache option), in favor of a more general solution for this: the new "nocache" command based on the rsync work with fadvise.

You just prepend "nocache" to any command you want. It also has nice utilities for describing and modifying the cache status of files. For example, here are the effects with and without nocache:

$ ./cachestats ~/file.mp3 pages in cache: 154/1945 (7.9%)  [filesize=7776.2K, pagesize=4K] $ ./nocache cp ~/file.mp3 /tmp $ ./cachestats ~/file.mp3 pages in cache: 154/1945 (7.9%)  [filesize=7776.2K, pagesize=4K]\ $ cp ~/file.mp3 /tmp $ ./cachestats ~/file.mp3 pages in cache: 1945/1945 (100.0%)  [filesize=7776.2K, pagesize=4K] 

So hopefully that will work for other backup programs (rsnapshot, duplicity, rdiff-backup, amanda, s3sync, s3ql, tar, etc.) and other commands that you don't want trashing your cache.

like image 183
nealmcb Avatar answered Oct 01 '22 15:10

nealmcb


Kristof Provost was very close, but in my situation, I didn't want to use dd or write my own software, so the solution was to use the "--drop-cache" option in rsync.

I have used this many times since creating this question, and it seems to fix the problem completely. One exception was when I am using rsync to copy from a FreeBSD machine, which doesn't support "--drop-cache". So I wrote a wrapper to replace the /usr/local/bin/rsync command, and remove that option, and now it works copying from there too.

It still uses huge amount of memory for buffers and seems to keep almost no cache, but it works smoothly anyway.

$ free              total       used       free     shared    buffers     cached Mem:      24731544   24531576     199968          0   15349680     850624 -/+ buffers/cache:    8331272   16400272 Swap:      4194300     602648    3591652 
like image 27
Peter Avatar answered Oct 01 '22 15:10

Peter