Linux :Identifying pages in memory

Question

I want to know what part of a huge file are cached in memory. I'm using some code from fincore for that, which works this way: the file is mmaped, then fincore loops over the address space and check pages with mincore, but it's very long (several minutes) because of the file size (several TB).

Is there a way to loop on used RAM pages instead? It would be much faster, but that means I should get the list of used pages from somewhere... However I can't find a convenient system call that would allow that.

Here comes the code:

#include <errno.h> 
#include <fcntl.h> 
#include <stdio.h> 
#include <stdlib.h> 
#include <string.h> 
#include <sys/stat.h> 
#include <sys/types.h> 
#include <unistd.h> 
/* } */

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/sysinfo.h>


void
fincore(char *filename) {
   int fd;
   struct stat st;

   struct sysinfo info;
   if (sysinfo(& info)) {
    perror("sysinfo");
    return;
   }

   void *pa = (char *)0;
   char *vec = (char *)0;
   size_t pageSize = getpagesize();
   register size_t pageIndex;

   fd = open(filename, 0);
   if (0 > fd) {
      perror("open");
      return;
   }

   if (0 != fstat(fd, &st)) {
      perror("fstat");
      close(fd);
      return;
   }

   pa = mmap((void *)0, st.st_size, PROT_NONE, MAP_SHARED, fd, 0);
   if (MAP_FAILED == pa) {
      perror("mmap");
      close(fd);
      return;
   }

   /* vec = calloc(1, 1+st.st_size/pageSize); */
   /* 2.2 sec for 8 TB */
   vec = calloc(1, (st.st_size+pageSize-1)/pageSize);
   if ((void *)0 == vec) {
      perror("calloc");
      close(fd);
      return;
   }

    /* 48 sec for 8 TB */
   if (0 != mincore(pa, st.st_size, vec)) {
      fprintf(stderr, "mincore(%p, %lu, %p): %s\n",
              pa, (unsigned long)st.st_size, vec, strerror(errno));
      free(vec);
      close(fd);
      return;
   }

   /* handle the results */
   /* 2m45s for 8 TB */
   for (pageIndex = 0; pageIndex <= st.st_size/pageSize; pageIndex++) {
      if (vec[pageIndex]&1) {
         printf("%zd\n", pageIndex);
      }
   }

   free(vec);
   vec = (char *)0;

   munmap(pa, st.st_size);
   close(fd);

   return;
}

int main(int argc, char *argv[]) {
    fincore(argv[1]);

    return 0;
}

Answer 1

The amount of information needed to represent a list is, for the pessimistic case when all or almost all pages are indeed in RAM, much higher than the bitmap - at least 64 vs 1 bits per entry. If there was such an API, when querying it about your 2 billion pages, you would have to be prepared to get 16 GB of data in the reply. Additionally, handling variable-length structures such as lists is more complex than handling a fixed-length array, so library functions, especially low-level system ones, tend to avoid the hassle.

I am also not quite sure about the implementation (how the OS interacts with the TLB and Co in this case), but it may well be that (even size difference aside) filling out the bitmap can be performed faster than creating a list due to the OS- and hardware-level structures the information is extracted from.

If you are not concerned about very fine granularity, you could have a look at /proc/<PID>/smaps. For each mapped region it shows some stats, including how much is loaded into memory (Rss field). If for the purpose of debugging you map some regions of a file with a separate mmap() call (in addition to the main mapping used for performing the actual task), you will probably get separate entries in smaps and thus see separate statistics for these regions. You almost certainly can't make billions of mappings without killing your system, but if the file is structured well, maybe having separate statistics for just a few dozen well-chosen regions can help you find the answers you are looking for.