Linux shared memory allocation on x86_64

Question

I have 64 bit REHL linux,
Linux ipms-sol1 2.6.32-71.el6.x86_64 #1 SMP x86_64 x86_64 x86_64 GNU/Linux

RAM size = ~38GB

I changed default shared memory limits as follows in /etc/sysctl.conf & loaded changed file in memory as sysctl -p

kernel.shmmni=81474836
kernel.shmmax=32212254720
kernel.shmall=7864320

Just for experimental basis I have changed shmmax size to 32GB and tried allocating 10GB in code using shmget() as given below, but it fails to get 10GB of shared memory in single shot but when I reduce my demand for shared space to 8GB it succeeds any clue as to where am I possibly going wrong?

#include <sys/types.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#include <stdio.h>

#define SHMSZ 10737418240

main()
{
    char c;
    int shmspaceid;
    key_t key;
    char *shm, *s;
    struct shmid_ds shmid;

    key = 5678;
    fprintf(stderr,"Changed code\n");

    if ((shmspaceid = shmget(key, SHMSZ, IPC_CREAT | 0666)) < 0) {
        fprintf(stderr,"ERROR memory allocation failed\n");
        return 1;
    }


    shmctl(shmspaceid, IPC_RMID, &shmid);
    return 0;
}

Regards Himanshu

Answer 1

I'm not sure that this solution is applicable to shared memory as well, but I know this phenomenon from normal malloc() calls.

It's pretty usual that you cannot allocate very large blocks of memory as you try it here. What the functions call means is "Allocate me a block of continuous memory of 10737418240 bytes". Often times, even if the total system memory could theoretically satisfy this need, the implied "a block of continuous memory" forces the limit of allocatable memory to be much lower.

The in-memory program structure, the number of programs loaded can all contribute to blocking certain areas of memory and not allow there to be 10 continuous gigabytes of memory allocatable.

I have found often times that a reboot will change that (as programs get loaded to a different position on the heap). You can try out your maximum allocatable block size with something like this:

int i=1024;
int error=0;
while(!error) {
  char *a=(char*)malloc(i);
  error=(a==null);
  if(!error)
    printf("Successfully allocated %i.\n", i);
  i*=2;
}

Hope this helps or is applicable here. I found this out while checking why I could not allocate close to maximum system memory to a JVM.

Answer 2

Shot in the dark: you don't have enough swap space. Shared memory, by default, requires reserving space in swap. You can disable this behavior using SHM_NORESERVE:

http://linux.die.net/man/2/shmget

SHM_NORESERVE (since Linux 2.6.15) This flag serves the same purpose as the mmap(2) MAP_NORESERVE flag. Do not reserve swap space for this segment. When swap space is reserved, one has the guarantee that it is possible to modify the segment. When swap space is not reserved one might get SIGSEGV upon a write if no physical memory is available. See also the discussion of the file /proc/sys/vm/overcommit_memory in proc(5).