malloc under linux, implicit limit

Question

Sorry if the title isn't as descriptive as it should be, the problem is hard to put in a few words. I am trying to find out how much mem i have available by malloc'ing and if that worked, writing to that segment. On certain systems (all linux on x86_64) i see segfaults when writing to the 2049th mib. The code is:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <errno.h>
#include <sys/mman.h>

int main (int argc, char **argv) {
    void *addr;
    int megasize = 2100
    /// allocate the memory via mmap same result
    //addr = mmap ((void *) 0, (size_t) megasize << 20, PROT_READ | PROT_WRITE,
    //                                                      MAP_PRIVATE | MAP_ANONYMOUS, (int) -1, (off_t) 0);
    addr = malloc(megasize << 20);
    if (addr == MAP_FAILED) {
            fprintf (stderr, "alloc of %d megabytes failed %s\n", megasize,
            strerror (errno));
            exit (1);
    };
    printf ("got %d megabytes at %p\n", megasize, addr);
    {
            int i;
            char *p = addr;
            printf("touching the %d Mb memory:\n", megasize);
            for (i = 0; i < megasize; i++) {
                    p[i << 20] = 0;
                    putchar('.');
                    if (i%64==63) // commenting out this line i see that it really is the 2049th mb
                            printf(" #%d\n", i);
                    fflush(stdout);
            };
            putchar('\n');
    };
    /// free the memory
    munmap (addr, (size_t) megasize << 20);
    return 0;
} 

It segfaults reliably on some systems, whereas on others it works fine. Reading the logs for the systems where it fails tells me it's not the oom killer. There are values for megasize that i can choose which will cause malloc to fail but those are larger. The segfault occurs reliably for any size bigger than 2gib and smaller than the limit where malloc returns -1 for those systems.

I believe there is a limit i am hitting that isn't observed by malloc and i can't figure out what it is. I tried reading out a few of the limits via getrlimit that seemed relevant like RLIMIT_AS and RLIMIT_DATA but those were way bigger.

This is the relevant part of my valgrindlog

==29126== Warning: set address range perms: large range [0x39436000, 0xbc836000) (defined)
==29126== Invalid write of size 1
==29126==    at 0x400AAD: main (in /home/max/source/scratch/memorytest)
==29126==  Address 0xffffffffb9436000 is not stack'd, malloc'd or (recently) free'd

Can anybody please give me a clue as to what the problem is?

Answer 1

You'll be getting an overflow when counting via int i, as int is 4 bytes wide here:

p[i << 20] = ...

Change

int i;

to be

size_t i;

size_t is the preferred type when addressing memory.