Choosing a size for a buffer

Question

I'm confused, when allocating memory with malloc you have to provide a size, however, it sometimes happen that you don't know the size you need at that point so you either alloc a huge amount of memory (which doesn't sound too wise since you're probably not going to use it all) or you use realloc when the initial buffer size becomes too small. Are this two options valid? The second one sounds good, however, the docs say that realloc ...may move the memory block to a new location which sounds like a very bad idea/hard to handle situation (e.g. if you have multiple pointers pointing to the same address the moment you call realloc they all become invalid) I'm a beginner with C, can somebody explain me how to handle the situation in which you have a buffer that may or may not grow to occupy a lot of memory.

Answer 1

it sometimes happen that you don't know the size you need at that point so you either alloc a huge amount of memory (which doesn't sound too wise since you're probably not going to use it all) or you use realloc when the initial buffer size becomes too small. Are this two options valid?

In principle yes. In practice, with modern OS kernels and default system configurations it doesn't matter how much you allocate with malloc. You see, malloc allocates address space, not memory. You can allocate as much address space as you want it will not actually consume memory; of course the OS will employ several sanity checks on the value, for example on a system with only 2GiB memory available (RAM + swap) you can't allocate 3GiB. The usual configuration is that the largest chunk of address space allocatable in a single chunk is 50% of the available system memory.

Only when you actually write something to it, the OS will reserve memory for it. So don't use calloc, because it initializes memory with, i.e. write something into it.

So if you don't know how much exactly you'll need, just malloc a big chunk of address space, for which you know, by the characteristics of the kind of data processed, that it will easily hold whatever you're expecting. Once you've got it in memory, you can use realloc to shrink the allocation. For all implemantations that matter realloc will never move data when shrinking an allocation.

One thing to be aware of is memory overcommitment: Say you got 5 processes running on a system with 4GiB RAM, each allocation 1GiB, but not immediately writing to it. The OS will give them this address space, i.e. it overcommits memory (just like airlines overcommit flight seats). Some time later the processes start writing to it. At some point the system runs out of memory and the OS has to do something about it: It will start killing processes until there's again room to "breathe".

You can switch off memory overcommitment though; strongly recommended on high reliability systems.

Answer 2

You're right - those are pretty much your two choices. You can work around the "multiple pointers" problem by abstracting a bit. If instead of passing around the pointer returned by malloc directly, you stick it into another data structure:

struct malloc_wrapper
{
    void *p;
} wrapper;

wrapper.p = malloc(INITIAL_SIZE);

And pass pointers to that data structure around instead, you can change p at any time, and anybody who shares the pointer to your new structure will be updated accordingly:

void *tmp = realloc(somepointertowrapper->p, NEW_SIZE);

/* check tmp to ensure it's not NULL. That indicates a failure
 * to realloc and the original pointer passed into realloc 
 * remains valid.
 */

somepointertowrapper->p = tmp;