Are large char buffers OK to use in C

Question

I need to store a 'large' file (say a little under 2mb) in memory and I want to know if it's OK to just use

char buffer[2048000];

Is there a better way to handle it? Any performance issues with using large char arrays?

Answer 1

Yes there could be issues, most C implementations will allocate that buffer on the stack if it's a variable of automatic storage duration, and the amount of "stack memory" available for use can be surprisingly small.

Your best bet is to use malloc for anything larger than a few kilobytes. Just remember to call free at an appropriate point.

Making it static is another possibility or making it global might help.

Answer 2

If you declare it in the global namespace, no problems. It's just a block of memory, so no performance issues whatsoever.

Within a function, the variable is going to be allocated on the stack, and you can easily run out of stack space. Avoid this approach. If you need it only in a local scope, allocate memory with malloc or use mmap to handle the file.

Answer 3

You can allocate a buffer in 3 ways:

1) On the stack. This happens when you define a non-static array within function scope. The stack size depends on your compiler and architecture, but the rule of thumb that one should not try and allocate big buffers there.

2) On the heap (using malloc). You can safely allocate big buffers there.

3) In the program data segment. This happens when you define a static array in any scope. It's ok to allocate big arrays there as well.

Answer 4

When you statically allocate the buffer in this way, you're asking the computer for 2048000 contiguous bytes free in memory. That isn't the same as asking for 2048000 bytes of free memory, since it must also be contiguous. You can therefore imagine that the larger the space to allocate is, the less likely you will find that amount of contiguous memory free.

The common approach is to use malloc to allocate the memory. This gives the program a chance to check the return value and act on it in the case in which NULL is returned (no memory given). This doesn't reduce the possibility of a memory problem from happening, but it also allows you, the programmer, to have more control in such a case.

In cases in which you need to allocate large amounts of memory (large enough that this could be an issue), one approach is to create a linked list of smaller char arrays. When you reach the end of one array, you follow the linked list to the next array and continue reading. This is best accompanied with a couple helper methods so the caller doesn't have to worry about these details.

This ensures that you don't need 2MB of contiguous memory but 10 contiguous arrays at 200 kb each, meaning they could potentially be in separate positions in memory.

In applications that pre-allocate memory used in the program (as in the case of games), the common approach is this, with multiple smaller allocated arrays in a linked list. The buffer is also written to initially since the operating system has been known to allocate memory lazily and not actually allocate the memory until it is accessed for the first time, which as you can imagine can create problems when the memory is actually required by the program. The big advantage in pre-allocating memory is that no new memory is usually required during the life of the program, and the entire linked list can be freed at the end without having to keep track of individual pointer allocations.