I want an arbitrarily-large buffer in Linux/C/C++

Question

Basically I want an arbitrarily large stack. I know that's not possible, but could I set aside a few terabytes of my virtual address space for it? I'd like to be able to start at the beginning and walk up the buffer as far as I need, and Linux can bring in pages from physical memory on an as-needed basis. Is something like that possible? Would it have the same performance as just malloc-ing a buffer? Would there be a way to signal to Linux that you're done with the memory once you pop the stack?

EDIT: I'm wanting this because I want to optimize a recursive/parallel algorithm that allocates lots of memory on each call. Malloc is too slow for me and I don't want all the threads to trip on each other inside of malloc's locks. So basically it would be my own run-time stack alongside the real one (one for each thread).

Actually, as long as the run-time stack is big enough that should be good enough. Is there a way to know/ensure the size of the stack? In a 64-bit address space there's enough room for several threads to be stack-allocating gigabytes of data. Is that doable?

It looks like pthread_attr_setstacksize could work for new threads, but that doesn't help much if you can be called from any thread ...

Answer 1

Well, if you want a large stack, you could always change ulimit -s. It can be very large, at least on 64-bit platforms.

If you mmap anonymous space, it won't actually be allocated physical pages until you touch it. You'll need to either set vm_overcommit or have enough swap (unless you have that much physical RAM). Of course, mmap'ing anonymous space is exactly what malloc does when you ask for a lot of memory.

You can tell Linux you're done with the pages using madvise. If you call MADV_DONTNEED, then Linux will discard the pages and they should be zero-filled next time you try to access them.