C++ replacement for C99 VLAs (goal: preserve performance)

Question

I am porting some C99 code that makes heavy use of variable length arrays (VLA) to C++.

I replaced the VLAs (stack allocation) with an array class that allocates memory on the heap. The performance hit was huge, a slowdown of a factor of 3.2 (see benchmarks below). What fast VLA replacement can I use in C++? My goal is to minimize performance hit when rewriting the code for C++.

One idea that was suggested to me was to write an array class that contains a fixed-size storage within the class (i.e. can be stack-allocated) and uses it for small arrays, and automatically switches to heap allocation for larger arrays. My implementation of this is at the end of the post. It works fairly well, but I still cannot reach the performance of the original C99 code. To come close to it, I must increase this fixed-size storage (MSL below) to sizes which I am not comfortable with. I don't want to allocate too-huge arrays on the stack even for the many small arrays that don't need it because I worry that it will trigger a stack overflow. A C99 VLA is actually less prone to this because it will never use more storage than needed.

I came upon std::dynarray, but my understanding is that it was not accepted into the standard (yet?).

I know that clang and gcc support VLAs in C++, but I need it to work with MSVC too. In fact better portability is one of the main goals of rewriting as C++ (the other goal being making the program, which was originally a command line tool, into a reusable library).

---

Benchmark

MSL refers to the array size above which I switch to heap-allocation. I use different values for 1D and 2D arrays.

Original C99 code: 115 seconds.
MSL = 0 (i.e. heap allocation): 367 seconds (3.2x).
1D-MSL = 50, 2D-MSL = 1000: 187 seconds (1.63x).
1D-MSL = 200, 2D-MSL = 4000: 143 seconds (1.24x).
1D-MSL = 1000, 2D-MSL = 20000: 131 (1.14x).

Increasing MSL further improves performance more, but eventually the program will start returning wrong results (I assume due to stack overflow).

These benchmarks are with clang 3.7 on OS X, but gcc 5 shows very similar results.

---

Code

This is the current "smallvector" implementation I use. I need 1D and 2D vectors. I switch to heap-allocation above size MSL.

template<typename T, size_t MSL=50> class lad_vector {     const size_t len;     T sdata[MSL];     T *data; public:     explicit lad_vector(size_t len_) : len(len_) {         if (len <= MSL)             data = &sdata[0];         else             data = new T[len];     }      ~lad_vector() {         if (len > MSL)             delete [] data;     }      const T &operator [] (size_t i) const { return data[i]; }     T &operator [] (size_t i) { return data[i]; }      operator T * () { return data; } };   template<typename T, size_t MSL=1000> class lad_matrix {     const size_t rows, cols;     T sdata[MSL];     T *data;  public:     explicit lad_matrix(size_t rows_, size_t cols_) : rows(rows_), cols(cols_) {         if (rows*cols <= MSL)             data = &sdata[0];         else             data = new T[rows*cols];     }      ~lad_matrix() {         if (rows*cols > MSL)             delete [] data;     }      T const * operator[] (size_t i) const { return &data[cols*i]; }     T * operator[] (size_t i) { return &data[cols*i]; } }; 

Answer 1

Create a large buffer (MB+) in thread-local storage. (Actual memory on heap, management in TLS).

Allow clients to request memory from it in FILO manner (stack-like). (this mimics how it works in C VLAs; and it is efficient, as each request/return is just an integer addition/subtraction).

Get your VLA storage from it.

Wrap it pretty, so you can say stack_array<T> x(1024);, and have that stack_array deal with construction/destruction (note that ->~T() where T is int is a legal noop, and construction can similarly be a noop), or make stack_array<T> wrap a std::vector<T, TLS_stack_allocator>.

Data will be not as local as the C VLA data is because it will be effectively on a separate stack. You can use SBO (small buffer optimization), which is when locality really matters.

A SBO stack_array<T> can be implemented with an allocator and a std vector unioned with a std array, or with a unique ptr and custom destroyer, or a myriad of other ways. You can probably retrofit your solution, replacing your new/malloc/free/delete with calls to the above TLS storage.

I say go with TLS as that removes need for synchronization overhead while allowing multi-threaded use, and mirrors the fact that the stack itself is implicitly TLS.

Stack-buffer based STL allocator? is a SO Q&A with at least two "stack" allocators in the answers. They will need some adaption to automatically get their buffer from TLS.

Note that the TLS being one large buffer is in a sense an implementation detail. You could do large allocations, and when you run out of space do another large allocation. You just need to keep track each "stack page" current capacity and a list of stack pages, so when you empty one you can move onto an earlier one. That lets you be a bit more conservative in your TLS initial allocation without worrying about running OOM; the important part is that you are FILO and allocate rarely, not that the entire FILO buffer is one contiguous one.