Initialize huge uint8_t array statically with reasonable compilation time

Question

I would like to statically initialize huge (megabytes) uint8_t array.

At the beginning I tried this:

constexpr uint8_t arr[HUGE_SIZE] = { 0, 255, ... };

Unfortunatelly, compilation time of above is very long (no optimization - around 30 seconds, optimizations on - above hour).

I found out that compilation time can be reduced to negligible (in both optimization off and on cases) if we use c style string initialization:

constexpr uint8_t arr[HUGE_SIZE + 1] = "\x00\xFF\x...";

Is this good approach in C++? Should I use some string literal to make types of both sides of above assignment equal?

Answer 1

If the array is really large, consider using a utility to directly generate an object file from the array. For example, with the GNU assembler you can do something like this:

    .section .rodata # or .data, as needed
    .globl arr
arr:
    .incbin "arr.bin" # assuming arr.bin is a file that contains the data
    .size arr,.-arr

Then assemble this file with the GNU assembler and link it to your program. To use this data elsewhere in your program, just declare it as extern "C":

extern "C" const uint8_t arr[];

Answer 2

Found that compile time for large arrays does improve a bit if the array is broken down into smaller chunks. Yet the string approach is still significantly faster. With such a scheme, the true array could union this array of arrays.

Posting the below as an example of how to test OP's problem without explicitly coding million byte source files. As this is not much of an answer but a resource for investigation, marking this community wiki.

#include <iostream>
using namespace std;

#include <cstdint>

#define METHOD 5

#if METHOD == 1
// 1 byte blocks 28 secs
#define ZZ16 65, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255,
#define ZZ256 ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16
#define ZZ4K  ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256
#define ARR constexpr uint8_t arr[]
#define COUT cout << arr << endl

#elif METHOD == 2
// 16 byte blocks 16 secs
#define ZZ16 {66, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255},
#define ZZ256 ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16
#define ZZ4K  ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256
#define ARR constexpr uint8_t arr[][16]
#define COUT cout << arr[0] << endl

#elif METHOD == 3
// 256 byte blocks 16 secs
#define ZZ16 67, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255,
#define ZZ256 {ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16},
#define ZZ4K  ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256
#define ARR constexpr uint8_t arr[][256]
#define COUT cout << arr[0] << endl

#elif METHOD == 4
// 4K byte blocks 13 secs
#define ZZ16 68, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255, 0, 255,
#define ZZ256 ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16
#define ZZ4K  {ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256},
#define ARR constexpr uint8_t arr[][4096]
#define COUT cout << arr[0] << endl

#elif METHOD == 5
// String 4 sec
#define ZZ16 "\x45\xFF\x00\xFF\x00\xFF\x00\xFF\x00\xFF\x00\xFF\x00\xFF\x00\xFF"
#define ZZ256 ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16  ZZ16
#define ZZ4K  ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256 ZZ256
#define ARR constexpr uint8_t arr[]
#define COUT cout << arr << endl
#endif

#define ZZ64K ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K  ZZ4K
#define ZZ1M  ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K ZZ64K
#define ZZ16M ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M  ZZ1M

// 3 million bytes
ARR = {
    ZZ1M ZZ1M ZZ1M
};

int main() {
  cout << "!!!Hello World!!!" << endl;
  COUT;
  cout << sizeof(arr) << endl;
  return 0;
}