Do FlatBuffers have size limits?

Question

Can I store a 100GB string in a FlatBuffer? 10 TB (in 2023 you can buy servers with more RAM than that)?

Is it just limited by how much RAM/swap my server has or is there a hard limit you need set like with Protocol Buffers (which you can't set above 2GiB - 1 byte)?

Answer 1

A single FlatBuffer is currently limited to 2GB (it uses 32-bit signed offsets). If you want to store more than that, you'd have to use a sequence of buffers.

This kinda makes sense, because FlatBuffers are meant to be contiguous in memory, so it puts more of a strain on your memory system than Protobuf (where you could stream 100GB data from disk which would then end up as discontinuous data in memory).

I agree, with mmap and friends, there are definitely use cases for >2GB nowadays. There are some plans for a 64-bit extension: https://github.com/google/flatbuffers/projects/10#card-14545298