Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

FlatBuffers: How to write giant files

Tags:

c++

flatbuffers

I'm experimenting with FlatBuffers to store OpenStreetMap data. So, this would be 50GB.

Since usually everything is kept in memory, how is it possible (if at all) to sequentially write data into a file?

I have a feeling this is not quite where FlatBuffers is good for.

like image 930
benjist Avatar asked Feb 10 '16 18:02

benjist

People also ask

How do you use Flatbuffer in C++?

To use FlatBuffers in your code, first generate the C++ classes from your schema with the --cpp option to flatc . Then you can include both FlatBuffers and the generated code to read or write FlatBuffers.

How does a Flatbuffer work?

FlatBuffers is a statically typed system, meaning the user of a buffer needs to know what kind of buffer it is. FlatBuffers can of course be wrapped inside other containers where needed, or you can use its union feature to dynamically identify multiple possible sub-objects stored.

How do you install FlatBuffers on a window?

Install FlatbuffersAt the beginning we have to get flatc — FlatBuffers compiler. It can be built from source code hosted in Google's flatbuffers repository. Let's download/clone it. Whole build process is described on FlatBuffers Building documentation.

1 Answers

There currently is no way to create a single FlatBuffer without having it all in memory at once. The only way to do it is to instead write out a chain of (length-prefixed) smaller independent FlatBuffers.

like image 72
Aardappel Avatar answered Oct 22 '22 05:10

Aardappel
Sign in to Comment

Related questions

C++ std::unique_ptr stored inside std::map use of deleted function ill formed
How to take advantage of multi-cpu in c++?
What can you do with templates with zero template parameters?
Using valgrind to measure cache misses [closed]
Making default constructor private in QT custom object
How to replace top element of heap efficiently withouth re-establishing heap invariant twice?
Ignore missing headers with clang AST parser
Overload resolution gets different result between gcc and clang
Is this trick, to make calling shared_from_this() in the constructor 'just work', dangerous?
Alternative to Fsync for windows c++
How to initialize a repeated element in protocol buffer?
C++ throwing class members
Why can I initialize two static class variables in a single file out of order but not three?
Function Pointer - 2 options
Template with default value of method pointer argument
Does the C++ standard allow this floating-point behaviour?
Why can't I declare templated type aliases inside of functions? [duplicate]
friend class cannot access protected member
Incomplete types and initializer_list
Protecting QML source code from plagiarism

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!