Optimal datafile format loading on a game console

Question

I need to load large models and other structured binary data on an older CD-based game console as efficiently as possible. What's the best way to do it? The data will be exported from a Python application. This is a pretty elaborate hobby project.

Requierements:

• no reliance on fully standard compliant STL - i might use uSTL though.
• as little overhead as possible. Aim for a solution so good. that it could be used on the original Playstation, and yet as modern and elegant as possible.
• no backward/forward compatibility necessary.
• no copying of large chunks around - preferably files get loaded into RAM in background, and all large chunks accessed directly from there later.
• should not rely on the target having the same endianness and alignment, i.e. a C plugin in Python which dumps its structs to disc would not be a very good idea.
• should allow to move the loaded data around, as with individual files 1/3 the RAM size, fragmentation might be an issue. No MMU to abuse.
• robustness is a great bonus, as my attention span is very short, i.e. i'd change saving part of the code and forget the loading one or vice versa, so at least a dumb safeguard would be nice.
• exchangeability between loaded data and runtime-generated data without runtime overhead and without severe memory management issues would be a nice bonus.

I kind of have a semi-plan of parsing in Python trivial, limited-syntax C headers which would use structs with offsets instead of pointers, and convenience wrapper structs/classes in the main app with getters which would convert offsets to properly typed pointers/references, but i'd like to hear your suggestions.

Clarification: the request is primarily about data loading framework and memory management issues.

Answer 1

On platforms like the Nintendo GameCube and DS, 3D models are usually stored in a very simple custom format:

• A brief header, containing a magic number identifying the file, the number of vertices, normals, etc., and optionally a checksum of the data following the header (Adler-32, CRC-16, etc).
• A possibly compressed list of 32-bit floating-point 3-tuples for each vector and normal.
• A possibly compressed list of edges or faces.
• All of the data is in the native endian format of the target platform.
• The compression format is often trivial (Huffman), simple (Arithmetic), or standard (gzip). All of these require very little memory or computational power.

You could take formats like that as a cue: it's quite a compact representation.

My suggestion is to use a format most similar to your in-memory data structures, to minimize post-processing and copying. If that means you create the format yourself, so be it. You have extreme needs, so extreme measures are needed.