Game Engine Runtime Decompression

Question

I have been trying to find out how game engines deal with asset compression. Obviously they compress all the assets when building the game. But how do they decompress them during runtime. The only thing I could think of was decompressing into memory, but that must be very memory intensive. If they decompressed onto the HDD a folder would be filling up while the game is playing? This doesn't sound very efficient.

Using a library like zlib (or any other) with c++, how is this runtime decompression done?

David

Answer 1

It's kinda like this, you have a game with so much data that it won't fit in a reasonable amount of memory, so you can't load all of them at once in memory, so you define some buffers for the data that you use as caches.

Now memory in order of speed from lowest to highest is as such:

• DVD
• Hard Drive
• RAM

Ideally you won't have to stream data from the DVD, but it's something to take into account if you have to make a console game for instance. So for each of these available storage spaces you define a buffer to be used as a cache.

When the game engine decides that it might need an asset, it should first look in the fastest cache to see if the asset is already loaded. If it is then you're in luck, you can immediately send it to be drawn. If it's not in the fastest cache, you have to go down a level to the hard drive cache. This is a file where you keep assets that were already decompressed and ready to be loaded into memory. If the fastest cache isn't fully occupied then you can just start loading the data and use it when it's ready. If there isn't enough space, then you will have to unload other assets first, I would recommend removing the least recently used assets until you have enough space to load the new one.

Now if the hard drive cache doesn't have the data loaded then you have to go down one more level to the archive, you will want to use the zip format to compress it because the zip file format doesn't force you to decompress the entire archive to have access to just one file so all you have to do is find the offset of said file within the archive and decompress it into the hard drive cache. Again if the cache is full you'd have to unload some other assets first, again I'd recommend removing the least recently used but you can try other algorithms as well if you think it will improve performance.

John Karmack had a keynote at QuakeCon 2011 where he explained this whole process maybe a little bit better than I can in a post (among other awesome things), you can find it here