Efficient way of storing and retrieving complex objects in C++?

Question

I'm currently working on a small dungeon simulation game. The game is quite detailed and i'm planning to have, over time, +200k instances of a class that represents a "monster". They contain perks, skills and history of that monster. Things like how many potions he has used, where he lives, what are his patrol routes, etc.

I started implementing this with SQLite and using a simple table called "monsters", which contained all the data. This allowed me to use SQL queries to find the monsters needed for simulation calculation on each frame. For example: finding all monsters who patrolled point A, or finding all monsters who used Potion X, etc. Unfortunately, querying SQLite several times on every frame quickly slowed down the game. Even though it's a 2D game, i need the precious milliseconds for simulation calculations.

Also, i was going to need JOIN's in the future to do graphs: i need to know if a monster has attacked another monster or if a monster is part of the team of another monster. That would slow things even further.

Does anyone have any suggestion on how to approach this?

My data resembles something like this:

Answer 1

If you are not using an entity component system, consider migrating your game to using one. Each bit of related data can be stored in a self-contained component, and the entity itself is some opaque identifier that identifies the component. When using an ECS, rather than have a game entity with a bunch of data hanging off it, you reverse the relationship. All components of a specific type exist in a big pool together, and your entity is just an identifier that specifies which component in this pool they care about. What this allows you to do is batch component updates. If you have an Inventory component on each monster with an inventory, all of your Inventory components can be stored more or less contiguously in memory. This means that when processing them, you have high cache coherency which can be a significant performance boost.

It also might be that you're just trying to do too much each frame. With an entity component system you can throttle specific subsystems to every X frames or every X seconds if you need to. Maybe the AI component only needs to run once a second to think about what to do next, but they need to keep moving continuously so you update position every frame. Or maybe putting together one of the charts and graphs takes too long to complete in one frame so you have it calculated every second frame, or split the processing over two frames so that you iterate over half your entities on frame and the rest on the second frame. There's a lot of ways you can split it up.

Here is some more reading on component systems if you haven't seen them before