Extending boost serialization

Question

I'm attempting to setup multi-purpose serialization for a networked video game for mobile devices. Because it's networked, during initial connection I need to serialize all data for the game state, however, once a game is in progress I will only need to serialize certain changes. The save and load methods that are part of the boost serialization library only have a version number as an parameter. What I'd like to be able to do is have more parameters so that I can change the conditions for what gets saved and loaded based on more than just a version number.

Boost serialization docs are here, for reference.

Here's what the plain boost serialization save method currently looks like:

template<class Archive>
void save(Archive& ar, const unsigned int version) const
{
    // Serialize stuff here
}

Here's what I'd like to accomplish:

template<class Archive>
void save(Archive& ar, const unsigned int version, const unsigned int state_flags) const
{
    if (state_flags & INITIAL_SERIALIZATION)
    {
        // Serialize only data needed for an initial serialization
    }

    // Other serialization
}

I doubt I can make the boost library call my serialization method that I want because it has overloaded operators made to call one with the specific signature in the first example above. I'm imagining calling my own version of save from within the call to save shown in the first example, and maybe grab the state_flags from a separate location. Does anyone have any ideas on how this could be done cleanly, or any good alternatives?

EDIT: I've run into another issue. I need to serialize objects that aren't necessarily members of a class, but the documentation doesn't mention any support for this.

Here's a simple example:

class Foo
{
private:
    SomeClass m_object;

    template<class Archive>
    void save(Archive& ar, const unsigned int version) const
    {
        Bar* pBar = m_object->getComponent<Bar>();
        ar & pBar;   // <--- But pBar isn't a member of Bar, it's part of SomeClass.
    }
};

I would just serialize SomeClass and let that trickle down to Bar, but in this case it is a class that is part of a third party library/engine, not something I can modify. Will Boost serialization allow me to serialize and deserialize this way?

Answer 1

EDIT: new answer added below to address the actual problem.

Your question implies that you deserialize to the same object repeatedly. It's circumstantial if that is clean or not. If, for example, you have a chess board, you would want to synchronize the initial position of the pieces (to continue from the last saved game). To communicate the moves while the game is played, it may be a better idea to send the individual moves as separate objects (that are then applied to the board object once received) instead of transmitting the whole board object which will only transfer what has changed if it's already 'initialized'. That way you can validate the input first and ignore invalid moves. Anyway, i just wanted to mention that, let's move on.

If you have an object that may be synchronized multiple times, with member data that only needs to be transferred once, let the object decide if it's 'initialized' or not (and consequently, if it needs to transmit everything or just a sub set) by using a flag (which is not serialized).

Then you can check the flag in the object's serialization code, just like in the code you posted (with the exception that the flag is not a parameter to the serialization method, but a member variable of the object you're de/serializing). If the flag is set, de/serialize everything and reset the flag. Both client and server must have the same state of the flag, or the serialization breaks.

Alternatively you could serialize the flag first, to tell the receiver how the deserialization must be performed (one bit for each member data group, for example).

Keep in mind, that the deserialization must match the serialization; You must extract the same objects, in the same order as they were serialized.

However, you can serialize polymorphic classes, given they're serialized through the same level in the class hierarchy as they are deserialized (when in doubt, cast to the base pointer when sending and deserialize through the base pointer as well).

Concerning your second question, what you're looking for is non-intrusive serialization. The non-intrusive serialization calls free-standing functions and passes the object to be serialized as parameter (that's how std::vector and boost::shared_ptr are serialized). You can use BOOST_SERIALIZATION_SPLIT_FREE to split the free-standing serialize() function into save() and load(). For intrusive serialization it's BOOST_SERIALIZATION_SPLIT_MEMBER.

To write a generalized de/serialization function (that transmits an objects through the network for example) you can use templates:

template<typename T>
void transmit( const T& data ) {
    // ...
    archive << data
    socket << archive_stream;
}

The restriction with this method is that the receiver must know what kind of object was sent. If you want to send random objects, make them polymorphic:

IData* data = 0;
archive >> data;
switch( data->type() ) {
case TYPE_INIT:
    return dispatch( static_cast<Board*>(data) );
case TYPE_MOVE:
    return dispatch( static_cast<Move*>(data) );
case TYPE_CHAT:
    return dispatch( static_cast<ChatMsg*>(data) );
}

---

UPDATE: If you need to control how your (custom) serialization methods/functions behave, based on a state unknown to the types being serialized, you can implement your own archive class which holds the state. The serialization functions can then query the state and act accordingly.

This state (or an appropriate substitution) must be serialized as well to indicate how the data must be deserialized. For example, this 'different behavior' of the serialization functions could be some sort of compression, and the state is the kind of compression used.

Here's a minimal example of a custom output archive. For more information you can read Derivation from an Existing Archive and dig through the boost sources.

Given a class you can't modify:

struct Foo {
    Foo() : i(42), s("foo") {}
    int i;
    std::string s;
};

You wish to serialize i and/or s based on a condition unknown to the class. You could create a wrapper to serialize it and add the state, but this won't work if the object is inside a vector (or other class for that matter).

It may be easier to make the archive aware of the state instead:

#include <boost/archive/text_oarchive.hpp>

// using struct to omit a bunch of friend declarations    
struct oarchive : boost::archive::text_oarchive_impl<oarchive>
{
    oarchive(std::ostream& os, unsigned flags=0)
      : boost::archive::text_oarchive_impl<oarchive>(os,flags),mask(0){}

    // forward to base class
    template<class T> void save( T& t ) {
        boost::archive::text_oarchive_impl<oarchive>::save(t);
    }

    // this is the 'state' that can be set on the archive
    // and queried by the serialization functions
    unsigned get_mask() const { return mask; }
    void set_mask(unsigned m) { mask = m; }
    void clear_mask() { mask = 0; }
private:
    unsigned mask;
};

// explicit instantiation of class templates involved
namespace boost { namespace archive {
   template class basic_text_oarchive<oarchive>;
   template class text_oarchive_impl<oarchive>;
   template class detail::archive_serializer_map<oarchive>;
} }

// template implementations (should go to the .cpp)
#include <boost/archive/impl/basic_text_oarchive.ipp>
#include <boost/archive/impl/text_oarchive_impl.ipp>
#include <boost/archive/impl/archive_serializer_map.ipp>

Now the state to set and query:

enum state { FULL=0x10, PARTIAL=0x20 };

And a method to set the state (this is just a very basic example):

oarchive& operator<<(oarchive& ar, state mask) {
    ar.set_mask(ar.get_mask()|mask);
    return ar;
}

Finally, the (non-intrusive) serialization function:

namespace boost { namespace serialization {

template<class Archive>
void save(Archive & ar, const Foo& foo, const unsigned int version)
{
    int mask = ar.get_mask(); // get state from the archive
    ar << mask; // serialize the state! when deserializing,
    // read the state first and extract the data accordingly

    if( mask & FULL )
        ar << foo.s; // only serialize s if FULL is set
    ar << foo.i;     // otherwise serialize i only
    ar.clear_mask(); // reset the state
}

} } // boost::serialization

BOOST_SERIALIZATION_SPLIT_FREE(Foo)

And this can be used as follows:

int main() {
    std::stringstream strm;
    oarchive ar(strm);

    Foo f;
    ar << PARTIAL << f << FULL << f;

    std::cout << strm.str();
}

The purpose of this example is just to illustrate the principle. It is too basic for production code.