Patterns: Create and translate between data objects and wire formats

Question

I was inclined to just use the mapper pattern which I use in various places in the app's code already. But I thought it might actually not be the best fit in this particular case:

The task:

• I need to implement data objects according to a given specification. The specification defines multiple versions for each object type, thus I have for example a class CarV1 and CarV2 representing each version of the specification.

• I need to translate these models between classes (C++ in this case, but the question is about general design) and wire formats (Json, Protocol Buffers), and vice versa.

• Construction of the objects is rather simple.

As I said, I'd normally use a mapper pattern, define a mapper interface and concrete mappers to map between each format. There are two things though why I ask for your opinion in this case:

• I'd use the mapper pattern only to map between two, and only two, types of formats, e.g. a database object and a model class. I already have a third format in this case, and it's possible that I have to add more formats to translate between in the near future.

• The versioning adds some complexity on top of the mapping, and I think there needs to be another indirection in between.

I've read about the Translator Pattern [1], but never used it. I think it fits to some degree, but not completely.

I also considered an Abstract Factory. This would allow to create similar objects (in my case versioned objects). But it is not a good fit for mapping between object representations.

What pattern should I use, and why?

[1] http://www.iro.umontreal.ca/~keller/Layla/translator.pdf

Answer 1

Implementation Target

We are going to write an automatic translator. Let's say we have an object representing our wire format:

JsonObject wire_data;

For convenience, we can imagine that our JsonObject has an add_field member function:

wire_data.add_field("name", "value"); 

However the actual interface of the JsonObject is actually irrelevant, and the rest of this post doesn't rely on it being implemented any particular way.

We want to be able to write this function:

template<class Car>
void add_car_info(JsonObject& object, Car car) {
    // Stuff goes here
}

with the following constraints:

• If Car has a field, e.g. Car::getMake(), our function add_car_info should add that field to the json object automatically
• If Car doesn't have a field, our function doesn't have to do anything.
• Our implementation shouldn't rely on Car being derived from anything, or being a base class of anything
• Our implementation should make it trivial to add new fields, without breaking backwards compatibility.

Example with four independent Car classes

Let's say you have four car classes. None of them share a base class; what fields they expose varies; and you may be adding more car classes in the future.

struct Car1
{
    std::string getMake() { return "Toyota"; }
    std::string getModel() { return "Prius"; }
    int getYear() { return 2013; }
};
struct Car2
{
    std::string getMake() { return "Toyota"; }
    int         getYear() { return 2017; };
};
struct Car3
{
    std::string getModel() { return "Prius"; }
    int         getYear() { return 2017; }
};
struct Car4
{
    long long getSerial() { return 2039809809820390; }
};

Now,

JsonObject wire_data;
Car1 car1;
add_field(wire_data, car1);

Should be equivilant to

Car1 car1; 
wire_data.add_field("car make", car1.getMake()); 
wire_data.add_field("car model", car1.getModel()); 
wire_data.add_field("year", car1.getYear()); 

While

Car2 car2;
add_field(wire_data, car2); 

Should be equivalent to

Car2 car2; 
wire_data.add_field("car make", car2.getMake()); 
wire_data.add_field("year", car2.getYear()); 

How do we implement add_car_info in a generic way?

Figuring out which cars have which fields is a tricky problem, especially because C++ doesn't have dynamic reflection, but we can do it using static reflection (and it'll be more efficient too)!

For now, I'm going to delegate the functionality to a object representing the translator.

template<class Car>
void add_car_info(JsonObject& wire_object, Car car) {
    auto translator = getCarTranslator(); 

    // This lambda adds the inputs to wire_object
    auto add_field = [&](std::string const& name, auto&& value) {
        wire_object.add_field(name, value); 
    };
    // Add the car's fields. 
    translator.translate(add_field, car); 
}         

It looks like the translator object just kicks, the can down the road, however having a translator object will make it easy to write translators for stuff other than cars.

How do we implement the magic translator?

Let's start off with getCarTranslator. With cars, there's four things we might care about: the make the model, the year, and the serial number.

auto getCarTranslator() {
    return makeTranslator(READ_FIELD("car make", getMake()),
                          READ_FIELD("car model", getModel()),
                          READ_FIELD("year", getYear()),
                          READ_FIELD("serial", getSerial()));
}

We're using a macro here, but I promise it's the only one, and it's not a complex macro:

// This class is used to tell our overload set we want the name of the field
class read_name_t
{
};

#define READ_FIELD(name, field)                                      \
    overload_set(                                                    \
        [](auto&& obj) -> decltype(obj.field) { return obj.field; }, \
        [](read_name_t) -> decltype(auto) { return name; })

We're defining an overload set over two lambdas. One of them gets the object's field, and the other one of them gets the name used for serialization.

Implementing an overload set for lambdas

This is pretty straight-forward. We just create a class that inherits from both lambdas:

template <class Base1, class Base2>
struct OverloadSet
  : public Base1
  , public Base2
{
    OverloadSet(Base1 const& b1, Base2 const& b2) : Base1(b1), Base2(b2) {}
    OverloadSet(Base1&& b1, Base2&& b2)
      : Base1(std::move(b1)), Base2(std::move(b2))
    {
    }
    using Base1::operator();
    using Base2::operator();
};

template <class F1, class F2>
auto overload_set(F1&& func1, F2&& func2)
    -> OverloadSet<typename std::decay<F1>::type, typename std::decay<F2>::type>
{
     return {std::forward<F1>(func1), std::forward<F2>(func2)};
}

Implementing a translator class using a tiny bit of SFINAE

The first step is to have a class that reads an individual field. It contains a lambda that does the reading. If we can apply the lambda, we apply it (reading the field). Otherwise, we don't apply it, and nothing happens.

template <class Reader>
class OptionalReader
{
public:
    Reader read;
    template <class Consumer, class Object>
    void maybeConsume(Consumer&& consume, Object&& obj) const
    {
        // The 0 is used to dispatch it so it considers both overloads
        maybeConsume(consume, obj, 0);
    }

private:
    // This is used to disable maybeConsume if we can't read it
    template <class...>
    using ignore_t = void;

    // This one gets called if we can read the object
    template <class Consumer, class Object>
    auto maybeConsume(Consumer& consume, Object& obj, int) const
        -> ignore_t<decltype(consume(read(read_name_t()), read(obj)))>
    {
        consume(read(read_name_t()), read(obj));
    }

    // This one gets called if we can't read it
    template <class Consumer, class Object>
    auto maybeConsume(Consumer&, Object&, long) const -> void
    {
    }
};

A translator takes a bunch of optional appliers, and just applies them in succession:

template <class... OptionalApplier>
class Translator : public OptionalApplier...
{
public:
    // Constructors
    Translator(OptionalApplier const&... appliers)
      : OptionalApplier(appliers)... {}

    Translator(OptionalApplier&&... appliers) 
      : OptionalApplier(appliers)... {}

    // translate fuction
    template <class Consumer, class Object>
    void translate(Consumer&& consume, Object&& o) const
    {
        // Apply each optional applier in turn
        char _[] = {((void)OptionalApplier::maybeConsume(consume, o), '\0')...};
        (void)_;
    }
};

Making the makeTranslator function is really simple now. We just take a bunch of readers, and use them to make optionalReaders.

template <class... Reader>
auto makeTranslator(Reader const&... readers)
    -> Translator<OptionalReader<Reader>...>
{
    return {OptionalReader<Reader>{readers}...};
}

Conclusion

This was a long post. There was a lot of infrastructure we had to build to get everything to work. It's really simple to use, though, and it doesn't require any knowledge about what classes we apply it on, except for what fields we're looking to use.

We can write translators for lots of stuff really easily!

Image translator example

For example, here's a translator for pictures and images that also takes into account different common names for things like the width and height of a picture.

Remember, any image class given to a translator can optionally implement any of these methods.

auto getImagesTranslator() {
    // Width and height might be implemented as `getWidth` and `getHeight`,
    // Or as `getRows` and `getCols`
    return makeTranslator(READ_FIELD("width", getWidth()),
                          READ_FIELD("height", getHeight()),
                          READ_FIELD("width", getCols()),
                          READ_FIELD("height", getRows()),
                          READ_FIELD("location", getLocation()),
                          READ_FIELD("pixel format", getPixelFormat()),
                          READ_FIELD("size", size()),
                          READ_FIELD("aspect ratio", getAspectRatio()),
                          READ_FIELD("pixel data", getPixelData()),
                          READ_FIELD("file format", getFileFormat())); 
}

Here's the complete implementation