Understanding mixins vs mixin templates

Question

In the process of learning the D language, I'm trying to make a generic Matrix class which supports type promotion of the contained object.

That is, when I multiply a Matrix!(int) to a Matrix!(real) I should get a Matrix!(real) as a result.

Since there are many different kinds of type promotions, reimplementing the opBinary method for every possible combination would be really tedious and a ton of boilerplate code. So mixins/mixin templates would seem to be the answer.

What I'm failing to understand is why this first code sample works

import std.stdio;
import std.string : format;


string define_opbinary(string other_type) {
    return "
        Matrix opBinary(string op)(Matrix!(%s) other) {
            if(op == \"*\") {
                Matrix result;
                if(this.columns == other.rows) {
                    result = new Matrix(this.rows, other.columns);
                } else {
                    result = new Matrix(0,0);
                }
                return result;
            } else assert(0, \"Operator \"~op~\" not implemented\");
        }
        ".format(other_type);
}


class Matrix(T) {
    T[][] storage;
    size_t rows;
    size_t columns;
    const string type = T.stringof;

    this(size_t rows, size_t columns) {
        this.storage = new T[][](rows, columns);
        this.rows = rows;
        this.columns = columns;
    }

    void opIndexAssign(T value, size_t row, size_t column) {
        storage[row][column] = value;
    }


    mixin(define_opbinary(int.stringof));
    mixin(define_opbinary(uint.stringof));
}



void main()
{
    Matrix!int mymat = new Matrix!(int)(2, 2);
    mymat[0,0] = 5;
    writeln(mymat.type);

    Matrix!uint mymat2 = new Matrix!(uint)(2, 2);
    writeln(mymat2.type);
    auto result = mymat * mymat2;
    writeln("result.rows=", result.rows);
    writeln("result.columns=", result.columns);
    auto result2 = mymat2 * mymat;
    writeln("result.type=",result.type);
    writeln("result2.type=",result2.type);
}

the dub output:

Performing "debug" build using /usr/bin/dmd for x86_64.
matrix ~master: building configuration "application"...
Linking...
Running ./matrix.exe 
50
00
int
uint
result.rows=2
result.columns=2
00
00
result.type=int
result2.type=uint

but the second code sample does not work

import std.stdio;
import std.string : format;


mixin template define_opbinary(alias other_type) {
    Matrix opBinary(string op)(Matrix!(other_type) other) {
        if(op == "*") {
            Matrix result;
            if(this.columns == other.rows) {
                result = new Matrix(this.rows, other.columns);
            } else {
                result = new Matrix(0,0);
            }
            return result;
        } else assert(0, "Operator "~op~" not implemented");
    }
}


class Matrix(T) {
    T[][] storage;
    size_t rows;
    size_t columns;
    const string type = T.stringof;

    this(size_t rows, size_t columns) {
        this.storage = new T[][](rows, columns);
        this.rows = rows;
        this.columns = columns;
    }

    void opIndexAssign(T value, size_t row, size_t column) {
        storage[row][column] = value;
    }

    mixin define_opbinary!(int);
    mixin define_opbinary!(uint);
}



void main()
{
    Matrix!int mymat = new Matrix!(int)(2, 2);
    mymat[0,0] = 5;
    writeln(mymat.type);

    Matrix!uint mymat2 = new Matrix!(uint)(2, 2);
    writeln(mymat2.type);
    auto result = mymat * mymat2;
    writeln("result.rows=", result.rows);
    writeln("result.columns=", result.columns);
    auto result2 = mymat2 * mymat;
    writeln("result.type=",result.type);
    writeln("result2.type=",result2.type);
}

the dub output:

source/app.d(60,19): Error: cast(Object)mymat is not of arithmetic type, it is a object.Object
source/app.d(60,27): Error: cast(Object)mymat2 is not of arithmetic type, it is a object.Object
source/app.d(64,20): Error: cast(Object)mymat2 is not of arithmetic type, it is a object.Object
source/app.d(64,29): Error: cast(Object)mymat is not of arithmetic type, it is a object.Object
/usr/bin/dmd failed with exit code 1.

What's extremely odd is that if I remove the mixin define_opbinary!(int); call, then I only get two arithmetic complaints (only the two complaints about line 60 (auto result = mymat * mymat2;) remain).

I have a feeling that somehow the compiler sees the two mixin calls as ambiguous and removes both but I'm not sure.

Any help would be greatly appreciated.

Answer 1

Oh I have a lot to say about this, including that I wouldn't use either type of mixin for this - I'd just use an ordinary template instead. I'll come back to that at the end.

I am going to try to be fairly comprehensive, so apologies if I describe stuff you already know, and on the other hand, I am probably going to give some irrelevant material too in the interests of providing comprehensive background material for a deeper understanding.

First, mixin vs template mixin. mixin() takes a string, parses it into a AST node (the AST btw is the compiler's internal data structure for representing code, it stands for "abstract syntax tree". foo() is an AST node like FunctionCall { args: [] }. if(foo) {} is one like IfStatement { condition: Expression { arg: Variable { name: foo }, body : EmptyStatement } - basically objects representing each part of the code).

Then it pastes that parsed AST node into the same slot where the mixin word appeared. You can often think of this as copy/pasting code strings, but with the restriction that the string must represent a complete element here, and it must be substituteable in the same context where the mixin was without errors. So like you can't do int a = bmixin(c) to make a variable with a b in front - the mixin must represent a complete node by itself.

Once it pastes in that AST node though, the compiler treats it as if the code was all written there originally. Any names referenced will be looked up in the pasted context, etc.

A template mixin, on the other hand, actually still has a container element in the AST, which is used for name lookups. It actually works similarly to a struct or class inside the compiler - they all have a list of child declarations that remain together as a unit.

The big difference is that a template mixin's contents are automatically accessible from the parent context... usually. It follows rules similar to class inheritance, where class Foo : Bar can see Bar's members as if they are its own, but they still remain separate. You can still do like super.method(); and call it independently of the child's overrides.

The "usually" comes in because of overloading and hijacking rules. Deep dive and rationale here: https://dlang.org/articles/hijack.html

But the short of it is in an effort to prevent third party code from silently being able to change your program's behavior when they add a new function, D requires all sets of function overloads to be merged at the usage point by the programmer, and it is particularly picky about operator overloads since they already have a default behavior that any mixin is going to be modifying.

mixin template B(T) {
   void foo(T t) {}
}
class A {
   mixin B!int;
   mixin B!string;
}

This is similar to the code you have, but with an ordinary function. If you compile and run, it will work. Now, let's add a foo overload directly to A:

mixin template B(T) {
   void foo(T t) {}
}
class A {
   mixin B!int;
   mixin B!string;

   void foo(float t) {}
}

If you try to compile this with a string argument, it will actually fail! "Error: function poi.A.foo(float t) is not callable using argument types (string)". Why won't it use the mixin one?

This is a rule of template mixins - remember the compiler still treats them as a unit, not just a pasted set of declarations. Any name present on the outer object - here, our class A - will be used instead of looking inside the template mixin.

Hence, it sees A.foo and doesn't bother looking into B to find a foo. This is kinda useful for overriding specific things from a template mixin, but can be a hassle when trying to add overloads. The solution is to add an alias line to the top-level to tell the compiler to specifically look inside. First, we need to give the mixin a name, then forward the name explicitly:

mixin template B(T) {
   void foo(T t) {}
}
class A {
   mixin B!int bint; // added a name here
   mixin B!string bstring; // and here

   alias foo = bint.foo; // forward foo to the template mixin
   alias foo = bstring.foo; // and this one too

   void foo(float t) {}
}

void main() {
    A a = new A;
    a.foo("a");
}

Now it works for float, int, and string.... but it also kinda defeats the purpose of template mixins for adding overloads. One trick you can to is to put a top-level template function in A, and it just forwards to the mixins... just they need a different name to register.

Which brings me back to your code. Like I said, D is particularly picky about operator overloads since they always override a normal behavior (even when that normal behavior is an error, like in classes). You need to be explicit about them at the top level.

Consider the following:

import std.stdio;
import std.string : format;


mixin template define_opbinary(alias other_type) {
    // I renamed this to opBinaryHelper since it will not be used directly
    // but rather called from the top level
    Matrix opBinaryHelper(string op)(Matrix!(other_type) other) {
        if(op == "*") {
            Matrix result;
            if(this.columns == other.rows) {
                result = new Matrix(this.rows, other.columns);
            } else {
                result = new Matrix(0,0);
            }
            return result;
        } else assert(0, "Operator "~op~" not implemented");
    }
}


class Matrix(T) {
    T[][] storage;
    size_t rows;
    size_t columns;
    const string type = T.stringof;

    this(size_t rows, size_t columns) {
        this.storage = new T[][](rows, columns);
        this.rows = rows;
        this.columns = columns;
    }

    void opIndexAssign(T value, size_t row, size_t column) {
        storage[row][column] = value;
    }

    mixin define_opbinary!(int);
    mixin define_opbinary!(uint);

    // and now here, we do a top-level opBinary that calls the helper
    auto opBinary(string op, M)(M rhs) {
        return this.opBinaryHelper!(op)(rhs);
    }
}



void main()
{
    Matrix!int mymat = new Matrix!(int)(2, 2);
    mymat[0,0] = 5;
    writeln(mymat.type);

    Matrix!uint mymat2 = new Matrix!(uint)(2, 2);
    writeln(mymat2.type);
    auto result = mymat * mymat2;
    writeln("result.rows=", result.rows);
    writeln("result.columns=", result.columns);
    auto result2 = mymat2 * mymat;
    writeln("result.type=",result.type);
    writeln("result2.type=",result2.type);
}

I pasted in the complete code, but there's actually only two changes there: the mixin template now defines a helper with a different name (opBinaryHelper), and the top-level class now has an explicit opBinary defined that forwards to said helper. (If you were to add other overloads btw, the alias trick from above may be necessary, but in this case, since it is all dispatched on if from inside the one name, it lets you merge all the helpers automatically.)

Finally, the code works.

Now, why wasn't any of this necessary with the string mixin? Well, back to the original definition: a string mixin parses it, then pastes in the AST node /as if it were originally written there/. That latter part lets it work (just at the cost of once you mixin a string, you are stuck with it, so if you don't like part of it, you must modify the library instead of just overriding a portion).

A template mixin maintains its own sub-namespace to allow for selective overriding, etc., and that triggers a foul with these stricter overloading rules.

---

And finally, here's the way I'd actually do it:

// this MatrixType : stuff magic means to accept any Matrix, and extract
// the other type out of it.
// a little docs: https://dlang.org/spec/template.html#alias_parameter_specialization

// basically, write a pattern that represents the type, then comma-separate
// a list of placeholders you declared in that pattern

auto opBinary(string op, MatrixType : Matrix!Other_Type, Other_Type)(MatrixType other) {
    // let the compiler do the promotion work for us!
    // we just fetch the type of regular multiplication between the two types
    // the .init just uses the initial default value of the types as a placeholder,
    // all we really care about is the type, just can't multiply types, only
    // values hence using that.
    alias PromotedType = typeof(T.init * Other_Type.init);

    // in your version, you used `if`, but since this is a compile-time
    // parameter, we can use `static if` instead and get more flexibility
    // on stuff like actually changing the return value per operation.
    //
    // Don't need it here, but wanted to point it out anyway.
    static if(op == "*") {
        // and now use that type for the result
        Matrix!PromotedType result;
        if(this.columns == other.rows) {
            result = new Matrix!PromotedType(this.rows, other.columns);
        } else {
            result = new Matrix!PromotedType(0,0);
        }
        return result;
    // and with static if, we can static assert to turn that runtime
    // exception into a compile-time error
    } else static assert(0, "Operator "~op~" not implemented");
}

Just put that opBinary in your class and now the one function can handle all the cases - no need to list specific types, so no more need for mixin magic at all! (....well unless you need virtual overriding with child classes, but that's a whole other topic. Short tip tho, it is possible to static foreach that, which I talked about in my last SO answer here: https://stackoverflow.com/a/57599398/1457000 )

There's a few D tricks in that little function, but I tried to explain in the comments of the code. Feel free to ask if you need more clarification though - those : patterns in template are IMO one of the more advanced D compile-time reflection things, so they're not easy to get at first, but for simple cases like this, it kinda makes sense, just think of it as a declaration with placeholders.