python SWIG object compare

Question

I have two lists of SWIG objects: a and b. I need to do set or comparison operations to find items in a that are not in b. (I have other operations to do also, but this is a good starting example).

set(a) -set(b) 

does not give accurate results so I tried:

[item for item in a if item not in b]

In both cases, it returns no items, even though a and b have no elements in common

I have one item in a with a value of:

<Swig Object of type 'OpenAccess_4::oaRect *' at 0x1ad6eea0>

and an item in b:

<Swig Object of type 'OpenAccess_4::oaRect *' at 0x1ad6eca8>

that are considered == when I compare.

The 'is' operator works correctly, but it will be very time consuming to do individual comparisons of the 2 lists, since they can be large, and the operation is repeated many times.

What am I missing about SWIG objects in Python that doesn't allow me to do '==' and 'set' operations?

Answer 1

Provided your objects implement the Python object protocol then the higher level container operations you care about will work automatically. To show how they might be implemented we will focus on operator==/__eq__.

We can set up a test case to investigate how comparison works in Python and SWIG:

import test
f1 = test.foo(1)
f2 = test.foo(2)
f3 = test.foo(1)
f4 = test.static_foo()
f5 = test.static_foo()
a = (f1,f2,f3,f4,f5)

compared = "\n".join((",".join(str(int(y==x)) for y in a) for x in a))

print compared

print "\n".join((str(x) for x in a))

With our naive implementation as a starting point:

%module test

%inline %{
struct foo {
  foo(const int v) : v(v) {}
  int v;
};

foo *static_foo() {
  static foo f{1};
  return &f;
}
%}

when run this gives:

1,0,0,0,0
0,1,0,0,0
0,0,1,0,0
0,0,0,1,0
0,0,0,0,1
<test.foo; proxy of <Swig Object of type 'foo *' at 0xb71e79f8> >
<test.foo; proxy of <Swig Object of type 'foo *' at 0xb71e7ad0> >
<test.foo; proxy of <Swig Object of type 'foo *' at 0xb71e7b78> >
<test.foo; proxy of <Swig Object of type 'foo *' at 0xb71e7b90> >
<test.foo; proxy of <Swig Object of type 'foo *' at 0xb71e7b60> >

This is hardly what we'd hope for, it's only the identity matrix.

So why does it behave like this? Well to start with SWIG, by default constructs a new proxy object for everything that gets returned from C++ to Python. Even in the static case the SWIG output at compile time can't prove that it's always the same object that gets returned so to be safe it always makes a new proxy.

At runtime we could add a typemap to check for and handle that case (e.g. with a std::map that instances are looked up in). That's a separate question though and a distraction from the real issue because it won't make f1==f3 ever since they are distinct but equivalent objects.

Notice here that in C++ we have the same problem, but for various design reasons we couldn't even compile a simple function using operator==:

bool bar() {
  static foo f1{2};
  static foo f2{2};
  return f1==f2;
}

Fails to compile with:

test.cxx:6 error: no match for ‘operator==’ in ‘f1 == f2’

Let's explore this a bit to understand how SWIG behaves when generating Python wrappers. If we add an operator== to our C++ class:

%module test

%inline %{
struct foo {
  foo(const int v) : v(v) {}
  int v;
  bool operator==(const foo& o) const { return v == o.v; }
};

foo *static_foo() {
  static foo f{1};
  return &f;
}

And suddenly SWIG does the right thing and passes that on to Python so our test case now produces:

1,0,1,1,1
0,1,0,0,0
1,0,1,1,1
1,0,1,1,1
1,0,1,1,1
<test.foo; proxy of <Swig Object of type 'foo *' at 0xb72869f8> >
<test.foo; proxy of <Swig Object of type 'foo *' at 0xb7286ad0> >
<test.foo; proxy of <Swig Object of type 'foo *' at 0xb7286a70> >
<test.foo; proxy of <Swig Object of type 'foo *' at 0xb7286bc0> >
<test.foo; proxy of <Swig Object of type 'foo *' at 0xb7286bd8> >

That's exactly the behaviour we'd hope for, all the instances where v is the same are equal, the other one is not. This is despite the proxy objects being different per instance.

What happens if the operator== we wrote is a non-member operator?

%module test

%inline %{
struct foo {
  foo(const int v) : v(v) {}
  int v;
};

foo *static_foo() {
  static foo f{1};
  return &f;
}

bool operator==(const foo& a, const foo& b) { return a.v== b.v; }

Suddenly we've lost the behaviour now and we're back wtih

1,0,0,0,0
0,1,0,0,0
0,0,1,0,0
0,0,0,1,0
0,0,0,0,1

Why? Because it's not so clear what to do with the operator== in this case. If you run SWIG with -Wall you'll see that we are getting a warning now:

test.i:15: Warning 503: Can't wrap 'operator ==' unless renamed to a valid identifier.

So let's assume that we can't edit the C++ code and look at "fixing" the problem. There are a few ways we could do this though.

1. We could ask SWIG to rename the operator== that it doesn't know how to wrap to be a function using %rename as hinted by the warning:

   %rename(compare_foo) operator==(const foo&, const foo&);
   

   This needs to be written anywhere before the declaration/definition of our operator== is seen by SWIG.

   In and of itself this is not sufficient to restore the behaviour we wanted though, so we fix it by adding some extra Python to the output of SWIG. Recall that the Python function __eq__ has the following form:

   object.__eq__(self, other)
   

   That's actually a pretty good match for our C++ operator== still, so we can simply add the following at the end of the SWIG interface file:

   %pythoncode %{
     foo.__eq__ = lambda a,b: compare_foo(a,b)
   %}
   

   Which restores the behaviour we're after. (Note: I'm not sure why the lambda is needed here, I wasn't expecting it to be required)

2. We could also do this by writing some more C++ in our interface, but not having to modify the actual code we're wrapping. Basically what we want to do is implement __eq__ inside foo. This can be done with %extend which extends a class, but only from the perspective of the target language. For completeness we use %ignore to suppress the function that we're getting a warning about since we have dealt with the problem.

   %module test
   
   %ignore operator==(const foo&, const foo&);
   %extend foo {
     bool __eq__(const foo& o) const {
       return *$self == o;
     }
   }
   
   %inline %{
   struct foo {
     foo(const int v) : v(v) {}
     int v;
   };
   
   foo *static_foo() {
     static foo f{1};
     return &f;
   }
   
   bool operator==(const foo& a, const foo& b) { return a.v== b.v; }
   
   bool bar() {
     static foo f1{2};
     static foo f2{2};
     return f1==f2;
   }
   %}
   

   Which again restores the behaviour we're after at, with the difference being that the glue is included as C++ glue rather than Python glue.

3. Finally if you're running SWIG Python with -builtin neither of those solutions will work for the non-member operator case. It's worth noting that the default SWIG output includes a tp_richcompare function. Still to use our operator== instead of an address comparison of the underlying objects you'll need to use the slots mechanism to register our own function similar to the above code.