boost::python and weak_ptr : stuff disappearing

Question

I would like to store a reference to an object as a weak_ptr. In pure C++, the following works :

#include <iostream>

#include <boost/shared_ptr.hpp>
#include <boost/weak_ptr.hpp>

using namespace std;
using namespace boost;

struct Empty
{
    Empty(){}
};

struct Store
{
    weak_ptr<Empty> value;
    Store(){};

    void setValue(shared_ptr<Empty> v) {
        cout << "storing " << v << endl;
        this->value = weak_ptr<Empty>(v);
        shared_ptr<Empty> v_ok = this->value.lock();
        if (v_ok) {
            cout << "ok, v has been stored" << endl;
        }
    }

    shared_ptr<Empty> getValue() {
        shared_ptr<Empty> p = this->value.lock();
        if (p) {
            cout << "stored value : " << p << endl;
        } else {
            cout << "there's nothing here !" << endl;
        }
        return p;
    }
};

int main()
{
    shared_ptr<Empty> e(new Empty);
    shared_ptr<Store> st(new Store);

    st->setValue(e);
    st->getValue();
    return 0;
}

compiling and running this will give you this :

%> ./a.out 
storing 0x8c6c008
ok, v has been stored
stored value : 0x8c6c008

Now, if I encapsulate that with boost python :

#include <iostream>

#include <boost/shared_ptr.hpp>
#include <boost/python.hpp>
#include <boost/weak_ptr.hpp>

using namespace std;
using namespace boost;
using namespace boost::python;

struct Empty
{
    Empty(){}
};

struct Store
{
    weak_ptr<Empty> value;
    Store(){};

    void setValue(shared_ptr<Empty> v) {
        cout << "storing " << v << endl;
        this->value = weak_ptr<Empty>(v);
        shared_ptr<Empty> v_ok = this->value.lock();
        if (v_ok) {
            cout << "ok, v has been stored" << endl;
        }
    }

    shared_ptr<Empty> getValue() {
        shared_ptr<Empty> p = this->value.lock();
        if (p) {
            cout << "stored value : " << p << endl;
        } else {
            cout << "there's nothing here !" << endl;
        }
        return p;
    }
};

BOOST_PYTHON_MODULE (test)
{
    class_< Empty, shared_ptr<Empty> >("Empty");

    class_< Store, shared_ptr<Store> >("Store")
    .def("get",&Store::getValue)
    .def("set",&Store::setValue);
}

and now a small python script to try it out

from test import *

e = Empty()
st = Store()

st.set(e)
st.get()

... and the result is ...

storing 0x9eb2a18
ok, v has been stored
there's nothing here !

so apparently while I'm still in the same method (setValue), there is no problem retrieving a shared_ptr from Store::value. But as soon as I get out of this context, there's nothing left !

How can this be ? Is python passing a brand new (and useless) shared_ptr as an argument to setValue, which is then destroyed at the end of the call ? I'm lost here.

Answer 1

Another workaround is to pass in a reference to the shared_ptr, not the shared_ptr object itself. In that case obviously boost::python-Wrapper is not creating this strange separate shared_ptr.

Btw. I you look at the type of the class pointed to by shared_ptr.pn.pi_ you'll see that the "original" shared_ptr handed out to Python contains a pointer to a sp_counted_impl_p<POINTEE_TYPE> object while the shared_ptr passed back in from Python contains a pointer to a sp_counted_impl_pd<void*, shared_ptr_deleter> object (see smart_ptr/detail/sp_counted_impl.hpp) The pd variant does not contain a reference to the pointee. I suspect that this sp_counted_impl_pd object is somehow referencing the sp_counted_impl_p object. This would explain why the destructor is not called when the reference count of the shared_ptr passed back in from Python drops to zero. The fact that weak_ptr does not work in this case might simply be a bug then..?

Answer 2

This is extremely curious. I've ruled out the std vs. boost shared pointer possibility, and played around with a few sanity checks, and as far as I can tell it's something that boost python is doing to the shared pointer that breaks it.

Tracing object constructors / destructors, the lifetimes of Empty and Store are being managed as you would expect (no copies occur).

An extremely interesting thing is that shared_from_this continues to work, even when weak_ptr<>.lock() doesn't, and, in fact, a new weak pointer created from a new shared pointer (from shared_from_this) does work.

So this lead me to the thread linked in the comments, it seems like there's something boost python is doing with the deleter and reference count that is breaking the weak pointer.

Inspecting the shared pointers in the debugger this is what we get:

When we call setValue, this is what the argument looks like:

1: p = (const 'boost::shared_ptr<Empty>' &) @0x7fff5fbfe720: {
  px = 0x100346900, 
  pn = {
    pi_ = 0x100338dd0
  }
}
> p *p.pn.pi_
$5 = (boost::detail::sp_counted_impl_pd<void*,boost::python::converter::shared_ptr_deleter>) {
  <boost::detail::sp_counted_base> = {
    _vptr$sp_counted_base = 0x10061aa30, 
    use_count_ = 2, 
    weak_count_ = 2
  }, 
  members of boost::detail::sp_counted_impl_pd<void*,boost::python::converter::shared_ptr_deleter>: 
  ptr = 0x0, 
  del = {
    owner = {
      m_p = 0x10049db90
    }
  }
}

If we create a shared pointer using shared_from_this on the argument, it looks like this:

1: p = (const 'boost::shared_ptr<Empty>' &) @0x7fff5fbfe5e0: {
  px = 0x100346900, 
  pn = {
    pi_ = 0x1003468e0
  }
}
> p *p.pn.pi_
$4 = (boost::detail::sp_counted_impl_pd<Empty*,boost::detail::sp_ms_deleter<Empty> >) {
  <boost::detail::sp_counted_base> = {
    _vptr$sp_counted_base = 0x10061b170, 
    use_count_ = 2, 
    weak_count_ = 2
  }, 
  members of boost::detail::sp_counted_impl_pd<Empty*,boost::detail::sp_ms_deleter<Empty> >: 
  ptr = 0x0, 
  del = {
    initialized_ = true, 
    storage_ = {
      data_ = "\000i4\000\001\000\000\000?h4\000\001\000\000", 
      align_ = {<No data fields>}
    }
  }
}

There's one thing to note here: the address of the shared count is different: this is a different shared pointer instance... so we've somehow created two different shared pointers to the same address. This is an extremely bad thing, since we'd expect this to shortly result in a double-free.

However, it doesn't. (I'm quite keen to understand this further, if anyone has any ideas?)

I don't know why it doesn't, to be honest (presumably there is something subtle going on here), but in any case, all this does point to a solution: We can use shared_from_this to create a shared pointer, from the value passed, which we can use to create a weak pointer that actually works.

So, to wrap up, here's the fix:

#include <iostream>
#include <boost/shared_ptr.hpp>
#include <boost/python.hpp>
#include <boost/weak_ptr.hpp>
#include <boost/enable_shared_from_this.hpp>

namespace bp = boost::python;

struct Empty: boost::enable_shared_from_this<Empty>{ };

struct Store
{
    boost::weak_ptr<Empty> value;

    void setValue(boost::shared_ptr<Empty> const& v) {
        value = boost::weak_ptr<Empty>(v->shared_from_this());
        boost::shared_ptr<Empty> v_ok = value.lock();
        if (v_ok) {
            std::cout << "ok, v has been stored" << std::endl;
        }
    }

    boost::shared_ptr<Empty> getValue() {
        boost::shared_ptr<Empty> p = value.lock();
        if (p) {
            std::cout << "stored value : " << p << std::endl;
        } else {
            std::cout << "there's nothing here !" << std::endl;
        }
        return p;
    }
};

BOOST_PYTHON_MODULE (libmylibinterface)
{
    bp::class_< Empty, boost::shared_ptr<Empty> >("Empty",bp::init<>())
        ;

    bp::class_< Store, boost::shared_ptr<Store> >("Store")
        .def("get",&Store::getValue)
        .def("set",&Store::setValue);

}