SWIG, C++, & Python: C++ temporary objects deleted too soon

Question

I'm having an issue with SWIG deleting temporary C++ objects too soon.

Example output from a Python test script:

--------------------------------------------------------------------------------
Works as expected:
    b0 =  Buffer(0, 0, 0, )
    b1 =  Buffer(1, 1, 1, )
    b0 =  Buffer(0, 0, 0, 1, 1, 1, )
    y  =  Buffer(0, 0, 0, 1, 1, 1, )
    b1 =  Buffer(1, 1, 1, )
    repr(b0) =  Buffer(id = 0, vector at 0x020bf450, data at 0x020aeb30, size = 6)
    repr(y)  =  Buffer(id = 0, vector at 0x020bf450, data at 0x020aeb30, size = 6)
Funny business:
    Deleting Buffer(id = 2)
    Deleting Buffer(id = 3)
    repr(b2) =  Buffer(id = 2, vector at 0x020bf790, data at 0x00, size = 4257068)
    Deleting Buffer(id = 4)
    repr(b3) =  Buffer(id = 4, vector at 0x02037040, data at 0x0204a4e0, size = 6)
    Deleting Buffer(id = 0)
    Deleting Buffer(id = 1)

The Deleting Buffer(id = X) is being generated from inside Buffer::~Buffer() C++ code, so we can see here that in the Funny business section, the C++ Buffer objects are getting deleted too early! The Python objects b2 and b3 should be holding references to the C++ Buffer objects with id=2 and id=4.

All my code is attached to my blog post about this issue. However, I'll summarize the code here:

Buffer.hpp:

#include <vector>
#include <string>

struct Buffer
{
    Buffer();
    Buffer(const Buffer & copy);
    ~Buffer();

    Buffer & operator=(const Buffer & rhs);

    Buffer & operator<<(const Buffer & rhs);
    Buffer & operator<<(double rhs);

    std::string __str__() const;
    std::string __repr__() const;

    private:

    std::vector<double> _data;
    int                 _id;
};

swig_test.i:

%module swig_test

%include "std_string.i"

%{
    #include "Buffer.hpp"
    #include <iostream>
%}

%ignore Buffer::operator=;

%include "Buffer.hpp"

go_test.py:

from swig_test import Buffer


def zeros(n):
    '''
    Returns a Buffer filled with 'n' zeros.
    '''

    b = Buffer()

    for i in xrange(n):
        b << 0.0

    return b


def ones(n):
    '''
    Returns a Buffer filled with 'n' ones.
    '''

    b = Buffer()

    for i in xrange(n):
        b << 1.0

    return b


def main():

    #--------------------------------------------------------------------------
    # This sections works as expected

    print "-" * 80
    print "Works as expected:"

    b0 = zeros(3)

    print "    b0 = ", b0

    b1 = ones(3)

    print "    b1 = ", b1

    y = b0 << b1

    print "    b0 = ", b0
    print "    y  = ", y
    print "    b1 = ", b1

    print "    repr(b0) = ", repr(b0)
    print "    repr(y)  = ", repr(y)

    #--------------------------------------------------------------------------
    # Funny things are happening here!

    print "Funny business:"

    b2 = zeros(3) << ones(3)

    print "    repr(b2) = ", repr(b2)

    b3 = zeros(3) << 4.0

    print "    repr(b3) = ", repr(b3)


if __name__ == "__main__":
    main()

I've tried a few things with SWIG as outlined on my blog post but my SWIG-foo skills have turned up short.

Save me SO community, you're my only hope!

Update 1

I suspect I have multiple PyObjects holding Buffer * to the same C++ Buffer object, so when the temporary PyObject is garbage collected, it deletes the C++ Buffer * along with it.

So, I think I need a Py_INCREF somewhere, but where?

Update 2

Trying to return by value as suggested by jarod42 breaks the concatenation paradigm, for example:

b = Buffer()
b << 1 << 2 << 3
print b

only produces:

Buffer(1, )

So that's not what I want.

The %newobject directive can be used to free a newly created object (preventing a memory leak) that a function or method creates. In this case, Buffer::operator<< isn't creating a new object.

Answer 1

After some more searching I came across this thread which eventually lead to a working solution.

Using a typemap(out) in combination with Py_INCREF did the trick.

swig_test.i:

%module swig_test

%include "std_string.i"

%{
    #include "Buffer.hpp"
    #include <iostream>
%}

%ignore Buffer::operator=;

%typemap(out) Buffer & operator<<
{
    if(result) { /* suppress unused warning */ }
    Py_INCREF($self);
    $result = $self;
}

%include "Buffer.hpp"

Now I get the behavior I want (which matches the pure Python implementation that works) and there are no memory leaks.