why boost::python iterator skips first element?

Question

I have encountered a strange problem when I have attempted to implement iterable object for Python in C++ (using boost::python). Python seems to always dereference one element ahead, so, in result it skips first element and also dereferences "end" element. I am also not confident if my return value policy is selected properly, but it is the only one that seems to work correctly if I replace int with std::string as an element type. The iterator tag has been chosen purposely - I intend to implement iterable object to access resource that can be traversed only once.

C++ code:

#include <Python.h>
#include <boost/python.hpp>

#include <iostream>
#include <iterator>

int nextInstance{0};

class Foo
{
public:
    class iterator : public std::iterator<std::input_iterator_tag, int>
    {
    public:
        iterator() = delete;
        iterator& operator=(const iterator&) = delete;

        iterator(const iterator& other)
        :
            instance_(nextInstance++),
            pos_(other.pos_)
        {
            std::cout << instance_ << " copy ctor " << other.instance_ << " (" << pos_ << ")\n";
        }

        explicit iterator(int pos)
        :
            instance_(nextInstance++),
            pos_(pos)
        {
            std::cout << instance_ << " ctor (" << pos_ << ")\n";
        }

        bool operator==(iterator& other)
        {
            std::cout << instance_ << " operator== " << other.instance_ << " (" << pos_ << ", " << other.pos_ << ")\n";

            return pos_ == other.pos_;
        }

        int& operator*()
        {
            std::cout << instance_ << " operator* (" << pos_ << ")\n";

            return pos_;
        }

        iterator operator++(int)
        {
            ++pos_;

            std::cout << instance_ << " operator++ (" << pos_ << ")\n";

            return *this;
        }

        ~iterator()
        {
            std::cout << instance_ << " dtor\n";
        }

    private:
        const int instance_;
        int       pos_{0};

    };

    iterator begin()
    {
        std::cout << "begin()\n";

        return iterator(0);
    }

    iterator end()
    {
        std::cout << "end()\n";

        return iterator(3);
    }
};

BOOST_PYTHON_MODULE(pythonIterator)
{
    boost::python::class_<Foo, boost::noncopyable>("Foo", boost::python::init<>())
        .def("__iter__", boost::python::iterator<Foo, boost::python::return_value_policy<boost::python::copy_non_const_reference>>{});
}

Python code:

#!/usr/bin/python

import pythonIterator

foo = pythonIterator.Foo()

for i in foo:
    print i

Output:

end()
0 ctor (3)
begin()
1 ctor (0)
2 copy ctor 1 (0)
3 copy ctor 0 (3)
1 dtor
0 dtor
4 copy ctor 2 (0)
5 copy ctor 3 (3)
3 dtor
2 dtor
4 operator== 5 (0, 3)
4 operator++ (1)
6 copy ctor 4 (1)
6 operator* (1)
6 dtor
1
4 operator== 5 (1, 3)
4 operator++ (2)
7 copy ctor 4 (2)
7 operator* (2)
7 dtor
2
4 operator== 5 (2, 3)
4 operator++ (3)
8 copy ctor 4 (3)
8 operator* (3)
8 dtor
3
4 operator== 5 (3, 3)
5 dtor
4 dtor

Answer 1

You have a bug in your post-increment operator. Specifically, what you implemented is pre-increment, not post-increment:

iterator operator++(int)
{
    ++pos_;
    return *this;  // return value *after* having incremented it
}

The correct implementation would be:

iterator operator++(int)
{
    iterator tmp(*this);
    ++pos_;
    return tmp; // return saved tmp *before* having incremented it
}

After that fix:

>>> list(pythonIterator.Foo())
... snip lots of output ...
[0, 1, 2]

Answer 2

Oh wow. Thanks for finally showing me the first self-contained Boost Python example.

So, let me repay you the service by suggesting to use Boost Iterator to handle the iterator complexity for you:

Live On Coliru

#include <Python.h>
#include <boost/python.hpp>
#include <boost/iterator/iterator_facade.hpp>

class Foo
{
public:
    struct iterator : boost::iterator_facade<iterator, int, boost::single_pass_traversal_tag, int>
    {
        iterator(int i) : current_(i) {}

        bool equal(iterator const& other) const { return current_ == other.current_; }
        int dereference() const { return current_; }
        void increment() { ++current_; }
    private:
        int current_;
    };

    iterator begin() { return 0; }
    iterator end()   { return 3; }
};

BOOST_PYTHON_MODULE(pythonIterator)
{
    boost::python::class_<Foo, boost::noncopyable>("Foo", boost::python::init<>())
        .def("__iter__", boost::python::iterator<Foo, boost::python::return_value_policy<boost::python::return_by_value>>{});
}

Prints:

$ ./test.py 
0
1
2

Of course, the choice to make the iterator return copies was inspired by the absense of a source range. (Obviously iterator_facade is fully geared towards lvalue-refs if you need them)