Python Overflow Error: iter index too large

Question

I have a problem with big iterator in for loop in the code below. It generates floats by reading a string list containing numbers.

def float_generator(tekstowe):
        x = ''
        for c in tekstowe:
            if c != ' ':
                x += c
            else:
                out = float(x)
                x = ''
                yield(out)

I'm getting a "OverflowError: iter index too large". I try to use really big iter numbers (like billions of values in a searched file). Is iter range somehow limited in for loops?

Using Python 2.7 64 bit. Thanks.

Answer 1

Looks like tekstowe is a sequence type that only implements __getitem__, not __iter__, so it's using the Python iterator wrapper that calls __getitem__ with 0, then 1, 2, 3, etc., until __getitem__ raises IndexError.

As an implementation detail, Python 2.7.11 and higher limits the value of the index passed by the iterator wrapper to LONG_MAX (before 2.7.11, it wasn't bounds checked but it still used a long for index storage, so it would wrap and start indexing with negative values). This doesn't matter on most non-Windows 64 bit builds, where LONG_MAX is 2**63 - 1 (larger than you'd likely encounter), but on Windows, C longs remain 32 bit quantities even on 64 bit builds, so LONG_MAX remains 2**31 - 1, which is low enough to be reached in human timescales.

Your options are:

1. Change the implementation of whatever class tekstowe is to give it a true __iter__ method, so it doesn't get wrapped by the sequence iterator wrapper when you use it
2. Upgrade to Python 3.4+, ideally 3.5 (2.7.10/3.4.3 and below lacks the check for overflow entirely, but this could mean wraparound causes infinite looping; 3.4.4/3.5.0 added the check, and they use a signed size_t, testing against PY_SSIZE_T_MAX, which means it will not error until the index reaches 2**63 - 1 on any 64 bit build, Windows or otherwise)

The changes to add the overflow checks were made to resolve Python bug #22939; the type change (from long to Py_ssize_t) for the sequence iterator's index storage occurred in 3.4.0's release, resolving Python bug #17932.