why does Contextmanager throws a runtime error 'generator didn't stop after throw()'?

Question

In my utility.py I have,

@contextmanager
def rate_limit_protection(max_tries=3, wait=300):
    tries = 0
    while max_tries > tries:
        try:
            yield
            break
        except FacebookRequestError as e:
            pprint.pprint(e)
            if e._body['error']['message'] == '(#17) User request limit reached':
                print("waiting...")
                time.sleep(wait)
                tries += 1

In my task.py I call:

for date in interval:
   with utility.rate_limit_protection():
      stats = account.get_insights(params=params)

After runing the task for a given date range, once Facebook rate limit kicks in, the program waits for 300 seconds after which it fails with the error.

File "/Users/kamal/.pyenv/versions/3.4.0/lib/python3.4/contextlib.py", line 78, in __exit__
    raise RuntimeError("generator didn't stop")
RuntimeError: generator didn't stop

Answer 1

The with statement is not a looping construct. It cannot be used to execute code repeatedly. A context manager created with @contextmanager should only yield once.

A context manager does (basically) three things:

1. It runs some code before a code block.
2. It runs some code after a code block.
3. Optionally, it suppresses exceptions raised within a code block.

If you want to do something like this, you need to rewrite it so that the loop is moved outside the context manager, or so that there is no context manager at all.

One option would be to write a function that accepts a callback as an argument, and then calls the callback in a loop like the one you currently have in your context manager:

def do_rate_protection(callback, max_tries=3):
    tries = 0
    while max_tries > tries:
        try:
            callback()
            break
        except FacebookRequestError as e:
            # etc.

You can then call it like this:

for date in interval:
    def callback():
        # code
    do_rate_protection(callback)

If the callback doesn't need the date variable, you can move it outside the loop to avoid repeatedly recreating the same function (which is wasteful of resources). You could also make date a parameter of the callback() function and pass it using functools.partial.