Google App Engine: task_retry_limit doesn't work?

Question

I have a Python GAE app.

I want my tasks to stop running or just retry once if they fail. Right now, they run forever despite what my yaml file is telling them!

Here is a queue.yaml entry:

 - name: globalPurchase
   rate: 10/s
   bucket_size: 100
   retry_parameters:
     task_retry_limit: 1

If globalPurchase task fails with a 500 error code, it is retried forever until it succeeds with this message in the logs:

"Task named "task14" on queue "globalPurchase" failed with code 500; will retry in 30 seconds"

Why is task_retry_limit not actually being used?

Answer 1

I had the same problem. The documentation and tooling in this area is lacking, but here's what I found:

• The retry parameters have no effect in the development server. I tried lots of different combinations, but it was always just indefinite retries 30s apart. The parameters did take effect when I deployed to the production server.
• I haven't found a way to disable all retries (other than by ensuring that my handler doesn't throw exceptions).
  • If task_retry_limit=0, then it still retries.
  • If task_retry_limit=0 and task_age_limit is set, then the queue.yaml is rejected with a message that task_retry_limit must be positive.
  • Similarly, it complains if task_age_limit=0.
  • If you set task_retry_limit=1 and task_age_limit=1s (apparently the minimum values), you still get one retry.
• The minimum retry time is 20 seconds. If the delay specified by your configuration is les than 20, than it will just wait 20 seconds.
• The time before the first retry is unpredictable; it seems to be randomly delayed by up to a minute. After that, the retries follow the configured schedule.

Answer 2

I had the same problem, strangely it seems to start working fine after I left it as is for few hours... Perhaps there is some time needed for GAE to refresh??

Anyway, the settings worked for me are:

# configure the default queue
- name: default
  rate: 1/s
  retry_parameters:
    # task will stop retrying ONLY when BOTH LIMITS ARE REACHED
    task_retry_limit: 1
    task_age_limit: 1s