Google App Engine - Use Task Queues or Deferred Jobs

Question

Google App Engine has two methods for running jobs at some later point, Task Queues and Deferred jops

They support all the same features as far as I can tell (e.g. a deferred job can be placed on a particular task queue so you can throttle execution) - but the deferred jobs look much easier to implement and more flexible.

Anyone know of the pro's con's of each method? Any circumstances where you would want to use task queues over deferred jobs?

Answer 1

I'm not sure if you noticed it, but the documentation for deferreds has this section in the end:

You may be wondering when to use ext.deferred, and when to stick with the built-in task queue API. Here are our suggestions.

You may want to use the deferred library if:

• You only use the task queue lightly.
• You want to refactor existing code to run on the Task Queue with a minimum of changes.
• You're writing a one off maintenance task, such as schema migration.
• Your app has many different types of background task, and writing a separate handler for each would be burdensome.
• Your task requires complex arguments that aren't easily serialized without using Pickle.
• You are writing a library for other apps that needs to do background work.

You may want to use the Task Queue API if:

• You need complete control over how tasks are queued and executed.
• You need better queue management or monitoring than deferred provides.
• You have high throughput, and overhead is important.
• You are building larger abstractions and need direct control over tasks.
• You like the webhook model better than the RPC model.

Naturally, you can use both the Task Queue API and the deferred library side-by-side, if your app has requirements that fit into both groups.