Nearly synchronous works, too; basically, I want to delegate the data access and processing behind a web app to a task queue for most jobs. What's the fastest latency that I can consider reasonable for celery tasks?
I guess for clarity I should explain that throughput -- while nice -- is not a necessary issue for me; I won't be needing scaling in that direction for a while, yet. Latency is the only criterion that I'll be evaluating at the moment. I'm content to use task.apply
if that's the only way it'll work, but I'd like to farm the work out a bit.
When I say throughput
I mean the average latency from sending a task until it's been executed. With roundtrip
I mean the average time it takes to send a task, executing it, sending the result back and retrieving the result.
As I said in the comments I currently don't have any official numbers to share, but with the right configuration Celery is low latency compared to many other solutions, but still it does come with more overhead than executing a function locally. This is something to take into account when designing the granularity of a task[1]
I'm currently writing a performance guide that may be of interest: http://ask.github.com/celery/userguide/optimizing.html
Feedback welcome, and would like to know about any other performance factors you are interested in.
[1] http://celeryq.org/docs/userguide/tasks.html#granularity
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With