Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is it possible to use celery for synchronous tasks?

Nearly synchronous works, too; basically, I want to delegate the data access and processing behind a web app to a task queue for most jobs. What's the fastest latency that I can consider reasonable for celery tasks?

Update (for clarification)

I guess for clarity I should explain that throughput -- while nice -- is not a necessary issue for me; I won't be needing scaling in that direction for a while, yet. Latency is the only criterion that I'll be evaluating at the moment. I'm content to use task.apply if that's the only way it'll work, but I'd like to farm the work out a bit.

like image 998
Chris R Avatar asked Nov 19 '10 06:11

Chris R


1 Answers

When I say throughput I mean the average latency from sending a task until it's been executed. With roundtrip I mean the average time it takes to send a task, executing it, sending the result back and retrieving the result.

As I said in the comments I currently don't have any official numbers to share, but with the right configuration Celery is low latency compared to many other solutions, but still it does come with more overhead than executing a function locally. This is something to take into account when designing the granularity of a task[1]

I'm currently writing a performance guide that may be of interest: http://ask.github.com/celery/userguide/optimizing.html

Feedback welcome, and would like to know about any other performance factors you are interested in.

[1] http://celeryq.org/docs/userguide/tasks.html#granularity

like image 116
asksol Avatar answered Nov 15 '22 13:11

asksol