celery vs pyro : is Pyro an alternative to Celery?

Question

I am trying to learn about Celery and was wondering if Celery and Pyro are trying to achieve the same thing ?

Could somebody please tell me if there is something which Celery can do which Pyro can not, or vice versa?

Answer 1

As I see in the official websites, Celery and Pyro, are intent to do different jobs but the confusion is pretty natural.

The objective in both of the packages is help you with distributed computing but with different approaches: Celery is intent to be a distributed task scheduler, it means, if you have a bunch of tasks (very uncorrelated) you can distribute them over a computer grid or over the network.

While, Pyro aims to establish a communication gateway between object over the network, it means, if you have a pretty big task, that you can't divide in little uncorrelated tasks, but with a bunch of objects, that are independent but usually need information about the others, then Pyro enables the communication between them, so you can perform the task distributing the objects in a computer grid or over the network.

You post this with the Django tag, so it will be relevant for you to say, that the requests that are performed to a web application can be seen as a bunch (a big one as the concurrency increases) of uncorrelated tasks, so Celery might be what you are looking for.

Answer 2

The answer above explains the differences between Pyro and Celery.

But in light of all the other changes that have happened over the years wrt to Python and the availability of Python ZeroMQ libraries and function picking, it might be worth taking a look at leveraging ZeroMQ and PiCloud's function pickling. This creates a whole new way to build distributed stacks.

See link sample code on jeffknupp.com blog

Yes, of course you can stick to Celery to develop distributed workers of tasks. And with Pyro, you can develop remote-procedure call applications. With Celery and Pyro, you are doing all of this in the Python world whereas with ZeroMQ they have implementations in a dozen different languages and it implements the common patterns for networking like PUB-SUB,REQ-RES,PIPES, etc. This opens up the possibility of creating language agnostic possibilities.