Python web framework + [a]sync tasks; long polling: not Django

Question

I need to create a project that has a web frontend to manage synchronous task execution (ala fabric), async tasks (AMQP), and long-polling/ajax for tabular viewing of results and queues/large, frequently changing datasets (think tail -f syslog). I have an existing Python codebase for a lot of the implementation-specific stuff.

After looking at a bunch of existing frameworks, the obvious answer appears to be Django+Celery. However, I do not want to "learn Django", nor do I need 95% of it's functionality. I just need simple auth, maybe sqlalchemy, easy ajax, amqp, xmlrpc would be helpful.

I would consider using Mongrel2, but I have a strong preference for RabbitMQ over 0MQ (for a few implementation-specific reasons).

I originally spent a great deal of time learning Twisted, and ended up getting a few hundred useful LOC out of it, but I found that I was twisting (lol) too much of my platform code to fit it's callback model. It actually 'fit the bill' very well (except with it's own amqp implementation), but it was so frustrating, and I went through so many iterations of code (one for each 'twisted ahah moment'), that it's 100% out.

Can somebody please help me wade through the mire? Tornado? Pylons? repoze? Pyramid? Flask? Bottle? CherryPy? Web2py? Paster/Webob? Anything else@ http://wiki.python.org/moin/WebFrameworks?

Edit:

To be clear, integration with RabbitMQ (or another amqp provider) is of the utmost importance, and is really the crux of problem.

Answer 1

I don't have a full vision of python web frameworks but just want to share my point of view on 2 of them :

• Bottle is light and works fine. If you want something easy to learn and easy to use that may be the right choice. I used it for quite simple front-end apps running locally and i liked it very much.

• Tornado seems to me as a very good non-blocking server for real-time web app. Combined with tornadio it makes ajax-long-polling quite easy. However, it may be a little harder to learn than Bottle. I would recommend to have a look to the chat app in the example folder of tornadio.

I hope it helps

Answer 2

If you are going to use AMQP long term then I would steer clear of Celery because they use AMQP in a wierd way that suggests the developers did not understand the AMQP model.

bottle is a nice framework for knocking together RESTful apps (I use it to create mock servers for testing) and if you already have the code that does the real work, you may be surprised at how short a bottle app can be.

I'm currently building Python apps using RabbitMQ and using amqplib by way of kombu. I originally chose kombu in case I wanted to swap libraries and use pika or something else, but now I wish that I had just gone with amqplib and built a proper Pythonic AMQP model on top of that.

Do spend some time on the RabbitMQ site reading some of the blogs and slide presentations on AMQP before you get too deep into coding or you won't really understand the AMQP model and will make things harder for yourself.

Please don't use xmlrpc unless you have to talk to other apps. Bottle makes simple RESTful apps so simple, that XMLRPC is just uneccessary complexity.