Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Advice on Python/Django and message queues [closed]

I have an application in Django, that needs to send a large number of emails to users in various use cases. I don't want to handle this synchronously within the application for obvious reasons.

Has anyone any recommendations for a message queuing server which integrates well with Python, or they have used on a Django project? The rest of my stack is Apache, mod_python, MySQL.

like image 204
Andy Hume Avatar asked Jan 18 '09 10:01

Andy Hume


People also ask

Does message queue decrease performance?

Message queues can significantly simplify coding of decoupled applications, while improving performance, reliability and scalability.

Does message queue increases the complexity of a system?

Implementing messages queues One thing to note is that decoupling of systems with message queues increases the complexity of the system's architecture, so that's one trade-off you will need to consider.

What are the ways in which message queue can be implemented?

Using Shared Memory or Message Queues depends on the need of the application and how effectively it can be utilized. Writing into the shared memory by one process and reading from the shared memory by another process. As we are aware, reading can be done with multiple processes as well.

Is Message Queue real time?

Queues are real time in nature If your subscribers pick messages off the queue at the rate they are added, then the additional latency added should be in the low milliseconds. In practical terms, a queue does not add latency.


2 Answers

In your specific case, where it's just an email queue, I wold take the easy way out and use django-mailer. As a nice side bonues there are other pluggable projects that are smart enough to take advantage of django-mailer when they see it in the stack.

As for more general queue solutions, I haven't been able to try any of these yet, but here's a list of ones that look more interesting to me:

  1. pybeanstalk/beanstalkd
  2. python interface to gearman (which is probably much more interesting now with the release of the C version of gearman)
  3. memcacheQ
  4. stomp
  5. Celery
like image 132
Van Gale Avatar answered Sep 21 '22 19:09

Van Gale


So far I have found no "nice" solution for this. I have some more strict soft realtime requirements (taking a picture from a cardboard box being labeled) so probably one of the approaches is fast enough for you. I assume emails can wait for a few minutes.

  • A "todo list" in the database processed by a cron job.
  • A "todo list" in the database processed permanently beeing polled by a daemon.
  • Using a custom daemon which gets notified by the webserver via an UDP packet (in Production today). Basically my own Queing system with the IP stack for handling the queue.
  • Using ActiveMQ as a message broker - this didn't work out because of stability issues. Also to me Java Daemons are generally somewhat plump
  • Using Update Triggers in CouchDB. Nice but Update Triggers are not meant to do heavy image processing, so no good fit for my problem.

So far I haven't tried RabbitMQ and XMPP/ejabebrd for handling the problem but they are on my list of next things to try. RabbitMQ got decent Python connectivity during 2008 and there are tons of XMPP libraries.

But perhaps all you need is a correctly configured mailserver on the local machine. This probably would allow you to dump mails synchronously into the local mailserver and thus make your whole software stack much more simple.

like image 41
max Avatar answered Sep 24 '22 19:09

max