How to run own daemon processes with Django?

Question

In my Django project I have to do repeatedly some processing in the background. This processing needs access to Django stuff, so I put it into Django's commands and run it as cronjob. Right now I realize, that I have to do some of them more frequently (cronjob has limitation to invoke command at most every 1 minute). Another problem is that I don't have enough control, to protect running the same command in one time. It's happen when one processing takes longer than one minute. I think that I should run them like daemons, but I am looking for pure way to do it with Django. Have you ever faced with this problem or know any clean solution for it?

Answer 1

We do a lot of background processing for django using Celery http://celeryproject.org/. It requires some effort to set up and there is a bit of a learning curve, but once it's up and running it's just awesome.

Answer 2

We took more simple approach - write the script as normal script with endless loop that iterate through a queryset and then use supervise to manage it as a daemon. Basically, this is all needed to have the daemon running:-

$ sudo apt-get install daemontools daemontools-run
$ mkdir /etc/service/sendmsevad
$ echo -> /etc/service/sendmsevad/run
#!/bin/bash
exec /usr/local/bin/sendmsgd
$ sudo svc -d  /etc/service/sendmsgd
$ sudo svc -u  /etc/service/sendmsgd
$ sudo svstat /etc/service/sendmsgd
/etc/service/sendmsg: up (pid 10521) 479 seconds

More about this - How do I daemonize an arbitrary script in unix?

Now, /usr/local/bin/sendmsgd may look like:-

def main(args=None):
    while True:
        process_messages()
        time.sleep(10)

if __name__ == '__main__':
    import signal
    def signal_handler(signal, frame):
        sys.exit(0)
    signal.signal(signal.SIGINT, signal_handler)

    main(sys.argv)