Django, Signals and another process

Question

I've got my Django project running well, and a separate background process which will collect data from various sources and store that data in an index.

I've got a model in a Django app called Sources which contains, essentially, a list of sources that data can come from! I've successfully managed to create a signal that is activated/called when a new entry is put in the Sources model.

My question is, is there a simple way that anybody knows of whereby I can send some form of signal/message to my background process indicating that the Sources model has been changed? Or should I just resort to polling for changes every x seconds, because it's so much simpler?

Many thanks for any help received.

Answer 1

It's unclear how are you running the background process you're talking about.

Anyway, I'd suggest that in your background task you use the Sources model directly. There are convenient ways to run the task without leaving realm of Django (so as to have an access to your models. You can use Celery [1], for example, or RQ [2].

Then you won't need to pass any messages, any changes to Sources model will take effect the next time your task is run.

[1] Celery is an open source asynchronous task queue/job queue, it isn't hard to set up and integrates with Django well.

• Celery: general introduction
• Django with celery introduction

[2] RQ means "Redis Queue", it is ‘a simple Python library for queueing jobs and processing them in the background with workers’.

• Introductory post
• GitHub repository

Answer 2

Polling is probably the easiest if you don't need split-second latency.

If you do, however, then you'll probably want to look into either, say,

• sending an UNIX signal (or other methods of IPC, depending on platform) to the process
• having the background process have a simple listening socket that you just send, say, a byte to (which is, admittedly, a form of IPC), and that triggers the action you want to trigger
• or some sort of task/message queue. Celery or ZeroMQ come to mind.