Subscribe to a redis channel in Django project

Question

I have multiple applications written with nodejs or python/django or ... These services are working fine. But need to have pub/sub Async communication with each other.

In nodejs there is no problem and easily can pub/sub to any redis channel.

Question: My question is how can i continuously subscribe to a redis channel and receive data published with other services?

Note: many links suggest to use django-channels. But I guess that's not the way to do it. If so can any one help me and give details on how to do it.

Update: Django by default is not event-based like nodejs. So if i use a redis client, I should for example check redis every second and see if anything is published or not. I don't think using just a redis client in python will be enough.

Really appreciate it.

Answer 1

There are a lot of alternatives. If you have FIFO issue you have to use queues in order to connect one microservice to another. For me, if you don’t have Big Data problem you can use RabbitMQ, It is very practical and very effective, otherwise if you have Big Data problem you can use Kafka. There are wide variety services.

If you want just Pub/Sub. The best tool is Redis, It is very fast and easy to integrate. If you are concerned how to implement it in Python just look at article

[Update]

It's possible to create a manage.py command in django and subscribe to redis in that management file and execute this script separated from django server:

class Command(BaseCommand):
def handle(self, *args, **options):

    r = redis.StrictRedis(host='localhost', port=6379, db=1)
    p = r.pubsub()
    p.psubscribe('topic.*')
    for message in p.listen():
        if message:
            print('message received, do anything you want with it.')

Answer 2

In order to handle subscriptions to redis, you will need to have a separate continuously running process (server) which listens to redis and then makes something with your data. django-channels will do the same by running the code in a worker

As pointed above, Django provides convenient way to run "servers" by using Django management command approach. When running a django management command, you have complete access to your code, i.e. to ORM as well.

Only detail that you mentioned Async communication. Here you need to take into account that Django's ORM is strictly sync code, and you need to pay attention how you want to use ORM with async code. Probably you need to clarify what do you mean by async here.

As for redis messages processing, you can use any libraries that work with it. For example, aioredis or redis-py