Menu
Part of an application I'm writing allows for users to upload images which I then resize and automatically upload to amazon s3. Currently the image resizing is happening right in the view and I'd like to offload this via celery to distributed workers. My question is whats the best way to get the image to the worker. My current thinking is to store the image directly in the database and then just pass the id to the worker and have it retrieve it. Is there a better practice then temporarily storing it in the database until it can get processed?
700
To pass arguments to task with apply_async() you need to wrap them in a list and then pass the list as first argument, I.e. apply_async([arg1, arg2, arg3]) . See the documentation for more details and examples. Use delay() as an alternative.
Bound tasks A task being bound means the first argument to the task will always be the task instance ( self ), just like Python bound methods: logger = get_task_logger(__name__) @app. task(bind=True) def add(self, x, y): logger.
Celery workers poll the queues and pulls available tasks. When a worker pulls a task, it removes it from the named queue and moves it to a special Redis unacked queue. The worker holds on to the task in memory, until it can process it.
Redis is the datastore and message broker between Celery and Django. In other words, Django and Celery use Redis to communicate with each other (instead of a SQL database). Redis can also be used as a cache as well. An alternative for Django & Celery is RabbitMQ (not covered here).
As stated in the celery docs, it is better not to pass the whole thing (image) as an argument, for this will cause extra overhead. So it is better to save it in first place, then pass the photo id as an argument, retrieve the image into the task and do the resizing / uploading.
145
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
