Django: File upload to s3 using boto

Question

I am trying to upload images (average size 12 MB) to my server (django using PIL)then I'm processing them to scale down images and creating thumbnails.

During this process I'm not saving images to server local disk instead, I'm using In Memory file and processing it. I'm uploading images to s3, using boto and I have original image, scaled image and the thumbnail which should be uploaded one by one (all these three are still present in my in Memory).

This process is very time consuming and sometimes the server which I am using (Digital Ocean 512 MB RAM) throws out of memory error.

Can I know is there any better way of uploading images

Any help is much appreciated. Thanks in advance.

Answer 1

I haven't worked a lot with django & boto but u can try few suggestions:

• Upgrade your server. if you are going to process that much big files, image processing requires a lot of memory.
• You can try implementing queues (i.e. whenever you will get any upload request and file is bigger then enqueue it and process after current process is complete). You can use celery for this work.
• If you don't need that much big files, then you can reduce image size at client side first and then pass it to the server.