Uploading Large files to AWS S3 Bucket with Django on Heroku without 30s request timeout

Question

I have a django app that allows users to upload videos. Its hosted on Heroku and the uploaded files stored on an S3 Bucket. I am using JavaScript to directly upload the files to S3 after obtaining a presigned request from Django app. This is due to Heroku 30s request timeout. Is there anyway that i can possibly upload large files through Django backend without using JavaScript and compromising the user experience?

Answer 1

You should consider some of the points below for the solution of your problem.

• Why your files should not come to your django-server then go to s3: Sending files to django server then sending them to s3 is just a waste of computational power and bandwidth both. Next con would be that why send files to django server when you can directly send them to your s3 storage.
• How can you upload files to s3 without compromising UX: Sending files to django server is certainly not an option so you have to handle this on your frontend side. But front end side has its own limitation like limited memory. It won't be able to handle very large file because everything gets loaded into RAM and browser will eventually run out of memory if its a very large file. I would suggest that you use something like dropzone.js. It won't solve the problem of memory but it certainly can provide good UX to the user like showing progress bars, number of files etc.

Answer 2

The points in the other answer are valid. The short answer to the question of "Is there anyway that i can possibly upload large files through Django backend without using JavaScript" is "not without switching away from Heroku".

Keep in mind that any data transmitted to your dynos goes through Heroku's routing mesh, which is what enforces the 30 second request limit to conserve its own finite resources. Long-running transactions of any kind use up bandwidth/compute/etc that could be used to serve other requests, so Heroku applies the limit to help keep things moving across the thousands of dynos. When uploading a file, you will first be constrained by client bandwidth to your server. Then, you will be constrained by the bandwidth between your dynos and S3, on top of any processing your dyno actually does.

The larger the file, the more likely it will be that transmitting the data will exceed the 30 second timeout, particularly in step 1 for clients on unreliable networks. Creating a direct path from client to S3 is a reasonable compromise.