Downloading dynamically generated large files in rails

Question

I have a big database with lots of rows, and users are generating queries on the database and then want to export the information (currently doing CSV). The problem is, as our database is growing, the queries take so long that the download attempts time out.

How can I set up Rails to download a file over time as it is created? These requests aren't common, so I don't mind a harsh hit to the server, but they do need to be dynamic (I can't generate the files in advance).

I found a lot of sites about how to download files in Rails, but they either deal with already-created files or smaller files that take no time to create. These can be very large (20MB+) files, so they need to "stream" the download, yet I can't find some way to do that.

Answer 1

I would suggest using a background worker service, like delayed job or sidekiq for rails,

Tell your users that their report will be ready shortly and schedule it to the background worker service, at the end of the report generation job initiate a notification to the user (websockets, emails, ...) and then allow the user to download the resulting file from whatever storage you are using - local, S3, etc

This has following benefits over streaming large responses:

1. 1 user from accounting that wants to make his quarterly reporting bundle of 20 huge reports will not freeze your web host for all traffic due to worker exhaustion so your CEO can still log in to see his numbers
2. you can LOWER the number of workers on the background service, so large complex queries will not overwhelm your database when the said user from point 1 comes to visit you
3. the serving of the files would be done by the web server in front of your service or S3 and not the worker process (rails can be slow)