Rails background image upload causing application timeout

Question

Unfortunately the bounty was awarded to an answer that does not solve this problem, for those with similar issues.

I have a form with an image upload (heroku to s3). When I submit the form, my rails server waits for the background job that uploads the image to finish before returning a response to the user. This causes an application timeout every single time there is an image upload.

Current order of events:

1. User submits form
2. Server receives form
3. If there is an image, server starts a background job
4. If a background job was started, the server waits for it to complete (rails times out here)
5. If started, the background job completes
6. The server processes the request
7. The server responds to the user

Desired order of events:

1. User submits form
2. Server receives form
3. Server processes non-image fields
4. If there is an image, server starts a background job
5. The server responds to the user
6. The background job completes and the server processes the uploaded image (saves URL)

Uploader code

class PhotoUploader < CarrierWave::Uploader::Base
  include ::CarrierWave::Backgrounder::Delay
  include CarrierWave::MimeTypes
  process :set_content_type
  storage :fog
end

Carrierwave::Backgrounder initializer

CarrierWave::Backgrounder.configure do |c|
  c.backend :sidekiq, queue: :carrierwave
end

User model

class User < ActiveRecord::Base
  mount_uploader :photo, PhotoUploader, delayed: true
  process_in_background :photo
end

There is no controller code because the form is handled by ActiveAdmin. I can override wherever is needed, but have not been able to figure out what needs to change.

What do I have to change to get the correct order of events?

Answer 1

The underlying issue here is Heroku has strict limits on how long a request may block without sending data back to the client. If you hit that limit (30 seconds for the initial byte), Heroku will timeout your request. For file uploads, it is very likely you will hit this limit.

The best approach is to have the user's browser directly upload the file to S3 first. There is some discussion here that is relevant to this: Direct Uploads to S3 using Carrierwave

If you use something like the jQuery File Upload plugin (https://github.com/blueimp/jQuery-File-Upload), the flow would be something like:

• User adds one or more files to the form before clicking submit.
• The files are uploaded directly to S3 and a file upload token is added for each file to your form.
• User submits form with the tokens of the uploaded files rather than the contents of the files.
• The server can move the files to their real home in S3 based on the submitted token.

This allows your web server to focus on just serving the request and not block on file uploads which can take a long time.

This requires a bit more work due to Heroku's limits - but ultimately I think is your only option to avoid the timeout limits.

Also, I'd recommend you create an upload S3 bucket and then set an S3 lifecycle policy to purge files older than some interval. When you are doing direct file uploads, it is common for some uploads to not be processed due to the user giving up etc, so the lifecycle does the job of cleaning up these files.