Uploading & Unzipping files to S3 through Rails hosted on Heroku?

Question

I'd like to be able to upload a zip file to my Rails application that contains a number of images. Then I'd like Rails to unzip that file and attach the images inside to my Photo's model via Paperclip, so that they are ultimately stored on my Amazon S3 account (configured through Paperclip).

I'd like do do this all on my Rails site hosted on Heroku, which unfortunately doesn't allow local storage of any kind (so far as I'm aware) to temporarily do the unzipping before the Paperclip parsing.

How would I do this??

Answer 1

I would recommend uploading directly to S3 which bypasses Heroku entirely so you're not restricted to the 30 second request timeout they enforce (which drops your uploads after that time is hit) or the 1gb /tmp directory limit. After the file is uploaded, you can make a POST to your Rails app with the file's name and location and then do your unzipping operation. If you'd like to use Paperclip for post-processing, I have attached a link below. If you end up going the route of uploading directly to S3 which offloads the work from your Rails server, please check out my sample projects:

Sample project using Rails 3, Flash and MooTools-based FancyUploader to upload directly to S3: https://github.com/iwasrobbed/Rails3-S3-Uploader-FancyUploader

Sample project using Rails 3, Flash/Silverlight/GoogleGears/BrowserPlus and jQuery-based Plupload to upload directly to S3: https://github.com/iwasrobbed/Rails3-S3-Uploader-Plupload

Here is the link for the Paperclip post processing for an example like images:

http://www.railstoolkit.com/posts/fancyupload-amazon-s3-uploader-with-paperclip

Answer 2

dmagkic is correct about the rails_root/tmp. I recommend something like the following:

• Upload files through heroku to S3
• Setup a background job to zip the files (store the file names that you need to group)
• run the BJ that downloads the files from S3, zips them, sends the zip to S3, removes the unzipped files.

That way your application will still be responsive'ish during the upload process.

If you try to upload multiple files, you COULD write to /tmp, but just make sure that all the files come across in the same post request.

Answer 3

Heroku does allow writing to #{RAILS_ROOT}/tmp.

But you need to take in mind that file will be there only as long as request lasts. Probably longer, but that is not guaranteed. You could try to block request while you unzip and send to S3, but you should take care of the time it takes.

It sounds to me like you need some flash uploader that can unzip and send to S3, without Heroku.