Uploading multiple files directly to Amazon S3 using Rails 3.2 and AJAX (non-flash upload solutions)

Question

This issue has been bothering me for many hours and I can't seem to find a solution to it.

I have a rails 3.2 app that allows users to upload files to an Amazon S3 account using carrierwave_direct, fog, and carrierwave (dependency for carrierwave_direct). Using carrierwave_direct allows the user to skip uploading the file to the server by POSTing it directly to Amazon S3 (saves server processing and timeouts like Heroku for large files).

It works fine if all you do is select 1 file, upload it to Amazon, and want a redirect_to a URL you provide Amazon. It does this by POSTing the form to Amazon S3, and Amazon responds to a provided URL (you specify this URL in your form) with some params in the URL, which are then stored as a pointer to the file on Amazon in your model.

So the lifecycle is: select 1 file, POST to Amazon, Amazon responds with a URL that sends you to another page, and you can then save a record with a pointer to the Amazon file.

What I've been trying to figure out is how do I allow multiple files to be selected and uploaded and update the upload progress? I'm trying to do this with pure javascript (using the file API provided by modern browsers) so I don't want any 3rd party tools. Also, in the pursuit of learning this in-depth, I'm avoiding any plugins and am trying to write the code myself.

The functionality I'm trying to obtain is:

1. User sees form with file field (or drag/drop)
2. User selects multiple files (either click file field or drag/drop)
3. Using Javascript (no servers yet), build a queue of selected files to upload (just filename and size, using browser File API)
4. User then clicks a "begin Upload" button
5. Iterate over each file in queue and POST the file to Amazon S3; Amazon will respond to each individual POST with a URL and that URL needs to be handled via Javascript, not as a standard request; the URL provided by Amazon will create a record that stores the pointer to the Amazon file; once the record has been created, the code goes to the next file in the queue until finished.

At this point, I could even do without an individual progress bar; I'd be happy just to get multiple files POSTed to Amazon S3 without page refreshes.

I am not partial to any of the gems. I'm actually afraid I'm going to have to write what I want to do from scratch if I really want it done in a specific way. The goal is multiple file uploads to an Amazon S3 account via AJAX. I would be ecstatic with even general concepts of how to approach the problem. I've spent many hours googling this and I just haven't found any solutions that do what I want. Any help at all would be greatly appreciated.

EDIT 2014-03-02

Raj asked how I implemented my multiple upload. It's been so long I don't recall all the "why" behind what I did (probably bad code anyway as it was my first time), but here is what I had going on.

The model I was uploading was a Testimonial, which has an associated image being stored in Amazon S3. It allowed a user to select multiple images (I think they were actually PDF files I converted to images) and drag/drop them onto the screen. While uploading, I displayed a modal that gave the user feedback about how long it would take.

I don't pretend to remember what I was doing on a lot of this, but if it helps feel free to use it.

# Gemfile
# For client-side multiple uploads
gem "jquery-fileupload-rails"

# For file uploads and Amazon S3 storage
gem "rmagick"
gem "carrierwave"
gem "fog"

Here's the view:

# app/views/testimonials/new.html.erb
<div id="main" class="padded">
  <div class="center">
    <div id="dropzone">
      Click or Drop Files here to Upload
    </div>

    <%= form_for @testimonial do |f| %>
      <div class="field">
        <%= file_field_tag :image, multiple: true, name: "testimonial[image]", id: "testimonial_image" %>
      </div>
    <% end %>
  </div>
</div>

<div id="mask"></div>
<div id="modal">
  <h1>
    Uploading <span id="global-upload-count">0</span> Files...
  </h1>
  <div id="global-progress">
    <div id="global-progress-bar" style="width: 0%">
      <div id="global-progress-percentage">0%</div>
    </div>
  </div>
  <div id="global-processing">
    <span class="spinner"></span> Processing...<span id="global-processing-count">0</span> sec
  </div>
</div>

<script id="template-upload" type="text/x-tmpl">
  <div class="upload">
    {%=o.name%} ({%=o.readable_size%})
    <div class="float-right percentage"></div>
    <div class="progress"><div class="bar" style="width: 0%"></div></div>
  </div>
</script>

And the JS:

number_to_human_size = (bytes) ->
  sizes = ['Bytes', 'KB', 'MB', 'GB', 'TB']
  i = parseInt(Math.floor(Math.log(bytes) / Math.log(1024)))
  return Math.round(bytes / Math.pow(1024, i), 2) + ' ' + sizes[i]

dropzone_hover = (e) ->
  e.preventDefault()
  $(this).addClass("dropzone-hover")

dropzone_leave = (e) ->
  e.preventDefault()
  $(this).removeClass("dropzone-hover")

jQuery ->
  global_count = 0
  seconds_to_process = 0
  processing_factor = 5 # seconds to convert/process each uploaded file

  $("#testimonial_image").hide()

  dropzone = $("#dropzone")

  dropzone.bind "click", (e) ->
    $("#testimonial_image").click()

  dropzone.bind("dragover", dropzone_hover)
  dropzone.bind("dragleave", dropzone_leave)
  dropzone.bind("drop", dropzone_leave)

  $("#new_testimonial").data("global-count", "0")

  $("#new_testimonial").fileupload
    dropZone: $("#dropzone")
    maxFileSize: 5000000 # 5 MB
    dataType: "script"

    add: (e, data) ->
      file = data.files[0]
      file.readable_size = number_to_human_size(file.size)
      data.context = $(tmpl("template-upload", file).trim())
      $("#new_testimonial").append(data.context)
      data.submit()
      global_count += 1

    progress: (e, data) ->
      if data.context
        progress = parseInt(data.loaded / data.total * 100, 10)
        data.context.find(".bar").css("width", progress + "%")
        data.context.find(".percentage").text(progress + "%")

    submit: (e, data) ->
      $("#mask").show()
      $("#modal").center().show()

    progressall: (e, data) ->
      $("#global-upload-count").text(global_count)
      global_progress = parseInt(data.loaded / data.total * 100, 10)
      $("#global-progress-bar").css("width", global_progress + "%")
      $("#global-progress-percentage").text(global_progress + "%")

      if global_progress >= 100
        seconds_to_process = global_count * processing_factor
        $("#global-processing-count").text(seconds_to_process)

        $("#global-processing").show()

        timer = setInterval(->
          seconds_to_process = seconds_to_process - 1
          $("#global-processing-count").text(seconds_to_process)

          if seconds_to_process == 0
            clearInterval(timer)
            global_count = 0
            seconds_to_process = 0
            $("#modal, #mask").hide(0)
        , 1000)

The Testimonial model:

class Testimonial < ActiveRecord::Base
  mount_uploader :image, ImageUploader

  def display_name
    if name.blank?
      return "Testimonial #{self.id}"
    else
      return name
    end
  end
end

Answer 1

As advised in comment, use jQuery Upload: http://blueimp.github.com/jQuery-File-Upload/