Migrating paperclip S3 images to new url/path format

Question

Is there a recommended technique for migrating a large set of paperclip S3 images to a new :url and :path format?

The reason for this is because after upgrading to rails 3.1, new versions of thumbs are not being shown after cropping (previously cached version is shown). This is because the filename no longer changes (since asset_timestamp was removed in rails 3.1). I'm using :fingerprint in the url/path format, but this is generated from the original, which doesn't change when cropping.

I was intending to insert :updated_at in the url/path format, and update attachment.updated_at during cropping, but after implementing that change all existing images would need to be moved to their new location. That's around half a million images to rename over S3.

At this point I'm considering copying them to their new location first, then deploying the code change, then moving any images which were missed (ie uploaded after the copy), but I'm hoping there's an easier way... any suggestions?

Answer 1

I had to change my paperclip path in order to support image cropping, I ended up creating a rake task to help out.

namespace :paperclip_migration do

  desc 'Migrate data'
  task :migrate_s3 => :environment do
    # Make sure that all of the models have been loaded so any attachments are registered
    puts 'Loading models...'
    Dir[Rails.root.join('app', 'models', '**/*')].each { |file| File.basename(file, '.rb').camelize.constantize }

    # Iterate through all of the registered attachments
    puts 'Migrating attachments...'
    attachment_registry.each_definition do |klass, name, options|
      puts "Migrating #{klass}: #{name}"
      klass.find_each(batch_size: 100) do |instance|
        attachment = instance.send(name)

        unless attachment.blank?
          attachment.styles.each do |style_name, style|
            old_path = interpolator.interpolate(old_path_option, attachment, style_name)
            new_path = interpolator.interpolate(new_path_option, attachment, style_name)
            # puts "#{style_name}:\n\told: #{old_path}\n\tnew: #{new_path}"
            s3_copy(s3_bucket, old_path, new_path)
          end
        end
      end
    end

    puts 'Completed migration.'
  end

  #############################################################################
  private

  # Paperclip Configuration
  def attachment_registry
    Paperclip::AttachmentRegistry
  end

  def s3_bucket
    ENV['S3_BUCKET']
  end

  def old_path_option
    ':class/:id_partition/:attachment/:hash.:extension'
  end

  def new_path_option
    ':class/:attachment/:id_partition/:style/:filename'
  end

  def interpolator
    Paperclip::Interpolations
  end

  # S3
  def s3
    AWS::S3.new(access_key_id: ENV['S3_KEY'], secret_access_key: ENV['S3_SECRET'])
  end

  def s3_copy(bucket, source, destination)
    source_object = s3.buckets[bucket].objects[source]
    destination_object = source_object.copy_to(destination, {metadata: source_object.metadata.to_h})
    destination_object.acl = source_object.acl
    puts "Copied #{source}"
  rescue Exception => e
    puts "*Unable to copy #{source} - #{e.message}"
  end

end

Answer 2

Didn't find a feasible method for migrating to a new url format. I ended up overriding Paperclip::Attachment#generate_fingerprint so it appends :updated_at.