inicio mail me! sindicaci;ón

Copying Files between S3 buckets

Building on this article here is a simple ruby script, that copies files between two buckets of the same S3 account, omitting files already present (by name).
This variant adds a list of path prefixes, so you can selectively copy only certain directories of your buckets.
Furthermore it copies the original buckets ACLs for each key.

require 'rubygems'
require 'right_aws'

aws_access_key_id     = 'YOUR AMAZON ACCESS KEY'
aws_secret_access_key = 'YOUR AMAZON SECRET ACCESS KEY'
source_bucket         = 'SOURCE BUCKET NAME'
target_bucket         = 'TARGET BUCKET NAME'
prefixes              = [PATH_PREFIX1, PATH_PREFIX2, ...]

s3 = RightAws::S3Interface.new(aws_access_key_id, aws_secret_access_key)

copied_keys = Array.new
(prefixes || ['']).each do |prefix|
  s3.incrementally_list_bucket(target_bucket, {:prefix => prefix}) do |key_set|
    copied_keys << key_set[:contents].map{|k| k[:key]}.flatten
  end
end
copied_keys.flatten!

(prefixes || ['']).each do |prefix|
  s3.incrementally_list_bucket(source_bucket, {:prefix => prefix}) do |key_set|
    key_set[:contents].each do |key|
      key = key[:key]
      if copied_keys.include?(key)
        puts "#{target_bucket} #{key} already exists. Skipping..."
      else

        puts "Copying #{source_bucket} #{key}, setting acl"

        retries=0
        begin
          s3.copy(source_bucket, key, target_bucket)
          acl = s3.get_acl(source_bucket, key)
          s3.put_acl(target_bucket, key, acl[:object])
        rescue Exception => e
          puts "cannot copy key, #{e.inspect}\nretrying #{retries} out of 10 times..."
          retries += 1
          retry if retries <= 10
        end
      end
    end
  end
end

Schreibe einen Kommentar