Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do I get the file / key size in boto S3?

There must be an easy way to get the file size (key size) without pulling over a whole file. I can see it in the Properties of the AWS S3 browser. And I think I can get it off the "Content-length" header of a "HEAD" request. But I'm not connecting the dots about how to do this with boto. Extra kudos if you post a link to some more comprehensive examples than are in the standard boto docs.

EDIT: So the following seems to do the trick (though from looking at source code I'm not completely sure.):

bk = conn.get_bucket('my_bucket_name') ky = boto.s3.key.Key(bk) ky.open_read()  ## This sends a GET request.  print ky.size 

For now I'll leave the question open for comments, better solutions, or pointers to examples.

like image 750
mjhm Avatar asked Mar 15 '11 17:03

mjhm


People also ask

How do I find out the size of my S3?

Open the AWS S3 console and click on your bucket. Click on the Metrics tab. The Total bucket size graph in the Bucket Metrics section shows the total size of the objects in the bucket.

What is S3 file key?

Keys. An object key (or key name) is the unique identifier for an object within a bucket. Every object in a bucket has exactly one key. The combination of a bucket, object key, and optionally, version ID (if S3 Versioning is enabled for the bucket) uniquely identify each object.

What is the size of object in S3?

Individual Amazon S3 objects can range in size from a minimum of 0 bytes to a maximum of 5 TB. The largest object that can be uploaded in a single PUT is 5 GB. For objects larger than 100 MB, customers should consider using the Multipart Upload capability.

What is boto3 key?

In your code example - key is the object reference to the unique identifier within a bucket.


1 Answers

This would work:

bk = conn.get_bucket('my_bucket_name') key = bk.lookup('my_key_name') print key.size 

The lookup method simply does a HEAD request on the bucket for the keyname so it will return all of the headers (including content-length) for the key but will not transfer any of the actual content of the key.

The S3 tutorial mentions this but not very explicitly and not in this exact context. I'll add a section on this to help make it easier to find.

Note: for every old link like http://boto.cloudhackers.com/s3_tut.html that returns a 404, add in "/en/latest" right after the ".com" : http://boto.cloudhackers.com/en/latest/s3_tut.html . (Someone needs to explore mod_rewrite...)

like image 154
garnaat Avatar answered Oct 04 '22 02:10

garnaat