Faster alternative to manage.py collectstatic (w/ s3boto storage backend) to sync static files to s3?

Question

I have been using s3boto's S3BotoStorage as my static files backend and syncing files to my aws s3 buckets (staging and production) using ./manage.py collectstatic. It works fine. However it is painfully slow. In addition to my own static files (just a few) and django admin, I have a few third party packages with many many static files (grappelli, django-redactor). And collectstatic can take upwards of 15 minutes each time I run it, depending on my internet connection. For instances where I'm syncing with my staging bucket and things aren't quite right, and I have to tweak something and re-sync, its a big time killer. Are there any good, fast, scriptable alternatives for syncing static files to s3?

Answer 1

I wrote a pluggable Django app, based on a djangosnippet, that caches the ETag of the remote file and compares the chached checksum instead of performing a lookup every time. It took me from about 1m30s to around 10s per call to manage.py collectstatic for a few hundred static files. Check it out here: https://github.com/antonagestam/collectfast

Answer 2

Set AWS_PRELOAD_METADATA to True in your settings so it pre-loads all files on s3 before syncing and only syncs the ones that are not already there (or have changed).