How to set-up a Django project with django-storages and Amazon S3, but with different folders for static files and media files?

Question

I'm configuring a Django project that were using the server filesystem for storing the apps static files (STATIC_ROOT) and user uploaded files (MEDIA_ROOT).

I need now to host all that content on Amazon's S3, so I have created a bucket for this. Using django-storages with the boto storage backend, I managed to upload collected statics to the S3 bucket:

MEDIA_ROOT = '/media/' STATIC_ROOT = '/static/'  DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage' AWS_ACCESS_KEY_ID = 'KEY_ID...' AWS_SECRET_ACCESS_KEY = 'ACCESS_KEY...' AWS_STORAGE_BUCKET_NAME = 'bucket-name' STATICFILES_STORAGE = 'storages.backends.s3boto.S3BotoStorage' 

Then, I got a problem: the MEDIA_ROOT and STATIC_ROOT are not used within the bucket, so the bucket root contains both the static files and user uploaded paths.

So then I could set:

S3_URL = 'http://s3.amazonaws.com/%s' % AWS_STORAGE_BUCKET_NAME STATIC_URL = S3_URL + STATIC_ROOT MEDIA_URL = 'S3_URL + MEDIA_ROOT 

And use those settings in the templates, but there is no distinction of static/media files when storing in S3 with django-storages.

How this can be done?

Thanks!

Answer 1

I think the following should work, and be simpler than Mandx's method, although it's very similar:

Create a s3utils.py file:

from storages.backends.s3boto import S3BotoStorage  StaticRootS3BotoStorage = lambda: S3BotoStorage(location='static') MediaRootS3BotoStorage  = lambda: S3BotoStorage(location='media') 

Then in your settings.py:

DEFAULT_FILE_STORAGE = 'myproject.s3utils.MediaRootS3BotoStorage' STATICFILES_STORAGE = 'myproject.s3utils.StaticRootS3BotoStorage' 

A different but related example (that I've actually tested) can be seen in the two example_ files here.

Answer 2

I'm currently using this code in a separated s3utils module:

from django.core.exceptions import SuspiciousOperation from django.utils.encoding import force_unicode  from storages.backends.s3boto import S3BotoStorage   def safe_join(base, *paths):     """     A version of django.utils._os.safe_join for S3 paths.      Joins one or more path components to the base path component intelligently.     Returns a normalized version of the final path.      The final path must be located inside of the base path component (otherwise     a ValueError is raised).      Paths outside the base path indicate a possible security sensitive operation.     """     from urlparse import urljoin     base_path = force_unicode(base)     paths = map(lambda p: force_unicode(p), paths)     final_path = urljoin(base_path + ("/" if not base_path.endswith("/") else ""), *paths)     # Ensure final_path starts with base_path and that the next character after     # the final path is '/' (or nothing, in which case final_path must be     # equal to base_path).     base_path_len = len(base_path) - 1     if not final_path.startswith(base_path) \        or final_path[base_path_len:base_path_len + 1] not in ('', '/'):         raise ValueError('the joined path is located outside of the base path'                          ' component')     return final_path   class StaticRootS3BotoStorage(S3BotoStorage):     def __init__(self, *args, **kwargs):         super(StaticRootS3BotoStorage, self).__init__(*args, **kwargs)         self.location = kwargs.get('location', '')         self.location = 'static/' + self.location.lstrip('/')      def _normalize_name(self, name):         try:             return safe_join(self.location, name).lstrip('/')         except ValueError:             raise SuspiciousOperation("Attempted access to '%s' denied." % name)   class MediaRootS3BotoStorage(S3BotoStorage):     def __init__(self, *args, **kwargs):         super(MediaRootS3BotoStorage, self).__init__(*args, **kwargs)         self.location = kwargs.get('location', '')         self.location = 'media/' + self.location.lstrip('/')      def _normalize_name(self, name):         try:             return safe_join(self.location, name).lstrip('/')         except ValueError:             raise SuspiciousOperation("Attempted access to '%s' denied." % name) 

Then, in my settings module:

DEFAULT_FILE_STORAGE = 'myproyect.s3utils.MediaRootS3BotoStorage' STATICFILES_STORAGE = 'myproyect.s3utils.StaticRootS3BotoStorage' 

I got to redefine the _normalize_name() private method to use a "fixed" version of the safe_join() function, since the original code is giving me SuspiciousOperation exceptions for legal paths.

I'm posting this for consideration, if anyone can give a better answer or improve this one, it will be very welcome.