how to get a 304 for images stored on amazon s3 when using django-storages on django app?

Question

I just started storing user uploaded images on Amazon's S3. It's pretty nice because it took care of my storage problem. However, I am struggling when it comes to having the browser cache the images.

I am using django-storages. In their docs they specify that you can put things on the request header for an image by setting the AWS_HEADER var in your settings. I am doing that and getting no results.

Basically when the app requests the image(s), I get a 200 EVERY TIME. ARG... when I take the browser straight to the image (copy and paste the link into a new window) I get a 200 then a 304 every time after that.

It's very frustrating because it re downloads the image every time. Some pages have up to 25 small thumbnails on them and it's redownloading everything every time the page is reloaded.

I am serving my static files using djangos staticfiles and they are working properly. I get a 200, then 304 after the file is cached.

here are my AWS settings in settings.py

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto.S3BotoStorage'
AWS_ACCESS_KEY_ID = '***'
AWS_SECRET_ACCESS_KEY = '***'
AWS_STORAGE_BUCKET_NAME = 'foobar_uploads'
AWS_HEADERS = {
    'Expires': 'Thu, 15 Apr 2020 20:00:00 GMT',
    'Cache-Control': 'max-age=86400',
}

AWS_CALLING_FORMAT = CallingFormat.SUBDOMAIN

here are the request and response headers for when the app requests the image: (i've replaced what i feel might be sensitive information with '*')

##request##
GET /user_uploads/*****/2012/3/17/14/46/thumb_a_28_DSC_0472.jpg?Signature=FVR6T%2BXFwHMmdQ9K3n7Ppp7QxoY%3D&Expires=1332023525&AWSAccessKeyId=***** HTTP/1.1
Host: *****_user_uploads_sandbox.s3.amazonaws.com
Connection: keep-alive
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.79 Safari/535.11
Accept: */*
Referer: http://localhost:8000/m/my-photos/
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3


##response##
HTTP/1.1 200 OK
x-amz-id-2: Hn3S+3gmeLHIjKCpz+2ocE6aPsLCVHh56jJYTsPHwxU98y89x+9X1Ml202evBUHT
x-amz-request-id: 528CEB880CA89AD3
Date: Sat, 17 Mar 2012 21:32:06 GMT
Cache-Control: max-age=86400
Expires: Thu, 15 Apr 2020 20:00:00 GMT
Last-Modified: Sat, 17 Mar 2012 20:46:29 GMT
ETag: "a3bc70e0c3fc0deb974edf95668e9030"
Accept-Ranges: bytes
Content-Type: image/jpeg
Content-Length: 8608
Server: AmazonS3

here are the request/response headers for when i manually request the image by copy and pasting link to the image:

##request##
GET /user_uploads/*****/2012/3/17/14/46/thumb_a_28_DSC_0472.jpg?Signature=FVR6T%2BXFwHMmdQ9K3n7Ppp7QxoY%3D&Expires=1332023525&AWSAccessKeyId=***** HTTP/1.1
Host: porlio_user_uploads_sandbox.s3.amazonaws.com
Connection: keep-alive
Cache-Control: max-age=0
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.79 Safari/535.11
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
If-None-Match: "a3bc70e0c3fc0deb974edf95668e9030"
If-Modified-Since: Sat, 17 Mar 2012 20:46:29 GMT



##response##
HTTP/1.1 304 Not Modified
x-amz-id-2: FZH0imrbNxziMznhl5zAoo38CaM7Z+TFnd8R6HtTYB3eTmVpCih+1IniKaliRo18
x-amz-request-id: 3CACF77FBB39D088
Date: Sat, 17 Mar 2012 21:33:22 GMT
Last-Modified: Sat, 17 Mar 2012 20:46:29 GMT
ETag: "a3bc70e0c3fc0deb974edf95668e9030"
Server: AmazonS3

I see there are a few differences such as the "If-None-Match:" or the "If-Modified-Since:" . I think that if I were to set those, then it should work like I'd like.

Is there an easy way to do this?

Thanks for any help!

EDIT 1: I read this article and couldn't translate it very well.. http://coder.cl/2012/01/django-and-amazon-s3/comment-page-1/

Answer 1

If you don't want django-storages to add the auth query to your static media, add the following to your settings.py:

AWS_QUERYSTRING_AUTH = False

This is caused by the S3 Boto backend, which will automatically append the query string unless instructed not to, via django-storages. If you look at the source code for the application, you'll notice that it looks for some extra undocumented settings in your settings.py file (around line 34).

I responded to this similar issue raised on the Github for django-compressor here.

Answer 2

It doesn't look like you're sending a "If-None-Match" or "If-Modified-Since" in your app's request, so S3 has no way to send you back a 304 since it has no idea what you have. As you can see your browser has the file cached, so it is sending both of those headers and getting a proper 304 response.

If you're keeping local copies of files, you'll need to store the Last-Modified and/or ETag and send them along with your request for the image.

As a related aside, I'd consider using just the max-age header and skipping Expires. According the to the spec, Expires should not be more then a year in the future. Using both Expires and max-age is redundant, and max-age can be set to a relative amount (as you have done).

I highly recommend this article on caching and his related REDbot tool for checking your configuration.