Caching & GZip on GAE (Community Wiki)

Question

Why does it seem like Google App Engine isn’t setting appropriate cache-friendly headers (like far-future expiration dates) on my CSS stylesheets and JavaScript files? When does GAE gzip those files? My app.yaml marks the respective directories as static_dirs, so the lack of far-future expiration dates is kind of surprising to me.

This is a community wiki to showcase the best practices regarding static file caching and gzipping on GAE!

Answer 1

How does GAE handle caching?

It seems GAE sets near-future cache expiration times, but does use the etag header. This is used so browsers can ask, “Has this file changed since when it had a etag of X68f0o?” and hear “Nope – 304 Not Modified” back in response.

As opposed to far-future expiration dates, this has the following trade-offs:

• Your end users will get the latest copies of your resources, even if they have the same name (unlike far-future expiration). This is good.
• Your end users will however still have to make a request to check on the status of that file. This does slow down your site, and is “pure overhead” when the content hasn’t changed. This is not ideal.

Opting for far-future cache expiration instead of (just) etag

To use far-future expiration dates takes two steps and a bit of understanding.

1. You have to manually update your app to request new versions of resources, by e.g. naming files like mysitesstyles.2011-02-11T0411.css instead of mysitestyles.css. There are tools to help automate this, but I’m not aware of any that directly relate to GAE.

2. Configure GAE to set the expiration times you want by using default_expiration and/or expiration in app.yaml. GAE docs on static files

A third option: Application manifests

Cache manifests are an HTML5 feature that overrides cache headers. MDN article, DiveIntoHTML5, W3C. This affects more than just your script and style files' caching, however. Use with care!

When does GAE gzip?

According to Google’s FAQ,

Google App Engine does its best to serve gzipped content to browsers that support it. Taking advantage of this scheme is automatic and requires no modifications to applications.

We use a combination of request headers (Accept-Encoding, User-Agent) and response headers (Content-Type) to determine whether or not the end-user can take advantage of gzipped content. This approach avoids some well-known bugs with gzipped content in popular browsers. To force gzipped content to be served, clients may supply 'gzip' as the value of both the Accept-Encoding and User-Agent request headers. Content will never be gzipped if no Accept-Encoding header is present.

This is covered further in the runtime environment documentation (Java | Python).

Some real-world observations do show this to generally be true. Assuming a gzip-capable browser:

• GAE gzips actual pages (if they have proper content-type headers like text/html; charset=utf-8)
• GAE gzips scripts and styles in static_dirs (defined in app.yaml).
• Note that you should not expect GAE to gzip images like GIFs or JPEGs as they are already compressed.