Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Amazon S3 and Cloudfront cache, how to clear cache or synchronize their cache

I have a cron job that runs every 10 minutes and updates the content-type and x-amz-meta. But since yesterday it seems like after the cron job run, Amazon is not picking up the changes made (refreshing his cache).

I even went and made the changes manually but no change...

When a video is uploaded it has a application/x-mp4 content-type and the cron job changes it to video/mp4.

Although S3 has the right content type video/mp4 cloudfront shows application/x-mp4(old content-type) ....

The cron job has been working for the last 6 months without a problem.

What is wrong with amazon caching? How can i synchronize the caching?

like image 442
mamesaye Avatar asked Feb 25 '14 17:02

mamesaye


People also ask

How do I clear AWS instance cache?

You can use invalidations to clear AWS S3 cache. To create invalidations login to AWS Console and go to Distribution Settings > Invalidations > Create Invalidation. Once invalidation is created just type path of file or a wildcard to purge the cache.

What is cache in CloudFront?

CloudFront caches your objects based on the values in all of the specified headers. CloudFront also forwards the headers that it forwards by default, but it caches your objects based only on the headers that you specify.

How do I automatically invalidate CloudFront cache?

Bynder can automatically invalidate your CloudFront CDN URLs when you upload a new version to an existing asset. No longer will you have to wait for its cache to expire. The assets you integrated externally using CloudFront will now always reflect the latest version.


7 Answers

Use Invalidations to clear the cache, you can put the path to the files you want to clear, or simply use wild cards to clear everything.

http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Invalidation.html#invalidating-objects-api

This can also be done using the API! http://docs.aws.amazon.com/cloudfront/latest/APIReference/API_CreateInvalidation.html

The AWS PHP SDK now has the methods but if you want to use something lighter check out this library: http://www.subchild.com/2010/09/17/amazon-cloudfront-php-invalidator/

user3305600's solution doesn't work as setting it to zero is the equivalent of Using the Origin Cache Headers.

like image 145
Neo Avatar answered Sep 24 '22 18:09

Neo


As to the actual code

get your CloudFront distribution id

aws cloudfront list-distributions

Invalidate all files in the distribution, so CloudFront fetches fresh ones

aws cloudfront create-invalidation --distribution-id=S11A16G5KZMEQD --paths /

My actual full release script is

#!/usr/bin/env bash

BUCKET=mysite.com
SOURCE_DIR=dist/

export AWS_ACCESS_KEY_ID=xxxxxxxxxxx
export AWS_SECRET_ACCESS_KEY=xxxxxxxxx
export AWS_DEFAULT_REGION=eu-west-1


echo "Building production"
if npm run build:prod ; then
   echo "Build Successful"
else
  echo "exiting.."
  exit 1
fi


echo "Removing all files on bucket"
aws s3 rm s3://${BUCKET} --recursive


echo "Attempting to upload site .."
echo "Command:  aws s3  sync $SOURCE_DIR s3://$BUCKET/"
aws s3  sync ${SOURCE_DIR} s3://${BUCKET}/
echo "S3 Upload complete"

echo "Invalidating cloudfrond distribution to get fresh cache"
aws cloudfront create-invalidation --distribution-id=S11A16G5KZMEQD --paths / --profile=myawsprofile

echo "Deployment complete"  

References

http://docs.aws.amazon.com/cli/latest/reference/cloudfront/get-invalidation.html

http://docs.aws.amazon.com/cli/latest/reference/cloudfront/create-invalidation.html

like image 24
Dr Manhattan Avatar answered Sep 22 '22 18:09

Dr Manhattan


Here is a manual way to invalidate the cache for all files on CloudFront via AWS

  1. Open your CloudFront Distributions list, and click the desired distribution ID (circled in red in screenshot below) you want to clear it's cache. enter image description here
  2. Click 'Invalidations' tab (see selected tab highlighted in blue in the screenshot below).
  3. Click 'Create invalidation' button (circled in red in the screenshot below) enter image description here
  4. Insert /* in the object paths input in order to clear cache of all files.
  5. Click 'Create invalidation' button. enter image description here
like image 26
Luke Phillips Avatar answered Sep 26 '22 18:09

Luke Phillips


S3 is not used for real time development but if you really want to test your freshly deployed website use

http://yourdomain.com/index.html?v=2
http://yourdomain.com/init.js?v=2

Adding a version parameter in the end will stop using the cached version of the file and the browser will get a fresh copy of the file from the server bucket

like image 32
HimalayanCoder Avatar answered Sep 22 '22 18:09

HimalayanCoder


Cloudfront will cache a file/object until the cache expiry. By default it is 24 hrs. If you have changed this to a large value, then it takes longer.

If you anytime needs to force clear the cache, use the invalidation. It is charged separately.

Another option is to change the URL (object key), so it fetches the new object always.

like image 33
Sony Kadavan Avatar answered Sep 25 '22 18:09

Sony Kadavan


If you're looking for a minimal solution that invalidates the cache, this edited version of Dr Manhattan's solution should be sufficient. Note that I'm specifying the root / directory to indicate I want the whole site refreshed.

export AWS_ACCESS_KEY_ID=<Key>
export AWS_SECRET_ACCESS_KEY=<Secret>
export AWS_DEFAULT_REGION=eu-west-1

echo "Invalidating cloudfrond distribution to get fresh cache"
aws cloudfront create-invalidation --distribution-id=<distributionId> --paths / --profile=<awsprofile>

Region Codes can be found here

You'll also need to create a profile using the aws cli. Use the aws configure --profile option. Below is an example snippet from Amazon.

$ aws configure --profile user2
AWS Access Key ID [None]: AKIAI44QH8DHBEXAMPLE
AWS Secret Access Key [None]: je7MtGbClwBF/2Zp9Utk/h3yCo8nvbEXAMPLEKEY
Default region name [None]: us-east-1
Default output format [None]: text
like image 23
Francis Avatar answered Sep 25 '22 18:09

Francis


(edit: Does not work)As of 2014, You can clear your cache whenever you want, Please go thorough the Documentation or just go to your distribution settings>Behaviors>Edit

Object Caching Use (Origin Cache Headers) Customize

Minimum TTL = 0

http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html

like image 23
user3305600 Avatar answered Sep 25 '22 18:09

user3305600