HOWTO Cache S3 Objects with Varnish

Many startups these days are using Amazon S3 to serve directly their static assets. S3 is being used as a simple CDN instead of more professional (and expensive) solutions (including Amazon’s own CloudFront) because it is very simple and cheap to use. Still if you have a high traffic site, this will no longer be so cheap since you will be paying for all those requests and the bandwidth. In such cases if you still want to use S3 for the storage advantage (like storing millions of files and see it as an unlimited storage space) but not have your bill go up like crazy, you can use a reverse proxy or web accelerator to cache your assets locally and reduce the number of direct hits on S3. We could use Squid or Varnish for this, and in this article I will show how we can configure Varnish for this. We are using varnish with S3 on various projects and it works very well, simplifying the setup and saving a lot of money in the Amazon S3 bill.

Varnish is a state-of-the-art, high-performance HTTP accelerator. It uses the advanced features in Linux 2.6, FreeBSD 6/7 and Solaris 10 to achieve its high performance. I will not go over the installation of varnish here, but I would highly recommend to use the latest version available at this time 2.0.4 as older versions have various issues.

This article was taken from:

http://www.ducea.com/2009/08/04/using-varnish-in-front-of-your-amazon-s3-static-content/

We could try to use something simple like this in a varnish vcl

A complete varnish vcl configuration to use with the Amazon S3 backend might look like this:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

backend s3 { .host = "s3.amazonaws.com"; .port = "80"; } sub vcl_recv { if (req.url ~ "\.(css|gif|ico|jpg|jpeg|js|png|swf|txt)$") { unset req.http.cookie; unset req.http.cache-control; unset req.http.pragma; unset req.http.expires; unset req.http.etag; unset req.http.X-Forwarded-For; set req.backend = s3; set req.http.host = "my_bucket.s3.amazonaws.com"; lookup; } } sub vcl_fetch { unset obj.http.X-Amz-Id-2; unset obj.http.X-Amz-Meta-Group; unset obj.http.X-Amz-Meta-Owner; unset obj.http.X-Amz-Meta-Permissions; unset obj.http.X-Amz-Request-Id; set obj.ttl = 1w; set obj.grace = 30s; }