Can Varnish be configured to try a second server for some resource if it gets a 404 from the primary server?

Question

This would be a perfect solution for me as I need to serve some generated content to web browsers. My plan is to generate the content on demand and store it for next time. I don't want the browsers to call my service (which generates the content) every time. I want them to go directly to the "cached" resource if it's available and only call the service if it's not. So I'd put Varinsh in front of server A which runs the service and server B which stores the previously generated content versions. If it gets a request for a resource it hasn't got cached it'll try server B. Upon getting a 404 response it'll request the same resource from server A.

Can Varnish be configured in such a way with VCL? If not is there a solution like that you know about?

P.S. I don't want to send 302 redirects to the browser plus I don't have control over server B to make it send such redirects instead of 404's

Answer 1

This is perfectly possible in Varnish. Make sure that in vcl_fetch (and possibly in vcl_error) you check the return status code (e.g. check for status > 400), do a restart if it failed, and in vcl_recv select the other backend if req.restarts > 0. For example:

backend serverA {
   .host="192.168.0.1";
   .port = "80";
}

backend serverB {
   .host = "192.168.0.2";
   .port = "80";
}

sub vcl_recv {
   if (req.restarts == 0) {
       set req.backend = serverB;
   } else {
       set req.backend = serverA;
   }
}

sub vcl_fetch {
    if (obj.status >= 400 && req.restarts == 0) {
        restart;
    }
}

sub vcl_error {
   if (req.restarts == 0) {
       restart;
   }
}

But this being said, it sounds like you're reinventing the concept of a cache server. And Varnish is great cache server. Why not have one back-end server (serverA) and have Varnish cache your generated entities? You can setup complex rules and you'll get expiration (of the cache), purge management and performance for free! :)