caching of restarted requests possible?

[Hauke Lampe]

Hi.

I have a virtual host "images.example.com" served from two backends:

- a backend "archive" which contains the bulk of images on fast read-only storage
- a backend "updates" holding additions and updates

A request for /foo.jpg should check the update backend first, even if the image was previously cached from the archive backend. A 404 status from the update backend restarts the request and fetches the image from the archive backend or delivers a cached copy.

My code so far is at:

http://cfg.openchaos.org/varnish/vcl/special/backend_select_updates.vcl

It basically does what I want, but because the update backend's 404 is not stored when vcl_fetch returns restart, it sends a backend query for every request.

I'd like to cache the 404 response for some time and immediately lookup the object under the next backend's hash key, so the update backend is only queried again after the TTL of the 404 object expires.

I figure that even if varnish would cache the request before restart, it would probably not go through vcl_fetch next time. I tried setting a magic header in vcl_fetch and restart the request in vcl_deliver. varnish didn't like that and died with "INCOMPLETE AT: cnt_deliver(196)"

For now, I use the original URL and Host: header as hash key and reduced the cache TTL. That works well enough, even though it produces more traffic on the backends than would be necessary.

Is there any way to remember the previous request status on restart and use it for backend selection in vcl_recv?


Hauke