Enable serving of graced objects if backend is down

[perbu]

We would like to be able to serve graced objects if the backend health polling says the backend is down.

Its not clear to me whether this should happen automatically or if it should be done explicitly in vcl ( if !req.backend.healthy { lookup;} ).

It seems to be a minor job enabling this (from my limited understanding of the code) but it would be a huge enhancement for varnish.

Per.

[anders]

I also want to be able to serve graced objects, and to control it from VCL. Please give us force_grace!

Excerpt from IRC conversation:

<perbu> phk: is it hard to lift the limitation on the grace that the object
+has to be busy in order for the graced object to be served?
<perbu> I understand that normally the limitation should be in place - but
+when the backend is down it would be really useful.
<perbu> i've tried forcing a lookup in vcl - but it doesn't seem to respect
+that.
<phk> perbu, yes, we should grace when the polling marks the backend down as
+well.
<phk> perbu, open a ticket so we remember
<perbu> phk: ack.
> This is very interesting for me too.
> It would be nice to control this grace in VCL as well, not just because of
+polling. For example if replies from the backend have the wrong http code,
+grace would be preferred..
> In such situations, it would preferable to deliver the graced object, and
+even extend the grave time.. :)
 <phk> anordby, in that case, you need to do a restart to take another attempt
 <phk> anordby, we are simply too far down the road at that time to make a
+determination
<phk> or said another way: we would have to do exactly the same work as you
+end up doing with restart
> phk: If all backends fail at around the same time, restart does not help.
 <phk> anordby, the trick is to make the restart do grace before trying to
+contact the backend.
> Yes. But something needs to be done for that to happen, I suppose ?
<phk> I guess you're right, we need to say "don't even try, just grace"
> I just recently started to play with Varnish' load balancing ability.
> One thing I found out quickly, default interval is 0? Meaning it'll poll as
+often as possible, if interval is not defined? This lead to huge amounts of
+traffic and CPU usage way beyond expected. :-)
> Maybe there should be a default .interval of one or two seconds?
 <phk> anordby, ticket
> Ok, I will. There could also be some issues related to loading VCL runtime,
+I'll check that too.
> It would be nice to also say "try restart 2 times, then grace" in VCL.
<phk> well, you can, it would be something like "sub vcl_recv { if (req.
+restart == 3) { force_grace; }}
<Mithrandir> s/restart/&s/, though
> Amen
> (Let it happen) :)

[anders]

We (Aftenposten & E24) are interested in helping sponsor this feature, provided that:

1) Forced grace always will work for any kind of backend error (network timeout, connection refused, response too slow, wrong http return code)

2) It works with a longer grace period, so that the backlog of cached data is longer. This shall not create problems for updates/purges for new content.

3) It is possible to control in VCL when forced grace should happen. For example if the backend delivers an unwanted 5xx http return code.

[donarae]

Replying to perbu:

We would like to be able to serve graced objects if the backend health polling says the backend is down. Its not clear to me whether this should happen automatically or if it should be done explicitly in vcl ( if !req.backend.healthy { lookup;} ). It seems to be a minor job enabling this (from my limited understanding of the code) but it would be a huge enhancement for varnish. Per.

[perbu]

20:08 <phk> perbu, what is the difference between ticket #369 and saint mode ? 20:11 <perbu> nothing, AFAIK. 20:11 <perbu> we didn't have the fancy name back then. :-) 20:12 <phk> ok, want to close it yourself then ? 20:12 <perbu> oh, may I? 20:12 <phk> you know how it is, when that time comes, real men shoot their own dogs.