retry on 503 -- but not right away please

[Jan-Frode Myklebust]

On Wed, Apr 25, 2012 at 03:08:35PM +0200, Andreas Plesner Jacobsen wrote:
> > 
> 
> The only effect of the above is probably that you're DoSing the backend :)

It's been happening only a few times a day (eight 503's on a cache that
server 1.7 million requests yesterday), so I think that small DoS might
be acceptable :-)

> And unless you changed the default, max_restarts will kick in before you reach 12.

Oh.. didn't know about that one...

> 
> > Any other ideas?
> 
> I don't understand why you're seeing this 503 in vcl_error. A backend
> responding with a 503 would just hit the regular fetch->deliver path, not
> vcl_error, unless you explicitly return error somewhere in your vcl, so maybe
> your initial analysis of the problem is wrong?

I'm uncertain where the error is coming from (backend or varnish), but
I think I  understand why we get it. The file served is PUT on the backend apache
server using webdav (mod_dav), so every now and then varnish will
request a file that is in the process of being written. Then f.ex. the
file length will change between apache stat'ing it, and reading it -- so
apache or varnish will bail out with a 503..


> I think my preferred solution would be to return some synthetic page with meta
> refresh (and that would require you to use error, since that's the only place
> you can do synthetic currently).

It's not html-files being served, so I don't know if the client will
understand meta refresh. I'm wondering if maybe we need to solve this on
the backend then.. write the files to a temp-name and use incron to move
them to the right place (we don't have much control over the client
writing the files).


  -jf