varnish using dropped backend connections?

[Nathan Warren]

On Mon Jun 17 10:55:37 CEST 2013, Lasse Karstensen wrote:

>> My suspicion was that varnish was creating a keepalive session to the play server that was persistent. This seems to be the case with tcp sessions visible via netstat) on both sides when no traffic is passing through. It seems as though the firewall was dropping the keepalive session, but varnish kept using it as if it were there. Then it kept re-using it over and over. After refreshing the page a few times (in error), the page will start working.

>The "proper" fix is to find out which NAT/connection tracking table that purges this entry and fix that.

>If you just want to fix the problem, you can ask Varnish to stop reusing backend connections. Of course this will give you TCP slow start on all backend fetches, but this is usually acceptable.

>Here is a VCL snippet for disabling reuse:

>"""
>sub vcl_miss {
>    set bereq.http.Connection = "close";
>}

>sub vcl_pass {
>    set bereq.http.Connection = "close";
>}
>"""

Thanks, Lasse. Your message didn't make it to my inbox strangely, but I caught it in the archive. This makes perfect sense, and I believe I was missing the vlc_miss definition to force each session closed. I don't think the additional network overhead in my situation would be much trouble.

As I stated earlier, I ended up using proxypass in apache as an intermediate backend, since it has a feature which passes the SO_KEEPALIVE flag to the OS. I had to drop the keepalive timer on the OS as well via /proc/sys/net/ipv4/tcp_keepalive_time. At some point I'll probably try to cut apache out of the loop.

I'm still curious if it would be worthwhile to make an optional parameter to the backend definition to set the SO_KEEPALIVE flag. It seems to be an option for frontend connections, but not backend.

Nate