Dropped connections with tcp_tw_recycle=1

[Nick Loman]

Hi Sven,

I don't know the basis precise for it, but I can vouch for the fact that 
tcp_tw_recycle is incompatible with NAT on the server side. I would 
guess it is because the NAT gateway keeps a connection tracking list and 
is unhappy that the webserver is trying to reuse the same ip:port hash 
whilst it is registered in TIME_WAIT mode.

There was a discussion of this previously:
http://projects.linpro.no/pipermail/varnish-misc/2009-April/002764.html

As you say tw_reuse works OK with NAT.

Cheers,

Nick.


Sven Ulland wrote:
> I was recently debugging an issue where several clients experienced
> sporadic problems connecting to a website cached by varnish. Every now
> and then (say, something like every 20-50th TCP connection) would time
> out, or sometimes take a few SYNs before being accepted.
>
> Here's a typical example. It's observed at the spot marked 'X' in this
> network structure from the client network's perspective:
>
>    [clients] -> [NAT gateway] -> [bridge firewall]X -> [Internet]
>
>   0.00 natgw-extip varni-extip TCP 4292 > http [SYN] TSV=283647429 
> TSER=0 WS=6
>   2.99 natgw-extip varni-extip TCP 4292 > http [SYN] TSV=283648179 
> TSER=0 WS=6
>   8.99 natgw-extip varni-extip TCP 4292 > http [SYN] TSV=283649679 
> TSER=0 WS=6
> 20.99 natgw-extip varni-extip TCP 4292 > http [SYN] TSV=283652679 TSER=0 
> WS=6
> 44.99 natgw-extip varni-extip TCP 4292 > http [SYN] TSV=283658679 TSER=0 
> WS=6
> 93.00 natgw-extip varni-extip TCP 4292 > http [SYN] TSV=283670679 TSER=0 
> WS=6
> 93.00 varni-extip natgw-extip TCP http > 4292 [SYN, ACK] TSV=2342207123 
> TSER=283670679
>
> Note: The NAT gateway didn't do port translation here. Also, the
> timestamp values were not touched by the NAT gateway. The varnish node
> is behind LVS-TUN, but the LVS was not the culprit.
>
> After troubleshooting with the website owner, tcpdumping at various
> points on both sides, it was clear that the packets were reaching the
> varnish node, but except the last SYN, they were all dropped. This
> turned out to be because the varnish node had the tcp_tw_recycle sysctl
> enabled. Switching it off fixed the problem.
>
> The performance page on the varnish wiki features recommends Linux
> sysctl settings, including enabling tcp_tw_recycle, since april 2008.
> The recycle setting was removed from that page recently, but I would
> think there are a lot of installations around the world that have it
> enabled.
>
> I tried to figure out exactly how the recycling mechanism works, but the
> code is too complex to figure out without time or kernel network
> experience. Recycling was introduced by David Miller in 2.3.15, ref
> <URL:http://lxr.linux.no/#linux-old+v2.3.15/net/ipv4/tcp_ipv4.c#L324>
> and e.g. <URL:http://lxr.linux.no/#linux+v2.6.31/net/ipv4/tcp_ipv4.c#L1255>.
> Do anyone have a good grasp on how it works, its connection to the RFC
> 1323 PAWS mechanism, and its claimed incompatibility with NAT (ref
> <URL:http://lkml.org/lkml/2008/11/15/83>)?
>
> When observing the same issue previously (dropped SYNs), I ditched
> tw_recycle in favour of tcp_tw_reuse, which doesn't seem to cause any
> problems (this was on a normal Apache system). It too is severely
> underdocumented, so I was hoping to shed some light on them both, and
> the exact circumstances where they are suitable for use.
>
> Sven
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email 
> ______________________________________________________________________
>