Varnish and TCP Incast Throughput Collapse

[Andrei]

Out of curiosity, what does ethtool show for the related nics on both
servers? I also have Varnish on a 10G server, and can reach around
7.7Gbit/s serving anywhere between 6-28k requests/second, however it did
take some sysctl tuning and the westwood TCP congestion control algo

On Wed, Jul 5, 2017 at 3:09 PM, John Salmon <John.Salmon at deshawresearch.com>
wrote:

> I've been using Varnish in an "intranet" application.  The picture is
> roughly:
>
>   origin <-> Varnish <-- 10G channel ---> switch <-- 1G channel --> client
>
> The machine running Varnish is a high-performance server.  It can
> easily saturate a 10Gbit channel.  The machine running the client is a
> more modest desktop workstation, but it's fully capable of saturating
> a 1Gbit channel.
>
> The client makes HTTP requests for objects of size 128kB.
>
> When the client makes those requests serially, "useful" data is
> transferred at about 80% of the channel bandwidth of the Gigabit
> link, which seems perfectly reasonable.
>
> But when the client makes the requests in parallel (typically
> 4-at-a-time, but it can vary), *total* throughput drops to about 25%
> of the channel bandwidth, i.e., about 30Mbyte/sec.
>
> After looking at traces and doing a fair amount of experimentation, we
> have reached the tentative conclusion that we're seeing "TCP Incast
> Throughput Collapse" (see references below)
>
> The literature on "TCP Incast Throughput Collapse" typically describes
> scenarios where a large number of servers overwhelm a single inbound
> port.  I haven't found any discussion of incast collapse with only one
> server, but it seems like a natural consequence of a 10Gigabit-capable
> server feeding a 1-Gigabit downlink.
>
> Has anybody else seen anything similar?  With Varnish or other single
> servers on 10Gbit to 1Gbit links.
>
> The literature offers a variety of mitigation strategies, but there are
> non-trivial tradeoffs and none appears to be a silver bullet.
>
> If anyone has seen TCP Incast Collapse with Varnish, were you able to work
> around it, and if so, how?
>
> Thanks,
> John Salmon
>
> References:
>
> http://www.pdl.cmu.edu/Incast/
>
> Annotated Bibliography in:
>    https://lists.freebsd.org/pipermail/freebsd-net/2015-
> November/043926.html
>
> --
> *.*
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20170705/3173ffe4/attachment.html>