System can't take more than 5k req /sec

[Jason Price]

Note: this isn't a varnish question, per se, because varnish indicates it
isn't the problem.

What OS parameters has the community found valuable to tweak?

Varnish 3.0.5, running on a c3.4xlarge amazon instance.  16 'cores', 30gb
ram, amazon linux (aka RHEL 6ish) and half of a 10gb card to play with.
 Very recent linux kernel.

We see approximately 50% cache hit.  Payload is approximately 3.5kBytes
compressed.  Response times are averaging 10 milliseconds... Backend
response time is around 17-20 milliseconds.  We aren't seeing any lru_nuked
objects, so the cache is sized appropriately for the cachability of the
data (we can only cache for 15ish minutes due to the nature of the data).

Threads set to 4k per thread pool.  Two thread pools, increased
sess-workspace.  I see queued messages in varnish stat, but I never see
dropped messages, and I don't see the thread count getting near max of 8k
(usually hovers around 3300-3500).  We have a very high requests to
connections ratio (approximately 100:1).

Vmod for url coding is in play, and occasionally some syslog messages.

Tcp socket reuse and recycle got us some improvement.  I've also played
with the txqueuelen on the network interface, and a couple other related
parameters to little avail.

Anything I can do?

-Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20140425/a083ec01/attachment.html>