[Varnish] #1738: Large number of ExpKills and nuking issues

[Varnish]

#1738: Large number of ExpKills and nuking issues
----------------------+--------------------
 Reporter:  coredump  |       Type:  defect
   Status:  new       |   Priority:  normal
Milestone:            |  Component:  build
  Version:  unknown   |   Severity:  normal
 Keywords:            |
----------------------+--------------------
 Last Friday we had a problem that may be a bug.

 Our setup uses two varnish servers, each one using malloc,65G storage.
 Same hardware/config with 96GB total memory. No swap used. There's a load
 balancer in front of the varnish servers doing consistent hashing. Using
 varnish-trunk revision 6f53e59

 One of our servers started increasing number of objects, expires stopped
 happening and we had a giant number of lru_nukes (those eventually went to
 zero) and worker thread going to the max number. Those are some graphs
 showing how it behaved:

 https://dl.dropboxusercontent.com/1/view/z9xmpv99p7oi6ws/Captured/3gD2H.png
 http://dsl.so/1B8AtVA

 Varnishlog showed requests going normally, except with some that had 10k+
 ExpKill messages, like this one:
 https://gist.github.com/coredump/6710e57aaf3dff7788e1
 And those are the params being used at the moment:
 https://gist.github.com/coredump/8a1c4d56ed79adb90956

 As a result, the load balancers started to get timeouts from that varnish
 server.

-- 
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1738>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator