[Varnish] #85: High number of "worker threads not created" and "overflowed work requests"

[Varnish]

#85: High number of "worker threads not created" and "overflowed work requests"
----------------------+-----------------------------------------------------
 Reporter:  Thias     |       Owner:  phk
     Type:  defect    |      Status:  new
 Priority:  normal    |   Milestone:     
Component:  varnishd  |     Version:  1.0
 Severity:  normal    |    Keywords:     
----------------------+-----------------------------------------------------
 Hi,

 I'm using varnish svn trunk on RHEL4 x86, and after about 5 minutes of
 sending it quite heavy traffic, things stop working so well and web pages
 become slow to load... here is what varnishstat reports :
 {{{
        10231        72.87        25.84 Client connections accepted
        64017        83.85       161.66 Client requests received
        46904        76.86       118.44 Cache hits
          115         0.00         0.29 Cache hits for pass
        16998         6.99        42.92 Cache misses
        17113         6.99        43.21 Backend connections success
            0         0.00         0.00 Backend connections failures
        16808         6.99        42.44 Backend connections reuses
        16823         6.99        42.48 Backend connections recycles
           17         0.00         0.04 Backend connections unused
         1187         2.00         3.00 N struct srcaddr
          220         1.00         0.56 N active struct srcaddr
          811         0.00         2.05 N struct sess_mem
          810         0.00         2.05 N struct sess
        17087         5.99        43.15 N struct object
        17086         5.99        43.15 N struct objecthead
        16955         6.99        42.82 N struct smf
            0         0.00         0.00 N small free smf
            3         0.00         0.01 N large free smf
           33         0.00         0.08 N struct vbe_conn
           93         0.00         0.23 N worker threads
           93         0.00         0.23 N worker threads created
        17205        65.88        43.45 N worker threads not created
            0         0.00         0.00 N worker threads limited
          632       -17.97         1.60 N queued work requests
        17298        65.88        43.68 N overflowed work requests
            0         0.00         0.00 N dropped work requests
            0         0.00         0.00 N expired objects
            0         0.00         0.00 N objects on deathrow
            0         0.00         0.00 HTTP header overflows
        26223        39.93        66.22 Objects sent with sendfile
        24092        23.96        60.84 Objects sent with write
         9355        27.95        23.62 Total Sessions
        63925        83.85       161.43 Total Requests
            0         0.00         0.00 Total pipe
          154         0.00         0.39 Total pass
        16948         5.99        42.80 Total fetch
     14946171     19597.00     37742.86 Total header bytes
   1625794755   2061759.36   4105542.31 Total body bytes
         1295        11.98         3.27 Session Closed
          265         0.00         0.67 Session Pipeline
           84         0.00         0.21 Session Read Ahead
        62339        71.87       157.42 Session herd
      2656226      3013.46      6707.64 SHM records
       129721       282.48       327.58 SHM writes
          127         1.00         0.32 SHM MTX contention
 }}}

 The "worker threads not created" and "overflowed work requests" start
 growing a lot when the problem appears. After switching away the traffic,
 but keeping varnishd running, things settle again and those values no
 longer grow, and things work fine again.

 Right now, even though things work fine (with no real/high traffic),
 varnishlog reports :
 {{{
     7 Debug          "Accept failed errno=24"
     7 Debug          "Accept failed errno=24"
     7 Debug          "Accept failed errno=24"
     7 Debug          "Accept failed errno=24"
     7 Debug          "Accept failed errno=24"
     7 Debug          "Accept failed errno=24"
     7 Debug          "Accept failed errno=24"
     7 Debug          "Accept failed errno=24"
     7 Debug          "Accept failed errno=24"
     7 Debug          "Accept failed errno=24"
     7 Debug          "Accept failed errno=24"
     7 Debug          "Accept failed errno=24"
     7 Debug          "Accept failed errno=24"
     7 Debug          "Accept failed errno=24"
     7 Debug          "Accept failed errno=24"
     7 Debug          "Accept failed errno=24"
 }}}

 ...and so on, LOTS of those, which seem to be "too many open files". If
 this is the issue, could we have a varnish command line option to have it
 increase the maximum number of allowed open files for itself? (possible
 when run as root)
 For now, I'll add "ulimit -n 500000" to /etc/rc.d/init.d/varnish and hope
 it'll fix it.

-- 
Ticket URL: <http://varnish.projects.linpro.no/ticket/85>
Varnish <http://varnish.projects.linpro.no/>
The Varnish HTTP Accelerator