too many clients

[Václav Bílek]

Helo

I have got a problem using varnish infront of webserwer cluster running
arroung 20K req/s; when i tried to put varnish(4 loadbalanced machines
8CPU cores each 32GB ram) infront of this farm it serves OK for few
seconds but then aper in syslog such messages:

kernel: TCP: drop open request from IP/port

and varnish stops serving


I thing it is related with number of clients (which was around 30K when
the problem apeard) and rate of request from each of this clients (
arround 1 request each 5 seconds).

I have tried tunning kernel TCP stack ... are there any recomandation on
varnish seting for many clients?

Vaclav Bilek


my params:
accept_fd_holdoff          50 [ms]
acceptor                   default (epoll, poll)
auto_restart               on [bool]
backend_http11             on [bool]
between_bytes_timeout      60.000000 [s]
cache_vbe_conns            off [bool]
cc_command                 "exec cc -fpic -shared -Wl,-x -o %o %s"
cli_buffer                 8192 [bytes]
cli_timeout                5 [seconds]
client_http11              off [bool]
clock_skew                 10 [s]
connect_timeout            0.400000 [s]
default_grace              10
default_ttl                60 [seconds]
diag_bitmap                0x0 [bitmap]
err_ttl                    0 [seconds]
esi_syntax                 0 [bitmap]
fetch_chunksize            128 [kilobytes]
first_byte_timeout         60.000000 [s]
group                      nogroup (65534)
listen_address             0.0.0.0:80
listen_depth               10240 [connections]
log_hashstring             off [bool]
log_local_address          off [bool]
lru_interval               2 [seconds]
max_esi_includes           5 [includes]
max_restarts               4 [restarts]
obj_workspace              8192 [bytes]
overflow_max               100 [%]
ping_interval              3 [seconds]
pipe_timeout               60 [seconds]
prefer_ipv6                off [bool]
purge_dups                 off [bool]
purge_hash                 on [bool]
rush_exponent              3 [requests per request]
send_timeout               600 [seconds]
sess_timeout               1 [seconds]
sess_workspace             16384 [bytes]
session_linger             0 [ms]
shm_reclen                 255 [bytes]
shm_workspace              8192 [bytes]
srcaddr_hash               1049 [buckets]
srcaddr_ttl                30 [seconds]
thread_pool_add_delay      20 [milliseconds]
thread_pool_add_threshold  2 [requests]
thread_pool_fail_delay     200 [milliseconds]
thread_pool_max            2048 [threads]
thread_pool_min            500 [threads]
thread_pool_purge_delay    1000 [milliseconds]
thread_pool_timeout        10 [seconds]
thread_pools               4 [pools]
user                       nobody (65534)
vcl_trace                  off [bool]


stats:
        7514  Client connections accepted
       35229  Client requests received
       27328  Cache hits
          57  Cache hits for pass
        5396  Cache misses
        7901  Backend connections success
           0  Backend connections not attempted
           0  Backend connections too many
           0  Backend connections failures
           0  Backend connections reuses
           0  Backend connections recycles
           0  Backend connections unused
        5203  N struct srcaddr
        3758  N active struct srcaddr
        4520  N struct sess_mem
        4298  N struct sess
        5433  N struct object
        1623  N struct objecthead
        6923  N struct smf
           0  N small free smf
          32  N large free smf
           2  N struct vbe_conn
          19  N struct bereq
        2000  N worker threads
        2000  N worker threads created
           0  N worker threads not created
           0  N worker threads limited
           0  N queued work requests
           4  N overflowed work requests
           0  N dropped work requests
           1  N backends
          34  N expired objects
           0  N LRU nuked objects
           0  N LRU saved objects
        3239  N LRU moved objects
           0  N objects on deathrow
           0  HTTP header overflows
           0  Objects sent with sendfile
        8973  Objects sent with write
           0  Objects overflowing workspace
        7514  Total Sessions
       35225  Total Requests
           0  Total pipe
        2505  Total pass
        7899  Total fetch
     3497006  Total header bytes
    32744211  Total body bytes
         631  Session Closed
           0  Session Pipeline
           0  Session Read Ahead
           0  Session Linger
       34676  Session herd
     1399198  SHM records
       96468  SHM writes
           2  SHM flushes due to overflow
          10  SHM MTX contention
           0  SHM cycles through buffer
       10926  allocator requests
        6891  outstanding allocations
    57233408  bytes allocated
 29953720320  bytes free
           0  SMA allocator requests
           0  SMA outstanding allocations
           0  SMA outstanding bytes
           0  SMA bytes allocated
           0  SMA bytes free
           0  SMS allocator requests
           0  SMS outstanding allocations
           0  SMS outstanding bytes
           0  SMS bytes allocated
           0  SMS bytes freed
        7900  Backend requests made
           1  N vcl total
           1  N vcl available
           0  N vcl discarded
         707  N total active purges
         741  N new purges added
          34  N old purges deleted
       38968  N objects tested
      504633  N regexps tested against
           0  N duplicate purges removed
           0  HCB Lookups without lock
           0  HCB Lookups with lock
           0  HCB Inserts
           0  Objects ESI parsed (unlock)
           0  ESI parse errors (unlock)