misc connection closures

[Janis Putrams]

Hi!

As this is my first post to this list I would like to thank to all developers of this excellent software.

We put our images behind varnish-1.0.4 serving ~1.5k req/sec and load is staying low which is great.

Though when reaching 2k req/sec we start to encounter connection closures. Connection gets closed immediately after connection is established.

I have raised thread limits but that did not help. I would look into logs if this happened for another application but I am not really sure where to look for varnish.

my configuration:
05:56:16 janis[pts/2]@host ~$ telnet localhost 81
Trying 127.0.0.1.81...
Connected to localhost.
Escape character is '^]'.
param.show
200 709
default_ttl          120 [seconds]
thread_pools         5 [pools]
thread_pool_max      100000 [threads]
thread_pool_min      100 [threads]
thread_pool_timeout  10 [seconds]
overflow_max         100 [%]
http_workspace       8192 [bytes]
sess_timeout         5 [seconds]
pipe_timeout         60 [seconds]
send_timeout         600 [seconds]
auto_restart         on [bool]
fetch_chunksize      128 [kilobytes]
sendfile_threshold   unlimited [bytes]
vcl_trace            off [bool]
listen_address       :80
listen_depth         4096 [connections]
srcaddr_hash         1049 [buckets]
srcaddr_ttl          30 [seconds]
backend_http11       off [bool]
client_http11        off [bool]
ping_interval        3 [seconds]

varnishstat:
00:01:22
Hitrate ratio:       10       19       19
Hitrate avg:     1.0059   1.0096   1.0096

       82740      1016.84      1009.02 Client connections accepted
      182469      2259.17      2225.23 Client requests received
      180928      2256.78      2206.44 Cache hits
           0         0.00         0.00 Cache hits for pass
        1541         2.40        18.79 Cache misses
        1541         2.40        18.79 Backend connections success
           2         0.00         0.02 Backend connections failures
        1507         2.40        18.38 Backend connections reuses
        1511         2.40        18.43 Backend connections recycles
           3         0.00         0.04 Backend connections unused
        1614         6.30        19.68 N struct srcaddr
         311        -3.70         3.79 N active struct srcaddr
         906         0.00        11.05 N struct sess_mem
         875        -1.00        10.67 N struct sess
        1604         2.60        19.56 N struct object
        1604         2.60        19.56 N struct objecthead
        1523         2.40        18.57 N struct smf
           0         0.00         0.00 N small free smf
           2         0.00         0.02 N large free smf
           6         0.00         0.07 N struct vbe_conn
          63         0.20         0.77 N worker threads
          63         0.20         0.77 N worker threads created
           0         0.00         0.00 N worker threads not created
           0         0.00         0.00 N worker threads limited
           0         0.00         0.00 N queued work requests
          63         0.20         0.77 N overflowed work requests
           0         0.00         0.00 N dropped work requests
           0         0.00         0.00 N expired objects
           0         0.00         0.00 N objects on deathrow
           0         0.00         0.00 HTTP header overflows
           0         0.00         0.00 Objects sent with sendfile
       95445      1159.97      1163.96 Objects sent with write
       77391       928.18       943.79 Total Sessions
      182469      2259.37      2225.23 Total Requests
           0         0.00         0.00 Total pipe
           0         0.00         0.00 Total pass
        1541         2.40        18.79 Total fetch
    38575489    476006.89    470432.79 Total header bytes
  7861511491  91999201.08  95872091.35 Total body bytes
       61101       712.18       745.13 Session Closed
         272         5.80         3.32 Session Pipeline
         137         3.00         1.67 Session Read Ahead
      121024      1539.50      1475.90 Session herd
     5833297     71817.11     71137.77 SHM records
      425680      5241.62      5191.22 SHM writes
         128         2.70         1.56 SHM MTX contention      

netstat summary:
      FIN_WAIT1: 93
      TIME_WAIT: 6825
     CLOSE_WAIT: 5
       LAST_ACK: 7
      FIN_WAIT2: 462
    ESTABLISHED: 751
       SYN_RECV: 112
         LISTEN: 7

load:
06:05:55 root[pts/1]@host ~# uptime
 06:05:58 up 52 days, 20:27,  4 users,  load average: 0.55, 0.39, 0.31

process info:
06:19:35 root[pts/1]@host ~/install# ps auxf|egrep [v]arnish
root     25226  0.0  0.0  10020   740 ?        Ss   05:33   0:00 varnishd -f /etc/varnishd/varnishd.conf -T 127.0.0.1:81 -P /var/run/varnishd.pid -s file,/var/spool/varnishd/varnishd_storage.bin,1574176153 -p thread_pools 5 -p listen_depth 4096 -w100,100000,10
root     26973 14.0  0.6 2282680 56608 ?       Sl   06:03   2:13  \_ varnishd -f /etc/varnishd/varnishd.conf -T 127.0.0.1:81 -P /var/run/varnishd.pid -s file,/var/spool/varnishd/varnishd_storage.bin,1574176153 -p thread_pools 5 -p listen_depth 4096 -w100,100000,10

06:19:42 root[pts/1]@host ~/install# uname -a
Linux host 2.6.16.52-1smp #1 SMP Thu May 31 19:32:54 CEST 2007 i686 Intel(R)_Xeon(R)_CPU____________5150__ at _2.66GHz Linux

06:23:04 root[pts/1]@host ~/install# free
             total       used       free     shared    buffers     cached
Mem:          8114        721       7392          0        306        314
-/+ buffers/cache:        101       8013
Swap:         9765          0       9765


connection closure:
06:05:58 root[pts/1]@host ~# telnet host.mydomain.lv 80
Trying 11.11.11.11.80...
Connected to 11.11.11.11.
Escape character is '^]'.
Connection closed by foreign host.

Also in varnishstat Client counters gets reset from time to time. Don't know if it is problem or not.

Thank you in advance.

--
Janis Putrams