Debugging restarts/crashes?

[Cloude Porteus]

Varnish seems to restart itself every so often, usually after a few days. We
finally installed some SSD's, which made all the difference and our system
load came down to the .02 - .2 range.

I don't currently have any logging turned on. If I did, would it have any
useful information for me?

We typically don't have more that 700k objects cached. I also never see the
worker threads go above 400. We are running on CentOS 5.

Our config:

NFILES=131072
MEMLOCK=90000
DAEMON_COREFILE_LIMIT="unlimited"
VARNISH_VCL_CONF=/etc/varnish/instructables.vcl
VARNISH_LISTEN_ADDRESS=
VARNISH_LISTEN_PORT=80
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=82
VARNISH_MIN_THREADS=400
VARNISH_MAX_THREADS=4000
VARNISH_THREAD_TIMEOUT=60
VARNISH_STORAGE_FILE=/var/lib/varnish/mmap
VARNISH_STORAGE_SIZE=50G
VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}"
VARNISH_TTL=1800

DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
             -f ${VARNISH_VCL_CONF} \
             -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT}
\
             -t ${VARNISH_TTL} \
             -w
${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \
             -u varnish -g varnish \
             -s ${VARNISH_STORAGE} \
             -p obj_workspace=4096 \
             -p sess_workspace=262144 \
             -p lru_interval=3600 \
             -p listen_depth=8192 \
             -p log_hashstring=off \
             -p sess_timeout=10 \
             -p shm_workspace=32768 \
             -p ping_interval=1 \
             -p thread_pools=4 \
             -p thread_pool_min=100 \
             -p thread_pool_max=4000 \
             -p srcaddr_ttl=0 \
             -p esi_syntax=1 "

I don't think out load is all that heavy and we have a high hitrate ratio.
Our varnishstat output.

0+11:02:14

Hitrate ratio:       10      100      133
Hitrate avg:     0.9875   0.9833   0.9832

    15370527       325.94       386.84 Client connections accepted
    15370517       325.94       386.84 Client requests received
    14604988       320.94       367.57 Cache hits
      182795         4.00         4.60 Cache hits for pass
      581426         1.00        14.63 Cache misses
      765513         5.00        19.27 Backend connections success
           1         0.00         0.00 Backend connections failures
         119          .            .   N struct sess_mem
          10          .            .   N struct sess
      447866          .            .   N struct object
      440545          .            .   N struct objecthead
      913238          .            .   N struct smf
        4490          .            .   N small free smf
           2          .            .   N large free smf
           0          .            .   N struct vbe_conn
         112          .            .   N struct bereq
         400          .            .   N worker threads
         400         0.00         0.01 N worker threads created
      313048         0.00         7.88 N worker threads limited
           1          .            .   N backends
      126326          .            .   N expired objects
     1250255          .            .   N LRU moved objects
          18         0.00         0.00 HTTP header overflows
    13645614       292.95       343.42 Objects sent with write
    15370518       327.94       386.84 Total Sessions
    15370510       327.94       386.84 Total Requests
           7         0.00         0.00 Total pipe
      191394         5.00         4.82 Total pass
      765481         9.00        19.27 Total fetch
  7252376430    156332.93    182523.19 Total header bytes
 74103505138    772233.39   1864989.81 Total body bytes
    15370160       327.94       386.83 Session Closed
         365         0.00         0.01 Session herd
   758091326     15625.09     19079.16 SHM records
    62365137      1317.75      1569.57 SHM writes
         468         0.00         0.01 SHM flushes due to overflow
         589         0.00         0.01 SHM MTX contention
         307         0.00         0.01 SHM cycles through buffer
     1573886        15.00        39.61 allocator requests
      908746          .            .   outstanding allocations
  7766196224          .            .   bytes allocated
 45920894976          .            .   bytes free


-- 
VP of Product Development
Instructables.com

http://www.instructables.com/member/lebowski
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20090513/87b9c46b/attachment-0001.html>