Varnish is restarting frequently

Sparsh Gupta sparshgupta at gmail.com
Tue May 8 20:02:15 CEST 2012


I have a bunch of Varnish servers (Exactly same configurations, caching
different keys on Intel 2650 series Computer + 16GB RAM + 32GB SSD each). I
am enclosing my configurations and settings along with my current
varnishstats.

I am looking for more debug information and how to find the route cause of
my varnish getting restarting every 6-8 hours (all varnish servers)

Please do let me know in case you need any more information. I am loosing
all my cache (I tried 20G rather than 32G but still it restarts flushing
all the cache and becoming unavailable for split seconds). I am using them
behind nginx load balancers (which also distribute traffic among different
varnish servers based on some keys). The varnish are fueled by 4 powerful
LAMP servers.

I have high miss rate because I keep loosing all my cache

Daemon Options:
START=yes
NFILES=131072
MEMLOCK=82000
INSTANCE=$(uname -n)
VARNISH_VCL_CONF=/etc/varnish/default.vcl
VARNISH_LISTEN_ADDRESS=10.56.140.12
VARNISH_LISTEN_PORT=80
VARNISH_ADMIN_LISTEN_ADDRESS=10.56.140.12
VARNISH_ADMIN_LISTEN_PORT=6082
VARNISH_MIN_THREADS=200
VARNISH_MAX_THREADS=4000
VARNISH_THREAD_TIMEOUT=60
VARNISH_STORAGE_SIZE=32G
VARNISH_STORAGE_MALLOC="malloc"
VARNISH_USER="varnish"
VARNISH_GROUP="varnish"

DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
             -f ${VARNISH_VCL_CONF} \
             -T
${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
             -w
${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \
             -s ${VARNISH_STORAGE_MALLOC},${VARNISH_STORAGE_SIZE} \
             -u ${VARNISH_USER} -g ${VARNISH_GROUP} \
             -i ${INSTANCE} \
             -p thread_pool_add_delay=2 \
             -p thread_pools=32 \
             -p thread_pool_min=25 \
             -p thread_pool_max=4000 \
             -p sess_timeout=3 \
             -p session_max=1000000 \
             -p session_linger=2000"


Varnishstat -1:
client_conn              49815        26.78 Client connections accepted
client_drop                  0         0.00 Connection dropped, no sess/wrk
client_req             1822206       979.68 Client requests received
cache_hit               253196       136.13 Cache hits
cache_hitpass             1379         0.74 Cache hits for pass
cache_miss             1567631       842.81 Cache misses
backend_conn             16163         8.69 Backend conn. success
backend_unhealthy            0         0.00 Backend conn. not attempted
backend_busy                 0         0.00 Backend conn. too many
backend_fail              1264         0.68 Backend conn. failures
backend_reuse          1552269       834.55 Backend conn. reuses
backend_toolate              3         0.00 Backend conn. was closed
backend_recycle        1553574       835.25 Backend conn. recycles
backend_retry                0         0.00 Backend conn. retry
fetch_head                   0         0.00 Fetch head
fetch_length           1567981       843.00 Fetch with Length
fetch_chunked              394         0.21 Fetch chunked
fetch_eof                    0         0.00 Fetch EOF
fetch_bad                    0         0.00 Fetch had bad headers
fetch_close                  0         0.00 Fetch wanted close
fetch_oldhttp                0         0.00 Fetch pre HTTP/1.1 closed
fetch_zero                   0         0.00 Fetch zero len
fetch_failed                 0         0.00 Fetch failed
fetch_1xx                    0         0.00 Fetch no body (1xx)
fetch_204                    0         0.00 Fetch no body (204)
fetch_304                    0         0.00 Fetch no body (304)
n_sess_mem                 968          .   N struct sess_mem
n_sess                     209          .   N struct sess
n_object               1565359          .   N struct object
n_vampireobject              0          .   N unresurrected objects
n_objectcore           1565551          .   N struct objectcore
n_objecthead           1565798          .   N struct objecthead
n_waitinglist              965          .   N struct waitinglist
n_vbc                       19          .   N struct vbc
n_wrk                      800          .   N worker threads
n_wrk_create               836         0.45 N worker threads created
n_wrk_failed                 0         0.00 N worker threads not created
n_wrk_max                    0         0.00 N worker threads limited
n_wrk_lqueue                 0         0.00 work request queue length
n_wrk_queued               210         0.11 N queued work requests
n_wrk_drop                   0         0.00 N dropped work requests
n_backend                    4          .   N backends
n_expired                 1637          .   N expired objects
n_lru_nuked                  0          .   N LRU nuked objects
n_lru_moved             177419          .   N LRU moved objects
losthdr                      0         0.00 HTTP header overflows
n_objsendfile                0         0.00 Objects sent with sendfile
n_objwrite             1817561       977.18 Objects sent with write
n_objoverflow                0         0.00 Objects overflowing workspace
s_sess                   49763        26.75 Total Sessions
s_req                  1822206       979.68 Total Requests
s_pipe                       0         0.00 Total pipe
s_pass                    1379         0.74 Total pass
s_fetch                1568375       843.21 Total fetch
s_hdrbytes           543007555    291939.55 Total header bytes
s_bodybytes          366746858    197175.73 Total body bytes
sess_closed               1973         1.06 Session Closed
sess_pipeline                0         0.00 Session Pipeline
sess_readahead               0         0.00 Session Read Ahead
sess_linger            1821411       979.25 Session Linger
sess_herd                50288        27.04 Session herd
shm_records          132731919     71361.25 SHM records
shm_writes             5168655      2778.85 SHM writes
shm_flushes                  0         0.00 SHM flushes due to overflow
shm_cont                 31135        16.74 SHM MTX contention
shm_cycles                  63         0.03 SHM cycles through buffer
sms_nreq                   635         0.34 SMS allocator requests
sms_nobj                     0          .   SMS outstanding allocations
sms_nbytes                   0          .   SMS outstanding bytes
sms_balloc              265430          .   SMS bytes allocated
sms_bfree               265430          .   SMS bytes freed
backend_req            1569626       843.88 Backend requests made
n_vcl                        2         0.00 N vcl total
n_vcl_avail                  2         0.00 N vcl available
n_vcl_discard                0         0.00 N vcl discarded
n_ban                       48          .   N total active bans
n_ban_add                   49         0.03 N new bans added
n_ban_retire                 1         0.00 N old bans deleted
n_ban_obj_test           82431        44.32 N objects tested
n_ban_re_test           211534       113.73 N regexps tested against
n_ban_dups                  35         0.02 N duplicate bans removed
hcb_nolock             1822862       980.03 HCB Lookups without lock
hcb_lock               1568249       843.14 HCB Lookups with lock
hcb_insert             1568249       843.14 HCB Inserts
esi_errors                   0         0.00 ESI parse errors (unlock)
esi_warnings                 0         0.00 ESI parse warnings (unlock)
accept_fail                  0         0.00 Accept failures
client_drop_late             0         0.00 Connection dropped late
uptime                    1860         1.00 Client uptime
dir_dns_lookups              0         0.00 DNS director lookups
dir_dns_failed               0         0.00 DNS director failed lookups
dir_dns_hit                  0         0.00 DNS director cached lookups hit
dir_dns_cache_full           0         0.00 DNS director full dnscache
vmods                        0          .   Loaded VMODs
n_gzip                       0         0.00 Gzip operations
n_gunzip               3377737      1815.99 Gunzip operations
LCK.sms.creat                4         0.00 Created locks
LCK.sms.destroy              0         0.00 Destroyed locks
LCK.sms.locks           360546       193.84 Lock Operations
LCK.sms.colls                0         0.00 Collisions
LCK.smp.creat                0         0.00 Created locks
LCK.smp.destroy              0         0.00 Destroyed locks
LCK.smp.locks                0         0.00 Lock Operations
LCK.smp.colls                0         0.00 Collisions
LCK.sma.creat                8         0.00 Created locks
LCK.sma.destroy              0         0.00 Destroyed locks
LCK.sma.locks        223989728    120424.58 Lock Operations
LCK.sma.colls                0         0.00 Collisions
LCK.smf.creat                0         0.00 Created locks
LCK.smf.destroy              0         0.00 Destroyed locks
LCK.smf.locks                0         0.00 Lock Operations
LCK.smf.colls                0         0.00 Collisions
LCK.hsl.creat                0         0.00 Created locks
LCK.hsl.destroy              0         0.00 Destroyed locks
LCK.hsl.locks                0         0.00 Lock Operations
LCK.hsl.colls                0         0.00 Collisions
LCK.hcb.creat                4         0.00 Created locks
LCK.hcb.destroy              0         0.00 Destroyed locks
LCK.hcb.locks         74720192     40172.15 Lock Operations
LCK.hcb.colls                0         0.00 Collisions
LCK.hcl.creat                0         0.00 Created locks
LCK.hcl.destroy              0         0.00 Destroyed locks
LCK.hcl.locks                0         0.00 Lock Operations
LCK.hcl.colls                0         0.00 Collisions
LCK.vcl.creat                4         0.00 Created locks
LCK.vcl.destroy              0         0.00 Destroyed locks
LCK.vcl.locks            19211        10.33 Lock Operations
LCK.vcl.colls                0         0.00 Collisions
LCK.stat.creat               4         0.00 Created locks
LCK.stat.destroy             0         0.00 Destroyed locks
LCK.stat.locks            6926         3.72 Lock Operations
LCK.stat.colls               0         0.00 Collisions
LCK.sessmem.creat            4         0.00 Created locks
LCK.sessmem.destroy            0         0.00 Destroyed locks
LCK.sessmem.locks        2970656      1597.13 Lock Operations
LCK.sessmem.colls              0         0.00 Collisions
LCK.wstat.creat                4         0.00 Created locks
LCK.wstat.destroy              0         0.00 Destroyed locks
LCK.wstat.locks          7972177      4286.12 Lock Operations
LCK.wstat.colls                0         0.00 Collisions
LCK.herder.creat               4         0.00 Created locks
LCK.herder.destroy             0         0.00 Destroyed locks
LCK.herder.locks            6214         3.34 Lock Operations
LCK.herder.colls               0         0.00 Collisions
LCK.wq.creat                 128         0.07 Created locks
LCK.wq.destroy                 0         0.00 Destroyed locks
LCK.wq.locks             9782145      5259.22 Lock Operations
LCK.wq.colls                   0         0.00 Collisions
LCK.objhdr.creat        74421535     40011.58 Created locks
LCK.objhdr.destroy        271865       146.16 Destroyed locks
LCK.objhdr.locks       349625261    187970.57 Lock Operations
LCK.objhdr.colls               0         0.00 Collisions
LCK.exp.creat                  4         0.00 Created locks
LCK.exp.destroy                0         0.00 Destroyed locks
LCK.exp.locks           74639057     40128.53 Lock Operations
LCK.exp.colls                  0         0.00 Collisions
LCK.lru.creat                  8         0.00 Created locks
LCK.lru.destroy                0         0.00 Destroyed locks
LCK.lru.locks           74353412     39974.95 Lock Operations
LCK.lru.colls                  0         0.00 Collisions
LCK.cli.creat                  4         0.00 Created locks
LCK.cli.destroy                0         0.00 Destroyed locks
LCK.cli.locks              32390        17.41 Lock Operations
LCK.cli.colls                  0         0.00 Collisions
LCK.ban.creat                  4         0.00 Created locks
LCK.ban.destroy                0         0.00 Destroyed locks
LCK.ban.locks           78392714     42146.62 Lock Operations
LCK.ban.colls                  0         0.00 Collisions
LCK.vbp.creat                  4         0.00 Created locks
LCK.vbp.destroy                0         0.00 Destroyed locks
LCK.vbp.locks              92827        49.91 Lock Operations
LCK.vbp.colls                  0         0.00 Collisions
LCK.vbe.creat                  4         0.00 Created locks
LCK.vbe.destroy                0         0.00 Destroyed locks
LCK.vbe.locks            4095777      2202.03 Lock Operations
LCK.vbe.colls                  0         0.00 Collisions
LCK.backend.creat             16         0.01 Created locks
LCK.backend.destroy            0         0.00 Destroyed locks
LCK.backend.locks      227718873    122429.50 Lock Operations
LCK.backend.colls              0         0.00 Collisions
SMA.s0.c_req             3136117      1686.08 Allocator requests
SMA.s0.c_fail                  0         0.00 Allocator failures
SMA.s0.c_bytes      206790032960 111177437.08 Bytes allocated
SMA.s0.c_freed      205374002652 110416130.46 Bytes freed
SMA.s0.g_alloc           3132208          .   Allocations outstanding
SMA.s0.g_bytes        1416030308          .   Bytes outstanding
SMA.s0.g_space       32943708060          .   Bytes available
SMA.Transient.c_req         2758         1.48 Allocator requests
SMA.Transient.c_fail           0         0.00 Allocator failures
SMA.Transient.c_bytes     56648986     30456.44 Bytes allocated
SMA.Transient.c_freed     56648986     30456.44 Bytes freed
SMA.Transient.g_alloc            0          .   Allocations outstanding
SMA.Transient.g_bytes            0          .   Bytes outstanding
SMA.Transient.g_space            0          .   Bytes available
VBE.dev1(10.56.140.8,,80).vcls            8          .   VCL references
VBE.dev1(10.56.140.8,,80).happy18446744073709551615          .   Happy
health probes
VBE.dev2(10.56.140.2,,80).vcls            8          .   VCL references
VBE.dev2(10.56.140.2,,80).happy18446744073709551615          .   Happy
health probes
VBE.dev3(10.56.140.4,,80).vcls            8          .   VCL references
VBE.dev3(10.56.140.4,,80).happy18446744073709551615          .   Happy
health probes
VBE.dev4(10.56.140.6,,80).vcls            8          .   VCL references
VBE.dev4(10.56.140.6,,80).happy18446744073709551615          .   Happy
health probes

Thanks
Sparsh Gupta
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20120508/b13607a5/attachment-0001.html>


More information about the varnish-misc mailing list