Varnish is restarting frequently
Sparsh Gupta
sparshgupta at gmail.com
Tue May 8 20:02:15 CEST 2012
I have a bunch of Varnish servers (Exactly same configurations, caching
different keys on Intel 2650 series Computer + 16GB RAM + 32GB SSD each). I
am enclosing my configurations and settings along with my current
varnishstats.
I am looking for more debug information and how to find the route cause of
my varnish getting restarting every 6-8 hours (all varnish servers)
Please do let me know in case you need any more information. I am loosing
all my cache (I tried 20G rather than 32G but still it restarts flushing
all the cache and becoming unavailable for split seconds). I am using them
behind nginx load balancers (which also distribute traffic among different
varnish servers based on some keys). The varnish are fueled by 4 powerful
LAMP servers.
I have high miss rate because I keep loosing all my cache
Daemon Options:
START=yes
NFILES=131072
MEMLOCK=82000
INSTANCE=$(uname -n)
VARNISH_VCL_CONF=/etc/varnish/default.vcl
VARNISH_LISTEN_ADDRESS=10.56.140.12
VARNISH_LISTEN_PORT=80
VARNISH_ADMIN_LISTEN_ADDRESS=10.56.140.12
VARNISH_ADMIN_LISTEN_PORT=6082
VARNISH_MIN_THREADS=200
VARNISH_MAX_THREADS=4000
VARNISH_THREAD_TIMEOUT=60
VARNISH_STORAGE_SIZE=32G
VARNISH_STORAGE_MALLOC="malloc"
VARNISH_USER="varnish"
VARNISH_GROUP="varnish"
DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
-f ${VARNISH_VCL_CONF} \
-T
${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
-w
${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \
-s ${VARNISH_STORAGE_MALLOC},${VARNISH_STORAGE_SIZE} \
-u ${VARNISH_USER} -g ${VARNISH_GROUP} \
-i ${INSTANCE} \
-p thread_pool_add_delay=2 \
-p thread_pools=32 \
-p thread_pool_min=25 \
-p thread_pool_max=4000 \
-p sess_timeout=3 \
-p session_max=1000000 \
-p session_linger=2000"
Varnishstat -1:
client_conn 49815 26.78 Client connections accepted
client_drop 0 0.00 Connection dropped, no sess/wrk
client_req 1822206 979.68 Client requests received
cache_hit 253196 136.13 Cache hits
cache_hitpass 1379 0.74 Cache hits for pass
cache_miss 1567631 842.81 Cache misses
backend_conn 16163 8.69 Backend conn. success
backend_unhealthy 0 0.00 Backend conn. not attempted
backend_busy 0 0.00 Backend conn. too many
backend_fail 1264 0.68 Backend conn. failures
backend_reuse 1552269 834.55 Backend conn. reuses
backend_toolate 3 0.00 Backend conn. was closed
backend_recycle 1553574 835.25 Backend conn. recycles
backend_retry 0 0.00 Backend conn. retry
fetch_head 0 0.00 Fetch head
fetch_length 1567981 843.00 Fetch with Length
fetch_chunked 394 0.21 Fetch chunked
fetch_eof 0 0.00 Fetch EOF
fetch_bad 0 0.00 Fetch had bad headers
fetch_close 0 0.00 Fetch wanted close
fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed
fetch_zero 0 0.00 Fetch zero len
fetch_failed 0 0.00 Fetch failed
fetch_1xx 0 0.00 Fetch no body (1xx)
fetch_204 0 0.00 Fetch no body (204)
fetch_304 0 0.00 Fetch no body (304)
n_sess_mem 968 . N struct sess_mem
n_sess 209 . N struct sess
n_object 1565359 . N struct object
n_vampireobject 0 . N unresurrected objects
n_objectcore 1565551 . N struct objectcore
n_objecthead 1565798 . N struct objecthead
n_waitinglist 965 . N struct waitinglist
n_vbc 19 . N struct vbc
n_wrk 800 . N worker threads
n_wrk_create 836 0.45 N worker threads created
n_wrk_failed 0 0.00 N worker threads not created
n_wrk_max 0 0.00 N worker threads limited
n_wrk_lqueue 0 0.00 work request queue length
n_wrk_queued 210 0.11 N queued work requests
n_wrk_drop 0 0.00 N dropped work requests
n_backend 4 . N backends
n_expired 1637 . N expired objects
n_lru_nuked 0 . N LRU nuked objects
n_lru_moved 177419 . N LRU moved objects
losthdr 0 0.00 HTTP header overflows
n_objsendfile 0 0.00 Objects sent with sendfile
n_objwrite 1817561 977.18 Objects sent with write
n_objoverflow 0 0.00 Objects overflowing workspace
s_sess 49763 26.75 Total Sessions
s_req 1822206 979.68 Total Requests
s_pipe 0 0.00 Total pipe
s_pass 1379 0.74 Total pass
s_fetch 1568375 843.21 Total fetch
s_hdrbytes 543007555 291939.55 Total header bytes
s_bodybytes 366746858 197175.73 Total body bytes
sess_closed 1973 1.06 Session Closed
sess_pipeline 0 0.00 Session Pipeline
sess_readahead 0 0.00 Session Read Ahead
sess_linger 1821411 979.25 Session Linger
sess_herd 50288 27.04 Session herd
shm_records 132731919 71361.25 SHM records
shm_writes 5168655 2778.85 SHM writes
shm_flushes 0 0.00 SHM flushes due to overflow
shm_cont 31135 16.74 SHM MTX contention
shm_cycles 63 0.03 SHM cycles through buffer
sms_nreq 635 0.34 SMS allocator requests
sms_nobj 0 . SMS outstanding allocations
sms_nbytes 0 . SMS outstanding bytes
sms_balloc 265430 . SMS bytes allocated
sms_bfree 265430 . SMS bytes freed
backend_req 1569626 843.88 Backend requests made
n_vcl 2 0.00 N vcl total
n_vcl_avail 2 0.00 N vcl available
n_vcl_discard 0 0.00 N vcl discarded
n_ban 48 . N total active bans
n_ban_add 49 0.03 N new bans added
n_ban_retire 1 0.00 N old bans deleted
n_ban_obj_test 82431 44.32 N objects tested
n_ban_re_test 211534 113.73 N regexps tested against
n_ban_dups 35 0.02 N duplicate bans removed
hcb_nolock 1822862 980.03 HCB Lookups without lock
hcb_lock 1568249 843.14 HCB Lookups with lock
hcb_insert 1568249 843.14 HCB Inserts
esi_errors 0 0.00 ESI parse errors (unlock)
esi_warnings 0 0.00 ESI parse warnings (unlock)
accept_fail 0 0.00 Accept failures
client_drop_late 0 0.00 Connection dropped late
uptime 1860 1.00 Client uptime
dir_dns_lookups 0 0.00 DNS director lookups
dir_dns_failed 0 0.00 DNS director failed lookups
dir_dns_hit 0 0.00 DNS director cached lookups hit
dir_dns_cache_full 0 0.00 DNS director full dnscache
vmods 0 . Loaded VMODs
n_gzip 0 0.00 Gzip operations
n_gunzip 3377737 1815.99 Gunzip operations
LCK.sms.creat 4 0.00 Created locks
LCK.sms.destroy 0 0.00 Destroyed locks
LCK.sms.locks 360546 193.84 Lock Operations
LCK.sms.colls 0 0.00 Collisions
LCK.smp.creat 0 0.00 Created locks
LCK.smp.destroy 0 0.00 Destroyed locks
LCK.smp.locks 0 0.00 Lock Operations
LCK.smp.colls 0 0.00 Collisions
LCK.sma.creat 8 0.00 Created locks
LCK.sma.destroy 0 0.00 Destroyed locks
LCK.sma.locks 223989728 120424.58 Lock Operations
LCK.sma.colls 0 0.00 Collisions
LCK.smf.creat 0 0.00 Created locks
LCK.smf.destroy 0 0.00 Destroyed locks
LCK.smf.locks 0 0.00 Lock Operations
LCK.smf.colls 0 0.00 Collisions
LCK.hsl.creat 0 0.00 Created locks
LCK.hsl.destroy 0 0.00 Destroyed locks
LCK.hsl.locks 0 0.00 Lock Operations
LCK.hsl.colls 0 0.00 Collisions
LCK.hcb.creat 4 0.00 Created locks
LCK.hcb.destroy 0 0.00 Destroyed locks
LCK.hcb.locks 74720192 40172.15 Lock Operations
LCK.hcb.colls 0 0.00 Collisions
LCK.hcl.creat 0 0.00 Created locks
LCK.hcl.destroy 0 0.00 Destroyed locks
LCK.hcl.locks 0 0.00 Lock Operations
LCK.hcl.colls 0 0.00 Collisions
LCK.vcl.creat 4 0.00 Created locks
LCK.vcl.destroy 0 0.00 Destroyed locks
LCK.vcl.locks 19211 10.33 Lock Operations
LCK.vcl.colls 0 0.00 Collisions
LCK.stat.creat 4 0.00 Created locks
LCK.stat.destroy 0 0.00 Destroyed locks
LCK.stat.locks 6926 3.72 Lock Operations
LCK.stat.colls 0 0.00 Collisions
LCK.sessmem.creat 4 0.00 Created locks
LCK.sessmem.destroy 0 0.00 Destroyed locks
LCK.sessmem.locks 2970656 1597.13 Lock Operations
LCK.sessmem.colls 0 0.00 Collisions
LCK.wstat.creat 4 0.00 Created locks
LCK.wstat.destroy 0 0.00 Destroyed locks
LCK.wstat.locks 7972177 4286.12 Lock Operations
LCK.wstat.colls 0 0.00 Collisions
LCK.herder.creat 4 0.00 Created locks
LCK.herder.destroy 0 0.00 Destroyed locks
LCK.herder.locks 6214 3.34 Lock Operations
LCK.herder.colls 0 0.00 Collisions
LCK.wq.creat 128 0.07 Created locks
LCK.wq.destroy 0 0.00 Destroyed locks
LCK.wq.locks 9782145 5259.22 Lock Operations
LCK.wq.colls 0 0.00 Collisions
LCK.objhdr.creat 74421535 40011.58 Created locks
LCK.objhdr.destroy 271865 146.16 Destroyed locks
LCK.objhdr.locks 349625261 187970.57 Lock Operations
LCK.objhdr.colls 0 0.00 Collisions
LCK.exp.creat 4 0.00 Created locks
LCK.exp.destroy 0 0.00 Destroyed locks
LCK.exp.locks 74639057 40128.53 Lock Operations
LCK.exp.colls 0 0.00 Collisions
LCK.lru.creat 8 0.00 Created locks
LCK.lru.destroy 0 0.00 Destroyed locks
LCK.lru.locks 74353412 39974.95 Lock Operations
LCK.lru.colls 0 0.00 Collisions
LCK.cli.creat 4 0.00 Created locks
LCK.cli.destroy 0 0.00 Destroyed locks
LCK.cli.locks 32390 17.41 Lock Operations
LCK.cli.colls 0 0.00 Collisions
LCK.ban.creat 4 0.00 Created locks
LCK.ban.destroy 0 0.00 Destroyed locks
LCK.ban.locks 78392714 42146.62 Lock Operations
LCK.ban.colls 0 0.00 Collisions
LCK.vbp.creat 4 0.00 Created locks
LCK.vbp.destroy 0 0.00 Destroyed locks
LCK.vbp.locks 92827 49.91 Lock Operations
LCK.vbp.colls 0 0.00 Collisions
LCK.vbe.creat 4 0.00 Created locks
LCK.vbe.destroy 0 0.00 Destroyed locks
LCK.vbe.locks 4095777 2202.03 Lock Operations
LCK.vbe.colls 0 0.00 Collisions
LCK.backend.creat 16 0.01 Created locks
LCK.backend.destroy 0 0.00 Destroyed locks
LCK.backend.locks 227718873 122429.50 Lock Operations
LCK.backend.colls 0 0.00 Collisions
SMA.s0.c_req 3136117 1686.08 Allocator requests
SMA.s0.c_fail 0 0.00 Allocator failures
SMA.s0.c_bytes 206790032960 111177437.08 Bytes allocated
SMA.s0.c_freed 205374002652 110416130.46 Bytes freed
SMA.s0.g_alloc 3132208 . Allocations outstanding
SMA.s0.g_bytes 1416030308 . Bytes outstanding
SMA.s0.g_space 32943708060 . Bytes available
SMA.Transient.c_req 2758 1.48 Allocator requests
SMA.Transient.c_fail 0 0.00 Allocator failures
SMA.Transient.c_bytes 56648986 30456.44 Bytes allocated
SMA.Transient.c_freed 56648986 30456.44 Bytes freed
SMA.Transient.g_alloc 0 . Allocations outstanding
SMA.Transient.g_bytes 0 . Bytes outstanding
SMA.Transient.g_space 0 . Bytes available
VBE.dev1(10.56.140.8,,80).vcls 8 . VCL references
VBE.dev1(10.56.140.8,,80).happy18446744073709551615 . Happy
health probes
VBE.dev2(10.56.140.2,,80).vcls 8 . VCL references
VBE.dev2(10.56.140.2,,80).happy18446744073709551615 . Happy
health probes
VBE.dev3(10.56.140.4,,80).vcls 8 . VCL references
VBE.dev3(10.56.140.4,,80).happy18446744073709551615 . Happy
health probes
VBE.dev4(10.56.140.6,,80).vcls 8 . VCL references
VBE.dev4(10.56.140.6,,80).happy18446744073709551615 . Happy
health probes
Thanks
Sparsh Gupta
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20120508/b13607a5/attachment-0001.html>
More information about the varnish-misc
mailing list