Varnish high thread problem，asking for help！
杨尧
yao.yang at autonavi.com
Tue Jun 5 18:26:08 CEST 2012
Hello，I have used Varnish3.0.2 for my work several months。It works fine until yesterday，here is some data statistics.
I couldn't find some error, how to trace the error with tools which can be easy used.
I need your help, thanks!
I run varnish with default run parameters.
/usr/local/varnish/sbin/varnishd -f /usr/local/varnish/etc/varnish/default.vcl -s malloc,36G -T 127.0.0.1:2000 -a 0.0.0.0:80
thread statistics of this week
[cid:_Foxmail.0 at 9747951E-1276-4891-AD2C-B97EC26F4784]
this is I used Varnishstat command for this log, you can see high thread below :
client_conn 3004906465 149.74 Client connections accepted
client_drop 15826 0.00 Connection dropped, no sess/wrk
client_req 7248344267 361.19 Client requests received
cache_hit 6890494918 343.36 Cache hits
cache_hitpass 237554 0.01 Cache hits for pass
cache_miss 14569361 0.73 Cache misses
backend_conn 2376506 0.12 Backend conn. success
backend_unhealthy 0 0.00 Backend conn. not attempted
backend_busy 0 0.00 Backend conn. too many
backend_fail 287 0.00 Backend conn. failures
backend_reuse 355143346 17.70 Backend conn. reuses
backend_toolate 220621 0.01 Backend conn. was closed
backend_recycle 355381547 17.71 Backend conn. recycles
backend_retry 1789 0.00 Backend conn. retry
fetch_head 532 0.00 Fetch head
fetch_length 127549059 6.36 Fetch with Length
fetch_chunked 74737573 3.72 Fetch chunked
fetch_eof 0 0.00 Fetch EOF
fetch_bad 0 0.00 Fetch had bad headers
fetch_close 398480 0.02 Fetch wanted close
fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed
fetch_zero 0 0.00 Fetch zero len
fetch_failed 33 0.00 Fetch failed
fetch_1xx 0 0.00 Fetch no body (1xx)
fetch_204 0 0.00 Fetch no body (204)
fetch_304 154791965 7.71 Fetch no body (304)
n_sess_mem 100000 . N struct sess_mem
n_sess 573047 . N struct sess
n_object 52928 . N struct object
n_vampireobject 0 . N unresurrected objects
n_objectcore 53843 . N struct objectcore
n_objecthead 53997 . N struct objecthead
n_waitinglist 263224 . N struct waitinglist
n_vbc 3 . N struct vbc
n_wrk 919 . N worker threads
n_wrk_create 207942 0.01 N worker threads created
n_wrk_failed 0 0.00 N worker threads not created
n_wrk_max 56590 0.00 N worker threads limited
n_wrk_lqueue 0 0.00 work request queue length
n_wrk_queued 2321505 0.12 N queued work requests
n_wrk_drop 16055 0.00 N dropped work requests
n_backend 2 . N backends
n_expired 14516344 . N expired objects
n_lru_nuked 0 . N LRU nuked objects
n_lru_moved 1228756392 . N LRU moved objects
losthdr 245059 0.01 HTTP header overflows
n_objsendfile 0 0.00 Objects sent with sendfile
n_objwrite 5263813707 262.30 Objects sent with write
n_objoverflow 0 0.00 Objects overflowing workspace
s_sess 3004890148 149.74 Total Sessions
s_req 7248344267 361.19 Total Requests
s_pipe 57317 0.00 Total pipe
s_pass 342909497 17.09 Total pass
s_fetch 357477576 17.81 Total fetch
s_hdrbytes 2417774736372 120479.68 Total header bytes
s_bodybytes 114294635264274 5695394.48 Total body bytes
sess_closed 486804044 24.26 Session Closed
sess_pipeline 1544047 0.08 Session Pipeline
sess_readahead 379428 0.02 Session Read Ahead
sess_linger 7012651760 349.45 Session Linger
sess_herd 6537770899 325.78 Session herd
shm_records 317672910554 15829.90 SHM records
shm_writes 23635386981 1177.77 SHM writes
shm_flushes 0 0.00 SHM flushes due to overflow
shm_cont 401864660 20.03 SHM MTX contention
shm_cycles 125523 0.01 SHM cycles through buffer
sms_nreq 246351 0.01 SMS allocator requests
sms_nobj 0 . SMS outstanding allocations
sms_nbytes 0 . SMS outstanding bytes
sms_balloc 97270490 . SMS bytes allocated
sms_bfree 97270490 . SMS bytes freed
backend_req 357478937 17.81 Backend requests made
n_vcl 1 0.00 N vcl total
n_vcl_avail 1 0.00 N vcl available
n_vcl_discard 0 0.00 N vcl discarded
n_ban 12 . N total active bans
n_ban_add 13 0.00 N new bans added
n_ban_retire 1 0.00 N old bans deleted
n_ban_obj_test 51233 0.00 N objects tested
n_ban_re_test 175003 0.01 N regexps tested against
n_ban_dups 5 0.00 N duplicate bans removed
hcb_nolock 6894873923 343.58 HCB Lookups without lock
hcb_lock 12193302 0.61 HCB Lookups with lock
hcb_insert 12192706 0.61 HCB Inserts
esi_errors 0 0.00 ESI parse errors (unlock)
esi_warnings 0 0.00 ESI parse warnings (unlock)
accept_fail 0 0.00 Accept failures
client_drop_late 229 0.00 Connection dropped late
uptime 20067905 1.00 Client uptime
dir_dns_lookups 0 0.00 DNS director lookups
dir_dns_failed 0 0.00 DNS director failed lookups
dir_dns_hit 0 0.00 DNS director cached lookups hit
dir_dns_cache_full 0 0.00 DNS director full dnscache
vmods 0 . Loaded VMODs
n_gzip 32483891 1.62 Gzip operations
n_gunzip 211871530 10.56 Gunzip operations
LCK.sms.creat 1 0.00 Created locks
LCK.sms.destroy 0 0.00 Destroyed locks
LCK.sms.locks 739053 0.04 Lock Operations
LCK.sms.colls 0 0.00 Collisions
LCK.smp.creat 0 0.00 Created locks
LCK.smp.destroy 0 0.00 Destroyed locks
LCK.smp.locks 0 0.00 Lock Operations
LCK.smp.colls 0 0.00 Collisions
LCK.sma.creat 2 0.00 Created locks
LCK.sma.destroy 0 0.00 Destroyed locks
LCK.sma.locks 1228521290 61.22 Lock Operations
LCK.sma.colls 0 0.00 Collisions
LCK.smf.creat 0 0.00 Created locks
LCK.smf.destroy 0 0.00 Destroyed locks
LCK.smf.locks 0 0.00 Lock Operations
LCK.smf.colls 0 0.00 Collisions
LCK.hsl.creat 0 0.00 Created locks
LCK.hsl.destroy 0 0.00 Destroyed locks
LCK.hsl.locks 0 0.00 Lock Operations
LCK.hsl.colls 0 0.00 Collisions
LCK.hcb.creat 1 0.00 Created locks
LCK.hcb.destroy 0 0.00 Destroyed locks
LCK.hcb.locks 24444497 1.22 Lock Operations
LCK.hcb.colls 0 0.00 Collisions
LCK.hcl.creat 0 0.00 Created locks
LCK.hcl.destroy 0 0.00 Destroyed locks
LCK.hcl.locks 0 0.00 Lock Operations
LCK.hcl.colls 0 0.00 Collisions
LCK.vcl.creat 1 0.00 Created locks
LCK.vcl.destroy 0 0.00 Destroyed locks
LCK.vcl.locks 606285 0.03 Lock Operations
LCK.vcl.colls 0 0.00 Collisions
LCK.stat.creat 1 0.00 Created locks
LCK.stat.destroy 0 0.00 Destroyed locks
LCK.stat.locks 100000 0.00 Lock Operations
LCK.stat.colls 0 0.00 Collisions
LCK.sessmem.creat 1 0.00 Created locks
LCK.sessmem.destroy 0 0.00 Destroyed locks
LCK.sessmem.locks 3005069436 149.75 Lock Operations
LCK.sessmem.colls 0 0.00 Collisions
LCK.wstat.creat 1 0.00 Created locks
LCK.wstat.destroy 0 0.00 Destroyed locks
LCK.wstat.locks 3824514186 190.58 Lock Operations
LCK.wstat.colls 0 0.00 Collisions
LCK.herder.creat 1 0.00 Created locks
LCK.herder.destroy 0 0.00 Destroyed locks
LCK.herder.locks 2128507 0.11 Lock Operations
LCK.herder.colls 0 0.00 Collisions
LCK.wq.creat 2 0.00 Created locks
LCK.wq.destroy 0 0.00 Destroyed locks
LCK.wq.locks 14076329081 701.43 Lock Operations
LCK.wq.colls 0 0.00 Collisions
LCK.objhdr.creat 12399795 0.62 Created locks
LCK.objhdr.destroy 12345878 0.62 Destroyed locks
LCK.objhdr.locks 27753000910 1382.95 Lock Operations
LCK.objhdr.colls 0 0.00 Collisions
LCK.exp.creat 1 0.00 Created locks
LCK.exp.destroy 0 0.00 Destroyed locks
LCK.exp.locks 49133198 2.45 Lock Operations
LCK.exp.colls 0 0.00 Collisions
LCK.lru.creat 2 0.00 Created locks
LCK.lru.destroy 0 0.00 Destroyed locks
LCK.lru.locks 14570647 0.73 Lock Operations
LCK.lru.colls 0 0.00 Collisions
LCK.cli.creat 1 0.00 Created locks
LCK.cli.destroy 0 0.00 Destroyed locks
LCK.cli.locks 6689114 0.33 Lock Operations
LCK.cli.colls 0 0.00 Collisions
LCK.ban.creat 1 0.00 Created locks
LCK.ban.destroy 0 0.00 Destroyed locks
LCK.ban.locks 1839997818 91.69 Lock Operations
LCK.ban.colls 0 0.00 Collisions
LCK.vbp.creat 1 0.00 Created locks
LCK.vbp.destroy 0 0.00 Destroyed locks
LCK.vbp.locks 0 0.00 Lock Operations
LCK.vbp.colls 0 0.00 Collisions
LCK.vbe.creat 1 0.00 Created locks
LCK.vbe.destroy 0 0.00 Destroyed locks
LCK.vbe.locks 4754121 0.24 Lock Operations
LCK.vbe.colls 0 0.00 Collisions
LCK.backend.creat 2 0.00 Created locks
LCK.backend.destroy 0 0.00 Destroyed locks
LCK.backend.locks 720311744 35.89 Lock Operations
LCK.backend.colls 0 0.00 Collisions
SMA.s0.nreq 29419933 1.47 Allocator requests
SMA.s0.nobj 107799 . Outstanding allocations
SMA.s0.nbytes 1652918591 . Outstanding bytes
SMA.s0.balloc 1249441221543 . Bytes allocated
SMA.s0.bfree 1247788302952 . Bytes free
SMA.Transient.nreq 531092059 26.46 Allocator requests
SMA.Transient.nobj 0 . Outstanding allocations
SMA.Transient.nbytes 0 . Outstanding bytes
SMA.Transient.balloc13406920173393 . Bytes allocated
SMA.Transient.bfree 13406920173393 . Bytes free
VBE.default(127.0.0.1,,8080).vcls 1 . VCL references
VBE.default(127.0.0.1,,8080).happy 0 . Happy health probes
VBE.api_mapabc(127.0.0.1,,8080).vcls 1 . VCL references
VBE.api_mapabc(127.0.0.1,,8080).happy 0 . Happy health probes
________________________________
[cid:image001.jpg at 01CC55C0.2961D160]
云中心-运维部 杨尧
Tel: 010-84107622
Mobile: 15011304105
MSN:spider_sea at 163.com
Mail: yao.yang at autonavi.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20120605/97e0c603/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Catch8D21.jpg
Type: image/jpeg
Size: 119210 bytes
Desc: Catch8D21.jpg
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20120605/97e0c603/attachment-0002.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 2016_image002(12-22-11-54-48).jpg
Type: image/jpeg
Size: 2016 bytes
Desc: 2016_image002(12-22-11-54-48).jpg
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20120605/97e0c603/attachment-0003.jpg>
More information about the varnish-misc
mailing list