varnish consumes more and more memory
Tobias Eichelbrönner
tobias.eichelbroenner at lamp-solutions.de
Mon Apr 18 10:21:50 CEST 2016
Hi together,
I have an old varnish 3.0.5 running. (I know its outdated, but my issue
is more understanding what happens here then fixing the problem on the
actual system). While running, the systems takes up more and more memory.
I already switched to file storage, but still, after one day running and
serving about 2 million requests, the varnish process takes more 20GB of
ram.
The first thing I am confused is, that as far as I see it varnishstats
does not show the memory consumed (I will attach an output at the
bottom). Tools like top and pmap show the output.
pmap -x ...
total kB 25368872 16712056 15197372
Beside some VCL-Code, the varnish uses the vmods redis, digest, and std.
Any idea how the find out what is causing the memory leak?
Thanks in advance,
Tobias
Here is my varnishstat -1 output
client_conn 2115657 18.63 Client connections accepted
client_drop 0 0.00 Connection dropped, no sess/wrk
client_req 5176344 45.58 Client requests received
cache_hit 2207169 19.43 Cache hits
cache_hitpass 18635 0.16 Cache hits for pass
cache_miss 508733 4.48 Cache misses
backend_conn 590509 5.20 Backend conn. success
backend_unhealthy 0 0.00 Backend conn. not attempted
backend_busy 0 0.00 Backend conn. too many
backend_fail 0 0.00 Backend conn. failures
backend_reuse 2391162 21.05 Backend conn. reuses
backend_toolate 581 0.01 Backend conn. was closed
backend_recycle 2392558 21.07 Backend conn. recycles
backend_retry 142 0.00 Backend conn. retry
fetch_head 604 0.01 Fetch head
fetch_length 1493458 13.15 Fetch with Length
fetch_chunked 1277164 11.25 Fetch chunked
fetch_eof 0 0.00 Fetch EOF
fetch_bad 0 0.00 Fetch had bad headers
fetch_close 9969 0.09 Fetch wanted close
fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed
fetch_zero 0 0.00 Fetch zero len
fetch_failed 528 0.00 Fetch failed
fetch_1xx 0 0.00 Fetch no body (1xx)
fetch_204 186330 1.64 Fetch no body (204)
fetch_304 14773 0.13 Fetch no body (304)
n_sess_mem 307 . N struct sess_mem
n_sess 59 . N struct sess
n_object 59401 . N struct object
n_vampireobject 0 . N unresurrected objects
n_objectcore 59439 . N struct objectcore
n_objecthead 60591 . N struct objecthead
n_waitinglist 44 . N struct waitinglist
n_vbc 2 . N struct vbc
n_wrk 44 . N worker threads
n_wrk_create 1296 0.01 N worker threads created
n_wrk_failed 0 0.00 N worker threads not created
n_wrk_max 0 0.00 N worker threads limited
n_wrk_lqueue 0 0.00 work request queue length
n_wrk_queued 7886 0.07 N queued work requests
n_wrk_drop 0 0.00 N dropped work requests
n_backend 2 . N backends
n_expired 449200 . N expired objects
n_lru_nuked 0 . N LRU nuked objects
n_lru_moved 1917928 . N LRU moved objects
losthdr 0 0.00 HTTP header overflows
n_objsendfile 0 0.00 Objects sent with sendfile
n_objwrite 4825557 42.49 Objects sent with write
n_objoverflow 0 0.00 Objects overflowing workspace
s_sess 2115659 18.63 Total Sessions
s_req 5176344 45.58 Total Requests
s_pipe 1304 0.01 Total pipe
s_pass 2473728 21.78 Total pass
s_fetch 2981770 26.25 Total fetch
s_hdrbytes 2065942876 18190.28 Total header bytes
s_bodybytes 133824929675 1178306.04 Total body bytes
sess_closed 418834 3.69 Session Closed
sess_pipeline 63774 0.56 Session Pipeline
sess_readahead 7618 0.07 Session Read Ahead
sess_linger 4929655 43.40 Session Linger
sess_herd 4533770 39.92 Session herd
shm_records 397316593 3498.31 SHM records
shm_writes 22134819 194.89 SHM writes
shm_flushes 0 0.00 SHM flushes due to overflow
shm_cont 83004 0.73 SHM MTX contention
shm_cycles 173 0.00 SHM cycles through buffer
sms_nreq 693 0.01 SMS allocator requests
sms_nobj 0 . SMS outstanding allocations
sms_nbytes 0 . SMS outstanding bytes
sms_balloc 472858 . SMS bytes allocated
sms_bfree 472858 . SMS bytes freed
backend_req 2982527 26.26 Backend requests made
n_vcl 2 0.00 N vcl total
n_vcl_avail 2 0.00 N vcl available
n_vcl_discard 0 0.00 N vcl discarded
n_ban 1 . N total active bans
n_ban_gone 1 . N total gone bans
n_ban_add 1 0.00 N new bans added
n_ban_retire 0 0.00 N old bans deleted
n_ban_obj_test 0 0.00 N objects tested
n_ban_re_test 0 0.00 N regexps tested against
n_ban_dups 0 0.00 N duplicate bans removed
hcb_nolock 2729927 24.04 HCB Lookups without lock
hcb_lock 470290 4.14 HCB Lookups with lock
hcb_insert 470044 4.14 HCB Inserts
esi_errors 0 0.00 ESI parse errors (unlock)
esi_warnings 0 0.00 ESI parse warnings (unlock)
accept_fail 0 0.00 Accept failures
client_drop_late 0 0.00 Connection dropped late
uptime 113574 1.00 Client uptime
dir_dns_lookups 0 0.00 DNS director lookups
dir_dns_failed 0 0.00 DNS director failed lookups
dir_dns_hit 0 0.00 DNS director cached lookups hit
dir_dns_cache_full 0 0.00 DNS director full dnscache
vmods 3 . Loaded VMODs
n_gzip 0 0.00 Gzip operations
n_gunzip 2590700 22.81 Gunzip operations
sess_pipe_overflow 0 . Dropped sessions due to
session pipe overflow
LCK.sms.creat 1 0.00 Created locks
LCK.sms.destroy 0 0.00 Destroyed locks
LCK.sms.locks 2079 0.02 Lock Operations
LCK.sms.colls 0 0.00 Collisions
LCK.smp.creat 0 0.00 Created locks
LCK.smp.destroy 0 0.00 Destroyed locks
LCK.smp.locks 0 0.00 Lock Operations
LCK.smp.colls 0 0.00 Collisions
LCK.sma.creat 1 0.00 Created locks
LCK.sma.destroy 0 0.00 Destroyed locks
LCK.sma.locks 11567121 101.85 Lock Operations
LCK.sma.colls 0 0.00 Collisions
LCK.smf.creat 1 0.00 Created locks
LCK.smf.destroy 0 0.00 Destroyed locks
LCK.smf.locks 2137363 18.82 Lock Operations
LCK.smf.colls 0 0.00 Collisions
LCK.hsl.creat 0 0.00 Created locks
LCK.hsl.destroy 0 0.00 Destroyed locks
LCK.hsl.locks 0 0.00 Lock Operations
LCK.hsl.colls 0 0.00 Collisions
LCK.hcb.creat 1 0.00 Created locks
LCK.hcb.destroy 0 0.00 Destroyed locks
LCK.hcb.locks 881563 7.76 Lock Operations
LCK.hcb.colls 0 0.00 Collisions
LCK.hcl.creat 0 0.00 Created locks
LCK.hcl.destroy 0 0.00 Destroyed locks
LCK.hcl.locks 0 0.00 Lock Operations
LCK.hcl.colls 0 0.00 Collisions
LCK.vcl.creat 1 0.00 Created locks
LCK.vcl.destroy 0 0.00 Destroyed locks
LCK.vcl.locks 10574 0.09 Lock Operations
LCK.vcl.colls 0 0.00 Collisions
LCK.stat.creat 1 0.00 Created locks
LCK.stat.destroy 0 0.00 Destroyed locks
LCK.stat.locks 2115908 18.63 Lock Operations
LCK.stat.colls 0 0.00 Collisions
LCK.sessmem.creat 1 0.00 Created locks
LCK.sessmem.destroy 0 0.00 Destroyed locks
LCK.sessmem.locks 2129936 18.75 Lock Operations
LCK.sessmem.colls 0 0.00 Collisions
LCK.wstat.creat 1 0.00 Created locks
LCK.wstat.destroy 0 0.00 Destroyed locks
LCK.wstat.locks 239816 2.11 Lock Operations
LCK.wstat.colls 0 0.00 Collisions
LCK.herder.creat 1 0.00 Created locks
LCK.herder.destroy 0 0.00 Destroyed locks
LCK.herder.locks 6034 0.05 Lock Operations
LCK.herder.colls 0 0.00 Collisions
LCK.wq.creat 2 0.00 Created locks
LCK.wq.destroy 0 0.00 Destroyed locks
LCK.wq.locks 10137930 89.26 Lock Operations
LCK.wq.colls 0 0.00 Collisions
LCK.objhdr.creat 471022 4.15 Created locks
LCK.objhdr.destroy 410522 3.61 Destroyed locks
LCK.objhdr.locks 11873362 104.54 Lock Operations
LCK.objhdr.colls 0 0.00 Collisions
LCK.exp.creat 1 0.00 Created locks
LCK.exp.destroy 0 0.00 Destroyed locks
LCK.exp.locks 1086349 9.57 Lock Operations
LCK.exp.colls 0 0.00 Collisions
LCK.lru.creat 2 0.00 Created locks
LCK.lru.destroy 0 0.00 Destroyed locks
LCK.lru.locks 523780 4.61 Lock Operations
LCK.lru.colls 0 0.00 Collisions
LCK.cli.creat 1 0.00 Created locks
LCK.cli.destroy 0 0.00 Destroyed locks
LCK.cli.locks 877 0.01 Lock Operations
LCK.cli.colls 0 0.00 Collisions
LCK.ban.creat 1 0.00 Created locks
LCK.ban.destroy 0 0.00 Destroyed locks
LCK.ban.locks 1071376 9.43 Lock Operations
LCK.ban.colls 0 0.00 Collisions
LCK.vbp.creat 1 0.00 Created locks
LCK.vbp.destroy 0 0.00 Destroyed locks
LCK.vbp.locks 0 0.00 Lock Operations
LCK.vbp.colls 0 0.00 Collisions
LCK.vbe.creat 1 0.00 Created locks
LCK.vbe.destroy 0 0.00 Destroyed locks
LCK.vbe.locks 1183870 10.42 Lock Operations
LCK.vbe.colls 0 0.00 Collisions
LCK.backend.creat 2 0.00 Created locks
LCK.backend.destroy 0 0.00 Destroyed locks
LCK.backend.locks 6560980 57.77 Lock Operations
LCK.backend.colls 0 0.00 Collisions
SMF.s0.c_req 943045 8.30 Allocator requests
SMF.s0.c_fail 0 0.00 Allocator failures
SMF.s0.c_bytes 57519628288 506450.67 Bytes allocated
SMF.s0.c_freed 54163292160 476898.69 Bytes freed
SMF.s0.g_alloc 129160 . Allocations outstanding
SMF.s0.g_bytes 3356336128 . Bytes outstanding
SMF.s0.g_space 938631168 . Bytes available
SMF.s0.g_smf 138281 . N struct smf
SMF.s0.g_smf_frag 9118 . N small free smf
SMF.s0.g_smf_large 3 . N large free smf
SMA.Transient.c_req 4780332 42.09 Allocator requests
SMA.Transient.c_fail 0 0.00 Allocator failures
SMA.Transient.c_bytes 284689405959 2506642.42 Bytes allocated
SMA.Transient.c_freed 284688351047 2506633.13 Bytes freed
SMA.Transient.g_alloc 85 . Allocations outstanding
SMA.Transient.g_bytes 1054912 . Bytes outstanding
SMA.Transient.g_space 0 . Bytes available
VBE.default(127.0.0.1,,8080).vcls 2 . VCL references
VBE.default(127.0.0.1,,8080).happy 0 . Happy health
probes
VBE.fpm(127.0.0.1,,8081).vcls 2 . VCL references
VBE.fpm(127.0.0.1,,8081).happy 0 . Happy health
probes
--
LAMP solutions GmbH
Gostenhofer Hauptstrasse 35
90443 Nuernberg
Amtsgericht Nuernberg: HRB 22366
Geschaeftsfuehrer: Heiko Schubert
Es gelten unsere allgemeinen Geschaeftsbedingungen.
http://www.lamp-solutions.de/agb/
Telefon : 0911 / 376 516 0
Fax : 0911 / 376 516 11
E-Mail : support at lamp-solutions.de
Web : www.lamp-solutions.de
Facebook : http://www.facebook.com/LAMPsolutions
Twitter : http://twitter.com/#!/lampsolutions
More information about the varnish-misc
mailing list