varnish consumes more and more memory
Jason Price
japrice at gmail.com
Mon Apr 18 14:19:50 CEST 2016
I'm curious what the Virtual Memory vs Resident Set Size from top show.
Most likely, the RSS will be stable and small. Virtual memory is almost
valueless as a measure.
ps -eo pid,user,vsz,rss,%cpu,%mem,cmd | egrep "PID|varnish"
-Jason
On Mon, Apr 18, 2016 at 4:21 AM, Tobias Eichelbrönner <
tobias.eichelbroenner at lamp-solutions.de> wrote:
> Hi together,
>
> I have an old varnish 3.0.5 running. (I know its outdated, but my issue
> is more understanding what happens here then fixing the problem on the
> actual system). While running, the systems takes up more and more memory.
>
> I already switched to file storage, but still, after one day running and
> serving about 2 million requests, the varnish process takes more 20GB of
> ram.
>
> The first thing I am confused is, that as far as I see it varnishstats
> does not show the memory consumed (I will attach an output at the
> bottom). Tools like top and pmap show the output.
>
> pmap -x ...
> total kB 25368872 16712056 15197372
>
> Beside some VCL-Code, the varnish uses the vmods redis, digest, and std.
>
> Any idea how the find out what is causing the memory leak?
>
> Thanks in advance,
>
> Tobias
>
> Here is my varnishstat -1 output
>
> client_conn 2115657 18.63 Client connections accepted
> client_drop 0 0.00 Connection dropped, no sess/wrk
> client_req 5176344 45.58 Client requests received
> cache_hit 2207169 19.43 Cache hits
> cache_hitpass 18635 0.16 Cache hits for pass
> cache_miss 508733 4.48 Cache misses
> backend_conn 590509 5.20 Backend conn. success
> backend_unhealthy 0 0.00 Backend conn. not attempted
> backend_busy 0 0.00 Backend conn. too many
> backend_fail 0 0.00 Backend conn. failures
> backend_reuse 2391162 21.05 Backend conn. reuses
> backend_toolate 581 0.01 Backend conn. was closed
> backend_recycle 2392558 21.07 Backend conn. recycles
> backend_retry 142 0.00 Backend conn. retry
> fetch_head 604 0.01 Fetch head
> fetch_length 1493458 13.15 Fetch with Length
> fetch_chunked 1277164 11.25 Fetch chunked
> fetch_eof 0 0.00 Fetch EOF
> fetch_bad 0 0.00 Fetch had bad headers
> fetch_close 9969 0.09 Fetch wanted close
> fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed
> fetch_zero 0 0.00 Fetch zero len
> fetch_failed 528 0.00 Fetch failed
> fetch_1xx 0 0.00 Fetch no body (1xx)
> fetch_204 186330 1.64 Fetch no body (204)
> fetch_304 14773 0.13 Fetch no body (304)
> n_sess_mem 307 . N struct sess_mem
> n_sess 59 . N struct sess
> n_object 59401 . N struct object
> n_vampireobject 0 . N unresurrected objects
> n_objectcore 59439 . N struct objectcore
> n_objecthead 60591 . N struct objecthead
> n_waitinglist 44 . N struct waitinglist
> n_vbc 2 . N struct vbc
> n_wrk 44 . N worker threads
> n_wrk_create 1296 0.01 N worker threads created
> n_wrk_failed 0 0.00 N worker threads not created
> n_wrk_max 0 0.00 N worker threads limited
> n_wrk_lqueue 0 0.00 work request queue length
> n_wrk_queued 7886 0.07 N queued work requests
> n_wrk_drop 0 0.00 N dropped work requests
> n_backend 2 . N backends
> n_expired 449200 . N expired objects
> n_lru_nuked 0 . N LRU nuked objects
> n_lru_moved 1917928 . N LRU moved objects
> losthdr 0 0.00 HTTP header overflows
> n_objsendfile 0 0.00 Objects sent with sendfile
> n_objwrite 4825557 42.49 Objects sent with write
> n_objoverflow 0 0.00 Objects overflowing workspace
> s_sess 2115659 18.63 Total Sessions
> s_req 5176344 45.58 Total Requests
> s_pipe 1304 0.01 Total pipe
> s_pass 2473728 21.78 Total pass
> s_fetch 2981770 26.25 Total fetch
> s_hdrbytes 2065942876 18190.28 Total header bytes
> s_bodybytes 133824929675 1178306.04 Total body bytes
> sess_closed 418834 3.69 Session Closed
> sess_pipeline 63774 0.56 Session Pipeline
> sess_readahead 7618 0.07 Session Read Ahead
> sess_linger 4929655 43.40 Session Linger
> sess_herd 4533770 39.92 Session herd
> shm_records 397316593 3498.31 SHM records
> shm_writes 22134819 194.89 SHM writes
> shm_flushes 0 0.00 SHM flushes due to overflow
> shm_cont 83004 0.73 SHM MTX contention
> shm_cycles 173 0.00 SHM cycles through buffer
> sms_nreq 693 0.01 SMS allocator requests
> sms_nobj 0 . SMS outstanding allocations
> sms_nbytes 0 . SMS outstanding bytes
> sms_balloc 472858 . SMS bytes allocated
> sms_bfree 472858 . SMS bytes freed
> backend_req 2982527 26.26 Backend requests made
> n_vcl 2 0.00 N vcl total
> n_vcl_avail 2 0.00 N vcl available
> n_vcl_discard 0 0.00 N vcl discarded
> n_ban 1 . N total active bans
> n_ban_gone 1 . N total gone bans
> n_ban_add 1 0.00 N new bans added
> n_ban_retire 0 0.00 N old bans deleted
> n_ban_obj_test 0 0.00 N objects tested
> n_ban_re_test 0 0.00 N regexps tested against
> n_ban_dups 0 0.00 N duplicate bans removed
> hcb_nolock 2729927 24.04 HCB Lookups without lock
> hcb_lock 470290 4.14 HCB Lookups with lock
> hcb_insert 470044 4.14 HCB Inserts
> esi_errors 0 0.00 ESI parse errors (unlock)
> esi_warnings 0 0.00 ESI parse warnings (unlock)
> accept_fail 0 0.00 Accept failures
> client_drop_late 0 0.00 Connection dropped late
> uptime 113574 1.00 Client uptime
> dir_dns_lookups 0 0.00 DNS director lookups
> dir_dns_failed 0 0.00 DNS director failed lookups
> dir_dns_hit 0 0.00 DNS director cached lookups hit
> dir_dns_cache_full 0 0.00 DNS director full dnscache
> vmods 3 . Loaded VMODs
> n_gzip 0 0.00 Gzip operations
> n_gunzip 2590700 22.81 Gunzip operations
> sess_pipe_overflow 0 . Dropped sessions due to
> session pipe overflow
> LCK.sms.creat 1 0.00 Created locks
> LCK.sms.destroy 0 0.00 Destroyed locks
> LCK.sms.locks 2079 0.02 Lock Operations
> LCK.sms.colls 0 0.00 Collisions
> LCK.smp.creat 0 0.00 Created locks
> LCK.smp.destroy 0 0.00 Destroyed locks
> LCK.smp.locks 0 0.00 Lock Operations
> LCK.smp.colls 0 0.00 Collisions
> LCK.sma.creat 1 0.00 Created locks
> LCK.sma.destroy 0 0.00 Destroyed locks
> LCK.sma.locks 11567121 101.85 Lock Operations
> LCK.sma.colls 0 0.00 Collisions
> LCK.smf.creat 1 0.00 Created locks
> LCK.smf.destroy 0 0.00 Destroyed locks
> LCK.smf.locks 2137363 18.82 Lock Operations
> LCK.smf.colls 0 0.00 Collisions
> LCK.hsl.creat 0 0.00 Created locks
> LCK.hsl.destroy 0 0.00 Destroyed locks
> LCK.hsl.locks 0 0.00 Lock Operations
> LCK.hsl.colls 0 0.00 Collisions
> LCK.hcb.creat 1 0.00 Created locks
> LCK.hcb.destroy 0 0.00 Destroyed locks
> LCK.hcb.locks 881563 7.76 Lock Operations
> LCK.hcb.colls 0 0.00 Collisions
> LCK.hcl.creat 0 0.00 Created locks
> LCK.hcl.destroy 0 0.00 Destroyed locks
> LCK.hcl.locks 0 0.00 Lock Operations
> LCK.hcl.colls 0 0.00 Collisions
> LCK.vcl.creat 1 0.00 Created locks
> LCK.vcl.destroy 0 0.00 Destroyed locks
> LCK.vcl.locks 10574 0.09 Lock Operations
> LCK.vcl.colls 0 0.00 Collisions
> LCK.stat.creat 1 0.00 Created locks
> LCK.stat.destroy 0 0.00 Destroyed locks
> LCK.stat.locks 2115908 18.63 Lock Operations
> LCK.stat.colls 0 0.00 Collisions
> LCK.sessmem.creat 1 0.00 Created locks
> LCK.sessmem.destroy 0 0.00 Destroyed locks
> LCK.sessmem.locks 2129936 18.75 Lock Operations
> LCK.sessmem.colls 0 0.00 Collisions
> LCK.wstat.creat 1 0.00 Created locks
> LCK.wstat.destroy 0 0.00 Destroyed locks
> LCK.wstat.locks 239816 2.11 Lock Operations
> LCK.wstat.colls 0 0.00 Collisions
> LCK.herder.creat 1 0.00 Created locks
> LCK.herder.destroy 0 0.00 Destroyed locks
> LCK.herder.locks 6034 0.05 Lock Operations
> LCK.herder.colls 0 0.00 Collisions
> LCK.wq.creat 2 0.00 Created locks
> LCK.wq.destroy 0 0.00 Destroyed locks
> LCK.wq.locks 10137930 89.26 Lock Operations
> LCK.wq.colls 0 0.00 Collisions
> LCK.objhdr.creat 471022 4.15 Created locks
> LCK.objhdr.destroy 410522 3.61 Destroyed locks
> LCK.objhdr.locks 11873362 104.54 Lock Operations
> LCK.objhdr.colls 0 0.00 Collisions
> LCK.exp.creat 1 0.00 Created locks
> LCK.exp.destroy 0 0.00 Destroyed locks
> LCK.exp.locks 1086349 9.57 Lock Operations
> LCK.exp.colls 0 0.00 Collisions
> LCK.lru.creat 2 0.00 Created locks
> LCK.lru.destroy 0 0.00 Destroyed locks
> LCK.lru.locks 523780 4.61 Lock Operations
> LCK.lru.colls 0 0.00 Collisions
> LCK.cli.creat 1 0.00 Created locks
> LCK.cli.destroy 0 0.00 Destroyed locks
> LCK.cli.locks 877 0.01 Lock Operations
> LCK.cli.colls 0 0.00 Collisions
> LCK.ban.creat 1 0.00 Created locks
> LCK.ban.destroy 0 0.00 Destroyed locks
> LCK.ban.locks 1071376 9.43 Lock Operations
> LCK.ban.colls 0 0.00 Collisions
> LCK.vbp.creat 1 0.00 Created locks
> LCK.vbp.destroy 0 0.00 Destroyed locks
> LCK.vbp.locks 0 0.00 Lock Operations
> LCK.vbp.colls 0 0.00 Collisions
> LCK.vbe.creat 1 0.00 Created locks
> LCK.vbe.destroy 0 0.00 Destroyed locks
> LCK.vbe.locks 1183870 10.42 Lock Operations
> LCK.vbe.colls 0 0.00 Collisions
> LCK.backend.creat 2 0.00 Created locks
> LCK.backend.destroy 0 0.00 Destroyed locks
> LCK.backend.locks 6560980 57.77 Lock Operations
> LCK.backend.colls 0 0.00 Collisions
> SMF.s0.c_req 943045 8.30 Allocator requests
> SMF.s0.c_fail 0 0.00 Allocator failures
> SMF.s0.c_bytes 57519628288 506450.67 Bytes allocated
> SMF.s0.c_freed 54163292160 476898.69 Bytes freed
> SMF.s0.g_alloc 129160 . Allocations outstanding
> SMF.s0.g_bytes 3356336128 . Bytes outstanding
> SMF.s0.g_space 938631168 . Bytes available
> SMF.s0.g_smf 138281 . N struct smf
> SMF.s0.g_smf_frag 9118 . N small free smf
> SMF.s0.g_smf_large 3 . N large free smf
> SMA.Transient.c_req 4780332 42.09 Allocator requests
> SMA.Transient.c_fail 0 0.00 Allocator failures
> SMA.Transient.c_bytes 284689405959 2506642.42 Bytes allocated
> SMA.Transient.c_freed 284688351047 2506633.13 Bytes freed
> SMA.Transient.g_alloc 85 . Allocations outstanding
> SMA.Transient.g_bytes 1054912 . Bytes outstanding
> SMA.Transient.g_space 0 . Bytes available
> VBE.default(127.0.0.1,,8080).vcls 2 . VCL references
> VBE.default(127.0.0.1,,8080).happy 0 . Happy health
> probes
> VBE.fpm(127.0.0.1,,8081).vcls 2 . VCL references
> VBE.fpm(127.0.0.1,,8081).happy 0 . Happy health
> probes
>
> --
> LAMP solutions GmbH
> Gostenhofer Hauptstrasse 35
> 90443 Nuernberg
>
> Amtsgericht Nuernberg: HRB 22366
> Geschaeftsfuehrer: Heiko Schubert
>
> Es gelten unsere allgemeinen Geschaeftsbedingungen.
> http://www.lamp-solutions.de/agb/
>
> Telefon : 0911 / 376 516 0
> Fax : 0911 / 376 516 11
> E-Mail : support at lamp-solutions.de
> Web : www.lamp-solutions.de
> Facebook : http://www.facebook.com/LAMPsolutions
> Twitter : http://twitter.com/#!/lampsolutions
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20160418/d9c755b3/attachment-0001.html>
More information about the varnish-misc
mailing list