varnish consumes more and more memory

Jason Price japrice at gmail.com
Mon Apr 18 14:19:50 CEST 2016


I'm curious what the Virtual Memory vs Resident Set Size from top show.
Most likely, the RSS will be stable and small.  Virtual memory is almost
valueless as a measure.

ps -eo pid,user,vsz,rss,%cpu,%mem,cmd | egrep "PID|varnish"

-Jason

On Mon, Apr 18, 2016 at 4:21 AM, Tobias Eichelbrönner <
tobias.eichelbroenner at lamp-solutions.de> wrote:

> Hi together,
>
> I have an old varnish 3.0.5 running. (I know its outdated, but my issue
> is more understanding what happens here then fixing the problem on the
> actual system). While running, the systems takes up more and more memory.
>
> I already switched to file storage, but still, after one day running and
> serving about 2 million requests, the varnish process takes more 20GB of
> ram.
>
> The first thing I am confused is, that as far as I see it varnishstats
> does not show the memory consumed (I will attach an output at the
> bottom). Tools like top and pmap show the output.
>
> pmap -x ...
> total kB         25368872 16712056 15197372
>
> Beside some VCL-Code, the varnish uses the vmods redis, digest, and std.
>
> Any idea how the find out what is causing the memory leak?
>
> Thanks in advance,
>
> Tobias
>
> Here is my varnishstat -1 output
>
> client_conn            2115657        18.63 Client connections accepted
> client_drop                  0         0.00 Connection dropped, no sess/wrk
> client_req             5176344        45.58 Client requests received
> cache_hit              2207169        19.43 Cache hits
> cache_hitpass            18635         0.16 Cache hits for pass
> cache_miss              508733         4.48 Cache misses
> backend_conn            590509         5.20 Backend conn. success
> backend_unhealthy            0         0.00 Backend conn. not attempted
> backend_busy                 0         0.00 Backend conn. too many
> backend_fail                 0         0.00 Backend conn. failures
> backend_reuse          2391162        21.05 Backend conn. reuses
> backend_toolate            581         0.01 Backend conn. was closed
> backend_recycle        2392558        21.07 Backend conn. recycles
> backend_retry              142         0.00 Backend conn. retry
> fetch_head                 604         0.01 Fetch head
> fetch_length           1493458        13.15 Fetch with Length
> fetch_chunked          1277164        11.25 Fetch chunked
> fetch_eof                    0         0.00 Fetch EOF
> fetch_bad                    0         0.00 Fetch had bad headers
> fetch_close               9969         0.09 Fetch wanted close
> fetch_oldhttp                0         0.00 Fetch pre HTTP/1.1 closed
> fetch_zero                   0         0.00 Fetch zero len
> fetch_failed               528         0.00 Fetch failed
> fetch_1xx                    0         0.00 Fetch no body (1xx)
> fetch_204               186330         1.64 Fetch no body (204)
> fetch_304                14773         0.13 Fetch no body (304)
> n_sess_mem                 307          .   N struct sess_mem
> n_sess                      59          .   N struct sess
> n_object                 59401          .   N struct object
> n_vampireobject              0          .   N unresurrected objects
> n_objectcore             59439          .   N struct objectcore
> n_objecthead             60591          .   N struct objecthead
> n_waitinglist               44          .   N struct waitinglist
> n_vbc                        2          .   N struct vbc
> n_wrk                       44          .   N worker threads
> n_wrk_create              1296         0.01 N worker threads created
> n_wrk_failed                 0         0.00 N worker threads not created
> n_wrk_max                    0         0.00 N worker threads limited
> n_wrk_lqueue                 0         0.00 work request queue length
> n_wrk_queued              7886         0.07 N queued work requests
> n_wrk_drop                   0         0.00 N dropped work requests
> n_backend                    2          .   N backends
> n_expired               449200          .   N expired objects
> n_lru_nuked                  0          .   N LRU nuked objects
> n_lru_moved            1917928          .   N LRU moved objects
> losthdr                      0         0.00 HTTP header overflows
> n_objsendfile                0         0.00 Objects sent with sendfile
> n_objwrite             4825557        42.49 Objects sent with write
> n_objoverflow                0         0.00 Objects overflowing workspace
> s_sess                 2115659        18.63 Total Sessions
> s_req                  5176344        45.58 Total Requests
> s_pipe                    1304         0.01 Total pipe
> s_pass                 2473728        21.78 Total pass
> s_fetch                2981770        26.25 Total fetch
> s_hdrbytes          2065942876     18190.28 Total header bytes
> s_bodybytes       133824929675   1178306.04 Total body bytes
> sess_closed             418834         3.69 Session Closed
> sess_pipeline            63774         0.56 Session Pipeline
> sess_readahead            7618         0.07 Session Read Ahead
> sess_linger            4929655        43.40 Session Linger
> sess_herd              4533770        39.92 Session herd
> shm_records          397316593      3498.31 SHM records
> shm_writes            22134819       194.89 SHM writes
> shm_flushes                  0         0.00 SHM flushes due to overflow
> shm_cont                 83004         0.73 SHM MTX contention
> shm_cycles                 173         0.00 SHM cycles through buffer
> sms_nreq                   693         0.01 SMS allocator requests
> sms_nobj                     0          .   SMS outstanding allocations
> sms_nbytes                   0          .   SMS outstanding bytes
> sms_balloc              472858          .   SMS bytes allocated
> sms_bfree               472858          .   SMS bytes freed
> backend_req            2982527        26.26 Backend requests made
> n_vcl                        2         0.00 N vcl total
> n_vcl_avail                  2         0.00 N vcl available
> n_vcl_discard                0         0.00 N vcl discarded
> n_ban                        1          .   N total active bans
> n_ban_gone                   1          .   N total gone bans
> n_ban_add                    1         0.00 N new bans added
> n_ban_retire                 0         0.00 N old bans deleted
> n_ban_obj_test               0         0.00 N objects tested
> n_ban_re_test                0         0.00 N regexps tested against
> n_ban_dups                   0         0.00 N duplicate bans removed
> hcb_nolock             2729927        24.04 HCB Lookups without lock
> hcb_lock                470290         4.14 HCB Lookups with lock
> hcb_insert              470044         4.14 HCB Inserts
> esi_errors                   0         0.00 ESI parse errors (unlock)
> esi_warnings                 0         0.00 ESI parse warnings (unlock)
> accept_fail                  0         0.00 Accept failures
> client_drop_late             0         0.00 Connection dropped late
> uptime                  113574         1.00 Client uptime
> dir_dns_lookups              0         0.00 DNS director lookups
> dir_dns_failed               0         0.00 DNS director failed lookups
> dir_dns_hit                  0         0.00 DNS director cached lookups hit
> dir_dns_cache_full           0         0.00 DNS director full dnscache
> vmods                        3          .   Loaded VMODs
> n_gzip                       0         0.00 Gzip operations
> n_gunzip               2590700        22.81 Gunzip operations
> sess_pipe_overflow           0          .   Dropped sessions due to
> session pipe overflow
> LCK.sms.creat                1         0.00 Created locks
> LCK.sms.destroy              0         0.00 Destroyed locks
> LCK.sms.locks             2079         0.02 Lock Operations
> LCK.sms.colls                0         0.00 Collisions
> LCK.smp.creat                0         0.00 Created locks
> LCK.smp.destroy              0         0.00 Destroyed locks
> LCK.smp.locks                0         0.00 Lock Operations
> LCK.smp.colls                0         0.00 Collisions
> LCK.sma.creat                1         0.00 Created locks
> LCK.sma.destroy              0         0.00 Destroyed locks
> LCK.sma.locks         11567121       101.85 Lock Operations
> LCK.sma.colls                0         0.00 Collisions
> LCK.smf.creat                1         0.00 Created locks
> LCK.smf.destroy              0         0.00 Destroyed locks
> LCK.smf.locks          2137363        18.82 Lock Operations
> LCK.smf.colls                0         0.00 Collisions
> LCK.hsl.creat                0         0.00 Created locks
> LCK.hsl.destroy              0         0.00 Destroyed locks
> LCK.hsl.locks                0         0.00 Lock Operations
> LCK.hsl.colls                0         0.00 Collisions
> LCK.hcb.creat                1         0.00 Created locks
> LCK.hcb.destroy              0         0.00 Destroyed locks
> LCK.hcb.locks           881563         7.76 Lock Operations
> LCK.hcb.colls                0         0.00 Collisions
> LCK.hcl.creat                0         0.00 Created locks
> LCK.hcl.destroy              0         0.00 Destroyed locks
> LCK.hcl.locks                0         0.00 Lock Operations
> LCK.hcl.colls                0         0.00 Collisions
> LCK.vcl.creat                1         0.00 Created locks
> LCK.vcl.destroy              0         0.00 Destroyed locks
> LCK.vcl.locks            10574         0.09 Lock Operations
> LCK.vcl.colls                0         0.00 Collisions
> LCK.stat.creat               1         0.00 Created locks
> LCK.stat.destroy             0         0.00 Destroyed locks
> LCK.stat.locks         2115908        18.63 Lock Operations
> LCK.stat.colls               0         0.00 Collisions
> LCK.sessmem.creat            1         0.00 Created locks
> LCK.sessmem.destroy            0         0.00 Destroyed locks
> LCK.sessmem.locks        2129936        18.75 Lock Operations
> LCK.sessmem.colls              0         0.00 Collisions
> LCK.wstat.creat                1         0.00 Created locks
> LCK.wstat.destroy              0         0.00 Destroyed locks
> LCK.wstat.locks           239816         2.11 Lock Operations
> LCK.wstat.colls                0         0.00 Collisions
> LCK.herder.creat               1         0.00 Created locks
> LCK.herder.destroy             0         0.00 Destroyed locks
> LCK.herder.locks            6034         0.05 Lock Operations
> LCK.herder.colls               0         0.00 Collisions
> LCK.wq.creat                   2         0.00 Created locks
> LCK.wq.destroy                 0         0.00 Destroyed locks
> LCK.wq.locks            10137930        89.26 Lock Operations
> LCK.wq.colls                   0         0.00 Collisions
> LCK.objhdr.creat          471022         4.15 Created locks
> LCK.objhdr.destroy        410522         3.61 Destroyed locks
> LCK.objhdr.locks        11873362       104.54 Lock Operations
> LCK.objhdr.colls               0         0.00 Collisions
> LCK.exp.creat                  1         0.00 Created locks
> LCK.exp.destroy                0         0.00 Destroyed locks
> LCK.exp.locks            1086349         9.57 Lock Operations
> LCK.exp.colls                  0         0.00 Collisions
> LCK.lru.creat                  2         0.00 Created locks
> LCK.lru.destroy                0         0.00 Destroyed locks
> LCK.lru.locks             523780         4.61 Lock Operations
> LCK.lru.colls                  0         0.00 Collisions
> LCK.cli.creat                  1         0.00 Created locks
> LCK.cli.destroy                0         0.00 Destroyed locks
> LCK.cli.locks                877         0.01 Lock Operations
> LCK.cli.colls                  0         0.00 Collisions
> LCK.ban.creat                  1         0.00 Created locks
> LCK.ban.destroy                0         0.00 Destroyed locks
> LCK.ban.locks            1071376         9.43 Lock Operations
> LCK.ban.colls                  0         0.00 Collisions
> LCK.vbp.creat                  1         0.00 Created locks
> LCK.vbp.destroy                0         0.00 Destroyed locks
> LCK.vbp.locks                  0         0.00 Lock Operations
> LCK.vbp.colls                  0         0.00 Collisions
> LCK.vbe.creat                  1         0.00 Created locks
> LCK.vbe.destroy                0         0.00 Destroyed locks
> LCK.vbe.locks            1183870        10.42 Lock Operations
> LCK.vbe.colls                  0         0.00 Collisions
> LCK.backend.creat              2         0.00 Created locks
> LCK.backend.destroy            0         0.00 Destroyed locks
> LCK.backend.locks        6560980        57.77 Lock Operations
> LCK.backend.colls              0         0.00 Collisions
> SMF.s0.c_req              943045         8.30 Allocator requests
> SMF.s0.c_fail                  0         0.00 Allocator failures
> SMF.s0.c_bytes       57519628288    506450.67 Bytes allocated
> SMF.s0.c_freed       54163292160    476898.69 Bytes freed
> SMF.s0.g_alloc            129160          .   Allocations outstanding
> SMF.s0.g_bytes        3356336128          .   Bytes outstanding
> SMF.s0.g_space         938631168          .   Bytes available
> SMF.s0.g_smf              138281          .   N struct smf
> SMF.s0.g_smf_frag           9118          .   N small free smf
> SMF.s0.g_smf_large             3          .   N large free smf
> SMA.Transient.c_req      4780332        42.09 Allocator requests
> SMA.Transient.c_fail           0         0.00 Allocator failures
> SMA.Transient.c_bytes 284689405959   2506642.42 Bytes allocated
> SMA.Transient.c_freed 284688351047   2506633.13 Bytes freed
> SMA.Transient.g_alloc           85          .   Allocations outstanding
> SMA.Transient.g_bytes      1054912          .   Bytes outstanding
> SMA.Transient.g_space            0          .   Bytes available
> VBE.default(127.0.0.1,,8080).vcls            2          .   VCL references
> VBE.default(127.0.0.1,,8080).happy           0          .   Happy health
> probes
> VBE.fpm(127.0.0.1,,8081).vcls                2          .   VCL references
> VBE.fpm(127.0.0.1,,8081).happy               0          .   Happy health
> probes
>
> --
> LAMP solutions GmbH
> Gostenhofer Hauptstrasse 35
> 90443 Nuernberg
>
> Amtsgericht Nuernberg: HRB 22366
> Geschaeftsfuehrer: Heiko Schubert
>
> Es gelten unsere allgemeinen Geschaeftsbedingungen.
> http://www.lamp-solutions.de/agb/
>
> Telefon      : 0911 / 376 516 0
> Fax          : 0911 / 376 516 11
> E-Mail       : support at lamp-solutions.de
> Web          : www.lamp-solutions.de
> Facebook     : http://www.facebook.com/LAMPsolutions
> Twitter      : http://twitter.com/#!/lampsolutions
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20160418/d9c755b3/attachment-0001.html>


More information about the varnish-misc mailing list