<div dir="ltr">I'm curious what the Virtual Memory vs Resident Set Size from top show. Most likely, the RSS will be stable and small. Virtual memory is almost valueless as a measure.<div><br></div><div>ps -eo pid,user,vsz,rss,%cpu,%mem,cmd | egrep "PID|varnish"<br></div><div><br></div><div>-Jason<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 18, 2016 at 4:21 AM, Tobias Eichelbrönner <span dir="ltr"><<a href="mailto:tobias.eichelbroenner@lamp-solutions.de" target="_blank">tobias.eichelbroenner@lamp-solutions.de</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi together,<br>
<br>
I have an old varnish 3.0.5 running. (I know its outdated, but my issue<br>
is more understanding what happens here then fixing the problem on the<br>
actual system). While running, the systems takes up more and more memory.<br>
<br>
I already switched to file storage, but still, after one day running and<br>
serving about 2 million requests, the varnish process takes more 20GB of<br>
ram.<br>
<br>
The first thing I am confused is, that as far as I see it varnishstats<br>
does not show the memory consumed (I will attach an output at the<br>
bottom). Tools like top and pmap show the output.<br>
<br>
pmap -x ...<br>
total kBÂ Â Â Â Â 25368872 16712056 15197372<br>
<br>
Beside some VCL-Code, the varnish uses the vmods redis, digest, and std.<br>
<br>
Any idea how the find out what is causing the memory leak?<br>
<br>
Thanks in advance,<br>
<br>
Tobias<br>
<br>
Here is my varnishstat -1 output<br>
<br>
client_conn      2115657    18.63 Client connections accepted<br>
client_drop         0     0.00 Connection dropped, no sess/wrk<br>
client_req       5176344    45.58 Client requests received<br>
cache_hit       2207169    19.43 Cache hits<br>
cache_hitpass      18635     0.16 Cache hits for pass<br>
cache_miss       508733     4.48 Cache misses<br>
backend_conn      590509     5.20 Backend conn. success<br>
backend_unhealthy      0     0.00 Backend conn. not attempted<br>
backend_busy         0     0.00 Backend conn. too many<br>
backend_fail         0     0.00 Backend conn. failures<br>
backend_reuse     2391162    21.05 Backend conn. reuses<br>
backend_toolate      581     0.01 Backend conn. was closed<br>
backend_recycle    2392558    21.07 Backend conn. recycles<br>
backend_retry       142     0.00 Backend conn. retry<br>
fetch_head         604     0.01 Fetch head<br>
fetch_length      1493458    13.15 Fetch with Length<br>
fetch_chunked     1277164    11.25 Fetch chunked<br>
fetch_eof          0     0.00 Fetch EOF<br>
fetch_bad          0     0.00 Fetch had bad headers<br>
fetch_close        9969     0.09 Fetch wanted close<br>
fetch_oldhttp        0     0.00 Fetch pre HTTP/1.1 closed<br>
fetch_zero          0     0.00 Fetch zero len<br>
fetch_failed        528     0.00 Fetch failed<br>
fetch_1xx          0     0.00 Fetch no body (1xx)<br>
fetch_204Â Â Â Â Â Â Â Â 186330Â Â Â Â Â 1.64 Fetch no body (204)<br>
fetch_304Â Â Â Â Â Â Â Â 14773Â Â Â Â Â 0.13 Fetch no body (304)<br>
n_sess_mem         307     .  N struct sess_mem<br>
n_sess           59     .  N struct sess<br>
n_object         59401     .  N struct object<br>
n_vampireobject       0     .  N unresurrected objects<br>
n_objectcore       59439     .  N struct objectcore<br>
n_objecthead       60591     .  N struct objecthead<br>
n_waitinglist        44     .  N struct waitinglist<br>
n_vbc            2     .  N struct vbc<br>
n_wrk            44     .  N worker threads<br>
n_wrk_create       1296     0.01 N worker threads created<br>
n_wrk_failed         0     0.00 N worker threads not created<br>
n_wrk_max          0     0.00 N worker threads limited<br>
n_wrk_lqueue         0     0.00 work request queue length<br>
n_wrk_queued       7886     0.07 N queued work requests<br>
n_wrk_drop          0     0.00 N dropped work requests<br>
n_backend          2     .  N backends<br>
n_expired        449200     .  N expired objects<br>
n_lru_nuked         0     .  N LRU nuked objects<br>
n_lru_moved      1917928     .  N LRU moved objects<br>
losthdr           0     0.00 HTTP header overflows<br>
n_objsendfile        0     0.00 Objects sent with sendfile<br>
n_objwrite       4825557    42.49 Objects sent with write<br>
n_objoverflow        0     0.00 Objects overflowing workspace<br>
s_sess         2115659    18.63 Total Sessions<br>
s_req         5176344    45.58 Total Requests<br>
s_pipe          1304     0.01 Total pipe<br>
s_pass         2473728    21.78 Total pass<br>
s_fetch        2981770    26.25 Total fetch<br>
s_hdrbytes     2065942876   18190.28 Total header bytes<br>
s_bodybytes    133824929675  1178306.04 Total body bytes<br>
sess_closed       418834     3.69 Session Closed<br>
sess_pipeline      63774     0.56 Session Pipeline<br>
sess_readahead      7618     0.07 Session Read Ahead<br>
sess_linger      4929655    43.40 Session Linger<br>
sess_herd       4533770    39.92 Session herd<br>
shm_records     397316593   3498.31 SHM records<br>
shm_writes      22134819    194.89 SHM writes<br>
shm_flushes         0     0.00 SHM flushes due to overflow<br>
shm_cont         83004     0.73 SHM MTX contention<br>
shm_cycles         173     0.00 SHM cycles through buffer<br>
sms_nreq          693     0.01 SMS allocator requests<br>
sms_nobj           0     .  SMS outstanding allocations<br>
sms_nbytes          0     .  SMS outstanding bytes<br>
sms_balloc       472858     .  SMS bytes allocated<br>
sms_bfree        472858     .  SMS bytes freed<br>
backend_req      2982527    26.26 Backend requests made<br>
n_vcl            2     0.00 N vcl total<br>
n_vcl_avail         2     0.00 N vcl available<br>
n_vcl_discard        0     0.00 N vcl discarded<br>
n_ban            1     .  N total active bans<br>
n_ban_gone          1     .  N total gone bans<br>
n_ban_add          1     0.00 N new bans added<br>
n_ban_retire         0     0.00 N old bans deleted<br>
n_ban_obj_test        0     0.00 N objects tested<br>
n_ban_re_test        0     0.00 N regexps tested against<br>
n_ban_dups          0     0.00 N duplicate bans removed<br>
hcb_nolock       2729927    24.04 HCB Lookups without lock<br>
hcb_lock        470290     4.14 HCB Lookups with lock<br>
hcb_insert       470044     4.14 HCB Inserts<br>
esi_errors          0     0.00 ESI parse errors (unlock)<br>
esi_warnings         0     0.00 ESI parse warnings (unlock)<br>
accept_fail         0     0.00 Accept failures<br>
client_drop_late       0     0.00 Connection dropped late<br>
uptime         113574     1.00 Client uptime<br>
dir_dns_lookups       0     0.00 DNS director lookups<br>
dir_dns_failed        0     0.00 DNS director failed lookups<br>
dir_dns_hit         0     0.00 DNS director cached lookups hit<br>
dir_dns_cache_full      0     0.00 DNS director full dnscache<br>
vmods            3     .  Loaded VMODs<br>
n_gzip            0     0.00 Gzip operations<br>
n_gunzip        2590700    22.81 Gunzip operations<br>
sess_pipe_overflow      0     .  Dropped sessions due to<br>
session pipe overflow<br>
LCK.sms.creat        1     0.00 Created locks<br>
LCK.sms.destroy       0     0.00 Destroyed locks<br>
LCK.sms.locks       2079     0.02 Lock Operations<br>
LCK.sms.colls        0     0.00 Collisions<br>
LCK.smp.creat        0     0.00 Created locks<br>
LCK.smp.destroy       0     0.00 Destroyed locks<br>
LCK.smp.locks        0     0.00 Lock Operations<br>
LCK.smp.colls        0     0.00 Collisions<br>
LCK.sma.creat        1     0.00 Created locks<br>
LCK.sma.destroy       0     0.00 Destroyed locks<br>
LCK.sma.locks     11567121    101.85 Lock Operations<br>
LCK.sma.colls        0     0.00 Collisions<br>
LCK.smf.creat        1     0.00 Created locks<br>
LCK.smf.destroy       0     0.00 Destroyed locks<br>
LCK.smf.locks     2137363    18.82 Lock Operations<br>
LCK.smf.colls        0     0.00 Collisions<br>
LCK.hsl.creat        0     0.00 Created locks<br>
LCK.hsl.destroy       0     0.00 Destroyed locks<br>
LCK.hsl.locks        0     0.00 Lock Operations<br>
LCK.hsl.colls        0     0.00 Collisions<br>
LCK.hcb.creat        1     0.00 Created locks<br>
LCK.hcb.destroy       0     0.00 Destroyed locks<br>
LCK.hcb.locks      881563     7.76 Lock Operations<br>
LCK.hcb.colls        0     0.00 Collisions<br>
LCK.hcl.creat        0     0.00 Created locks<br>
LCK.hcl.destroy       0     0.00 Destroyed locks<br>
LCK.hcl.locks        0     0.00 Lock Operations<br>
LCK.hcl.colls        0     0.00 Collisions<br>
LCK.vcl.creat        1     0.00 Created locks<br>
LCK.vcl.destroy       0     0.00 Destroyed locks<br>
LCK.vcl.locks      10574     0.09 Lock Operations<br>
LCK.vcl.colls        0     0.00 Collisions<br>
LCK.stat.creat        1     0.00 Created locks<br>
LCK.stat.destroy       0     0.00 Destroyed locks<br>
LCK.stat.locks     2115908    18.63 Lock Operations<br>
LCK.stat.colls        0     0.00 Collisions<br>
LCK.sessmem.creat      1     0.00 Created locks<br>
LCK.sessmem.destroy      0     0.00 Destroyed locks<br>
LCK.sessmem.locks    2129936    18.75 Lock Operations<br>
LCK.sessmem.colls       0     0.00 Collisions<br>
LCK.wstat.creat        1     0.00 Created locks<br>
LCK.wstat.destroy       0     0.00 Destroyed locks<br>
LCK.wstat.locks      239816     2.11 Lock Operations<br>
LCK.wstat.colls        0     0.00 Collisions<br>
LCK.herder.creat        1     0.00 Created locks<br>
LCK.herder.destroy       0     0.00 Destroyed locks<br>
LCK.herder.locks      6034     0.05 Lock Operations<br>
LCK.herder.colls        0     0.00 Collisions<br>
LCK.wq.creat          2     0.00 Created locks<br>
LCK.wq.destroy         0     0.00 Destroyed locks<br>
LCK.wq.locks      10137930    89.26 Lock Operations<br>
LCK.wq.colls          0     0.00 Collisions<br>
LCK.objhdr.creat     471022     4.15 Created locks<br>
LCK.objhdr.destroy    410522     3.61 Destroyed locks<br>
LCK.objhdr.locks    11873362    104.54 Lock Operations<br>
LCK.objhdr.colls        0     0.00 Collisions<br>
LCK.exp.creat         1     0.00 Created locks<br>
LCK.exp.destroy        0     0.00 Destroyed locks<br>
LCK.exp.locks      1086349     9.57 Lock Operations<br>
LCK.exp.colls         0     0.00 Collisions<br>
LCK.lru.creat         2     0.00 Created locks<br>
LCK.lru.destroy        0     0.00 Destroyed locks<br>
LCK.lru.locks       523780     4.61 Lock Operations<br>
LCK.lru.colls         0     0.00 Collisions<br>
LCK.cli.creat         1     0.00 Created locks<br>
LCK.cli.destroy        0     0.00 Destroyed locks<br>
LCK.cli.locks        877     0.01 Lock Operations<br>
LCK.cli.colls         0     0.00 Collisions<br>
LCK.ban.creat         1     0.00 Created locks<br>
LCK.ban.destroy        0     0.00 Destroyed locks<br>
LCK.ban.locks      1071376     9.43 Lock Operations<br>
LCK.ban.colls         0     0.00 Collisions<br>
LCK.vbp.creat         1     0.00 Created locks<br>
LCK.vbp.destroy        0     0.00 Destroyed locks<br>
LCK.vbp.locks         0     0.00 Lock Operations<br>
LCK.vbp.colls         0     0.00 Collisions<br>
LCK.vbe.creat         1     0.00 Created locks<br>
LCK.vbe.destroy        0     0.00 Destroyed locks<br>
LCK.vbe.locks      1183870    10.42 Lock Operations<br>
LCK.vbe.colls         0     0.00 Collisions<br>
LCK.backend.creat       2     0.00 Created locks<br>
LCK.backend.destroy      0     0.00 Destroyed locks<br>
LCK.backend.locks    6560980    57.77 Lock Operations<br>
LCK.backend.colls       0     0.00 Collisions<br>
SMF.s0.c_req       943045     8.30 Allocator requests<br>
SMF.s0.c_fail         0     0.00 Allocator failures<br>
SMF.s0.c_bytes    57519628288  506450.67 Bytes allocated<br>
SMF.s0.c_freed    54163292160  476898.69 Bytes freed<br>
SMF.s0.g_alloc      129160     .  Allocations outstanding<br>
SMF.s0.g_bytes    3356336128     .  Bytes outstanding<br>
SMF.s0.g_space     938631168     .  Bytes available<br>
SMF.s0.g_smf       138281     .  N struct smf<br>
SMF.s0.g_smf_frag      9118     .  N small free smf<br>
SMF.s0.g_smf_large       3     .  N large free smf<br>
SMA.Transient.c_req   4780332    42.09 Allocator requests<br>
SMA.Transient.c_fail      0     0.00 Allocator failures<br>
SMA.Transient.c_bytes 284689405959Â Â 2506642.42 Bytes allocated<br>
SMA.Transient.c_freed 284688351047Â Â 2506633.13 Bytes freed<br>
SMA.Transient.g_alloc      85     .  Allocations outstanding<br>
SMA.Transient.g_bytes   1054912     .  Bytes outstanding<br>
SMA.Transient.g_space      0     .  Bytes available<br>
VBE.default(127.0.0.1,,8080).vcls      2     .  VCL references<br>
VBE.default(127.0.0.1,,8080).happy      0     .  Happy health<br>
probes<br>
VBE.fpm(127.0.0.1,,8081).vcls        2     .  VCL references<br>
VBE.fpm(127.0.0.1,,8081).happy        0     .  Happy health<br>
probes<br>
<br>
--<br>
LAMP solutions GmbH<br>
Gostenhofer Hauptstrasse 35<br>
90443 Nuernberg<br>
<br>
Amtsgericht Nuernberg: HRB 22366<br>
Geschaeftsfuehrer: Heiko Schubert<br>
<br>
Es gelten unsere allgemeinen Geschaeftsbedingungen.<br>
<a href="http://www.lamp-solutions.de/agb/" rel="noreferrer" target="_blank">http://www.lamp-solutions.de/agb/</a><br>
<br>
Telefon   : 0911 / 376 516 0<br>
Fax     : 0911 / 376 516 11<br>
E-Mail    : <a href="mailto:support@lamp-solutions.de">support@lamp-solutions.de</a><br>
Web     : <a href="http://www.lamp-solutions.de" rel="noreferrer" target="_blank">www.lamp-solutions.de</a><br>
Facebook   : <a href="http://www.facebook.com/LAMPsolutions" rel="noreferrer" target="_blank">http://www.facebook.com/LAMPsolutions</a><br>
Twitter   : <a href="http://twitter.com/#!/lampsolutions" rel="noreferrer" target="_blank">http://twitter.com/#!/lampsolutions</a><br>
<br>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
</blockquote></div><br></div>