Varnish 3.0.5 memory leak?
Louis-Philippe Gauthier
louis-philippe.gauthier at adgear.com
Mon Jun 2 15:04:56 CEST 2014
Hi, we've been experimenting with varnish 3.0.5 and we seem to be
experiencing a memory leak.
We're running varnish-3.0.5-r4 (from portage):
http://sources.gentoo.org/cgi-bin/viewvc.cgi/gentoo-x86/www-servers/varnish/varnish-3.0.5-r4.ebuild
In the graph bellow you can see that the memory kept growing even after the
malloc memory (12G) was full. I understand that there's a 1K overhead per
object and that Varnish will use more then 12G, but it keeps growing even
after the object space is full (now at 43.4G).
[image: Inline image 2]
box:
$ uname -a
Linux 3.8.13-gentoo #3 SMP Wed Jun 26 11:29:09 EDT 2013 x86_64 Intel(R)
Xeon(R) CPU E5-2450 0 @ 2.10GHz GenuineIntel GNU/Linux
configs:
$ cat /etc/conf.d/varnishd
# /etc/conf.d/varnishd
VARNISHD="/usr/sbin/varnishd"
VARNISHADM="/usr/bin/varnishadm"
CONFIGFILE="/etc/varnish/default.vcl"
VARNISHD_OPTS="-a :80 -p default_grace=600 -s malloc,12G -p
thread_pool_max=2000 -p sess_timeout=120 -p thread_pool_min=100 -f
$CONFIGFILE"
rc_ulimit="-n 32786 -l 82000"
$ cat /etc/varnish/default.vcl
sub vcl_fetch {
unset beresp.http.access-control-allow-origin;
unset beresp.http.cache-control;
unset beresp.http.content-type;
unset beresp.http.p3p;
unset beresp.http.pragma;
unset beresp.http.server;
set beresp.do_stream = false;
set beresp.http.Content-Type = "application/json";
set beresp.ttl = 12h;
return (deliver);
}
sub vcl_recv {
if (req.url ~ "^/seg\?") {
set req.url = regsub(req.url, "^/seg\?",
"/db/client/1/seg.json?adsafe_url=");
}
else if (req.url ~ "^/raw\?") {
set req.url = regsub(req.url, "^/raw\?",
"/db/client/1/itgrl.json?adsafe_url=");
}
else {
error 404 "Nope";
}
}
backend default {
.host = "localhost";
.port = "8080";
}
varnishstat:
$ varnishstat -1
client_conn 118007 1.43 Client connections accepted
client_drop 0 0.00 Connection dropped, no sess/wrk
client_req 2674303461 32451.20 Client requests received
cache_hit 2581606982 31326.38 Cache hits
cache_hitpass 0 0.00 Cache hits for pass
cache_miss 92696487 1124.82 Cache misses
backend_conn 143901 1.75 Backend conn. success
backend_unhealthy 0 0.00 Backend conn. not attempted
backend_busy 0 0.00 Backend conn. too many
backend_fail 113149 1.37 Backend conn. failures
backend_reuse 92437824 1121.68 Backend conn. reuses
backend_toolate 143808 1.75 Backend conn. was closed
backend_recycle 92583388 1123.45 Backend conn. recycles
backend_retry 44 0.00 Backend conn. retry
fetch_head 0 0.00 Fetch head
fetch_length 92583297 1123.45 Fetch with Length
fetch_chunked 0 0.00 Fetch chunked
fetch_eof 0 0.00 Fetch EOF
fetch_bad 0 0.00 Fetch had bad headers
fetch_close 0 0.00 Fetch wanted close
fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed
fetch_zero 0 0.00 Fetch zero len
fetch_failed 0 0.00 Fetch failed
fetch_1xx 0 0.00 Fetch no body (1xx)
fetch_204 0 0.00 Fetch no body (204)
fetch_304 0 0.00 Fetch no body (304)
n_sess_mem 2193 . N struct sess_mem
n_sess 1743 . N struct sess
n_object 17035658 . N struct object
n_vampireobject 0 . N unresurrected objects
n_objectcore 17036857 . N struct objectcore
n_objecthead 17287988 . N struct objecthead
n_waitinglist 1224 . N struct waitinglist
n_vbc 21 . N struct vbc
n_wrk 1225 . N worker threads
n_wrk_create 4831 0.06 N worker threads created
n_wrk_failed 0 0.00 N worker threads not created
n_wrk_max 0 0.00 N worker threads limited
n_wrk_lqueue 0 0.00 work request queue length
n_wrk_queued 97676 1.19 N queued work requests
n_wrk_drop 0 0.00 N dropped work requests
n_backend 1 . N backends
n_expired 1913697 . N expired objects
n_lru_nuked 73634035 . N LRU nuked objects
n_lru_moved 597753138 . N LRU moved objects
losthdr 0 0.00 HTTP header overflows
n_objsendfile 0 0.00 Objects sent with sendfile
n_objwrite 2670567550 32405.87 Objects sent with write
n_objoverflow 0 0.00 Objects overflowing workspace
s_sess 118079 1.43 Total Sessions
s_req 2674303508 32451.20 Total Requests
s_pipe 0 0.00 Total pipe
s_pass 0 0.00 Total pass
s_fetch 92583297 1123.45 Total fetch
s_hdrbytes 582930172361 7073536.86 Total header bytes
s_bodybytes 447671320223 5432245.12 Total body bytes
sess_closed 113192 1.37 Session Closed
sess_pipeline 0 0.00 Session Pipeline
sess_readahead 0 0.00 Session Read Ahead
sess_linger 2674190310 32449.83 Session Linger
sess_herd 1005183056 12197.34 Session herd
shm_records 84088575128 1020368.59 SHM records
shm_writes 3867468359 46929.60 SHM writes
shm_flushes 3637 0.04 SHM flushes due to overflow
shm_cont 47998203 582.43 SHM MTX contention
shm_cycles 28175 0.34 SHM cycles through buffer
sms_nreq 113190 1.37 SMS allocator requests
sms_nobj 0 . SMS outstanding allocations
sms_nbytes 0 . SMS outstanding bytes
sms_balloc 47426610 . SMS bytes allocated
sms_bfree 47426610 . SMS bytes freed
backend_req 92566803 1123.25 Backend requests made
n_vcl 1 0.00 N vcl total
n_vcl_avail 1 0.00 N vcl available
n_vcl_discard 0 0.00 N vcl discarded
n_ban 1 . N total active bans
n_ban_gone 1 . N total gone bans
n_ban_add 1 0.00 N new bans added
n_ban_retire 0 0.00 N old bans deleted
n_ban_obj_test 0 0.00 N objects tested
n_ban_re_test 0 0.00 N regexps tested against
n_ban_dups 0 0.00 N duplicate bans removed
hcb_nolock 2665700150 32346.80 HCB Lookups without lock
hcb_lock 91663289 1112.28 HCB Lookups with lock
hcb_insert 91656230 1112.20 HCB Inserts
esi_errors 0 0.00 ESI parse errors (unlock)
esi_warnings 0 0.00 ESI parse warnings (unlock)
accept_fail 0 0.00 Accept failures
client_drop_late 0 0.00 Connection dropped late
uptime 82410 1.00 Client uptime
dir_dns_lookups 0 0.00 DNS director lookups
dir_dns_failed 0 0.00 DNS director failed lookups
dir_dns_hit 0 0.00 DNS director cached lookups hit
dir_dns_cache_full 0 0.00 DNS director full dnscache
vmods 0 . Loaded VMODs
n_gzip 0 0.00 Gzip operations
n_gunzip 0 0.00 Gunzip operations
sess_pipe_overflow 0 . Dropped sessions due to session
pipe overflow
LCK.sms.creat 1 0.00 Created locks
LCK.sms.destroy 0 0.00 Destroyed locks
LCK.sms.locks 339570 4.12 Lock Operations
LCK.sms.colls 0 0.00 Collisions
LCK.smp.creat 0 0.00 Created locks
LCK.smp.destroy 0 0.00 Destroyed locks
LCK.smp.locks 0 0.00 Lock Operations
LCK.smp.colls 0 0.00 Collisions
LCK.sma.creat 2 0.00 Created locks
LCK.sma.destroy 0 0.00 Destroyed locks
LCK.sma.locks 466325781 5658.61 Lock Operations
LCK.sma.colls 0 0.00 Collisions
LCK.smf.creat 0 0.00 Created locks
LCK.smf.destroy 0 0.00 Destroyed locks
LCK.smf.locks 0 0.00 Lock Operations
LCK.smf.colls 0 0.00 Collisions
LCK.hsl.creat 0 0.00 Created locks
LCK.hsl.destroy 0 0.00 Destroyed locks
LCK.hsl.locks 0 0.00 Lock Operations
LCK.hsl.colls 0 0.00 Collisions
LCK.hcb.creat 1 0.00 Created locks
LCK.hcb.destroy 0 0.00 Destroyed locks
LCK.hcb.locks 166286374 2017.79 Lock Operations
LCK.hcb.colls 0 0.00 Collisions
LCK.hcl.creat 0 0.00 Created locks
LCK.hcl.destroy 0 0.00 Destroyed locks
LCK.hcl.locks 0 0.00 Lock Operations
LCK.hcl.colls 0 0.00 Collisions
LCK.vcl.creat 1 0.00 Created locks
LCK.vcl.destroy 0 0.00 Destroyed locks
LCK.vcl.locks 4238534 51.43 Lock Operations
LCK.vcl.colls 0 0.00 Collisions
LCK.stat.creat 1 0.00 Created locks
LCK.stat.destroy 0 0.00 Destroyed locks
LCK.stat.locks 118472 1.44 Lock Operations
LCK.stat.colls 0 0.00 Collisions
LCK.sessmem.creat 1 0.00 Created locks
LCK.sessmem.destroy 0 0.00 Destroyed locks
LCK.sessmem.locks 118731 1.44 Lock Operations
LCK.sessmem.colls 0 0.00 Collisions
LCK.wstat.creat 1 0.00 Created locks
LCK.wstat.destroy 0 0.00 Destroyed locks
LCK.wstat.locks 35382849 429.35 Lock Operations
LCK.wstat.colls 0 0.00 Collisions
LCK.herder.creat 1 0.00 Created locks
LCK.herder.destroy 0 0.00 Destroyed locks
LCK.herder.locks 24026 0.29 Lock Operations
LCK.herder.colls 0 0.00 Collisions
LCK.wq.creat 2 0.00 Created locks
LCK.wq.destroy 0 0.00 Destroyed locks
LCK.wq.locks 2013141564 24428.37 Lock Operations
LCK.wq.colls 0 0.00 Collisions
LCK.objhdr.creat 91644710 1112.06 Created locks
LCK.objhdr.destroy 74373301 902.48 Destroyed locks
LCK.objhdr.locks 10786415928 130887.22 Lock Operations
LCK.objhdr.colls 0 0.00 Collisions
LCK.exp.creat 1 0.00 Created locks
LCK.exp.destroy 0 0.00 Destroyed locks
LCK.exp.locks 168213500 2041.18 Lock Operations
LCK.exp.colls 0 0.00 Collisions
LCK.lru.creat 2 0.00 Created locks
LCK.lru.destroy 0 0.00 Destroyed locks
LCK.lru.locks 166217433 2016.96 Lock Operations
LCK.lru.colls 0 0.00 Collisions
LCK.cli.creat 1 0.00 Created locks
LCK.cli.destroy 0 0.00 Destroyed locks
LCK.cli.locks 27464 0.33 Lock Operations
LCK.cli.colls 0 0.00 Collisions
LCK.ban.creat 1 0.00 Created locks
LCK.ban.destroy 0 0.00 Destroyed locks
LCK.ban.locks 168213537 2041.18 Lock Operations
LCK.ban.colls 0 0.00 Collisions
LCK.vbp.creat 1 0.00 Created locks
LCK.vbp.destroy 0 0.00 Destroyed locks
LCK.vbp.locks 0 0.00 Lock Operations
LCK.vbp.colls 0 0.00 Collisions
LCK.vbe.creat 1 0.00 Created locks
LCK.vbe.destroy 0 0.00 Destroyed locks
LCK.vbe.locks 514115 6.24 Lock Operations
LCK.vbe.colls 0 0.00 Collisions
LCK.backend.creat 1 0.00 Created locks
LCK.backend.destroy 0 0.00 Destroyed locks
LCK.backend.locks 185937951 2256.25 Lock Operations
LCK.backend.colls 0 0.00 Collisions
SMA.s0.c_req 315117125 3823.77 Allocator requests
SMA.s0.c_fail 129837159 1575.50 Allocator failures
SMA.s0.c_bytes 73800148989 895524.20 Bytes allocated
SMA.s0.c_freed 60915247331 739173.00 Bytes freed
SMA.s0.g_alloc 34071316 . Allocations outstanding
SMA.s0.g_bytes 12884901658 . Bytes outstanding
SMA.s0.g_space 230 . Bytes available
SMA.Transient.c_req 12 0.00 Allocator requests
SMA.Transient.c_fail 0 0.00 Allocator failures
SMA.Transient.c_bytes 409344 4.97 Bytes allocated
SMA.Transient.c_freed 409344 4.97 Bytes freed
SMA.Transient.g_alloc 0 . Allocations outstanding
SMA.Transient.g_bytes 0 . Bytes outstanding
SMA.Transient.g_space 0 . Bytes available
VBE.default(38.74.193.146,,8080).vcls 1 . VCL
references
VBE.default(38.74.193.146,,8080).happy 0 . Happy
health probes
Are we doing something stupid? We tried running Varnish 4.0.0-r1 but it was
leaking threads so we downgraded...
Thanks,
Louis-Philippe Gauthier
Team Lead, AdGear Trader
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20140602/bf1d3b43/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image-1.png
Type: image/png
Size: 18251 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20140602/bf1d3b43/attachment-0001.png>
More information about the varnish-dev
mailing list