Question about varnish memory usage
Adam Schumacher
adam.schumacher at flightaware.com
Fri Sep 19 18:36:26 CEST 2014
>>
>> The system is currently fully using available physical memory and has burned
>> through about 15% of the swap space. The varnishstat was taken after the
>> physical memory was exhausted and with just a few Gigs of swap in use. Here
>> is a current snapshot from top and varnishstat –1 if that helps.
>
> Could it be that your memory allocator is leaking memory? The varnishstat
> looks decent, there are no counters that are off the charts. Last time we saw
> behaviour like that was when we were seeing seeing leaks with malloc on Linux,
> before we switched to jemalloc.
>
> I think Varnish will try to link with jemalloc if it is available on your
> system. Is it? As you can see there are quite a few allocations happening and
> if that memory is not reclaimed and reused you'll be leaking quite a bit.
>
My understanding is that the base malloc in freebsd 9.2 is jemalloc — the
code was ported back into the base malloc.c. Between FreeBSD 9.2 and
FreeBSD 10.0, they changed how it is done, and now is using the pure
jemalloc.c code that should be the same as in Linux.
https://svnweb.freebsd.org/base/release/9.2.0/lib/libc/stdlib/malloc.c?revis
ion=255898&view=markup
Vs
https://svnweb.freebsd.org/base/release/10.0.0/contrib/jemalloc/src/jemalloc
.c?revision=260789&view=markup
That said, we have a freebsd 10 server running the same version of varnishd
that has the same problem (the reason I had discounted freebsd as the
culprit initially). This server’s hardware is much newer and as you can see
has quite a bit more physical memory as well. It is running with a few more
threads as you can see, because it has more cpu cores and was configured
with more thread_pools. Otherwise, it is the same as our 9.2 boxes.
>From our FreeBSD 10 box running varnish:
CPU: 4.0% user, 0.0% nice, 0.7% system, 0.2% interrupt, 95.1% idle
Mem: 4509M Active, 111G Inact, 6229M Wired, 73M Cache, 2757M Free
ARC: 1911M Total, 464M MFU, 730M MRU, 400K Anon, 390M Header, 326M Other
Swap: 32G Total, 8353M Used, 24G Free, 25% Inuse
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU
COMMAND
1311 www 1291 20 0 354G 116G uwait 8 476.0H 21.19%
varnishd
$ uname -v
> FreeBSD 10.0-RELEASE-p7 #0: Tue Jul 8 06:37:44 UTC 2014
root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC
varnishstat –1:
client_conn 317914959 133.36 Client connections accepted
client_drop 0 0.00 Connection dropped, no sess/wrk
client_req 507413109 212.85 Client requests received
cache_hit 236222630 99.09 Cache hits
cache_hitpass 21505999 9.02 Cache hits for pass
cache_miss 213793146 89.68 Cache misses
backend_conn 111736980 46.87 Backend conn. success
backend_unhealthy 0 0.00 Backend conn. not attempted
backend_busy 8690564 3.65 Backend conn. too many
backend_fail 516882 0.22 Backend conn. failures
backend_reuse 265640872 111.43 Backend conn. reuses
backend_toolate 14139104 5.93 Backend conn. was closed
backend_recycle 279780619 117.36 Backend conn. recycles
backend_retry 25784 0.01 Backend conn. retry
fetch_head 1770 0.00 Fetch head
fetch_length 121268619 50.87 Fetch with Length
fetch_chunked 254268690 106.66 Fetch chunked
fetch_eof 0 0.00 Fetch EOF
fetch_bad 0 0.00 Fetch had bad headers
fetch_close 1590881 0.67 Fetch wanted close
fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed
fetch_zero 0 0.00 Fetch zero len
fetch_failed 167507 0.07 Fetch failed
fetch_1xx 0 0.00 Fetch no body (1xx)
fetch_204 0 0.00 Fetch no body (204)
fetch_304 172230 0.07 Fetch no body (304)
n_sess_mem 6946 . N struct sess_mem
n_sess 593 . N struct sess
n_object 992149 . N struct object
n_vampireobject 0 . N unresurrected objects
n_objectcore 993155 . N struct objectcore
n_objecthead 988241 . N struct objecthead
n_waitinglist 1273 . N struct waitinglist
n_vbc 71 . N struct vbc
n_wrk 1280 . N worker threads
n_wrk_create 1595 0.00 N worker threads created
n_wrk_failed 0 0.00 N worker threads not created
n_wrk_max 0 0.00 N worker threads limited
n_wrk_lqueue 0 0.00 work request queue length
n_wrk_queued 4530 0.00 N queued work requests
n_wrk_drop 0 0.00 N dropped work requests
n_backend 7 . N backends
n_expired 135419385 . N expired objects
n_lru_nuked 76410300 . N LRU nuked objects
n_lru_moved 128024378 . N LRU moved objects
losthdr 28 0.00 HTTP header overflows
n_objsendfile 0 0.00 Objects sent with sendfile
n_objwrite 355076718 148.95 Objects sent with write
n_objoverflow 0 0.00 Objects overflowing workspace
s_sess 317934238 133.37 Total Sessions
s_req 507413109 212.85 Total Requests
s_pipe 0 0.00 Total pipe
s_pass 165619470 69.47 Total pass
s_fetch 377135515 158.20 Total fetch
s_hdrbytes 189267454297 79393.14 Total header bytes
s_bodybytes 3732756334613 1565801.44 Total body bytes
sess_closed 35770111 15.00 Session Closed
sess_pipeline 24574 0.01 Session Pipeline
sess_readahead 7384 0.00 Session Read Ahead
sess_linger 476097250 199.71 Session Linger
sess_herd 494921706 207.61 Session herd
shm_records 45611717569 19133.02 SHM records
shm_writes 2842104094 1192.19 SHM writes
shm_flushes 2794 0.00 SHM flushes due to overflow
shm_cont 5861694 2.46 SHM MTX contention
shm_cycles 19426 0.01 SHM cycles through buffer
sms_nreq 2859746 1.20 SMS allocator requests
sms_nobj 0 . SMS outstanding allocations
sms_nbytes 0 . SMS outstanding bytes
sms_balloc 8118447295 . SMS bytes allocated
sms_bfree 8118447295 . SMS bytes freed
backend_req 377315426 158.27 Backend requests made
n_vcl 1 0.00 N vcl total
n_vcl_avail 1 0.00 N vcl available
n_vcl_discard 0 0.00 N vcl discarded
n_ban 1 . N total active bans
n_ban_gone 1 . N total gone bans
n_ban_add 1 0.00 N new bans added
n_ban_retire 0 0.00 N old bans deleted
n_ban_obj_test 0 0.00 N objects tested
n_ban_re_test 0 0.00 N regexps tested against
n_ban_dups 0 0.00 N duplicate bans removed
hcb_nolock 0 0.00 HCB Lookups without lock
hcb_lock 0 0.00 HCB Lookups with lock
hcb_insert 0 0.00 HCB Inserts
esi_errors 0 0.00 ESI parse errors (unlock)
esi_warnings 0 0.00 ESI parse warnings (unlock)
accept_fail 459 0.00 Accept failures
client_drop_late 0 0.00 Connection dropped late
uptime 2383927 1.00 Client uptime
dir_dns_lookups 0 0.00 DNS director lookups
dir_dns_failed 0 0.00 DNS director failed lookups
dir_dns_hit 0 0.00 DNS director cached lookups hit
dir_dns_cache_full 0 0.00 DNS director full dnscache
vmods 0 . Loaded VMODs
n_gzip 257811067 108.15 Gzip operations
n_gunzip 276663353 116.05 Gunzip operations
sess_pipe_overflow 0 . Dropped sessions due to session
pipe overflow
LCK.sms.creat 1 0.00 Created locks
LCK.sms.destroy 0 0.00 Destroyed locks
LCK.sms.locks 8579238 3.60 Lock Operations
LCK.sms.colls 0 0.00 Collisions
LCK.smp.creat 0 0.00 Created locks
LCK.smp.destroy 0 0.00 Destroyed locks
LCK.smp.locks 0 0.00 Lock Operations
LCK.smp.colls 0 0.00 Collisions
LCK.sma.creat 2 0.00 Created locks
LCK.sma.destroy 0 0.00 Destroyed locks
LCK.sma.locks 2111376064 885.67 Lock Operations
LCK.sma.colls 0 0.00 Collisions
LCK.smf.creat 0 0.00 Created locks
LCK.smf.destroy 0 0.00 Destroyed locks
LCK.smf.locks 0 0.00 Lock Operations
LCK.smf.colls 0 0.00 Collisions
LCK.hsl.creat 0 0.00 Created locks
LCK.hsl.destroy 0 0.00 Destroyed locks
LCK.hsl.locks 0 0.00 Lock Operations
LCK.hsl.colls 0 0.00 Collisions
LCK.hcb.creat 0 0.00 Created locks
LCK.hcb.destroy 0 0.00 Destroyed locks
LCK.hcb.locks 0 0.00 Lock Operations
LCK.hcb.colls 0 0.00 Collisions
LCK.hcl.creat 250007 0.10 Created locks
LCK.hcl.destroy 0 0.00 Destroyed locks
LCK.hcl.locks 942040132 395.16 Lock Operations
LCK.hcl.colls 0 0.00 Collisions
LCK.vcl.creat 1 0.00 Created locks
LCK.vcl.destroy 0 0.00 Destroyed locks
LCK.vcl.locks 4406502 1.85 Lock Operations
LCK.vcl.colls 0 0.00 Collisions
LCK.stat.creat 1 0.00 Created locks
LCK.stat.destroy 0 0.00 Destroyed locks
LCK.stat.locks 317940634 133.37 Lock Operations
LCK.stat.colls 0 0.00 Collisions
LCK.sessmem.creat 1 0.00 Created locks
LCK.sessmem.destroy 0 0.00 Destroyed locks
LCK.sessmem.locks 318157037 133.46 Lock Operations
LCK.sessmem.colls 0 0.00 Collisions
LCK.wstat.creat 1 0.00 Created locks
LCK.wstat.destroy 0 0.00 Destroyed locks
LCK.wstat.locks 6160352 2.58 Lock Operations
LCK.wstat.colls 0 0.00 Collisions
LCK.herder.creat 1 0.00 Created locks
LCK.herder.destroy 0 0.00 Destroyed locks
LCK.herder.locks 1734 0.00 Lock Operations
LCK.herder.colls 0 0.00 Collisions
LCK.wq.creat 32 0.00 Created locks
LCK.wq.destroy 0 0.00 Destroyed locks
LCK.wq.locks 1140526679 478.42 Lock Operations
LCK.wq.colls 0 0.00 Collisions
LCK.objhdr.creat 183579053 77.01 Created locks
LCK.objhdr.destroy 182591336 76.59 Destroyed locks
LCK.objhdr.locks 1584172928 664.52 Lock Operations
LCK.objhdr.colls 0 0.00 Collisions
LCK.exp.creat 1 0.00 Created locks
LCK.exp.destroy 0 0.00 Destroyed locks
LCK.exp.locks 426984689 179.11 Lock Operations
LCK.exp.colls 0 0.00 Collisions
LCK.lru.creat 2 0.00 Created locks
LCK.lru.destroy 0 0.00 Destroyed locks
LCK.lru.locks 289232005 121.33 Lock Operations
LCK.lru.colls 0 0.00 Collisions
LCK.cli.creat 1 0.00 Created locks
LCK.cli.destroy 0 0.00 Destroyed locks
LCK.cli.locks 1180095 0.50 Lock Operations
LCK.cli.colls 0 0.00 Collisions
LCK.ban.creat 1 0.00 Created locks
LCK.ban.destroy 0 0.00 Destroyed locks
LCK.ban.locks 427224495 179.21 Lock Operations
LCK.ban.colls 0 0.00 Collisions
LCK.vbp.creat 1 0.00 Created locks
LCK.vbp.destroy 0 0.00 Destroyed locks
LCK.vbp.locks 1809670 0.76 Lock Operations
LCK.vbp.colls 0 0.00 Collisions
LCK.vbe.creat 1 0.00 Created locks
LCK.vbe.destroy 0 0.00 Destroyed locks
LCK.vbe.locks 224508313 94.18 Lock Operations
LCK.vbe.colls 0 0.00 Collisions
LCK.backend.creat 7 0.00 Created locks
LCK.backend.destroy 0 0.00 Destroyed locks
LCK.backend.locks 904980852 379.62 Lock Operations
LCK.backend.colls 0 0.00 Collisions
SMA.s0.c_req 578846458 242.81 Allocator requests
SMA.s0.c_fail 76630001 32.14 Allocator failures
SMA.s0.c_bytes 25682079523063 10773014.24 Bytes allocated
SMA.s0.c_freed 25675638298667 10770312.30 Bytes freed
SMA.s0.g_alloc 2023035 . Allocations outstanding
SMA.s0.g_bytes 6441224396 . Bytes outstanding
SMA.s0.g_space 1226548 . Bytes available
SMA.Transient.c_req 343584190 144.13 Allocator requests
SMA.Transient.c_fail 0 0.00 Allocator failures
SMA.Transient.c_bytes 21240906064202 8910048.87 Bytes allocated
SMA.Transient.c_freed 21240900389430 8910046.49 Bytes freed
SMA.Transient.g_alloc 5520 . Allocations outstanding
SMA.Transient.g_bytes 5674772 . Bytes outstanding
SMA.Transient.g_space 0 . Bytes available
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20140919/fc57cdd8/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5831 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20140919/fc57cdd8/attachment-0001.bin>
More information about the varnish-misc
mailing list