Question about varnish memory usage
Adam Schumacher
adam.schumacher at flightaware.com
Thu Sep 18 22:30:25 CEST 2014
From: Per Buer <perbu at varnish-software.com>
>
>
> Wrt, Dridis point about threads might also not be that relevant as only
> 700-800 threads are being created. Session workspace is virtual until
> allocated so I don't think that is where your memory is disappearing.
>
> 24 thread pools is a bit odd (you only need two) but don't know of any leaks
> it might trigger.
>
> I'd stay away from classic hashing as cribit is stable and you're the only one
> I know that uses it.
>
> When is the varnishstat taken? Right before running out of memory? That would
> probably be useful.
There are 24 thread pools because there are 24 cores in the server and I’m
sure someone read somewhere that a pool per was the way to go. Probably
that was a long time ago in a galaxy far, far away….
What are the significant differences between classic and cribit hashing?
The system is currently fully using available physical memory and has burned
through about 15% of the swap space. The varnishstat was taken after the
physical memory was exhausted and with just a few Gigs of swap in use. Here
is a current snapshot from top and varnishstat –1 if that helps.
Mem: 24G Active, 2857M Inact, 3757M Wired, 433M Cache, 510M Free
ARC: 1715M Total, 555M MFU, 274M MRU, 1042K Anon, 216M Header, 670M Other
Swap: 32G Total, 5174M Used, 27G Free, 15% Inuse
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU
COMMAND
87309 www 803 20 0 49189M 23483M select 22 130.1H 5.27%
varnishd
client_conn 115492418 106.06 Client connections accepted
client_drop 0 0.00 Connection dropped, no sess/wrk
client_req 195339845 179.39 Client requests received
cache_hit 94748999 87.01 Cache hits
cache_hitpass 8249240 7.58 Cache hits for pass
cache_miss 79893031 73.37 Cache misses
backend_conn 39130304 35.93 Backend conn. success
backend_unhealthy 0 0.00 Backend conn. not attempted
backend_busy 132086 0.12 Backend conn. too many
backend_fail 0 0.00 Backend conn. failures
backend_reuse 98773816 90.71 Backend conn. reuses
backend_toolate 6237866 5.73 Backend conn. was closed
backend_recycle 105011602 96.44 Backend conn. recycles
backend_retry 11220 0.01 Backend conn. retry
fetch_head 577 0.00 Fetch head
fetch_length 48290509 44.35 Fetch with Length
fetch_chunked 89008238 81.74 Fetch chunked
fetch_eof 0 0.00 Fetch EOF
fetch_bad 0 0.00 Fetch had bad headers
fetch_close 509908 0.47 Fetch wanted close
fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed
fetch_zero 0 0.00 Fetch zero len
fetch_failed 45097 0.04 Fetch failed
fetch_1xx 0 0.00 Fetch no body (1xx)
fetch_204 0 0.00 Fetch no body (204)
fetch_304 68372 0.06 Fetch no body (304)
n_sess_mem 1150 . N struct sess_mem
n_sess 722 . N struct sess
n_object 942029 . N struct object
n_vampireobject 0 . N unresurrected objects
n_objectcore 942471 . N struct objectcore
n_objecthead 936744 . N struct objecthead
n_waitinglist 624 . N struct waitinglist
n_vbc 117 . N struct vbc
n_wrk 792 . N worker threads
n_wrk_create 792 0.00 N worker threads created
n_wrk_failed 0 0.00 N worker threads not created
n_wrk_max 0 0.00 N worker threads limited
n_wrk_lqueue 0 0.00 work request queue length
n_wrk_queued 0 0.00 N queued work requests
n_wrk_drop 0 0.00 N dropped work requests
n_backend 7 . N backends
n_expired 50061996 . N expired objects
n_lru_nuked 28851249 . N LRU nuked objects
n_lru_moved 56196515 . N LRU moved objects
losthdr 7 0.00 HTTP header overflows
n_objsendfile 0 0.00 Objects sent with sendfile
n_objwrite 139062543 127.71 Objects sent with write
n_objoverflow 0 0.00 Objects overflowing workspace
s_sess 115493404 106.06 Total Sessions
s_req 195339845 179.39 Total Requests
s_pipe 0 0.00 Total pipe
s_pass 58017202 53.28 Total pass
s_fetch 137832710 126.58 Total fetch
s_hdrbytes 73466348331 67467.23 Total header bytes
s_bodybytes 1475428907061 1354948.26 Total body bytes
sess_closed 11135893 10.23 Session Closed
sess_pipeline 8340 0.01 Session Pipeline
sess_readahead 3402 0.00 Session Read Ahead
sess_linger 186082340 170.89 Session Linger
sess_herd 193483825 177.68 Session herd
shm_records 17182889691 15779.77 SHM records
shm_writes 1058678573 972.23 SHM writes
shm_flushes 4719 0.00 SHM flushes due to overflow
shm_cont 335494 0.31 SHM MTX contention
shm_cycles 7346 0.01 SHM cycles through buffer
sms_nreq 405919 0.37 SMS allocator requests
sms_nobj 0 . SMS outstanding allocations
sms_nbytes 0 . SMS outstanding bytes
sms_balloc 1127850835 . SMS bytes allocated
sms_bfree 1127850835 . SMS bytes freed
backend_req 137878631 126.62 Backend requests made
n_vcl 1 0.00 N vcl total
n_vcl_avail 1 0.00 N vcl available
n_vcl_discard 0 0.00 N vcl discarded
n_ban 1 . N total active bans
n_ban_gone 1 . N total gone bans
n_ban_add 1 0.00 N new bans added
n_ban_retire 0 0.00 N old bans deleted
n_ban_obj_test 0 0.00 N objects tested
n_ban_re_test 0 0.00 N regexps tested against
n_ban_dups 0 0.00 N duplicate bans removed
hcb_nolock 0 0.00 HCB Lookups without lock
hcb_lock 0 0.00 HCB Lookups with lock
hcb_insert 0 0.00 HCB Inserts
esi_errors 0 0.00 ESI parse errors (unlock)
esi_warnings 0 0.00 ESI parse warnings (unlock)
accept_fail 547 0.00 Accept failures
client_drop_late 0 0.00 Connection dropped late
uptime 1088919 1.00 Client uptime
dir_dns_lookups 0 0.00 DNS director lookups
dir_dns_failed 0 0.00 DNS director failed lookups
dir_dns_hit 0 0.00 DNS director cached lookups hit
dir_dns_cache_full 0 0.00 DNS director full dnscache
vmods 0 . Loaded VMODs
n_gzip 92821054 85.24 Gzip operations
n_gunzip 98789399 90.72 Gunzip operations
sess_pipe_overflow 0 . Dropped sessions due to session
pipe overflow
LCK.sms.creat 1 0.00 Created locks
LCK.sms.destroy 0 0.00 Destroyed locks
LCK.sms.locks 1217757 1.12 Lock Operations
LCK.sms.colls 0 0.00 Collisions
LCK.smp.creat 0 0.00 Created locks
LCK.smp.destroy 0 0.00 Destroyed locks
LCK.smp.locks 0 0.00 Lock Operations
LCK.smp.colls 0 0.00 Collisions
LCK.sma.creat 2 0.00 Created locks
LCK.sma.destroy 0 0.00 Destroyed locks
LCK.sma.locks 762080017 699.85 Lock Operations
LCK.sma.colls 0 0.00 Collisions
LCK.smf.creat 0 0.00 Created locks
LCK.smf.destroy 0 0.00 Destroyed locks
LCK.smf.locks 0 0.00 Lock Operations
LCK.smf.colls 0 0.00 Collisions
LCK.hsl.creat 0 0.00 Created locks
LCK.hsl.destroy 0 0.00 Destroyed locks
LCK.hsl.locks 0 0.00 Lock Operations
LCK.hsl.colls 0 0.00 Collisions
LCK.hcb.creat 0 0.00 Created locks
LCK.hcb.destroy 0 0.00 Destroyed locks
LCK.hcb.locks 0 0.00 Lock Operations
LCK.hcb.colls 0 0.00 Collisions
LCK.hcl.creat 250007 0.23 Created locks
LCK.hcl.destroy 0 0.00 Destroyed locks
LCK.hcl.locks 364838330 335.05 Lock Operations
LCK.hcl.colls 0 0.00 Collisions
LCK.vcl.creat 1 0.00 Created locks
LCK.vcl.destroy 0 0.00 Destroyed locks
LCK.vcl.locks 1404720 1.29 Lock Operations
LCK.vcl.colls 0 0.00 Collisions
LCK.stat.creat 1 0.00 Created locks
LCK.stat.destroy 0 0.00 Destroyed locks
LCK.stat.locks 115493914 106.06 Lock Operations
LCK.stat.colls 0 0.00 Collisions
LCK.sessmem.creat 1 0.00 Created locks
LCK.sessmem.destroy 0 0.00 Destroyed locks
LCK.sessmem.locks 115660082 106.22 Lock Operations
LCK.sessmem.colls 0 0.00 Collisions
LCK.wstat.creat 1 0.00 Created locks
LCK.wstat.destroy 0 0.00 Destroyed locks
LCK.wstat.locks 3815488 3.50 Lock Operations
LCK.wstat.colls 0 0.00 Collisions
LCK.herder.creat 1 0.00 Created locks
LCK.herder.destroy 0 0.00 Destroyed locks
LCK.herder.locks 1 0.00 Lock Operations
LCK.herder.colls 0 0.00 Collisions
LCK.wq.creat 24 0.00 Created locks
LCK.wq.destroy 0 0.00 Destroyed locks
LCK.wq.locks 436653782 401.00 Lock Operations
LCK.wq.colls 0 0.00 Collisions
LCK.objhdr.creat 69117038 63.47 Created locks
LCK.objhdr.destroy 68180433 62.61 Destroyed locks
LCK.objhdr.locks 605204037 555.78 Lock Operations
LCK.objhdr.colls 0 0.00 Collisions
LCK.exp.creat 1 0.00 Created locks
LCK.exp.destroy 0 0.00 Destroyed locks
LCK.exp.locks 159855422 146.80 Lock Operations
LCK.exp.colls 0 0.00 Collisions
LCK.lru.creat 2 0.00 Created locks
LCK.lru.destroy 0 0.00 Destroyed locks
LCK.lru.locks 108706511 99.83 Lock Operations
LCK.lru.colls 0 0.00 Collisions
LCK.cli.creat 1 0.00 Created locks
LCK.cli.destroy 0 0.00 Destroyed locks
LCK.cli.locks 544317 0.50 Lock Operations
LCK.cli.colls 0 0.00 Collisions
LCK.ban.creat 1 0.00 Created locks
LCK.ban.destroy 0 0.00 Destroyed locks
LCK.ban.locks 159911741 146.85 Lock Operations
LCK.ban.colls 0 0.00 Collisions
LCK.vbp.creat 1 0.00 Created locks
LCK.vbp.destroy 0 0.00 Destroyed locks
LCK.vbp.locks 826695 0.76 Lock Operations
LCK.vbp.colls 0 0.00 Collisions
LCK.vbe.creat 1 0.00 Created locks
LCK.vbe.destroy 0 0.00 Destroyed locks
LCK.vbe.locks 78260491 71.87 Lock Operations
LCK.vbe.colls 0 0.00 Collisions
LCK.backend.creat 7 0.00 Created locks
LCK.backend.destroy 0 0.00 Destroyed locks
LCK.backend.locks 327541647 300.80 Lock Operations
LCK.backend.colls 0 0.00 Collisions
SMA.s0.c_req 211189969 193.94 Allocator requests
SMA.s0.c_fail 28921596 26.56 Allocator failures
SMA.s0.c_bytes 9340561806222 8577829.76 Bytes allocated
SMA.s0.c_freed 9334120384877 8571914.33 Bytes freed
SMA.s0.g_alloc 1924886 . Allocations outstanding
SMA.s0.g_bytes 6441310878 . Bytes outstanding
SMA.s0.g_space 1140066 . Bytes available
SMA.Transient.c_req 122683830 112.67 Allocator requests
SMA.Transient.c_fail 0 0.00 Allocator failures
SMA.Transient.c_bytes 7588011522470 6968389.31 Bytes allocated
SMA.Transient.c_freed 7588004757463 6968383.10 Bytes freed
SMA.Transient.g_alloc 6530 . Allocations outstanding
SMA.Transient.g_bytes 6765007 . Bytes outstanding
SMA.Transient.g_space 0 . Bytes available
>
>>
>>>> >As a hotfix I would cap Transient at, say, 2G. "varnishd -s
>>>> Transient=malloc,2G". You can probably up your main storage quite a bit
>>>> with this in place.
>>
>> I’ll give it a shot and let you know. It usually takes a couple days before
>> filling up all the physical memory.
>
> Well, I'm somewhat doubtful it it will actually work.
>
I guess it is a good thing I hadn’t actually made the change then. Since
this is a production system, I was going to wait until our maintenance
period tonight.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20140918/a1cfae8f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5831 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20140918/a1cfae8f/attachment-0001.bin>
More information about the varnish-misc
mailing list