[Varnish] #295: Varnishd using all available memory, even using limited malloc
Varnish
varnish-bugs at projects.linpro.no
Fri Sep 12 11:00:22 CEST 2008
#295: Varnishd using all available memory, even using limited malloc
-------------------------------------+--------------------------------------
Reporter: ay | Owner: des
Type: defect | Status: new
Priority: normal | Milestone: Varnish 2.0 release
Component: build | Version: trunk
Severity: major | Resolution:
Keywords: leak memory malloc swap |
-------------------------------------+--------------------------------------
Old description:
> I need some help to understand this
>
> I have a 64bit server with 32GB of RAM and 60GB SWAP
>
> And varnish is running with these parameters
>
> /usr/sbin/varnishd -a :80 -f /etc/varnish/nettby.vcl -T 127.0.0.1:82 -t
> 120 -u varnish -g varnish -p thread_pool_add_delay 100 -p
> thread_pool_timeout 600 -p client_http11 on -p lru_interval 3600 -s
> malloc,30G -P /var/run/varnish.pid
>
> malloc'ed 30G of memory.. So in theory I should not even need swap.
>
> But
>
> [root at sol ~]# free -m
> total used free shared buffers cached
> Mem: 32242 31735 507 0 0 92
> -/+ buffers/cache: 31642 600
> Swap: 59871 51322 8549
>
> The machine is using 51GB of swap and is swapping in pages frequently
>
> [root at sol ~]# vmstat 1
> procs -----------memory---------- ---swap-- -----io---- --system--
> -----cpu------
> r b swpd free buff cache si so bi bo in cs us sy
> id wa st
> 0 0 52549748 446992 780 96284 29 11 48 157 1 1 4
> 2 92 1 0
> 0 0 52549708 530360 780 96348 280 0 280 0 9654 6935 4
> 8 87 1 0
> 0 0 52549708 528160 780 96420 0 0 0 0 6785 7560 2
> 2 96 0 0
> 0 0 52549664 526668 780 96316 316 0 316 0 6570 7350 2
> 2 95 1 0
> 0 0 52549648 524932 780 96396 104 0 104 0 6300 7456 2
> 1 96 0 0
> 0 0 52549644 523372 780 96400 28 0 28 0 6013 7099 2
> 2 96 0 0
> 0 0 52549640 521884 788 96424 28 0 28 28 5649 7343 2
> 2 96 0 0
>
>
> [root at sol ~]# varnishstat -1
> client_conn 2089633996 362.23 Client connections accepted
> client_req 8345199749 1446.62 Client requests received
> cache_hit 8289156139 1436.90 Cache hits
> cache_hitpass 1610 0.00 Cache hits for pass
> cache_miss 53623537 9.30 Cache misses
> backend_conn 53621117 9.30 Backend connections success
> backend_fail 4031 0.00 Backend connections failures
> backend_reuse 0 0.00 Backend connections reuses
> backend_recycle 0 0.00 Backend connections recycles
> backend_unused 0 0.00 Backend connections unused
> n_srcaddr 7689 . N struct srcaddr
> n_srcaddr_act 4164 . N active struct srcaddr
> n_sess_mem 59601 . N struct sess_mem
> n_sess 64719 . N struct sess
> n_object 2025596 . N struct object
> n_objecthead 2025576 . N struct objecthead
> n_smf 0 . N struct smf
> n_smf_frag 0 . N small free smf
> n_smf_large 0 . N large free smf
> n_vbe_conn 106 . N struct vbe_conn
> n_bereq 96 . N struct bereq
> n_wrk 78 . N worker threads
> n_wrk_create 15968 0.00 N worker threads created
> n_wrk_failed 0 0.00 N worker threads not created
> n_wrk_max 191919689 33.27 N worker threads limited
> n_wrk_queue 0 0.00 N queued work requests
> n_wrk_overflow 210128619 36.43 N overflowed work requests
> n_wrk_drop 31202342 5.41 N dropped work requests
> n_backend 1 . N backends
> n_expired 31600718 . N expired objects
> n_lru_nuked 20008124 . N LRU nuked objects
> n_lru_saved 0 . N LRU saved objects
> n_lru_moved 779599235 . N LRU moved objects
> n_deathrow 0 . N objects on deathrow
> losthdr 196 0.00 HTTP header overflows
> n_objsendfile 0 0.00 Objects sent with sendfile
> n_objwrite 6674989721 1157.09 Objects sent with write
> n_objoverflow 0 0.00 Objects overflowing workspace
> s_sess 2073648637 359.46 Total Sessions
> s_req 8345234372 1446.62 Total Requests
> s_pipe 0 0.00 Total pipe
> s_pass 0 0.00 Total pass
> s_fetch 53620083 9.29 Total fetch
> s_hdrbytes 2924438007698 506943.34 Total header bytes
> s_bodybytes 13839780950697 2399088.22 Total body bytes
> sess_closed 54123498 9.38 Session Closed
> sess_pipeline 0 0.00 Session Pipeline
> sess_readahead 42322579 7.34 Session Read Ahead
> sess_herd 8270331676 1433.64 Session herd
> shm_records 347828089203 60295.05 SHM records
> shm_writes 14631506809 2536.33 SHM writes
> shm_flushes 908314 0.16 SHM flushes due to overflow
> shm_cont 14245861 2.47 SHM MTX contention
> sm_nreq 0 0.00 allocator requests
> sm_nobj 0 . outstanding allocations
> sm_balloc 0 . bytes allocated
> sm_bfree 0 . bytes free
> sma_nreq 127269201 22.06 SMA allocator requests
> sma_nobj 4019319 . SMA outstanding allocations
> sma_nbytes 32212251961 . SMA outstanding bytes
> sma_balloc 891934739829 . SMA bytes allocated
> sma_bfree 859722487868 . SMA bytes free
> backend_req 53621117 9.30 Backend requests made
New description:
I need some help to understand this
I have a 64bit server with 32GB of RAM and 60GB SWAP
And varnish is running with these parameters
{{{
/usr/sbin/varnishd -a :80 -f /etc/varnish/nettby.vcl -T 127.0.0.1:82 -t
120 -u varnish -g varnish -p thread_pool_add_delay 100 -p
thread_pool_timeout 600 -p client_http11 on -p lru_interval 3600 -s
malloc,30G -P /var/run/varnish.pid
malloc'ed 30G of memory.. So in theory I should not even need swap.
But
[root at sol ~]# free -m
total used free shared buffers cached
Mem: 32242 31735 507 0 0 92
-/+ buffers/cache: 31642 600
Swap: 59871 51322 8549
The machine is using 51GB of swap and is swapping in pages frequently
[root at sol ~]# vmstat 1
procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu------
r b swpd free buff cache si so bi bo in cs us sy
id wa st
0 0 52549748 446992 780 96284 29 11 48 157 1 1 4
2 92 1 0
0 0 52549708 530360 780 96348 280 0 280 0 9654 6935 4
8 87 1 0
0 0 52549708 528160 780 96420 0 0 0 0 6785 7560 2
2 96 0 0
0 0 52549664 526668 780 96316 316 0 316 0 6570 7350 2
2 95 1 0
0 0 52549648 524932 780 96396 104 0 104 0 6300 7456 2
1 96 0 0
0 0 52549644 523372 780 96400 28 0 28 0 6013 7099 2
2 96 0 0
0 0 52549640 521884 788 96424 28 0 28 28 5649 7343 2
2 96 0 0
[root at sol ~]# varnishstat -1
client_conn 2089633996 362.23 Client connections accepted
client_req 8345199749 1446.62 Client requests received
cache_hit 8289156139 1436.90 Cache hits
cache_hitpass 1610 0.00 Cache hits for pass
cache_miss 53623537 9.30 Cache misses
backend_conn 53621117 9.30 Backend connections success
backend_fail 4031 0.00 Backend connections failures
backend_reuse 0 0.00 Backend connections reuses
backend_recycle 0 0.00 Backend connections recycles
backend_unused 0 0.00 Backend connections unused
n_srcaddr 7689 . N struct srcaddr
n_srcaddr_act 4164 . N active struct srcaddr
n_sess_mem 59601 . N struct sess_mem
n_sess 64719 . N struct sess
n_object 2025596 . N struct object
n_objecthead 2025576 . N struct objecthead
n_smf 0 . N struct smf
n_smf_frag 0 . N small free smf
n_smf_large 0 . N large free smf
n_vbe_conn 106 . N struct vbe_conn
n_bereq 96 . N struct bereq
n_wrk 78 . N worker threads
n_wrk_create 15968 0.00 N worker threads created
n_wrk_failed 0 0.00 N worker threads not created
n_wrk_max 191919689 33.27 N worker threads limited
n_wrk_queue 0 0.00 N queued work requests
n_wrk_overflow 210128619 36.43 N overflowed work requests
n_wrk_drop 31202342 5.41 N dropped work requests
n_backend 1 . N backends
n_expired 31600718 . N expired objects
n_lru_nuked 20008124 . N LRU nuked objects
n_lru_saved 0 . N LRU saved objects
n_lru_moved 779599235 . N LRU moved objects
n_deathrow 0 . N objects on deathrow
losthdr 196 0.00 HTTP header overflows
n_objsendfile 0 0.00 Objects sent with sendfile
n_objwrite 6674989721 1157.09 Objects sent with write
n_objoverflow 0 0.00 Objects overflowing workspace
s_sess 2073648637 359.46 Total Sessions
s_req 8345234372 1446.62 Total Requests
s_pipe 0 0.00 Total pipe
s_pass 0 0.00 Total pass
s_fetch 53620083 9.29 Total fetch
s_hdrbytes 2924438007698 506943.34 Total header bytes
s_bodybytes 13839780950697 2399088.22 Total body bytes
sess_closed 54123498 9.38 Session Closed
sess_pipeline 0 0.00 Session Pipeline
sess_readahead 42322579 7.34 Session Read Ahead
sess_herd 8270331676 1433.64 Session herd
shm_records 347828089203 60295.05 SHM records
shm_writes 14631506809 2536.33 SHM writes
shm_flushes 908314 0.16 SHM flushes due to overflow
shm_cont 14245861 2.47 SHM MTX contention
sm_nreq 0 0.00 allocator requests
sm_nobj 0 . outstanding allocations
sm_balloc 0 . bytes allocated
sm_bfree 0 . bytes free
sma_nreq 127269201 22.06 SMA allocator requests
sma_nobj 4019319 . SMA outstanding allocations
sma_nbytes 32212251961 . SMA outstanding bytes
sma_balloc 891934739829 . SMA bytes allocated
sma_bfree 859722487868 . SMA bytes free
backend_req 53621117 9.30 Backend requests made
}}}
--
Ticket URL: <http://varnish.projects.linpro.no/ticket/295#comment:1>
Varnish <http://varnish.projects.linpro.no/>
The Varnish HTTP Accelerator
More information about the varnish-bugs
mailing list