Varnish virtual memory usage
Henry Paulissen
h.paulissen at qbell.nl
Thu Nov 5 00:48:43 CET 2009
Our load balancer transforms all connections from keep-alive to close.
So keep-alive connections aren’t the issue here.
Also, if I limit the thread count I still see the same behavior.
-----Oorspronkelijk bericht-----
Van: Ken Brownfield [mailto:kb at slide.com]
Verzonden: donderdag 5 november 2009 0:31
Aan: Henry Paulissen
CC: varnish-misc at projects.linpro.no
Onderwerp: Re: Varnish virtual memory usage
Looks like varnish is allocating ~1.5GB of RAM for pure cache (which
may roughly match your "-s file" option) but 1,610 threads with your
1MB stack limit will use 1.7GB of RAM. Pmap is reporting the
footprint of this instance as roughly 3.6GB, and I'm assuming top/ps
agree with that number.
Unless your "-s file" option is significantly less than 1-1.5GB, the
sheer thread count explains your memory usage: maybe using a stacksize
of 512K or 256K could help, and/or disable keepalives on the client
side?
Also, if you happen to be using a load balancer, TCP Buffering
(NetScaler) or Proxy Buffering? (BigIP) or the like can drastically
reduce the thread count (and they can handle the persistent keepalives
as well).
But IMHO, an event-based (for example) handler for "idle" or "slow"
threads is probably the next important feature, just below
persistence. Without something like TCP buffering, the memory
available for actual caching is dwarfed by the thread stacksize alloc
overhead.
Ken
On Nov 4, 2009, at 3:18 PM, Henry Paulissen wrote:
> I attached the memory dump.
>
> Child processes count gives me 1610 processes (on this instance).
> Currently the server isn’t so busy (~175 requests / sec).
>
> Varnishstat -1:
> =
> =
> =
> =
> =
> =
> ======================================================================
> ======
> =
> =
> =
> =
> =
> =
> ======================================================================
> ======
> uptime 3090 . Child uptime
> client_conn 435325 140.88 Client connections accepted
> client_drop 0 0.00 Connection dropped, no sess
> client_req 435294 140.87 Client requests received
> cache_hit 45740 14.80 Cache hits
> cache_hitpass 0 0.00 Cache hits for pass
> cache_miss 126445 40.92 Cache misses
> backend_conn 355277 114.98 Backend conn. success
> backend_unhealthy 0 0.00 Backend conn. not
> attempted
> backend_busy 0 0.00 Backend conn. too many
> backend_fail 0 0.00 Backend conn. failures
> backend_reuse 34331 11.11 Backend conn. reuses
> backend_toolate 690 0.22 Backend conn. was closed
> backend_recycle 35021 11.33 Backend conn. recycles
> backend_unused 0 0.00 Backend conn. unused
> fetch_head 0 0.00 Fetch head
> fetch_length 384525 124.44 Fetch with Length
> fetch_chunked 2441 0.79 Fetch chunked
> fetch_eof 0 0.00 Fetch EOF
> fetch_bad 0 0.00 Fetch had bad headers
> fetch_close 2028 0.66 Fetch wanted close
> fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed
> fetch_zero 0 0.00 Fetch zero len
> fetch_failed 0 0.00 Fetch failed
> n_sess_mem 989 . N struct sess_mem
> n_sess 94 . N struct sess
> n_object 89296 . N struct object
> n_vampireobject 0 . N unresurrected objects
> n_objectcore 89640 . N struct objectcore
> n_objecthead 25379 . N struct objecthead
> n_smf 0 . N struct smf
> n_smf_frag 0 . N small free smf
> n_smf_large 0 . N large free smf
> n_vbe_conn 26 . N struct vbe_conn
> n_wrk 1600 . N worker threads
> n_wrk_create 1600 0.52 N worker threads created
> n_wrk_failed 0 0.00 N worker threads not
> created
> n_wrk_max 1274 0.41 N worker threads limited
> n_wrk_queue 0 0.00 N queued work requests
> n_wrk_overflow 1342 0.43 N overflowed work requests
> n_wrk_drop 0 0.00 N dropped work requests
> n_backend 5 . N backends
> n_expired 1393 . N expired objects
> n_lru_nuked 35678 . N LRU nuked objects
> n_lru_saved 0 . N LRU saved objects
> n_lru_moved 20020 . N LRU moved objects
> n_deathrow 0 . N objects on deathrow
> losthdr 11 0.00 HTTP header overflows
> n_objsendfile 0 0.00 Objects sent with sendfile
> n_objwrite 433558 140.31 Objects sent with write
> n_objoverflow 0 0.00 Objects overflowing
> workspace
> s_sess 435298 140.87 Total Sessions
> s_req 435294 140.87 Total Requests
> s_pipe 0 0.00 Total pipe
> s_pass 263190 85.17 Total pass
> s_fetch 388994 125.89 Total fetch
> s_hdrbytes 157405143 50940.18 Total header bytes
> s_bodybytes 533077018 172516.83 Total body bytes
> sess_closed 435291 140.87 Session Closed
> sess_pipeline 0 0.00 Session Pipeline
> sess_readahead 0 0.00 Session Read Ahead
> sess_linger 0 0.00 Session Linger
> sess_herd 69 0.02 Session herd
> shm_records 37936743 12277.26 SHM records
> shm_writes 2141029 692.89 SHM writes
> shm_flushes 0 0.00 SHM flushes due to overflow
> shm_cont 3956 1.28 SHM MTX contention
> shm_cycles 16 0.01 SHM cycles through buffer
> sm_nreq 0 0.00 allocator requests
> sm_nobj 0 . outstanding allocations
> sm_balloc 0 . bytes allocated
> sm_bfree 0 . bytes free
> sma_nreq 550879 178.28 SMA allocator requests
> sma_nobj 178590 . SMA outstanding allocations
> sma_nbytes 1073690180 . SMA outstanding bytes
> sma_balloc 2066782844 . SMA bytes allocated
> sma_bfree 993092664 . SMA bytes free
> sms_nreq 649 0.21 SMS allocator requests
> sms_nobj 0 . SMS outstanding allocations
> sms_nbytes 0 . SMS outstanding bytes
> sms_balloc 378848 . SMS bytes allocated
> sms_bfree 378848 . SMS bytes freed
> backend_req 389342 126.00 Backend requests made
> n_vcl 1 0.00 N vcl total
> n_vcl_avail 1 0.00 N vcl available
> n_vcl_discard 0 0.00 N vcl discarded
> n_purge 1 . N total active purges
> n_purge_add 1 0.00 N new purges added
> n_purge_retire 0 0.00 N old purges deleted
> n_purge_obj_test 0 0.00 N objects tested
> n_purge_re_test 0 0.00 N regexps tested against
> n_purge_dups 0 0.00 N duplicate purges removed
> hcb_nolock 0 0.00 HCB Lookups without lock
> hcb_lock 0 0.00 HCB Lookups with lock
> hcb_insert 0 0.00 HCB Inserts
> esi_parse 0 0.00 Objects ESI parsed (unlock)
> esi_errors 0 0.00 ESI parse errors (unlock)
> =
> =
> =
> =
> =
> =
> ======================================================================
> ======
> =
> =
> =
> =
> =
> =
> ======================================================================
> ======
>
>
>
> -----Oorspronkelijk bericht-----
> Van: Ken Brownfield [mailto:kb at slide.com]
> Verzonden: donderdag 5 november 2009 0:01
> Aan: Henry Paulissen
> CC: Rogério Schneider
> Onderwerp: Re: Varnish virtual memory usage
>
> Curious: For a heavily leaked varnish instance, can you run "pmap -x
> PID" on the parent PID and child PID, and record how many threads are
> active (something like 'ps -efT | grep varnish | wc -l')? Might help
> isolate the RAM usage.
>
> Sorry if you have done this already; didn't find it in my email
> archive.
>
> Ken
>
> On Nov 4, 2009, at 2:53 PM, Henry Paulissen wrote:
>
>> No, varnishd still usages way more than allowed.
>> The only solutions I found at the moment are:
>>
>> Run on x64 linux and restart varnish every 4 hours (crontab).
>> Run on x32 linux (all is working as expected but you cant allocate
>> more as
>> 4G each instance).
>>
>>
>> I hope linpro will find this issue and address it.
>>
>>
>>
>> Again @ linpro: if you need a machine (with live traffic) to run
>> some tests,
>> please contact me.
>> We have multiple machines in high availability, so testing and
>> rebooting a
>> instance wouldn’t hurt us.
>>
>>
>> Regards.
>>
>> -----Oorspronkelijk bericht-----
>> Van: Rogério Schneider [mailto:stockrt at gmail.com]
>> Verzonden: woensdag 4 november 2009 22:04
>> Aan: Henry Paulissen
>> CC: Scott Wilson; varnish-misc at projects.linpro.no
>> Onderwerp: Re: Varnish virtual memory usage
>>
>> On Thu, Oct 22, 2009 at 6:04 AM, Henry Paulissen
>> <h.paulissen at qbell.nl>
>> wrote:
>>> I will report back.
>>
>> Did this solve the problem?
>>
>> Removing this?
>>
>>>> if (req.http.Cache-Control == "no-cache" || req.http.Pragma ==
>> "no-cache") {
>>>> purge_url(req.url);
>>>> }
>>>>
>>
>> Cheers
>>
>> Att,
>> --
>> Rogério Schneider
>>
>> MSN: stockrt at hotmail.com
>> GTalk: stockrt at gmail.com
>> Skype: stockrt
>> http://stockrt.github.com
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at projects.linpro.no
>> http://projects.linpro.no/mailman/listinfo/varnish-misc
> <pmap.txt>
More information about the varnish-misc
mailing list