Varnish virtual memory usage
Ken Brownfield
kb+varnish at slide.com
Thu Nov 5 02:16:05 CET 2009
Ah, sorry, missed that the command-line was in there.
Given 1G of cache, a large sess_workspace and shm_workspace buffers,
and the number of threads, the math adds up correctly. Do you
definitely need those large buffers?
Your memory footprint will simply increase with thread count; reducing
active simultaneous connections, reducing the stack size, and reducing
the large sess_workspace are the only ways I know of for you to
control the memory. I'm really not seeing a leak or malfunction,
IMHO. The reason behind your high/growing worker count is worth
investigating (lower send_timeout? slow/disconnecting clients? strace
the threads to see what they're doing?)
Minor thing: overflow_max is a percentage, so 10000 is probably ignored?
--
Ken
On Nov 4, 2009, at 4:47 PM, Henry Paulissen wrote:
> See the pmap.txt attachment.
> The startup command is in the beginning of the file.
>
>
> /usr/local/varnish/sbin/varnishd -P /var/run/xxx.pid -a 0.0.0.0:xxx -f
> /usr/local/varnish/etc/varnish/xxx.xxx.xxx.vcl -T 0.0.0.0:xxx -s
> malloc,1G
> -i xxx -n /usr/local/varnish/var/varnish/xxx -p obj_workspace 8192 -p
> sess_workspace 262144 -p listen_depth 8192 -p lru_interval 60 -p
> sess_timeout 10 -p shm_workspace 32768 -p ping_interval 2 -p
> thread_pools 4
> -p thread_pool_min 50 -p thread_pool_max 4000 -p esi_syntax 1 -p
> overflow_max 10000
>
>
> -----Oorspronkelijk bericht-----
> Van: Ken Brownfield [mailto:kb+varnish at slide.com]
> Verzonden: donderdag 5 november 2009 1:42
> Aan: Henry Paulissen
> Onderwerp: Re: Varnish virtual memory usage
>
> Is your -s set at 1.5GB? What's your varnishd command line?
>
> I'm not sure if you realize that thread_pool does not control the
> number of threads, only the number of pools (and mutexes). I think
> thread_pool_max is what you're looking for?
> --
> Ken
>
> On Nov 4, 2009, at 4:37 PM, Henry Paulissen wrote:
>
>> Running varnishd now for abount 30 minutes with a thread_pool of 4.
>>
>> =
>> =
>> =
>> =
>> =
>> =
>> =
>> =====================================================================
>> ========================
>> =
>> =
>> =
>> =
>> =
>> =
>> =
>> =====================================================================
>> ========================
>> uptime 2637 . Child uptime
>> client_conn 316759 120.12 Client connections
>> accepted
>> client_drop 0 0.00 Connection dropped, no
>> sess
>> client_req 316738 120.11 Client requests received
>> cache_hit 32477 12.32 Cache hits
>> cache_hitpass 0 0.00 Cache hits for pass
>> cache_miss 93703 35.53 Cache misses
>> backend_conn 261033 98.99 Backend conn. success
>> backend_unhealthy 0 0.00 Backend conn. not
>> attempted
>> backend_busy 0 0.00 Backend conn. too many
>> backend_fail 0 0.00 Backend conn. failures
>> backend_reuse 23305 8.84 Backend conn. reuses
>> backend_toolate 528 0.20 Backend conn. was closed
>> backend_recycle 23833 9.04 Backend conn. recycles
>> backend_unused 0 0.00 Backend conn. unused
>> fetch_head 0 0.00 Fetch head
>> fetch_length 280973 106.55 Fetch with Length
>> fetch_chunked 1801 0.68 Fetch chunked
>> fetch_eof 0 0.00 Fetch EOF
>> fetch_bad 0 0.00 Fetch had bad headers
>> fetch_close 1329 0.50 Fetch wanted close
>> fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed
>> fetch_zero 0 0.00 Fetch zero len
>> fetch_failed 0 0.00 Fetch failed
>> n_sess_mem 284 . N struct sess_mem
>> n_sess 35 . N struct sess
>> n_object 90560 . N struct object
>> n_vampireobject 0 . N unresurrected objects
>> n_objectcore 90616 . N struct objectcore
>> n_objecthead 25146 . N struct objecthead
>> n_smf 0 . N struct smf
>> n_smf_frag 0 . N small free smf
>> n_smf_large 0 . N large free smf
>> n_vbe_conn 10 . N struct vbe_conn
>> n_wrk 200 . N worker threads
>> n_wrk_create 248 0.09 N worker threads created
>> n_wrk_failed 0 0.00 N worker threads not
>> created
>> n_wrk_max 100988 38.30 N worker threads limited
>> n_wrk_queue 0 0.00 N queued work requests
>> n_wrk_overflow 630 0.24 N overflowed work requests
>> n_wrk_drop 0 0.00 N dropped work requests
>> n_backend 5 . N backends
>> n_expired 1027 . N expired objects
>> n_lru_nuked 2108 . N LRU nuked objects
>> n_lru_saved 0 . N LRU saved objects
>> n_lru_moved 12558 . N LRU moved objects
>> n_deathrow 0 . N objects on deathrow
>> losthdr 5 0.00 HTTP header overflows
>> n_objsendfile 0 0.00 Objects sent with sendfile
>> n_objwrite 315222 119.54 Objects sent with write
>> n_objoverflow 0 0.00 Objects overflowing
>> workspace
>> s_sess 316740 120.11 Total Sessions
>> s_req 316738 120.11 Total Requests
>> s_pipe 0 0.00 Total pipe
>> s_pass 190664 72.30 Total pass
>> s_fetch 284103 107.74 Total fetch
>> s_hdrbytes 114236150 43320.50 Total header bytes
>> s_bodybytes 355198316 134697.88 Total body bytes
>> sess_closed 316740 120.11 Session Closed
>> sess_pipeline 0 0.00 Session Pipeline
>> sess_readahead 0 0.00 Session Read Ahead
>> sess_linger 0 0.00 Session Linger
>> sess_herd 33 0.01 Session herd
>> shm_records 27534992 10441.79 SHM records
>> shm_writes 1555265 589.79 SHM writes
>> shm_flushes 0 0.00 SHM flushes due to
>> overflow
>> shm_cont 1689 0.64 SHM MTX contention
>> shm_cycles 12 0.00 SHM cycles through buffer
>> sm_nreq 0 0.00 allocator requests
>> sm_nobj 0 . outstanding allocations
>> sm_balloc 0 . bytes allocated
>> sm_bfree 0 . bytes free
>> sma_nreq 379783 144.02 SMA allocator requests
>> sma_nobj 181121 . SMA outstanding
>> allocations
>> sma_nbytes 1073735584 . SMA outstanding bytes
>> sma_balloc 1488895305 . SMA bytes allocated
>> sma_bfree 415159721 . SMA bytes free
>> sms_nreq 268 0.10 SMS allocator requests
>> sms_nobj 0 . SMS outstanding
>> allocations
>> sms_nbytes 0 . SMS outstanding bytes
>> sms_balloc 156684 . SMS bytes allocated
>> sms_bfree 156684 . SMS bytes freed
>> backend_req 284202 107.77 Backend requests made
>> n_vcl 1 0.00 N vcl total
>> n_vcl_avail 1 0.00 N vcl available
>> n_vcl_discard 0 0.00 N vcl discarded
>> n_purge 1 . N total active purges
>> n_purge_add 1 0.00 N new purges added
>> n_purge_retire 0 0.00 N old purges deleted
>> n_purge_obj_test 0 0.00 N objects tested
>> n_purge_re_test 0 0.00 N regexps tested against
>> n_purge_dups 0 0.00 N duplicate purges removed
>> hcb_nolock 0 0.00 HCB Lookups without lock
>> hcb_lock 0 0.00 HCB Lookups with lock
>> hcb_insert 0 0.00 HCB Inserts
>> esi_parse 0 0.00 Objects ESI parsed
>> (unlock)
>> esi_errors 0 0.00 ESI parse errors (unlock)
>> =
>> =
>> =
>> =
>> =
>> =
>> =
>> =====================================================================
>> ========================
>> =
>> =
>> =
>> =
>> =
>> =
>> =
>> =====================================================================
>> ========================
>>
>> As you can see I have now 200 worker threads.
>> Still its using 1.8G and is still increasing (~1 to 5 mb/s)
>>
>>
>> -----Oorspronkelijk bericht-----
>> Van: Ken Brownfield [mailto:kb+varnish at slide.com]
>> Verzonden: donderdag 5 november 2009 1:18
>> Aan: Henry Paulissen
>> CC: varnish-misc at projects.linpro.no
>> Onderwerp: Re: Varnish virtual memory usage
>>
>> Hmm, well the memory adds up to a 1.5G -s option (can you confirm
>> what
>> you use with -s?) and memory required to run the number of threads
>> you're running. Unless your -s is drastically smaller than 1.5GB,
>> the
>> pmap you sent is of a normal, non-leaking process.
>>
>> Ken
>>
>> On Nov 4, 2009, at 3:48 PM, Henry Paulissen wrote:
>>
>>> Our load balancer transforms all connections from keep-alive to
>>> close.
>>> So keep-alive connections aren’t the issue here.
>>>
>>> Also, if I limit the thread count I still see the same behavior.
>>>
>>> -----Oorspronkelijk bericht-----
>>> Van: Ken Brownfield [mailto:kb at slide.com]
>>> Verzonden: donderdag 5 november 2009 0:31
>>> Aan: Henry Paulissen
>>> CC: varnish-misc at projects.linpro.no
>>> Onderwerp: Re: Varnish virtual memory usage
>>>
>>> Looks like varnish is allocating ~1.5GB of RAM for pure cache (which
>>> may roughly match your "-s file" option) but 1,610 threads with your
>>> 1MB stack limit will use 1.7GB of RAM. Pmap is reporting the
>>> footprint of this instance as roughly 3.6GB, and I'm assuming top/ps
>>> agree with that number.
>>>
>>> Unless your "-s file" option is significantly less than 1-1.5GB, the
>>> sheer thread count explains your memory usage: maybe using a
>>> stacksize
>>> of 512K or 256K could help, and/or disable keepalives on the client
>>> side?
>>>
>>> Also, if you happen to be using a load balancer, TCP Buffering
>>> (NetScaler) or Proxy Buffering? (BigIP) or the like can drastically
>>> reduce the thread count (and they can handle the persistent
>>> keepalives
>>> as well).
>>>
>>> But IMHO, an event-based (for example) handler for "idle" or "slow"
>>> threads is probably the next important feature, just below
>>> persistence. Without something like TCP buffering, the memory
>>> available for actual caching is dwarfed by the thread stacksize
>>> alloc
>>> overhead.
>>>
>>> Ken
>>>
>>> On Nov 4, 2009, at 3:18 PM, Henry Paulissen wrote:
>>>
>>>> I attached the memory dump.
>>>>
>>>> Child processes count gives me 1610 processes (on this instance).
>>>> Currently the server isn’t so busy (~175 requests / sec).
>>>>
>>>> Varnishstat -1:
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> ===================================================================
>>>> ======
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> ===================================================================
>>>> ======
>>>> uptime 3090 . Child uptime
>>>> client_conn 435325 140.88 Client connections
>>>> accepted
>>>> client_drop 0 0.00 Connection dropped, no
>>>> sess
>>>> client_req 435294 140.87 Client requests received
>>>> cache_hit 45740 14.80 Cache hits
>>>> cache_hitpass 0 0.00 Cache hits for pass
>>>> cache_miss 126445 40.92 Cache misses
>>>> backend_conn 355277 114.98 Backend conn. success
>>>> backend_unhealthy 0 0.00 Backend conn. not
>>>> attempted
>>>> backend_busy 0 0.00 Backend conn. too many
>>>> backend_fail 0 0.00 Backend conn. failures
>>>> backend_reuse 34331 11.11 Backend conn. reuses
>>>> backend_toolate 690 0.22 Backend conn. was closed
>>>> backend_recycle 35021 11.33 Backend conn. recycles
>>>> backend_unused 0 0.00 Backend conn. unused
>>>> fetch_head 0 0.00 Fetch head
>>>> fetch_length 384525 124.44 Fetch with Length
>>>> fetch_chunked 2441 0.79 Fetch chunked
>>>> fetch_eof 0 0.00 Fetch EOF
>>>> fetch_bad 0 0.00 Fetch had bad headers
>>>> fetch_close 2028 0.66 Fetch wanted close
>>>> fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1
>>>> closed
>>>> fetch_zero 0 0.00 Fetch zero len
>>>> fetch_failed 0 0.00 Fetch failed
>>>> n_sess_mem 989 . N struct sess_mem
>>>> n_sess 94 . N struct sess
>>>> n_object 89296 . N struct object
>>>> n_vampireobject 0 . N unresurrected objects
>>>> n_objectcore 89640 . N struct objectcore
>>>> n_objecthead 25379 . N struct objecthead
>>>> n_smf 0 . N struct smf
>>>> n_smf_frag 0 . N small free smf
>>>> n_smf_large 0 . N large free smf
>>>> n_vbe_conn 26 . N struct vbe_conn
>>>> n_wrk 1600 . N worker threads
>>>> n_wrk_create 1600 0.52 N worker threads created
>>>> n_wrk_failed 0 0.00 N worker threads not
>>>> created
>>>> n_wrk_max 1274 0.41 N worker threads limited
>>>> n_wrk_queue 0 0.00 N queued work requests
>>>> n_wrk_overflow 1342 0.43 N overflowed work
>>>> requests
>>>> n_wrk_drop 0 0.00 N dropped work requests
>>>> n_backend 5 . N backends
>>>> n_expired 1393 . N expired objects
>>>> n_lru_nuked 35678 . N LRU nuked objects
>>>> n_lru_saved 0 . N LRU saved objects
>>>> n_lru_moved 20020 . N LRU moved objects
>>>> n_deathrow 0 . N objects on deathrow
>>>> losthdr 11 0.00 HTTP header overflows
>>>> n_objsendfile 0 0.00 Objects sent with
>>>> sendfile
>>>> n_objwrite 433558 140.31 Objects sent with write
>>>> n_objoverflow 0 0.00 Objects overflowing
>>>> workspace
>>>> s_sess 435298 140.87 Total Sessions
>>>> s_req 435294 140.87 Total Requests
>>>> s_pipe 0 0.00 Total pipe
>>>> s_pass 263190 85.17 Total pass
>>>> s_fetch 388994 125.89 Total fetch
>>>> s_hdrbytes 157405143 50940.18 Total header bytes
>>>> s_bodybytes 533077018 172516.83 Total body bytes
>>>> sess_closed 435291 140.87 Session Closed
>>>> sess_pipeline 0 0.00 Session Pipeline
>>>> sess_readahead 0 0.00 Session Read Ahead
>>>> sess_linger 0 0.00 Session Linger
>>>> sess_herd 69 0.02 Session herd
>>>> shm_records 37936743 12277.26 SHM records
>>>> shm_writes 2141029 692.89 SHM writes
>>>> shm_flushes 0 0.00 SHM flushes due to
>>>> overflow
>>>> shm_cont 3956 1.28 SHM MTX contention
>>>> shm_cycles 16 0.01 SHM cycles through
>>>> buffer
>>>> sm_nreq 0 0.00 allocator requests
>>>> sm_nobj 0 . outstanding allocations
>>>> sm_balloc 0 . bytes allocated
>>>> sm_bfree 0 . bytes free
>>>> sma_nreq 550879 178.28 SMA allocator requests
>>>> sma_nobj 178590 . SMA outstanding
>>>> allocations
>>>> sma_nbytes 1073690180 . SMA outstanding bytes
>>>> sma_balloc 2066782844 . SMA bytes allocated
>>>> sma_bfree 993092664 . SMA bytes free
>>>> sms_nreq 649 0.21 SMS allocator requests
>>>> sms_nobj 0 . SMS outstanding
>>>> allocations
>>>> sms_nbytes 0 . SMS outstanding bytes
>>>> sms_balloc 378848 . SMS bytes allocated
>>>> sms_bfree 378848 . SMS bytes freed
>>>> backend_req 389342 126.00 Backend requests made
>>>> n_vcl 1 0.00 N vcl total
>>>> n_vcl_avail 1 0.00 N vcl available
>>>> n_vcl_discard 0 0.00 N vcl discarded
>>>> n_purge 1 . N total active purges
>>>> n_purge_add 1 0.00 N new purges added
>>>> n_purge_retire 0 0.00 N old purges deleted
>>>> n_purge_obj_test 0 0.00 N objects tested
>>>> n_purge_re_test 0 0.00 N regexps tested against
>>>> n_purge_dups 0 0.00 N duplicate purges
>>>> removed
>>>> hcb_nolock 0 0.00 HCB Lookups without lock
>>>> hcb_lock 0 0.00 HCB Lookups with lock
>>>> hcb_insert 0 0.00 HCB Inserts
>>>> esi_parse 0 0.00 Objects ESI parsed
>>>> (unlock)
>>>> esi_errors 0 0.00 ESI parse errors
>>>> (unlock)
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> ===================================================================
>>>> ======
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> =
>>>> ===================================================================
>>>> ======
>>>>
>>>>
>>>>
>>>> -----Oorspronkelijk bericht-----
>>>> Van: Ken Brownfield [mailto:kb at slide.com]
>>>> Verzonden: donderdag 5 november 2009 0:01
>>>> Aan: Henry Paulissen
>>>> CC: Rogério Schneider
>>>> Onderwerp: Re: Varnish virtual memory usage
>>>>
>>>> Curious: For a heavily leaked varnish instance, can you run "pmap
>>>> -x
>>>> PID" on the parent PID and child PID, and record how many threads
>>>> are
>>>> active (something like 'ps -efT | grep varnish | wc -l')? Might
>>>> help
>>>> isolate the RAM usage.
>>>>
>>>> Sorry if you have done this already; didn't find it in my email
>>>> archive.
>>>>
>>>> Ken
>>>>
>>>> On Nov 4, 2009, at 2:53 PM, Henry Paulissen wrote:
>>>>
>>>>> No, varnishd still usages way more than allowed.
>>>>> The only solutions I found at the moment are:
>>>>>
>>>>> Run on x64 linux and restart varnish every 4 hours (crontab).
>>>>> Run on x32 linux (all is working as expected but you cant allocate
>>>>> more as
>>>>> 4G each instance).
>>>>>
>>>>>
>>>>> I hope linpro will find this issue and address it.
>>>>>
>>>>>
>>>>>
>>>>> Again @ linpro: if you need a machine (with live traffic) to run
>>>>> some tests,
>>>>> please contact me.
>>>>> We have multiple machines in high availability, so testing and
>>>>> rebooting a
>>>>> instance wouldn’t hurt us.
>>>>>
>>>>>
>>>>> Regards.
>>>>>
>>>>> -----Oorspronkelijk bericht-----
>>>>> Van: Rogério Schneider [mailto:stockrt at gmail.com]
>>>>> Verzonden: woensdag 4 november 2009 22:04
>>>>> Aan: Henry Paulissen
>>>>> CC: Scott Wilson; varnish-misc at projects.linpro.no
>>>>> Onderwerp: Re: Varnish virtual memory usage
>>>>>
>>>>> On Thu, Oct 22, 2009 at 6:04 AM, Henry Paulissen
>>>>> <h.paulissen at qbell.nl>
>>>>> wrote:
>>>>>> I will report back.
>>>>>
>>>>> Did this solve the problem?
>>>>>
>>>>> Removing this?
>>>>>
>>>>>>> if (req.http.Cache-Control == "no-cache" || req.http.Pragma ==
>>>>> "no-cache") {
>>>>>>> purge_url(req.url);
>>>>>>> }
>>>>>>>
>>>>>
>>>>> Cheers
>>>>>
>>>>> Att,
>>>>> --
>>>>> Rogério Schneider
>>>>>
>>>>> MSN: stockrt at hotmail.com
>>>>> GTalk: stockrt at gmail.com
>>>>> Skype: stockrt
>>>>> http://stockrt.github.com
>>>>>
>>>>> _______________________________________________
>>>>> varnish-misc mailing list
>>>>> varnish-misc at projects.linpro.no
>>>>> http://projects.linpro.no/mailman/listinfo/varnish-misc
>>>> <pmap.txt>
>>>
>> <pmap.txt>
>
More information about the varnish-misc
mailing list