<div class="gmail_quote">On Thu, Sep 15, 2011 at 11:29, Aurélien Lemaire <span dir="ltr"><<a href="mailto:aurelien.lemaire@smile.fr">aurelien.lemaire@smile.fr</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
<u></u>
<div bgcolor="#ffffff" text="#000000">
Good day folks,<br>
<br>
First of all, varnish is an outstanding piece of software that my
company and i are addicted to. So thanks to all the coders.<br>
<br>
Here is my problem :<br>
I allocated varnish 1G of RAM on a website that can have more than 2
Go of possible cacheable objects . Not to worry though as any
proxy-cache system should smartly nuke old objects to make place to
new one to live peacefully within its allocated RAM. And that's
where Varnish behave unexpectedly : each time it need to nuke SOME
objects : it nukes ALMOST ALL of them (often ~80% of my 35k objects)
which is quite aggressive ; thus i lost almost all my cache....IRK !<br>
<br>
3 Munin graphs attached to see the problem clearly : big drop each
time a nuking happens.<br>
<br>
To make sure my pbr is about varnish nuking system : i increased
from 1G to 3G(more than the max possible 2G cacheable objects) on
another varnish of this platefom (this website is delivered by
multiple front/varnish server all stricly similar and independant)
and this issue disappeared (no more nuking : no lost of ~80%of my
objects)<br>
<br>
Here is my env : <br>
Debian 5.0.8 64 bits on 2.6.32-5-openvz-amd64 kernel<br>
Varnish 2.1.3 SVN 5049:5055(debian package 2.1.3-8) <br>
200 varnish 's worker threads running constantly (no issue on
workers)<br>
30req/s average with 60/s in peak<br>
<br>
Daemon run as such :<br>
/usr/sbin/varnishd -P /var/run/varnishd.pid -a :80 -T localhost:6082
-S /etc/varnish/secret -f /etc/varnish/serverx.vcl -w 100,1024 -s
file,/var/lib/varnish/serverx/varnish_storage.bin,3G<br>
<br>
Here a quick varnishstat -1 :<br>
<blockquote><tt>client_conn 17177518 9.26 Client
connections accepted</tt><br>
<tt>client_drop 0 0.00 Connection dropped,
no sess/wrk</tt><br>
<tt>client_req 31277524 16.86 Client requests
received</tt><br>
<tt>cache_hit 20909485 11.27 Cache hits</tt><br>
<tt>cache_hitpass 2558712 1.38 Cache hits for pass</tt><br>
<tt>cache_miss 5422557 2.92 Cache misses</tt><br>
<tt>backend_conn 175175 0.09 Backend conn.
success</tt><br>
<tt>backend_unhealthy 0 0.00 Backend conn. not
attempted</tt><br>
<tt>backend_busy 0 0.00 Backend conn. too
many</tt><br>
<tt>backend_fail 0 0.00 Backend conn.
failures</tt><br>
<tt>backend_reuse 10192538 5.49 Backend conn.
reuses</tt><br>
<tt>backend_toolate 98629 0.05 Backend conn. was
closed</tt><br>
<tt>backend_recycle 10291261 5.55 Backend conn.
recycles</tt><br>
<tt>backend_unused 0 0.00 Backend conn.
unused</tt><br>
<tt>fetch_head 459553 0.25 Fetch head</tt><br>
<tt>fetch_length 9907522 5.34 Fetch with Length</tt><br>
<tt>fetch_chunked 33 0.00 Fetch chunked</tt><br>
<tt>fetch_eof 0 0.00 Fetch EOF</tt><br>
<tt>fetch_bad 0 0.00 Fetch had bad
headers</tt><br>
<tt>fetch_close 7 0.00 Fetch wanted close</tt><br>
<tt>fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1
closed</tt><br>
<tt>fetch_zero 638 0.00 Fetch zero len</tt><br>
<tt>fetch_failed 0 0.00 Fetch failed</tt><br>
<tt>n_sess_mem 1189 . N struct sess_mem</tt><br>
<tt>n_sess 890 . N struct sess</tt><br>
<tt>n_object 32678 . N struct object</tt><br>
<tt>n_vampireobject 0 . N unresurrected
objects</tt><br>
<tt>n_objectcore 32794 . N struct objectcore</tt><br>
<tt>n_objecthead 30698 . N struct objecthead</tt><br>
<tt>n_smf 68810 . N struct smf</tt><br>
<tt>n_smf_frag 3002 . N small free smf</tt><br>
<tt>n_smf_large 490 . N large free smf</tt><br>
<tt>n_vbe_conn 8 . N struct vbe_conn</tt><br>
<tt>n_wrk 200 . N worker threads</tt><br>
<tt>n_wrk_create 200 0.00 N worker threads
created</tt><br>
<tt>n_wrk_failed 0 0.00 N worker threads
not created</tt><br>
<tt>n_wrk_max 4133 0.00 N worker threads
limited</tt><br>
<tt>n_wrk_queue 0 0.00 N queued work
requests</tt><br>
<tt>n_wrk_overflow 5 0.00 N overflowed work
requests</tt><br>
<tt>n_wrk_drop 0 0.00 N dropped work
requests</tt><br>
<tt>n_backend 1 . N backends</tt><br>
<tt>n_expired 2291302 . N expired objects</tt><br>
<b><tt>n_lru_nuked 3098052 . N LRU nuked
objects</tt></b><br>
<tt>n_lru_saved 0 . N LRU saved objects</tt><br>
<tt>n_lru_moved 19894617 . N LRU moved objects</tt><br>
<tt>n_deathrow 0 . N objects on
deathrow</tt><br>
<tt>losthdr 0 0.00 HTTP header
overflows</tt><br>
<tt>n_objsendfile 0 0.00 Objects sent with
sendfile</tt><br>
<tt>n_objwrite 23874235 12.87 Objects sent with
write</tt><br>
<tt>n_objoverflow 0 0.00 Objects overflowing
workspace</tt><br>
<tt>s_sess 17177512 9.26 Total Sessions</tt><br>
<tt>s_req 31277524 16.86 Total Requests</tt><br>
<tt>s_pipe 0 0.00 Total pipe</tt><br>
<tt>s_pass 4945241 2.67 Total pass</tt><br>
<tt>s_fetch 10367753 5.59 Total fetch</tt><br>
<tt>s_hdrbytes 10732679592 5784.04 Total header bytes</tt><br>
<tt>s_bodybytes 316828226025 170744.51 Total body bytes</tt><br>
<tt>sess_closed 7071576 3.81 Session Closed</tt><br>
<tt>sess_pipeline 13905 0.01 Session Pipeline</tt><br>
<tt>sess_readahead 8085 0.00 Session Read Ahead</tt><br>
<tt>sess_linger 24485521 13.20 Session Linger</tt><br>
<tt>sess_herd 23385907 12.60 Session herd</tt><br>
<tt>shm_records 1766166234 951.82 SHM records</tt><br>
<tt>shm_writes 128900505 69.47 SHM writes</tt><br>
<tt>shm_flushes 13215 0.01 SHM flushes due to
overflow</tt><br>
<tt>shm_cont 78095 0.04 SHM MTX contention</tt><br>
<tt>shm_cycles 769 0.00 SHM cycles through
buffer</tt><br>
<tt>sm_nreq 18413160 9.92 allocator requests</tt><br>
<tt>sm_nobj 65318 . outstanding
allocations</tt><br>
<tt>sm_balloc 824430592 . bytes allocated</tt><br>
<tt>sm_bfree 249311232 . bytes free</tt><br>
<tt>sma_nreq 0 0.00 SMA allocator
requests</tt><br>
<tt>sma_nobj 0 . SMA outstanding
allocations</tt><br>
<tt>sma_nbytes 0 . SMA outstanding
bytes</tt><br>
<tt>sma_balloc 0 . SMA bytes allocated</tt><br>
<tt>sma_bfree 0 . SMA bytes free</tt><br>
<tt>sms_nreq 61 0.00 SMS allocator
requests</tt><br>
<tt>sms_nobj 0 . SMS outstanding
allocations</tt><br>
<tt>sms_nbytes 0 . SMS outstanding
bytes</tt><br>
<tt>sms_balloc 16409 . SMS bytes allocated</tt><br>
<tt>sms_bfree 16409 . SMS bytes freed</tt><br>
<tt>backend_req 10367752 5.59 Backend requests
made</tt><br>
<tt>n_vcl 1 0.00 N vcl total</tt><br>
<tt>n_vcl_avail 1 0.00 N vcl available</tt><br>
<tt>n_vcl_discard 0 0.00 N vcl discarded</tt><br>
<tt>n_purge 1 . N total active
purges</tt><br>
<tt>n_purge_add 12314 0.01 N new purges added</tt><br>
<tt>n_purge_retire 12313 0.01 N old purges
deleted</tt><br>
<tt>n_purge_obj_test 2448163 1.32 N objects tested</tt><br>
<tt>n_purge_re_test 62064275 33.45 N regexps tested
against</tt><br>
<tt>n_purge_dups 9524 0.01 N duplicate purges
removed</tt><br>
<tt>hcb_nolock 28886624 15.57 HCB Lookups without
lock</tt><br>
<tt>hcb_lock 4243837 2.29 HCB Lookups with
lock</tt><br>
<tt>hcb_insert 4243834 2.29 HCB Inserts</tt><br>
<tt>esi_parse 0 0.00 Objects ESI parsed
(unlock)</tt><br>
<tt>esi_errors 0 0.00 ESI parse errors
(unlock)</tt><br>
<tt>accept_fail 0 0.00 Accept failures</tt><br>
<tt>client_drop_late 0 0.00 Connection dropped
late</tt><br>
<tt>uptime 1855569 1.00 Client uptime</tt><br>
</blockquote>
<br>
Is it normal varnish behaviour ? sounds like a bug to me.<br>
Am i missing some tuning (lru_interval) to soften the nuking algo ?<br>
Do you need more info ?<br>
helps appreciated here ;-)<br>
<br>
Regards, Aurelien Lemaire<br>
</div>
<br>_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br></blockquote></div><br>It could be someone downloading a large file (like a ~700Mb iso file)
and varnish nuking objects to make room for this file (even if its
configured to not cache it).<br><br>Try to get a varnishlog trace of the moment the nuking begins.<br clear="all"><br>-- <br>Jorge Nerín<br><<a href="mailto:jnerin@gmail.com">jnerin@gmail.com</a>><br>