<div dir="ltr">In case someone runs into a similar issue regarding BANs accumulating, I found an experimental `ban_cutoff` runtime parameter <span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">(see below)</span> that specifically exists for this issue.<div><br></div>A search for `ban_cutoff` in this mailing list archive also yields a good thread about this (thread: "Varnish Lurker is getting slower / Ban lists keeps increasing")<div><br></div><div><div><div><div>ban_cutoff</div><div> Value is: 0 [bans]</div><div> Default is: 0</div><div> Minimum is: 0</div><div><br></div><div> Expurge long tail content from the cache to keep the number of</div><div> bans below this value. 0 disables.</div><div><br></div><div> When this parameter is set to a non-zero value, the ban lurker</div><div> continues to work the ban list as usual top to bottom, but when</div><div> it reaches the ban_cutoff-th ban, it treats all objects as if</div><div> they matched a ban and expurges them from cache. As actively</div><div> used objects get tested against the ban list at request time</div><div> and thus are likely to be associated with bans near the top of</div><div> the ban list, with ban_cutoff, least recently accessed objects</div><div> (the "long tail") are removed.</div><div><br></div><div> This parameter is a safety net to avoid bad response times due</div><div> to bans being tested at lookup time. Setting a cutoff trades</div><div> response time for cache efficiency. The recommended value is</div><div> proportional to rate(bans_lurker_tests_tested) / n_objects</div><div> while the ban lurker is working, which is the number of bans</div><div> the system can sustain. The additional latency due to request</div><div> ban testing is in the order of ban_cutoff /</div><div> rate(bans_lurker_tests_tested). For example, for</div><div> rate(bans_lurker_tests_tested) = 2M/s and a tolerable latency</div><div> of 100ms, a good value for ban_cutoff may be 200K.</div><div><br></div><div> NB: We do not know yet if it is a good idea to change this</div><div> parameter, or if the default value is even sensible. Caution is</div><div> advised, and feedback is most welcome.</div><div><br></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, May 31, 2018 at 3:49 PM Hugues Alary <<a href="mailto:hugues@betabrand.com" target="_blank">hugues@betabrand.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi there, <div><br></div><div>Re-opening this thread from 2016!</div><div><br></div><div>A few days ago, I ran into an issue where varnish (varnish-6.0.0 revision a068361dff0d25a0d85cf82a6e5fdaf315e06a7d) became completely unresponsive.</div><div><br></div><div>While investigating, I noticed a lot of errors in varnishlog `VSL: store overflow`, but most importantly, I noticed our BAN list was starting to accumulate with somewhere around 130K BANs reported by varnishstat.</div><div><br></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">The load on the website was pretty low. Restarting Varnish fixed the issue. </span>It is possible I had been running some script generating a lot of BANs, but it still seems weird it would generate so many BANs that varnish would become unresponsive.</div><div><br></div><div>After restarting Varnish, I watched `varnishstat -1 | grep bans` and could see MAIN.bans going up and down, but staying fairly stable on average.</div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Today, I decided to check again and I noticed:</span></div><div><br></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans 66863 . Count of bans</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_completed 66834 . Number of bans marked 'completed'</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_obj 66849 . Number of bans using obj.*</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_req 14 . Number of bans using req.*</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_added 170939 1.18 Bans added</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_deleted 104076 0.72 Bans deleted</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_tested 3767852 26.08 Bans tested against objects (lookup)</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_obj_killed 62024 0.43 Objects killed by bans (lookup)</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_lurker_tested 3344073359 23148.62 Bans tested against objects (lurker)</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_tests_tested 94711644 655.62 Ban tests tested against objects (lookup)</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_lurker_tests_tested 4573910595 31661.91 Ban tests tested against objects (lurker)</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_lurker_obj_killed 16627 0.12 Objects killed by bans (lurker)</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_lurker_obj_killed_cutoff 0 0.00 Objects killed by bans for cutoff (lurker)</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_dups 113845 0.79 Bans superseded by other bans</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_lurker_contention 3 0.00 Lurker gave way for lookup</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_persisted_bytes 163879330 . Bytes used by the persisted ban lists</font></div><div style="text-decoration-style:initial;text-decoration-color:initial"><font face="monospace, monospace">MAIN.bans_persisted_fragmentation 162795194 . Extra bytes in persisted ban lists due to fragmentation</font></div><br></span></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><br></span></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">1- MAIN.bans is almost at 70k and varnish has been running for 1 day 16 hours</span></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">2- MAIN.bans_req is > 0, but nowhere in my configuration do I ever submit bans using req.*</span></div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><font face="monospace, monospace"><br></font></span></div><div>So I have 3 questions: </div><div><br></div><div>- is 70k+ bans un-reasonable and should I find a way to reduce them?</div><div>- why am I seeing 14 MAIN.bans_req when my config doesn't ever emit such bans?</div><div><br></div><div>Thanks,</div><div>-Hugues</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 3, 2016 at 3:04 AM, Dridi Boukelmoune <span dir="ltr"><<a href="mailto:dridi@varni.sh" target="_blank">dridi@varni.sh</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On Thu, Oct 27, 2016 at 6:02 PM, Nils Goroll <<a href="mailto:slink@schokola.de" target="_blank">slink@schokola.de</a>> wrote:<br>
> Hi,<br>
><br>
> we've added a bunch of ban and ban lurker improvements which did not get<br>
> backported to 4.1.3<br>
><br>
> Please upgrade to 5.0 or master.<br>
<br>
</span>Hi Nils,<br>
<br>
I was thinking (against the too-many-knobs trend) that maybe we should<br>
have some sort of ban_queue_limit parameter to avoid piling up too<br>
many bans. It would also play nice with your std.ban() initiative<br>
where hitting the queue limit could be one of the reasons returned by<br>
the function when it fails.<br>
<br>
The default could be 0 for unlimited, or a fairly large number.<br>
<span class="m_6416731546397539288m_1256840884578708246HOEnZb"><font color="#888888"><br>
Dridi<br>
</font></span><div class="m_6416731546397539288m_1256840884578708246HOEnZb"><div class="m_6416731546397539288m_1256840884578708246h5"><br>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org" target="_blank">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
</div></div></blockquote></div><br></div>
</blockquote></div>