<div dir="ltr">Hello James,<div><br></div><div>Varnish hit the lru limit, meaning it delete N objects to make room for the new one, and decided it was enough. N is called "nuke_limit", you can change it at launch time using the -p switch.</div><div><br></div><div>If your objects are indeed very diverse in size, it makes sense to have multiple storages, one for small content, and another for larger content. That way you won't need to go through a million objects to push a new one in cache.</div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>-- <br></div>Guillaume Quintard<br></div></div></div>
<br><div class="gmail_quote">On Fri, Nov 17, 2017 at 3:54 PM, James Mathiesen <span dir="ltr"><<a href="mailto:jmathiesen@tripadvisor.com" target="_blank">jmathiesen@tripadvisor.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="m_5585345984887977926WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">This is running RPM varnish-4.1.8-1.el7.x86_64 in a kubernetes container.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We had user complaints that a binary object (~100MB) was coming back truncated on every fetch.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Fetching the object I saw the following in the varnish logs:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- ExpKill LRU_Cand p=0x7feffe41abc0 f=0x0 r=1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- ExpKill LRU x=435945906<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- ExpKill LRU_Cand p=0x7ff018c63880 f=0x0 r=1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- ExpKill LRU x=423595256<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- ExpKill LRU_Cand p=0x7ff0129314c0 f=0x0 r=1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- ExpKill LRU x=432965548<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- ExpKill LRU_Exhausted<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- FetchError Could not get storage<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- BackendClose 29 boot.default<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- BereqAcct 260 0 260 399 2789376 2789775<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- End <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Which seems consistent with the symptom -- the backend transfer starts and gets streamed to the client but partway through the transfer the backend and frontend connections are reset.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The problem is going to be difficult to reproduce as the symptom only appears after the container has been running for months.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The dataset is media objects of various sizes (from tiny thumbnails to a few hundred MB) and the cache available is 5GB. The cache is much, much smaller than the working set. The object we noticed the problem
with is about 100MB.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Before I spent a lot of time gathering further data I wanted to understand if I'm hitting a known behavior and whether there's any value to anyone if I try and gather more information vs. just planning an
upgrade to 5.2 and seeing if the problem goes away.<span class="HOEnZb"><font color="#888888"><u></u><u></u></font></span></span></p><span class="HOEnZb"><font color="#888888">
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">james<u></u><u></u></span></p>
</font></span></div>
</div>
<br>______________________________<wbr>_________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer" target="_blank">https://www.varnish-cache.org/<wbr>lists/mailman/listinfo/<wbr>varnish-misc</a><br></blockquote></div><br></div>