<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">This is running RPM varnish-4.1.8-1.el7.x86_64 in a kubernetes container.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We had user complaints that a binary object (~100MB) was coming back truncated on every fetch.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Fetching the object I saw the following in the varnish logs:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- ExpKill LRU_Cand p=0x7feffe41abc0 f=0x0 r=1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- ExpKill LRU x=435945906<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- ExpKill LRU_Cand p=0x7ff018c63880 f=0x0 r=1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- ExpKill LRU x=423595256<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- ExpKill LRU_Cand p=0x7ff0129314c0 f=0x0 r=1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- ExpKill LRU x=432965548<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- ExpKill LRU_Exhausted<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- FetchError Could not get storage<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- BackendClose 29 boot.default<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- BereqAcct 260 0 260 399 2789376 2789775<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">- End <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Which seems consistent with the symptom -- the backend transfer starts and gets streamed to the client but partway through the transfer the backend and frontend connections are reset.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The problem is going to be difficult to reproduce as the symptom only appears after the container has been running for months.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The dataset is media objects of various sizes (from tiny thumbnails to a few hundred MB) and the cache available is 5GB. The cache is much, much smaller than the working set. The object we noticed the problem
with is about 100MB.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Before I spent a lot of time gathering further data I wanted to understand if I'm hitting a known behavior and whether there's any value to anyone if I try and gather more information vs. just planning an
upgrade to 5.2 and seeing if the problem goes away.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">james<o:p></o:p></span></p>
</div>
</body>
</html>