<div dir="ltr">I cannot cut the LB out due to network design. There is no log on the LB also. Problematic requests are not large (js/css files), I suspect that this is a miscommunication issue between Varnish and Apache but I am not able to detect the reason yet.</div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Apr 3, 2017 at 10:32 AM, Guillaume Quintard <span dir="ltr"><<a href="mailto:guillaume@varnish-software.com" target="_blank">guillaume@varnish-software.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Large requests/responses are dropped by the LB, maybe?</div><div class="gmail_extra"><br clear="all"><div><div class="m_-7691884057122096493gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>-- <br></div>Guillaume Quintard<br></div></div></div>
<br><div class="gmail_quote">On Mon, Apr 3, 2017 at 9:12 AM, Andrei <span dir="ltr"><<a href="mailto:lagged@gmail.com" target="_blank">lagged@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">So the Varnish backend requests go through a load balancer before reaching Apache? What about those logs? What if you cut that LB out, and just use directors to LB in Varnish directly?</div><div class="m_-7691884057122096493HOEnZb"><div class="m_-7691884057122096493h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Apr 2, 2017 at 12:39 PM, Hazar Güney <span dir="ltr"><<a href="mailto:hazarguney@gmail.com" target="_blank">hazarguney@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><span style="font-size:12.8px">Btw, I need to also note that traffic is routed to Varnish from load balancer:</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">LB -> Varnish -> LB -> Backend pool</span></div><span style="font-size:12.8px"><div><span style="font-size:12.8px"><br></span></div></span><div><span style="font-size:12.8px">Time does not matter.. It occurs during both peak and regular hours. </span><span style="font-size:12.8px">Even during peak hours we do not reach the "local ports" limit. </span><span style="font-size:12.8px">Unfortunately there is no any clue in the logs. </span><span style="font-size:12.8px">There is no evidence that Apache restarts on the backend pool during occurence of the issue.</span></div></div><div class="m_-7691884057122096493m_7587588026377711803HOEnZb"><div class="m_-7691884057122096493m_7587588026377711803h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Apr 1, 2017 at 9:44 PM, Andrei <span dir="ltr"><<a href="mailto:lagged@gmail.com" target="_blank">lagged@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">If it's during peak hours are you sure there aren't any rate limits being reached? Perhaps net.ipv4.ip_local_port<wbr>_range might need a bump? Are Apache or syslog logging anything around those times? No silly periodic (Apache) graceful restarts? Just a few thoughts :)</div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Apr 1, 2017 at 2:14 PM, Hazar Güney <span dir="ltr"><<a href="mailto:hazarguney@gmail.com" target="_blank">hazarguney@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">We see this error a few times in a day on a highly busy production environment. Unfortunately there is too much traffic on the server to keep tcpdump/ngrep running and we cannot re-produce it on test environment :(<div><br></div><div>I have started tcpdump on a test environment of another implementation and will let you as soon as the issue gets triggerred again.</div></div><div class="m_-7691884057122096493m_7587588026377711803m_3929882159414030332m_3347131316372979983HOEnZb"><div class="m_-7691884057122096493m_7587588026377711803m_3929882159414030332m_3347131316372979983h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 31, 2017 at 4:17 PM, Andrei <span dir="ltr"><<a href="mailto:lagged@gmail.com" target="_blank">lagged@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Can you provide a tcpdump/ngrep of the requests between Client/Varnish/Apache along with the varnishlog entry to see if that uncovers anything?</div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 31, 2017 at 7:25 AM, Hazar Güney <span dir="ltr"><<a href="mailto:hazarguney@gmail.com" target="_blank">hazarguney@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Any idea?</div><div class="m_-7691884057122096493m_7587588026377711803m_3929882159414030332m_3347131316372979983m_6468746499687868456m_40741382235127568HOEnZb"><div class="m_-7691884057122096493m_7587588026377711803m_3929882159414030332m_3347131316372979983m_6468746499687868456m_40741382235127568h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 30, 2017 at 3:41 PM, Hazar Güney <span dir="ltr"><<a href="mailto:hazarguney@gmail.com" target="_blank">hazarguney@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">It did not work either:<div><br></div><div><div> * << BeReq >> 127418176 </div><div> - Begin bereq 127418175 fetch</div><div> - Timestamp Start: 1490877149.450124 0.000000 0.000000</div><div> - BereqMethod GET</div><div> - BereqURL XXXX</div><div> - BereqProtocol HTTP/1.1</div><div> - BereqHeader Accept: text/css,*/*;q=0.1</div><div> - BereqHeader User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 10_2 like Mac OS X) AppleWebKit/602.3.12 (KHTML, like Gecko) Version/10.0 Mobile/14C92 Safari/602.1</div><div> - BereqHeader Accept-Language: tr-tr</div><div> - BereqHeader Referer: XXXX</div><div> - BereqHeader Host: XXXX</div><div> - BereqHeader RIP: XXXX</div><div> - BereqHeader X-Forwarded-For: XXXX</div><div> - BereqHeader Accept-Encoding: gzip</div><div> - BereqHeader X-Varnish: 127418176</div><div> - VCL_call BACKEND_FETCH</div><div> - BereqHeader connection: Close</div><div> - VCL_return fetch</div><div> - BackendOpen 25 reload_2017-03-30T14:53:46.st2 10.35.78.11 80 172.17.0.2 59152</div><div> - BackendStart 10.35.78.11 80</div><div> - Timestamp Bereq: 1490877149.450594 0.000470 0.000470</div><div> - FetchError http first read error: EOF</div><div> - BackendClose 25 reload_2017-03-30T14:53:46.st2</div><div> - Timestamp Beresp: 1490877149.451184 0.001060 0.000590</div><div> - Timestamp Error: 1490877149.451189 0.001065 0.000005</div><div> - BerespProtocol HTTP/1.1</div><div> - BerespStatus 503</div><div> - BerespReason Service Unavailable</div><div> - BerespReason Backend fetch failed</div><div> - BerespHeader Date: Thu, 30 Mar 2017 12:32:29 GMT</div><div> - BerespHeader Server: Varnish</div><div> - VCL_call BACKEND_ERROR</div><div> - BereqHeader X-Varnish-Backend-5xx: 1</div><div> - VCL_return retry</div><div> - Timestamp Retry: 1490877149.451205 0.001081 0.000016</div><div> - Link bereq 127298071 retry</div><div> - End </div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 30, 2017 at 2:34 PM, Guillaume Quintard <span dir="ltr"><<a href="mailto:guillaume@varnish-software.com" target="_blank">guillaume@varnish-software.co<wbr>m</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">It does, I'm suspecting that the connection reuse is creating some issues, probably because Apache is doing some non-standard stuff (protip: always blame Apache).</div><div class="gmail_extra"><br clear="all"><div><div class="m_-7691884057122096493m_7587588026377711803m_3929882159414030332m_3347131316372979983m_6468746499687868456m_40741382235127568m_7464633421317302291m_442754893298189936m_-1123069974190272813gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>-- <br></div>Guillaume Quintard<br></div></div></div>
<br><div class="gmail_quote">On Thu, Mar 30, 2017 at 1:17 PM, Hazar Güney <span dir="ltr"><<a href="mailto:hazarguney@gmail.com" target="_blank">hazarguney@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">"Connection: close" supersedes keep-alive behavior, is that correct?</div><div class="m_-7691884057122096493m_7587588026377711803m_3929882159414030332m_3347131316372979983m_6468746499687868456m_40741382235127568m_7464633421317302291m_442754893298189936m_-1123069974190272813HOEnZb"><div class="m_-7691884057122096493m_7587588026377711803m_3929882159414030332m_3347131316372979983m_6468746499687868456m_40741382235127568m_7464633421317302291m_442754893298189936m_-1123069974190272813h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 30, 2017 at 2:08 PM, Guillaume Quintard <span dir="ltr"><<a href="mailto:guillaume@varnish-software.com" target="_blank">guillaume@varnish-software.co<wbr>m</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Can you try something: add 'set bereq.http.connection = "Close"; ' at the beginning of vcl_backend_fetch and see if that helps?</div><div class="gmail_extra"><br clear="all"><div><div class="m_-7691884057122096493m_7587588026377711803m_3929882159414030332m_3347131316372979983m_6468746499687868456m_40741382235127568m_7464633421317302291m_442754893298189936m_-1123069974190272813m_7684082798158834847m_7609544558872835544gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>-- <br></div>Guillaume Quintard<br></div></div></div>
<br><div class="gmail_quote">On Thu, Mar 30, 2017 at 1:04 PM, Hazar Güney <span dir="ltr"><<a href="mailto:hazarguney@gmail.com" target="_blank">hazarguney@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">MaxKeepAliveRequests 20<br><div>KeepAliveTimeout 2</div><div><br></div><div>Version is "4.1.3 revision 5e3b6d2". We have also seen "<span style="font-size:12.8px">straight insufficient bytes" error with POST requests to a specific php script hosted by another backend and fixed it by using "pipe" instead of "pass" but this specific backend gives "http first read error: EOF" error. Another example from today:</span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">* << BeReq >> 126635444 </span></div><div><span style="font-size:12.8px">- Begin bereq 126635443 fetch</span></div><div><span style="font-size:12.8px">- Timestamp Start: 1490870598.921499 0.000000 0.000000</span></div><span><div><span style="font-size:12.8px">- BereqMethod GET</span></div><div><span style="font-size:12.8px">- BereqURL XXXX</span></div><div><span style="font-size:12.8px">- BereqProtocol HTTP/1.1</span></div></span><div><span style="font-size:12.8px">- BereqHeader Host: XXXX</span></div><div><span style="font-size:12.8px">- BereqHeader User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36</span></div><div><span style="font-size:12.8px">- BereqHeader Accept: image/webp,image/*,*/*;q=0.8</span></div><div><span style="font-size:12.8px">- BereqHeader Referer: XXXX</span></div><div><span style="font-size:12.8px">- BereqHeader Accept-Language: tr-TR,tr;q=0.8,en-US;q=0.6,en;<wbr>q=0.4</span></div><div><span style="font-size:12.8px">- BereqHeader RIP: XXXX</span></div><div><span style="font-size:12.8px">- BereqHeader X-Forwarded-For: XXXX</span></div><div><span style="font-size:12.8px">- BereqHeader Accept-Encoding: gzip</span></div><div><span style="font-size:12.8px">- BereqHeader X-Varnish: 126635444</span></div><span><div><span style="font-size:12.8px">- VCL_call BACKEND_FETCH</span></div><div><span style="font-size:12.8px">- VCL_return fetch</span></div></span><div><span style="font-size:12.8px">- BackendOpen 35 reload_2017-03-20T11:32:44.st2 10.35.78.11 80 172.17.0.2 48896</span></div><div><span style="font-size:12.8px">- BackendStart 10.35.78.11 80</span></div><div><span style="font-size:12.8px">- Timestamp Bereq: 1490870598.922050 0.000552 0.000552</span></div><span><div><span style="font-size:12.8px"><b>- FetchError http first read error: EOF</b></span></div></span><div><span style="font-size:12.8px">- BackendClose 35 reload_2017-03-20T11:32:44.st2</span></div><div><span style="font-size:12.8px">- Timestamp Beresp: 1490870598.922622 0.001124 0.000572</span></div><div><span style="font-size:12.8px">- Timestamp Error: 1490870598.922627 0.001129 0.000005</span></div><span><div><span style="font-size:12.8px">- BerespProtocol HTTP/1.1</span></div><div><span style="font-size:12.8px">- BerespStatus 503</span></div><div><span style="font-size:12.8px">- BerespReason Service Unavailable</span></div><div><span style="font-size:12.8px">- BerespReason Backend fetch failed</span></div></span><div><span style="font-size:12.8px">- BerespHeader Date: Thu, 30 Mar 2017 10:43:18 GMT</span></div><span><div><span style="font-size:12.8px">- BerespHeader Server: Varnish</span></div><div><span style="font-size:12.8px">- VCL_call BACKEND_ERROR</span></div><div><span style="font-size:12.8px">- BereqHeader X-Varnish-Backend-5xx: 1</span></div><div><span style="font-size:12.8px">- VCL_return retry</span></div></span><div><span style="font-size:12.8px">- Timestamp Retry: 1490870598.922657 0.001159 0.000030</span></div><div><span style="font-size:12.8px">- Link bereq 126832283 retry</span></div><div><span style="font-size:12.8px">- End </span><br></div></div><div class="m_-7691884057122096493m_7587588026377711803m_3929882159414030332m_3347131316372979983m_6468746499687868456m_40741382235127568m_7464633421317302291m_442754893298189936m_-1123069974190272813m_7684082798158834847m_7609544558872835544HOEnZb"><div class="m_-7691884057122096493m_7587588026377711803m_3929882159414030332m_3347131316372979983m_6468746499687868456m_40741382235127568m_7464633421317302291m_442754893298189936m_-1123069974190272813m_7684082798158834847m_7609544558872835544h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 29, 2017 at 12:03 PM, Mattias Geniar <span dir="ltr"><<a href="mailto:mattias@nucleus.be" target="_blank">mattias@nucleus.be</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">> Backend is Apache.<br>
<br>
In older Varnish versions, you could sometimes see a similar error;<br>
<br>
> 11 FetchError c straight insufficient bytes<br>
<br>
The error message you’re seeing might be related, as it mentions the EOF.<br>
<br>
This happens when the backend sends a Content-Length header that doesn’t match the _actual_ content length it’s sending. In Apache, this was commonly caused by a mod_deflate misconfiguration.<br>
<br>
For testing, could you try disabling Gzip either in your backend or strip the Accept-Encoding header in Varnish to force a plain text response?<br>
<span class="m_-7691884057122096493m_7587588026377711803m_3929882159414030332m_3347131316372979983m_6468746499687868456m_40741382235127568m_7464633421317302291m_442754893298189936m_-1123069974190272813m_7684082798158834847m_7609544558872835544m_-8440447935089024071HOEnZb"><font color="#888888"><br>
Mattias<br>
<br>
</font></span></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</blockquote></div><br></div>
</blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org" target="_blank">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer" target="_blank">https://www.varnish-cache.org/<wbr>lists/mailman/listinfo/varnish<wbr>-misc</a><br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</blockquote></div><br></div>