Intermittent http first read error: EOF
Jason Price
japrice at gmail.com
Fri Jan 22 19:54:45 CET 2016
That backend timeout feature does sound promising... Nescalers do reap
connections.
To be completely clear, I was arguing that you should run tcpdump on the
varnish server itself, listening for all traffic to it's direct backend.
You could do that... That'd give you a better idea of what's happening on
that server. To augment it, you'd also need to run 'nstcpdump.sh' on the
netscaler and ideally on the backend itself. Then you'd need to line up
the HTTP transaction across all of them. It'd won't be much fun, but
you'll see exactly where the problem is, and know who to yell at to fix it.
-Jason
On Fri, Jan 22, 2016 at 9:38 AM, L Cruzero <lcruzero at gmail.com> wrote:
> thanks Jason, I've not found any 400+ or 500+ errors in any of web app
> server logs that correlate to varnishlog timestamps. also, I should have
> included this bit of info. the backend that varnish connects to is a
> Netscaler LB, both thread_pool_timeout in varnish and tcp persistent
> connection timeout in LB are set to 120secs. pointing varnish directly to a
> web app server to get a good tcpdump output wont be as straight forward as
> it sounds, due to content switching policies being applied at the NS to
> server various page components from different web app endpoints. I'm
> considering upgrading to 4.1, which list "backend connection timeout" as
> one of the changes.
>
> https://www.varnish-cache.org/docs/trunk/whats-new/changes.html
>
>
> thanks.
> -LC
>
>
> On Fri, Jan 22, 2016 at 8:55 AM, Jason Price <japrice at gmail.com> wrote:
>
>> Is there any chance the web app behind varnish is doing this?
>>
>> I hate to suggest this, but capturing the problem with tcpdump and
>> finding it in wireshark may be the best way of proving which side has the
>> problem. Capture the pcap file with full timestamps, and correlate the
>> times with varnish log, and follow the TCP connection.
>>
>> -Jason
>>
>> On Thu, Jan 21, 2016 at 2:45 PM, L Cruzero <lcruzero at gmail.com> wrote:
>>
>>> Hi, I'm occasionally seeing this error " http first read error: EOF" in
>>> varnishlog for content that exist, and not exceeding "first_byte_timeout"
>>> TTL
>>> I was considering issuing a restart < 4 on 503's with a URL condition
>>> match since this is happening pretty rarely, I'm seeing the error just 2-3
>>> times while also getting a 200 for the same html asset 3,000+ times within
>>> a couple of mins of logging.
>>>
>>>
>>> varnish-4.0.3
>>>
>>>
>>> # varnishadm "param.show first_byte_timeout"
>>>
>>> first_byte_timeout
>>>
>>> Value is: 60.000 [seconds] (default)
>>>
>>> Default is: 60.000
>>>
>>> Minimum is: 0.000
>>>
>>> # varnishadm "param.show thread_pool_timeout"
>>>
>>> thread_pool_timeout
>>>
>>> Value is: 120.000 [seconds]
>>>
>>> Default is: 300.000
>>>
>>> Minimum is: 10.000
>>>
>>>
>>> VCL code used to define and use backend where 503 errors are being
>>> generated..
>>>
>>> backend wwwdot {
>>>
>>> .host = "web-prod-ssf.domain.ly";
>>>
>>> .port = "80";
>>>
>>> }
>>>
>>>
>>> if (req.http.host ~ "^(origin-www|www)") {
>>>
>>> set req.backend_hint = wwwdot;
>>>
>>> return(pass);
>>>
>>> }
>>>
>>> << BeReq >> 575646365
>>>
>>> - Begin bereq 575646364 pass
>>>
>>> - Timestamp Start: 1453229310.040839 0.000000 0.000000
>>>
>>> - BereqMethod GET
>>>
>>> - BereqURL /toprail-domain.html
>>>
>>> - BereqProtocol HTTP/1.1
>>>
>>> - BereqHeader DNT: 1
>>>
>>> - BereqHeader Cookie: bknx_fa=1453197478872; bknx_ss=1453229297581;
>>> CPN_crispkey=;
>>> CPN_geo=eyJpcF9hZGRyZXNzIjoiNzAuMTk2LjEzMi4yMyIsImlwX3R5cGUiOiJNYXBwZWQiLCJOZXR3b3JrIjp7ImNvbm5lY3Rpb25fdHlwZSI6Im1vYmlsZSB3aXJlbGVzcyIsImxpbmVfc3BlZWQiOiJsb3ciLCJpcF9yb3V0aW5nX3R5cGUiO
>>>
>>> - BereqHeader Accept: */*
>>>
>>> - BereqHeader User-Agent: Mozilla/5.0 (iPhone; CPU iPhone OS 9_2
>>> like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0
>>> Mobile/13C75 Safari/601.1
>>>
>>> - BereqHeader Accept-Language: en-us
>>>
>>> - BereqHeader Referer:
>>> http://www.domain.com/outdoors/index.ssf/2014/02/wild_boar_attacks_slidell_man.html
>>>
>>> - BereqHeader X-Client-Dest-Addr: 69.164.6.110
>>>
>>> - BereqHeader True-Client-IP: 70.196.132.44
>>>
>>> - BereqHeader X-Via: 1.1 sw.cds943.dal.llnw.net:8000 (EdgePrism/
>>> 4.3.1.0), 1.1 cds1158.dal.llnw.net:80 (EdgePrism/4.3.1.0), 1.1
>>> cds1079.lga.llnw.net:80 (EdgePrism/4.3.1.0)
>>>
>>> - BereqHeader Host: www.domain.com
>>>
>>> - BereqHeader Accept-Encoding: identity
>>>
>>> - BereqHeader X-Forwarded-For: 70.196.132.44, 69.164.7.89,
>>> 69.164.43.169, 69.164.48.181, 10.51.13.254
>>>
>>> - BereqHeader X-Varnish: 575646365
>>>
>>> - VCL_call BACKEND_FETCH
>>>
>>> - VCL_return fetch
>>>
>>> - BackendOpen 86 wwwdot(69.4.99.100,,80) 10.51.13.97 56085
>>>
>>> - Backend 86 wwwdot wwwdot(69.2.99.10,,80)
>>>
>>> - Timestamp Bereq: 1453229310.041475 0.000635 0.000635
>>>
>>> - FetchError http first read error: EOF
>>>
>>> - BackendClose 86 wwwdot(69.4.99.100,,80)
>>>
>>> - Timestamp Beresp: 1453229310.041717 0.000877 0.000242
>>>
>>> - Timestamp Error: 1453229310.041720 0.000881 0.000003
>>>
>>> - BerespProtocol HTTP/1.1
>>>
>>> - BerespStatus 503
>>>
>>> - BerespReason Service Unavailable
>>>
>>> - BerespReason Backend fetch failed
>>>
>>> - BerespHeader Date: Tue, 19 Jan 2016 18:48:30 GMT
>>>
>>> - BerespHeader Server: Varnish
>>>
>>> - VCL_call BACKEND_ERROR
>>>
>>> - BerespHeader Content-Type: text/html; charset=utf-8
>>>
>>> - BerespHeader Retry-After: 5
>>>
>>> - VCL_return deliver
>>>
>>> - Storage malloc Transient
>>>
>>> - ObjProtocol HTTP/1.1
>>>
>>> - ObjStatus 503
>>>
>>> - ObjReason Backend fetch failed
>>>
>>> - ObjHeader Date: Tue, 19 Jan 2016 18:48:30 GMT
>>>
>>> - ObjHeader Server: Varnish
>>>
>>> - ObjHeader Content-Type: text/html; charset=utf-8
>>>
>>> - ObjHeader Retry-After: 5
>>>
>>> - Length 286
>>>
>>> - BereqAcct 5960 0 5960 0 0 0
>>>
>>> - End
>>>
>>>
>>> any suggestions and or ideas on solving this issue would be much
>>> appreciated.
>>>
>>> Thanks
>>>
>>> -LC
>>>
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20160122/6ac04eaa/attachment.html>
More information about the varnish-misc
mailing list