Varnish 503 on POST requests once in a while
Maninder Singh
mandys at gmail.com
Thu Dec 26 11:57:10 UTC 2019
Got one in the log.
varnishlog -r /var/log/varnish/503.log
* << BeReq >> 21993384
- Begin bereq 21993383 pass
- Timestamp Start: 1577360704.829258 0.000000 0.000000
- BereqMethod POST
- BereqURL /corp/contact.php?cf=exitpopup
- BereqProtocol HTTP/1.1
- BereqHeader User-Agent: Mozilla/5.0 (Windows NT 6.1; Trident/7.0;
rv:11.0) like Gecko
- BereqHeader Accept: image/gif, image/x-xbitmap, image/jpeg,
image/pjpeg, application/x-shockwave-flash, */*
- BereqHeader SSLClientCertStatus: NoClientCert
- BereqHeader X-Forwarded-Proto: https
- BereqHeader Content-Type: application/x-www-form-urlencoded
- BereqHeader SSLClientCipher: SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA,
version=TLSv1, bits=256
- BereqHeader Host: www.mydomain.com
- BereqHeader Referer: http://mydomain.com/solutions/enterprise.php
- BereqHeader SSLSessionID:
3C119EF3FE70318789F82E9B583581C876B4DDE520C0182C33CD6AF1AA46E63A
- BereqHeader X-Cluster-Client-Ip: X.X.X.X
- BereqHeader Cookie:
X-Mapping-fjhppofk=E8F494E9A630955F1975EEDF11EBC4D7
- BereqHeader X-Forwarded-Port: 443
- BereqHeader Accept-Encoding: gzip, deflate
- BereqHeader Content-Length: 1366
- BereqHeader X-Forwarded-For: X.X.X.X, 10.187.187.36
- BereqHeader browser: other
- BereqHeader serverIp: 10.208.224.192
- BereqHeader X-Varnish: 21993384
- VCL_call BACKEND_FETCH
- VCL_return fetch
- BackendOpen 26 reload_2019-12-18T054253.default 127.0.0.1 8080
127.0.0.1 59998
- BackendStart 127.0.0.1 8080
- Timestamp Bereq: 1577360705.664932 0.835674 0.835674
- FetchError http first read error: EOF
- BackendClose 26 reload_2019-12-18T054253.default
- Timestamp Beresp: 1577360705.665020 0.835763 0.000089
- Timestamp Error: 1577360705.665028 0.835770 0.000008
- BerespProtocol HTTP/1.1
- BerespStatus 503
- BerespReason Service Unavailable
- BerespReason Backend fetch failed
- BerespHeader Date: Thu, 26 Dec 2019 11:45:05 GMT
- BerespHeader Server: Varnish
- VCL_call BACKEND_ERROR
- BerespHeader Content-Type: text/html; charset=utf-8
- BerespHeader Retry-After: 5
- VCL_return deliver
- Storage malloc Transient
- ObjProtocol HTTP/1.1
- ObjStatus 503
- ObjReason Backend fetch failed
- ObjHeader Date: Thu, 26 Dec 2019 11:45:05 GMT
- ObjHeader Server: Varnish
- ObjHeader Content-Type: text/html; charset=utf-8
- ObjHeader Retry-After: 5
- Length 1288
- BereqAcct 849 1366 2215 0 0 0
- End
On Thu, 26 Dec 2019 at 15:21, Maninder Singh <mandys at gmail.com> wrote:
> Okay. Let me set it up and get back to you once i see something in that
> log. It happens only once/twice a day so could be a while before i get
> back.
>
> On Thu, 26 Dec, 2019, 3:10 pm Guillaume Quintard, <
> guillaume at varnish-software.com> wrote:
>
>> Hi,
>>
>> Most probably, varnishncsa isn't going to give you the full picture, so
>> let's go for the exhaustive way:
>>
>> varnishlog -q "BerespStatus eq 503" -w /whatever/file
>>
>>
>> you can then read the logs with
>>
>> varnishlog -r /whatever/file
>>
>>
>> (more logging pointers here:
>> https://docs.varnish-software.com/tutorials/vsl-query/)
>>
>> If you can get your hands on a failed transaction, we should be able to
>> tell you what went wrong. Most probably, you hit a timeout of some sort. Or
>> it could be that the backend was declared sick, if you have probes.
>>
>> --
>> Guillaume Quintard
>>
>>
>> On Thu, Dec 26, 2019 at 6:33 AM Maninder Singh <mandys at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I am facing a strange problem.
>>>
>>> I am running varnish 4.1.11 on port 80 which is behind a LB.
>>>
>>> Varnish then connects to apache 2.4 ( on port 8080 ) and also have
>>> php-fpm running.
>>>
>>> Now, everything works fine but once in a while I am seeing 503 response
>>> from varnish for a POST request. During this time there is no log in apache
>>> also ( neither access log nor error log ).
>>>
>>> Where does this request vanish ?
>>>
>>> How do I debug this issue ?
>>>
>>> Some more details:
>>>
>>> * I have varnishncsa turned on with following flags.
>>>
>>> ExecStart=/usr/bin/varnishncsa -q 'not (ReqURL ~
>>> "\.(js|css|png|jpeg|jpg|ico|gif|swf|woff)" or ReqHeader:User-Agent ~
>>> "HTTP-Monitor" or ReqHeader:User-Agent ~ "Monitoring/1.1")' -a -w
>>> /var/log/varnish/varnishncsa.log -D -f /etc/sysconfig/varnishncsa
>>>
>>> Here is the entry in varnish log during that time.
>>>
>>> X.X.X.X, LB.IP.0.0 - - [25/Dec/2019:00:26:40 -0600] 2629363 "POST
>>> http://mydomain.com/LetsCelebrate HTTP/1.1" 503 1288 "
>>> https://referringdomain.com/LetsCelebrate?hostingUrl=https%3A%2F%2Fxy.abc.com"
>>> "Mozilla/5.0 (Linux; Android 9; SM-A920F) AppleWebKit/537.36 (KHTML, like
>>> Gecko) Chrome/79.0.3945.93 Mobile Safari/537.36" "mydomain.com" miss
>>>
>>> In .vcl file I have following set.
>>> .first_byte_timeout = 120s;
>>>
>>> Now, I am not logging varnish to apache requests ( which backend fetch ).
>>> As you can see above I am only logging client requests.
>>>
>>> Is their a way to log client requests like above but then also log
>>> backend fetch failures to another log file ?
>>> [ Running 2 instances of varnishncsa ? ]
>>>
>>> Any pointers ?
>>>
>>> Please let me know.
>>>
>>> Thanks.
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20191226/92d93452/attachment-0001.html>
More information about the varnish-misc
mailing list