From varnish at funzi.de Thu Mar 15 07:01:55 2018 From: varnish at funzi.de (Christopher Beppler) Date: Thu, 15 Mar 2018 08:01:55 +0100 Subject: Assert error in VPX_Send_Proxy() Message-ID: <8891bb34-c6d6-85e9-0e39-fb96e8c6b9ff@funzi.org> Hi, I am new to this list and I hope that someone here can shed some light on my issue. I have googled the error message but didn't find anything related to VPX_Send_Proxy(). My setup is as follows: Haproxy --(proxy-v2)-> varnish --(proxy-v1)-> nginx -> webapps I am running FreeBSD 11.1-RELEASE-p4 and varnish5-5.2.1 from ports. Since a few days, I get the following error message. Not all the time, but always around the same time. So it might be an issue related to a certain load: Mar 14 23:44:04 varnish varnishd[15349]: Child (24895) Panic at: Wed, 14 Mar 2018 22:44:04 GMT Assert error in VPX_Send_Proxy(), proxy/cache_proxy_proto.c line 425: Condition((sp) != NULL) not true. version = varnish-5.2.1 revision NOGIT, vrt api = 6.1 ident = FreeBSD,11.1-RELEASE-p4,amd64,-junix,-sfile,-smalloc,-hcritbit,kqueue now = 4710006.893845 (mono), 1521067443.500616 (real) Backtrace: 0x437e60: at /usr/local/sbin/varnishd 0x47e0bc: at /usr/local/sbin/varnishd 0x46fb96: at /usr/local/sbin/varnishd 0x4173bf: at /usr/local/sbin/varnishd 0x4162f7: at /usr/local/sbin/varnishd 0x43d184: at /usr/local/sbin/varnishd 0x458841: at /usr/local/sbin/varnishd 0x451f96: at /usr/local/sbin/varnishd 0x451d15: at /usr/local/sbin/varnishd thread = (cache-worker) thr.req = 0x80f902ba0 { vxid = 98306, transport This results in a client disconnect. If I retry loading the page it sometimes work. If I am not mistaken, varnish kills a process/thread if an assertion fails and spawns a new one to stay in a clean state. That's why I don't have to restart anything. After some time, it seems to recover but during the time this error appears the page is almost unusable. Why question would now be what I can do to resolve that and what it means that sp == NULL at this particular assertion? If you need more logs or details, I am happy to provide them. Thanks in advance, Christopher From daghf at varnish-software.com Thu Mar 15 12:39:09 2018 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Thu, 15 Mar 2018 13:39:09 +0100 Subject: Assert error in VPX_Send_Proxy() In-Reply-To: <8891bb34-c6d6-85e9-0e39-fb96e8c6b9ff@funzi.org> References: <8891bb34-c6d6-85e9-0e39-fb96e8c6b9ff@funzi.org> Message-ID: Hi Christopher, This certainly looks like a bug. Could you please report this in https://github.com/varnishcache/varnish-cache/issues ? Please include as much detail as possible, and the full panic output from 'varnishadm panic.show'. On Thu, Mar 15, 2018 at 8:01 AM, Christopher Beppler wrote: > Hi, > > I am new to this list and I hope that someone here can shed some light > on my issue. I have googled the error message but didn't find anything > related to VPX_Send_Proxy(). > > My setup is as follows: > > Haproxy --(proxy-v2)-> varnish --(proxy-v1)-> nginx -> webapps > > I am running FreeBSD 11.1-RELEASE-p4 and varnish5-5.2.1 from ports. > > Since a few days, I get the following error message. Not all the time, > but always around the same time. So it might be an issue related to a > certain load: > > Mar 14 23:44:04 varnish varnishd[15349]: Child (24895) Panic at: Wed, 14 > Mar 2018 22:44:04 GMT Assert error in VPX_Send_Proxy(), > proxy/cache_proxy_proto.c line 425: Condition((sp) != NULL) not true. > version = varnish-5.2.1 revision NOGIT, vrt api = 6.1 ident = > FreeBSD,11.1-RELEASE-p4,amd64,-junix,-sfile,-smalloc,-hcritbit,kqueue > now = 4710006.893845 (mono), 1521067443.500616 (real) Backtrace: > 0x437e60: at /usr/local/sbin/varnishd 0x47e0bc: > at /usr/local/sbin/varnishd 0x46fb96: > at /usr/local/sbin/varnishd 0x4173bf: > at /usr/local/sbin/varnishd 0x4162f7: > at /usr/local/sbin/varnishd 0x43d184: > at /usr/local/sbin/varnishd 0x458841: > at /usr/local/sbin/varnishd 0x451f96: > at /usr/local/sbin/varnishd 0x451d15: > at /usr/local/sbin/varnishd thread = (cache-worker) > thr.req = 0x80f902ba0 { vxid = 98306, transport > > This results in a client disconnect. If I retry loading the page it > sometimes work. > > If I am not mistaken, varnish kills a process/thread if an assertion > fails and spawns a new one to stay in a clean state. That's why I don't > have to restart anything. After some time, it seems to recover but > during the time this error appears the page is almost unusable. > > Why question would now be what I can do to resolve that and what it > means that sp == NULL at this particular assertion? > > If you need more logs or details, I am happy to provide them. > > Thanks in advance, > Christopher > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Dag Haavi Finstad Software Developer | Varnish Software Mobile: +47 476 64 134 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From varnish at funzi.de Thu Mar 15 15:37:12 2018 From: varnish at funzi.de (Christopher Beppler) Date: Thu, 15 Mar 2018 16:37:12 +0100 Subject: Assert error in VPX_Send_Proxy() In-Reply-To: References: <8891bb34-c6d6-85e9-0e39-fb96e8c6b9ff@funzi.org> Message-ID: Hi, Thank you. I've submitted it as #2613. While looking at the panic, it might be triggered by a PROPFIND from an OpenOffice client. But it could also just be a coincident. Regards, Christopher Am 15.03.2018 um 13:39 schrieb Dag Haavi Finstad: > Hi Christopher, > > This certainly looks like a bug. > > Could you please report this in > https://github.com/varnishcache/varnish-cache/issues ? > > Please include as much detail as possible, and the full panic output > from 'varnishadm panic.show'. > > On Thu, Mar 15, 2018 at 8:01 AM, Christopher Beppler > wrote: > > Hi, > > I am new to this list and I hope that someone here can shed some light > on my issue. I have googled the error message but didn't find anything > related to VPX_Send_Proxy(). > > My setup is as follows: > > Haproxy --(proxy-v2)-> varnish --(proxy-v1)-> nginx -> webapps > > I am running FreeBSD 11.1-RELEASE-p4 and varnish5-5.2.1 from ports. > > Since a few days, I get the following error message. Not all the time, > but always around the same time. So it might be an issue related to a > certain load: > > Mar 14 23:44:04 varnish varnishd[15349]: Child (24895) Panic at: Wed, 14 > Mar 2018 22:44:04 GMT Assert error in VPX_Send_Proxy(), > proxy/cache_proxy_proto.c line 425:? ?Condition((sp) != NULL) not true. > version = varnish-5.2.1 revision NOGIT, vrt api = 6.1 ident = > FreeBSD,11.1-RELEASE-p4,amd64,-junix,-sfile,-smalloc,-hcritbit,kqueue > now = 4710006.893845 (mono), 1521067443.500616 (real) Backtrace: > 0x437e60: at /usr/local/sbin/varnishd? ?0x47e0bc: > at /usr/local/sbin/varnishd? ?0x46fb96: > at /usr/local/sbin/varnishd? ?0x4173bf: > at /usr/local/sbin/varnishd? ?0x4162f7: > at /usr/local/sbin/varnishd? ?0x43d184: > at /usr/local/sbin/varnishd? ?0x458841: > at /usr/local/sbin/varnishd? ?0x451f96: > at /usr/local/sbin/varnishd? ?0x451d15: > at /usr/local/sbin/varnishd thread = (cache-worker) > thr.req = 0x80f902ba0 {? ?vxid = 98306, transport > > This results in a client disconnect. If I retry loading the page it > sometimes work. > > If I am not mistaken, varnish kills a process/thread if an assertion > fails and spawns a new one to stay in a clean state. That's why I don't > have to restart anything. After some time, it seems to recover but > during the time this error appears the page is almost unusable. > > Why question would now be what I can do to resolve that and what it > means that sp == NULL at this particular assertion? > > If you need more logs or details, I am happy to provide them. > > Thanks in advance, > Christopher > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > > > -- > Dag Haavi Finstad > Software Developer | Varnish Software > Mobile: +47 476 64 134 > We Make Websites Fly! From hugues at betabrand.com Thu Mar 29 18:24:41 2018 From: hugues at betabrand.com (Hugues Alary) Date: Thu, 29 Mar 2018 11:24:41 -0700 Subject: VCL 4.1 and beresp.backend.ip Message-ID: Hi there, First, congratulations and thanks for releasing Varnish 6.0! VCL 4.1 retired beresp.backend.ip. Is there any way in VCL 4.1 to get the IP Address of the backend used? If not, it means I'll need to use VCL 4.0. Is there any reason for me to prefer 4.1 over 4.0? Cheers! -Hugues -------------- next part -------------- An HTML attachment was scrubbed... URL: From contact at awa.solutions Fri Mar 30 15:11:35 2018 From: contact at awa.solutions (AWA SOLUTIONS) Date: Fri, 30 Mar 2018 17:11:35 +0200 Subject: Docs : Only handle actual PURGE HTTP methods, everything else is discarded Message-ID: <001a01d3c839$64cc5480$2e64fd80$@solutions> Hi there, While reading the docs, I found this in https://www.varnish-software.com/wiki/content/tutorials/varnish/sample_vclTe mplate.html : HANDLING HTTP PURGE sub vcl_purge { # Only handle actual PURGE HTTP methods, everything else is discarded if (req.method != "PURGE") { # restart request set req.http.X-Purge = "Yes"; return(restart); } } But in mattiasgeniar/varnish-4.0-configuration-templates https://github.com/mattiasgeniar/varnish-4.0-configuration-templates/blob/ma ster/default.vcl This is just the opposite logic : sub vcl_purge { # Only handle actual PURGE HTTP methods, everything else is discarded if (req.method == "PURGE") { # restart request set req.http.X-Purge = "Yes"; return (restart); } } So I assume the latter, wiki page, has a typo ?! Last, in https://book.varnish-software.com/4.0/chapters/Cache_Invalidation.html , after similar vcl_recv conditions than above to return purge, vcl_purge subroutine sets req.method instead of req.http.X-Purge : sub vcl_purge { set req.method = "GET"; return (restart); } Sounds logical to change req.method to avoid infinite loops, so why isn't it the same in above examples ? Thank you ! -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Fri Mar 30 15:42:23 2018 From: geoff at uplex.de (Geoff Simmons) Date: Fri, 30 Mar 2018 17:42:23 +0200 Subject: VCL 4.1 and beresp.backend.ip In-Reply-To: References: Message-ID: <039739d6-df08-1034-c14b-bf70a0320262@uplex.de> On 03/29/2018 08:24 PM, Hugues Alary wrote: > > VCL 4.1 retired beresp.backend.ip. > > Is there any way in VCL 4.1 to get the IP Address of the backend used? No. There have been some thoughts about adding something for "get backend info" in a future version, which would expose the IP address among other things, but that's not available for now. > If not, it means I'll need to use VCL 4.0. > > Is there any reason for me to prefer 4.1 over 4.0? There may or may not be, depending on your needs. You'll need VCL 4.1 if: - You're using Unix domain sockets - You have more than one listen address, and you want to use local.socket and/or local.endpoint in VCL to tell them apart. - You want to use sess.xid. - You want to selectively disable ESIs in client responses, and prefer to use resp.do_esi rather than req.esi for that. The *.proto variables are read-only in 4.1; if you want to write to them, you have to stay with 4.0. beresp.storage_hint is still around in 4.0, not in 4.1, but you should be using beresp.storage anyway. AFAIK that's all of it. HTH, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From hugues at betabrand.com Fri Mar 30 19:12:42 2018 From: hugues at betabrand.com (Hugues Alary) Date: Fri, 30 Mar 2018 12:12:42 -0700 Subject: VCL 4.1 and beresp.backend.ip In-Reply-To: <039739d6-df08-1034-c14b-bf70a0320262@uplex.de> References: <039739d6-df08-1034-c14b-bf70a0320262@uplex.de> Message-ID: Great, thank you Geoff! On Fri, Mar 30, 2018 at 8:42 AM, Geoff Simmons wrote: > On 03/29/2018 08:24 PM, Hugues Alary wrote: > > > > VCL 4.1 retired beresp.backend.ip. > > > > Is there any way in VCL 4.1 to get the IP Address of the backend used? > > No. There have been some thoughts about adding something for "get > backend info" in a future version, which would expose the IP address > among other things, but that's not available for now. > > > If not, it means I'll need to use VCL 4.0. > > > > Is there any reason for me to prefer 4.1 over 4.0? > > There may or may not be, depending on your needs. > > You'll need VCL 4.1 if: > > - You're using Unix domain sockets > - You have more than one listen address, and you want to use > local.socket and/or local.endpoint in VCL to tell them apart. > - You want to use sess.xid. > - You want to selectively disable ESIs in client responses, and > prefer to use resp.do_esi rather than req.esi for that. > > The *.proto variables are read-only in 4.1; if you want to write to > them, you have to stay with 4.0. > > beresp.storage_hint is still around in 4.0, not in 4.1, but you should > be using beresp.storage anyway. > > AFAIK that's all of it. > > > HTH, > Geoff > -- > ** * * UPLEX - Nils Goroll Systemoptimierung > > Scheffelstra?e 32 > 22301 Hamburg > > Tel +49 40 2880 5731 > Mob +49 176 636 90917 > Fax +49 40 42949753 > > http://uplex.de > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ciapnz at gmail.com Sat Mar 31 08:21:16 2018 From: ciapnz at gmail.com (Danila Vershinin) Date: Sat, 31 Mar 2018 11:21:16 +0300 Subject: Varnish error for failed backend Message-ID: <1314224D-6234-473D-94D2-D91F43524FA5@gmail.com> Hi, I have configured Varnish backend probes + grace mode. Works great. One thing I?d like to address with this setup (this is Magento website, but this ?issue? applies to other website types) is more relevant error pages. If a constant PHP error occurs in the backend and there is no cache - we see Backend fetch failed. In some cases (e.g. when adding a store view to Magento website, and not yet configured CMS page, Magento returns 404 which of course fails the probes) - we see Backend fetch failed as well. Finally, if Magento sends too long header and we haven?t upped some Varnish settings, etc. (misconfiguration issue) - we see Backend fetch failed. While I?m working with Varnish setups quite often, it always takes time to explain to the clients that the Varnish is not to blame and ideally there would be a way to deliver different Varnish error page for these 3 cases: * 500 error-ed backend * 404-ed backend * actual problem talking to the backend (HTTP etc.) Is there any way to achieve this with grace mode on? I assume this requires at very least being able to get information from last probe... Best Regards, Danila -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From geoff at uplex.de Sat Mar 31 11:25:31 2018 From: geoff at uplex.de (Geoff Simmons) Date: Sat, 31 Mar 2018 13:25:31 +0200 Subject: Varnish error for failed backend In-Reply-To: <1314224D-6234-473D-94D2-D91F43524FA5@gmail.com> References: <1314224D-6234-473D-94D2-D91F43524FA5@gmail.com> Message-ID: <156d1c45-c498-2af2-a5c9-5d8da802b19a@uplex.de> On 03/31/2018 10:21 AM, Danila Vershinin wrote: > > If a constant PHP error occurs in the backend and there is no cache - > we see Backend fetch failed. [...] > ... it always takes time to explain to the clients that the Varnish > is not to blame ... Welcome to Varnish administration! Such is the life. When the backend fetch fails, always look to the FetchError entry in the backend log, which diagnoses the problem. It helps with those conversations. > ... and ideally there would be a way to deliver different Varnish > error page for these 3 cases:> > * 500 error-ed backend > * 404-ed backend > * actual problem talking to the backend (HTTP etc.) Varnish can generate the response itself with vcl_synth -- when you see the Guru Meditation and "Backend fetch failed", you're seeing the buitlin.vcl version of vcl_synth. vcl_synth is called automatically for response codes 500 & 503, for 404 or anything else, your VCL has to direct to there: sub vcl_deliver { if (resp.status == 404) { return (synth(404)); } } So you could have something like this: sub vcl_synth { if (resp.status == 500) { synthetic("500 error-ed backend"); } elsif (resp.status == 404) { synthetic("404-ed backend"); } elsif (resp.reason == "Backend fetch failed") { synthetic("actual problem talking to the backend"); } } Since I believe about Varnish 5.0 or so you can use also this syntax to generate the synthetic response (synthetic() works as well): set resp.body = "mumble"; Of course you'll probably want to have HTML markup in the generated response -- look to the Guru Meditation in builtin.vcl's vcl_synth for an example. HTH, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From ciapnz at gmail.com Sat Mar 31 12:36:41 2018 From: ciapnz at gmail.com (Danila Vershinin) Date: Sat, 31 Mar 2018 15:36:41 +0300 Subject: Varnish error for failed backend In-Reply-To: <156d1c45-c498-2af2-a5c9-5d8da802b19a@uplex.de> References: <1314224D-6234-473D-94D2-D91F43524FA5@gmail.com> <156d1c45-c498-2af2-a5c9-5d8da802b19a@uplex.de> Message-ID: <9ED34EEC-96BD-491C-84B6-3A078C530785@gmail.com> Hi Geoff, Thank you. I understand about vcl_synth being used when getting 500 & 503 better :) However, I?m also interested how can we distinguish backend errors while using grace mode and failed probe. Suppose that you have configured grace mode and a health check against homepage. And homepage fails with 500. The backend is marked as sick, then synthetic response (guru meditation) is delivered from vcl_backend_error with resp.status pre-filled to 503. So there seems to be no way to do the same conditional error display in that case? Best Regards, Danila > On 31 Mar 2018, at 14:25, Geoff Simmons wrote: > > On 03/31/2018 10:21 AM, Danila Vershinin wrote: >> >> If a constant PHP error occurs in the backend and there is no cache - >> we see Backend fetch failed. > [...] > >> ... it always takes time to explain to the clients that the Varnish >> is not to blame ... > Welcome to Varnish administration! Such is the life. > > When the backend fetch fails, always look to the FetchError entry in the > backend log, which diagnoses the problem. It helps with those conversations. > >> ... and ideally there would be a way to deliver different Varnish >> error page for these 3 cases:> >> * 500 error-ed backend >> * 404-ed backend >> * actual problem talking to the backend (HTTP etc.) > > Varnish can generate the response itself with vcl_synth -- when you see > the Guru Meditation and "Backend fetch failed", you're seeing the > buitlin.vcl version of vcl_synth. > > vcl_synth is called automatically for response codes 500 & 503, for 404 > or anything else, your VCL has to direct to there: > > sub vcl_deliver { > if (resp.status == 404) { > return (synth(404)); > } > } > > So you could have something like this: > > sub vcl_synth { > if (resp.status == 500) { > synthetic("500 error-ed backend"); > } > elsif (resp.status == 404) { > synthetic("404-ed backend"); > } > elsif (resp.reason == "Backend fetch failed") { > synthetic("actual problem talking to the backend"); > } > } > > Since I believe about Varnish 5.0 or so you can use also this syntax to > generate the synthetic response (synthetic() works as well): > > set resp.body = "mumble"; > > Of course you'll probably want to have HTML markup in the generated > response -- look to the Guru Meditation in builtin.vcl's vcl_synth for > an example. > > > HTH, > Geoff > -- > ** * * UPLEX - Nils Goroll Systemoptimierung > > Scheffelstra?e 32 > 22301 Hamburg > > Tel +49 40 2880 5731 > Mob +49 176 636 90917 > Fax +49 40 42949753 > > http://uplex.de > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From geoff at uplex.de Sat Mar 31 14:10:15 2018 From: geoff at uplex.de (Geoff Simmons) Date: Sat, 31 Mar 2018 16:10:15 +0200 Subject: Varnish error for failed backend In-Reply-To: <9ED34EEC-96BD-491C-84B6-3A078C530785@gmail.com> References: <1314224D-6234-473D-94D2-D91F43524FA5@gmail.com> <156d1c45-c498-2af2-a5c9-5d8da802b19a@uplex.de> <9ED34EEC-96BD-491C-84B6-3A078C530785@gmail.com> Message-ID: On 03/31/2018 02:36 PM, Danila Vershinin wrote: > > However, I?m also interested how can we distinguish backend errors > while using grace mode and failed probe. > > Suppose that you have configured grace mode and a health check > against homepage. And homepage fails with 500. > The backend is marked as sick, then synthetic response (guru > meditation) is delivered from vcl_backend_error with resp.status > pre-filled to 503. > So there seems to be no way to do the same conditional error display > in that case? Oh I'm sorry, I think I may have misunderstood your question. I assume in this scenario that Varnish never got a valid response that could be cached -- otherwise, the cached object could have been delivered while it's still in grace. As long as the health probe is failing, Varnish doesn't attempt any fetch in response to a client request at all. Varnish makes a note of the backend sick state, and then any request meant for that backend goes right to the 503 response and vcl_backend_error, logging FetchError=="no backend connection". There's no attempt to re-use or initiate a network connection, let alone forward a request, it just becomes 503 and that's it, until the health probe goes back to healthy. So the 503 you get in that situation is only indirectly related to the 500 health probe response -- the health probes see the 500, running in their own threads and independent of any request/response traffic. That means that you can't establish a relation between the 503 fetch failure and the 500 health probe response in VCL. What you can do is look at std.healthy(backend), something like: import std; if (!std.healthy(mybackend)) { set resp.body = "..." + "... because the backend is sick ..."; } And of course there's the log -- FetchError=="no backend connection" and the Backend_health entries. Hope that helped a little better this time, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: