From navneet.kashyap at webners.com Fri Nov 3 07:30:09 2017 From: navneet.kashyap at webners.com (Navneet Kashyap) Date: Fri, 3 Nov 2017 13:00:09 +0530 Subject: using varnish4 for HTTPS wordpress site Message-ID: Hi I was using varnish-cahe (open source) for my wordpress website, it was running well when its on HTTP only, but when i turn it to HTTPS its giving me error message when testing status using plugin in wordpress i.e. *Varnish HTTP Purge* Error: This request cannot be performed: cURL error 60: Issuer certificate is invalid. please check the screenshot also. [image: Inline image 1] Currently the flow is like this: client--> HTTPS request--> AWS load balancer --> Varnish--> apache2. we are using self-signed certs for backend authentication settings in AWS-load balancer, and using AMAZON provided CA-certs (using certificate manager) Note: Is this possible in varnish-cache software (open-Source) or else we have to buy varnish-cache plus software (Paid Version). or we have to busy CA-certs for that domain. ? kindly guide us. Thanks and Regards Navneet Kashyap Sr. System Administrator - Webner Solutions Pvt. Ltd. Web - www.webnersolutions.com [image: Zoho Development, Salesforce Development, Web and Mobile App Development] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 71209 bytes Desc: not available URL: From luca.gervasi at gmail.com Fri Nov 3 08:18:03 2017 From: luca.gervasi at gmail.com (Luca Gervasi) Date: Fri, 03 Nov 2017 08:18:03 +0000 Subject: using varnish4 for HTTPS wordpress site In-Reply-To: References: Message-ID: Hi, this is totally unrelated to varnish. If you want to achieve purging, you can safely stay http and purge locally (if you are using a load balancer in front of your delivery, you either have just one varnish - hence purge locally - or your purges are inconsistently distributed among all your caches). Bye On Fri, 3 Nov 2017 at 08:31 Navneet Kashyap wrote: > Hi > > I was using varnish-cahe (open source) for my wordpress website, it was > running well when its on HTTP only, but when i turn it to HTTPS its giving > me error message when testing status using plugin in wordpress i.e. *Varnish > HTTP Purge* > Error: This request cannot be performed: cURL error 60: Issuer certificate > is invalid. > > please check the screenshot also. > [image: Inline image 1] > > Currently the flow is like this: > client--> HTTPS request--> AWS load balancer --> Varnish--> apache2. > > we are using self-signed certs for backend authentication settings in > AWS-load balancer, and using AMAZON provided CA-certs (using certificate > manager) > > Note: Is this possible in varnish-cache software (open-Source) or else we > have to buy varnish-cache plus software (Paid Version). or we have to busy > CA-certs for that domain. ? > > kindly guide us. > > > Thanks and Regards > > Navneet Kashyap > Sr. System Administrator - Webner Solutions Pvt. Ltd. > Web - www.webnersolutions.com > [image: Zoho Development, Salesforce Development, Web and Mobile App > Development] > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 71209 bytes Desc: not available URL: From guillaume at varnish-software.com Fri Nov 3 08:28:08 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Fri, 3 Nov 2017 09:28:08 +0100 Subject: using varnish4 for HTTPS wordpress site In-Reply-To: References: Message-ID: To answer the HTTPS question, client-side HTTPS is easily done using hitch ( hitch-tls.org), for the backend-side, you would need Varnish Plus. But you are saying Amazon provides the certificates, so my guess is that the LB does the termination arrive unencrypted to Varnish. Do the curl error, does the host used matches the certificate? -- Guillaume Quintard On Fri, Nov 3, 2017 at 9:18 AM, Luca Gervasi wrote: > Hi, > this is totally unrelated to varnish. If you want to achieve purging, you > can safely stay http and purge locally (if you are using a load balancer in > front of your delivery, you either have just one varnish - hence purge > locally - or your purges are inconsistently distributed among all your > caches). > > Bye > > On Fri, 3 Nov 2017 at 08:31 Navneet Kashyap > wrote: > >> Hi >> >> I was using varnish-cahe (open source) for my wordpress website, it was >> running well when its on HTTP only, but when i turn it to HTTPS its giving >> me error message when testing status using plugin in wordpress i.e. *Varnish >> HTTP Purge* >> Error: This request cannot be performed: cURL error 60: Issuer >> certificate is invalid. >> >> please check the screenshot also. >> [image: Inline image 1] >> >> Currently the flow is like this: >> client--> HTTPS request--> AWS load balancer --> Varnish--> apache2. >> >> we are using self-signed certs for backend authentication settings in >> AWS-load balancer, and using AMAZON provided CA-certs (using certificate >> manager) >> >> Note: Is this possible in varnish-cache software (open-Source) or else we >> have to buy varnish-cache plus software (Paid Version). or we have to busy >> CA-certs for that domain. ? >> >> kindly guide us. >> >> >> Thanks and Regards >> >> Navneet Kashyap >> Sr. System Administrator - Webner Solutions Pvt. Ltd. >> Web - www.webnersolutions.com >> [image: Zoho Development, Salesforce Development, Web and Mobile App >> Development] >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 71209 bytes Desc: not available URL: From jdh132 at psu.edu Fri Nov 3 12:03:08 2017 From: jdh132 at psu.edu (Jason Heffner) Date: Fri, 3 Nov 2017 08:03:08 -0400 Subject: using varnish4 for HTTPS wordpress site In-Reply-To: References: Message-ID: Hi, I had a similar issue using the wordpress-varnish plugin and our large multi-site install. I had to completely re-write the code to use curl as opposed to fsockopen, but for the certificate issue I used the curl option to ignore certificate errors. You may be able to simply add a line to the plugin code. We were able to do this since we had security in place that wouldn?t allow BAN from any other systems. Jason > On Nov 3, 2017, at 4:28 AM, Guillaume Quintard wrote: > > To answer the HTTPS question, client-side HTTPS is easily done using hitch (hitch-tls.org ), for the backend-side, you would need Varnish Plus. > > But you are saying Amazon provides the certificates, so my guess is that the LB does the termination arrive unencrypted to Varnish. > > Do the curl error, does the host used matches the certificate? > > -- > Guillaume Quintard > > On Fri, Nov 3, 2017 at 9:18 AM, Luca Gervasi > wrote: > Hi, > this is totally unrelated to varnish. If you want to achieve purging, you can safely stay http and purge locally (if you are using a load balancer in front of your delivery, you either have just one varnish - hence purge locally - or your purges are inconsistently distributed among all your caches). > > Bye > > On Fri, 3 Nov 2017 at 08:31 Navneet Kashyap > wrote: > Hi > > I was using varnish-cahe (open source) for my wordpress website, it was running well when its on HTTP only, but when i turn it to HTTPS its giving me error message when testing status using plugin in wordpress i.e. > Varnish HTTP Purge > Error: This request cannot be performed: cURL error 60: Issuer certificate is invalid. > > please check the screenshot also. > > > Currently the flow is like this: > client--> HTTPS request--> AWS load balancer --> Varnish--> apache2. > > we are using self-signed certs for backend authentication settings in AWS-load balancer, and using AMAZON provided CA-certs (using certificate manager) > > Note: Is this possible in varnish-cache software (open-Source) or else we have to buy varnish-cache plus software (Paid Version). or we have to busy CA-certs for that domain. ? > > kindly guide us. > > > Thanks and Regards > > Navneet Kashyap > Sr. System Administrator - Webner Solutions Pvt. Ltd. > Web - www.webnersolutions.com > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From cservin-varnish at cromagnon.com Fri Nov 3 15:42:16 2017 From: cservin-varnish at cromagnon.com (Craig Servin) Date: Fri, 03 Nov 2017 10:42:16 -0500 Subject: Any way to know if you are in a backgound fetch in Varnish 4.1 Message-ID: <5a086e9e5fafa1e334eb287ad9e85f59@www.cromagnon.com> Hi All, Is there any way for me to know if I am in a background fetch while in vcl_backend_response? I thought about setting a header in vcl_hit, but I thought there might be a less hacky way to do it. Thanks, Craig From guillaume at varnish-software.com Fri Nov 3 15:45:13 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Fri, 3 Nov 2017 16:45:13 +0100 Subject: Any way to know if you are in a backgound fetch in Varnish 4.1 In-Reply-To: <5a086e9e5fafa1e334eb287ad9e85f59@www.cromagnon.com> References: <5a086e9e5fafa1e334eb287ad9e85f59@www.cromagnon.com> Message-ID: Hi, No, bereq.is_bgfetch was added in 5.2. Cheers, -- Guillaume Quintard On Fri, Nov 3, 2017 at 4:42 PM, Craig Servin wrote: > Hi All, > > Is there any way for me to know if I am in a background fetch while in > vcl_backend_response? > > I thought about setting a header in vcl_hit, but I thought there might be > a less hacky way to do it. > > Thanks, > > Craig > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagged at gmail.com Sun Nov 5 08:12:35 2017 From: lagged at gmail.com (Andrei) Date: Sun, 5 Nov 2017 10:12:35 +0200 Subject: backend timeouts/503s vs grace cache Message-ID: Hello everyone, One of the backends we have configured, runs through an SSH tunnel which occasionally gets restarted. When the tunnel is restarted, Varnish is returning a 503 since it can't reach the backend for pages which would normally be cached (we force cache on the front page of the related site). I believe our grace implementation might be incorrect, as we would expect a grace period cache return instead of 503. Our grace ttl is set to 21600 seconds based on a global variable: sub vcl_backend_response { set beresp.grace = std.duration(variable.global_get("ttl_grace") + "s", 6h); } Our grace implementation in sub vcl_hit is: sub vcl_hit { # We have no fresh fish. Lets look at the stale ones. if (std.healthy(req.backend_hint)) { # Backend is healthy. Limit age to 10s. if (obj.ttl + 10s > 0s) { #set req.http.grace = "normal(limited)"; std.log("OKHITDELIVER: obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace); return (deliver); } else { # No candidate for grace. Fetch a fresh object. std.log("No candidate for grace. Fetch a fresh object. obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace); return(miss); } } else { # backend is sick - use full grace if (obj.ttl + obj.grace > 0s) { #set req.http.grace = "full"; std.log("SICK DELIVERY: obj.hits: " + obj.hits + " obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace); return (deliver); } else { # no graced object. std.log("No graced object. obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace); return (miss); } } # fetch & deliver once we get the result return (miss); # Dead code, keep as a safeguard } Occasionally we see: - VCL_Log No candidate for grace. Fetch a fresh object. obj.ttl:-1369.659 obj.keep: 0.000 obj.grace: 21600.000 For the most part, it's: - VCL_Log OKHITDELIVER: obj.ttl:26.872 obj.keep: 0.000 obj.grace: 21600.000 Are we setting the grace ttl too low perhaps? -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagged at gmail.com Sun Nov 5 08:18:28 2017 From: lagged at gmail.com (Andrei) Date: Sun, 5 Nov 2017 10:18:28 +0200 Subject: backend timeouts/503s vs grace cache In-Reply-To: References: Message-ID: I wanted to add that I don't believe our grace TTL is too low, as the site has active monitoring requesting the front page every minute, so this should keep cache primed. Problem is the 3rd party monitoring service intermittently reports downtime errors due to the 503 replies sent by Varnish while the ssh tunnel to the bakend is restarted On Nov 5, 2017 10:12, "Andrei" wrote: Hello everyone, One of the backends we have configured, runs through an SSH tunnel which occasionally gets restarted. When the tunnel is restarted, Varnish is returning a 503 since it can't reach the backend for pages which would normally be cached (we force cache on the front page of the related site). I believe our grace implementation might be incorrect, as we would expect a grace period cache return instead of 503. Our grace ttl is set to 21600 seconds based on a global variable: sub vcl_backend_response { set beresp.grace = std.duration(variable.global_get("ttl_grace") + "s", 6h); } Our grace implementation in sub vcl_hit is: sub vcl_hit { # We have no fresh fish. Lets look at the stale ones. if (std.healthy(req.backend_hint)) { # Backend is healthy. Limit age to 10s. if (obj.ttl + 10s > 0s) { #set req.http.grace = "normal(limited)"; std.log("OKHITDELIVER: obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace); return (deliver); } else { # No candidate for grace. Fetch a fresh object. std.log("No candidate for grace. Fetch a fresh object. obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace); return(miss); } } else { # backend is sick - use full grace if (obj.ttl + obj.grace > 0s) { #set req.http.grace = "full"; std.log("SICK DELIVERY: obj.hits: " + obj.hits + " obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace); return (deliver); } else { # no graced object. std.log("No graced object. obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace); return (miss); } } # fetch & deliver once we get the result return (miss); # Dead code, keep as a safeguard } Occasionally we see: - VCL_Log No candidate for grace. Fetch a fresh object. obj.ttl:-1369.659 obj.keep: 0.000 obj.grace: 21600.000 For the most part, it's: - VCL_Log OKHITDELIVER: obj.ttl:26.872 obj.keep: 0.000 obj.grace: 21600.000 Are we setting the grace ttl too low perhaps? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hermunn at varnish-software.com Sun Nov 5 22:50:00 2017 From: hermunn at varnish-software.com (=?UTF-8?Q?P=C3=A5l_Hermunn_Johansen?=) Date: Sun, 5 Nov 2017 23:50:00 +0100 Subject: Any way to know if you are in a backgound fetch in Varnish 4.1 In-Reply-To: References: <5a086e9e5fafa1e334eb287ad9e85f59@www.cromagnon.com> Message-ID: Actually, this was back ported in https://github.com/varnishcache/varnish-cache/commit/bf167ec4cec2315 , and it will be part of 4.1.9 when it is released. P?l 2017-11-03 16:45 GMT+01:00 Guillaume Quintard : > Hi, > > No, bereq.is_bgfetch was added in 5.2. > > Cheers, > > -- > Guillaume Quintard > > On Fri, Nov 3, 2017 at 4:42 PM, Craig Servin > wrote: >> >> Hi All, >> >> Is there any way for me to know if I am in a background fetch while in >> vcl_backend_response? >> >> I thought about setting a header in vcl_hit, but I thought there might be >> a less hacky way to do it. >> >> Thanks, >> >> Craig >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From slink at schokola.de Mon Nov 6 09:13:08 2017 From: slink at schokola.de (Nils Goroll) Date: Mon, 6 Nov 2017 10:13:08 +0100 Subject: VCDK In-Reply-To: References: Message-ID: Hi, I've based my latest vmod project (soon to be released) on https://github.com/dridi/vcdk and must say that from my first impression I am super happy with it. VCDK makes the initial steps to creating a vmod so much easier - and not just that, it also supports varnish tools (which I did not use yet). For those interested, my comments are https://github.com/Dridi/vcdk/issues/1 I can really recommend VCDK! Thank you, Dridi! From cservin-varnish at cromagnon.com Mon Nov 6 16:55:40 2017 From: cservin-varnish at cromagnon.com (Craig Servin) Date: Mon, 06 Nov 2017 10:55:40 -0600 Subject: Any way to know if you are in a backgound fetch in Varnish 4.1 In-Reply-To: References: <5a086e9e5fafa1e334eb287ad9e85f59@www.cromagnon.com> Message-ID: <7ede88f12d37d88cad089cdc1bd40a2d@www.cromagnon.com> Thanks for the info, In the interim I'm just doing this: sub vcl_hit { if (obj.ttl < 0s && ( obj.ttl + obj.grace > 0s ) ) { set req.http.X-IN-GRACE = "true"; } } I realize there is a race condition between my vcl and the builtin.vcl fallthrough, but it shouldn't affect my use case. Thanks again, Craig On 2017-11-05 16:50, P?l Hermunn Johansen wrote: > Actually, this was back ported in > https://github.com/varnishcache/varnish-cache/commit/bf167ec4cec2315 , > and it will be part of 4.1.9 when it is released. > > P?l > > 2017-11-03 16:45 GMT+01:00 Guillaume Quintard > : >> Hi, >> >> No, bereq.is_bgfetch was added in 5.2. >> >> Cheers, >> >> -- >> Guillaume Quintard >> >> On Fri, Nov 3, 2017 at 4:42 PM, Craig Servin >> >> wrote: >>> >>> Hi All, >>> >>> Is there any way for me to know if I am in a background fetch while >>> in >>> vcl_backend_response? >>> >>> I thought about setting a header in vcl_hit, but I thought there >>> might be >>> a less hacky way to do it. >>> >>> Thanks, >>> >>> Craig >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From jason at codeassassin.com Tue Nov 7 05:22:22 2017 From: jason at codeassassin.com (Jason Stangroome) Date: Tue, 7 Nov 2017 16:22:22 +1100 Subject: Understanding cache utilisation Message-ID: Hello, For at least one site that I am using Varnish Cache for, the allocated cache size is fully utilised and objects are now being nuked/evicted from the cache. Before I blindly supply more resources to provide a larger cache, I'd like to understand if the cache space is being used optimally. For example, of the objects that are expiring naturally, were they ever used to serve a cache hit? If not, it was probably wasteful to store them in the cache in the first place. Of the objects that are being nuked/evicted, were they close to their natural expiration time anyway? Were they ever used to serve a cache hit? Of the objects that have been nuked, are they requested shortly after, suggesting it may have been worth having a larger cache in order to retain them a bit longer? I did not see an easy way to answer these questions with the standard Varnish toolset - am I missing something? I looked through the code paths in Varnish 5.1 responsible for expiration and nuking and it seems the object hit count and ttl is available in that context - would it be worth writing a patch to log these fields to the VSL? How do others ensure their allocated cache size is well utilised before growing the cache? Regards, Jason From dridi at varni.sh Tue Nov 7 06:55:59 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 7 Nov 2017 07:55:59 +0100 Subject: Understanding cache utilisation In-Reply-To: References: Message-ID: > Of the objects that are being nuked/evicted, were they close to their > natural expiration time anyway? Were they ever used to serve a cache > hit? > > Of the objects that have been nuked, are they requested shortly after, > suggesting it may have been worth having a larger cache in order to > retain them a bit longer? > > I did not see an easy way to answer these questions with the standard > Varnish toolset - am I missing something? I think you are correct. The grace period could also be responsible for keeping expired objects around and making them good candidates for LRU eviction. > I looked through the code paths in Varnish 5.1 responsible for > expiration and nuking and it seems the object hit count and ttl is > available in that context - would it be worth writing a patch to log > these fields to the VSL? You could add a `t=%f` to the `ExpKill LRU` log records, and introduce something like a `h=%d` for hits and add it to the LRU and EXP_* variants of the ExpKill record. See `man vsl` for Expkill. I'm not sure such a patch would be accepted as we usually are cautious with the VSL throughput but at least I would gladly review it (merging's not my call in this area). Dridi From guillaume at varnish-software.com Tue Nov 7 07:09:06 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Tue, 7 Nov 2017 08:09:06 +0100 Subject: Understanding cache utilisation In-Reply-To: References: Message-ID: Hello Jason, Just an idea, before patching: log obj.hits and obj.age to get an idea of what you are delivering. -- Guillaume Quintard On Nov 7, 2017 07:23, "Jason Stangroome" wrote: > Hello, > > For at least one site that I am using Varnish Cache for, the allocated > cache size is fully utilised and objects are now being nuked/evicted > from the cache. Before I blindly supply more resources to provide a > larger cache, I'd like to understand if the cache space is being used > optimally. > > For example, of the objects that are expiring naturally, were they > ever used to serve a cache hit? If not, it was probably wasteful to > store them in the cache in the first place. > > Of the objects that are being nuked/evicted, were they close to their > natural expiration time anyway? Were they ever used to serve a cache > hit? > > Of the objects that have been nuked, are they requested shortly after, > suggesting it may have been worth having a larger cache in order to > retain them a bit longer? > > I did not see an easy way to answer these questions with the standard > Varnish toolset - am I missing something? > > I looked through the code paths in Varnish 5.1 responsible for > expiration and nuking and it seems the object hit count and ttl is > available in that context - would it be worth writing a patch to log > these fields to the VSL? > > How do others ensure their allocated cache size is well utilised > before growing the cache? > > Regards, > > Jason > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Tue Nov 7 08:04:43 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 7 Nov 2017 09:04:43 +0100 Subject: Understanding cache utilisation In-Reply-To: References: Message-ID: On Tue, Nov 7, 2017 at 8:09 AM, Guillaume Quintard wrote: > Hello Jason, > > Just an idea, before patching: log obj.hits and obj.age to get an idea of > what you are delivering. Indeed, if you keep track of the relevant bits (like varnishncsa only keeps track of what's needed by its format) you could cheaply (for a given amount of "cheap") correlate this with LRU evictions. I'm in the general opinion of making the logs useful, that's why I submitted a change in 5.2 to get at least the VXID on Hit* records, not just the fact that we hit something. Dridi From info+varnish at shee.org Sat Nov 11 20:48:24 2017 From: info+varnish at shee.org (info+varnish at shee.org) Date: Sat, 11 Nov 2017 21:48:24 +0100 Subject: backend web node keep-alive option Message-ID: <76A6807F-2821-4D0E-9E4D-C00CA22BA947@shee.org> I was observing a behavior of a PHP script on one backend server that got executed/requested twice (but not by the client/browser). The plain script just processes data and only outputs a response until its done. The observed behavior is; when first_byte_timeout passes but the script is still processing data, the varnish node answers with an 503 and the script gets listed twice in the webserver process list. So, either it gets requested on this event again or the TCP connection close triggers an new execution. In any case its a bug and because of missing resources to dive into a more deeper analyses, I just disabled the enabled keep-alive option of the backend webserver (apache/httpd). The results: it helped to avoid a further execution of the script. The question is: What is the best practice to configure the backend webservers? Do you keep the keep-alive option enabled or not? I'd appreciate hearing your suggestions. Leon From guillaume at varnish-software.com Sun Nov 12 14:42:15 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Sun, 12 Nov 2017 15:42:15 +0100 Subject: backend web node keep-alive option In-Reply-To: <76A6807F-2821-4D0E-9E4D-C00CA22BA947@shee.org> References: <76A6807F-2821-4D0E-9E4D-C00CA22BA947@shee.org> Message-ID: Hi Leon, Why not just up the timeout value? Usually, you want to keep the keep-alive as it's much more efficient. Cheers, -- Guillaume Quintard On Sat, Nov 11, 2017 at 9:48 PM, wrote: > I was observing a behavior of a PHP script on one backend server that got > executed/requested > twice (but not by the client/browser). > > The plain script just processes data and only outputs a response until its > done. The observed > behavior is; when first_byte_timeout passes but the script is still > processing data, the varnish > node answers with an 503 and the script gets listed twice in the webserver > process list. So, either > it gets requested on this event again or the TCP connection close triggers > an new execution. In any > case its a bug and because of missing resources to dive into a more deeper > analyses, I just disabled > the enabled keep-alive option of the backend webserver (apache/httpd). The > results: it helped to avoid > a further execution of the script. > > The question is: What is the best practice to configure the backend > webservers? Do you keep the > keep-alive option enabled or not? > > I'd appreciate hearing your suggestions. > > Leon > > > > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Mon Nov 13 07:11:25 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 13 Nov 2017 08:11:25 +0100 Subject: backend web node keep-alive option In-Reply-To: <76A6807F-2821-4D0E-9E4D-C00CA22BA947@shee.org> References: <76A6807F-2821-4D0E-9E4D-C00CA22BA947@shee.org> Message-ID: On Sat, Nov 11, 2017 at 9:48 PM, wrote: > I was observing a behavior of a PHP script on one backend server that got executed/requested > twice (but not by the client/browser). > > The plain script just processes data and only outputs a response until its done. The observed > behavior is; when first_byte_timeout passes but the script is still processing data, the varnish > node answers with an 503 and the script gets listed twice in the webserver process list. So, either > it gets requested on this event again or the TCP connection close triggers an new execution. In any > case its a bug and because of missing resources to dive into a more deeper analyses, I just disabled > the enabled keep-alive option of the backend webserver (apache/httpd). The results: it helped to avoid > a further execution of the script. > > The question is: What is the best practice to configure the backend webservers? Do you keep the > keep-alive option enabled or not? > > I'd appreciate hearing your suggestions. I think you're getting a retry from varnishd, independent from VCL's return(retry) which only happens on GET/HEAD requests that are supposed to be side-effect-free and safely retry-able. On top of that you might be hitting a bug where Varnish retries more than once. Seeing some log would help confirm this hypothesis. https://github.com/varnishcache/varnish-cache/pull/2135 Dridi From raph at futomaki.net Tue Nov 14 22:24:48 2017 From: raph at futomaki.net (Raphael Mazelier) Date: Tue, 14 Nov 2017 23:24:48 +0100 Subject: Varnish nighmare after upgrading : need help Message-ID: <620367f1-0247-b457-f65c-08fc898ea6b3@futomaki.net> Hello list, First of all despite my mail subject I really appreciate varnish. We use it a lot at work (hundred of instances) with success and unfortunately some pain these time. TLDR; upgrading from varnish 2 to varnish 4 and 5 on one of our infrastructure brought us some serious trouble and instability on this platform. And we are a bit desperate/frustrated Long story. A bit of context : This a very complex platform serving an IPTV service with some traffic. (8k req/s in peak, even more when it work well). It is compose of a two stage reverse proxy cache (3 x 2 varnish for stage 1), 2 varnish for stage 2, (so 8 in total) and a lot of different backends (php applications, nodejs apps, remote backends *sigh*, and even pipe one). This a big historical spaghetti app. We plan to rebuild it from scratch in 2018. The first stage varnish are separate in two pool handling different topology of clients. A lot of the logic is in varnish/vcl itself, lot of url rewrite, lot of manipulation of headers, choice of a backend, and even ESI processing... The VCL of the stage 1 varnish are almost 3000 lines long. But for now we have to leave/deal with it. History of the problem : At the beginning all varnish are in 2.x version. Things works almost well. This summer we need to upgrade the varnish version to handle very long header (a product requirement). So after a short battle porting our vcl to vcl4.0 we start using varnish 4. Shortly after thing begun to goes very bad. The first issue we hit, is a memory exhaustion on both stage, and oom-killer... We test a lot of things, and in the battle we upgrade to varnish5. We fix it, resizing the pool, and using now file backend (from memory before). Memory is now stable (we have large pool, 32G, and strange thing, we never have object being nuke, which it good or bad it depend). We have also fix a lot of things in our vcl. The problem we fight against now is only on the stage1 varnish, and specifically on one pool (the busiest one). When everything goes well the average cpu usage is 30%, memory stabilize around 12G, hit cache is around 0.85. Problem happen randomly (not everyday) but during our peaks. The cpu increase fasly to reach 350% (4 core) and load > 3/ When the problem is here varnish still deliver requests (we didn't see dropped or reject connections) but our application begin to lost user, including a big lot of business. I suspect this is because timeout are very aggressive on the client side and varnish should answer slowly -first question : how see response time of request of the varnish server ?. (varnishnsca something ?) I also suspect some kind of request queuing, also stracing varnish when it happen show a lot of futex wait ?!. The frustrating part is restarting varnish fix the problem immediately, and the cpu remains normal after, even if the trafic peak is not finish. So there is clearly something stacked in varnish which cause our problem. -second question : how to see number of stacked connections, long connections and so on ? At this stage we accept all kind of help / hints for debuging (and regarding the business impact we can evaluate the help of a professional support) PS : I always have the option to scale out, popping a lot of new varnish instance, but this seems very frustrating... Best, -- Raphael Mazelier From guillaume at varnish-software.com Tue Nov 14 22:41:53 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Tue, 14 Nov 2017 23:41:53 +0100 Subject: Varnish nighmare after upgrading : need help In-Reply-To: <620367f1-0247-b457-f65c-08fc898ea6b3@futomaki.net> References: <620367f1-0247-b457-f65c-08fc898ea6b3@futomaki.net> Message-ID: Hi, Let's look at the usual suspects first, can we get the output of "ps aux |grep varnish" and a pastebin of "varnishncsa -1"? Are you using any vmod? man varnishncsa will help craft a format line with the response time (on mobile now, I don't have access to it) Cheers, -- Guillaume Quintard On Nov 14, 2017 23:25, "Raphael Mazelier" wrote: > Hello list, > > First of all despite my mail subject I really appreciate varnish. > We use it a lot at work (hundred of instances) with success and > unfortunately some pain these time. > > TLDR; upgrading from varnish 2 to varnish 4 and 5 on one of our > infrastructure brought us some serious trouble and instability on this > platform. > And we are a bit desperate/frustrated > > > Long story. > > A bit of context : > > This a very complex platform serving an IPTV service with some traffic. > (8k req/s in peak, even more when it work well). > It is compose of a two stage reverse proxy cache (3 x 2 varnish for stage > 1), 2 varnish for stage 2, (so 8 in total) and a lot of different backends > (php applications, nodejs apps, remote backends *sigh*, and even pipe one). > This a big historical spaghetti app. We plan to rebuild it from scratch in > 2018. > The first stage varnish are separate in two pool handling different > topology of clients. > > A lot of the logic is in varnish/vcl itself, lot of url rewrite, lot of > manipulation of headers, choice of a backend, and even ESI processing... > The VCL of the stage 1 varnish are almost 3000 lines long. > > But for now we have to leave/deal with it. > > History of the problem : > > At the beginning all varnish are in 2.x version. Things works almost well. > This summer we need to upgrade the varnish version to handle very long > header (a product requirement). > So after a short battle porting our vcl to vcl4.0 we start using varnish 4. > Shortly after thing begun to goes very bad. > > The first issue we hit, is a memory exhaustion on both stage, and > oom-killer... > We test a lot of things, and in the battle we upgrade to varnish5. > We fix it, resizing the pool, and using now file backend (from memory > before). > Memory is now stable (we have large pool, 32G, and strange thing, we never > have object being nuke, which it good or bad it depend). > We have also fix a lot of things in our vcl. > > The problem we fight against now is only on the stage1 varnish, and > specifically on one pool (the busiest one). > When everything goes well the average cpu usage is 30%, memory stabilize > around 12G, hit cache is around 0.85. > Problem happen randomly (not everyday) but during our peaks. The cpu > increase fasly to reach 350% (4 core) and load > 3/ > When the problem is here varnish still deliver requests (we didn't see > dropped or reject connections) but our application begin to lost user, > including a big lot of business. I suspect this is because timeout are very > aggressive on the client side and varnish should answer slowly > > -first question : how see response time of request of the varnish server > ?. (varnishnsca something ?) > > I also suspect some kind of request queuing, also stracing varnish when it > happen show a lot of futex wait ?!. > The frustrating part is restarting varnish fix the problem immediately, > and the cpu remains normal after, even if the trafic peak is not finish. > So there is clearly something stacked in varnish which cause our problem. > > -second question : how to see number of stacked connections, long > connections and so on ? > > At this stage we accept all kind of help / hints for debuging (and > regarding the business impact we can evaluate the help of a professional > support) > > PS : I always have the option to scale out, popping a lot of new varnish > instance, but this seems very frustrating... > > Best, > > -- > Raphael Mazelier > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagged at gmail.com Wed Nov 15 05:02:56 2017 From: lagged at gmail.com (Andrei) Date: Tue, 14 Nov 2017 23:02:56 -0600 Subject: backend timeouts/503s vs grace cache In-Reply-To: References: Message-ID: bump On Sun, Nov 5, 2017 at 2:12 AM, Andrei wrote: > Hello everyone, > > One of the backends we have configured, runs through an SSH tunnel which > occasionally gets restarted. When the tunnel is restarted, Varnish is > returning a 503 since it can't reach the backend for pages which would > normally be cached (we force cache on the front page of the related site). > I believe our grace implementation might be incorrect, as we would expect a > grace period cache return instead of 503. > > Our grace ttl is set to 21600 seconds based on a global variable: > > sub vcl_backend_response { > set beresp.grace = std.duration(variable.global_get("ttl_grace") + "s", > 6h); > } > > Our grace implementation in sub vcl_hit is: > > sub vcl_hit { > # We have no fresh fish. Lets look at the stale ones. > if (std.healthy(req.backend_hint)) { > # Backend is healthy. Limit age to 10s. > if (obj.ttl + 10s > 0s) { > #set req.http.grace = "normal(limited)"; > std.log("OKHITDELIVER: obj.ttl:" + obj.ttl + " obj.keep: " + > obj.keep + " obj.grace: " + obj.grace); > return (deliver); > } else { > # No candidate for grace. Fetch a fresh object. > std.log("No candidate for grace. Fetch a fresh object. obj.ttl:" + > obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace); > return(miss); > } > } else { > # backend is sick - use full grace > if (obj.ttl + obj.grace > 0s) { > #set req.http.grace = "full"; > std.log("SICK DELIVERY: obj.hits: " + obj.hits + " obj.ttl:" + > obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace); > return (deliver); > } else { > # no graced object. > std.log("No graced object. obj.ttl:" + obj.ttl + " obj.keep: " + > obj.keep + " obj.grace: " + obj.grace); > return (miss); > } > } > > # fetch & deliver once we get the result > return (miss); # Dead code, keep as a safeguard > } > > > Occasionally we see: > - VCL_Log No candidate for grace. Fetch a fresh object. > obj.ttl:-1369.659 obj.keep: 0.000 obj.grace: 21600.000 > > For the most part, it's: > - VCL_Log OKHITDELIVER: obj.ttl:26.872 obj.keep: 0.000 obj.grace: > 21600.000 > > Are we setting the grace ttl too low perhaps? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Wed Nov 15 14:11:29 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 15 Nov 2017 15:11:29 +0100 Subject: backend timeouts/503s vs grace cache In-Reply-To: References: Message-ID: Hi, Why bother with the complex vcl_hit? Since you are saying that the cache is regularly primed, I don't really see the added value. (note, after a quick glance at it, I think it could just be a race condition where the backend appears up in vcl_hit and is down by the time you ask it the content) -- Guillaume Quintard On Wed, Nov 15, 2017 at 6:02 AM, Andrei wrote: > bump > > On Sun, Nov 5, 2017 at 2:12 AM, Andrei wrote: > >> Hello everyone, >> >> One of the backends we have configured, runs through an SSH tunnel which >> occasionally gets restarted. When the tunnel is restarted, Varnish is >> returning a 503 since it can't reach the backend for pages which would >> normally be cached (we force cache on the front page of the related site). >> I believe our grace implementation might be incorrect, as we would expect a >> grace period cache return instead of 503. >> >> Our grace ttl is set to 21600 seconds based on a global variable: >> >> sub vcl_backend_response { >> set beresp.grace = std.duration(variable.global_get("ttl_grace") + >> "s", 6h); >> } >> >> Our grace implementation in sub vcl_hit is: >> >> sub vcl_hit { >> # We have no fresh fish. Lets look at the stale ones. >> if (std.healthy(req.backend_hint)) { >> # Backend is healthy. Limit age to 10s. >> if (obj.ttl + 10s > 0s) { >> #set req.http.grace = "normal(limited)"; >> std.log("OKHITDELIVER: obj.ttl:" + obj.ttl + " obj.keep: " + >> obj.keep + " obj.grace: " + obj.grace); >> return (deliver); >> } else { >> # No candidate for grace. Fetch a fresh object. >> std.log("No candidate for grace. Fetch a fresh object. obj.ttl:" >> + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace); >> return(miss); >> } >> } else { >> # backend is sick - use full grace >> if (obj.ttl + obj.grace > 0s) { >> #set req.http.grace = "full"; >> std.log("SICK DELIVERY: obj.hits: " + obj.hits + " obj.ttl:" + >> obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace); >> return (deliver); >> } else { >> # no graced object. >> std.log("No graced object. obj.ttl:" + obj.ttl + " obj.keep: " + >> obj.keep + " obj.grace: " + obj.grace); >> return (miss); >> } >> } >> >> # fetch & deliver once we get the result >> return (miss); # Dead code, keep as a safeguard >> } >> >> >> Occasionally we see: >> - VCL_Log No candidate for grace. Fetch a fresh object. >> obj.ttl:-1369.659 obj.keep: 0.000 obj.grace: 21600.000 >> >> For the most part, it's: >> - VCL_Log OKHITDELIVER: obj.ttl:26.872 obj.keep: 0.000 >> obj.grace: 21600.000 >> >> Are we setting the grace ttl too low perhaps? >> > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagged at gmail.com Wed Nov 15 14:29:00 2017 From: lagged at gmail.com (Andrei) Date: Wed, 15 Nov 2017 08:29:00 -0600 Subject: backend timeouts/503s vs grace cache In-Reply-To: References: Message-ID: Hi Guillaume, Thanks for getting back to me On Wed, Nov 15, 2017 at 8:11 AM, Guillaume Quintard < guillaume at varnish-software.com> wrote: > Hi, > > Why bother with the complex vcl_hit? Since you are saying that the cache > is regularly primed, I don't really see the added value. > I was mainly going by an example a while back, and not all sites/urls are primed in the same manner. It just stuck in the conf ever since > > (note, after a quick glance at it, I think it could just be a race > condition where the backend appears up in vcl_hit and is down by the time > you ask it the content) > How would you suggest "restarting" the request to try and force a grace cache object to be returned if present in that case? > > -- > Guillaume Quintard > > On Wed, Nov 15, 2017 at 6:02 AM, Andrei wrote: > >> bump >> >> On Sun, Nov 5, 2017 at 2:12 AM, Andrei wrote: >> >>> Hello everyone, >>> >>> One of the backends we have configured, runs through an SSH tunnel which >>> occasionally gets restarted. When the tunnel is restarted, Varnish is >>> returning a 503 since it can't reach the backend for pages which would >>> normally be cached (we force cache on the front page of the related site). >>> I believe our grace implementation might be incorrect, as we would expect a >>> grace period cache return instead of 503. >>> >>> Our grace ttl is set to 21600 seconds based on a global variable: >>> >>> sub vcl_backend_response { >>> set beresp.grace = std.duration(variable.global_get("ttl_grace") + >>> "s", 6h); >>> } >>> >>> Our grace implementation in sub vcl_hit is: >>> >>> sub vcl_hit { >>> # We have no fresh fish. Lets look at the stale ones. >>> if (std.healthy(req.backend_hint)) { >>> # Backend is healthy. Limit age to 10s. >>> if (obj.ttl + 10s > 0s) { >>> #set req.http.grace = "normal(limited)"; >>> std.log("OKHITDELIVER: obj.ttl:" + obj.ttl + " obj.keep: " + >>> obj.keep + " obj.grace: " + obj.grace); >>> return (deliver); >>> } else { >>> # No candidate for grace. Fetch a fresh object. >>> std.log("No candidate for grace. Fetch a fresh object. obj.ttl:" >>> + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace); >>> return(miss); >>> } >>> } else { >>> # backend is sick - use full grace >>> if (obj.ttl + obj.grace > 0s) { >>> #set req.http.grace = "full"; >>> std.log("SICK DELIVERY: obj.hits: " + obj.hits + " obj.ttl:" + >>> obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace); >>> return (deliver); >>> } else { >>> # no graced object. >>> std.log("No graced object. obj.ttl:" + obj.ttl + " obj.keep: " + >>> obj.keep + " obj.grace: " + obj.grace); >>> return (miss); >>> } >>> } >>> >>> # fetch & deliver once we get the result >>> return (miss); # Dead code, keep as a safeguard >>> } >>> >>> >>> Occasionally we see: >>> - VCL_Log No candidate for grace. Fetch a fresh object. >>> obj.ttl:-1369.659 obj.keep: 0.000 obj.grace: 21600.000 >>> >>> For the most part, it's: >>> - VCL_Log OKHITDELIVER: obj.ttl:26.872 obj.keep: 0.000 >>> obj.grace: 21600.000 >>> >>> Are we setting the grace ttl too low perhaps? >>> >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Wed Nov 15 14:34:41 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 15 Nov 2017 15:34:41 +0100 Subject: backend timeouts/503s vs grace cache In-Reply-To: References: Message-ID: You can wait until vcl_deliver and do a restart, possibly adding a marker saying "don't bother with the backend, serve from cache". The actual solution would be to mark the backend as sick before restarting the ssh tunnel, and draining the connections, but I guess that's not an option here, is it? -- Guillaume Quintard On Wed, Nov 15, 2017 at 3:29 PM, Andrei wrote: > Hi Guillaume, > > Thanks for getting back to me > > On Wed, Nov 15, 2017 at 8:11 AM, Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> Hi, >> >> Why bother with the complex vcl_hit? Since you are saying that the cache >> is regularly primed, I don't really see the added value. >> > I was mainly going by an example a while back, and not all sites/urls are > primed in the same manner. It just stuck in the conf ever since > > >> >> (note, after a quick glance at it, I think it could just be a race >> condition where the backend appears up in vcl_hit and is down by the time >> you ask it the content) >> > How would you suggest "restarting" the request to try and force a grace > cache object to be returned if present in that case? > > > >> >> -- >> Guillaume Quintard >> >> On Wed, Nov 15, 2017 at 6:02 AM, Andrei wrote: >> >>> bump >>> >>> On Sun, Nov 5, 2017 at 2:12 AM, Andrei wrote: >>> >>>> Hello everyone, >>>> >>>> One of the backends we have configured, runs through an SSH tunnel >>>> which occasionally gets restarted. When the tunnel is restarted, Varnish is >>>> returning a 503 since it can't reach the backend for pages which would >>>> normally be cached (we force cache on the front page of the related site). >>>> I believe our grace implementation might be incorrect, as we would expect a >>>> grace period cache return instead of 503. >>>> >>>> Our grace ttl is set to 21600 seconds based on a global variable: >>>> >>>> sub vcl_backend_response { >>>> set beresp.grace = std.duration(variable.global_get("ttl_grace") + >>>> "s", 6h); >>>> } >>>> >>>> Our grace implementation in sub vcl_hit is: >>>> >>>> sub vcl_hit { >>>> # We have no fresh fish. Lets look at the stale ones. >>>> if (std.healthy(req.backend_hint)) { >>>> # Backend is healthy. Limit age to 10s. >>>> if (obj.ttl + 10s > 0s) { >>>> #set req.http.grace = "normal(limited)"; >>>> std.log("OKHITDELIVER: obj.ttl:" + obj.ttl + " obj.keep: " + >>>> obj.keep + " obj.grace: " + obj.grace); >>>> return (deliver); >>>> } else { >>>> # No candidate for grace. Fetch a fresh object. >>>> std.log("No candidate for grace. Fetch a fresh object. >>>> obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + >>>> obj.grace); >>>> return(miss); >>>> } >>>> } else { >>>> # backend is sick - use full grace >>>> if (obj.ttl + obj.grace > 0s) { >>>> #set req.http.grace = "full"; >>>> std.log("SICK DELIVERY: obj.hits: " + obj.hits + " obj.ttl:" >>>> + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace); >>>> return (deliver); >>>> } else { >>>> # no graced object. >>>> std.log("No graced object. obj.ttl:" + obj.ttl + " obj.keep: " >>>> + obj.keep + " obj.grace: " + obj.grace); >>>> return (miss); >>>> } >>>> } >>>> >>>> # fetch & deliver once we get the result >>>> return (miss); # Dead code, keep as a safeguard >>>> } >>>> >>>> >>>> Occasionally we see: >>>> - VCL_Log No candidate for grace. Fetch a fresh object. >>>> obj.ttl:-1369.659 obj.keep: 0.000 obj.grace: 21600.000 >>>> >>>> For the most part, it's: >>>> - VCL_Log OKHITDELIVER: obj.ttl:26.872 obj.keep: 0.000 >>>> obj.grace: 21600.000 >>>> >>>> Are we setting the grace ttl too low perhaps? >>>> >>> >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagged at gmail.com Wed Nov 15 14:42:04 2017 From: lagged at gmail.com (Andrei) Date: Wed, 15 Nov 2017 08:42:04 -0600 Subject: backend timeouts/503s vs grace cache In-Reply-To: References: Message-ID: Thanks for the pointers! The tunnel setup is pretty flexible so I'll go ahead and mark the backend sick before restarting the tunnel, then healthy once confirmed up. On Wed, Nov 15, 2017 at 8:34 AM, Guillaume Quintard < guillaume at varnish-software.com> wrote: > You can wait until vcl_deliver and do a restart, possibly adding a marker > saying "don't bother with the backend, serve from cache". > > The actual solution would be to mark the backend as sick before restarting > the ssh tunnel, and draining the connections, but I guess that's not an > option here, is it? > > -- > Guillaume Quintard > > On Wed, Nov 15, 2017 at 3:29 PM, Andrei wrote: > >> Hi Guillaume, >> >> Thanks for getting back to me >> >> On Wed, Nov 15, 2017 at 8:11 AM, Guillaume Quintard < >> guillaume at varnish-software.com> wrote: >> >>> Hi, >>> >>> Why bother with the complex vcl_hit? Since you are saying that the cache >>> is regularly primed, I don't really see the added value. >>> >> I was mainly going by an example a while back, and not all sites/urls are >> primed in the same manner. It just stuck in the conf ever since >> >> >>> >>> (note, after a quick glance at it, I think it could just be a race >>> condition where the backend appears up in vcl_hit and is down by the time >>> you ask it the content) >>> >> How would you suggest "restarting" the request to try and force a grace >> cache object to be returned if present in that case? >> >> >> >>> >>> -- >>> Guillaume Quintard >>> >>> On Wed, Nov 15, 2017 at 6:02 AM, Andrei wrote: >>> >>>> bump >>>> >>>> On Sun, Nov 5, 2017 at 2:12 AM, Andrei wrote: >>>> >>>>> Hello everyone, >>>>> >>>>> One of the backends we have configured, runs through an SSH tunnel >>>>> which occasionally gets restarted. When the tunnel is restarted, Varnish is >>>>> returning a 503 since it can't reach the backend for pages which would >>>>> normally be cached (we force cache on the front page of the related site). >>>>> I believe our grace implementation might be incorrect, as we would expect a >>>>> grace period cache return instead of 503. >>>>> >>>>> Our grace ttl is set to 21600 seconds based on a global variable: >>>>> >>>>> sub vcl_backend_response { >>>>> set beresp.grace = std.duration(variable.global_get("ttl_grace") + >>>>> "s", 6h); >>>>> } >>>>> >>>>> Our grace implementation in sub vcl_hit is: >>>>> >>>>> sub vcl_hit { >>>>> # We have no fresh fish. Lets look at the stale ones. >>>>> if (std.healthy(req.backend_hint)) { >>>>> # Backend is healthy. Limit age to 10s. >>>>> if (obj.ttl + 10s > 0s) { >>>>> #set req.http.grace = "normal(limited)"; >>>>> std.log("OKHITDELIVER: obj.ttl:" + obj.ttl + " obj.keep: " + >>>>> obj.keep + " obj.grace: " + obj.grace); >>>>> return (deliver); >>>>> } else { >>>>> # No candidate for grace. Fetch a fresh object. >>>>> std.log("No candidate for grace. Fetch a fresh object. >>>>> obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + >>>>> obj.grace); >>>>> return(miss); >>>>> } >>>>> } else { >>>>> # backend is sick - use full grace >>>>> if (obj.ttl + obj.grace > 0s) { >>>>> #set req.http.grace = "full"; >>>>> std.log("SICK DELIVERY: obj.hits: " + obj.hits + " obj.ttl:" >>>>> + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + obj.grace); >>>>> return (deliver); >>>>> } else { >>>>> # no graced object. >>>>> std.log("No graced object. obj.ttl:" + obj.ttl + " obj.keep: " >>>>> + obj.keep + " obj.grace: " + obj.grace); >>>>> return (miss); >>>>> } >>>>> } >>>>> >>>>> # fetch & deliver once we get the result >>>>> return (miss); # Dead code, keep as a safeguard >>>>> } >>>>> >>>>> >>>>> Occasionally we see: >>>>> - VCL_Log No candidate for grace. Fetch a fresh object. >>>>> obj.ttl:-1369.659 obj.keep: 0.000 obj.grace: 21600.000 >>>>> >>>>> For the most part, it's: >>>>> - VCL_Log OKHITDELIVER: obj.ttl:26.872 obj.keep: 0.000 >>>>> obj.grace: 21600.000 >>>>> >>>>> Are we setting the grace ttl too low perhaps? >>>>> >>>> >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>> >>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Wed Nov 15 14:46:36 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 15 Nov 2017 15:46:36 +0100 Subject: backend timeouts/503s vs grace cache In-Reply-To: References: Message-ID: Oh, then your life should be easier then! Don't forget to drain the connections, varnishstat will give you the number of open connections open to any backend. -- Guillaume Quintard On Wed, Nov 15, 2017 at 3:42 PM, Andrei wrote: > Thanks for the pointers! The tunnel setup is pretty flexible so I'll go > ahead and mark the backend sick before restarting the tunnel, then healthy > once confirmed up. > > > On Wed, Nov 15, 2017 at 8:34 AM, Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> You can wait until vcl_deliver and do a restart, possibly adding a marker >> saying "don't bother with the backend, serve from cache". >> >> The actual solution would be to mark the backend as sick before >> restarting the ssh tunnel, and draining the connections, but I guess that's >> not an option here, is it? >> >> -- >> Guillaume Quintard >> >> On Wed, Nov 15, 2017 at 3:29 PM, Andrei wrote: >> >>> Hi Guillaume, >>> >>> Thanks for getting back to me >>> >>> On Wed, Nov 15, 2017 at 8:11 AM, Guillaume Quintard < >>> guillaume at varnish-software.com> wrote: >>> >>>> Hi, >>>> >>>> Why bother with the complex vcl_hit? Since you are saying that the >>>> cache is regularly primed, I don't really see the added value. >>>> >>> I was mainly going by an example a while back, and not all sites/urls >>> are primed in the same manner. It just stuck in the conf ever since >>> >>> >>>> >>>> (note, after a quick glance at it, I think it could just be a race >>>> condition where the backend appears up in vcl_hit and is down by the time >>>> you ask it the content) >>>> >>> How would you suggest "restarting" the request to try and force a grace >>> cache object to be returned if present in that case? >>> >>> >>> >>>> >>>> -- >>>> Guillaume Quintard >>>> >>>> On Wed, Nov 15, 2017 at 6:02 AM, Andrei wrote: >>>> >>>>> bump >>>>> >>>>> On Sun, Nov 5, 2017 at 2:12 AM, Andrei wrote: >>>>> >>>>>> Hello everyone, >>>>>> >>>>>> One of the backends we have configured, runs through an SSH tunnel >>>>>> which occasionally gets restarted. When the tunnel is restarted, Varnish is >>>>>> returning a 503 since it can't reach the backend for pages which would >>>>>> normally be cached (we force cache on the front page of the related site). >>>>>> I believe our grace implementation might be incorrect, as we would expect a >>>>>> grace period cache return instead of 503. >>>>>> >>>>>> Our grace ttl is set to 21600 seconds based on a global variable: >>>>>> >>>>>> sub vcl_backend_response { >>>>>> set beresp.grace = std.duration(variable.global_get("ttl_grace") + >>>>>> "s", 6h); >>>>>> } >>>>>> >>>>>> Our grace implementation in sub vcl_hit is: >>>>>> >>>>>> sub vcl_hit { >>>>>> # We have no fresh fish. Lets look at the stale ones. >>>>>> if (std.healthy(req.backend_hint)) { >>>>>> # Backend is healthy. Limit age to 10s. >>>>>> if (obj.ttl + 10s > 0s) { >>>>>> #set req.http.grace = "normal(limited)"; >>>>>> std.log("OKHITDELIVER: obj.ttl:" + obj.ttl + " obj.keep: " + >>>>>> obj.keep + " obj.grace: " + obj.grace); >>>>>> return (deliver); >>>>>> } else { >>>>>> # No candidate for grace. Fetch a fresh object. >>>>>> std.log("No candidate for grace. Fetch a fresh object. >>>>>> obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + >>>>>> obj.grace); >>>>>> return(miss); >>>>>> } >>>>>> } else { >>>>>> # backend is sick - use full grace >>>>>> if (obj.ttl + obj.grace > 0s) { >>>>>> #set req.http.grace = "full"; >>>>>> std.log("SICK DELIVERY: obj.hits: " + obj.hits + " >>>>>> obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + >>>>>> obj.grace); >>>>>> return (deliver); >>>>>> } else { >>>>>> # no graced object. >>>>>> std.log("No graced object. obj.ttl:" + obj.ttl + " obj.keep: >>>>>> " + obj.keep + " obj.grace: " + obj.grace); >>>>>> return (miss); >>>>>> } >>>>>> } >>>>>> >>>>>> # fetch & deliver once we get the result >>>>>> return (miss); # Dead code, keep as a safeguard >>>>>> } >>>>>> >>>>>> >>>>>> Occasionally we see: >>>>>> - VCL_Log No candidate for grace. Fetch a fresh object. >>>>>> obj.ttl:-1369.659 obj.keep: 0.000 obj.grace: 21600.000 >>>>>> >>>>>> For the most part, it's: >>>>>> - VCL_Log OKHITDELIVER: obj.ttl:26.872 obj.keep: 0.000 >>>>>> obj.grace: 21600.000 >>>>>> >>>>>> Are we setting the grace ttl too low perhaps? >>>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> varnish-misc mailing list >>>>> varnish-misc at varnish-cache.org >>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>> >>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagged at gmail.com Wed Nov 15 14:48:38 2017 From: lagged at gmail.com (Andrei) Date: Wed, 15 Nov 2017 08:48:38 -0600 Subject: backend timeouts/503s vs grace cache In-Reply-To: References: Message-ID: What do you mean exactly when you say "drain the connections"? :D On Wed, Nov 15, 2017 at 8:46 AM, Guillaume Quintard < guillaume at varnish-software.com> wrote: > Oh, then your life should be easier then! Don't forget to drain the > connections, varnishstat will give you the number of open connections open > to any backend. > > -- > Guillaume Quintard > > On Wed, Nov 15, 2017 at 3:42 PM, Andrei wrote: > >> Thanks for the pointers! The tunnel setup is pretty flexible so I'll go >> ahead and mark the backend sick before restarting the tunnel, then healthy >> once confirmed up. >> >> >> On Wed, Nov 15, 2017 at 8:34 AM, Guillaume Quintard < >> guillaume at varnish-software.com> wrote: >> >>> You can wait until vcl_deliver and do a restart, possibly adding a >>> marker saying "don't bother with the backend, serve from cache". >>> >>> The actual solution would be to mark the backend as sick before >>> restarting the ssh tunnel, and draining the connections, but I guess that's >>> not an option here, is it? >>> >>> -- >>> Guillaume Quintard >>> >>> On Wed, Nov 15, 2017 at 3:29 PM, Andrei wrote: >>> >>>> Hi Guillaume, >>>> >>>> Thanks for getting back to me >>>> >>>> On Wed, Nov 15, 2017 at 8:11 AM, Guillaume Quintard < >>>> guillaume at varnish-software.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> Why bother with the complex vcl_hit? Since you are saying that the >>>>> cache is regularly primed, I don't really see the added value. >>>>> >>>> I was mainly going by an example a while back, and not all sites/urls >>>> are primed in the same manner. It just stuck in the conf ever since >>>> >>>> >>>>> >>>>> (note, after a quick glance at it, I think it could just be a race >>>>> condition where the backend appears up in vcl_hit and is down by the time >>>>> you ask it the content) >>>>> >>>> How would you suggest "restarting" the request to try and force a grace >>>> cache object to be returned if present in that case? >>>> >>>> >>>> >>>>> >>>>> -- >>>>> Guillaume Quintard >>>>> >>>>> On Wed, Nov 15, 2017 at 6:02 AM, Andrei wrote: >>>>> >>>>>> bump >>>>>> >>>>>> On Sun, Nov 5, 2017 at 2:12 AM, Andrei wrote: >>>>>> >>>>>>> Hello everyone, >>>>>>> >>>>>>> One of the backends we have configured, runs through an SSH tunnel >>>>>>> which occasionally gets restarted. When the tunnel is restarted, Varnish is >>>>>>> returning a 503 since it can't reach the backend for pages which would >>>>>>> normally be cached (we force cache on the front page of the related site). >>>>>>> I believe our grace implementation might be incorrect, as we would expect a >>>>>>> grace period cache return instead of 503. >>>>>>> >>>>>>> Our grace ttl is set to 21600 seconds based on a global variable: >>>>>>> >>>>>>> sub vcl_backend_response { >>>>>>> set beresp.grace = std.duration(variable.global_get("ttl_grace") >>>>>>> + "s", 6h); >>>>>>> } >>>>>>> >>>>>>> Our grace implementation in sub vcl_hit is: >>>>>>> >>>>>>> sub vcl_hit { >>>>>>> # We have no fresh fish. Lets look at the stale ones. >>>>>>> if (std.healthy(req.backend_hint)) { >>>>>>> # Backend is healthy. Limit age to 10s. >>>>>>> if (obj.ttl + 10s > 0s) { >>>>>>> #set req.http.grace = "normal(limited)"; >>>>>>> std.log("OKHITDELIVER: obj.ttl:" + obj.ttl + " obj.keep: " + >>>>>>> obj.keep + " obj.grace: " + obj.grace); >>>>>>> return (deliver); >>>>>>> } else { >>>>>>> # No candidate for grace. Fetch a fresh object. >>>>>>> std.log("No candidate for grace. Fetch a fresh object. >>>>>>> obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + >>>>>>> obj.grace); >>>>>>> return(miss); >>>>>>> } >>>>>>> } else { >>>>>>> # backend is sick - use full grace >>>>>>> if (obj.ttl + obj.grace > 0s) { >>>>>>> #set req.http.grace = "full"; >>>>>>> std.log("SICK DELIVERY: obj.hits: " + obj.hits + " >>>>>>> obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + >>>>>>> obj.grace); >>>>>>> return (deliver); >>>>>>> } else { >>>>>>> # no graced object. >>>>>>> std.log("No graced object. obj.ttl:" + obj.ttl + " obj.keep: >>>>>>> " + obj.keep + " obj.grace: " + obj.grace); >>>>>>> return (miss); >>>>>>> } >>>>>>> } >>>>>>> >>>>>>> # fetch & deliver once we get the result >>>>>>> return (miss); # Dead code, keep as a safeguard >>>>>>> } >>>>>>> >>>>>>> >>>>>>> Occasionally we see: >>>>>>> - VCL_Log No candidate for grace. Fetch a fresh object. >>>>>>> obj.ttl:-1369.659 obj.keep: 0.000 obj.grace: 21600.000 >>>>>>> >>>>>>> For the most part, it's: >>>>>>> - VCL_Log OKHITDELIVER: obj.ttl:26.872 obj.keep: 0.000 >>>>>>> obj.grace: 21600.000 >>>>>>> >>>>>>> Are we setting the grace ttl too low perhaps? >>>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> varnish-misc mailing list >>>>>> varnish-misc at varnish-cache.org >>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>> >>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Wed Nov 15 14:56:34 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 15 Nov 2017 15:56:34 +0100 Subject: backend timeouts/503s vs grace cache In-Reply-To: References: Message-ID: Once you set the backend to sick, you will probably still have requests in-flight to this backend, so you don't want to restart your SSH tunnel just yet. Instead, monitor the VBE.*.BACKENDNAME.conn lines, and wait for them to drop to 0, then you can reset the SSH tunnel. -- Guillaume Quintard On Wed, Nov 15, 2017 at 3:48 PM, Andrei wrote: > What do you mean exactly when you say "drain the connections"? :D > > On Wed, Nov 15, 2017 at 8:46 AM, Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> Oh, then your life should be easier then! Don't forget to drain the >> connections, varnishstat will give you the number of open connections open >> to any backend. >> >> -- >> Guillaume Quintard >> >> On Wed, Nov 15, 2017 at 3:42 PM, Andrei wrote: >> >>> Thanks for the pointers! The tunnel setup is pretty flexible so I'll go >>> ahead and mark the backend sick before restarting the tunnel, then healthy >>> once confirmed up. >>> >>> >>> On Wed, Nov 15, 2017 at 8:34 AM, Guillaume Quintard < >>> guillaume at varnish-software.com> wrote: >>> >>>> You can wait until vcl_deliver and do a restart, possibly adding a >>>> marker saying "don't bother with the backend, serve from cache". >>>> >>>> The actual solution would be to mark the backend as sick before >>>> restarting the ssh tunnel, and draining the connections, but I guess that's >>>> not an option here, is it? >>>> >>>> -- >>>> Guillaume Quintard >>>> >>>> On Wed, Nov 15, 2017 at 3:29 PM, Andrei wrote: >>>> >>>>> Hi Guillaume, >>>>> >>>>> Thanks for getting back to me >>>>> >>>>> On Wed, Nov 15, 2017 at 8:11 AM, Guillaume Quintard < >>>>> guillaume at varnish-software.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Why bother with the complex vcl_hit? Since you are saying that the >>>>>> cache is regularly primed, I don't really see the added value. >>>>>> >>>>> I was mainly going by an example a while back, and not all sites/urls >>>>> are primed in the same manner. It just stuck in the conf ever since >>>>> >>>>> >>>>>> >>>>>> (note, after a quick glance at it, I think it could just be a race >>>>>> condition where the backend appears up in vcl_hit and is down by the time >>>>>> you ask it the content) >>>>>> >>>>> How would you suggest "restarting" the request to try and force a >>>>> grace cache object to be returned if present in that case? >>>>> >>>>> >>>>> >>>>>> >>>>>> -- >>>>>> Guillaume Quintard >>>>>> >>>>>> On Wed, Nov 15, 2017 at 6:02 AM, Andrei wrote: >>>>>> >>>>>>> bump >>>>>>> >>>>>>> On Sun, Nov 5, 2017 at 2:12 AM, Andrei wrote: >>>>>>> >>>>>>>> Hello everyone, >>>>>>>> >>>>>>>> One of the backends we have configured, runs through an SSH tunnel >>>>>>>> which occasionally gets restarted. When the tunnel is restarted, Varnish is >>>>>>>> returning a 503 since it can't reach the backend for pages which would >>>>>>>> normally be cached (we force cache on the front page of the related site). >>>>>>>> I believe our grace implementation might be incorrect, as we would expect a >>>>>>>> grace period cache return instead of 503. >>>>>>>> >>>>>>>> Our grace ttl is set to 21600 seconds based on a global variable: >>>>>>>> >>>>>>>> sub vcl_backend_response { >>>>>>>> set beresp.grace = std.duration(variable.global_get("ttl_grace") >>>>>>>> + "s", 6h); >>>>>>>> } >>>>>>>> >>>>>>>> Our grace implementation in sub vcl_hit is: >>>>>>>> >>>>>>>> sub vcl_hit { >>>>>>>> # We have no fresh fish. Lets look at the stale ones. >>>>>>>> if (std.healthy(req.backend_hint)) { >>>>>>>> # Backend is healthy. Limit age to 10s. >>>>>>>> if (obj.ttl + 10s > 0s) { >>>>>>>> #set req.http.grace = "normal(limited)"; >>>>>>>> std.log("OKHITDELIVER: obj.ttl:" + obj.ttl + " obj.keep: " >>>>>>>> + obj.keep + " obj.grace: " + obj.grace); >>>>>>>> return (deliver); >>>>>>>> } else { >>>>>>>> # No candidate for grace. Fetch a fresh object. >>>>>>>> std.log("No candidate for grace. Fetch a fresh object. >>>>>>>> obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + >>>>>>>> obj.grace); >>>>>>>> return(miss); >>>>>>>> } >>>>>>>> } else { >>>>>>>> # backend is sick - use full grace >>>>>>>> if (obj.ttl + obj.grace > 0s) { >>>>>>>> #set req.http.grace = "full"; >>>>>>>> std.log("SICK DELIVERY: obj.hits: " + obj.hits + " >>>>>>>> obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + >>>>>>>> obj.grace); >>>>>>>> return (deliver); >>>>>>>> } else { >>>>>>>> # no graced object. >>>>>>>> std.log("No graced object. obj.ttl:" + obj.ttl + " >>>>>>>> obj.keep: " + obj.keep + " obj.grace: " + obj.grace); >>>>>>>> return (miss); >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> # fetch & deliver once we get the result >>>>>>>> return (miss); # Dead code, keep as a safeguard >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> Occasionally we see: >>>>>>>> - VCL_Log No candidate for grace. Fetch a fresh object. >>>>>>>> obj.ttl:-1369.659 obj.keep: 0.000 obj.grace: 21600.000 >>>>>>>> >>>>>>>> For the most part, it's: >>>>>>>> - VCL_Log OKHITDELIVER: obj.ttl:26.872 obj.keep: 0.000 >>>>>>>> obj.grace: 21600.000 >>>>>>>> >>>>>>>> Are we setting the grace ttl too low perhaps? >>>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> varnish-misc mailing list >>>>>>> varnish-misc at varnish-cache.org >>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From lagged at gmail.com Wed Nov 15 15:03:10 2017 From: lagged at gmail.com (Andrei) Date: Wed, 15 Nov 2017 09:03:10 -0600 Subject: backend timeouts/503s vs grace cache In-Reply-To: References: Message-ID: I'm afraid I'll have to skip that step since the tunnel is restarted based on connection errors from probes sent every 3s, and those "pending requests" shouldn't be completing correctly anyways as those are half open connections at that point. Not to mention I want to minimize visibility/downtime, and waiting on the lingering requests also means more timeouts to worry about. Thanks again for your input though. I should definitely see better results from request tagging/restarts and marking the backends accordingly during the tunnel reatarts On Nov 15, 2017 16:56, "Guillaume Quintard" wrote: Once you set the backend to sick, you will probably still have requests in-flight to this backend, so you don't want to restart your SSH tunnel just yet. Instead, monitor the VBE.*.BACKENDNAME.conn lines, and wait for them to drop to 0, then you can reset the SSH tunnel. -- Guillaume Quintard On Wed, Nov 15, 2017 at 3:48 PM, Andrei wrote: > What do you mean exactly when you say "drain the connections"? :D > > On Wed, Nov 15, 2017 at 8:46 AM, Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> Oh, then your life should be easier then! Don't forget to drain the >> connections, varnishstat will give you the number of open connections open >> to any backend. >> >> -- >> Guillaume Quintard >> >> On Wed, Nov 15, 2017 at 3:42 PM, Andrei wrote: >> >>> Thanks for the pointers! The tunnel setup is pretty flexible so I'll go >>> ahead and mark the backend sick before restarting the tunnel, then healthy >>> once confirmed up. >>> >>> >>> On Wed, Nov 15, 2017 at 8:34 AM, Guillaume Quintard < >>> guillaume at varnish-software.com> wrote: >>> >>>> You can wait until vcl_deliver and do a restart, possibly adding a >>>> marker saying "don't bother with the backend, serve from cache". >>>> >>>> The actual solution would be to mark the backend as sick before >>>> restarting the ssh tunnel, and draining the connections, but I guess that's >>>> not an option here, is it? >>>> >>>> -- >>>> Guillaume Quintard >>>> >>>> On Wed, Nov 15, 2017 at 3:29 PM, Andrei wrote: >>>> >>>>> Hi Guillaume, >>>>> >>>>> Thanks for getting back to me >>>>> >>>>> On Wed, Nov 15, 2017 at 8:11 AM, Guillaume Quintard < >>>>> guillaume at varnish-software.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Why bother with the complex vcl_hit? Since you are saying that the >>>>>> cache is regularly primed, I don't really see the added value. >>>>>> >>>>> I was mainly going by an example a while back, and not all sites/urls >>>>> are primed in the same manner. It just stuck in the conf ever since >>>>> >>>>> >>>>>> >>>>>> (note, after a quick glance at it, I think it could just be a race >>>>>> condition where the backend appears up in vcl_hit and is down by the time >>>>>> you ask it the content) >>>>>> >>>>> How would you suggest "restarting" the request to try and force a >>>>> grace cache object to be returned if present in that case? >>>>> >>>>> >>>>> >>>>>> >>>>>> -- >>>>>> Guillaume Quintard >>>>>> >>>>>> On Wed, Nov 15, 2017 at 6:02 AM, Andrei wrote: >>>>>> >>>>>>> bump >>>>>>> >>>>>>> On Sun, Nov 5, 2017 at 2:12 AM, Andrei wrote: >>>>>>> >>>>>>>> Hello everyone, >>>>>>>> >>>>>>>> One of the backends we have configured, runs through an SSH tunnel >>>>>>>> which occasionally gets restarted. When the tunnel is restarted, Varnish is >>>>>>>> returning a 503 since it can't reach the backend for pages which would >>>>>>>> normally be cached (we force cache on the front page of the related site). >>>>>>>> I believe our grace implementation might be incorrect, as we would expect a >>>>>>>> grace period cache return instead of 503. >>>>>>>> >>>>>>>> Our grace ttl is set to 21600 seconds based on a global variable: >>>>>>>> >>>>>>>> sub vcl_backend_response { >>>>>>>> set beresp.grace = std.duration(variable.global_get("ttl_grace") >>>>>>>> + "s", 6h); >>>>>>>> } >>>>>>>> >>>>>>>> Our grace implementation in sub vcl_hit is: >>>>>>>> >>>>>>>> sub vcl_hit { >>>>>>>> # We have no fresh fish. Lets look at the stale ones. >>>>>>>> if (std.healthy(req.backend_hint)) { >>>>>>>> # Backend is healthy. Limit age to 10s. >>>>>>>> if (obj.ttl + 10s > 0s) { >>>>>>>> #set req.http.grace = "normal(limited)"; >>>>>>>> std.log("OKHITDELIVER: obj.ttl:" + obj.ttl + " obj.keep: " >>>>>>>> + obj.keep + " obj.grace: " + obj.grace); >>>>>>>> return (deliver); >>>>>>>> } else { >>>>>>>> # No candidate for grace. Fetch a fresh object. >>>>>>>> std.log("No candidate for grace. Fetch a fresh object. >>>>>>>> obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + >>>>>>>> obj.grace); >>>>>>>> return(miss); >>>>>>>> } >>>>>>>> } else { >>>>>>>> # backend is sick - use full grace >>>>>>>> if (obj.ttl + obj.grace > 0s) { >>>>>>>> #set req.http.grace = "full"; >>>>>>>> std.log("SICK DELIVERY: obj.hits: " + obj.hits + " >>>>>>>> obj.ttl:" + obj.ttl + " obj.keep: " + obj.keep + " obj.grace: " + >>>>>>>> obj.grace); >>>>>>>> return (deliver); >>>>>>>> } else { >>>>>>>> # no graced object. >>>>>>>> std.log("No graced object. obj.ttl:" + obj.ttl + " >>>>>>>> obj.keep: " + obj.keep + " obj.grace: " + obj.grace); >>>>>>>> return (miss); >>>>>>>> } >>>>>>>> } >>>>>>>> >>>>>>>> # fetch & deliver once we get the result >>>>>>>> return (miss); # Dead code, keep as a safeguard >>>>>>>> } >>>>>>>> >>>>>>>> >>>>>>>> Occasionally we see: >>>>>>>> - VCL_Log No candidate for grace. Fetch a fresh object. >>>>>>>> obj.ttl:-1369.659 obj.keep: 0.000 obj.grace: 21600.000 >>>>>>>> >>>>>>>> For the most part, it's: >>>>>>>> - VCL_Log OKHITDELIVER: obj.ttl:26.872 obj.keep: 0.000 >>>>>>>> obj.grace: 21600.000 >>>>>>>> >>>>>>>> Are we setting the grace ttl too low perhaps? >>>>>>>> >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> varnish-misc mailing list >>>>>>> varnish-misc at varnish-cache.org >>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From raph at futomaki.net Wed Nov 15 22:52:50 2017 From: raph at futomaki.net (Raphael Mazelier) Date: Wed, 15 Nov 2017 23:52:50 +0100 Subject: Varnish nighmare after upgrading : need help In-Reply-To: References: <620367f1-0247-b457-f65c-08fc898ea6b3@futomaki.net> Message-ID: <89923832-10d5-2a3f-617e-22e9a9f31f7b@futomaki.net> Hi, Of course the evening was quite quiet and I have no spurious output to show. (schrodinger effect) Anyway here the pastebin of the busiest period this night https://pastebin.com/536LM9Nx. We use std, and director vmod. Btw : I found the correct format for varnishncsa (varnishncsa -F '%h %r %s %{Varnish:handling}x %{Varnish:side}x %T %D' does the job). Side question : why not include hit/miss in the default output ? Thks for the help. Best, -- Raphael Mazelier On 14/11/2017 23:41, Guillaume Quintard wrote: > Hi, > > Let's look at the usual suspects first, can we get the output of "ps > aux |grep varnish" and a pastebin of "varnishncsa -1"? > > Are you using any vmod? > > man varnishncsa will help craft a format line with the response time > (on mobile now, I don't have access to it) > > Cheers, > > -- > Guillaume Quintard > > On Nov 14, 2017 23:25, "Raphael Mazelier" > wrote: > > Hello list, > > First of all despite my mail subject I really appreciate varnish. > We use it a lot at work (hundred of instances) with success and > unfortunately some pain these time. > > TLDR; upgrading from varnish 2 to varnish 4 and 5 on one of our > infrastructure brought us some serious trouble and instability on > this platform. > And we are a bit desperate/frustrated > > > Long story. > > A bit of context : > > This a very complex platform serving an IPTV service with some > traffic. (8k req/s in peak, even more when it work well). > It is compose of a two stage reverse proxy cache (3 x 2 varnish > for stage 1), 2 varnish for stage 2, (so 8 in total) and a lot of > different backends (php applications, nodejs apps, remote backends > *sigh*, and even pipe one). This a big historical spaghetti app. > We plan to rebuild it from scratch in 2018. > The first stage varnish are separate in two pool handling > different topology of clients. > > A lot of the logic is in varnish/vcl itself, lot of url rewrite, > lot of manipulation of headers, choice of a backend, and even ESI > processing... > The VCL of the stage 1 varnish are almost 3000 lines long. > > But for now we have to leave/deal with it. > > History of the problem : > > At the beginning all varnish are in 2.x version. Things works > almost well. > This summer we need to upgrade the varnish version to handle very > long header (a product requirement). > So after a short battle porting our vcl to vcl4.0 we start using > varnish 4. > Shortly after thing begun to goes very bad. > > The first issue we hit, is a memory exhaustion on both stage, and > oom-killer... > We test a lot of things, and in the battle we upgrade to varnish5. > We fix it, resizing the pool, and using now file backend (from > memory before). > Memory is now stable (we have large pool, 32G, and strange thing, > we never have object being nuke, which it good or bad it depend). > We have also fix a lot of things in our vcl. > > The problem we fight against now is only on the stage1 varnish, > and specifically on one pool (the busiest one). > When everything goes well the average cpu usage is 30%, memory > stabilize around 12G, hit cache is around 0.85. > Problem happen randomly (not everyday) but during our peaks. The > cpu increase fasly to reach 350% (4 core) and load > 3/ > When the problem is here varnish still deliver requests (we didn't > see dropped or reject connections) but our application begin to > lost user, including a big lot of business. I suspect this is > because timeout are very aggressive on the client side and varnish > should answer slowly > > -first question : how see response time of request of the varnish > server ?. (varnishnsca something ?) > > I also suspect some kind of request queuing, also stracing varnish > when it happen show a lot of futex wait ?!. > The frustrating part is restarting varnish fix the problem > immediately, and the cpu remains normal after, even if the trafic > peak is not finish. > So there is clearly something stacked in varnish which cause our > problem. > > -second question : how to see number of stacked connections, long > connections and so on ? > > At this stage we accept all kind of help / hints for debuging (and > regarding the business impact we can evaluate the help of a > professional support) > > PS : I always have the option to scale out, popping a lot of new > varnish instance, but this seems very frustrating... > > Best, > > -- > Raphael Mazelier > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Nov 16 08:09:31 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 16 Nov 2017 09:09:31 +0100 Subject: Varnish nighmare after upgrading : need help In-Reply-To: <89923832-10d5-2a3f-617e-22e9a9f31f7b@futomaki.net> References: <620367f1-0247-b457-f65c-08fc898ea6b3@futomaki.net> <89923832-10d5-2a3f-617e-22e9a9f31f7b@futomaki.net> Message-ID: I think we just replicate the ncsa default format line -- Guillaume Quintard On Nov 15, 2017 23:52, "Raphael Mazelier" wrote: > Hi, > > Of course the evening was quite quiet and I have no spurious output to > show. (schrodinger effect) > > Anyway here the pastebin of the busiest period this night > https://pastebin.com/536LM9Nx. > > We use std, and director vmod. > > Btw : I found the correct format for varnishncsa (varnishncsa -F '%h %r > %s %{Varnish:handling}x %{Varnish:side}x %T %D' does the job). > Side question : why not include hit/miss in the default output ? > > > Thks for the help. > > Best, > > -- > Raphael Mazelier > > On 14/11/2017 23:41, Guillaume Quintard wrote: > > Hi, > > Let's look at the usual suspects first, can we get the output of "ps aux > |grep varnish" and a pastebin of "varnishncsa -1"? > > Are you using any vmod? > > man varnishncsa will help craft a format line with the response time (on > mobile now, I don't have access to it) > > Cheers, > > -- > Guillaume Quintard > > On Nov 14, 2017 23:25, "Raphael Mazelier" wrote: > >> Hello list, >> >> First of all despite my mail subject I really appreciate varnish. >> We use it a lot at work (hundred of instances) with success and >> unfortunately some pain these time. >> >> TLDR; upgrading from varnish 2 to varnish 4 and 5 on one of our >> infrastructure brought us some serious trouble and instability on this >> platform. >> And we are a bit desperate/frustrated >> >> >> Long story. >> >> A bit of context : >> >> This a very complex platform serving an IPTV service with some traffic. >> (8k req/s in peak, even more when it work well). >> It is compose of a two stage reverse proxy cache (3 x 2 varnish for stage >> 1), 2 varnish for stage 2, (so 8 in total) and a lot of different backends >> (php applications, nodejs apps, remote backends *sigh*, and even pipe one). >> This a big historical spaghetti app. We plan to rebuild it from scratch in >> 2018. >> The first stage varnish are separate in two pool handling different >> topology of clients. >> >> A lot of the logic is in varnish/vcl itself, lot of url rewrite, lot of >> manipulation of headers, choice of a backend, and even ESI processing... >> The VCL of the stage 1 varnish are almost 3000 lines long. >> >> But for now we have to leave/deal with it. >> >> History of the problem : >> >> At the beginning all varnish are in 2.x version. Things works almost well. >> This summer we need to upgrade the varnish version to handle very long >> header (a product requirement). >> So after a short battle porting our vcl to vcl4.0 we start using varnish >> 4. >> Shortly after thing begun to goes very bad. >> >> The first issue we hit, is a memory exhaustion on both stage, and >> oom-killer... >> We test a lot of things, and in the battle we upgrade to varnish5. >> We fix it, resizing the pool, and using now file backend (from memory >> before). >> Memory is now stable (we have large pool, 32G, and strange thing, we >> never have object being nuke, which it good or bad it depend). >> We have also fix a lot of things in our vcl. >> >> The problem we fight against now is only on the stage1 varnish, and >> specifically on one pool (the busiest one). >> When everything goes well the average cpu usage is 30%, memory stabilize >> around 12G, hit cache is around 0.85. >> Problem happen randomly (not everyday) but during our peaks. The cpu >> increase fasly to reach 350% (4 core) and load > 3/ >> When the problem is here varnish still deliver requests (we didn't see >> dropped or reject connections) but our application begin to lost user, >> including a big lot of business. I suspect this is because timeout are very >> aggressive on the client side and varnish should answer slowly >> >> -first question : how see response time of request of the varnish server >> ?. (varnishnsca something ?) >> >> I also suspect some kind of request queuing, also stracing varnish when >> it happen show a lot of futex wait ?!. >> The frustrating part is restarting varnish fix the problem immediately, >> and the cpu remains normal after, even if the trafic peak is not finish. >> So there is clearly something stacked in varnish which cause our problem. >> >> -second question : how to see number of stacked connections, long >> connections and so on ? >> >> At this stage we accept all kind of help / hints for debuging (and >> regarding the business impact we can evaluate the help of a professional >> support) >> >> PS : I always have the option to scale out, popping a lot of new varnish >> instance, but this seems very frustrating... >> >> Best, >> >> -- >> Raphael Mazelier >> >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From info+varnish at shee.org Thu Nov 16 11:57:13 2017 From: info+varnish at shee.org (info+varnish at shee.org) Date: Thu, 16 Nov 2017 12:57:13 +0100 Subject: backend web node keep-alive option In-Reply-To: References: <76A6807F-2821-4D0E-9E4D-C00CA22BA947@shee.org> Message-ID: Am 12.11.2017 um 15:42 schrieb Guillaume Quintard : > > On Sat, Nov 11, 2017 at 9:48 PM, wrote: > >> I was observing a behavior of a PHP script on one backend server that got executed/requested >> twice (but not by the client/browser). >> >> The plain script just processes data and only outputs a response until its done. The observed >> behavior is; when first_byte_timeout passes but the script is still processing data, the varnish >> node answers with an 503 and the script gets listed twice in the webserver process list. So, either >> it gets requested on this event again or the TCP connection close triggers an new execution. In any >> case its a bug and because of missing resources to dive into a more deeper analyses, I just disabled >> the enabled keep-alive option of the backend webserver (apache/httpd). The results: it helped to avoid >> a further execution of the script. >> >> The question is: What is the best practice to configure the backend webservers? Do you keep the >> keep-alive option enabled or not? >> Hi Guillaume, > > Why not just up the timeout value? Sure, the mentioned script(task) is for my taste conceptually bad designed and I could convinced the developer to run some kind of a task queue. The primary focus of my question targeted the keep-alive part but while here: does it have negative implications having e.g. a 2h first_byte_timeout configuration? Is this common? > Usually, you want to keep the keep-alive as it's much more efficient. Also sure, but my big picture while asking was - heavy traffic. While the costs of the TCP handshake persists, the scenario is at the backend of the architecture where high bandwidth links exits. So, i can imagine that in heavy traffic scenarios, the keep-alive feature would lead to more memory consumption, more file descriptors and be more prone to denial of service attacks. Any experiences out there? -- Thanks, Leon From info+varnish at shee.org Thu Nov 16 12:05:48 2017 From: info+varnish at shee.org (info+varnish at shee.org) Date: Thu, 16 Nov 2017 13:05:48 +0100 Subject: backend web node keep-alive option In-Reply-To: References: <76A6807F-2821-4D0E-9E4D-C00CA22BA947@shee.org> Message-ID: > Am 13.11.2017 um 08:11 schrieb Dridi Boukelmoune : > > On Sat, Nov 11, 2017 at 9:48 PM, wrote: >> I was observing a behavior of a PHP script on one backend server that got executed/requested >> twice (but not by the client/browser). >> >> The plain script just processes data and only outputs a response until its done. The observed >> behavior is; when first_byte_timeout passes but the script is still processing data, the varnish >> node answers with an 503 and the script gets listed twice in the webserver process list. So, either >> it gets requested on this event again or the TCP connection close triggers an new execution. In any >> case its a bug and because of missing resources to dive into a more deeper analyses, I just disabled >> the enabled keep-alive option of the backend webserver (apache/httpd). The results: it helped to avoid >> a further execution of the script. >> >> The question is: What is the best practice to configure the backend webservers? Do you keep the >> keep-alive option enabled or not? >> >> I'd appreciate hearing your suggestions. > > I think you're getting a retry from varnishd, independent from VCL's > return(retry) which only happens on GET/HEAD requests that are > supposed to be side-effect-free and safely retry-able. On top of that > you might be hitting a bug where Varnish retries more than once. > Seeing some log would help confirm this hypothesis. > > https://github.com/varnishcache/varnish-cache/pull/2135 Ah, okay. Is this still valid for version 5.1/5.2? The mentioned script doesn't exist anymore. I will try to spend some time to reconstruct this case. -- Thanks, Leon From jmathiesen at tripadvisor.com Fri Nov 17 14:54:03 2017 From: jmathiesen at tripadvisor.com (James Mathiesen) Date: Fri, 17 Nov 2017 14:54:03 +0000 Subject: Question about an LRU error Message-ID: This is running RPM varnish-4.1.8-1.el7.x86_64 in a kubernetes container. We had user complaints that a binary object (~100MB) was coming back truncated on every fetch. Fetching the object I saw the following in the varnish logs: - ExpKill LRU_Cand p=0x7feffe41abc0 f=0x0 r=1 - ExpKill LRU x=435945906 - ExpKill LRU_Cand p=0x7ff018c63880 f=0x0 r=1 - ExpKill LRU x=423595256 - ExpKill LRU_Cand p=0x7ff0129314c0 f=0x0 r=1 - ExpKill LRU x=432965548 - ExpKill LRU_Exhausted - FetchError Could not get storage - BackendClose 29 boot.default - BereqAcct 260 0 260 399 2789376 2789775 - End Which seems consistent with the symptom -- the backend transfer starts and gets streamed to the client but partway through the transfer the backend and frontend connections are reset. The problem is going to be difficult to reproduce as the symptom only appears after the container has been running for months. The dataset is media objects of various sizes (from tiny thumbnails to a few hundred MB) and the cache available is 5GB. The cache is much, much smaller than the working set. The object we noticed the problem with is about 100MB. Before I spent a lot of time gathering further data I wanted to understand if I'm hitting a known behavior and whether there's any value to anyone if I try and gather more information vs. just planning an upgrade to 5.2 and seeing if the problem goes away. james -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Fri Nov 17 15:10:07 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Fri, 17 Nov 2017 16:10:07 +0100 Subject: Question about an LRU error In-Reply-To: References: Message-ID: Hello James, Varnish hit the lru limit, meaning it delete N objects to make room for the new one, and decided it was enough. N is called "nuke_limit", you can change it at launch time using the -p switch. If your objects are indeed very diverse in size, it makes sense to have multiple storages, one for small content, and another for larger content. That way you won't need to go through a million objects to push a new one in cache. -- Guillaume Quintard On Fri, Nov 17, 2017 at 3:54 PM, James Mathiesen wrote: > This is running RPM varnish-4.1.8-1.el7.x86_64 in a kubernetes container. > > > > We had user complaints that a binary object (~100MB) was coming back > truncated on every fetch. > > > > Fetching the object I saw the following in the varnish logs: > > > > - ExpKill LRU_Cand p=0x7feffe41abc0 f=0x0 r=1 > > - ExpKill LRU x=435945906 > > - ExpKill LRU_Cand p=0x7ff018c63880 f=0x0 r=1 > > - ExpKill LRU x=423595256 > > - ExpKill LRU_Cand p=0x7ff0129314c0 f=0x0 r=1 > > - ExpKill LRU x=432965548 > > - ExpKill LRU_Exhausted > > - FetchError Could not get storage > > - BackendClose 29 boot.default > > - BereqAcct 260 0 260 399 2789376 2789775 > > - End > > > > Which seems consistent with the symptom -- the backend transfer starts and > gets streamed to the client but partway through the transfer the backend > and frontend connections are reset. > > > > The problem is going to be difficult to reproduce as the symptom only > appears after the container has been running for months. > > > > The dataset is media objects of various sizes (from tiny thumbnails to a > few hundred MB) and the cache available is 5GB. The cache is much, much > smaller than the working set. The object we noticed the problem with is > about 100MB. > > > > Before I spent a lot of time gathering further data I wanted to understand > if I'm hitting a known behavior and whether there's any value to anyone if > I try and gather more information vs. just planning an upgrade to 5.2 and > seeing if the problem goes away. > > > > james > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jmathiesen at tripadvisor.com Fri Nov 17 15:13:07 2017 From: jmathiesen at tripadvisor.com (James Mathiesen) Date: Fri, 17 Nov 2017 15:13:07 +0000 Subject: Question about an LRU error In-Reply-To: References: Message-ID: <78AB5F3C-A407-410D-B791-BFF62CFF733C@tripadvisor.com> Hi Guillaume, Thanks very much for your response. I'll look into the multiple cache option. james From: Guillaume Quintard Date: Friday, November 17, 2017 at 10:10 AM To: James Mathiesen Cc: "varnish-misc at varnish-cache.org" Subject: Re: Question about an LRU error Hello James, Varnish hit the lru limit, meaning it delete N objects to make room for the new one, and decided it was enough. N is called "nuke_limit", you can change it at launch time using the -p switch. If your objects are indeed very diverse in size, it makes sense to have multiple storages, one for small content, and another for larger content. That way you won't need to go through a million objects to push a new one in cache. -- Guillaume Quintard On Fri, Nov 17, 2017 at 3:54 PM, James Mathiesen > wrote: This is running RPM varnish-4.1.8-1.el7.x86_64 in a kubernetes container. We had user complaints that a binary object (~100MB) was coming back truncated on every fetch. Fetching the object I saw the following in the varnish logs: - ExpKill LRU_Cand p=0x7feffe41abc0 f=0x0 r=1 - ExpKill LRU x=435945906 - ExpKill LRU_Cand p=0x7ff018c63880 f=0x0 r=1 - ExpKill LRU x=423595256 - ExpKill LRU_Cand p=0x7ff0129314c0 f=0x0 r=1 - ExpKill LRU x=432965548 - ExpKill LRU_Exhausted - FetchError Could not get storage - BackendClose 29 boot.default - BereqAcct 260 0 260 399 2789376 2789775 - End Which seems consistent with the symptom -- the backend transfer starts and gets streamed to the client but partway through the transfer the backend and frontend connections are reset. The problem is going to be difficult to reproduce as the symptom only appears after the container has been running for months. The dataset is media objects of various sizes (from tiny thumbnails to a few hundred MB) and the cache available is 5GB. The cache is much, much smaller than the working set. The object we noticed the problem with is about 100MB. Before I spent a lot of time gathering further data I wanted to understand if I'm hitting a known behavior and whether there's any value to anyone if I try and gather more information vs. just planning an upgrade to 5.2 and seeing if the problem goes away. james _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From noelle at uni-wuppertal.de Tue Nov 21 07:59:38 2017 From: noelle at uni-wuppertal.de (=?UTF-8?Q?Christian_N=c3=b6lle?=) Date: Tue, 21 Nov 2017 08:59:38 +0100 Subject: munin plugin not working with 5.2 In-Reply-To: References: Message-ID: <09db60c6-522b-9293-f573-acd7d9ee8068@uni-wuppertal.de> Am 28.09.2017 um 12:39 schrieb Christian N?lle: > after upgrading to 5.2 from 5.1.x our munin plugin stopped working. Are > the changes to the stats system - announced in the release notes for 5.2 > - the cause? Does anyone know where to get a working version? Is no one using 5.2 and munin? I filed a bug with munin, see https://github.com/munin-monitoring/contrib/issues/876, addressing the potential problems but I am not able to fix this. Is there any maintainer of that munin plugin? We would really appreciate it - otherwise we have to switch back to 5.1... :/ Greeetings! -- -c -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5397 bytes Desc: S/MIME Cryptographic Signature URL: From n.premkumar.me at gmail.com Wed Nov 22 05:49:04 2017 From: n.premkumar.me at gmail.com (Prem Kumar) Date: Wed, 22 Nov 2017 11:19:04 +0530 Subject: why default_header part of vrt_backend is removed in varnish 5 Message-ID: Hi All, I'm using default headers specific to backend type in 3.1. For example backend 1 ,backend2 where default header for backend1 will be "xyz" and backend2 will be "abc". Now I don't find a way to add these headers in bereq. because I don't know what is the backend chosen and value should be added for a header is like adding multiple if else condition in the vcl_backend_featch. Looks like if i have more number of backends ,it will not scale. Please any could explain why this is been dropped in 5 and how could it be handled? Thanks, Prem -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Wed Nov 22 08:15:09 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 22 Nov 2017 09:15:09 +0100 Subject: why default_header part of vrt_backend is removed in varnish 5 In-Reply-To: References: Message-ID: On Wed, Nov 22, 2017 at 6:49 AM, Prem Kumar wrote: > Hi All, > > > I'm using default headers specific to backend type in 3.1. For example > backend 1 ,backend2 where default header for backend1 will be "xyz" and > backend2 will be "abc". > Now I don't find a way to add these headers in bereq. because I don't know > what is the backend chosen and value should be added for a header is like > adding multiple if else condition in the vcl_backend_featch. Looks like if i > have more number of backends ,it will not scale. > Please any could explain why this is been dropped in 5 and how could it be > handled? Hello, Not sure to understand, are you referring to `.host_header`? It's still arround. You mention Varnish 3.1 but there is no such release, are you upgrading from 2.1 or 3.0? Dridi From dridi at varni.sh Wed Nov 22 08:18:38 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 22 Nov 2017 09:18:38 +0100 Subject: backend web node keep-alive option In-Reply-To: References: <76A6807F-2821-4D0E-9E4D-C00CA22BA947@shee.org> Message-ID: >> I think you're getting a retry from varnishd, independent from VCL's >> return(retry) which only happens on GET/HEAD requests that are >> supposed to be side-effect-free and safely retry-able. On top of that >> you might be hitting a bug where Varnish retries more than once. >> Seeing some log would help confirm this hypothesis. >> >> https://github.com/varnishcache/varnish-cache/pull/2135 > > > Ah, okay. Is this still valid for version 5.1/5.2? Yes, but fixed two days ago so it will be part of the next release. https://github.com/varnishcache/varnish-cache/commit/fa3884391f85565cb7d564cfa5f22d066948d7bc > The mentioned script doesn't exist anymore. I will try to spend some time to reconstruct this case. Happy to see you worked around the problem on your side. Cheers From raph at futomaki.net Thu Nov 23 20:57:46 2017 From: raph at futomaki.net (Raphael Mazelier) Date: Thu, 23 Nov 2017 21:57:46 +0100 Subject: Varnish nighmare after upgrading : need help In-Reply-To: <89923832-10d5-2a3f-617e-22e9a9f31f7b@futomaki.net> References: <620367f1-0247-b457-f65c-08fc898ea6b3@futomaki.net> <89923832-10d5-2a3f-617e-22e9a9f31f7b@futomaki.net> Message-ID: <1a28c6e7-29b2-3998-b7d4-049d0eaf1fec@futomaki.net> > > > On 14/11/2017 23:41, Guillaume Quintard wrote: >> Hi, >> >> Let's look at the usual suspects first, can we get the output of "ps >> aux |grep varnish" and a pastebin of "varnishncsa -1"? >> >> Are you using any vmod? >> >> man varnishncsa will help craft a format line with the response time >> (on mobile now, I don't have access to it) >> >> Cheers, Hi All, A short following of the situation and what seems to mitigate the problem/make this platform works. After lot of testing and A/B testing, the solution for us was to make more smaller instances. We basically double all the servers(vms) , but in the other hand divide by two (or more) the ram, and the memory allocated to varnish. We also revert using malloc with little ram (4G) , 12g on the vm(s). We also make scheduled task to flush the cache (restarting varnish). This is a completely counter intuitive, because nuking some entries seems better to handle a big cache with no nuke. In my understating it means that our hot content remains in the cache, and nuking object is OK. This may also means that our ttl in objects are completly wrong. Anyway it seems working. Thanks a lot for the people who help us. (and I'm sure we can find a way to re-tribute this). Best, PS : in the battle we upgrade to 5.2 (seems ok), just a quick question on varnihstat. I read the changelog and understand that the varnishstat communication with varnish have change. I have just a little glith. When launching varnishstat the hitrate counter and average reset randomly at the beginning, and after a while stabilize. This is a bit annoying because we often want to have a quick view on the hitrate. -- Raphael Mazelier From matrix at matrix2000.name Mon Nov 27 11:15:24 2017 From: matrix at matrix2000.name (matrix) Date: Mon, 27 Nov 2017 20:15:24 +0900 Subject: vmod_header can be used in vcl_backend_fetch? Message-ID: I would like to control bereq header below.(varnish4.1.8) header.append(bereq.http.Accept-Encoding, "aaa"); header.append(bereq.http.Accept-Encoding, "bbb"); But bereq.http.Accept-Encoding header is "aaa". (not "aaa,bbb") is it possible to handle the bereq using vmod_header ? or is it varnish-modules bug? Best regards. From guillaume at varnish-software.com Mon Nov 27 11:21:38 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 27 Nov 2017 12:21:38 +0100 Subject: vmod_header can be used in vcl_backend_fetch? In-Reply-To: References: Message-ID: Hi, I would expect to have two AE headers, one with aaa and one with bbb, would you have the varnishlog of that particular bereq? Also, it's probably out of scope, but are you actually changing the Accept-Encoding header, or is that just for the example? -- Guillaume Quintard On Mon, Nov 27, 2017 at 12:15 PM, matrix wrote: > I would like to control bereq header below.(varnish4.1.8) > > header.append(bereq.http.Accept-Encoding, "aaa"); > header.append(bereq.http.Accept-Encoding, "bbb"); > > But bereq.http.Accept-Encoding header is "aaa". (not "aaa,bbb") > is it possible to handle the bereq using vmod_header ? > > or is it varnish-modules bug? > > Best regards. > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matrix at matrix2000.name Mon Nov 27 12:04:21 2017 From: matrix at matrix2000.name (matrix) Date: Mon, 27 Nov 2017 21:04:21 +0900 Subject: vmod_header can be used in vcl_backend_fetch? In-Reply-To: References: Message-ID: hi thank you for your reply According to varnishlog Bereq header are displayed but origin server log, only Accept-Encoding: aaaa . - BereqHeader Accept-Encoding: aaaa - BereqHeader Accept-Encoding: bbbb I would like to change Accept-Encoding Header to br,gzip, because the varnish is force replaced Accept-Encoding gzip when backend request. VCL unset bereq.http.Accept-Encoding; header.append(bereq.http.Accept-Encoding, "aaaa"); header.append(bereq.http.Accept-Encoding, "bbbb"); Full varnishlog below * << BeReq >> 3 - Begin bereq 2 fetch - Timestamp Start: 1511783436.988432 0.000000 0.000000 - BereqMethod GET - BereqURL /cache/200.php - BereqProtocol HTTP/1.1 - BereqHeader User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 - BereqHeader Accept: */* - BereqHeader host: test.com - BereqHeader Accept-Encoding: gzip - BereqHeader Surrogate-Capability: key=ESI/1.0 - BereqHeader X-Cache-Key: /cache/200.php - BereqHeader X-Varnish: 3 - VCL_call BACKEND_FETCH - BereqUnset Accept-Encoding: gzip - BereqHeader Accept-Encoding: aaaa - BereqHeader Accept-Encoding: bbbb - VCL_return fetch - BackendOpen 21 boot.backend1 127.0.0.1 81 127.0.0.1 51782 - BackendStart 127.0.0.1 81 - Timestamp Bereq: 1511783436.988549 0.000118 0.000118 - Timestamp Beresp: 1511783437.198227 0.209795 0.209677 - BerespProtocol HTTP/1.1 - BerespStatus 404 - BerespReason Not Found - BerespHeader Date: Mon, 27 Nov 2017 11:50:37 GMT - BerespHeader Content-Type: text/html; charset=UTF-8 - BerespHeader Transfer-Encoding: chunked - BerespHeader Connection: keep-alive - BerespHeader Vary: Accept-Encoding - TTL RFC 86400 10 -1 1511783437 1511783437 1511783437 0 0 - VCL_call BACKEND_RESPONSE - TTL VCL 300 10 0 1511783437 - VCL_return deliver - Storage file disk - ObjProtocol HTTP/1.1 - ObjStatus 404 - ObjReason Not Found - ObjHeader Date: Mon, 27 Nov 2017 11:50:37 GMT - ObjHeader Content-Type: text/html; charset=UTF-8 - ObjHeader Vary: Accept-Encoding - Fetch_Body 2 chunked stream - BackendReuse 21 boot.backend1 - Timestamp BerespBody: 1511783437.198340 0.209908 0.000113 - Length 16 - BereqAcct 569 0 569 178 16 194 - End * << Request >> 2 - Begin req 1 rxreq - Timestamp Start: 1511783436.988227 0.000000 0.000000 - Timestamp Req: 1511783436.988227 0.000000 0.000000 - ReqStart x.x.x.x 62656 - ReqMethod GET - ReqURL /cache/200.php?1234 - ReqProtocol HTTP/1.1 - ReqHeader User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 - ReqHeader Accept: */* - ReqHeader Host:test.com - ReqHeader Accept-Encoding: gzip,deflate - VCL_call RECV - ReqUnset Host:test.com - ReqHeader host: test.com - ReqUnset Accept-Encoding: gzip,deflate - ReqHeader Accept-Encoding: gzip - ReqHeader Surrogate-Capability: key=ESI/1.0 - ReqURL /cache/200.php - VCL_return hash - VCL_call HASH - ReqHeader X-Cache-Key: /cache/200.php - VCL_Log redhash - ReqUnset X-Cache-Key: /cache/200.php - ReqHeader X-Cache-Key: /cache/200.php - VCL_return lookup - VCL_call MISS - VCL_Log redmiss - VCL_return fetch - Link bereq 3 fetch - Timestamp Fetch: 1511783437.198397 0.210170 0.210170 - RespProtocol HTTP/1.1 - RespStatus 404 - RespReason Not Found - RespHeader Date: Mon, 27 Nov 2017 11:50:37 GMT - RespHeader Content-Type: text/html; charset=UTF-8 - RespHeader Vary: Accept-Encoding - RespHeader X-Varnish: 2 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_Log reddeliver - RespUnset X-Varnish: 2 - RespUnset Via: 1.1 varnish-v4 - VCL_return deliver - Timestamp Process: 1511783437.198471 0.210244 0.000074 - RespHeader Content-Length: 16 - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1511783437.198515 0.210288 0.000044 - ReqAcct 219 0 219 282 16 298 - End * << Session >> 1 - Begin sess 0 HTTP/1 - SessOpen x.x.x.x 62656 0.0.0.0:80 172.26.12.250 80 1511783436.987762 20 - Link req 2 rxreq - SessClose REM_CLOSE 0.217 - End 2017-11-27 20:21 GMT+09:00 Guillaume Quintard : > Hi, > > I would expect to have two AE headers, one with aaa and one with bbb, would > you have the varnishlog of that particular bereq? > > Also, it's probably out of scope, but are you actually changing the > Accept-Encoding header, or is that just for the example? > > -- > Guillaume Quintard > > On Mon, Nov 27, 2017 at 12:15 PM, matrix wrote: >> >> I would like to control bereq header below.(varnish4.1.8) >> >> header.append(bereq.http.Accept-Encoding, "aaa"); >> header.append(bereq.http.Accept-Encoding, "bbb"); >> >> But bereq.http.Accept-Encoding header is "aaa". (not "aaa,bbb") >> is it possible to handle the bereq using vmod_header ? >> >> or is it varnish-modules bug? >> >> Best regards. >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > From geoff at uplex.de Mon Nov 27 12:21:49 2017 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 27 Nov 2017 13:21:49 +0100 Subject: vmod_header can be used in vcl_backend_fetch? In-Reply-To: References: Message-ID: On 11/27/2017 01:04 PM, matrix wrote: > > According to varnishlog Bereq header are displayed but origin server > log, only Accept-Encoding: aaaa . > > - BereqHeader Accept-Encoding: aaaa > - BereqHeader Accept-Encoding: bbbb > > I would like to change Accept-Encoding Header to br,gzip, because the > varnish is force replaced Accept-Encoding gzip when backend request. > > VCL > unset bereq.http.Accept-Encoding; > header.append(bereq.http.Accept-Encoding, "aaaa"); > header.append(bereq.http.Accept-Encoding, "bbbb"); After the header.append()s you can use: std.collect(bereq.http.Accept-Encoding) ... to combine them into a single header with comma-separated values. https://varnish-cache.org/docs/5.2/reference/vmod_std.generated.html#func-collect Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From guillaume at varnish-software.com Mon Nov 27 12:23:42 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 27 Nov 2017 13:23:42 +0100 Subject: vmod_header can be used in vcl_backend_fetch? In-Reply-To: References: Message-ID: So the varnishlog confirms the vmods works as expected: - BereqUnset Accept-Encoding: gzip - BereqHeader Accept-Encoding: aaaa - BereqHeader Accept-Encoding: bbbb bereq is "BackEnd REQuest", if you want to change the "BackEnd RESPonse", use beresp.http.*. And you can simply to: set bereq.http.accept-encoding = "br, gzip"; Just know that while varnish natively support gzip, it's not the case for brotli. You can have a look at this blog post: https://info.varnish-software.com/blog/varnish-cache-brotli-compression -- Guillaume Quintard On Mon, Nov 27, 2017 at 1:04 PM, matrix wrote: > hi thank you for your reply > > According to varnishlog Bereq header are displayed but origin server > log, only Accept-Encoding: aaaa . > > - BereqHeader Accept-Encoding: aaaa > - BereqHeader Accept-Encoding: bbbb > > I would like to change Accept-Encoding Header to br,gzip, because the > varnish is force replaced Accept-Encoding gzip when backend request. > > VCL > unset bereq.http.Accept-Encoding; > header.append(bereq.http.Accept-Encoding, "aaaa"); > header.append(bereq.http.Accept-Encoding, "bbbb"); > > > Full varnishlog below > > * << BeReq >> 3 > - Begin bereq 2 fetch > - Timestamp Start: 1511783436.988432 0.000000 0.000000 > - BereqMethod GET > - BereqURL /cache/200.php > - BereqProtocol HTTP/1.1 > - BereqHeader User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) > libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > - BereqHeader Accept: */* > - BereqHeader host: test.com > - BereqHeader Accept-Encoding: gzip > - BereqHeader Surrogate-Capability: key=ESI/1.0 > - BereqHeader X-Cache-Key: /cache/200.php > - BereqHeader X-Varnish: 3 > - VCL_call BACKEND_FETCH > - BereqUnset Accept-Encoding: gzip > - BereqHeader Accept-Encoding: aaaa > - BereqHeader Accept-Encoding: bbbb > - VCL_return fetch > - BackendOpen 21 boot.backend1 127.0.0.1 81 127.0.0.1 51782 > - BackendStart 127.0.0.1 81 > - Timestamp Bereq: 1511783436.988549 0.000118 0.000118 > - Timestamp Beresp: 1511783437.198227 0.209795 0.209677 > - BerespProtocol HTTP/1.1 > - BerespStatus 404 > - BerespReason Not Found > - BerespHeader Date: Mon, 27 Nov 2017 11:50:37 GMT > - BerespHeader Content-Type: text/html; charset=UTF-8 > - BerespHeader Transfer-Encoding: chunked > - BerespHeader Connection: keep-alive > - BerespHeader Vary: Accept-Encoding > - TTL RFC 86400 10 -1 1511783437 1511783437 1511783437 0 0 > - VCL_call BACKEND_RESPONSE > - TTL VCL 300 10 0 1511783437 > - VCL_return deliver > - Storage file disk > - ObjProtocol HTTP/1.1 > - ObjStatus 404 > - ObjReason Not Found > - ObjHeader Date: Mon, 27 Nov 2017 11:50:37 GMT > - ObjHeader Content-Type: text/html; charset=UTF-8 > - ObjHeader Vary: Accept-Encoding > - Fetch_Body 2 chunked stream > - BackendReuse 21 boot.backend1 > - Timestamp BerespBody: 1511783437.198340 0.209908 0.000113 > - Length 16 > - BereqAcct 569 0 569 178 16 194 > - End > > * << Request >> 2 > - Begin req 1 rxreq > - Timestamp Start: 1511783436.988227 0.000000 0.000000 > - Timestamp Req: 1511783436.988227 0.000000 0.000000 > - ReqStart x.x.x.x 62656 > - ReqMethod GET > - ReqURL /cache/200.php?1234 > - ReqProtocol HTTP/1.1 > - ReqHeader User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) > libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > - ReqHeader Accept: */* > - ReqHeader Host:test.com > - ReqHeader Accept-Encoding: gzip,deflate > - VCL_call RECV > - ReqUnset Host:test.com > - ReqHeader host: test.com > - ReqUnset Accept-Encoding: gzip,deflate > - ReqHeader Accept-Encoding: gzip > - ReqHeader Surrogate-Capability: key=ESI/1.0 > - ReqURL /cache/200.php > - VCL_return hash > - VCL_call HASH > - ReqHeader X-Cache-Key: /cache/200.php > - VCL_Log redhash > - ReqUnset X-Cache-Key: /cache/200.php > - ReqHeader X-Cache-Key: /cache/200.php > - VCL_return lookup > - VCL_call MISS > - VCL_Log redmiss > - VCL_return fetch > - Link bereq 3 fetch > - Timestamp Fetch: 1511783437.198397 0.210170 0.210170 > - RespProtocol HTTP/1.1 > - RespStatus 404 > - RespReason Not Found > - RespHeader Date: Mon, 27 Nov 2017 11:50:37 GMT > - RespHeader Content-Type: text/html; charset=UTF-8 > - RespHeader Vary: Accept-Encoding > - RespHeader X-Varnish: 2 > - RespHeader Age: 0 > - RespHeader Via: 1.1 varnish-v4 > - VCL_call DELIVER > - VCL_Log reddeliver > - RespUnset X-Varnish: 2 > - RespUnset Via: 1.1 varnish-v4 > - VCL_return deliver > - Timestamp Process: 1511783437.198471 0.210244 0.000074 > - RespHeader Content-Length: 16 > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1511783437.198515 0.210288 0.000044 > - ReqAcct 219 0 219 282 16 298 > - End > > * << Session >> 1 > - Begin sess 0 HTTP/1 > - SessOpen x.x.x.x 62656 0.0.0.0:80 172.26.12.250 80 > 1511783436.987762 20 > - Link req 2 rxreq > - SessClose REM_CLOSE 0.217 > - End > > > 2017-11-27 20:21 GMT+09:00 Guillaume Quintard com>: > > Hi, > > > > I would expect to have two AE headers, one with aaa and one with bbb, > would > > you have the varnishlog of that particular bereq? > > > > Also, it's probably out of scope, but are you actually changing the > > Accept-Encoding header, or is that just for the example? > > > > -- > > Guillaume Quintard > > > > On Mon, Nov 27, 2017 at 12:15 PM, matrix wrote: > >> > >> I would like to control bereq header below.(varnish4.1.8) > >> > >> header.append(bereq.http.Accept-Encoding, "aaa"); > >> header.append(bereq.http.Accept-Encoding, "bbb"); > >> > >> But bereq.http.Accept-Encoding header is "aaa". (not "aaa,bbb") > >> is it possible to handle the bereq using vmod_header ? > >> > >> or is it varnish-modules bug? > >> > >> Best regards. > >> _______________________________________________ > >> varnish-misc mailing list > >> varnish-misc at varnish-cache.org > >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matrix at matrix2000.name Tue Nov 28 07:03:15 2017 From: matrix at matrix2000.name (matrix) Date: Tue, 28 Nov 2017 16:03:15 +0900 Subject: vmod_header can be used in vcl_backend_fetch? In-Reply-To: References: Message-ID: thank you for checking. I would like to make it clear about vmod header action. Does ?header append? mean to append value of each header? or to append the entire header ? I complete thought that handled value of existing header below. header.append(bereq.http.Accept-Encoding, "aaaa"); Vary:http.Accept-Encoding"aaaa" header.append(bereq.http.Accept-Encoding, "bbbb"); Vary:http.Accept-Encoding"aaaa,bbbb" *thank you for information about br. I already check! 2017-11-27 21:23 GMT+09:00 Guillaume Quintard : > So the varnishlog confirms the vmods works as expected: > > - BereqUnset Accept-Encoding: gzip > - BereqHeader Accept-Encoding: aaaa > - BereqHeader Accept-Encoding: bbbb > > bereq is "BackEnd REQuest", if you want to change the "BackEnd RESPonse", > use beresp.http.*. > > And you can simply to: > > set bereq.http.accept-encoding = "br, gzip"; > > Just know that while varnish natively support gzip, it's not the case for > brotli. You can have a look at this blog post: > https://info.varnish-software.com/blog/varnish-cache-brotli-compression > > -- > Guillaume Quintard > > On Mon, Nov 27, 2017 at 1:04 PM, matrix wrote: >> >> hi thank you for your reply >> >> According to varnishlog Bereq header are displayed but origin server >> log, only Accept-Encoding: aaaa . >> >> - BereqHeader Accept-Encoding: aaaa >> - BereqHeader Accept-Encoding: bbbb >> >> I would like to change Accept-Encoding Header to br,gzip, because the >> varnish is force replaced Accept-Encoding gzip when backend request. >> >> VCL >> unset bereq.http.Accept-Encoding; >> header.append(bereq.http.Accept-Encoding, "aaaa"); >> header.append(bereq.http.Accept-Encoding, "bbbb"); >> >> >> Full varnishlog below >> >> * << BeReq >> 3 >> - Begin bereq 2 fetch >> - Timestamp Start: 1511783436.988432 0.000000 0.000000 >> - BereqMethod GET >> - BereqURL /cache/200.php >> - BereqProtocol HTTP/1.1 >> - BereqHeader User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) >> libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 >> - BereqHeader Accept: */* >> - BereqHeader host: test.com >> - BereqHeader Accept-Encoding: gzip >> - BereqHeader Surrogate-Capability: key=ESI/1.0 >> - BereqHeader X-Cache-Key: /cache/200.php >> - BereqHeader X-Varnish: 3 >> - VCL_call BACKEND_FETCH >> - BereqUnset Accept-Encoding: gzip >> - BereqHeader Accept-Encoding: aaaa >> - BereqHeader Accept-Encoding: bbbb >> - VCL_return fetch >> - BackendOpen 21 boot.backend1 127.0.0.1 81 127.0.0.1 51782 >> - BackendStart 127.0.0.1 81 >> - Timestamp Bereq: 1511783436.988549 0.000118 0.000118 >> - Timestamp Beresp: 1511783437.198227 0.209795 0.209677 >> - BerespProtocol HTTP/1.1 >> - BerespStatus 404 >> - BerespReason Not Found >> - BerespHeader Date: Mon, 27 Nov 2017 11:50:37 GMT >> - BerespHeader Content-Type: text/html; charset=UTF-8 >> - BerespHeader Transfer-Encoding: chunked >> - BerespHeader Connection: keep-alive >> - BerespHeader Vary: Accept-Encoding >> - TTL RFC 86400 10 -1 1511783437 1511783437 1511783437 0 0 >> - VCL_call BACKEND_RESPONSE >> - TTL VCL 300 10 0 1511783437 >> - VCL_return deliver >> - Storage file disk >> - ObjProtocol HTTP/1.1 >> - ObjStatus 404 >> - ObjReason Not Found >> - ObjHeader Date: Mon, 27 Nov 2017 11:50:37 GMT >> - ObjHeader Content-Type: text/html; charset=UTF-8 >> - ObjHeader Vary: Accept-Encoding >> - Fetch_Body 2 chunked stream >> - BackendReuse 21 boot.backend1 >> - Timestamp BerespBody: 1511783437.198340 0.209908 0.000113 >> - Length 16 >> - BereqAcct 569 0 569 178 16 194 >> - End >> >> * << Request >> 2 >> - Begin req 1 rxreq >> - Timestamp Start: 1511783436.988227 0.000000 0.000000 >> - Timestamp Req: 1511783436.988227 0.000000 0.000000 >> - ReqStart x.x.x.x 62656 >> - ReqMethod GET >> - ReqURL /cache/200.php?1234 >> - ReqProtocol HTTP/1.1 >> - ReqHeader User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) >> libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 >> - ReqHeader Accept: */* >> - ReqHeader Host:test.com >> - ReqHeader Accept-Encoding: gzip,deflate >> - VCL_call RECV >> - ReqUnset Host:test.com >> - ReqHeader host: test.com >> - ReqUnset Accept-Encoding: gzip,deflate >> - ReqHeader Accept-Encoding: gzip >> - ReqHeader Surrogate-Capability: key=ESI/1.0 >> - ReqURL /cache/200.php >> - VCL_return hash >> - VCL_call HASH >> - ReqHeader X-Cache-Key: /cache/200.php >> - VCL_Log redhash >> - ReqUnset X-Cache-Key: /cache/200.php >> - ReqHeader X-Cache-Key: /cache/200.php >> - VCL_return lookup >> - VCL_call MISS >> - VCL_Log redmiss >> - VCL_return fetch >> - Link bereq 3 fetch >> - Timestamp Fetch: 1511783437.198397 0.210170 0.210170 >> - RespProtocol HTTP/1.1 >> - RespStatus 404 >> - RespReason Not Found >> - RespHeader Date: Mon, 27 Nov 2017 11:50:37 GMT >> - RespHeader Content-Type: text/html; charset=UTF-8 >> - RespHeader Vary: Accept-Encoding >> - RespHeader X-Varnish: 2 >> - RespHeader Age: 0 >> - RespHeader Via: 1.1 varnish-v4 >> - VCL_call DELIVER >> - VCL_Log reddeliver >> - RespUnset X-Varnish: 2 >> - RespUnset Via: 1.1 varnish-v4 >> - VCL_return deliver >> - Timestamp Process: 1511783437.198471 0.210244 0.000074 >> - RespHeader Content-Length: 16 >> - Debug "RES_MODE 2" >> - RespHeader Connection: keep-alive >> - Timestamp Resp: 1511783437.198515 0.210288 0.000044 >> - ReqAcct 219 0 219 282 16 298 >> - End >> >> * << Session >> 1 >> - Begin sess 0 HTTP/1 >> - SessOpen x.x.x.x 62656 0.0.0.0:80 172.26.12.250 80 >> 1511783436.987762 20 >> - Link req 2 rxreq >> - SessClose REM_CLOSE 0.217 >> - End >> >> >> 2017-11-27 20:21 GMT+09:00 Guillaume Quintard >> : >> > Hi, >> > >> > I would expect to have two AE headers, one with aaa and one with bbb, >> > would >> > you have the varnishlog of that particular bereq? >> > >> > Also, it's probably out of scope, but are you actually changing the >> > Accept-Encoding header, or is that just for the example? >> > >> > -- >> > Guillaume Quintard >> > >> > On Mon, Nov 27, 2017 at 12:15 PM, matrix wrote: >> >> >> >> I would like to control bereq header below.(varnish4.1.8) >> >> >> >> header.append(bereq.http.Accept-Encoding, "aaa"); >> >> header.append(bereq.http.Accept-Encoding, "bbb"); >> >> >> >> But bereq.http.Accept-Encoding header is "aaa". (not "aaa,bbb") >> >> is it possible to handle the bereq using vmod_header ? >> >> >> >> or is it varnish-modules bug? >> >> >> >> Best regards. >> >> _______________________________________________ >> >> varnish-misc mailing list >> >> varnish-misc at varnish-cache.org >> >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > >> > > > From guillaume at varnish-software.com Tue Nov 28 09:03:01 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Tue, 28 Nov 2017 10:03:01 +0100 Subject: vmod_header can be used in vcl_backend_fetch? In-Reply-To: References: Message-ID: No: https://github.com/varnish/varnish-modules/blob/master/src/vmod_header.vcc#L35 Check ``man vmod_headers`` for more info. -- Guillaume Quintard On Tue, Nov 28, 2017 at 8:03 AM, matrix wrote: > thank you for checking. > > I would like to make it clear about vmod header action. > Does ?header append? mean to append value of each header? or to append > the entire header ? > > I complete thought that handled value of existing header below. > > header.append(bereq.http.Accept-Encoding, "aaaa"); > > Vary:http.Accept-Encoding"aaaa" > > header.append(bereq.http.Accept-Encoding, "bbbb"); > > Vary:http.Accept-Encoding"aaaa,bbbb" > > *thank you for information about br. I already check! > > > > 2017-11-27 21:23 GMT+09:00 Guillaume Quintard com>: > > So the varnishlog confirms the vmods works as expected: > > > > - BereqUnset Accept-Encoding: gzip > > - BereqHeader Accept-Encoding: aaaa > > - BereqHeader Accept-Encoding: bbbb > > > > bereq is "BackEnd REQuest", if you want to change the "BackEnd RESPonse", > > use beresp.http.*. > > > > And you can simply to: > > > > set bereq.http.accept-encoding = "br, gzip"; > > > > Just know that while varnish natively support gzip, it's not the case for > > brotli. You can have a look at this blog post: > > https://info.varnish-software.com/blog/varnish-cache-brotli-compression > > > > -- > > Guillaume Quintard > > > > On Mon, Nov 27, 2017 at 1:04 PM, matrix wrote: > >> > >> hi thank you for your reply > >> > >> According to varnishlog Bereq header are displayed but origin server > >> log, only Accept-Encoding: aaaa . > >> > >> - BereqHeader Accept-Encoding: aaaa > >> - BereqHeader Accept-Encoding: bbbb > >> > >> I would like to change Accept-Encoding Header to br,gzip, because the > >> varnish is force replaced Accept-Encoding gzip when backend request. > >> > >> VCL > >> unset bereq.http.Accept-Encoding; > >> header.append(bereq.http.Accept-Encoding, "aaaa"); > >> header.append(bereq.http.Accept-Encoding, "bbbb"); > >> > >> > >> Full varnishlog below > >> > >> * << BeReq >> 3 > >> - Begin bereq 2 fetch > >> - Timestamp Start: 1511783436.988432 0.000000 0.000000 > >> - BereqMethod GET > >> - BereqURL /cache/200.php > >> - BereqProtocol HTTP/1.1 > >> - BereqHeader User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) > >> libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > >> - BereqHeader Accept: */* > >> - BereqHeader host: test.com > >> - BereqHeader Accept-Encoding: gzip > >> - BereqHeader Surrogate-Capability: key=ESI/1.0 > >> - BereqHeader X-Cache-Key: /cache/200.php > >> - BereqHeader X-Varnish: 3 > >> - VCL_call BACKEND_FETCH > >> - BereqUnset Accept-Encoding: gzip > >> - BereqHeader Accept-Encoding: aaaa > >> - BereqHeader Accept-Encoding: bbbb > >> - VCL_return fetch > >> - BackendOpen 21 boot.backend1 127.0.0.1 81 127.0.0.1 51782 > >> - BackendStart 127.0.0.1 81 > >> - Timestamp Bereq: 1511783436.988549 0.000118 0.000118 > >> - Timestamp Beresp: 1511783437.198227 0.209795 0.209677 > >> - BerespProtocol HTTP/1.1 > >> - BerespStatus 404 > >> - BerespReason Not Found > >> - BerespHeader Date: Mon, 27 Nov 2017 11:50:37 GMT > >> - BerespHeader Content-Type: text/html; charset=UTF-8 > >> - BerespHeader Transfer-Encoding: chunked > >> - BerespHeader Connection: keep-alive > >> - BerespHeader Vary: Accept-Encoding > >> - TTL RFC 86400 10 -1 1511783437 1511783437 1511783437 0 0 > >> - VCL_call BACKEND_RESPONSE > >> - TTL VCL 300 10 0 1511783437 > >> - VCL_return deliver > >> - Storage file disk > >> - ObjProtocol HTTP/1.1 > >> - ObjStatus 404 > >> - ObjReason Not Found > >> - ObjHeader Date: Mon, 27 Nov 2017 11:50:37 GMT > >> - ObjHeader Content-Type: text/html; charset=UTF-8 > >> - ObjHeader Vary: Accept-Encoding > >> - Fetch_Body 2 chunked stream > >> - BackendReuse 21 boot.backend1 > >> - Timestamp BerespBody: 1511783437.198340 0.209908 0.000113 > >> - Length 16 > >> - BereqAcct 569 0 569 178 16 194 > >> - End > >> > >> * << Request >> 2 > >> - Begin req 1 rxreq > >> - Timestamp Start: 1511783436.988227 0.000000 0.000000 > >> - Timestamp Req: 1511783436.988227 0.000000 0.000000 > >> - ReqStart x.x.x.x 62656 > >> - ReqMethod GET > >> - ReqURL /cache/200.php?1234 > >> - ReqProtocol HTTP/1.1 > >> - ReqHeader User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) > >> libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > >> - ReqHeader Accept: */* > >> - ReqHeader Host:test.com > >> - ReqHeader Accept-Encoding: gzip,deflate > >> - VCL_call RECV > >> - ReqUnset Host:test.com > >> - ReqHeader host: test.com > >> - ReqUnset Accept-Encoding: gzip,deflate > >> - ReqHeader Accept-Encoding: gzip > >> - ReqHeader Surrogate-Capability: key=ESI/1.0 > >> - ReqURL /cache/200.php > >> - VCL_return hash > >> - VCL_call HASH > >> - ReqHeader X-Cache-Key: /cache/200.php > >> - VCL_Log redhash > >> - ReqUnset X-Cache-Key: /cache/200.php > >> - ReqHeader X-Cache-Key: /cache/200.php > >> - VCL_return lookup > >> - VCL_call MISS > >> - VCL_Log redmiss > >> - VCL_return fetch > >> - Link bereq 3 fetch > >> - Timestamp Fetch: 1511783437.198397 0.210170 0.210170 > >> - RespProtocol HTTP/1.1 > >> - RespStatus 404 > >> - RespReason Not Found > >> - RespHeader Date: Mon, 27 Nov 2017 11:50:37 GMT > >> - RespHeader Content-Type: text/html; charset=UTF-8 > >> - RespHeader Vary: Accept-Encoding > >> - RespHeader X-Varnish: 2 > >> - RespHeader Age: 0 > >> - RespHeader Via: 1.1 varnish-v4 > >> - VCL_call DELIVER > >> - VCL_Log reddeliver > >> - RespUnset X-Varnish: 2 > >> - RespUnset Via: 1.1 varnish-v4 > >> - VCL_return deliver > >> - Timestamp Process: 1511783437.198471 0.210244 0.000074 > >> - RespHeader Content-Length: 16 > >> - Debug "RES_MODE 2" > >> - RespHeader Connection: keep-alive > >> - Timestamp Resp: 1511783437.198515 0.210288 0.000044 > >> - ReqAcct 219 0 219 282 16 298 > >> - End > >> > >> * << Session >> 1 > >> - Begin sess 0 HTTP/1 > >> - SessOpen x.x.x.x 62656 0.0.0.0:80 172.26.12.250 80 > >> 1511783436.987762 20 > >> - Link req 2 rxreq > >> - SessClose REM_CLOSE 0.217 > >> - End > >> > >> > >> 2017-11-27 20:21 GMT+09:00 Guillaume Quintard > >> : > >> > Hi, > >> > > >> > I would expect to have two AE headers, one with aaa and one with bbb, > >> > would > >> > you have the varnishlog of that particular bereq? > >> > > >> > Also, it's probably out of scope, but are you actually changing the > >> > Accept-Encoding header, or is that just for the example? > >> > > >> > -- > >> > Guillaume Quintard > >> > > >> > On Mon, Nov 27, 2017 at 12:15 PM, matrix > wrote: > >> >> > >> >> I would like to control bereq header below.(varnish4.1.8) > >> >> > >> >> header.append(bereq.http.Accept-Encoding, "aaa"); > >> >> header.append(bereq.http.Accept-Encoding, "bbb"); > >> >> > >> >> But bereq.http.Accept-Encoding header is "aaa". (not "aaa,bbb") > >> >> is it possible to handle the bereq using vmod_header ? > >> >> > >> >> or is it varnish-modules bug? > >> >> > >> >> Best regards. > >> >> _______________________________________________ > >> >> varnish-misc mailing list > >> >> varnish-misc at varnish-cache.org > >> >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > >> > > >> > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From matrix at matrix2000.name Tue Nov 28 09:39:08 2017 From: matrix at matrix2000.name (matrix) Date: Tue, 28 Nov 2017 18:39:08 +0900 Subject: vmod_header can be used in vcl_backend_fetch? In-Reply-To: References: Message-ID: I was misunderstanding vmod_header action. The Vmod is correct action. thank you Guillaume ! 2017-11-28 18:03 GMT+09:00 Guillaume Quintard : > No: > https://github.com/varnish/varnish-modules/blob/master/src/vmod_header.vcc#L35 > > Check ``man vmod_headers`` for more info. > > -- > Guillaume Quintard > > On Tue, Nov 28, 2017 at 8:03 AM, matrix wrote: >> >> thank you for checking. >> >> I would like to make it clear about vmod header action. >> Does ?header append? mean to append value of each header? or to append >> the entire header ? >> >> I complete thought that handled value of existing header below. >> >> header.append(bereq.http.Accept-Encoding, "aaaa"); >> >> Vary:http.Accept-Encoding"aaaa" >> >> header.append(bereq.http.Accept-Encoding, "bbbb"); >> >> Vary:http.Accept-Encoding"aaaa,bbbb" >> >> *thank you for information about br. I already check! >> >> >> >> 2017-11-27 21:23 GMT+09:00 Guillaume Quintard >> : >> > So the varnishlog confirms the vmods works as expected: >> > >> > - BereqUnset Accept-Encoding: gzip >> > - BereqHeader Accept-Encoding: aaaa >> > - BereqHeader Accept-Encoding: bbbb >> > >> > bereq is "BackEnd REQuest", if you want to change the "BackEnd >> > RESPonse", >> > use beresp.http.*. >> > >> > And you can simply to: >> > >> > set bereq.http.accept-encoding = "br, gzip"; >> > >> > Just know that while varnish natively support gzip, it's not the case >> > for >> > brotli. You can have a look at this blog post: >> > https://info.varnish-software.com/blog/varnish-cache-brotli-compression >> > >> > -- >> > Guillaume Quintard >> > >> > On Mon, Nov 27, 2017 at 1:04 PM, matrix wrote: >> >> >> >> hi thank you for your reply >> >> >> >> According to varnishlog Bereq header are displayed but origin server >> >> log, only Accept-Encoding: aaaa . >> >> >> >> - BereqHeader Accept-Encoding: aaaa >> >> - BereqHeader Accept-Encoding: bbbb >> >> >> >> I would like to change Accept-Encoding Header to br,gzip, because the >> >> varnish is force replaced Accept-Encoding gzip when backend request. >> >> >> >> VCL >> >> unset bereq.http.Accept-Encoding; >> >> header.append(bereq.http.Accept-Encoding, "aaaa"); >> >> header.append(bereq.http.Accept-Encoding, "bbbb"); >> >> >> >> >> >> Full varnishlog below >> >> >> >> * << BeReq >> 3 >> >> - Begin bereq 2 fetch >> >> - Timestamp Start: 1511783436.988432 0.000000 0.000000 >> >> - BereqMethod GET >> >> - BereqURL /cache/200.php >> >> - BereqProtocol HTTP/1.1 >> >> - BereqHeader User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) >> >> libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 >> >> - BereqHeader Accept: */* >> >> - BereqHeader host: test.com >> >> - BereqHeader Accept-Encoding: gzip >> >> - BereqHeader Surrogate-Capability: key=ESI/1.0 >> >> - BereqHeader X-Cache-Key: /cache/200.php >> >> - BereqHeader X-Varnish: 3 >> >> - VCL_call BACKEND_FETCH >> >> - BereqUnset Accept-Encoding: gzip >> >> - BereqHeader Accept-Encoding: aaaa >> >> - BereqHeader Accept-Encoding: bbbb >> >> - VCL_return fetch >> >> - BackendOpen 21 boot.backend1 127.0.0.1 81 127.0.0.1 51782 >> >> - BackendStart 127.0.0.1 81 >> >> - Timestamp Bereq: 1511783436.988549 0.000118 0.000118 >> >> - Timestamp Beresp: 1511783437.198227 0.209795 0.209677 >> >> - BerespProtocol HTTP/1.1 >> >> - BerespStatus 404 >> >> - BerespReason Not Found >> >> - BerespHeader Date: Mon, 27 Nov 2017 11:50:37 GMT >> >> - BerespHeader Content-Type: text/html; charset=UTF-8 >> >> - BerespHeader Transfer-Encoding: chunked >> >> - BerespHeader Connection: keep-alive >> >> - BerespHeader Vary: Accept-Encoding >> >> - TTL RFC 86400 10 -1 1511783437 1511783437 1511783437 0 0 >> >> - VCL_call BACKEND_RESPONSE >> >> - TTL VCL 300 10 0 1511783437 >> >> - VCL_return deliver >> >> - Storage file disk >> >> - ObjProtocol HTTP/1.1 >> >> - ObjStatus 404 >> >> - ObjReason Not Found >> >> - ObjHeader Date: Mon, 27 Nov 2017 11:50:37 GMT >> >> - ObjHeader Content-Type: text/html; charset=UTF-8 >> >> - ObjHeader Vary: Accept-Encoding >> >> - Fetch_Body 2 chunked stream >> >> - BackendReuse 21 boot.backend1 >> >> - Timestamp BerespBody: 1511783437.198340 0.209908 0.000113 >> >> - Length 16 >> >> - BereqAcct 569 0 569 178 16 194 >> >> - End >> >> >> >> * << Request >> 2 >> >> - Begin req 1 rxreq >> >> - Timestamp Start: 1511783436.988227 0.000000 0.000000 >> >> - Timestamp Req: 1511783436.988227 0.000000 0.000000 >> >> - ReqStart x.x.x.x 62656 >> >> - ReqMethod GET >> >> - ReqURL /cache/200.php?1234 >> >> - ReqProtocol HTTP/1.1 >> >> - ReqHeader User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) >> >> libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 >> >> - ReqHeader Accept: */* >> >> - ReqHeader Host:test.com >> >> - ReqHeader Accept-Encoding: gzip,deflate >> >> - VCL_call RECV >> >> - ReqUnset Host:test.com >> >> - ReqHeader host: test.com >> >> - ReqUnset Accept-Encoding: gzip,deflate >> >> - ReqHeader Accept-Encoding: gzip >> >> - ReqHeader Surrogate-Capability: key=ESI/1.0 >> >> - ReqURL /cache/200.php >> >> - VCL_return hash >> >> - VCL_call HASH >> >> - ReqHeader X-Cache-Key: /cache/200.php >> >> - VCL_Log redhash >> >> - ReqUnset X-Cache-Key: /cache/200.php >> >> - ReqHeader X-Cache-Key: /cache/200.php >> >> - VCL_return lookup >> >> - VCL_call MISS >> >> - VCL_Log redmiss >> >> - VCL_return fetch >> >> - Link bereq 3 fetch >> >> - Timestamp Fetch: 1511783437.198397 0.210170 0.210170 >> >> - RespProtocol HTTP/1.1 >> >> - RespStatus 404 >> >> - RespReason Not Found >> >> - RespHeader Date: Mon, 27 Nov 2017 11:50:37 GMT >> >> - RespHeader Content-Type: text/html; charset=UTF-8 >> >> - RespHeader Vary: Accept-Encoding >> >> - RespHeader X-Varnish: 2 >> >> - RespHeader Age: 0 >> >> - RespHeader Via: 1.1 varnish-v4 >> >> - VCL_call DELIVER >> >> - VCL_Log reddeliver >> >> - RespUnset X-Varnish: 2 >> >> - RespUnset Via: 1.1 varnish-v4 >> >> - VCL_return deliver >> >> - Timestamp Process: 1511783437.198471 0.210244 0.000074 >> >> - RespHeader Content-Length: 16 >> >> - Debug "RES_MODE 2" >> >> - RespHeader Connection: keep-alive >> >> - Timestamp Resp: 1511783437.198515 0.210288 0.000044 >> >> - ReqAcct 219 0 219 282 16 298 >> >> - End >> >> >> >> * << Session >> 1 >> >> - Begin sess 0 HTTP/1 >> >> - SessOpen x.x.x.x 62656 0.0.0.0:80 172.26.12.250 80 >> >> 1511783436.987762 20 >> >> - Link req 2 rxreq >> >> - SessClose REM_CLOSE 0.217 >> >> - End >> >> >> >> >> >> 2017-11-27 20:21 GMT+09:00 Guillaume Quintard >> >> : >> >> > Hi, >> >> > >> >> > I would expect to have two AE headers, one with aaa and one with bbb, >> >> > would >> >> > you have the varnishlog of that particular bereq? >> >> > >> >> > Also, it's probably out of scope, but are you actually changing the >> >> > Accept-Encoding header, or is that just for the example? >> >> > >> >> > -- >> >> > Guillaume Quintard >> >> > >> >> > On Mon, Nov 27, 2017 at 12:15 PM, matrix >> >> > wrote: >> >> >> >> >> >> I would like to control bereq header below.(varnish4.1.8) >> >> >> >> >> >> header.append(bereq.http.Accept-Encoding, "aaa"); >> >> >> header.append(bereq.http.Accept-Encoding, "bbb"); >> >> >> >> >> >> But bereq.http.Accept-Encoding header is "aaa". (not "aaa,bbb") >> >> >> is it possible to handle the bereq using vmod_header ? >> >> >> >> >> >> or is it varnish-modules bug? >> >> >> >> >> >> Best regards. >> >> >> _______________________________________________ >> >> >> varnish-misc mailing list >> >> >> varnish-misc at varnish-cache.org >> >> >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> >> > >> >> > >> > >> > > > From hugues at betabrand.com Tue Nov 28 19:48:15 2017 From: hugues at betabrand.com (Hugues Alary) Date: Tue, 28 Nov 2017 11:48:15 -0800 Subject: Varnish 5.2.0 child panic Message-ID: Hi there! In the last 18 days, varnish reports its child died 8 times. We had more traffic than usual those past few days, but I don't think it was anything Varnish can't handle. We run Varnish on Kubernetes on GCP. I was able to retrieve the last panic via varnishadm (see below). The other panics are the same as the one below (same assertion failing, same line number). (I would send them too, but it's amazingly difficult to get logs out of google cloud). If the panic is not enough, I can send our configuration file on request. Panic at: Tue, 28 Nov 2017 13:44:30 GMT Assert error in HSH_Lookup(), cache/cache_hash.c line 432: Condition((vary) != 0) not true. version = varnish-5.2.0 revision 4c4875cbf, vrt api = 6.1 ident = Linux,4.4.64+,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll now = 1755374.451854 (mono), 1511876668.180978 (real) Backtrace: 0x556f4d169e36: varnishd(+0x4ae36) [0x556f4d169e36] 0x556f4d1b4b80: varnishd(VAS_Fail+0x40) [0x556f4d1b4b80] 0x556f4d15f1b2: varnishd(HSH_Lookup+0xcb2) [0x556f4d15f1b2] 0x556f4d16e14f: varnishd(CNT_Request+0xedf) [0x556f4d16e14f] 0x556f4d18dda2: varnishd(+0x6eda2) [0x556f4d18dda2] 0x556f4d18525c: varnishd(+0x6625c) [0x556f4d18525c] 0x556f4d185780: varnishd(+0x66780) [0x556f4d185780] 0x7f13aa27c494: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7494) [0x7f13aa27c494] 0x7f13a9fbeaff: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f13a9fbeaff] thread = (cache-worker) thr.req = 0x7f0c615bf020 { vxid = 59420066, transport = HTTP/1 { state = HTTP1::Proc } step = R_STP_LOOKUP, req_body = R_BODY_NONE, restarts = 0, esi_level = 0, sp = 0x7f12be06de20 { fd = 82, vxid = 59420065, t_open = 1511876667.587429, t_idle = 1511876667.587429, transport = HTTP/1 { state = HTTP1::Proc } client = 10.44.11.6 50566, privs = 0x7f12be06de88 { }, }, worker = 0x7f1374734dd0 { stack = {0x7f1374735000 -> 0x7f13746a2000}, ws = 0x7f1374734e78 { id = \"wrk\", {s, f, r, e} = {0x7f1374734190, +0, (nil), +2040}, }, VCL::method = DELIVER, VCL::return = deliver, VCL::methods = {}, }, ws = 0x7f0c615bf208 { id = \"req\", {s, f, r, e} = {0x7f0c615c1008, +3920, +516080, +516080}, }, http_conn = 0x7f0c615bf130 { fd = 82 (@0x7f12be06de38), doclose = NULL, ws = 0x7f0c615bf208 { [Already dumped, see above] }, {rxbuf_b, rxbuf_e} = {0x7f0c615c1008, 0x7f0c615c1e27}, {pipeline_b, pipeline_e} = {(nil), (nil)}, content_length = -1, body_status = none, first_byte_timeout = 0.000000, between_bytes_timeout = 0.000000, }, http[req] = 0x7f0c615bf2a0 { ws = 0x7f0c615bf208 { [Already dumped, see above] }, hdrs { \"GET\", \"/api/rest/reviews/product/6430\", \"HTTP/1.1\", \"Host: www.betabrand.com\", \"Accept-Encoding: gzip\", \"CF-IPCountry: US\", \"CF-RAY: 3c4dc30d9aae55b2-ORD\", \"CF-Visitor: {\"scheme\":\"https\"}\", \"user-agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_0_3 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A432 Safari/604.1\", \"accept-language: en-us\", \"referer: https://www.betabrand.com/womens/pants/womens-black-boot-flare-dress-pant-yoga-pants\ ", \"cookie: optimizelyPendingLogEvents=%5B%22n%3Dimage_impression%26u%3Doeu1511755011199r0.44405056492852124%26wxhr%3Dtrue%26time%3D1511876665.858%26f%3D8443224466%2C8247617066%2C8411440184%2C7780900332%2C8106662898%26g%3D%22%5D; sailthru_content=d9d08cd819b052bb46ae11d6f4f23820f9f47e5805766810fea9aeef193727f0f94155821eef8e7bdb5607beb554d9f12fd3caaaf239d3b03d10e1dc4b5fe514; NaN_hash=d1015888FXVD1484095712135476; __utma=207700493.554833224.1511755020.1511755021.1511876545.2; __utmb=207700493.5.10.1511876545; __utmc=207700493; __utmz=207700493.1511876545.2.2.utmcsr=Google|utmgclid=EAIaIQobChMIwbet_7Lh1wIVk7fACh0pGAaaEAAYASAAEgLFrPD_BwE|utmccn=Brand_General|utmcmd=Retention|utmctr=(not%20provided); _ga=GA1.2.554833224.1511755020; _gac_UA-17580748-1=1.1511876545.EAIaIQobChMIwbet_7Lh1wIVk7fACh0pGAaaEAAYASAAEgLFrPD_BwE; _gac_UA-17580748-8=1.1511876545.EAIaIQobChMIwbet_7Lh1wIVk7fACh0pGAaaEAAYASAAEgLFrPD_BwE; _gid=GA1.2.1787521728.1511876545; _hp2_id.777713614=%7B%22userId%22%3A%221402506040644741%22%2C%22pageviewId%22%3A%226182215729724641%22%2C%22sessionId%22%3A%224386307704701158%22%2C%22identity%22%3Anull%2C%22trackerVersion%22%3A%223.0%22%7D; _hp2_ses_props.777713614=%7B%22r%22%3A%22https%3A%2F%2Fwww.google.com %2F%22%2C%22us%22%3A%22Google%22%2C%22um%22%3A%22Retention%22%2C%22ua%22%3A%22Brand_General%22%2C%22ts%22%3A1511876545351%2C%22d%22%3A% 22www.betabrand.com%22%2C%22h%22%3A%22%2Fsale%22%7D; _sp_id.ac93=0115679b-fdb3-417c-9994-b1387273d811.1511755021.2.1511876663.1511755037.2396b5db-5ad9-4e6e-9061-f19c13780fc6; _sp_ses.ac93=*; optimizelyBuckets=%7B%228443224466%22%3A%228449793114%22%2C%228247617066%22%3A%228249041800%22%2C%228411440184%22%3A%228407050657%22%2C%228106662898%22%3A%228109860945%22%7D; _gat_UA-17580748-8=1; frontend=deadbeefdeadbeefdeadbeefaa; __zlcmid=jhgvWpO31r9clR; __utmt_UA-17580748-1=1; _uetsid=_ueta633ebda; _bbli=0; fs_uid=www.fullstory.com`1bTW`4950547299041280:5668600916475904; uuid=740c5206-78d1-4902-8896-a5e8e90eb7c8; bb-seed=0.01224735359031548; betabrand-campaign-first=Brand_General,Google,Retention,undefined,undefined,Tue Nov 28 2017 07:42:23 GMT-0600 (CST); betabrand-campaign-last=Brand_General,Google,Retention,undefined,undefined,Tue Nov 28 2017 07:42:23 GMT-0600 (CST); betabrand-campaign-session=Brand_General,Google,Retention,undefined,undefined,Tue Nov 28 2017 07:42:23 GMT-0600 (CST); optimizelyEndUserId=oeu1511755011199r0.44405056492852124; optimizelySegments=%7B%22172841198%22%3A%22campaign%22%2C%22172864896%22%3A%22safari%22%2C%22173083061%22%3A%22true%22%2C%223972941021%22%3A%22brand_general%22%7D; __qca=P0-1874833837-1511755021423; sailthru_visitor=b78950af-8d6d-4cb6-8d54-a497f97dd6ba; betabrand-timing=1; 3060738.3440491=e5d3965e-bb79-4cde-b307-d532ffdec44c; tracker_device=34b4b287-87e9-4418-bef5-28ab27d9038c; EcommTest1=2; __cfduid=dd480648d530b3001597f1e574200f3611511755011\", \"CF-Connecting-IP: 73.44.212.19\", \"X-Forwarded-Proto: https\", \"Connection: close\", \"X-Request-Start: t=1511876667586\", \"X-Queue-Start: t=1511876667586\", \"X-Unique-ID: 0A80002E:8771_0A2C0B06:01BB_5A1D683B_F1ABD22:0009\", \"X-Forwarded-For: 73.44.212.19, 10.128.0.46, 10.44.11.6, 10.44.11.6\", \"X-PSA-Blocking-Rewrite: betabrand-pagespeed\", \"Accept: application/json\", }, }, vcl = { name = \"boot\", busy = 92, discard = 0, state = auto, temp = warm, conf = { srcname = { \"/etc/varnish/default.vcl\", \"Builtin\", }, }, }, vmods = { std = {Varnish 5.2.0 4c4875cbf, 0.0}, directors = {Varnish 5.2.0 4c4875cbf, 0.0}, }, flags = { }, }, thr.busyobj = (nil) { }, Thanks for your help! Cheers, -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Tue Nov 28 21:12:31 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Tue, 28 Nov 2017 22:12:31 +0100 Subject: Max_esi_depth and nested Esi In-Reply-To: References: Message-ID: Moving to varnish-misc -- Guillaume Quintard On Nov 28, 2017 7:23 PM, "Jos? Mar?a C.A." wrote: > Hi, I am working with varnish 4.1.8. Im using nested esis and the default > level was 5. > I made several html components that they need 9 depth levels and I have > changed the max_esi_depth to 9. My question is: which are the consequences > of this parameter regarding to this increment of level, maybe increase the > processing time per esi? > > Thnaks in advance ?? > > _______________________________________________ > varnish-dev mailing list > varnish-dev at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugues at betabrand.com Tue Nov 28 23:35:49 2017 From: hugues at betabrand.com (Hugues Alary) Date: Tue, 28 Nov 2017 15:35:49 -0800 Subject: Varnish 5.2.0 child panic In-Reply-To: References: Message-ID: Just realized this might be better as a bug report, I'll submit one if needed. Also, I just had another panic: Panic at: Tue, 28 Nov 2017 22:18:49 GMT Assert error in HSH_Lookup(), cache/cache_hash.c line 432: Condition((vary) != 0) not true. version = varnish-5.2.0 revision 4c4875cbf, vrt api = 6.1 ident = Linux,4.4.64+,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll now = 1786235.707578 (mono), 1511907529.436702 (real) Backtrace: 0x556f4d169e36: varnishd(+0x4ae36) [0x556f4d169e36] 0x556f4d1b4b80: varnishd(VAS_Fail+0x40) [0x556f4d1b4b80] 0x556f4d15f1b2: varnishd(HSH_Lookup+0xcb2) [0x556f4d15f1b2] 0x556f4d16e14f: varnishd(CNT_Request+0xedf) [0x556f4d16e14f] 0x556f4d18dda2: varnishd(+0x6eda2) [0x556f4d18dda2] 0x556f4d18525c: varnishd(+0x6625c) [0x556f4d18525c] 0x556f4d185780: varnishd(+0x66780) [0x556f4d185780] 0x7f13aa27c494: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7494) [0x7f13aa27c494] 0x7f13a9fbeaff: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f13a9fbeaff] thread = (cache-worker) thr.req = 0x7f1245e0a020 { vxid = 2869347, transport = HTTP/1 { state = HTTP1::Proc } step = R_STP_LOOKUP, req_body = R_BODY_NONE, restarts = 0, esi_level = 0, sp = 0x7f137a00ea20 { fd = 65, vxid = 2869346, t_open = 1511907453.132658, t_idle = 1511907453.132658, transport = HTTP/1 { state = HTTP1::Proc } client = 10.44.43.4 45520, privs = 0x7f137a00ea88 { }, }, worker = 0x7f1388213dd0 { stack = {0x7f1388214000 -> 0x7f1388181000}, ws = 0x7f1388213e78 { id = \"wrk\", {s, f, r, e} = {0x7f1388213190, +0, (nil), +2040}, }, VCL::method = DELIVER, VCL::return = deliver, VCL::methods = {}, }, ws = 0x7f1245e0a208 { id = \"req\", {s, f, r, e} = {0x7f1245e0c008, +4144, +516080, +516080}, }, http_conn = 0x7f1245e0a130 { fd = 65 (@0x7f137a00ea38), doclose = NULL, ws = 0x7f1245e0a208 { [Already dumped, see above] }, {rxbuf_b, rxbuf_e} = {0x7f1245e0c008, 0x7f1245e0cf01}, {pipeline_b, pipeline_e} = {(nil), (nil)}, content_length = -1, body_status = none, first_byte_timeout = 0.000000, between_bytes_timeout = 0.000000, }, http[req] = 0x7f1245e0a2a0 { ws = 0x7f1245e0a208 { [Already dumped, see above] }, hdrs { \"GET\", \"/api/rest/reviews/product/6430\", \"HTTP/1.1\", \"Host: www.betabrand.com\", \"Accept-Encoding: gzip\", \"CF-IPCountry: US\", \"CF-RAY: 3c50b2adfddd5a56-BOS\", \"CF-Visitor: {\"scheme\":\"https\"}\", \"user-agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_1_2 like Mac OS X) AppleWebKit/604.3.5 (KHTML, like Gecko) Mobile/15B202 [FBAN/FBIOS;FBAV/150.0.0.32.132;FBBV/80278251;FBDV/iPhone9,1;FBMD/iPhone;FBSN/iOS;FBSV/11.1.2;FBSS/2;FBCR/Verizon;FBID/phone;FBLC/en_US;FBOP/5;FBRV/0]\", \"accept-language: en-us\", \"referer: https://www.betabrand.com/womens/pants/dress-pant-yoga-pants-collection/womens-black-boot-flare-dress-pant-yoga-pants\ ", \"cookie: NaN_hash=d7023912CNFF1437826184182031; __utma=207700493.1027173649.1511907207.1511907209.1511907209.1; __utmb=207700493.16.10.1511907209; __utmc=207700493; __utmz=207700493.1511907209.1.1.utmcsr=Facebook|utmccn=BlackSheepWrap_Booyah|utmcmd=Acquisition|utmctr=ACQ_WMN; _ga=GA1.2.1027173649.1511907207; _gat_UA-17580748-8=1; _gid=GA1.2.363913430.1511907207; _sp_id.ac93=16411174-95e0-4636-a008-9e3cba433d59.1511907211.1.1511907405.1511907211.756129b3-5717-460b-98c9-c7597c30c819; _sp_ses.ac93=*; optimizelyBuckets=%7B%223353291595%22%3A%223339030766%22%2C%223397350584%22%3A%223376220538%22%2C%223528387311%22%3A%223528387312%22%2C%228443224466%22%3A%228449793114%22%2C%228247617066%22%3A%228240095367%22%2C%228411440184%22%3A%228414131467%22%2C%228106662898%22%3A%228109860945%22%7D; sailthru_content=bfba62eaf5e186fe0219ddc33264f66df9f47e5805766810fea9aeef193727f0d9d08cd819b052bb46ae11d6f4f2382032527e3cf48859c6a3a1764af88774438fc9ab569691ee6c97a8146c72e75c31c8d1082e407b7c337fe9b9af43106beb9eb5c66e1957b87d5165561970fe74c12fd3caaaf239d3b03d10e1dc4b5fe514; _hp2_id.777713614=%7B%22userId%22%3A%220743002369627522%22%2C%22pageviewId%22%3A%225740794725429989%22%2C%22sessionId%22%3A%221322112537315924%22%2C%22identity%22%3Anull%2C%22trackerVersion%22%3A%223.0%22%7D; _hp2_ses_props.777713614=%7B%22r%22%3A%22http%3A%2F%2Fm.facebook.com %22%2C%22us%22%3A%22Facebook%22%2C%22um%22%3A%22Acquisition%22%2C%22ut%22%3A%22ACQ_WMN%22%2C%22ua%22%3A%22BlackSheepWrap_Booyah%22%2C%22ts%22%3A1511907219368%2C%22d%22%3A% 22www.betabrand.com%22%2C%22h%22%3A%22%2Fwomens%2Fpants%2Fdress-pant-yoga-pants-collection%22%7D; __zlcmid=jigvnRPnf0yTRh; _uetsid=_uetf208ea35; _bbli=0; fs_uid= www.fullstory.com`1bTW`5677711459876864:5629499534213120; frontend=deadbeefdeadbeefdeadbeefaa; uuid=a1d09da5-7506-41c2-94b6-6b5acdba0f9c; betabrand-campaign-last=BlackSheepWrap_Booyah,Facebook,Acquisition,ACQ_WMN,undefined,Tue Nov 28 2017 17:14:57 GMT-0500 (EST); betabrand-campaign-session=BlackSheepWrap_Booyah,Facebook,Acquisition,ACQ_WMN,undefined,Tue Nov 28 2017 17:14:57 GMT-0500 (EST); optimizelyEndUserId=oeu1444061797597r0.2150009439792484; optimizelySegments=%7B%22172841198%22%3A%22campaign%22%2C%22172864896%22%3A%22unknown%22%2C%22173083061%22%3A%22true%22%2C%223972941021%22%3A%22blacksheepwrap_booya%22%7D; sailthru_visitor=94d5d458-c946-4228-9886-1b57e519beeb; 3060738.3440491=67c18f82-2818-44ef-ab9b-f2667055fada; __qca=P0-1450722180-1511907220479; tracker_device=ce48cfcc-5b82-483b-8a1b-562959838be8; __utmt_UA-17580748-1=1; betabrand-timing=1; EcommTest1=2; betabrand-campaign-first=BlackSheepWrap_Booyah,Facebook,Acquisition,ACQ_WMN,undefined,Tue Nov 28 2017 17:13:23 GMT-0500 (EST); betabrand-introduction=true; __cfduid=dcc19056430e6c8dfd790803695cfb1bf1511907198; _gr_ep_sent=1; _gr_er_sent=1; granify.lasts at 809=1444061806057; granify.uuid=53534c95-2046-47f9-b9f8-556dbd6e4b4d; a1ashgd=c9de5eb25c19369cd2a04d2babab5888\", \"CF-Connecting-IP: 208.64.112.35\", \"X-Forwarded-Proto: https\", \"Connection: close\", \"X-Request-Start: t=1511907453131\", \"X-Queue-Start: t=1511907453131\", \"X-Unique-ID: 0A800027:8911_0A2C2B04:01BB_5A1DE07D_30A0ADB:0009\", \"X-Forwarded-For: 208.64.112.35, 10.128.0.39, 10.44.43.4, 10.44.43.4\", \"X-PSA-Blocking-Rewrite: betabrand-pagespeed\", \"Accept: application/json\", }, }, vcl = { name = \"boot\", busy = 135, discard = 0, state = auto, temp = warm, conf = { srcname = { \"/etc/varnish/default.vcl\", \"Builtin\", }, }, }, vmods = { std = {Varnish 5.2.0 4c4875cbf, 0.0}, directors = {Varnish 5.2.0 4c4875cbf, 0.0}, }, flags = { }, }, thr.busyobj = (nil) { }, On Tue, Nov 28, 2017 at 11:48 AM, Hugues Alary wrote: > Hi there! > > In the last 18 days, varnish reports its child died 8 times. > > We had more traffic than usual those past few days, but I don't think it > was anything Varnish can't handle. > > We run Varnish on Kubernetes on GCP. > > I was able to retrieve the last panic via varnishadm (see below). The > other panics are the same as the one below (same assertion failing, same > line number). (I would send them too, but it's amazingly difficult to get > logs out of google cloud). > > If the panic is not enough, I can send our configuration file on request. > > Panic at: Tue, 28 Nov 2017 13:44:30 GMT > Assert error in HSH_Lookup(), cache/cache_hash.c line 432: > Condition((vary) != 0) not true. > version = varnish-5.2.0 revision 4c4875cbf, vrt api = 6.1 > ident = Linux,4.4.64+,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll > now = 1755374.451854 (mono), 1511876668.180978 (real) > Backtrace: > 0x556f4d169e36: varnishd(+0x4ae36) [0x556f4d169e36] > 0x556f4d1b4b80: varnishd(VAS_Fail+0x40) [0x556f4d1b4b80] > 0x556f4d15f1b2: varnishd(HSH_Lookup+0xcb2) [0x556f4d15f1b2] > 0x556f4d16e14f: varnishd(CNT_Request+0xedf) [0x556f4d16e14f] > 0x556f4d18dda2: varnishd(+0x6eda2) [0x556f4d18dda2] > 0x556f4d18525c: varnishd(+0x6625c) [0x556f4d18525c] > 0x556f4d185780: varnishd(+0x66780) [0x556f4d185780] > 0x7f13aa27c494: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7494) > [0x7f13aa27c494] > 0x7f13a9fbeaff: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) > [0x7f13a9fbeaff] > thread = (cache-worker) > thr.req = 0x7f0c615bf020 { > vxid = 59420066, transport = HTTP/1 { > state = HTTP1::Proc > } > step = R_STP_LOOKUP, > req_body = R_BODY_NONE, > restarts = 0, esi_level = 0, > sp = 0x7f12be06de20 { > fd = 82, vxid = 59420065, > t_open = 1511876667.587429, > t_idle = 1511876667.587429, > transport = HTTP/1 { > state = HTTP1::Proc > } > client = 10.44.11.6 50566, > privs = 0x7f12be06de88 { > }, > }, > worker = 0x7f1374734dd0 { > stack = {0x7f1374735000 -> 0x7f13746a2000}, > ws = 0x7f1374734e78 { > id = \"wrk\", > {s, f, r, e} = {0x7f1374734190, +0, (nil), +2040}, > }, > VCL::method = DELIVER, > VCL::return = deliver, > VCL::methods = {}, > }, > ws = 0x7f0c615bf208 { > id = \"req\", > {s, f, r, e} = {0x7f0c615c1008, +3920, +516080, +516080}, > }, > http_conn = 0x7f0c615bf130 { > fd = 82 (@0x7f12be06de38), > doclose = NULL, > ws = 0x7f0c615bf208 { > [Already dumped, see above] > }, > {rxbuf_b, rxbuf_e} = {0x7f0c615c1008, 0x7f0c615c1e27}, > {pipeline_b, pipeline_e} = {(nil), (nil)}, > content_length = -1, > body_status = none, > first_byte_timeout = 0.000000, > between_bytes_timeout = 0.000000, > }, > http[req] = 0x7f0c615bf2a0 { > ws = 0x7f0c615bf208 { > [Already dumped, see above] > }, > hdrs { > \"GET\", > \"/api/rest/reviews/product/6430\", > \"HTTP/1.1\", > \"Host: www.betabrand.com\", > \"Accept-Encoding: gzip\", > \"CF-IPCountry: US\", > \"CF-RAY: 3c4dc30d9aae55b2-ORD\", > \"CF-Visitor: {\"scheme\":\"https\"}\", > \"user-agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_0_3 like Mac OS > X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A432 > Safari/604.1\", > \"accept-language: en-us\", > \"referer: https://www.betabrand.com/womens/pants/womens-black- > boot-flare-dress-pant-yoga-pants\", > \"cookie: optimizelyPendingLogEvents=%5B%22n%3Dimage_impression%26u% > 3Doeu1511755011199r0.44405056492852124%26wxhr%3Dtrue%26time%3D1511876665. > 858%26f%3D8443224466%2C8247617066%2C8411440184% > 2C7780900332%2C8106662898%26g%3D%22%5D; sailthru_content= > d9d08cd819b052bb46ae11d6f4f23820f9f47e5805766810fea9aeef1937 > 27f0f94155821eef8e7bdb5607beb554d9f12fd3caaaf239d3b03d10e1dc4b5fe514; > NaN_hash=d1015888FXVD1484095712135476; __utma=207700493.554833224. > 1511755020.1511755021.1511876545.2; __utmb=207700493.5.10.1511876545; > __utmc=207700493; __utmz=207700493.1511876545.2.2.utmcsr=Google|utmgclid= > EAIaIQobChMIwbet_7Lh1wIVk7fACh0pGAaaEAAYASAAEgL > FrPD_BwE|utmccn=Brand_General|utmcmd=Retention|utmctr=(not%20provided); > _ga=GA1.2.554833224.1511755020; _gac_UA-17580748-1=1. > 1511876545.EAIaIQobChMIwbet_7Lh1wIVk7fACh0pGAaaEAAYASAAEgLFrPD_BwE; > _gac_UA-17580748-8=1.1511876545.EAIaIQobChMIwbet_ > 7Lh1wIVk7fACh0pGAaaEAAYASAAEgLFrPD_BwE; _gid=GA1.2.1787521728.1511876545; > _hp2_id.777713614=%7B%22userId%22%3A%221402506040644741%22%2C% > 22pageviewId%22%3A%226182215729724641%22%2C%22sessionId%22%3A% > 224386307704701158%22%2C%22identity%22%3Anull%2C% > 22trackerVersion%22%3A%223.0%22%7D; _hp2_ses_props.777713614=%7B% > 22r%22%3A%22https%3A%2F%2Fwww.google.com%2F%22%2C%22us%22% > 3A%22Google%22%2C%22um%22%3A%22Retention%22%2C%22ua%22%3A% > 22Brand_General%22%2C%22ts%22%3A1511876545351%2C%22d%22%3A%2 > 2www.betabrand.com%22%2C%22h%22%3A%22%2Fsale%22%7D; > _sp_id.ac93=0115679b-fdb3-417c-9994-b1387273d811.1511755021.2.1511876663. > 1511755037.2396b5db-5ad9-4e6e-9061-f19c13780fc6; _sp_ses.ac93=*; > optimizelyBuckets=%7B%228443224466%22%3A%228449793114%22%2C% > 228247617066%22%3A%228249041800%22%2C%228411440184%22%3A% > 228407050657%22%2C%228106662898%22%3A%228109860945%22%7D; > _gat_UA-17580748-8=1; frontend=deadbeefdeadbeefdeadbeefaa; > __zlcmid=jhgvWpO31r9clR; __utmt_UA-17580748-1=1; _uetsid=_ueta633ebda; > _bbli=0; fs_uid=www.fullstory.com`1bTW`4950547299041280:5668600916475904; > uuid=740c5206-78d1-4902-8896-a5e8e90eb7c8; bb-seed=0.01224735359031548; > betabrand-campaign-first=Brand_General,Google, > Retention,undefined,undefined,Tue Nov 28 2017 07:42:23 GMT-0600 (CST); > betabrand-campaign-last=Brand_General,Google,Retention,undefined,undefined,Tue > Nov 28 2017 07:42:23 GMT-0600 (CST); betabrand-campaign-session= > Brand_General,Google,Retention,undefined,undefined,Tue Nov 28 2017 > 07:42:23 GMT-0600 (CST); optimizelyEndUserId=oeu1511755011199r0.44405056492852124; > optimizelySegments=%7B%22172841198%22%3A%22campaign% > 22%2C%22172864896%22%3A%22safari%22%2C%22173083061%22% > 3A%22true%22%2C%223972941021%22%3A%22brand_general%22%7D; > __qca=P0-1874833837-1511755021423; sailthru_visitor=b78950af-8d6d-4cb6-8d54-a497f97dd6ba; > betabrand-timing=1; 3060738.3440491=e5d3965e-bb79-4cde-b307-d532ffdec44c; > tracker_device=34b4b287-87e9-4418-bef5-28ab27d9038c; EcommTest1=2; > __cfduid=dd480648d530b3001597f1e574200f3611511755011\", > \"CF-Connecting-IP: 73.44.212.19\", > \"X-Forwarded-Proto: https\", > \"Connection: close\", > \"X-Request-Start: t=1511876667586\", > \"X-Queue-Start: t=1511876667586\", > \"X-Unique-ID: 0A80002E:8771_0A2C0B06:01BB_5A1D683B_F1ABD22:0009\", > \"X-Forwarded-For: 73.44.212.19, 10.128.0.46, 10.44.11.6, > 10.44.11.6\", > \"X-PSA-Blocking-Rewrite: betabrand-pagespeed\", > \"Accept: application/json\", > }, > }, > vcl = { > name = \"boot\", > busy = 92, > discard = 0, > state = auto, > temp = warm, > conf = { > srcname = { > \"/etc/varnish/default.vcl\", > \"Builtin\", > }, > }, > }, > vmods = { > std = {Varnish 5.2.0 4c4875cbf, 0.0}, > directors = {Varnish 5.2.0 4c4875cbf, 0.0}, > }, > flags = { > }, > }, > thr.busyobj = (nil) { > }, > > Thanks for your help! > Cheers, > -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.premkumar.me at gmail.com Wed Nov 29 08:23:45 2017 From: n.premkumar.me at gmail.com (Prem Kumar) Date: Wed, 29 Nov 2017 13:53:45 +0530 Subject: Upstream connection between nginx to varnish - HTTP2 Message-ID: Hi , Trying to use nginx to varnish connection as http2 instead of http1.1. https://www.nginx.com/blog/http2-module-nginx/#QandA Currently nginx is not supporting the upstream http2. Anyone knows hitch SSL proxy will do upstream connection to varnish through http2?. or any other SSL proxy would do http2 connection?. Thanks for the help. Thanks, Prem -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Wed Nov 29 08:28:33 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 29 Nov 2017 09:28:33 +0100 Subject: Varnish 5.2.0 child panic In-Reply-To: References: Message-ID: Looks like this hase been fixed in master, and it will be ported to 4.1 https://github.com/varnishcache/varnish-cache/issues/2319 -- Guillaume Quintard On Wed, Nov 29, 2017 at 12:35 AM, Hugues Alary wrote: > Just realized this might be better as a bug report, I'll submit one if > needed. > > Also, I just had another panic: > > Panic at: Tue, 28 Nov 2017 22:18:49 GMT > Assert error in HSH_Lookup(), cache/cache_hash.c line 432: > Condition((vary) != 0) not true. > version = varnish-5.2.0 revision 4c4875cbf, vrt api = 6.1 > ident = Linux,4.4.64+,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll > now = 1786235.707578 (mono), 1511907529.436702 (real) > Backtrace: > 0x556f4d169e36: varnishd(+0x4ae36) [0x556f4d169e36] > 0x556f4d1b4b80: varnishd(VAS_Fail+0x40) [0x556f4d1b4b80] > 0x556f4d15f1b2: varnishd(HSH_Lookup+0xcb2) [0x556f4d15f1b2] > 0x556f4d16e14f: varnishd(CNT_Request+0xedf) [0x556f4d16e14f] > 0x556f4d18dda2: varnishd(+0x6eda2) [0x556f4d18dda2] > 0x556f4d18525c: varnishd(+0x6625c) [0x556f4d18525c] > 0x556f4d185780: varnishd(+0x66780) [0x556f4d185780] > 0x7f13aa27c494: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7494) > [0x7f13aa27c494] > 0x7f13a9fbeaff: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) > [0x7f13a9fbeaff] > thread = (cache-worker) > thr.req = 0x7f1245e0a020 { > vxid = 2869347, transport = HTTP/1 { > state = HTTP1::Proc > } > step = R_STP_LOOKUP, > req_body = R_BODY_NONE, > restarts = 0, esi_level = 0, > sp = 0x7f137a00ea20 { > fd = 65, vxid = 2869346, > t_open = 1511907453.132658, > t_idle = 1511907453.132658, > transport = HTTP/1 { > state = HTTP1::Proc > } > client = 10.44.43.4 45520, > privs = 0x7f137a00ea88 { > }, > }, > worker = 0x7f1388213dd0 { > stack = {0x7f1388214000 -> 0x7f1388181000}, > ws = 0x7f1388213e78 { > id = \"wrk\", > {s, f, r, e} = {0x7f1388213190, +0, (nil), +2040}, > }, > VCL::method = DELIVER, > VCL::return = deliver, > VCL::methods = {}, > }, > ws = 0x7f1245e0a208 { > id = \"req\", > {s, f, r, e} = {0x7f1245e0c008, +4144, +516080, +516080}, > }, > http_conn = 0x7f1245e0a130 { > fd = 65 (@0x7f137a00ea38), > doclose = NULL, > ws = 0x7f1245e0a208 { > [Already dumped, see above] > }, > {rxbuf_b, rxbuf_e} = {0x7f1245e0c008, 0x7f1245e0cf01}, > {pipeline_b, pipeline_e} = {(nil), (nil)}, > content_length = -1, > body_status = none, > first_byte_timeout = 0.000000, > between_bytes_timeout = 0.000000, > }, > http[req] = 0x7f1245e0a2a0 { > ws = 0x7f1245e0a208 { > [Already dumped, see above] > }, > hdrs { > \"GET\", > \"/api/rest/reviews/product/6430\", > \"HTTP/1.1\", > \"Host: www.betabrand.com\", > \"Accept-Encoding: gzip\", > \"CF-IPCountry: US\", > \"CF-RAY: 3c50b2adfddd5a56-BOS\", > \"CF-Visitor: {\"scheme\":\"https\"}\", > \"user-agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_1_2 like Mac OS > X) AppleWebKit/604.3.5 (KHTML, like Gecko) Mobile/15B202 [FBAN/FBIOS;FBAV/ > 150.0.0.32.132;FBBV/80278251;FBDV/iPhone9,1;FBMD/iPhone;FBSN/ > iOS;FBSV/11.1.2;FBSS/2;FBCR/Verizon;FBID/phone;FBLC/en_US; > FBOP/5;FBRV/0]\", > \"accept-language: en-us\", > \"referer: https://www.betabrand.com/womens/pants/dress-pant-yoga- > pants-collection/womens-black-boot-flare-dress-pant-yoga-pants\", > \"cookie: NaN_hash=d7023912CNFF1437826184182031; > __utma=207700493.1027173649.1511907207.1511907209.1511907209.1; > __utmb=207700493.16.10.1511907209; __utmc=207700493; > __utmz=207700493.1511907209.1.1.utmcsr=Facebook|utmccn= > BlackSheepWrap_Booyah|utmcmd=Acquisition|utmctr=ACQ_WMN; > _ga=GA1.2.1027173649.1511907207; _gat_UA-17580748-8=1; > _gid=GA1.2.363913430.1511907207; _sp_id.ac93=16411174-95e0- > 4636-a008-9e3cba433d59.1511907211.1.1511907405. > 1511907211.756129b3-5717-460b-98c9-c7597c30c819; _sp_ses.ac93=*; > optimizelyBuckets=%7B%223353291595%22%3A%223339030766%22%2C% > 223397350584%22%3A%223376220538%22%2C%223528387311%22%3A% > 223528387312%22%2C%228443224466%22%3A%228449793114%22%2C% > 228247617066%22%3A%228240095367%22%2C%228411440184%22%3A% > 228414131467%22%2C%228106662898%22%3A%228109860945%22%7D; > sailthru_content=bfba62eaf5e186fe0219ddc33264f6 > 6df9f47e5805766810fea9aeef193727f0d9d08cd819b052bb46ae11d6f4 > f2382032527e3cf48859c6a3a1764af88774438fc9ab569691ee6c97a814 > 6c72e75c31c8d1082e407b7c337fe9b9af43106beb9eb5c66e1957b87d51 > 65561970fe74c12fd3caaaf239d3b03d10e1dc4b5fe514; _hp2_id.777713614=%7B% > 22userId%22%3A%220743002369627522%22%2C%22pageviewId%22%3A% > 225740794725429989%22%2C%22sessionId%22%3A%221322112537315924%22%2C% > 22identity%22%3Anull%2C%22trackerVersion%22%3A%223.0%22%7D; > _hp2_ses_props.777713614=%7B%22r%22%3A%22http%3A%2F%2Fm.facebook.com > %22%2C%22us%22%3A%22Facebook%22%2C%22um%22%3A%22Acquisition%22%2C%22ut%22% > 3A%22ACQ_WMN%22%2C%22ua%22%3A%22BlackSheepWrap_Booyah%22%2C% > 22ts%22%3A1511907219368%2C%22d%22%3A%22www.betabrand.com% > 22%2C%22h%22%3A%22%2Fwomens%2Fpants%2Fdress-pant-yoga-pants-collection%22%7D; > __zlcmid=jigvnRPnf0yTRh; _uetsid=_uetf208ea35; _bbli=0; fs_uid= > www.fullstory.com`1bTW`5677711459876864:5629499534213120; frontend=deadbeefdeadbeefdeadbeefaa; > uuid=a1d09da5-7506-41c2-94b6-6b5acdba0f9c; betabrand-campaign-last= > BlackSheepWrap_Booyah,Facebook,Acquisition,ACQ_WMN,undefined,Tue Nov 28 > 2017 17:14:57 GMT-0500 (EST); betabrand-campaign-session= > BlackSheepWrap_Booyah,Facebook,Acquisition,ACQ_WMN,undefined,Tue Nov 28 > 2017 17:14:57 GMT-0500 (EST); optimizelyEndUserId=oeu1444061797597r0.2150009439792484; > optimizelySegments=%7B%22172841198%22%3A%22campaign% > 22%2C%22172864896%22%3A%22unknown%22%2C%22173083061%22%3A%22true%22%2C% > 223972941021%22%3A%22blacksheepwrap_booya%22%7D; > sailthru_visitor=94d5d458-c946-4228-9886-1b57e519beeb; > 3060738.3440491=67c18f82-2818-44ef-ab9b-f2667055fada; __qca=P0-1450722180-1511907220479; > tracker_device=ce48cfcc-5b82-483b-8a1b-562959838be8; > __utmt_UA-17580748-1=1; betabrand-timing=1; EcommTest1=2; > betabrand-campaign-first=BlackSheepWrap_Booyah, > Facebook,Acquisition,ACQ_WMN,undefined,Tue Nov 28 2017 17:13:23 GMT-0500 > (EST); betabrand-introduction=true; __cfduid= > dcc19056430e6c8dfd790803695cfb1bf1511907198; _gr_ep_sent=1; > _gr_er_sent=1; granify.lasts at 809=1444061806057; > granify.uuid=53534c95-2046-47f9-b9f8-556dbd6e4b4d; a1ashgd= > c9de5eb25c19369cd2a04d2babab5888\", > \"CF-Connecting-IP: 208.64.112.35\", > \"X-Forwarded-Proto: https\", > \"Connection: close\", > \"X-Request-Start: t=1511907453131\", > \"X-Queue-Start: t=1511907453131\", > \"X-Unique-ID: 0A800027:8911_0A2C2B04:01BB_5A1DE07D_30A0ADB:0009\", > \"X-Forwarded-For: 208.64.112.35, 10.128.0.39, 10.44.43.4, > 10.44.43.4\", > \"X-PSA-Blocking-Rewrite: betabrand-pagespeed\", > \"Accept: application/json\", > }, > }, > vcl = { > name = \"boot\", > busy = 135, > discard = 0, > state = auto, > temp = warm, > conf = { > srcname = { > \"/etc/varnish/default.vcl\", > \"Builtin\", > }, > }, > }, > vmods = { > std = {Varnish 5.2.0 4c4875cbf, 0.0}, > directors = {Varnish 5.2.0 4c4875cbf, 0.0}, > }, > flags = { > }, > }, > thr.busyobj = (nil) { > }, > > > > > On Tue, Nov 28, 2017 at 11:48 AM, Hugues Alary > wrote: > >> Hi there! >> >> In the last 18 days, varnish reports its child died 8 times. >> >> We had more traffic than usual those past few days, but I don't think it >> was anything Varnish can't handle. >> >> We run Varnish on Kubernetes on GCP. >> >> I was able to retrieve the last panic via varnishadm (see below). The >> other panics are the same as the one below (same assertion failing, same >> line number). (I would send them too, but it's amazingly difficult to get >> logs out of google cloud). >> >> If the panic is not enough, I can send our configuration file on request. >> >> Panic at: Tue, 28 Nov 2017 13:44:30 GMT >> Assert error in HSH_Lookup(), cache/cache_hash.c line 432: >> Condition((vary) != 0) not true. >> version = varnish-5.2.0 revision 4c4875cbf, vrt api = 6.1 >> ident = Linux,4.4.64+,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll >> now = 1755374.451854 (mono), 1511876668.180978 (real) >> Backtrace: >> 0x556f4d169e36: varnishd(+0x4ae36) [0x556f4d169e36] >> 0x556f4d1b4b80: varnishd(VAS_Fail+0x40) [0x556f4d1b4b80] >> 0x556f4d15f1b2: varnishd(HSH_Lookup+0xcb2) [0x556f4d15f1b2] >> 0x556f4d16e14f: varnishd(CNT_Request+0xedf) [0x556f4d16e14f] >> 0x556f4d18dda2: varnishd(+0x6eda2) [0x556f4d18dda2] >> 0x556f4d18525c: varnishd(+0x6625c) [0x556f4d18525c] >> 0x556f4d185780: varnishd(+0x66780) [0x556f4d185780] >> 0x7f13aa27c494: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7494) >> [0x7f13aa27c494] >> 0x7f13a9fbeaff: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) >> [0x7f13a9fbeaff] >> thread = (cache-worker) >> thr.req = 0x7f0c615bf020 { >> vxid = 59420066, transport = HTTP/1 { >> state = HTTP1::Proc >> } >> step = R_STP_LOOKUP, >> req_body = R_BODY_NONE, >> restarts = 0, esi_level = 0, >> sp = 0x7f12be06de20 { >> fd = 82, vxid = 59420065, >> t_open = 1511876667.587429, >> t_idle = 1511876667.587429, >> transport = HTTP/1 { >> state = HTTP1::Proc >> } >> client = 10.44.11.6 50566, >> privs = 0x7f12be06de88 { >> }, >> }, >> worker = 0x7f1374734dd0 { >> stack = {0x7f1374735000 -> 0x7f13746a2000}, >> ws = 0x7f1374734e78 { >> id = \"wrk\", >> {s, f, r, e} = {0x7f1374734190, +0, (nil), +2040}, >> }, >> VCL::method = DELIVER, >> VCL::return = deliver, >> VCL::methods = {}, >> }, >> ws = 0x7f0c615bf208 { >> id = \"req\", >> {s, f, r, e} = {0x7f0c615c1008, +3920, +516080, +516080}, >> }, >> http_conn = 0x7f0c615bf130 { >> fd = 82 (@0x7f12be06de38), >> doclose = NULL, >> ws = 0x7f0c615bf208 { >> [Already dumped, see above] >> }, >> {rxbuf_b, rxbuf_e} = {0x7f0c615c1008, 0x7f0c615c1e27}, >> {pipeline_b, pipeline_e} = {(nil), (nil)}, >> content_length = -1, >> body_status = none, >> first_byte_timeout = 0.000000, >> between_bytes_timeout = 0.000000, >> }, >> http[req] = 0x7f0c615bf2a0 { >> ws = 0x7f0c615bf208 { >> [Already dumped, see above] >> }, >> hdrs { >> \"GET\", >> \"/api/rest/reviews/product/6430\", >> \"HTTP/1.1\", >> \"Host: www.betabrand.com\", >> \"Accept-Encoding: gzip\", >> \"CF-IPCountry: US\", >> \"CF-RAY: 3c4dc30d9aae55b2-ORD\", >> \"CF-Visitor: {\"scheme\":\"https\"}\", >> \"user-agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_0_3 like Mac OS >> X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A432 >> Safari/604.1\", >> \"accept-language: en-us\", >> \"referer: https://www.betabrand.com/wome >> ns/pants/womens-black-boot-flare-dress-pant-yoga-pants\", >> \"cookie: optimizelyPendingLogEvents=%5B >> %22n%3Dimage_impression%26u%3Doeu1511755011199r0.44405056492 >> 852124%26wxhr%3Dtrue%26time%3D1511876665.858%26f% >> 3D8443224466%2C8247617066%2C8411440184%2C7780900332%2C8106662898%26g%3D%22%5D; >> sailthru_content=d9d08cd819b052bb46ae11d6f4f23820f9f47e58057 >> 66810fea9aeef193727f0f94155821eef8e7bdb5607beb554d9f12fd3caaaf239d3b03d10e1dc4b5fe514; >> NaN_hash=d1015888FXVD1484095712135476; __utma=207700493.554833224.151 >> 1755020.1511755021.1511876545.2; __utmb=207700493.5.10.1511876545; >> __utmc=207700493; __utmz=207700493.1511876545.2. >> 2.utmcsr=Google|utmgclid=EAIaIQobChMIwbet_7Lh1wIVk7fACh0pGAa >> aEAAYASAAEgLFrPD_BwE|utmccn=Brand_General|utmcmd= >> Retention|utmctr=(not%20provided); _ga=GA1.2.554833224.1511755020; >> _gac_UA-17580748-1=1.1511876545.EAIaIQobChMIwbet_7Lh1wIVk7fACh0pGAaaEAAYASAAEgLFrPD_BwE; >> _gac_UA-17580748-8=1.1511876545.EAIaIQobChMIwbet_7Lh1wIVk7fACh0pGAaaEAAYASAAEgLFrPD_BwE; >> _gid=GA1.2.1787521728.1511876545; _hp2_id.777713614=%7B%22userId >> %22%3A%221402506040644741%22%2C%22pageviewId%22%3A%22618221 >> 5729724641%22%2C%22sessionId%22%3A%224386307704701158%22% >> 2C%22identity%22%3Anull%2C%22trackerVersion%22%3A%223.0%22%7D; >> _hp2_ses_props.777713614=%7B%22r%22%3A%22https%3A%2F%2Fwww.google.com >> %2F%22%2C%22us%22%3A%22Google%22%2C%22um%22%3A%22Re >> tention%22%2C%22ua%22%3A%22Brand_General%22%2C%22ts%22%3A151 >> 1876545351%2C%22d%22%3A%22www.betabrand.com%22%2C%22h%22%3A%22%2Fsale%22%7D; >> _sp_id.ac93=0115679b-fdb3-417c-9994-b1387273d811.1511755021. >> 2.1511876663.1511755037.2396b5db-5ad9-4e6e-9061-f19c13780fc6; >> _sp_ses.ac93=*; optimizelyBuckets=%7B%22844322 >> 4466%22%3A%228449793114%22%2C%228247617066%22%3A%22824904180 >> 0%22%2C%228411440184%22%3A%228407050657%22%2C%228106662898%22%3A%228109860945%22%7D; >> _gat_UA-17580748-8=1; frontend=deadbeefdeadbeefdeadbeefaa; >> __zlcmid=jhgvWpO31r9clR; __utmt_UA-17580748-1=1; _uetsid=_ueta633ebda; >> _bbli=0; fs_uid=www.fullstory.com`1bTW`4950547299041280:5668600916475904; >> uuid=740c5206-78d1-4902-8896-a5e8e90eb7c8; bb-seed=0.01224735359031548; >> betabrand-campaign-first=Brand_General,Google,Retention,undefined,undefined,Tue >> Nov 28 2017 07:42:23 GMT-0600 (CST); betabrand-campaign-last=Brand_ >> General,Google,Retention,undefined,undefined,Tue Nov 28 2017 07:42:23 >> GMT-0600 (CST); betabrand-campaign-session=Bra >> nd_General,Google,Retention,undefined,undefined,Tue Nov 28 2017 07:42:23 >> GMT-0600 (CST); optimizelyEndUserId=oeu1511755011199r0.44405056492852124; >> optimizelySegments=%7B%22172841198%22%3A%22campaign%22%2C% >> 22172864896%22%3A%22safari%22%2C%22173083061%22%3A%22true% >> 22%2C%223972941021%22%3A%22brand_general%22%7D; >> __qca=P0-1874833837-1511755021423; sailthru_visitor=b78950af-8d6d-4cb6-8d54-a497f97dd6ba; >> betabrand-timing=1; 3060738.3440491=e5d3965e-bb79-4cde-b307-d532ffdec44c; >> tracker_device=34b4b287-87e9-4418-bef5-28ab27d9038c; EcommTest1=2; >> __cfduid=dd480648d530b3001597f1e574200f3611511755011\", >> \"CF-Connecting-IP: 73.44.212.19\", >> \"X-Forwarded-Proto: https\", >> \"Connection: close\", >> \"X-Request-Start: t=1511876667586\", >> \"X-Queue-Start: t=1511876667586\", >> \"X-Unique-ID: 0A80002E:8771_0A2C0B06:01BB_5A1D683B_F1ABD22:0009\", >> \"X-Forwarded-For: 73.44.212.19, 10.128.0.46, 10.44.11.6, >> 10.44.11.6\", >> \"X-PSA-Blocking-Rewrite: betabrand-pagespeed\", >> \"Accept: application/json\", >> }, >> }, >> vcl = { >> name = \"boot\", >> busy = 92, >> discard = 0, >> state = auto, >> temp = warm, >> conf = { >> srcname = { >> \"/etc/varnish/default.vcl\", >> \"Builtin\", >> }, >> }, >> }, >> vmods = { >> std = {Varnish 5.2.0 4c4875cbf, 0.0}, >> directors = {Varnish 5.2.0 4c4875cbf, 0.0}, >> }, >> flags = { >> }, >> }, >> thr.busyobj = (nil) { >> }, >> >> Thanks for your help! >> Cheers, >> > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Wed Nov 29 08:56:25 2017 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 29 Nov 2017 09:56:25 +0100 Subject: Varnish 5.2.0 child panic In-Reply-To: References: Message-ID: On Wed, Nov 29, 2017 at 12:35 AM, Hugues Alary wrote: > Just realized this might be better as a bug report, I'll submit one if > needed. > > Also, I just had another panic: Hi, You should sanitize the panic output to not disclose user cookies publicly! Replace the value with junk next time. > Panic at: Tue, 28 Nov 2017 22:18:49 GMT > Assert error in HSH_Lookup(), cache/cache_hash.c line 432: > Condition((vary) != 0) not true. > version = varnish-5.2.0 revision 4c4875cbf, vrt api = 6.1 > ident = Linux,4.4.64+,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll > now = 1786235.707578 (mono), 1511907529.436702 (real) > Backtrace: > 0x556f4d169e36: varnishd(+0x4ae36) [0x556f4d169e36] > 0x556f4d1b4b80: varnishd(VAS_Fail+0x40) [0x556f4d1b4b80] > 0x556f4d15f1b2: varnishd(HSH_Lookup+0xcb2) [0x556f4d15f1b2] > 0x556f4d16e14f: varnishd(CNT_Request+0xedf) [0x556f4d16e14f] > 0x556f4d18dda2: varnishd(+0x6eda2) [0x556f4d18dda2] > 0x556f4d18525c: varnishd(+0x6625c) [0x556f4d18525c] > 0x556f4d185780: varnishd(+0x66780) [0x556f4d185780] > 0x7f13aa27c494: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7494) > [0x7f13aa27c494] > 0x7f13a9fbeaff: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) > [0x7f13a9fbeaff] > thread = (cache-worker) > thr.req = 0x7f1245e0a020 { > vxid = 2869347, transport = HTTP/1 { > state = HTTP1::Proc > } > step = R_STP_LOOKUP, > req_body = R_BODY_NONE, > restarts = 0, esi_level = 0, > sp = 0x7f137a00ea20 { > fd = 65, vxid = 2869346, > t_open = 1511907453.132658, > t_idle = 1511907453.132658, > transport = HTTP/1 { > state = HTTP1::Proc > } > client = 10.44.43.4 45520, > privs = 0x7f137a00ea88 { > }, > }, > worker = 0x7f1388213dd0 { > stack = {0x7f1388214000 -> 0x7f1388181000}, > ws = 0x7f1388213e78 { > id = \"wrk\", > {s, f, r, e} = {0x7f1388213190, +0, (nil), +2040}, > }, > VCL::method = DELIVER, > VCL::return = deliver, > VCL::methods = {}, > }, > ws = 0x7f1245e0a208 { > id = \"req\", > {s, f, r, e} = {0x7f1245e0c008, +4144, +516080, +516080}, > }, > http_conn = 0x7f1245e0a130 { > fd = 65 (@0x7f137a00ea38), > doclose = NULL, > ws = 0x7f1245e0a208 { > [Already dumped, see above] > }, > {rxbuf_b, rxbuf_e} = {0x7f1245e0c008, 0x7f1245e0cf01}, > {pipeline_b, pipeline_e} = {(nil), (nil)}, > content_length = -1, > body_status = none, > first_byte_timeout = 0.000000, > between_bytes_timeout = 0.000000, > }, > http[req] = 0x7f1245e0a2a0 { > ws = 0x7f1245e0a208 { > [Already dumped, see above] > }, > hdrs { > \"GET\", > \"/api/rest/reviews/product/6430\", > \"HTTP/1.1\", > \"Host: www.betabrand.com\", > \"Accept-Encoding: gzip\", > \"CF-IPCountry: US\", > \"CF-RAY: 3c50b2adfddd5a56-BOS\", > \"CF-Visitor: {\"scheme\":\"https\"}\", > \"user-agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_1_2 like Mac OS X) > AppleWebKit/604.3.5 (KHTML, like Gecko) Mobile/15B202 > [FBAN/FBIOS;FBAV/150.0.0.32.132;FBBV/80278251;FBDV/iPhone9,1;FBMD/iPhone;FBSN/iOS;FBSV/11.1.2;FBSS/2;FBCR/Verizon;FBID/phone;FBLC/en_US;FBOP/5;FBRV/0]\", > \"accept-language: en-us\", > \"referer: > https://www.betabrand.com/womens/pants/dress-pant-yoga-pants-collection/womens-black-boot-flare-dress-pant-yoga-pants\", For example: > \"cookie: XXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXX; > XXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > XXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXX; XXXXXX=XXXXXXXXX; > XXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXX=XXXXXXX; > XXX=XXXXXXXXXXXXXXXXXXXXXXXXXXX; XXXXXXXXXXXXXXXXXX=X; > XXXX=XXXXXXXXXXXXXXXXXXXXXXXXXX; > XXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > XXXXXXXXXXXX=X; > XXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > XXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > XXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > XXXXXXXXXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > XXXXXXXX=XXXXXXXXXXXXXX; XXXXXXX=XXXXXXXXXXXX; XXXXX=X; > XXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > XXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXX; > XXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > XXXXXXXXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXX XX XXXX XXXXXXXX XXXXXXXX XXXXX; > XXXXXXXXXXXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXX XX XXXX XXXXXXXX XXXXXXXX XXXXX; > XXXXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > XXXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > XXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > XXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > XXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXX; > XXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; XXXXXXXXXXXXXXXXXXXX=X; > XXXXXXXXXXXXXXXX=X; XXXXXXXXXX=X; > XXXXXXXXXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXX XX XXXX XXXXXXXX XXXXXXXX XXXXX; XXXXXXXXXXXXXXXXXXXXXX=XXXX; > XXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; XXXXXXXXXXX=X; > XXXXXXXXXXX=X; XXXXXXXXXXXXXXXXX=XXXXXXXXXXXXX; > XXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > XXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\", And anything sensitive in general like IP addresses... > \"CF-Connecting-IP: 208.64.112.35\", > \"X-Forwarded-Proto: https\", > \"Connection: close\", > \"X-Request-Start: t=1511907453131\", > \"X-Queue-Start: t=1511907453131\", > \"X-Unique-ID: 0A800027:8911_0A2C2B04:01BB_5A1DE07D_30A0ADB:0009\", > \"X-Forwarded-For: 208.64.112.35, 10.128.0.39, 10.44.43.4, > 10.44.43.4\", > \"X-PSA-Blocking-Rewrite: betabrand-pagespeed\", > \"Accept: application/json\", > }, > }, > vcl = { > name = \"boot\", > busy = 135, > discard = 0, > state = auto, > temp = warm, > conf = { > srcname = { > \"/etc/varnish/default.vcl\", > \"Builtin\", > }, > }, > }, > vmods = { > std = {Varnish 5.2.0 4c4875cbf, 0.0}, > directors = {Varnish 5.2.0 4c4875cbf, 0.0}, > }, > flags = { > }, > }, > thr.busyobj = (nil) { > }, Dridi From hermunn at varnish-software.com Wed Nov 29 09:07:00 2017 From: hermunn at varnish-software.com (=?UTF-8?Q?P=C3=A5l_Hermunn_Johansen?=) Date: Wed, 29 Nov 2017 10:07:00 +0100 Subject: Varnish 5.2.0 child panic In-Reply-To: References: Message-ID: 2017-11-29 9:28 GMT+01:00 Guillaume Quintard : > Looks like this hase been fixed in master, and it will be ported to 4.1 > https://github.com/varnishcache/varnish-cache/issues/2319 Right now it looks to me that the bug is not in 4.1, but I will need to investigate if the other (associated) changes need to go in. Please let us know (in the GitHub ticket) if the problem is observed in 4.1 too. P?l From olivier.hanesse at gmail.com Wed Nov 29 13:53:55 2017 From: olivier.hanesse at gmail.com (Olivier Hanesse) Date: Wed, 29 Nov 2017 14:53:55 +0100 Subject: Varnish Lurker is getting slower / Ban lists keeps increasing In-Reply-To: References: <12ca81a6-44fa-bfa0-046d-b051cabc92b6@schokola.de> Message-ID: Hello, I am still on this issue. Today, I disable the loadbalancing to a specific varnish server whose ban list was around 500K (n_object is around 600K) After 4 hours without any requests other than "BAN", the ban list is still increasing, and I got a system load around 1.5. A "top" with thread show that the ban_lurker is eating 100% of 1 CPU (8 cpu computer) top - 14:47:53 up 91 days, 22:36, 2 users, load average: 1.57, 1.45, 1.49 Threads: 702 total, 2 running, 700 sleeping, 0 stopped, 0 zombie %Cpu(s): 11.6 us, 0.2 sy, 0.0 ni, 88.1 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st KiB Mem: 8196832 total, 5532436 used, 2664396 free, 98028 buffers KiB Swap: 499708 total, 0 used, 499708 free. 1077884 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4606 vcache 20 0 4212992 3.553g 85300 R 99.9 45.5 6991:50 ban-lurker Is is possible that the ban_lurker is locked or in an infinite loop (I know it is single threaded) ? What kind of dump can I provide to help understand this issue ? Regards Olivier 2017-09-04 16:54 GMT+02:00 Olivier Hanesse : > In this case, that means that as long as the ban lurker is working, no > statistics are updated right ? > > So if I don't see any updates of statistics such as "bans_deleted", or > "bans_lurker_obj_killed_cutoff" during a long period, it doesn't mean > that the lurker is sleeping, hanged or waiting for a lock, it means that > the lurker worker is working pretty "hard", is that correct ? > > > 2017-09-04 16:11 GMT+02:00 Dridi Boukelmoune : > >> On Mon, Sep 4, 2017 at 2:12 PM, Olivier Hanesse >> wrote: >> > Are the stats used by varnishstat about the lurker "well" updated >> "every minute" ? The fact that the statistics was only updated once is >> kinda strange : the ban list size is higher than the cutoff value everyday >> :( >> >> No, that's a limitation of the statistics, serving HTTP traffic has >> higher priority than committing updates of the counters. >> >> See this for reference: >> >> https://github.com/varnishcache/varnish-cache/pull/2290 >> >> Dridi >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugues at betabrand.com Wed Nov 29 18:04:46 2017 From: hugues at betabrand.com (Hugues Alary) Date: Wed, 29 Nov 2017 10:04:46 -0800 Subject: Varnish 5.2.0 child panic In-Reply-To: References: Message-ID: Hi Dridi, I actually did think about this originally, but decided to only obfuscate one cookie ("frontend", which I replaced with deadbeefdeadbeefdeadbeefaa in both panics). Now, thinking back about it (thanks to your email), even though it would be hard to infer anything of value from any of the other cookies, I probably should have obfuscated them too. As for the IP address, this was definitely an omission on my part. I appreciate the reminder. Cheers, -Hugues On Wed, Nov 29, 2017 at 12:56 AM, Dridi Boukelmoune wrote: > On Wed, Nov 29, 2017 at 12:35 AM, Hugues Alary > wrote: > > Just realized this might be better as a bug report, I'll submit one if > > needed. > > > > Also, I just had another panic: > > Hi, > > You should sanitize the panic output to not disclose user cookies > publicly! Replace the value with junk next time. > > > Panic at: Tue, 28 Nov 2017 22:18:49 GMT > > Assert error in HSH_Lookup(), cache/cache_hash.c line 432: > > Condition((vary) != 0) not true. > > version = varnish-5.2.0 revision 4c4875cbf, vrt api = 6.1 > > ident = Linux,4.4.64+,x86_64,-junix,-smalloc,-smalloc,-hcritbit,epoll > > now = 1786235.707578 (mono), 1511907529.436702 (real) > > Backtrace: > > 0x556f4d169e36: varnishd(+0x4ae36) [0x556f4d169e36] > > 0x556f4d1b4b80: varnishd(VAS_Fail+0x40) [0x556f4d1b4b80] > > 0x556f4d15f1b2: varnishd(HSH_Lookup+0xcb2) [0x556f4d15f1b2] > > 0x556f4d16e14f: varnishd(CNT_Request+0xedf) [0x556f4d16e14f] > > 0x556f4d18dda2: varnishd(+0x6eda2) [0x556f4d18dda2] > > 0x556f4d18525c: varnishd(+0x6625c) [0x556f4d18525c] > > 0x556f4d185780: varnishd(+0x66780) [0x556f4d185780] > > 0x7f13aa27c494: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7494) > > [0x7f13aa27c494] > > 0x7f13a9fbeaff: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) > > [0x7f13a9fbeaff] > > thread = (cache-worker) > > thr.req = 0x7f1245e0a020 { > > vxid = 2869347, transport = HTTP/1 { > > state = HTTP1::Proc > > } > > step = R_STP_LOOKUP, > > req_body = R_BODY_NONE, > > restarts = 0, esi_level = 0, > > sp = 0x7f137a00ea20 { > > fd = 65, vxid = 2869346, > > t_open = 1511907453.132658, > > t_idle = 1511907453.132658, > > transport = HTTP/1 { > > state = HTTP1::Proc > > } > > client = 10.44.43.4 45520, > > privs = 0x7f137a00ea88 { > > }, > > }, > > worker = 0x7f1388213dd0 { > > stack = {0x7f1388214000 -> 0x7f1388181000}, > > ws = 0x7f1388213e78 { > > id = \"wrk\", > > {s, f, r, e} = {0x7f1388213190, +0, (nil), +2040}, > > }, > > VCL::method = DELIVER, > > VCL::return = deliver, > > VCL::methods = {}, > > }, > > ws = 0x7f1245e0a208 { > > id = \"req\", > > {s, f, r, e} = {0x7f1245e0c008, +4144, +516080, +516080}, > > }, > > http_conn = 0x7f1245e0a130 { > > fd = 65 (@0x7f137a00ea38), > > doclose = NULL, > > ws = 0x7f1245e0a208 { > > [Already dumped, see above] > > }, > > {rxbuf_b, rxbuf_e} = {0x7f1245e0c008, 0x7f1245e0cf01}, > > {pipeline_b, pipeline_e} = {(nil), (nil)}, > > content_length = -1, > > body_status = none, > > first_byte_timeout = 0.000000, > > between_bytes_timeout = 0.000000, > > }, > > http[req] = 0x7f1245e0a2a0 { > > ws = 0x7f1245e0a208 { > > [Already dumped, see above] > > }, > > hdrs { > > \"GET\", > > \"/api/rest/reviews/product/6430\", > > \"HTTP/1.1\", > > \"Host: www.betabrand.com\", > > \"Accept-Encoding: gzip\", > > \"CF-IPCountry: US\", > > \"CF-RAY: 3c50b2adfddd5a56-BOS\", > > \"CF-Visitor: {\"scheme\":\"https\"}\", > > \"user-agent: Mozilla/5.0 (iPhone; CPU iPhone OS 11_1_2 like Mac > OS X) > > AppleWebKit/604.3.5 (KHTML, like Gecko) Mobile/15B202 > > [FBAN/FBIOS;FBAV/150.0.0.32.132;FBBV/80278251;FBDV/ > iPhone9,1;FBMD/iPhone;FBSN/iOS;FBSV/11.1.2;FBSS/2;FBCR/ > Verizon;FBID/phone;FBLC/en_US;FBOP/5;FBRV/0]\", > > \"accept-language: en-us\", > > \"referer: > > https://www.betabrand.com/womens/pants/dress-pant-yoga- > pants-collection/womens-black-boot-flare-dress-pant-yoga-pants\", > > For example: > > > \"cookie: XXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXX; > > XXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > > XXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXX; XXXXXX=XXXXXXXXX; > > XXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXX= > XXXXXXXXXXXXXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXX=XXXXXXX; > > XXX=XXXXXXXXXXXXXXXXXXXXXXXXXXX; XXXXXXXXXXXXXXXXXX=X; > > XXXX=XXXXXXXXXXXXXXXXXXXXXXXXXX; > > XXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > > XXXXXXXXXXXX=X; > > XXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > > XXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > > XXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > > XXXXXXXXXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > > XXXXXXXX=XXXXXXXXXXXXXX; XXXXXXX=XXXXXXXXXXXX; XXXXX=X; > > XXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > > XXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXX; > > XXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > > XXXXXXXXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > > XXX XX XXXX XXXXXXXX XXXXXXXX XXXXX; > > XXXXXXXXXXXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > > XXX XX XXXX XXXXXXXX XXXXXXXX XXXXX; > > XXXXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > > XXXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > > XXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > > XXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > > XXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXX; > > XXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > XXXXXXXXXXXXXXXXXXXX=X; > > XXXXXXXXXXXXXXXX=X; XXXXXXXXXX=X; > > XXXXXXXXXXXXXXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > > XXX XX XXXX XXXXXXXX XXXXXXXX XXXXX; XXXXXXXXXXXXXXXXXXXXXX=XXXX; > > XXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; XXXXXXXXXXX=X; > > XXXXXXXXXXX=X; XXXXXXXXXXXXXXXXX=XXXXXXXXXXXXX; > > XXXXXXXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX; > > XXXXXXX=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\", > > And anything sensitive in general like IP addresses... > > > \"CF-Connecting-IP: 208.64.112.35\", > > \"X-Forwarded-Proto: https\", > > \"Connection: close\", > > \"X-Request-Start: t=1511907453131\", > > \"X-Queue-Start: t=1511907453131\", > > \"X-Unique-ID: 0A800027:8911_0A2C2B04:01BB_ > 5A1DE07D_30A0ADB:0009\", > > \"X-Forwarded-For: 208.64.112.35, 10.128.0.39, 10.44.43.4, > > 10.44.43.4\", > > \"X-PSA-Blocking-Rewrite: betabrand-pagespeed\", > > \"Accept: application/json\", > > }, > > }, > > vcl = { > > name = \"boot\", > > busy = 135, > > discard = 0, > > state = auto, > > temp = warm, > > conf = { > > srcname = { > > \"/etc/varnish/default.vcl\", > > \"Builtin\", > > }, > > }, > > }, > > vmods = { > > std = {Varnish 5.2.0 4c4875cbf, 0.0}, > > directors = {Varnish 5.2.0 4c4875cbf, 0.0}, > > }, > > flags = { > > }, > > }, > > thr.busyobj = (nil) { > > }, > > Dridi > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugues at betabrand.com Wed Nov 29 18:11:56 2017 From: hugues at betabrand.com (Hugues Alary) Date: Wed, 29 Nov 2017 10:11:56 -0800 Subject: Varnish 5.2.0 child panic In-Reply-To: References: Message-ID: >From the github bug report, it seems the bug was reported against 5.1 and 4.1 is not affected (the reporter went back to 4.1, from 5.1). @Guillaume, was "4.1" a typo? As for the possible resolution for this, it seems a patch was indeed written and merged in master, in which case I will wait for the next release and upgrade. I don't think there's any real solution to avoid this bug until the next varnish release, but, if there is, I'd love to know since this happens quite regularly on my end. Thanks! -Hugues On Wed, Nov 29, 2017 at 1:07 AM, P?l Hermunn Johansen < hermunn at varnish-software.com> wrote: > 2017-11-29 9:28 GMT+01:00 Guillaume Quintard com>: > > Looks like this hase been fixed in master, and it will be ported to 4.1 > > https://github.com/varnishcache/varnish-cache/issues/2319 > > Right now it looks to me that the bug is not in 4.1, but I will need > to investigate if the other (associated) changes need to go in. Please > let us know (in the GitHub ticket) if the problem is observed in 4.1 > too. > > > P?l > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Wed Nov 29 21:13:17 2017 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 29 Nov 2017 22:13:17 +0100 Subject: Varnish 5.2.0 child panic In-Reply-To: References: Message-ID: Hi Hughes, It was not a typo, just a mistake, I misread the backport-review-4.1 tag on the issue. 4.1 is indeed unaffected. -- Guillaume Quintard On Nov 29, 2017 19:12, "Hugues Alary" wrote: > From the github bug report, it seems the bug was reported against 5.1 and > 4.1 is not affected (the reporter went back to 4.1, from 5.1). > > @Guillaume, was "4.1" a typo? > > As for the possible resolution for this, it seems a patch was indeed > written and merged in master, in which case I will wait for the next > release and upgrade. > > I don't think there's any real solution to avoid this bug until the next > varnish release, but, if there is, I'd love to know since this happens > quite regularly on my end. > > Thanks! > -Hugues > > > > On Wed, Nov 29, 2017 at 1:07 AM, P?l Hermunn Johansen < > hermunn at varnish-software.com> wrote: > >> 2017-11-29 9:28 GMT+01:00 Guillaume Quintard < >> guillaume at varnish-software.com>: >> > Looks like this hase been fixed in master, and it will be ported to 4.1 >> > https://github.com/varnishcache/varnish-cache/issues/2319 >> >> Right now it looks to me that the bug is not in 4.1, but I will need >> to investigate if the other (associated) changes need to go in. Please >> let us know (in the GitHub ticket) if the problem is observed in 4.1 >> too. >> >> >> P?l >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From joh.hendriks at gmail.com Thu Nov 30 08:46:36 2017 From: joh.hendriks at gmail.com (Johan Hendriks) Date: Thu, 30 Nov 2017 09:46:36 +0100 Subject: Varnish 5.2.0 child panic In-Reply-To: References: Message-ID: <622a4a86-e251-165a-161b-ff96183a7f63@gmail.com> Op 29/11/2017 om 19:11 schreef Hugues Alary: > From the github bug report, it seems the bug was reported against 5.1 > and 4.1 is not affected (the reporter went back to 4.1, from 5.1).? > > @Guillaume, was "4.1" a typo? > > As for the possible resolution for this, it seems a patch was indeed > written and merged in master, in which case I will wait for the next > release and upgrade.? > > I don't think there's any real solution to avoid this bug until the > next varnish release, but, if there is, I'd love to know since this > happens quite regularly on my end. > > Thanks! > -Hugues > > Maybe you can cherry pick the patch and build varnish yourself by patching the source. If it affects you that much it is worth the try. From slink at schokola.de Thu Nov 30 12:30:54 2017 From: slink at schokola.de (Nils Goroll) Date: Thu, 30 Nov 2017 13:30:54 +0100 Subject: Varnish Lurker is getting slower / Ban lists keeps increasing In-Reply-To: References: <12ca81a6-44fa-bfa0-046d-b051cabc92b6@schokola.de> Message-ID: Hi Olivier On 29/11/17 14:53, Olivier Hanesse wrote: > Is is possible that the ban_lurker is locked or in an infinite loop (I know it > is single threaded) ? What kind of dump can I provide to help understand this > issue ?? Basically we'd need the output of varnishadm ban.list optionally a second run with varnishadm param.set debug +lurker Trouble here is that, by default, the CLI interface will truncate the output after 48k which is way too soon for any ban list of an order of magnitude as you are seeing it. The highest value currently accepted for the cli limit is just under 100MB, which can be set using varnishadm param.set cli_limit 99999999b As a precaution, I would advise to raise the ping timeout with such huge CLI responses: varnishadm param.set cli_timeout 3600 This should be undone back to the default or whatever you had before after gathering the data. Feel free to send me the results via private email, bzip2 or xz compression would probably be a good idea. Nils From slink at schokola.de Thu Nov 30 12:44:05 2017 From: slink at schokola.de (Nils Goroll) Date: Thu, 30 Nov 2017 13:44:05 +0100 Subject: Varnish Lurker is getting slower / Ban lists keeps increasing In-Reply-To: References: <12ca81a6-44fa-bfa0-046d-b051cabc92b6@schokola.de> Message-ID: <8007cef2-a4f6-85a0-4807-0a9ae0c8678a@schokola.de> On 29/11/17 14:53, Olivier Hanesse wrote: > After 4 hours without any requests other than "BAN" side note: As explained before and as documented since 5d713b2d8a796c6d21c2066d8ef1e0beab690dd4 / Wed Aug 30 15:18:13 2017 +0200, even with ban_cutoff, we need to work the ban list and evaluate bans until we reach the cutoff. Any once we reached that, all we ware saving is the evaluation of the bans, we still need to do the work of nuking objects. So my guess would be that the bans you are seeing are so expensive to evaluate that your ban lurker has not even reached the cutoff. While I certainly do want to understand if the issue you are reporting is due to a genuine bug or inefficiency, I would recommend to get the issued bans under control in the first place. Also, taking traffic from a server with a busy ban lurker will most likely not help, but rather lead to a longer total time to work all bans: Cache access for requests tests bans concurrently to the ban lurker. Nils From daghf at varnish-software.com Thu Nov 30 14:20:10 2017 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Thu, 30 Nov 2017 15:20:10 +0100 Subject: Varnish 5.2.0 child panic In-Reply-To: <622a4a86-e251-165a-161b-ff96183a7f63@gmail.com> References: <622a4a86-e251-165a-161b-ff96183a7f63@gmail.com> Message-ID: I'd just like to mention we now have weekly builds based off the Varnish master branch. See https://packagecloud.io/varnishcache/varnish-weekly If you're already on 5.2.0, there shouldn't be too much of a divergence. On Thu, Nov 30, 2017 at 9:46 AM, Johan Hendriks wrote: > > Op 29/11/2017 om 19:11 schreef Hugues Alary: > > From the github bug report, it seems the bug was reported against 5.1 > > and 4.1 is not affected (the reporter went back to 4.1, from 5.1). > > > > @Guillaume, was "4.1" a typo? > > > > As for the possible resolution for this, it seems a patch was indeed > > written and merged in master, in which case I will wait for the next > > release and upgrade. > > > > I don't think there's any real solution to avoid this bug until the > > next varnish release, but, if there is, I'd love to know since this > > happens quite regularly on my end. > > > > Thanks! > > -Hugues > > > > > Maybe you can cherry pick the patch and build varnish yourself by > patching the source. > If it affects you that much it is worth the try. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Dag Haavi Finstad Software Developer | Varnish Software Mobile: +47 476 64 134 We Make Websites Fly! -------------- next part -------------- An HTML attachment was scrubbed... URL: From A.Hongens at netmatch.nl Thu Nov 30 19:09:38 2017 From: A.Hongens at netmatch.nl (=?utf-8?B?QW5nZWxvIEjDtm5nZW5z?=) Date: Thu, 30 Nov 2017 19:09:38 +0000 Subject: Upstream connection between nginx to varnish - HTTP2 In-Reply-To: References: Message-ID: <36bf3412c60d472fbb777daaffbc7e53@netmatch.nl> Yup, hitch works fine. From: varnish-misc [mailto:varnish-misc-bounces+a.hongens=netmatch.nl at varnish-cache.org] On Behalf Of Prem Kumar Sent: Wednesday, 29 November, 2017 09:24 To: varnish-misc at varnish-cache.org Subject: Upstream connection between nginx to varnish - HTTP2 Hi , Trying to use nginx to varnish connection as http2 instead of http1.1. https://www.nginx.com/blog/http2-module-nginx/#QandA Currently nginx is not supporting the upstream http2. Anyone knows hitch SSL proxy will do upstream connection to varnish through http2?. or any other SSL proxy would do http2 connection?. Thanks for the help. Thanks, Prem -------------- next part -------------- An HTML attachment was scrubbed... URL: From hugues at betabrand.com Thu Nov 30 19:16:13 2017 From: hugues at betabrand.com (Hugues Alary) Date: Thu, 30 Nov 2017 11:16:13 -0800 Subject: Varnish 5.2.0 child panic In-Reply-To: References: <622a4a86-e251-165a-161b-ff96183a7f63@gmail.com> Message-ID: That's great to know Dag, I'll use one of those packages, thanks! -H On Thu, Nov 30, 2017 at 6:20 AM, Dag Haavi Finstad < daghf at varnish-software.com> wrote: > I'd just like to mention we now have weekly builds based off the Varnish > master branch. > > See https://packagecloud.io/varnishcache/varnish-weekly > > If you're already on 5.2.0, there shouldn't be too much of a divergence. > > On Thu, Nov 30, 2017 at 9:46 AM, Johan Hendriks > wrote: > >> >> Op 29/11/2017 om 19:11 schreef Hugues Alary: >> > From the github bug report, it seems the bug was reported against 5.1 >> > and 4.1 is not affected (the reporter went back to 4.1, from 5.1). >> > >> > @Guillaume, was "4.1" a typo? >> > >> > As for the possible resolution for this, it seems a patch was indeed >> > written and merged in master, in which case I will wait for the next >> > release and upgrade. >> > >> > I don't think there's any real solution to avoid this bug until the >> > next varnish release, but, if there is, I'd love to know since this >> > happens quite regularly on my end. >> > >> > Thanks! >> > -Hugues >> > >> > >> Maybe you can cherry pick the patch and build varnish yourself by >> patching the source. >> If it affects you that much it is worth the try. >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > > > -- > Dag Haavi Finstad > Software Developer | Varnish Software > Mobile: +47 476 64 134 <+47%20476%2064%20134> > We Make Websites Fly! > -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Thu Nov 30 19:40:29 2017 From: slink at schokola.de (Nils Goroll) Date: Thu, 30 Nov 2017 20:40:29 +0100 Subject: Varnish Lurker is getting slower / Ban lists keeps increasing In-Reply-To: References: <12ca81a6-44fa-bfa0-046d-b051cabc92b6@schokola.de> Message-ID: <996d26d4-29ba-6c75-cc11-682d99bc08c2@schokola.de> FTR: I'll continue working on this with Olivier 1:1 and will report the outcome here. From slink at schokola.de Thu Nov 30 19:42:20 2017 From: slink at schokola.de (Nils Goroll) Date: Thu, 30 Nov 2017 20:42:20 +0100 Subject: Varnish 5.2.0 child panic In-Reply-To: References: Message-ID: <31615eeb-dd5d-fa42-f6b5-2bc3f9415d96@schokola.de> On 29/11/17 19:11, Hugues Alary wrote: > > I don't think there's any real solution to avoid this bug until the next varnish > release, but, if there is, I'd love to know since this happens quite regularly > on my end. This bug can only be avoided by not getting into vcl_backend_error with a busy object which has a Vary header.