From FULLERD at parliament.uk Wed Aug 1 11:05:25 2018 From: FULLERD at parliament.uk (FULLER, David) Date: Wed, 1 Aug 2018 11:05:25 +0000 Subject: Memory utilisation gradually increasing In-Reply-To: References:

<92AD7F3D-D5F9-435B-9712-48F8C735642A@parliament.uk>

<99B4A9D4-5D48-46BC-9900-E38122BC36A0@parliament.uk> <83715EF0-9A49-4F3A-9632-DBC8F2511B90@parliament.uk> <892D91DE-D9E0-4E3F-BE71-3A202D257368@parliament.uk> Message-ID: <9DE7BDAA-3464-4C03-9E70-2A98FC9EB0B1@parliament.uk> That?s great, thanks. Does the first VCL listed (labelled boot in column 4 ? ?available cold/cold 0 boot?) need to stay loaded and is there a recommended amount of VCLs to keep loaded? From: Dridi Boukelmoune Date: Tuesday, 31 July 2018 at 10:03 To: "FULLER, David" Cc: Guillaume Quintard , varnish-misc Subject: Re: Memory utilisation gradually increasing On Mon, Jul 30, 2018 at 2:50 PM, FULLER, David wrote: > Thanks Dridi, > > After further investigation I found that we do have a python/cron job running that checks for backend changes and if so does a vcl.load. This has resulted in a growing number of VCLs being loaded, as you suspected. After a redeploying the Varnish container this morning and then monitoring varnishadm we have gone from around 30 VCLs loaded to over 200 in a few hours. Is there a way to limit the number of VCLs that can be loaded, with older ones being dropped as new ones are loaded? For Varnish 6.0 we released a new varnishreload script, see its usage: https://github.com/varnishcache/pkg-varnish-cache/blob/0ad2f22629c4a368959c423a19e352c9c6c79682/systemd/varnishreload#L46-L74 The main goals were to unify the reload script on all the platforms we support and simplify the logic by only supporting the privileged "service manager" use case. As a result it was also easy to add the -m option to discard old reloads if have more than "maximum" reload VCLs. In other words, it's up to the python script scheduled by your cron job to keep track of reloads and discard older ones according to your policy. Dridi UK Parliament Disclaimer: This e-mail is confidential to the intended recipient. If you have received it in error, please notify the sender and delete it from your system. Any unauthorised use, disclosure, or copying is not permitted. This e-mail has been checked for viruses, but no liability is accepted for any damage caused by any virus transmitted by this e-mail. This e-mail address is not secure, is not encrypted and should not be used for sensitive data. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Thu Aug 2 10:19:43 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 2 Aug 2018 12:19:43 +0200 Subject: Memory utilisation gradually increasing In-Reply-To: <9DE7BDAA-3464-4C03-9E70-2A98FC9EB0B1@parliament.uk> References:

<92AD7F3D-D5F9-435B-9712-48F8C735642A@parliament.uk>

<99B4A9D4-5D48-46BC-9900-E38122BC36A0@parliament.uk> <83715EF0-9A49-4F3A-9632-DBC8F2511B90@parliament.uk> <892D91DE-D9E0-4E3F-BE71-3A202D257368@parliament.uk> <9DE7BDAA-3464-4C03-9E70-2A98FC9EB0B1@parliament.uk> Message-ID: On Wed, Aug 1, 2018 at 1:05 PM, FULLER, David wrote: > That?s great, thanks. Does the first VCL listed (labelled boot in column 4 > ? ?available cold/cold 0 boot?) need to stay loaded and is there > a recommended amount of VCLs to keep loaded? There's nothing special about "boot", it's just the VCL name given to the one loaded at launch time using the -f or -b option. For the amount of VCLs to keep loaded, I'd say enough to feel confident about rollbacks. If it takes you 30m to link a problem to a VCL change, then keep 1h worth of reloads for example. Again, your policy, your rules :) Cheers From junaid.mukhtar at gmail.com Tue Aug 7 07:43:46 2018 From: junaid.mukhtar at gmail.com (Junaid Mukhtar) Date: Tue, 7 Aug 2018 08:43:46 +0100 Subject: libvmod_saintmode.so Availability Message-ID: Hi I am in process of upgrading from Varnish3 to Varnish4. As part of change saintmode is moved out to varnish modules and we need to include libvmod_saintmode.so to use the saintmode functionality. The problem is I am unable to find "libvmod_saintmode.so" file. I tried to compile it from the source code but got stuck with rst2man package as it's a required package that comes with python-docutils but even with latest python-docutils I am unable to compile from source code. Any help in getting the libvmod_saintmode.so file would be really helpful. -------- Regards, Junaid -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Tue Aug 7 11:34:19 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 7 Aug 2018 13:34:19 +0200 Subject: libvmod_saintmode.so Availability In-Reply-To: References: Message-ID: > Any help in getting the libvmod_saintmode.so file would be really helpful. It's part of varnish-modules now: https://github.com/varnish/varnish-modules You can find the manual here: https://github.com/varnish/varnish-modules/blob/master/docs/vmod_saintmode.rst#vmod_saintmode Cheers From s.priebe at profihost.ag Tue Aug 7 13:27:10 2018 From: s.priebe at profihost.ag (Stefan Priebe - Profihost AG) Date: Tue, 7 Aug 2018 15:27:10 +0200 Subject: vcl broken after upgrade from 4.1 to 5.0 Message-ID: Hello, our varnish vcl is broken after upgrading from 4.1 to 5.0 but i can't find any documentation hint and i also do not find any solution. The old varnish conf uses std.collect to concatenate alls Set-Cookies header from the backend. Then does a lot of "magic" using regsuball. At the end it uses the following Code to split the header into multiple headers again: rebsuball(cookies, ", ", {" Set-Cookies: "}) { allows \n / new lines so every new line starts with Set-Cookies: do have multiple Set-Cookies headers. This does not seem to work anymore with 5.0 the Set-Cookies Header is still a one line. I was also not able to find the opposite of std.collect to split the line again into multiple headers. Thanks! Greets, Stefan From dridi at varni.sh Tue Aug 7 18:10:07 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 7 Aug 2018 20:10:07 +0200 Subject: vcl broken after upgrade from 4.1 to 5.0 In-Reply-To: References: Message-ID: On Tue, Aug 7, 2018 at 3:27 PM, Stefan Priebe - Profihost AG wrote: > Hello, > > our varnish vcl is broken after upgrading from 4.1 to 5.0 but i can't > find any documentation hint and i also do not find any solution. > > The old varnish conf uses std.collect to concatenate alls Set-Cookies > header from the backend. > > Then does a lot of "magic" using regsuball. > > At the end it uses the following Code to split the header into multiple > headers again: > > rebsuball(cookies, ", ", {" > Set-Cookies: "}) > > { allows \n / new lines so every new line starts with Set-Cookies: do > have multiple Set-Cookies headers. Wow, that's like evil clever! > This does not seem to work anymore with 5.0 the Set-Cookies Header is > still a one line. I was also not able to find the opposite of > std.collect to split the line again into multiple headers. That's a nice trick, and technically illegal although, well... Good job for such a hack! Some time after 4.1 std.collect grew support for an optional separator other than ", " especially for brain-dead cookie use cases. FYI cookies are such a mess because they contain a date formatted with a comma followed by a space, but not quoted, so they can't be split like other headers such as the Accept* headers. To add insult to the injury, HTTP/2 requires cookies to be individual headers nevertheless [1] to help HPACK compression effectiveness. Since 5.0 is the first release with experimental h2 support, we may have tightened headers handling and broken your (illegal but oh so clever) hack. I don't have time to look [2] closer now, but it would indeed be nice to have the reciprocal operation to std.collect, so you could use a safe [3] separator to collect, make your changes and then split using the same separator. std.collect(beresp.http.set-cookie, "~SEP~"); /* do stuff to beresp.http.set-cookie */ std.split(beresp.http.set-cookie, "~SEP~"); Dridi [1] remember, impossible for h1 [2] but I'll find time to review a patch ;-) [3] if you also control the backend in that regard From s.priebe at profihost.ag Wed Aug 8 08:41:36 2018 From: s.priebe at profihost.ag (Stefan Priebe - Profihost AG) Date: Wed, 8 Aug 2018 10:41:36 +0200 Subject: vcl broken after upgrade from 4.1 to 5.0 In-Reply-To: References:

Message-ID: Hello, Am 07.08.2018 um 20:10 schrieb Dridi Boukelmoune: > On Tue, Aug 7, 2018 at 3:27 PM, Stefan Priebe - Profihost AG > wrote: >> Hello, >> >> our varnish vcl is broken after upgrading from 4.1 to 5.0 but i can't >> find any documentation hint and i also do not find any solution. >> >> The old varnish conf uses std.collect to concatenate alls Set-Cookies >> header from the backend. >> >> Then does a lot of "magic" using regsuball. >> >> At the end it uses the following Code to split the header into multiple >> headers again: >> >> rebsuball(cookies, ", ", {" >> Set-Cookies: "}) >> >> { allows \n / new lines so every new line starts with Set-Cookies: do >> have multiple Set-Cookies headers. > > Wow, that's like evil clever! Not my code ;-) I'm just the person who should fix it. >> This does not seem to work anymore with 5.0 the Set-Cookies Header is >> still a one line. I was also not able to find the opposite of >> std.collect to split the line again into multiple headers. > > That's a nice trick, and technically illegal although, well... Good > job for such a hack! > > Some time after 4.1 std.collect grew support for an optional separator > other than ", " especially for brain-dead cookie use cases. FYI > cookies are such a mess because they contain a date formatted with a > comma followed by a space, but not quoted, so they can't be split like > other headers such as the Accept* headers. > > To add insult to the injury, HTTP/2 requires cookies to be individual > headers nevertheless [1] to help HPACK compression effectiveness. > Since 5.0 is the first release with experimental h2 support, we may > have tightened headers handling and broken your (illegal but oh so > clever) hack. Yes i know but we can't change the backend application so we need to mess with their cookie stuff inside varnish ;-( and the vendor just says use varnish 4.1 we don't support 5.X - most probably because the hack isn't working anymore. > I don't have time to look [2] closer now, but it would indeed be nice > to have the reciprocal operation to std.collect, so you could use a > safe [3] separator to collect, make your changes and then split using > the same separator. > > std.collect(beresp.http.set-cookie, "~SEP~"); > /* do stuff to beresp.http.set-cookie */ > std.split(beresp.http.set-cookie, "~SEP~"); Yes but i'm sorry i'm unable to provide code. Is there any way to use something like a loop or recursion? In that case i would be able to modify a single set-cookie header like we need it. I also wasn't able to solve it by mod_headers. Thanks! Greets, Stefan From dridi at varni.sh Wed Aug 8 12:14:48 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 8 Aug 2018 14:14:48 +0200 Subject: vcl broken after upgrade from 4.1 to 5.0 In-Reply-To: References:

Message-ID: > Yes but i'm sorry i'm unable to provide code. Is there any way to use > something like a loop or recursion? In that case i would be able to > modify a single set-cookie header like we need it. A "foreach" construct would indeed help for the case of multiple headers and it has been discussed in the past but to no avail. It's not trivial to add this to VCL and ensure soundness. > I also wasn't able to solve it by mod_headers. I'm not really surprised. This would require a VMOD of its own. Did you look at alternatives such as vmod-cookie and vmod-cookieplus? https://github.com/varnish/varnish-modules/blob/master/docs/vmod_cookie.rst#vmod_cookie https://docs.varnish-software.com/varnish-cache-plus/vmods/cookieplus/#description Maybe one of those provides enough capabilities for what you need to do. Dridi From s.priebe at profihost.ag Wed Aug 8 13:07:28 2018 From: s.priebe at profihost.ag (Stefan Priebe - Profihost AG) Date: Wed, 8 Aug 2018 15:07:28 +0200 Subject: vcl broken after upgrade from 4.1 to 5.0 In-Reply-To: References:

Message-ID: Am 08.08.2018 um 14:14 schrieb Dridi Boukelmoune: >> Yes but i'm sorry i'm unable to provide code. Is there any way to use >> something like a loop or recursion? In that case i would be able to >> modify a single set-cookie header like we need it. > > A "foreach" construct would indeed help for the case of multiple > headers and it has been discussed in the past but to no avail. It's > not trivial to add this to VCL and ensure soundness. > >> I also wasn't able to solve it by mod_headers. > > I'm not really surprised. This would require a VMOD of its own. > > Did you look at alternatives such as vmod-cookie and vmod-cookieplus? > > https://github.com/varnish/varnish-modules/blob/master/docs/vmod_cookie.rst#vmod_cookie > https://docs.varnish-software.com/varnish-cache-plus/vmods/cookieplus/#description > > Maybe one of those provides enough capabilities for what you need to do. Thanks again. Sadly it seems they do not help. The Problem is: 1.) there are multiple set-cookies headers from the backend 2.) the vcl than uses multiple regsuball to modify them 3.) the multiple set-cookies headers have to be part of the response in the same order as before except for those who are empty after the regsuball ;-( very ugly. Stefan > > Dridi > From neovichnn at gmail.com Mon Aug 20 04:49:10 2018 From: neovichnn at gmail.com (neo) Date: Mon, 20 Aug 2018 07:49:10 +0300 Subject: how to install varnish6 on ubuntu 16.04 Message-ID: <22b44ad0-5949-141f-ea77-90aed0b3bf81@gmail.com> I used this instruction https://packagecloud.io/varnishcache/varnish60/install#manual-deb put to /etc/apt/sources.list.d/varnishcache_varnish60.list deb https://packagecloud.io/varnishcache/varnish60/ubuntu/xenial 16.04 LTS Xenial Xerus main deb-src https://packagecloud.io/varnishcache/varnish60/ubuntu/xenial 16.04 LTS Xenial Xerus main and when do sudo apt-get update got an error: ... W: The repository 'https://packagecloud.io/varnishcache/varnish60/ubuntu/xenial 16.04 Release' does not have a Release file. N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use. N: See apt-secure(8) manpage for repository creation and user configuration details. E: Failed to fetch https://packagecloud.io/varnishcache/varnish60/ubuntu/xenial/dists/16.04/LTS/source/Sources 404? Not Found E: Some index files failed to download. They have been ignored, or old ones used instead. neo at neo:/tmp/varnish-cache$ The whole code: https://paste.ubuntu.com/p/6FkQZbCPf6/ How to install varnish6 ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Tue Aug 21 18:57:33 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Tue, 21 Aug 2018 11:57:33 -0700 Subject: how to install varnish6 on ubuntu 16.04 In-Reply-To: <22b44ad0-5949-141f-ea77-90aed0b3bf81@gmail.com> References: <22b44ad0-5949-141f-ea77-90aed0b3bf81@gmail.com> Message-ID: Hi, Why aren't you following the instruction you link to? the file should contain: deb https://packagecloud.io/varnishcache/varnish60/ubuntu/ xenial main deb-src https://packagecloud.io/varnishcache/varnish60/ubuntu/ xenial main also, note that there's a script automating the process that would save you this hassle: https://packagecloud.io/varnishcache/varnish60/install Kind regards, -- Guillaume Quintard On Sun, Aug 19, 2018 at 9:49 PM, neo wrote: > I used this instruction https://packagecloud.io/varnishcache/varnish60/ > install#manual-deb > > put to /etc/apt/sources.list.d/varnishcache_varnish60.list > > deb https://packagecloud.io/varnishcache/varnish60/ubuntu/xenial 16.04 > LTS Xenial Xerus main > deb-src https://packagecloud.io/varnishcache/varnish60/ubuntu/xenial > 16.04 LTS Xenial Xerus main > > and when do sudo apt-get update got an error: > > ... > > W: The repository 'https://packagecloud.io/varnishcache/varnish60/ubuntu/ > xenial 16.04 Release' does not have a Release file. > N: Data from such a repository can't be authenticated and is therefore > potentially dangerous to use. > N: See apt-secure(8) manpage for repository creation and user > configuration details. > E: Failed to fetch https://packagecloud.io/varnishcache/varnish60/ubuntu/ > xenial/dists/16.04/LTS/source/Sources 404 Not Found > E: Some index files failed to download. They have been ignored, or old > ones used instead. > neo at neo:/tmp/varnish-cache$ > > > The whole code: https://paste.ubuntu.com/p/6FkQZbCPf6/ > > How to install varnish6 ? > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jboyle at quotient-inc.com Fri Aug 24 20:40:04 2018 From: jboyle at quotient-inc.com (James Boyle) Date: Fri, 24 Aug 2018 16:40:04 -0400 Subject: GPG signatures for Varnish 4.1 respository Message-ID: <44d6cabd-8efb-2987-372f-8ad899a661ac@quotient-inc.com> Hello, I was wondering if the Varnish maintainers would consider adding GPG signatures to the packages in the Varnish 4.1 repository (https://packagecloud.io/varnishcache/varnish41/el/7/x86_64). It would increase the level of confidence that those packages have not been tampered with since being built. For custom repositories I maintain, it is as simple as running the following in the appropriate directory after the build process is complete, though, admittedly, I'm unfamiliar with the build process in use on your side. rpmsign -D '_gpg_name jboyle at quotient-inc.com' --addsign *.rpm Also, I contacted the folks at packagecloud.io first -- they recommended I share that they also have some support for GPG (public) keys.? They gave me this link: https://blog.packagecloud.io/eng/2017/06/08/announcing-package-signing-gpg-key-support/ However, I'd most like to have signatures embedded in the packages so I can set gpgcheck=1 in my yum repository configuration. Thank you! --James