From guillaume at varnish-software.com Thu Apr 1 01:13:58 2021 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 31 Mar 2021 18:13:58 -0700 Subject: Introducing bob Message-ID: Hi everyone, Sometime ago I built and started to use a little tool that I'd like to share with you, in the hope that it'll make your life easier too. This tool is named "bob", and is available here: https://github.com/varnish/toolbox/tree/master/bob The short of it is that on compatible repositories, if you have bob and docker installed, you can start building without worrying about dependencies. For varnish, you can just go: bob ./autogen.des bob make bob make check And things should just work, the files are created by the container in your work directory, with the right owner and permissions, but without you having to install anything extra. And that allows you to code from your host machine with the IDE of your choice, and to rely on bob only for the compilation steps. Internally, bob just: - grabs a a Dockerfile in the .circleci/ directory and builds the corresponding image - creates a container user with the same UID/GID that is running on the host - mounts your home as the container's home - run a command - destroy the container The goal here is to provide a frictionless start to new users so they can get cracking faster, but also to allow testing on other platforms. For example, you can set the BOB_DIST env variable to run other containers: BOB_DIST=ubuntu bob ./autogen.des BOB_DIST=ubuntu bob make BOB_DIST=ubuntu bob make check This can prove very useful to test different environments (cf. new compiler version issues) If you want to recreate the image (which includes a "docker pull"), just run it without arguments: bob If you need to get into the container to run a couple of commands: bob bash I've just pushed the necessary Dockerfiles to enable it on: - https://github.com/varnishcache/varnish-cache (centos by default, and BOB_DIST support for ubuntu, archlinux and alpine) - https://github.com/varnish/varnish-modules (centos only) - https://github.com/varnish/hitch/ (centos by default, and BOB_DIST support for ubuntu, archlinux and alpine) So those are ready to go, but the tool is meant to be generic and should work with pretty much any repo as long as you create a Dockerfile for it in bob/, .bob/, .circleci/ or .github. Of course, this is completely optional, and if you don't care about it, this should be completely invisible to you. However, if you do find it useful, or PRs and bug trackers are open, I'm eager to get more feedback about it. Cheers! -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: From joh.hendriks at gmail.com Sun Apr 4 07:49:56 2021 From: joh.hendriks at gmail.com (Johan Hendriks) Date: Sun, 4 Apr 2021 09:49:56 +0200 Subject: Certificate error on varnish-cache.org Message-ID: Hello all. I get a certificate error on the webpage of varnish-cache.org The certificate has been expired. Regards, Johan -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Sun Apr 4 15:56:48 2021 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Sun, 04 Apr 2021 15:56:48 +0000 Subject: Certificate error on varnish-cache.org In-Reply-To: References: Message-ID: <10828.1617551808@critter.freebsd.dk> -------- Fixed. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From justinl at arena.net Tue Apr 6 16:10:12 2021 From: justinl at arena.net (Justin Lloyd) Date: Tue, 6 Apr 2021 16:10:12 +0000 Subject: Cookie VMOD keep/filter documentation issue Message-ID: Hi all, I'm confused on whether cookie.keep_re? and cookie.filter_re? should work with CSV strings. The documentation at https://varnish-cache.org/docs/trunk/reference/vmod_cookie.html does not explicitly say that those two functions can take a CSV string like the docs for keep? and filter? do, but the example for keep_re? uses such string, indicating that it should be able to. However, testing with varnishtest? definitely shows that the _re? functions do not work with CSV strings. Can anyone clarify what the truth is supposed to be? Thanks, Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Tue Apr 6 17:29:23 2021 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 6 Apr 2021 17:29:23 +0000 Subject: Cookie VMOD keep/filter documentation issue In-Reply-To: References: Message-ID: On Tue, Apr 6, 2021 at 4:11 PM Justin Lloyd wrote: > > Hi all, > > I'm confused on whether cookie.keep_re and cookie.filter_re should work with CSV strings. The documentation at https://varnish-cache.org/docs/trunk/reference/vmod_cookie.html does not explicitly say that those two functions can take a CSV string like the docs for keep and filter do, but the example for keep_re uses such string, indicating that it should be able to. However, testing with varnishtest definitely shows that the _re functions do not work with CSV strings. Can anyone clarify what the truth is supposed to be? Hi, Looking at the examples right now it is clear [1] that filter() and keep() each take a CSV string. It is also clear [1] that filter_re() takes a single regular expression [2]. However, the keep_re() example is both misleading and wrong. Thanks for bringing this to our attention! https://github.com/varnishcache/varnish-cache/commit/606977bbfb624ead38e9c8648beac0b3906a4294 Dridi [1] to me [2] not to be confused with a singular expression From justinl at arena.net Tue Apr 6 17:35:19 2021 From: justinl at arena.net (Justin Lloyd) Date: Tue, 6 Apr 2021 17:35:19 +0000 Subject: Cookie VMOD keep/filter documentation issue In-Reply-To: References: , Message-ID: Hi Dridi, Thanks for confirming that! However, is there then a way to get the effect of being able to pass a regex to keep_re() and filter_re()? I just started working with the cookie vmod this morning and this would be a useful feature. As I understand it, Varnish 6.4 (we're on 6.5) replaced filter() and filter_except() with keep() and keep_re() and that the previous functions could take CSV strings. Thanks, Justin ________________________________ From: Dridi Boukelmoune Sent: Tuesday, April 6, 2021 10:29 AM To: Justin Lloyd Cc: varnish-misc at varnish-cache.org Subject: Re: Cookie VMOD keep/filter documentation issue On Tue, Apr 6, 2021 at 4:11 PM Justin Lloyd wrote: > > Hi all, > > I'm confused on whether cookie.keep_re and cookie.filter_re should work with CSV strings. The documentation at https://varnish-cache.org/docs/trunk/reference/vmod_cookie.html does not explicitly say that those two functions can take a CSV string like the docs for keep and filter do, but the example for keep_re uses such string, indicating that it should be able to. However, testing with varnishtest definitely shows that the _re functions do not work with CSV strings. Can anyone clarify what the truth is supposed to be? Hi, Looking at the examples right now it is clear [1] that filter() and keep() each take a CSV string. It is also clear [1] that filter_re() takes a single regular expression [2]. However, the keep_re() example is both misleading and wrong. Thanks for bringing this to our attention! https://github.com/varnishcache/varnish-cache/commit/606977bbfb624ead38e9c8648beac0b3906a4294 Dridi [1] to me [2] not to be confused with a singular expression -------------- next part -------------- An HTML attachment was scrubbed... URL: From justinl at arena.net Tue Apr 6 17:41:29 2021 From: justinl at arena.net (Justin Lloyd) Date: Tue, 6 Apr 2021 17:41:29 +0000 Subject: Cookie VMOD keep/filter documentation issue In-Reply-To: References: , , Message-ID: Forget I asked, that was a dumb question since it's a regex and | can be used. /sigh Justin ________________________________ From: Justin Lloyd Sent: Tuesday, April 6, 2021 10:35 AM To: Dridi Boukelmoune Cc: varnish-misc at varnish-cache.org Subject: Re: Cookie VMOD keep/filter documentation issue Hi Dridi, Thanks for confirming that! However, is there then a way to get the effect of being able to pass a regex to keep_re() and filter_re()? I just started working with the cookie vmod this morning and this would be a useful feature. As I understand it, Varnish 6.4 (we're on 6.5) replaced filter() and filter_except() with keep() and keep_re() and that the previous functions could take CSV strings. Thanks, Justin ________________________________ From: Dridi Boukelmoune Sent: Tuesday, April 6, 2021 10:29 AM To: Justin Lloyd Cc: varnish-misc at varnish-cache.org Subject: Re: Cookie VMOD keep/filter documentation issue On Tue, Apr 6, 2021 at 4:11 PM Justin Lloyd wrote: > > Hi all, > > I'm confused on whether cookie.keep_re and cookie.filter_re should work with CSV strings. The documentation at https://varnish-cache.org/docs/trunk/reference/vmod_cookie.html does not explicitly say that those two functions can take a CSV string like the docs for keep and filter do, but the example for keep_re uses such string, indicating that it should be able to. However, testing with varnishtest definitely shows that the _re functions do not work with CSV strings. Can anyone clarify what the truth is supposed to be? Hi, Looking at the examples right now it is clear [1] that filter() and keep() each take a CSV string. It is also clear [1] that filter_re() takes a single regular expression [2]. However, the keep_re() example is both misleading and wrong. Thanks for bringing this to our attention! https://github.com/varnishcache/varnish-cache/commit/606977bbfb624ead38e9c8648beac0b3906a4294 Dridi [1] to me [2] not to be confused with a singular expression -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Tue Apr 6 18:28:47 2021 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 6 Apr 2021 18:28:47 +0000 Subject: Cookie VMOD keep/filter documentation issue In-Reply-To: References: Message-ID: On Tue, Apr 6, 2021 at 5:41 PM Justin Lloyd wrote: > > Forget I asked, that was a dumb question since it's a regex and | can be used. /sigh Yes, "|" is an option. > Hi Dridi, > > Thanks for confirming that! However, is there then a way to get the effect of being able to pass a regex to keep_re() and filter_re()? I just started working with the cookie vmod this morning and this would be a useful feature. As I understand it, Varnish 6.4 (we're on 6.5) replaced filter() and filter_except() with keep() and keep_re() and that the previous functions could take CSV strings. I think there is a misunderstanding here. We renamed filter_except() to keep() because it had better semantics, much easier to understand. The filter_re() and keep_re() functions are regular expression variants for basically the same operations: filtering some cookies out, or keeping a subset of the cookies. Cheers, Dridi From mandys at gmail.com Thu Apr 15 06:07:01 2021 From: mandys at gmail.com (Maninder Singh) Date: Thu, 15 Apr 2021 11:37:01 +0530 Subject: Understanding 503s Message-ID: Hi, I need some help understanding why the below 503 is happening. I am logging 503s to a separate file and then querying as below. varnishlog -q "BerespStatus eq 503" -w /whatever/file varnishlog -r /whatever/file What's going wrong here ? What should I be looking at ? Please let me know. * << BeReq >> 45926757 - Begin bereq 45926756 pass - Timestamp Start: 1618461576.793111 0.000000 0.000000 - BereqMethod POST - BereqURL /index.php?&u=85&tr=aa - BereqProtocol HTTP/1.1 - BereqHeader X-Forwarded-Proto: https - BereqHeader X-Forwarded-Port: 443 - BereqHeader Host: graph.com - BereqHeader X-Amzn-Trace-Id: Root=1-xxx - BereqHeader Content-Length: 793 - BereqHeader sec-ch-ua: "Google Chrome";v="89", "Chromium";v="89", ";Not A Brand";v="99" - BereqHeader accept: application/json, text/javascript, */*; q=0.01 - BereqHeader sec-ch-ua-mobile: ?0 - BereqHeader user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36 - BereqHeader content-type: application/x-www-form-urlencoded; charset=UTF-8 - BereqHeader origin: https://mandy..com - BereqHeader sec-fetch-site: same-site - BereqHeader sec-fetch-mode: cors - BereqHeader sec-fetch-dest: empty - BereqHeader referer: https://mandy.com/ - BereqHeader accept-encoding: gzip, deflate, br - BereqHeader accept-language: en-US,en;q=0.9 - BereqHeader X-Forwarded-For: 103.67.157.20, 10.0.0.170 - BereqHeader browser: other - BereqHeader serverIp: 10.0.3.237 - BereqHeader serverId: abc01 - BereqHeader X-Varnish: 45926757 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 32 reload_2021-04-13T130756.default 127.0.0.1 8080 127.0.0.1 56176 - BackendStart 127.0.0.1 8080 - Timestamp Bereq: 1618461577.074387 0.281276 0.281276 - FetchError http first read error: EOF - BackendClose 32 reload_2021-04-13T130756.default - Timestamp Beresp: 1618461577.074430 0.281319 0.000043 - Timestamp Error: 1618461577.074434 0.281323 0.000004 - BerespProtocol HTTP/1.1 - BerespStatus 503 - BerespReason Service Unavailable - BerespReason Backend fetch failed - BerespHeader Date: Thu, 15 Apr 2021 04:39:37 GMT - BerespHeader Server: Varnish - VCL_call BACKEND_ERROR - BerespHeader Content-Type: text/html; charset=utf-8 - BerespHeader Retry-After: 5 - VCL_return deliver - Storage malloc Transient - ObjProtocol HTTP/1.1 - ObjStatus 503 - ObjReason Backend fetch failed - ObjHeader Date: Thu, 15 Apr 2021 04:39:37 GMT - ObjHeader Server: Varnish - ObjHeader Content-Type: text/html; charset=utf-8 - ObjHeader Retry-After: 5 - Length 1288 - BereqAcct 948 793 1741 0 0 0 - End -------------- next part -------------- An HTML attachment was scrubbed... URL: From mandys at gmail.com Thu Apr 15 06:22:29 2021 From: mandys at gmail.com (Maninder Singh) Date: Thu, 15 Apr 2021 11:52:29 +0530 Subject: Understanding 503s In-Reply-To: References: <5074fde7-c89f-4158-a23e-f3ffdfb8c5c4.de476353-2903-4c68-bec2-3738a7d98eb8.872b7609-9509-4d43-9e4c-d8e73d5cd0d1@emailsignatures365.codetwo.com> Message-ID: I have that defined as 2 minutes. backend default { .host = "127.0.0.1"; .port = "8080"; .first_byte_timeout = 120s; } That's why this error is puzzling. Any other timeouts ( connect ? ) etc to look at ? Also, in the above dump, how much time did it take ? To me it looks like it was closed within a second ? - BackendOpen 32 reload_2021-04-13T130756.default 127.0.0.1 8080 127.0.0.1 56176 - BackendStart 127.0.0.1 8080 - Timestamp Bereq: 1618461577.074387 0.281276 0.281276 - FetchError http first read error: EOF - BackendClose 32 reload_2021-04-13T130756.default - Timestamp Beresp: 1618461577.074430 0.281319 0.000043 - Timestamp Error: 1618461577.074434 0.281323 0.000004 On Thu, 15 Apr 2021 at 11:42, Frands Bjerring Hansen wrote: > Look at the fetch error: http first read error: EOF > > > > Perhaps the backend closed the connection before sending any data or the > first_byte_timeout defined for the backend has been reached. The default is > 60 seconds. > > > > > > > > */ Frands Bjerring Hansen* > Head of Technology, Linux > Office: +45 70 40 00 00 > frands.hansen at team.blue > > *Operations - Linux* > team.blue Denmark A/S > H?jvangen 4 > 8660 Skanderborg > Denmark > CVR: 29412006 > > > > > On 15/04/2021, 08.08, "varnish-misc" zitcom.dk at varnish-cache.org> wrote: > > Hi, > > I need some help understanding why the below 503 is happening. > > > > I am logging 503s to a separate file and then querying as below. > > > > varnishlog -q "BerespStatus eq 503" -w /whatever/file > > varnishlog -r /whatever/file > > > > What's going wrong here ? > > What should I be looking at ? > > > > Please let me know. > > > > * << BeReq >> 45926757 > > - Begin bereq 45926756 pass > > - Timestamp Start: 1618461576.793111 0.000000 0.000000 > > - BereqMethod POST > > - BereqURL /index.php?&u=85&tr=aa > > - BereqProtocol HTTP/1.1 > > - BereqHeader X-Forwarded-Proto: https > > - BereqHeader X-Forwarded-Port: 443 > > - BereqHeader Host: graph.com > > - BereqHeader X-Amzn-Trace-Id: Root=1-xxx > > - BereqHeader Content-Length: 793 > > - BereqHeader sec-ch-ua: "Google Chrome";v="89", "Chromium";v="89", > ";Not A Brand";v="99" > > - BereqHeader accept: application/json, text/javascript, */*; q=0.01 > > - BereqHeader sec-ch-ua-mobile: ?0 > > - BereqHeader user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36 > > - BereqHeader content-type: application/x-www-form-urlencoded; > charset=UTF-8 > > - BereqHeader origin: https://mandy..com > > - BereqHeader sec-fetch-site: same-site > > - BereqHeader sec-fetch-mode: cors > > - BereqHeader sec-fetch-dest: empty > > - BereqHeader referer: https://mandy.com/ > > - BereqHeader accept-encoding: gzip, deflate, br > > - BereqHeader accept-language: en-US,en;q=0.9 > > - BereqHeader X-Forwarded-For: 103.67.157.20, 10.0.0.170 > > - BereqHeader browser: other > > - BereqHeader serverIp: 10.0.3.237 > > - BereqHeader serverId: abc01 > > - BereqHeader X-Varnish: 45926757 > > - VCL_call BACKEND_FETCH > > - VCL_return fetch > > - BackendOpen 32 reload_2021-04-13T130756.default 127.0.0.1 8080 > 127.0.0.1 56176 > > - BackendStart 127.0.0.1 8080 > > - Timestamp Bereq: 1618461577.074387 0.281276 0.281276 > > - FetchError http first read error: EOF > > - BackendClose 32 reload_2021-04-13T130756.default > > - Timestamp Beresp: 1618461577.074430 0.281319 0.000043 > > - Timestamp Error: 1618461577.074434 0.281323 0.000004 > > - BerespProtocol HTTP/1.1 > > - BerespStatus 503 > > - BerespReason Service Unavailable > > - BerespReason Backend fetch failed > > - BerespHeader Date: Thu, 15 Apr 2021 04:39:37 GMT > > - BerespHeader Server: Varnish > > - VCL_call BACKEND_ERROR > > - BerespHeader Content-Type: text/html; charset=utf-8 > > - BerespHeader Retry-After: 5 > > - VCL_return deliver > > - Storage malloc Transient > > - ObjProtocol HTTP/1.1 > > - ObjStatus 503 > > - ObjReason Backend fetch failed > > - ObjHeader Date: Thu, 15 Apr 2021 04:39:37 GMT > > - ObjHeader Server: Varnish > > - ObjHeader Content-Type: text/html; charset=utf-8 > > - ObjHeader Retry-After: 5 > > - Length 1288 > > - BereqAcct 948 793 1741 0 0 0 > > - End > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: team.blue-color-rgb-small-mail-signature_78509216-de0a-4033-a847-aa8f4610dc1b.png Type: image/png Size: 3918 bytes Desc: not available URL: From mandys at gmail.com Thu Apr 15 06:23:48 2021 From: mandys at gmail.com (Maninder Singh) Date: Thu, 15 Apr 2021 11:53:48 +0530 Subject: Understanding 503s In-Reply-To: References: <5074fde7-c89f-4158-a23e-f3ffdfb8c5c4.de476353-2903-4c68-bec2-3738a7d98eb8.872b7609-9509-4d43-9e4c-d8e73d5cd0d1@emailsignatures365.codetwo.com> Message-ID: Also, backend is apache 2.4 and running php-fpm. On Thu, 15 Apr 2021 at 11:52, Maninder Singh wrote: > I have that defined as 2 minutes. > backend default { > .host = "127.0.0.1"; > .port = "8080"; > .first_byte_timeout = 120s; > } > > That's why this error is puzzling. > > Any other timeouts ( connect ? ) etc to look at ? > > Also, in the above dump, how much time did it take ? > > To me it looks like it was closed within a second ? > > > - BackendOpen 32 reload_2021-04-13T130756.default 127.0.0.1 8080 > 127.0.0.1 56176 > - BackendStart 127.0.0.1 8080 > - Timestamp Bereq: 1618461577.074387 0.281276 0.281276 > - FetchError http first read error: EOF > - BackendClose 32 reload_2021-04-13T130756.default > - Timestamp Beresp: 1618461577.074430 0.281319 0.000043 > - Timestamp Error: 1618461577.074434 0.281323 0.000004 > > On Thu, 15 Apr 2021 at 11:42, Frands Bjerring Hansen > wrote: > >> Look at the fetch error: http first read error: EOF >> >> >> >> Perhaps the backend closed the connection before sending any data or the >> first_byte_timeout defined for the backend has been reached. The default is >> 60 seconds. >> >> >> >> >> >> >> >> */ Frands Bjerring Hansen* >> Head of Technology, Linux >> Office: +45 70 40 00 00 >> frands.hansen at team.blue >> >> *Operations - Linux* >> team.blue Denmark A/S >> H?jvangen 4 >> 8660 Skanderborg >> Denmark >> CVR: 29412006 >> >> >> >> >> On 15/04/2021, 08.08, "varnish-misc" > zitcom.dk at varnish-cache.org> wrote: >> >> Hi, >> >> I need some help understanding why the below 503 is happening. >> >> >> >> I am logging 503s to a separate file and then querying as below. >> >> >> >> varnishlog -q "BerespStatus eq 503" -w /whatever/file >> >> varnishlog -r /whatever/file >> >> >> >> What's going wrong here ? >> >> What should I be looking at ? >> >> >> >> Please let me know. >> >> >> >> * << BeReq >> 45926757 >> >> - Begin bereq 45926756 pass >> >> - Timestamp Start: 1618461576.793111 0.000000 0.000000 >> >> - BereqMethod POST >> >> - BereqURL /index.php?&u=85&tr=aa >> >> - BereqProtocol HTTP/1.1 >> >> - BereqHeader X-Forwarded-Proto: https >> >> - BereqHeader X-Forwarded-Port: 443 >> >> - BereqHeader Host: graph.com >> >> - BereqHeader X-Amzn-Trace-Id: Root=1-xxx >> >> - BereqHeader Content-Length: 793 >> >> - BereqHeader sec-ch-ua: "Google Chrome";v="89", "Chromium";v="89", >> ";Not A Brand";v="99" >> >> - BereqHeader accept: application/json, text/javascript, */*; q=0.01 >> >> - BereqHeader sec-ch-ua-mobile: ?0 >> >> - BereqHeader user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) >> AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36 >> >> - BereqHeader content-type: application/x-www-form-urlencoded; >> charset=UTF-8 >> >> - BereqHeader origin: https://mandy..com >> >> - BereqHeader sec-fetch-site: same-site >> >> - BereqHeader sec-fetch-mode: cors >> >> - BereqHeader sec-fetch-dest: empty >> >> - BereqHeader referer: https://mandy.com/ >> >> - BereqHeader accept-encoding: gzip, deflate, br >> >> - BereqHeader accept-language: en-US,en;q=0.9 >> >> - BereqHeader X-Forwarded-For: 103.67.157.20, 10.0.0.170 >> >> - BereqHeader browser: other >> >> - BereqHeader serverIp: 10.0.3.237 >> >> - BereqHeader serverId: abc01 >> >> - BereqHeader X-Varnish: 45926757 >> >> - VCL_call BACKEND_FETCH >> >> - VCL_return fetch >> >> - BackendOpen 32 reload_2021-04-13T130756.default 127.0.0.1 8080 >> 127.0.0.1 56176 >> >> - BackendStart 127.0.0.1 8080 >> >> - Timestamp Bereq: 1618461577.074387 0.281276 0.281276 >> >> - FetchError http first read error: EOF >> >> - BackendClose 32 reload_2021-04-13T130756.default >> >> - Timestamp Beresp: 1618461577.074430 0.281319 0.000043 >> >> - Timestamp Error: 1618461577.074434 0.281323 0.000004 >> >> - BerespProtocol HTTP/1.1 >> >> - BerespStatus 503 >> >> - BerespReason Service Unavailable >> >> - BerespReason Backend fetch failed >> >> - BerespHeader Date: Thu, 15 Apr 2021 04:39:37 GMT >> >> - BerespHeader Server: Varnish >> >> - VCL_call BACKEND_ERROR >> >> - BerespHeader Content-Type: text/html; charset=utf-8 >> >> - BerespHeader Retry-After: 5 >> >> - VCL_return deliver >> >> - Storage malloc Transient >> >> - ObjProtocol HTTP/1.1 >> >> - ObjStatus 503 >> >> - ObjReason Backend fetch failed >> >> - ObjHeader Date: Thu, 15 Apr 2021 04:39:37 GMT >> >> - ObjHeader Server: Varnish >> >> - ObjHeader Content-Type: text/html; charset=utf-8 >> >> - ObjHeader Retry-After: 5 >> >> - Length 1288 >> >> - BereqAcct 948 793 1741 0 0 0 >> >> - End >> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: team.blue-color-rgb-small-mail-signature_78509216-de0a-4033-a847-aa8f4610dc1b.png Type: image/png Size: 3918 bytes Desc: not available URL: From dridi at varni.sh Thu Apr 15 07:21:01 2021 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 15 Apr 2021 07:21:01 +0000 Subject: Understanding 503s In-Reply-To: References: <5074fde7-c89f-4158-a23e-f3ffdfb8c5c4.de476353-2903-4c68-bec2-3738a7d98eb8.872b7609-9509-4d43-9e4c-d8e73d5cd0d1@emailsignatures365.codetwo.com> Message-ID: Hello, For global timeouts: https://varnish-cache.org/docs/trunk/reference/varnishd.html#run-time-parameters They contain "timeout" in the name. For VCL-defined timeouts: https://varnish-cache.org/docs/trunk/reference/vcl-backend.html#timeout-attributes https://varnish-cache.org/docs/trunk/reference/vcl-probe.html#attribute-timeout https://varnish-cache.org/docs/trunk/reference/vcl-var.html If the problem was a timeout, you would see this in the log: FetchError first byte timeout If you are using apache httpd there?s a good chance that incoming connections are closed after 5s of inactivity by default. But when varnish pools backend connections, the default backend_idle_timeout is 60s so you may end up reusing a connection that was closed by your httpd server. You should increase your keep-alive timeout to meet varnish?s expectations. https://httpd.apache.org/docs/2.4/mod/core.html#keepalivetimeout Please note that varnish has a similar timeout_idle that defaults to 5s too. So stacking varnish servers can lead to the same problem if you rely on the defaults. Dridi On Thu, Apr 15, 2021 at 6:27 AM Maninder Singh wrote: > > Also, backend is apache 2.4 > and running php-fpm. > > On Thu, 15 Apr 2021 at 11:52, Maninder Singh wrote: >> >> I have that defined as 2 minutes. >> backend default { >> .host = "127.0.0.1"; >> .port = "8080"; >> .first_byte_timeout = 120s; >> } >> >> That's why this error is puzzling. >> >> Any other timeouts ( connect ? ) etc to look at ? >> >> Also, in the above dump, how much time did it take ? >> >> To me it looks like it was closed within a second ? >> >> >> - BackendOpen 32 reload_2021-04-13T130756.default 127.0.0.1 8080 127.0.0.1 56176 >> - BackendStart 127.0.0.1 8080 >> - Timestamp Bereq: 1618461577.074387 0.281276 0.281276 >> - FetchError http first read error: EOF >> - BackendClose 32 reload_2021-04-13T130756.default >> - Timestamp Beresp: 1618461577.074430 0.281319 0.000043 >> - Timestamp Error: 1618461577.074434 0.281323 0.000004 >> >> On Thu, 15 Apr 2021 at 11:42, Frands Bjerring Hansen wrote: >>> >>> Look at the fetch error: http first read error: EOF >>> >>> >>> >>> Perhaps the backend closed the connection before sending any data or the first_byte_timeout defined for the backend has been reached. The default is 60 seconds. >>> >>> >>> >>> >>> >>> >>> >>> >>> / Frands Bjerring Hansen >>> Head of Technology, Linux >>> Office: +45 70 40 00 00 >>> frands.hansen at team.blue >>> >>> Operations - Linux >>> team.blue Denmark A/S >>> H?jvangen 4 >>> 8660 Skanderborg >>> Denmark >>> CVR: 29412006 >>> >>> >>> >>> >>> On 15/04/2021, 08.08, "varnish-misc" wrote: >>> >>> Hi, >>> >>> I need some help understanding why the below 503 is happening. >>> >>> >>> >>> I am logging 503s to a separate file and then querying as below. >>> >>> >>> >>> varnishlog -q "BerespStatus eq 503" -w /whatever/file >>> >>> varnishlog -r /whatever/file >>> >>> >>> >>> What's going wrong here ? >>> >>> What should I be looking at ? >>> >>> >>> >>> Please let me know. >>> >>> >>> >>> * << BeReq >> 45926757 >>> >>> - Begin bereq 45926756 pass >>> >>> - Timestamp Start: 1618461576.793111 0.000000 0.000000 >>> >>> - BereqMethod POST >>> >>> - BereqURL /index.php?&u=85&tr=aa >>> >>> - BereqProtocol HTTP/1.1 >>> >>> - BereqHeader X-Forwarded-Proto: https >>> >>> - BereqHeader X-Forwarded-Port: 443 >>> >>> - BereqHeader Host: graph.com >>> >>> - BereqHeader X-Amzn-Trace-Id: Root=1-xxx >>> >>> - BereqHeader Content-Length: 793 >>> >>> - BereqHeader sec-ch-ua: "Google Chrome";v="89", "Chromium";v="89", ";Not A Brand";v="99" >>> >>> - BereqHeader accept: application/json, text/javascript, */*; q=0.01 >>> >>> - BereqHeader sec-ch-ua-mobile: ?0 >>> >>> - BereqHeader user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36 >>> >>> - BereqHeader content-type: application/x-www-form-urlencoded; charset=UTF-8 >>> >>> - BereqHeader origin: https://mandy..com >>> >>> - BereqHeader sec-fetch-site: same-site >>> >>> - BereqHeader sec-fetch-mode: cors >>> >>> - BereqHeader sec-fetch-dest: empty >>> >>> - BereqHeader referer: https://mandy.com/ >>> >>> - BereqHeader accept-encoding: gzip, deflate, br >>> >>> - BereqHeader accept-language: en-US,en;q=0.9 >>> >>> - BereqHeader X-Forwarded-For: 103.67.157.20, 10.0.0.170 >>> >>> - BereqHeader browser: other >>> >>> - BereqHeader serverIp: 10.0.3.237 >>> >>> - BereqHeader serverId: abc01 >>> >>> - BereqHeader X-Varnish: 45926757 >>> >>> - VCL_call BACKEND_FETCH >>> >>> - VCL_return fetch >>> >>> - BackendOpen 32 reload_2021-04-13T130756.default 127.0.0.1 8080 127.0.0.1 56176 >>> >>> - BackendStart 127.0.0.1 8080 >>> >>> - Timestamp Bereq: 1618461577.074387 0.281276 0.281276 >>> >>> - FetchError http first read error: EOF >>> >>> - BackendClose 32 reload_2021-04-13T130756.default >>> >>> - Timestamp Beresp: 1618461577.074430 0.281319 0.000043 >>> >>> - Timestamp Error: 1618461577.074434 0.281323 0.000004 >>> >>> - BerespProtocol HTTP/1.1 >>> >>> - BerespStatus 503 >>> >>> - BerespReason Service Unavailable >>> >>> - BerespReason Backend fetch failed >>> >>> - BerespHeader Date: Thu, 15 Apr 2021 04:39:37 GMT >>> >>> - BerespHeader Server: Varnish >>> >>> - VCL_call BACKEND_ERROR >>> >>> - BerespHeader Content-Type: text/html; charset=utf-8 >>> >>> - BerespHeader Retry-After: 5 >>> >>> - VCL_return deliver >>> >>> - Storage malloc Transient >>> >>> - ObjProtocol HTTP/1.1 >>> >>> - ObjStatus 503 >>> >>> - ObjReason Backend fetch failed >>> >>> - ObjHeader Date: Thu, 15 Apr 2021 04:39:37 GMT >>> >>> - ObjHeader Server: Varnish >>> >>> - ObjHeader Content-Type: text/html; charset=utf-8 >>> >>> - ObjHeader Retry-After: 5 >>> >>> - Length 1288 >>> >>> - BereqAcct 948 793 1741 0 0 0 >>> >>> - End >>> >>> > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From mandys at gmail.com Thu Apr 15 07:27:06 2021 From: mandys at gmail.com (Maninder Singh) Date: Thu, 15 Apr 2021 12:57:06 +0530 Subject: Understanding 503s In-Reply-To: References: <5074fde7-c89f-4158-a23e-f3ffdfb8c5c4.de476353-2903-4c68-bec2-3738a7d98eb8.872b7609-9509-4d43-9e4c-d8e73d5cd0d1@emailsignatures365.codetwo.com> Message-ID: Thank you Dridi. This is very helpful. FYI - My apache keepalive is KeepAliveTimeout 3 You would suggest increasing this to 5-10 ? We had lowered the KeepAliveTimeout as the server is a very busy one and we want to handle many connections. On Thu, 15 Apr 2021 at 12:51, Dridi Boukelmoune wrote: > Hello, > > For global timeouts: > > > https://varnish-cache.org/docs/trunk/reference/varnishd.html#run-time-parameters > > They contain "timeout" in the name. > > For VCL-defined timeouts: > > > https://varnish-cache.org/docs/trunk/reference/vcl-backend.html#timeout-attributes > > https://varnish-cache.org/docs/trunk/reference/vcl-probe.html#attribute-timeout > https://varnish-cache.org/docs/trunk/reference/vcl-var.html > > If the problem was a timeout, you would see this in the log: > > FetchError first byte timeout > > If you are using apache httpd there?s a good chance that incoming > connections are closed after 5s of inactivity by default. But when > varnish pools backend connections, the default backend_idle_timeout is > 60s so you may end up reusing a connection that was closed by your > httpd server. You should increase your keep-alive timeout to meet > varnish?s expectations. > > https://httpd.apache.org/docs/2.4/mod/core.html#keepalivetimeout > > Please note that varnish has a similar timeout_idle that defaults to > 5s too. So stacking varnish servers can lead to the same problem if > you rely on the defaults. > > Dridi > > On Thu, Apr 15, 2021 at 6:27 AM Maninder Singh wrote: > > > > Also, backend is apache 2.4 > > and running php-fpm. > > > > On Thu, 15 Apr 2021 at 11:52, Maninder Singh wrote: > >> > >> I have that defined as 2 minutes. > >> backend default { > >> .host = "127.0.0.1"; > >> .port = "8080"; > >> .first_byte_timeout = 120s; > >> } > >> > >> That's why this error is puzzling. > >> > >> Any other timeouts ( connect ? ) etc to look at ? > >> > >> Also, in the above dump, how much time did it take ? > >> > >> To me it looks like it was closed within a second ? > >> > >> > >> - BackendOpen 32 reload_2021-04-13T130756.default 127.0.0.1 8080 > 127.0.0.1 56176 > >> - BackendStart 127.0.0.1 8080 > >> - Timestamp Bereq: 1618461577.074387 0.281276 0.281276 > >> - FetchError http first read error: EOF > >> - BackendClose 32 reload_2021-04-13T130756.default > >> - Timestamp Beresp: 1618461577.074430 0.281319 0.000043 > >> - Timestamp Error: 1618461577.074434 0.281323 0.000004 > >> > >> On Thu, 15 Apr 2021 at 11:42, Frands Bjerring Hansen > wrote: > >>> > >>> Look at the fetch error: http first read error: EOF > >>> > >>> > >>> > >>> Perhaps the backend closed the connection before sending any data or > the first_byte_timeout defined for the backend has been reached. The > default is 60 seconds. > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> / Frands Bjerring Hansen > >>> Head of Technology, Linux > >>> Office: +45 70 40 00 00 > >>> frands.hansen at team.blue > >>> > >>> Operations - Linux > >>> team.blue Denmark A/S > >>> H?jvangen 4 > >>> 8660 Skanderborg > >>> Denmark > >>> CVR: 29412006 > >>> > >>> > >>> > >>> > >>> On 15/04/2021, 08.08, "varnish-misc" zitcom.dk at varnish-cache.org> wrote: > >>> > >>> Hi, > >>> > >>> I need some help understanding why the below 503 is happening. > >>> > >>> > >>> > >>> I am logging 503s to a separate file and then querying as below. > >>> > >>> > >>> > >>> varnishlog -q "BerespStatus eq 503" -w /whatever/file > >>> > >>> varnishlog -r /whatever/file > >>> > >>> > >>> > >>> What's going wrong here ? > >>> > >>> What should I be looking at ? > >>> > >>> > >>> > >>> Please let me know. > >>> > >>> > >>> > >>> * << BeReq >> 45926757 > >>> > >>> - Begin bereq 45926756 pass > >>> > >>> - Timestamp Start: 1618461576.793111 0.000000 0.000000 > >>> > >>> - BereqMethod POST > >>> > >>> - BereqURL /index.php?&u=85&tr=aa > >>> > >>> - BereqProtocol HTTP/1.1 > >>> > >>> - BereqHeader X-Forwarded-Proto: https > >>> > >>> - BereqHeader X-Forwarded-Port: 443 > >>> > >>> - BereqHeader Host: graph.com > >>> > >>> - BereqHeader X-Amzn-Trace-Id: Root=1-xxx > >>> > >>> - BereqHeader Content-Length: 793 > >>> > >>> - BereqHeader sec-ch-ua: "Google Chrome";v="89", > "Chromium";v="89", ";Not A Brand";v="99" > >>> > >>> - BereqHeader accept: application/json, text/javascript, */*; > q=0.01 > >>> > >>> - BereqHeader sec-ch-ua-mobile: ?0 > >>> > >>> - BereqHeader user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; > x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 > Safari/537.36 > >>> > >>> - BereqHeader content-type: application/x-www-form-urlencoded; > charset=UTF-8 > >>> > >>> - BereqHeader origin: https://mandy..com > >>> > >>> - BereqHeader sec-fetch-site: same-site > >>> > >>> - BereqHeader sec-fetch-mode: cors > >>> > >>> - BereqHeader sec-fetch-dest: empty > >>> > >>> - BereqHeader referer: https://mandy.com/ > >>> > >>> - BereqHeader accept-encoding: gzip, deflate, br > >>> > >>> - BereqHeader accept-language: en-US,en;q=0.9 > >>> > >>> - BereqHeader X-Forwarded-For: 103.67.157.20, 10.0.0.170 > >>> > >>> - BereqHeader browser: other > >>> > >>> - BereqHeader serverIp: 10.0.3.237 > >>> > >>> - BereqHeader serverId: abc01 > >>> > >>> - BereqHeader X-Varnish: 45926757 > >>> > >>> - VCL_call BACKEND_FETCH > >>> > >>> - VCL_return fetch > >>> > >>> - BackendOpen 32 reload_2021-04-13T130756.default 127.0.0.1 8080 > 127.0.0.1 56176 > >>> > >>> - BackendStart 127.0.0.1 8080 > >>> > >>> - Timestamp Bereq: 1618461577.074387 0.281276 0.281276 > >>> > >>> - FetchError http first read error: EOF > >>> > >>> - BackendClose 32 reload_2021-04-13T130756.default > >>> > >>> - Timestamp Beresp: 1618461577.074430 0.281319 0.000043 > >>> > >>> - Timestamp Error: 1618461577.074434 0.281323 0.000004 > >>> > >>> - BerespProtocol HTTP/1.1 > >>> > >>> - BerespStatus 503 > >>> > >>> - BerespReason Service Unavailable > >>> > >>> - BerespReason Backend fetch failed > >>> > >>> - BerespHeader Date: Thu, 15 Apr 2021 04:39:37 GMT > >>> > >>> - BerespHeader Server: Varnish > >>> > >>> - VCL_call BACKEND_ERROR > >>> > >>> - BerespHeader Content-Type: text/html; charset=utf-8 > >>> > >>> - BerespHeader Retry-After: 5 > >>> > >>> - VCL_return deliver > >>> > >>> - Storage malloc Transient > >>> > >>> - ObjProtocol HTTP/1.1 > >>> > >>> - ObjStatus 503 > >>> > >>> - ObjReason Backend fetch failed > >>> > >>> - ObjHeader Date: Thu, 15 Apr 2021 04:39:37 GMT > >>> > >>> - ObjHeader Server: Varnish > >>> > >>> - ObjHeader Content-Type: text/html; charset=utf-8 > >>> > >>> - ObjHeader Retry-After: 5 > >>> > >>> - Length 1288 > >>> > >>> - BereqAcct 948 793 1741 0 0 0 > >>> > >>> - End > >>> > >>> > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Thu Apr 15 08:14:30 2021 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 15 Apr 2021 08:14:30 +0000 Subject: Understanding 503s In-Reply-To: References: <5074fde7-c89f-4158-a23e-f3ffdfb8c5c4.de476353-2903-4c68-bec2-3738a7d98eb8.872b7609-9509-4d43-9e4c-d8e73d5cd0d1@emailsignatures365.codetwo.com> Message-ID: On Thu, Apr 15, 2021 at 7:27 AM Maninder Singh wrote: > > Thank you Dridi. > This is very helpful. > > FYI - My apache keepalive is > KeepAliveTimeout 3 > > You would suggest increasing this to 5-10 ? If varnish is httpd?s only client then increase it to 70s. Varnish will close unused connections after 60s by default, and if it?s really really busy that gives a 10s window for the actual shutdown to happen. If there are other direct clients in front of your httpd server, then decrease backend_idle_timeout in varnishd to 2s, but then you will force varnish to establish connections more often. This is already the case of course, but at least that will reduce the risk of reusing a closed connection and failing backend fetches for this reason. > We had lowered the KeepAliveTimeout as the server is a very busy one and we want to handle many connections. I understand, and there?s a good reason to have a low default when you can?t trust the clients. It boils down to whether your httpd server is openly accessible to more than just varnish, including potentially malicious clients. Dridi From mandys at gmail.com Thu Apr 15 08:33:56 2021 From: mandys at gmail.com (Maninder Singh) Date: Thu, 15 Apr 2021 14:03:56 +0530 Subject: Understanding 503s In-Reply-To: References: <5074fde7-c89f-4158-a23e-f3ffdfb8c5c4.de476353-2903-4c68-bec2-3738a7d98eb8.872b7609-9509-4d43-9e4c-d8e73d5cd0d1@emailsignatures365.codetwo.com> Message-ID: Apache runs on port 8080 and is not open to the outside world. All requests are routed through varnish but then not all requests are cached. I guess in that case, varnish becomes the only client for apache. So, I should increase the KeepAliveTimeout. On Thu, 15 Apr 2021 at 13:45, Dridi Boukelmoune wrote: > On Thu, Apr 15, 2021 at 7:27 AM Maninder Singh wrote: > > > > Thank you Dridi. > > This is very helpful. > > > > FYI - My apache keepalive is > > KeepAliveTimeout 3 > > > > You would suggest increasing this to 5-10 ? > > If varnish is httpd?s only client then increase it to 70s. Varnish > will close unused connections after 60s by default, and if it?s really > really busy that gives a 10s window for the actual shutdown to happen. > > If there are other direct clients in front of your httpd server, then > decrease backend_idle_timeout in varnishd to 2s, but then you will > force varnish to establish connections more often. This is already the > case of course, but at least that will reduce the risk of reusing a > closed connection and failing backend fetches for this reason. > > > We had lowered the KeepAliveTimeout as the server is a very busy one and > we want to handle many connections. > > I understand, and there?s a good reason to have a low default when you > can?t trust the clients. It boils down to whether your httpd server is > openly accessible to more than just varnish, including potentially > malicious clients. > > Dridi > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mandys at gmail.com Fri Apr 23 07:02:38 2021 From: mandys at gmail.com (Maninder Singh) Date: Fri, 23 Apr 2021 12:32:38 +0530 Subject: Understanding 503s In-Reply-To: References: <5074fde7-c89f-4158-a23e-f3ffdfb8c5c4.de476353-2903-4c68-bec2-3738a7d98eb8.872b7609-9509-4d43-9e4c-d8e73d5cd0d1@emailsignatures365.codetwo.com> Message-ID: I finally figured out why this was happening. Hope this helps someone. We were running php-fpm and had the following configuration. pm = dynamic pm.max_children = 166 pm.start_servers = 16 pm.min_spare_servers = 8 pm.max_spare_servers = 16 This was working fine with the usual load. But, we found that whenever there was a spike it led to an increase in 503s. This was due to start_servers set to 16. php-fpm takes a sec to spawn more processes and during that time we see 503s. For a high traffic site ( like ours ), we had to set this to pm = static pm.max_children = 125 The above values are kept keeping in mind our RAM. These would be different for others. Now, we don't see any 503s as the server is prepared to handle more connections. Thanks, On Thu, 15 Apr 2021 at 14:03, Maninder Singh wrote: > Apache runs on port 8080 and is not open to the outside world. > All requests are routed through varnish but then not all requests are > cached. > > I guess in that case, varnish becomes the only client for apache. > So, I should increase the KeepAliveTimeout. > > > > On Thu, 15 Apr 2021 at 13:45, Dridi Boukelmoune wrote: > >> On Thu, Apr 15, 2021 at 7:27 AM Maninder Singh wrote: >> > >> > Thank you Dridi. >> > This is very helpful. >> > >> > FYI - My apache keepalive is >> > KeepAliveTimeout 3 >> > >> > You would suggest increasing this to 5-10 ? >> >> If varnish is httpd?s only client then increase it to 70s. Varnish >> will close unused connections after 60s by default, and if it?s really >> really busy that gives a 10s window for the actual shutdown to happen. >> >> If there are other direct clients in front of your httpd server, then >> decrease backend_idle_timeout in varnishd to 2s, but then you will >> force varnish to establish connections more often. This is already the >> case of course, but at least that will reduce the risk of reusing a >> closed connection and failing backend fetches for this reason. >> >> > We had lowered the KeepAliveTimeout as the server is a very busy one >> and we want to handle many connections. >> >> I understand, and there?s a good reason to have a low default when you >> can?t trust the clients. It boils down to whether your httpd server is >> openly accessible to more than just varnish, including potentially >> malicious clients. >> >> Dridi >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Fri Apr 23 16:59:20 2021 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Fri, 23 Apr 2021 09:59:20 -0700 Subject: Understanding 503s In-Reply-To: References: <5074fde7-c89f-4158-a23e-f3ffdfb8c5c4.de476353-2903-4c68-bec2-3738a7d98eb8.872b7609-9509-4d43-9e4c-d8e73d5cd0d1@emailsignatures365.codetwo.com> Message-ID: Thanks for taking the time to debrief, I'm sure that will be useful in the future! -- Guillaume Quintard On Fri, Apr 23, 2021, 00:04 Maninder Singh wrote: > I finally figured out why this was happening. > > Hope this helps someone. > > We were running php-fpm and had the following configuration. > > pm = dynamic > pm.max_children = 166 > pm.start_servers = 16 > pm.min_spare_servers = 8 > pm.max_spare_servers = 16 > > This was working fine with the usual load. > But, we found that whenever there was a spike it led to an increase in > 503s. > > This was due to start_servers set to 16. > > php-fpm takes a sec to spawn more processes and during that time we see > 503s. > > For a high traffic site ( like ours ), we had to set this to > pm = static > pm.max_children = 125 > > The above values are kept keeping in mind our RAM. > These would be different for others. > > Now, we don't see any 503s as the server is prepared to handle more > connections. > > Thanks, > > On Thu, 15 Apr 2021 at 14:03, Maninder Singh wrote: > >> Apache runs on port 8080 and is not open to the outside world. >> All requests are routed through varnish but then not all requests are >> cached. >> >> I guess in that case, varnish becomes the only client for apache. >> So, I should increase the KeepAliveTimeout. >> >> >> >> On Thu, 15 Apr 2021 at 13:45, Dridi Boukelmoune wrote: >> >>> On Thu, Apr 15, 2021 at 7:27 AM Maninder Singh wrote: >>> > >>> > Thank you Dridi. >>> > This is very helpful. >>> > >>> > FYI - My apache keepalive is >>> > KeepAliveTimeout 3 >>> > >>> > You would suggest increasing this to 5-10 ? >>> >>> If varnish is httpd?s only client then increase it to 70s. Varnish >>> will close unused connections after 60s by default, and if it?s really >>> really busy that gives a 10s window for the actual shutdown to happen. >>> >>> If there are other direct clients in front of your httpd server, then >>> decrease backend_idle_timeout in varnishd to 2s, but then you will >>> force varnish to establish connections more often. This is already the >>> case of course, but at least that will reduce the risk of reusing a >>> closed connection and failing backend fetches for this reason. >>> >>> > We had lowered the KeepAliveTimeout as the server is a very busy one >>> and we want to handle many connections. >>> >>> I understand, and there?s a good reason to have a low default when you >>> can?t trust the clients. It boils down to whether your httpd server is >>> openly accessible to more than just varnish, including potentially >>> malicious clients. >>> >>> Dridi >>> >> _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marco.dickert at evolver.de Thu Apr 29 15:21:04 2021 From: marco.dickert at evolver.de (Marco Dickert - evolver group) Date: Thu, 29 Apr 2021 17:21:04 +0200 Subject: varnish eats all RAM Message-ID: <20210429152103.GD3147@marco.evolver.de> Hi all, we have a weird problem with one of our varnish instances running on debian buster. The problem is, that varnish consumes all RAM over time and does not free it again. This leads to service unavailability, until varnish gets either killed by the oom-killer or restarted via some monitoring service (which we set up to avoid errors). The system runs on an Intel(R) Xeon(R) CPU E5335 and has 16GB RAM. We initially experienced this problem with varnish 6.1 (default debian version), but we upgraded to varnish 6.6 using the official varnish repos, and this behaviour persists. I attached a graph of the RAM usage, in which you can see the continuous allocation of RAM until the process gets restarted. Has anyone an idea how we could examine the RAM usage further? Used varnish modules: - directors - std - digest - cookie Used varnish options: -p listen_depth=4096 \ -p thread_pool_min=200 \ -p thread_pool_max=300 \ -p workspace_backend=64k \ -p workspace_client=128k \ -p nuke_limit=1000 \ -p thread_pools=2 \ -s malloc,3G Greetings, Marco Dickert -- Mit freundlichen Gr??en Marco Dickert Administration und Technik evolver services GmbH Fon +49 / (0)3 71 / 4 00 03 78 24 Fax +49 / (0)3 71 / 4 00 03 79 E-Mail marco.dickert at evolver.de Web https://www.evolver.de Sitz der Gesellschaft: Chemnitz Handelsregister: Amtsgericht Chemnitz, HRB 22649 Gesch?ftsf?hrer: Torsten Gramann und Mathias M?ckel -------------- next part -------------- A non-text attachment was scrubbed... Name: lb-ram-sixhours.png Type: image/png Size: 30340 bytes Desc: not available URL: From geoff at uplex.de Thu Apr 29 16:00:39 2021 From: geoff at uplex.de (Geoff Simmons) Date: Thu, 29 Apr 2021 18:00:39 +0200 Subject: varnish eats all RAM In-Reply-To: <20210429152103.GD3147@marco.evolver.de> References: <20210429152103.GD3147@marco.evolver.de> Message-ID: <0985b590-037b-eed5-0500-1b216bed6451@uplex.de> On 4/29/21 17:21, Marco Dickert - evolver group wrote: > > we have a weird problem with one of our varnish instances running on debian > buster. The problem is, that varnish consumes all RAM over time and does not > free it again. Just a hunch, but this is reminiscent of problems that I've seen caused by transparent huge pages, which are enabled by default in recent Linux versions. Why THP can be a problem with Varnish & jemalloc takes some explaining, but maybe first check if that could be your problem. On Linux you can look for the value of AnonHugePages in /proc/meminfo, value in KB. You can also look for rss_huge in /sys/fs/cgroup/memory/memory.stat, where the value is the number of huge pages. So you'd have to multiply the latter value by the size of a hugepage, which appears as Hugepagesize in /proc/meminfo (usually it's 2MB on Linux). If those values are very large and seem to account for most of your high RAM consumption, considerably higher than you would have expected for Varnish, then THP may be your problem. If you have jemalloc since version 4.5, you can turn off THP specifically for Varnish with the configuration thp:never for jemalloc, either in /etc/jemalloc.conf, or by passing it in the environment variable MALLOC_CONF to the varnishd invocation -- set MALLOC_CONF=thp:never and make sure Varnish inherits that in its environment. thp:never is not supported by jemalloc 3.6.0, which is still very widely deployed on many systems (unfortunately, since jemalloc is up to version 5.2.1 now). You can also turn off THP system-wide (for all processes): $ echo never > /sys/kernel/mm/transparent_hugepage/enabled That might be your only option if the jemalloc version is too old. Either way, then try it and see if your RAM issues improve. HTH, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 840 bytes Desc: OpenPGP digital signature URL: