From sebastien at sdv.fr Mon Nov 2 13:13:29 2020 From: sebastien at sdv.fr (=?UTF-8?B?U8OpYmFzdGllbg==?= EISSLER) Date: Mon, 2 Nov 2020 14:13:29 +0100 Subject: Traffic drop issue In-Reply-To: References: <20201026161140.4a4c80ef@symposium.sdv.fr> <20201029160540.64febdc5@symposium.sdv.fr> Message-ID: <20201102141329.652ac7b9@symposium.sdv.fr> Hello, > On Thu, Oct 29, 2020 at 3:06 PM S?bastien EISSLER wrote: > > > > Hello, > > > > Yes, we have a load-balancer in front, we already investigate that way and don't dectect any error or configuration limitation. > > The fact that threads_limited counter strongly increase and that restart of Varnish restore the traffic suggest an issue on varnish side. > > Did you try to increase thread_pool_max? > Yes, we increased thread_pool_max and thread_pool_min after that issue. For the moment all work fine. We think about adjusting other params concerning threads like thread_pool_reserve. Did you have advise about that? Regards -- S?bastien From dridi at varni.sh Mon Nov 2 13:32:52 2020 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 2 Nov 2020 13:32:52 +0000 Subject: Traffic drop issue In-Reply-To: <20201102141329.652ac7b9@symposium.sdv.fr> References: <20201026161140.4a4c80ef@symposium.sdv.fr> <20201029160540.64febdc5@symposium.sdv.fr> <20201102141329.652ac7b9@symposium.sdv.fr> Message-ID: > > Did you try to increase thread_pool_max? > > > > Yes, we increased thread_pool_max and thread_pool_min after that issue. > For the moment all work fine. FWIW, it's in the documentation of the threads_limited counter, which you (and Guillaume) didn't seem to notice (or remember) before I brought this up. Correct me if I'm wrong of course, but more importantly please let me know how we could improve the documentation if that was not enough. > We think about adjusting other params concerning threads like thread_pool_reserve. > Did you have advise about that? If you already increased thread_pool_min, you mechanically increased the size of the reserve (5% of thread_pool_min by default) so I'd suggest you keep monitoring threads_limited and see whether you need more workers for your workload. Dridi From guillaume at varnish-software.com Mon Nov 2 15:59:32 2020 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 2 Nov 2020 07:59:32 -0800 Subject: Traffic drop issue In-Reply-To: References: <20201026161140.4a4c80ef@symposium.sdv.fr> <20201029160540.64febdc5@symposium.sdv.fr> <20201102141329.652ac7b9@symposium.sdv.fr> Message-ID: What intrigues me is why the number of requests decreased. Hitting thread_pool_max should make the number of request plateau, not go down. If there's a loadbalancer that realizes requests are getting dropped, and so takes traffic away, it makes sense, just wanted to make sure. If that's the case, the loadbalancer will probably give you some info there. On top of threas_limited, sess_dropped and sess_queued are probably good counters to check. -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Mon Nov 2 17:31:00 2020 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 2 Nov 2020 17:31:00 +0000 Subject: Traffic drop issue In-Reply-To: References: <20201026161140.4a4c80ef@symposium.sdv.fr> <20201029160540.64febdc5@symposium.sdv.fr> <20201102141329.652ac7b9@symposium.sdv.fr> Message-ID: On Mon, Nov 2, 2020 at 3:59 PM Guillaume Quintard wrote: > > What intrigues me is why the number of requests decreased. Hitting thread_pool_max should make the number of request plateau, not go down. > > If there's a loadbalancer that realizes requests are getting dropped, and so takes traffic away, it makes sense, just wanted to make sure. > If that's the case, the loadbalancer will probably give you some info there. Likewise if too many backend fetches are triggered (low hit ratio?) and pile up, they will get higher priority than client tasks. > On top of threas_limited, sess_dropped and sess_queued are probably good counters to check. I think this doesn't apply to h2 requests. From guillaume at varnish-software.com Thu Nov 12 16:29:45 2020 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 12 Nov 2020 08:29:45 -0800 Subject: Varnish Cache 6.0.7 Message-ID: Hi all, Almost a week ago, we released Varnish Cache 6.0.7, this is the latest entry in the LTS series. As expected, the changelog ( https://github.com/varnishcache/varnish-cache/blob/6.0/doc/changes.rst#varnish-cache-607-2020-11-06) mainly contains bug fixes, making the release even more robust, but it also contains a couple of treats like vmod_shard supporting weighted backends and the new "pid" command for varnishadm. You will be able to find packages for this release on the usual packagecloud page: https://packagecloud.io/varnishcache/varnish60lts However, you will see something unusual here: we have added support for RHEL/CentOS 8, Debian Buster and Ubuntu Focal. We usually don't add support for new distribution once a series is started, but we are making an exception for LTS as we don't want to encourage people running it on EOL'd distributions. We have also pushed the Official Docker images here: https://hub.docker.com/_/varnish Note that the underlying Debian image switch from Stretch to Buster as the Stretch image won't be maintained anymore. Kind regards, -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrhosseini at hotmail.com Wed Nov 18 12:19:02 2020 From: hrhosseini at hotmail.com (Hamidreza Hosseini) Date: Wed, 18 Nov 2020 12:19:02 +0000 Subject: Varnish restart because of ram filled issue Message-ID: Hi, I have some varnish ram in production that I'm using it for 1 Year I had this problem since first time that I install it but Now because we have so many request , the varnish ram will filled and crash and it destroys my backend cluster The whole ram is 54GB but I adjust 43 GB ram to it in systemd: ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :6081 -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,43g -i varnish what should I do for this problem?? why varnish doesnt prevent ram filled? Is there any config in varnish that I tell it you can not eceed from this amount and if you reach that amount you MUST free your cache from the first input?(This feature is really important because I think because I have 5 core cpu onj server, systemd cant understand the limitation that I'm adjusting, it should balance in varnish...) OS: ubuntu Varnish version: varnishd (varnish-6.0.5 revision 3065ccaacc4bb537fb976a524bd808db42c5fe40) Copyright (c) 2006 Verdens Gang AS Copyright (c) 2006-2019 Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Wed Nov 18 19:35:11 2020 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 18 Nov 2020 11:35:11 -0800 Subject: Varnish restart because of ram filled issue In-Reply-To: References: Message-ID: Hi, The cache isn't the only one taking RAM, I would recommend having a look at this article to understand what would cost you here: https://info.varnish-software.com/blog/understanding-varnish-cache-memory-usage Main culprit is possibly going to be Transient storage ( https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage) but there could be other reasons Kind regards, -- Guillaume Quintard On Wed, Nov 18, 2020 at 4:20 AM Hamidreza Hosseini wrote: > Hi, > I have some varnish ram in production that I'm using it for 1 Year > I had this problem since first time that I install it but Now because we > have so many request , the varnish ram will filled and crash and it > destroys my backend cluster > The whole ram is 54GB but I adjust 43 GB ram to it in systemd: > > ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :6081 -T > localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s > malloc,43g -i varnish > > what should I do for this problem?? > why varnish doesnt prevent ram filled? > Is there any config in varnish that I tell it you can not eceed from this > amount and if you reach that amount you MUST free your cache from the first > input?(This feature is really important because I think because I have 5 > core cpu onj server, systemd cant understand the limitation that I'm > adjusting, it should balance in varnish...) > > OS: ubuntu > Varnish version: > varnishd (varnish-6.0.5 revision 3065ccaacc4bb537fb976a524bd808db42c5fe40) > Copyright (c) 2006 Verdens Gang AS > Copyright (c) 2006-2019 Varnish Software AS > > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: