From joh.hendriks at gmail.com Tue Sep 18 07:35:19 2018 From: joh.hendriks at gmail.com (Johan Hendriks) Date: Tue, 18 Sep 2018 09:35:19 +0200 Subject: varnishhist output not correctly in varnish 6.0.1 and varnish 6.1 Message-ID: I noticed that in varnish 6.0.1 and 6.1.0 the varnishhist ar not displayed like they should or at least not as in 6.0.0 I attached two screenshots one from 6.0.0 and one from 6.0.1 regards Johan Hendriks -------------- next part -------------- A non-text attachment was scrubbed... Name: varnishhist-6.0.1.png Type: image/png Size: 20533 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: varnishhist-6.0.0.png Type: image/png Size: 18756 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pEpkey.asc Type: application/pgp-keys Size: 1769 bytes Desc: not available URL: From phk at phk.freebsd.dk Tue Sep 18 07:46:22 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 18 Sep 2018 07:46:22 +0000 Subject: varnishhist output not correctly in varnish 6.0.1 and varnish 6.1 In-Reply-To: References: Message-ID: <95700.1537256782@critter.freebsd.dk> -------- In message , Johan Hendriks writes: >I noticed that in varnish 6.0.1 and 6.1.0 the varnishhist ar not >displayed like they should or at least not as in 6.0.0 > >I attached two screenshots one from 6.0.0 and one from 6.0.1 Please open a ticket on this. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From adrien.bigot at smile.fr Tue Sep 18 09:35:11 2018 From: adrien.bigot at smile.fr (BIGOT Adrien) Date: Tue, 18 Sep 2018 11:35:11 +0200 Subject: CLI start with multiple VCL on Varnish 6 Failed. Message-ID: <8f57e251-a4aa-b6a4-71d5-deb9b5d7407c@smile.fr> Hello, Based on this blog article : https://info.varnish-software.com/blog/one-vcl-per-domain?? I've been using multiple VCL with Varnish 5.2 without any problem. However, I don't manage to get it working with Varnish 6. I tried with the latest stable version on packagecloud.io and even with the latest weekly-build. Tested on Debian Strecth, Ubuntu 16.04 and Centos 7. [root at ip-172-31-137-19 varnish]# varnishd -I start.cli -F -a :80 -f '' Debug: Version: varnish-6.0.1 revision 8d54bec5330c29304979ebf2c425ae14ab80493c Debug: Platform: Linux,3.10.0-862.3.2.el7.x86_64,x86_64,-junix,-sdefault,-sdefault,-hcritbit BEGIN of -I file processing 200 326 ----------------------------- Varnish Cache CLI 1.0 ----------------------------- Linux,3.10.0-862.3.2.el7.x86_64,x86_64,-junix,-sdefault,-sdefault,-hcritbit varnish-6.0.1 revision 8d54bec5330c29304979ebf2c425ae14ab80493c Type 'help' for command list. Type 'quit' to close CLI session. Type 'start' to launch worker process. > vcl.load vcl-api-orig /etc/varnish/api.vcl 200 14 VCL compiled. > vcl.load vcl-catz-orig /etc/varnish/catz.vcl 200 14 VCL compiled. > vcl.label label-api vcl-api-orig 200 0 > vcl.label label-catz vcl-catz-orig 200 0 > vcl.load vcl-root-orig /etc/varnish/root.vcl 200 14 VCL compiled. > vcl.use vcl-root-orig 200 30 VCL 'vcl-root-orig' now active END of -I file processing CLI result = 400 In the logs : On Centos sept. 18 08:35:37 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx su[11820]: pam_unix(su:session): session opened for user root by ec2-user(uid=0) sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: Version: varnish-6.0.1 revision 8d54bec5330c29304979ebf2c425ae14ab80493c sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: Platform: Linux,3.10.0-862.3.2.el7.x86_64,x86_64,-junix,-sdefault,-sdefault,-hcritbit sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Rd vcl.load vcl-api-orig /etc/varnish/api.vcl sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Wr 200 VCL compiled. sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Rd vcl.load vcl-catz-orig /etc/varnish/catz.vcl sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Wr 200 VCL compiled. sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Rd vcl.label label-api vcl-api-orig sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Wr 200 sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Rd vcl.label label-catz vcl-catz-orig sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Wr 200 sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Rd vcl.load vcl-root-orig /etc/varnish/root.vcl sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Wr 200 VCL compiled. sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Rd vcl.use vcl-root-orig sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxxm varnishd[11846]: CLI -I file Wr 200 VCL 'vcl-root-orig' now active sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: Child (11870) Started sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: Child (11870) not responding to CLI, killed it. sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: Child (11870) Pushing vcls failed: CLI communication error sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: Stopping Child sept. 18 08:36:35 ip-172-31-137-19.xxxxxxxxxxxxxxxxxxm varnishd[11846]: Child (11870) died signal=3 sept. 18 08:36:35 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: Child cleanup complete On Debian : Sep 18 10:07:01 ip-172-31-137-84 varnishd[6783]: Child (6809) said Child starts Sep 18 10:07:01 ip-172-31-137-84 varnishd[6783]: Child cleanup complete Sep 18 10:07:02 ip-172-31-137-84 varnishd[6816]: Version: varnish-6.0.1 revision 8d54bec5330c29304979ebf2c425ae14ab80493c Sep 18 10:07:02 ip-172-31-137-84 varnishd[6816]: Platform: Linux,4.9.0-8-amd64,x86_64,-junix,-sdefault,-sdefault,-hcritbit Sep 18 10:07:02 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.load vcl-api-orig /etc/varnish/api.vcl Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 VCL compiled. Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.load vcl-catz-orig /etc/varnish/catz.vcl Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 VCL compiled. Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.label label-api vcl-api-orig Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.label label-catz vcl-catz-orig Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.load vcl-root-orig /etc/varnish/root.vcl Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 VCL compiled. Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.use vcl-root-orig Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 VCL 'vcl-root-orig' now active Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: Child (6842) Started Sep 18 10:08:03 ip-172-31-137-84 varnishd[6816]: Child (6842) not responding to CLI, killed it. Sep 18 10:08:03 ip-172-31-137-84 varnishd[6816]: Child (6842) Pushing vcls failed: ???????????????????????????????????????????????? CLI communication error (hdr) Sep 18 10:08:03 ip-172-31-137-84 varnishd[6816]: Stopping Child Sep 18 10:08:04 ip-172-31-137-84 varnishd[6816]: Child (6842) died signal=3 Did anyone manage to get it working with varnish 6 ? Thank you. Best regards, -- Bigot Adrien From n.premkumar.me at gmail.com Wed Sep 19 11:38:17 2018 From: n.premkumar.me at gmail.com (Prem Kumar) Date: Wed, 19 Sep 2018 17:08:17 +0530 Subject: Doubt on mmap file backend Message-ID: We are trying to map the file and content is written in to specific offset. In meanwhile, if try to access the mmap'd content at specified offset, we are getting NULL value. we have not called msync() yet. the mmap called with MAP_SHARED. Is it possible that mmap content can differ on read after write or is this the problem happening because read is called before the msync().Sorry ,it is very generic question ..Wanted to know if any idea on the behavior. Thanks, Prem -------------- next part -------------- An HTML attachment was scrubbed... URL: From phk at phk.freebsd.dk Wed Sep 19 11:42:50 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Wed, 19 Sep 2018 11:42:50 +0000 Subject: Doubt on mmap file backend In-Reply-To: References: Message-ID: <1830.1537357370@critter.freebsd.dk> -------- In message , Prem Kumar writes: >Is it possible that mmap content can differ on read after write or is this >the problem happening because read is called before the msync().Sorry ,it >is very generic question ..Wanted to know if any idea on the behavior. If you mean "read by another process", then yes. Only kernels with integrated VM/buffers provide consistency without msync. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From t.winkelmann at ffh.de Wed Sep 19 12:30:09 2018 From: t.winkelmann at ffh.de (Winkelmann, Thomas (RADIO TELE FFH - Online)) Date: Wed, 19 Sep 2018 14:30:09 +0200 Subject: Varnish VMods for Version 6.1? Message-ID: <8A76B2DB46634944BEEBB218DAE22BC265522A6968@exchange-srv2.office.radioteleffh.de> Hello, just wanted to test Varnish 6.1 regarding H2 support, but I'm unable to compile the vmods on top of it. Is there a compatible version varnish-modules already available for 6.1? Or: Are there big differences between 6.0.1 and 6.1 regarding H/2? Thanks, Thomas ________________________________ RADIO / TELE FFH GmbH & Co. Betriebs-KG FFH-Platz 1, 61111 Bad Vilbel HRA - Nr. 26092 Frankfurt/Main USt.IdNr. DE 112152620 Gesch?ftsf?hrer / Programmdirektor: Hans-Dieter Hillmoth -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Wed Sep 19 16:58:58 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 19 Sep 2018 18:58:58 +0200 Subject: Varnish VMods for Version 6.1? In-Reply-To: <8A76B2DB46634944BEEBB218DAE22BC265522A6968@exchange-srv2.office.radioteleffh.de> References: <8A76B2DB46634944BEEBB218DAE22BC265522A6968@exchange-srv2.office.radioteleffh.de> Message-ID: On Wed, Sep 19, 2018 at 2:32 PM Winkelmann, Thomas (RADIO TELE FFH - Online) wrote: > > Hello, > > just wanted to test Varnish 6.1 regarding H2 support, but I?m unable to compile the vmods on top of it. > > Is there a compatible version varnish-modules already available for 6.1? We haven't added support for 6.1 yet, but we have a pull request to review. > Or: Are there big differences between 6.0.1 and 6.1 regarding H/2? The 6.0.1 release should contain all the latest h2 bug fixes. Please note that the 6.0 branch is a long term support branch and 6.1 is not so ignore varnish-modules for now, please be aware of this before choosing one. Also please note that there are a few remaining known h2 bugs, but stability has greatly improved since the 6.0.0 release. Cheers From t.winkelmann at ffh.de Thu Sep 20 08:22:26 2018 From: t.winkelmann at ffh.de (Winkelmann, Thomas (RADIO TELE FFH - Online)) Date: Thu, 20 Sep 2018 10:22:26 +0200 Subject: AW: Varnish VMods for Version 6.1? In-Reply-To: References: <8A76B2DB46634944BEEBB218DAE22BC265522A6968@exchange-srv2.office.radioteleffh.de> Message-ID: <8A76B2DB46634944BEEBB218DAE22BC265522A6971@exchange-srv2.office.radioteleffh.de> Thanks for the feedback. We tried 6.0.1. So far H/2 seems to be working fine under normal conditions. But as soon as the request are increasing ( > 800 reg/s) the traffic will stop. Also HTTP 1.1. traffic. Sound like these two problems: https://github.com/varnishcache/varnish-cache/issues/2431 https://github.com/varnishcache/varnish-cache/issues/2418 Let us know if we can help to solve these problems. -----Urspr?ngliche Nachricht----- Von: Dridi Boukelmoune [mailto:dridi at varni.sh] Gesendet: Mittwoch, 19. September 2018 18:59 An: Winkelmann, Thomas (RADIO TELE FFH - Online) Cc: varnish-misc Betreff: Re: Varnish VMods for Version 6.1? On Wed, Sep 19, 2018 at 2:32 PM Winkelmann, Thomas (RADIO TELE FFH - Online) wrote: > > Hello, > > just wanted to test Varnish 6.1 regarding H2 support, but I?m unable to compile the vmods on top of it. > > Is there a compatible version varnish-modules already available for 6.1? We haven't added support for 6.1 yet, but we have a pull request to review. > Or: Are there big differences between 6.0.1 and 6.1 regarding H/2? The 6.0.1 release should contain all the latest h2 bug fixes. Please note that the 6.0 branch is a long term support branch and 6.1 is not so ignore varnish-modules for now, please be aware of this before choosing one. Also please note that there are a few remaining known h2 bugs, but stability has greatly improved since the 6.0.0 release. Cheers RADIO / TELE FFH GmbH & Co. Betriebs-KG FFH-Platz 1, 61111 Bad Vilbel HRA - Nr. 26092 Frankfurt/Main USt.IdNr. DE 112152620 Gesch?ftsf?hrer / Programmdirektor: Hans-Dieter Hillmoth From dridi at varni.sh Thu Sep 20 09:18:48 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 20 Sep 2018 11:18:48 +0200 Subject: Varnish VMods for Version 6.1? In-Reply-To: <8A76B2DB46634944BEEBB218DAE22BC265522A6971@exchange-srv2.office.radioteleffh.de> References: <8A76B2DB46634944BEEBB218DAE22BC265522A6968@exchange-srv2.office.radioteleffh.de> <8A76B2DB46634944BEEBB218DAE22BC265522A6971@exchange-srv2.office.radioteleffh.de> Message-ID: On Thu, Sep 20, 2018 at 10:22 AM Winkelmann, Thomas (RADIO TELE FFH - Online) wrote: > > Thanks for the feedback. We tried 6.0.1. So far H/2 seems to be working fine under normal conditions. But as soon as the request are increasing ( > 800 reg/s) the traffic will stop. Also HTTP 1.1. traffic. Sound like these two problems: > > https://github.com/varnishcache/varnish-cache/issues/2431 > https://github.com/varnishcache/varnish-cache/issues/2418 2418 may very well be responsible. You could try working around this by adding more worker threads to your pools. That would postpone the deadlock to higher concurrent traffic. And possibly increase the thread reserve as well. > Let us know if we can help to solve these problems. I'm afraid we already have a consistent reproducer, so I guess all that remains is that we (Dag) roll up our (his) sleeves and do something about it. Joke aside, we've looked at it then but it's not an easy task. In my previous message I said: > so ignore varnish-modules for now, please be aware of this before > choosing one. What I really meant was: > so ignor[ing] varnish-modules for now, please be aware of this before > choosing one. In other words, once varnish-modules supports 6.1, please consider that 6.0 is an LTS series before you pick your target. Not that there is anything wrong about picking 6.1 if you are planning to periodically have major upgrades (I wouldn't recommend staying on an unmaintained branch once it reaches EOL). Dridi From t.winkelmann at ffh.de Thu Sep 20 09:39:13 2018 From: t.winkelmann at ffh.de (Winkelmann, Thomas (RADIO TELE FFH - Online)) Date: Thu, 20 Sep 2018 11:39:13 +0200 Subject: AW: Varnish VMods for Version 6.1? In-Reply-To: References: <8A76B2DB46634944BEEBB218DAE22BC265522A6968@exchange-srv2.office.radioteleffh.de> <8A76B2DB46634944BEEBB218DAE22BC265522A6971@exchange-srv2.office.radioteleffh.de> Message-ID: <8A76B2DB46634944BEEBB218DAE22BC265522A6975@exchange-srv2.office.radioteleffh.de> Thanks again! We already use -p thread_pools=2 -p thread_pool_min=200 -p thread_pool_max=5000 I think this results already in a high number of threads. So it's probably the best to wait until you have fixed the bug. -----Urspr?ngliche Nachricht----- Von: Dridi Boukelmoune [mailto:dridi at varni.sh] Gesendet: Donnerstag, 20. September 2018 11:19 An: Winkelmann, Thomas (RADIO TELE FFH - Online) Cc: varnish-misc Betreff: Re: Varnish VMods for Version 6.1? On Thu, Sep 20, 2018 at 10:22 AM Winkelmann, Thomas (RADIO TELE FFH - Online) wrote: > > Thanks for the feedback. We tried 6.0.1. So far H/2 seems to be working fine under normal conditions. But as soon as the request are increasing ( > 800 reg/s) the traffic will stop. Also HTTP 1.1. traffic. Sound like these two problems: > > https://github.com/varnishcache/varnish-cache/issues/2431 > https://github.com/varnishcache/varnish-cache/issues/2418 2418 may very well be responsible. You could try working around this by adding more worker threads to your pools. That would postpone the deadlock to higher concurrent traffic. And possibly increase the thread reserve as well. > Let us know if we can help to solve these problems. I'm afraid we already have a consistent reproducer, so I guess all that remains is that we (Dag) roll up our (his) sleeves and do something about it. Joke aside, we've looked at it then but it's not an easy task. In my previous message I said: > so ignore varnish-modules for now, please be aware of this before > choosing one. What I really meant was: > so ignor[ing] varnish-modules for now, please be aware of this before > choosing one. In other words, once varnish-modules supports 6.1, please consider that 6.0 is an LTS series before you pick your target. Not that there is anything wrong about picking 6.1 if you are planning to periodically have major upgrades (I wouldn't recommend staying on an unmaintained branch once it reaches EOL). Dridi RADIO / TELE FFH GmbH & Co. Betriebs-KG FFH-Platz 1, 61111 Bad Vilbel HRA - Nr. 26092 Frankfurt/Main USt.IdNr. DE 112152620 Gesch?ftsf?hrer / Programmdirektor: Hans-Dieter Hillmoth From dridi at varni.sh Thu Sep 20 12:28:18 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 20 Sep 2018 14:28:18 +0200 Subject: Varnish VMods for Version 6.1? In-Reply-To: <8A76B2DB46634944BEEBB218DAE22BC265522A6975@exchange-srv2.office.radioteleffh.de> References: <8A76B2DB46634944BEEBB218DAE22BC265522A6968@exchange-srv2.office.radioteleffh.de> <8A76B2DB46634944BEEBB218DAE22BC265522A6971@exchange-srv2.office.radioteleffh.de> <8A76B2DB46634944BEEBB218DAE22BC265522A6975@exchange-srv2.office.radioteleffh.de> Message-ID: On Thu, Sep 20, 2018 at 11:39 AM Winkelmann, Thomas (RADIO TELE FFH - Online) wrote: > > Thanks again! We already use > > -p thread_pools=2 > -p thread_pool_min=200 > -p thread_pool_max=5000 > > I think this results already in a high number of threads. > > So it's probably the best to wait until you have fixed the bug. Did you try increasing thread_pool_reserve as well? The default is 5% of thread_pool_min and the maximum is 95%. To stay with the same maximum, you could try something like: -p thread_pools=2 -p thread_pool_min=2000 -p thread_pool_max=5000 -p thread_pool_reserve=500 Dridi From t.winkelmann at ffh.de Thu Sep 20 13:51:14 2018 From: t.winkelmann at ffh.de (Winkelmann, Thomas (RADIO TELE FFH - Online)) Date: Thu, 20 Sep 2018 15:51:14 +0200 Subject: AW: Varnish VMods for Version 6.1? In-Reply-To: References: <8A76B2DB46634944BEEBB218DAE22BC265522A6968@exchange-srv2.office.radioteleffh.de> <8A76B2DB46634944BEEBB218DAE22BC265522A6971@exchange-srv2.office.radioteleffh.de> <8A76B2DB46634944BEEBB218DAE22BC265522A6975@exchange-srv2.office.radioteleffh.de> Message-ID: <8A76B2DB46634944BEEBB218DAE22BC265522A697B@exchange-srv2.office.radioteleffh.de> Just tested these values. Varnish is nearly dead after restarting with these new values. Only a few requests will be served. I think, we have to wait :) -----Urspr?ngliche Nachricht----- Von: Dridi Boukelmoune [mailto:dridi at varni.sh] Gesendet: Donnerstag, 20. September 2018 14:28 An: Winkelmann, Thomas (RADIO TELE FFH - Online) Cc: varnish-misc Betreff: Re: Varnish VMods for Version 6.1? On Thu, Sep 20, 2018 at 11:39 AM Winkelmann, Thomas (RADIO TELE FFH - Online) wrote: > > Thanks again! We already use > > -p thread_pools=2 > -p thread_pool_min=200 > -p thread_pool_max=5000 > > I think this results already in a high number of threads. > > So it's probably the best to wait until you have fixed the bug. Did you try increasing thread_pool_reserve as well? The default is 5% of thread_pool_min and the maximum is 95%. To stay with the same maximum, you could try something like: -p thread_pools=2 -p thread_pool_min=2000 -p thread_pool_max=5000 -p thread_pool_reserve=500 Dridi RADIO / TELE FFH GmbH & Co. Betriebs-KG FFH-Platz 1, 61111 Bad Vilbel HRA - Nr. 26092 Frankfurt/Main USt.IdNr. DE 112152620 Gesch?ftsf?hrer / Programmdirektor: Hans-Dieter Hillmoth From junaid.mukhtar at gmail.com Thu Sep 20 14:11:31 2018 From: junaid.mukhtar at gmail.com (Junaid Mukhtar) Date: Thu, 20 Sep 2018 15:11:31 +0100 Subject: Varnish 4 Performance Issues Message-ID: Hi we are in middle of upgrading from varnish 3.0.7 to varnish 4.1.10; but unfortunately all of the response times in the performance test are indicating an increase of at least 100% We have analyzed the logs and everything but can't get around this; the issue is not only for the non-cached pages but it's also impacting the cached pages. We are also struggling to analyze it further as well, any guidenace as to what we can do in terms of putting debug logging in would be helpful. One of the areas we are considering is to output response times, but it's proving difficult for me. Any suggestions -------- Regards, Junaid -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Thu Sep 20 14:36:25 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 20 Sep 2018 16:36:25 +0200 Subject: Varnish 4 Performance Issues In-Reply-To: References: Message-ID: On Thu, Sep 20, 2018 at 4:14 PM Junaid Mukhtar wrote: > > Hi > > we are in middle of upgrading from varnish 3.0.7 to varnish 4.1.10; but unfortunately all of the response times in the performance test are indicating an increase of at least 100% > > We have analyzed the logs and everything but can't get around this; the issue is not only for the non-cached pages but it's also impacting the cached pages. We are also struggling to analyze it further as well, any guidenace as to what we can do in terms of putting debug logging in would be helpful. > > One of the areas we are considering is to output response times, but it's proving difficult for me. Any suggestions Hello, What is the output of `varnishadm param.show` ? Dridi From junaid.mukhtar at gmail.com Thu Sep 20 14:54:41 2018 From: junaid.mukhtar at gmail.com (Junaid Mukhtar) Date: Thu, 20 Sep 2018 15:54:41 +0100 Subject: Varnish 4 Performance Issues In-Reply-To: References: Message-ID: Below is the output; all of them are default except i was trying to up the threads_pool to 4 but didn't feel any imporvement in performance degradation accept_filter off [bool] (default) acceptor_sleep_decay 0.9 (default) acceptor_sleep_incr 0.000 [seconds] (default) acceptor_sleep_max 0.050 [seconds] (default) auto_restart on [bool] (default) backend_idle_timeout 60.000 [seconds] (default) ban_dups on [bool] (default) ban_lurker_age 60.000 [seconds] (default) ban_lurker_batch 1000 (default) ban_lurker_sleep 0.010 [seconds] (default) between_bytes_timeout 60.000 [seconds] (default) cc_command "exec gcc -std=gnu99 -O2 -g -Wp,-D_FORTIFY_SOURCE=0 -Wall -Werror -pthread -fpic -shared -Wl,-x -o %o %s" (default) cli_buffer 8k [bytes] (default) cli_limit 48k [bytes] (default) cli_timeout 60.000 [seconds] (default) clock_skew 10 [seconds] (default) clock_step 1.000 [seconds] (default) connect_timeout 3.500 [seconds] (default) critbit_cooloff 180.000 [seconds] (default) debug none (default) default_grace 10.000 [seconds] (default) default_keep 0.000 [seconds] (default) default_ttl 120.000 [seconds] (default) feature none (default) fetch_chunksize 16k [bytes] (default) fetch_maxchunksize 0.25G [bytes] (default) first_byte_timeout 60.000 [seconds] (default) gzip_buffer 32k [bytes] (default) gzip_level 6 (default) gzip_memlevel 8 (default) http_gzip_support on [bool] (default) http_max_hdr 64 [header lines] (default) http_range_support on [bool] (default) http_req_hdr_len 8k [bytes] (default) http_req_size 32k [bytes] (default) http_resp_hdr_len 8k [bytes] (default) http_resp_size 32k [bytes] (default) idle_send_timeout 60.000 [seconds] (default) listen_depth 1024 [connections] (default) lru_interval 2.000 [seconds] (default) max_esi_depth 5 [levels] (default) max_restarts 4 [restarts] (default) max_retries 4 [retries] (default) nuke_limit 50 [allocations] (default) pcre_match_limit 10000 (default) pcre_match_limit_recursion 20 (default) ping_interval 3 [seconds] (default) pipe_timeout 60.000 [seconds] (default) pool_req 10,100,10 (default) pool_sess 10,100,10 (default) pool_vbo 10,100,10 (default) prefer_ipv6 off [bool] (default) rush_exponent 3 [requests per request] (default) send_timeout 600.000 [seconds] (default) session_max 100000 [sessions] (default) shm_reclen 255b [bytes] (default) shortlived 10.000 [seconds] (default) sigsegv_handler on [bool] (default) syslog_cli_traffic on [bool] (default) tcp_fastopen off [bool] (default) tcp_keepalive_intvl 75.000 [seconds] (default) tcp_keepalive_probes 9 [probes] (default) tcp_keepalive_time 7200.000 [seconds] (default) thread_pool_add_delay 0.000 [seconds] (default) thread_pool_destroy_delay 1.000 [seconds] (default) thread_pool_fail_delay 0.200 [seconds] (default) thread_pool_max 5000 [threads] (default) thread_pool_min 100 [threads] (default) thread_pool_reserve 0 [threads] (default) thread_pool_stack 48k [bytes] (default) thread_pool_timeout 300.000 [seconds] (default) thread_pools 2 [pools] (default) thread_queue_limit 20 (default) thread_stats_rate 10 [requests] (default) timeout_idle 5.000 [seconds] (default) timeout_linger 0.050 [seconds] (default) vcc_allow_inline_c off [bool] (default) vcc_err_unref on [bool] (default) vcc_unsafe_path on [bool] (default) vcl_cooldown 600.000 [seconds] (default) vcl_dir /etc/varnish (default) vmod_dir /usr/lib64/varnish/vmods (default) vsl_buffer 4k [bytes] (default) vsl_mask -VCL_trace,-WorkThread,-Hash,-VfpAcct (default) vsl_reclen 255b [bytes] (default) vsl_space 80M [bytes] (default) vsm_free_cooldown 60.000 [seconds] (default) vsm_space 1M [bytes] (default) workspace_backend 64k [bytes] (default) workspace_client 64k [bytes] (default) workspace_session 0.50k [bytes] (default) workspace_thread 2k [bytes] (default) -------- Regards, Junaid On Thu, Sep 20, 2018 at 3:37 PM Dridi Boukelmoune wrote: > On Thu, Sep 20, 2018 at 4:14 PM Junaid Mukhtar > wrote: > > > > Hi > > > > we are in middle of upgrading from varnish 3.0.7 to varnish 4.1.10; but > unfortunately all of the response times in the performance test are > indicating an increase of at least 100% > > > > We have analyzed the logs and everything but can't get around this; the > issue is not only for the non-cached pages but it's also impacting the > cached pages. We are also struggling to analyze it further as well, any > guidenace as to what we can do in terms of putting debug logging in would > be helpful. > > > > One of the areas we are considering is to output response times, but > it's proving difficult for me. Any suggestions > > Hello, > > What is the output of `varnishadm param.show` ? > > Dridi > -------------- next part -------------- An HTML attachment was scrubbed... URL: From junaid.mukhtar at gmail.com Thu Sep 20 14:57:31 2018 From: junaid.mukhtar at gmail.com (Junaid Mukhtar) Date: Thu, 20 Sep 2018 15:57:31 +0100 Subject: Varnish 4 Performance Issues In-Reply-To: References: Message-ID: This is what we get in varnishhist 280_ | 230_ | | | | | 180_ | || || ||| ||| 130_ ||| ||| |||||| |||||| |||||| 80_ |||||| |||||| # |||||| ### ||||||| # ### ||||||||| ##### # 30_ ||||||||| ######## ||||||||| ######### |||||||||| ########### |||||||||||| ## # ############# # +---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+--------------------- |1e-6 |1e-5 |1e-4 |1e-3 |1e-2 |1e-1 |1e0 |1e1 |1e2 -------- Regards, Junaid On Thu, Sep 20, 2018 at 3:54 PM Junaid Mukhtar wrote: > Below is the output; all of them are default except i was trying to up the > threads_pool to 4 but didn't feel any imporvement in performance degradation > > accept_filter off [bool] (default) > acceptor_sleep_decay 0.9 (default) > acceptor_sleep_incr 0.000 [seconds] (default) > acceptor_sleep_max 0.050 [seconds] (default) > auto_restart on [bool] (default) > backend_idle_timeout 60.000 [seconds] (default) > ban_dups on [bool] (default) > ban_lurker_age 60.000 [seconds] (default) > ban_lurker_batch 1000 (default) > ban_lurker_sleep 0.010 [seconds] (default) > between_bytes_timeout 60.000 [seconds] (default) > cc_command "exec gcc -std=gnu99 -O2 -g > -Wp,-D_FORTIFY_SOURCE=0 -Wall -Werror -pthread -fpic -shared -Wl,-x -o %o > %s" (default) > cli_buffer 8k [bytes] (default) > cli_limit 48k [bytes] (default) > cli_timeout 60.000 [seconds] (default) > clock_skew 10 [seconds] (default) > clock_step 1.000 [seconds] (default) > connect_timeout 3.500 [seconds] (default) > critbit_cooloff 180.000 [seconds] (default) > debug none (default) > default_grace 10.000 [seconds] (default) > default_keep 0.000 [seconds] (default) > default_ttl 120.000 [seconds] (default) > feature none (default) > fetch_chunksize 16k [bytes] (default) > fetch_maxchunksize 0.25G [bytes] (default) > first_byte_timeout 60.000 [seconds] (default) > gzip_buffer 32k [bytes] (default) > gzip_level 6 (default) > gzip_memlevel 8 (default) > http_gzip_support on [bool] (default) > http_max_hdr 64 [header lines] (default) > http_range_support on [bool] (default) > http_req_hdr_len 8k [bytes] (default) > http_req_size 32k [bytes] (default) > http_resp_hdr_len 8k [bytes] (default) > http_resp_size 32k [bytes] (default) > idle_send_timeout 60.000 [seconds] (default) > listen_depth 1024 [connections] (default) > lru_interval 2.000 [seconds] (default) > max_esi_depth 5 [levels] (default) > max_restarts 4 [restarts] (default) > max_retries 4 [retries] (default) > nuke_limit 50 [allocations] (default) > pcre_match_limit 10000 (default) > pcre_match_limit_recursion 20 (default) > ping_interval 3 [seconds] (default) > pipe_timeout 60.000 [seconds] (default) > pool_req 10,100,10 (default) > pool_sess 10,100,10 (default) > pool_vbo 10,100,10 (default) > prefer_ipv6 off [bool] (default) > rush_exponent 3 [requests per request] (default) > send_timeout 600.000 [seconds] (default) > session_max 100000 [sessions] (default) > shm_reclen 255b [bytes] (default) > shortlived 10.000 [seconds] (default) > sigsegv_handler on [bool] (default) > syslog_cli_traffic on [bool] (default) > tcp_fastopen off [bool] (default) > tcp_keepalive_intvl 75.000 [seconds] (default) > tcp_keepalive_probes 9 [probes] (default) > tcp_keepalive_time 7200.000 [seconds] (default) > thread_pool_add_delay 0.000 [seconds] (default) > thread_pool_destroy_delay 1.000 [seconds] (default) > thread_pool_fail_delay 0.200 [seconds] (default) > thread_pool_max 5000 [threads] (default) > thread_pool_min 100 [threads] (default) > thread_pool_reserve 0 [threads] (default) > thread_pool_stack 48k [bytes] (default) > thread_pool_timeout 300.000 [seconds] (default) > thread_pools 2 [pools] (default) > thread_queue_limit 20 (default) > thread_stats_rate 10 [requests] (default) > timeout_idle 5.000 [seconds] (default) > timeout_linger 0.050 [seconds] (default) > vcc_allow_inline_c off [bool] (default) > vcc_err_unref on [bool] (default) > vcc_unsafe_path on [bool] (default) > vcl_cooldown 600.000 [seconds] (default) > vcl_dir /etc/varnish (default) > vmod_dir /usr/lib64/varnish/vmods (default) > vsl_buffer 4k [bytes] (default) > vsl_mask -VCL_trace,-WorkThread,-Hash,-VfpAcct (default) > vsl_reclen 255b [bytes] (default) > vsl_space 80M [bytes] (default) > vsm_free_cooldown 60.000 [seconds] (default) > vsm_space 1M [bytes] (default) > workspace_backend 64k [bytes] (default) > workspace_client 64k [bytes] (default) > workspace_session 0.50k [bytes] (default) > workspace_thread 2k [bytes] (default) > > > -------- > Regards, > Junaid > > > On Thu, Sep 20, 2018 at 3:37 PM Dridi Boukelmoune wrote: > >> On Thu, Sep 20, 2018 at 4:14 PM Junaid Mukhtar >> wrote: >> > >> > Hi >> > >> > we are in middle of upgrading from varnish 3.0.7 to varnish 4.1.10; but >> unfortunately all of the response times in the performance test are >> indicating an increase of at least 100% >> > >> > We have analyzed the logs and everything but can't get around this; the >> issue is not only for the non-cached pages but it's also impacting the >> cached pages. We are also struggling to analyze it further as well, any >> guidenace as to what we can do in terms of putting debug logging in would >> be helpful. >> > >> > One of the areas we are considering is to output response times, but >> it's proving difficult for me. Any suggestions >> >> Hello, >> >> What is the output of `varnishadm param.show` ? >> >> Dridi >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adrien.bigot at smile.fr Thu Sep 20 20:03:55 2018 From: adrien.bigot at smile.fr (BIGOT Adrien) Date: Thu, 20 Sep 2018 22:03:55 +0200 Subject: CLI start with multiple VCL on Varnish 6 Failed. Message-ID: Hello, Based on this blog article : https://info.varnish-software.com/blog/one-vcl-per-domain?? I've been using multiple VCL with Varnish 5.2 without any problem. However, I don't manage to get it working with Varnish 6. I tried with the latest stable version on packagecloud.io and even with the latest weekly-build. Tested on Debian Strecth, Ubuntu 16.04 and Centos 7. [root at ip-172-31-137-19 varnish]# varnishd -I start.cli -F -a :80 -f '' Debug: Version: varnish-6.0.1 revision 8d54bec5330c29304979ebf2c425ae14ab80493c Debug: Platform: Linux,3.10.0-862.3.2.el7.x86_64,x86_64,-junix,-sdefault,-sdefault,-hcritbit BEGIN of -I file processing 200 326 ----------------------------- Varnish Cache CLI 1.0 ----------------------------- Linux,3.10.0-862.3.2.el7.x86_64,x86_64,-junix,-sdefault,-sdefault,-hcritbit varnish-6.0.1 revision 8d54bec5330c29304979ebf2c425ae14ab80493c Type 'help' for command list. Type 'quit' to close CLI session. Type 'start' to launch worker process. > vcl.load vcl-api-orig /etc/varnish/api.vcl 200 14 VCL compiled. > vcl.load vcl-catz-orig /etc/varnish/catz.vcl 200 14 VCL compiled. > vcl.label label-api vcl-api-orig 200 0 > vcl.label label-catz vcl-catz-orig 200 0 > vcl.load vcl-root-orig /etc/varnish/root.vcl 200 14 VCL compiled. > vcl.use vcl-root-orig 200 30 VCL 'vcl-root-orig' now active END of -I file processing CLI result = 400 In the logs : On Centos sept. 18 08:35:37 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx su[11820]: pam_unix(su:session): session opened for user root by ec2-user(uid=0) sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: Version: varnish-6.0.1 revision 8d54bec5330c29304979ebf2c425ae14ab80493c sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: Platform: Linux,3.10.0-862.3.2.el7.x86_64,x86_64,-junix,-sdefault,-sdefault,-hcritbit sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Rd vcl.load vcl-api-orig /etc/varnish/api.vcl sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Wr 200 VCL compiled. sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Rd vcl.load vcl-catz-orig /etc/varnish/catz.vcl sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Wr 200 VCL compiled. sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Rd vcl.label label-api vcl-api-orig sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Wr 200 sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Rd vcl.label label-catz vcl-catz-orig sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Wr 200 sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Rd vcl.load vcl-root-orig /etc/varnish/root.vcl sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Wr 200 VCL compiled. sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: CLI -I file Rd vcl.use vcl-root-orig sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxxm varnishd[11846]: CLI -I file Wr 200 VCL 'vcl-root-orig' now active sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: Child (11870) Started sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: Child (11870) not responding to CLI, killed it. sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: Child (11870) Pushing vcls failed: CLI communication error sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: Stopping Child sept. 18 08:36:35 ip-172-31-137-19.xxxxxxxxxxxxxxxxxxm varnishd[11846]: Child (11870) died signal=3 sept. 18 08:36:35 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: Child cleanup complete On Debian : Sep 18 10:07:01 ip-172-31-137-84 varnishd[6783]: Child (6809) said Child starts Sep 18 10:07:01 ip-172-31-137-84 varnishd[6783]: Child cleanup complete Sep 18 10:07:02 ip-172-31-137-84 varnishd[6816]: Version: varnish-6.0.1 revision 8d54bec5330c29304979ebf2c425ae14ab80493c Sep 18 10:07:02 ip-172-31-137-84 varnishd[6816]: Platform: Linux,4.9.0-8-amd64,x86_64,-junix,-sdefault,-sdefault,-hcritbit Sep 18 10:07:02 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.load vcl-api-orig /etc/varnish/api.vcl Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 VCL compiled. Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.load vcl-catz-orig /etc/varnish/catz.vcl Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 VCL compiled. Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.label label-api vcl-api-orig Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.label label-catz vcl-catz-orig Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.load vcl-root-orig /etc/varnish/root.vcl Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 VCL compiled. Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.use vcl-root-orig Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 VCL 'vcl-root-orig' now active Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: Child (6842) Started Sep 18 10:08:03 ip-172-31-137-84 varnishd[6816]: Child (6842) not responding to CLI, killed it. Sep 18 10:08:03 ip-172-31-137-84 varnishd[6816]: Child (6842) Pushing vcls failed: ???????????????????????????????????????????????? CLI communication error (hdr) Sep 18 10:08:03 ip-172-31-137-84 varnishd[6816]: Stopping Child Sep 18 10:08:04 ip-172-31-137-84 varnishd[6816]: Child (6842) died signal=3 Did anyone manage to get it working with varnish 6 ? Thank you. Best regards, -- Bigot Adrien From dridi at varni.sh Fri Sep 21 08:56:28 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 21 Sep 2018 10:56:28 +0200 Subject: Varnish VMods for Version 6.1? In-Reply-To: <8A76B2DB46634944BEEBB218DAE22BC265522A697B@exchange-srv2.office.radioteleffh.de> References: <8A76B2DB46634944BEEBB218DAE22BC265522A6968@exchange-srv2.office.radioteleffh.de> <8A76B2DB46634944BEEBB218DAE22BC265522A6971@exchange-srv2.office.radioteleffh.de> <8A76B2DB46634944BEEBB218DAE22BC265522A6975@exchange-srv2.office.radioteleffh.de> <8A76B2DB46634944BEEBB218DAE22BC265522A697B@exchange-srv2.office.radioteleffh.de> Message-ID: On Thu, Sep 20, 2018 at 3:51 PM Winkelmann, Thomas (RADIO TELE FFH - Online) wrote: > > Just tested these values. Varnish is nearly dead after restarting with these new values. Only a few requests will be served. I think, we have to wait :) I guess now would be a good time to bring back h2load. If you increase thread_pool_min to 2000 (a value that shouldn't be unreasonable) that brings the thread_pool_reserve to 400, so even if I didn't bump it further to 500 it'd be in the same ballpark. And if that is enough to bring Varnish to a crawl we may have a deeper problem here. As a general rule of thumb we recommend a thread_pool_min sized for high (for a given definition of high, let's say above average) traffic so that sudden peaks of traffic wouldn't wait too long for threads to be created. I don't see why that should become a problem with Varnish 6 :-/ Dridi From t.winkelmann at ffh.de Fri Sep 21 11:50:43 2018 From: t.winkelmann at ffh.de (Winkelmann, Thomas (RADIO TELE FFH - Online)) Date: Fri, 21 Sep 2018 13:50:43 +0200 Subject: AW: Varnish VMods for Version 6.1? In-Reply-To: References: <8A76B2DB46634944BEEBB218DAE22BC265522A6968@exchange-srv2.office.radioteleffh.de> <8A76B2DB46634944BEEBB218DAE22BC265522A6971@exchange-srv2.office.radioteleffh.de> <8A76B2DB46634944BEEBB218DAE22BC265522A6975@exchange-srv2.office.radioteleffh.de> <8A76B2DB46634944BEEBB218DAE22BC265522A697B@exchange-srv2.office.radioteleffh.de> Message-ID: <8A76B2DB46634944BEEBB218DAE22BC265522A698A@exchange-srv2.office.radioteleffh.de> I again did some testing by settings the params to -p thread_pools=4 -p thread_pool_min=2000 -p thread_pool_max=5000 -p thread_pool_reserve=95 The document says https://varnish-cache.org/docs/trunk/reference/varnishd.html that thread_pool_reserve is a value from 0 to 95. It's working fine on one of our two servers which have less load (300 req/s) than the primaray one (> 800req/s). On the primary one you can see decreasing the client.req in varnishstat two seconds after restarting hitch with H/2 enabled. It jumps from 100req/s to 1700req/s from one second to the next. -----Urspr?ngliche Nachricht----- Von: Dridi Boukelmoune [mailto:dridi at varni.sh] Gesendet: Freitag, 21. September 2018 10:56 An: Winkelmann, Thomas (RADIO TELE FFH - Online) Cc: varnish-misc Betreff: Re: Varnish VMods for Version 6.1? On Thu, Sep 20, 2018 at 3:51 PM Winkelmann, Thomas (RADIO TELE FFH - Online) wrote: > > Just tested these values. Varnish is nearly dead after restarting with these new values. Only a few requests will be served. I think, we have to wait :) I guess now would be a good time to bring back h2load. If you increase thread_pool_min to 2000 (a value that shouldn't be unreasonable) that brings the thread_pool_reserve to 400, so even if I didn't bump it further to 500 it'd be in the same ballpark. And if that is enough to bring Varnish to a crawl we may have a deeper problem here. As a general rule of thumb we recommend a thread_pool_min sized for high (for a given definition of high, let's say above average) traffic so that sudden peaks of traffic wouldn't wait too long for threads to be created. I don't see why that should become a problem with Varnish 6 :-/ Dridi RADIO / TELE FFH GmbH & Co. Betriebs-KG FFH-Platz 1, 61111 Bad Vilbel HRA - Nr. 26092 Frankfurt/Main USt.IdNr. DE 112152620 Gesch?ftsf?hrer / Programmdirektor: Hans-Dieter Hillmoth From junaid.mukhtar at gmail.com Fri Sep 21 14:52:04 2018 From: junaid.mukhtar at gmail.com (Junaid Mukhtar) Date: Fri, 21 Sep 2018 15:52:04 +0100 Subject: Varnish 4 Performance Issues In-Reply-To: References: Message-ID: Any help guys; this is really getting to us..... -------- Regards, Junaid On Thu, Sep 20, 2018 at 3:57 PM Junaid Mukhtar wrote: > > This is what we get in varnishhist > > 280_ > > > > | > 230_ | > | > | > | > | > 180_ | > || > || > ||| > ||| > 130_ ||| > ||| > |||||| > |||||| > |||||| > 80_ |||||| > > |||||| # > > |||||| ### > > ||||||| # ### > > ||||||||| ##### # > 30_ > ||||||||| ######## > > ||||||||| > ######### > > |||||||||| > ########### > > |||||||||||| ## # > ############# # > > +---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+--------------------- > |1e-6 |1e-5 |1e-4 > |1e-3 |1e-2 |1e-1 > |1e0 |1e1 |1e2 > -------- > Regards, > Junaid > > > On Thu, Sep 20, 2018 at 3:54 PM Junaid Mukhtar > wrote: > >> Below is the output; all of them are default except i was trying to up >> the threads_pool to 4 but didn't feel any imporvement in performance >> degradation >> >> accept_filter off [bool] (default) >> acceptor_sleep_decay 0.9 (default) >> acceptor_sleep_incr 0.000 [seconds] (default) >> acceptor_sleep_max 0.050 [seconds] (default) >> auto_restart on [bool] (default) >> backend_idle_timeout 60.000 [seconds] (default) >> ban_dups on [bool] (default) >> ban_lurker_age 60.000 [seconds] (default) >> ban_lurker_batch 1000 (default) >> ban_lurker_sleep 0.010 [seconds] (default) >> between_bytes_timeout 60.000 [seconds] (default) >> cc_command "exec gcc -std=gnu99 -O2 -g >> -Wp,-D_FORTIFY_SOURCE=0 -Wall -Werror -pthread -fpic -shared -Wl,-x -o %o >> %s" (default) >> cli_buffer 8k [bytes] (default) >> cli_limit 48k [bytes] (default) >> cli_timeout 60.000 [seconds] (default) >> clock_skew 10 [seconds] (default) >> clock_step 1.000 [seconds] (default) >> connect_timeout 3.500 [seconds] (default) >> critbit_cooloff 180.000 [seconds] (default) >> debug none (default) >> default_grace 10.000 [seconds] (default) >> default_keep 0.000 [seconds] (default) >> default_ttl 120.000 [seconds] (default) >> feature none (default) >> fetch_chunksize 16k [bytes] (default) >> fetch_maxchunksize 0.25G [bytes] (default) >> first_byte_timeout 60.000 [seconds] (default) >> gzip_buffer 32k [bytes] (default) >> gzip_level 6 (default) >> gzip_memlevel 8 (default) >> http_gzip_support on [bool] (default) >> http_max_hdr 64 [header lines] (default) >> http_range_support on [bool] (default) >> http_req_hdr_len 8k [bytes] (default) >> http_req_size 32k [bytes] (default) >> http_resp_hdr_len 8k [bytes] (default) >> http_resp_size 32k [bytes] (default) >> idle_send_timeout 60.000 [seconds] (default) >> listen_depth 1024 [connections] (default) >> lru_interval 2.000 [seconds] (default) >> max_esi_depth 5 [levels] (default) >> max_restarts 4 [restarts] (default) >> max_retries 4 [retries] (default) >> nuke_limit 50 [allocations] (default) >> pcre_match_limit 10000 (default) >> pcre_match_limit_recursion 20 (default) >> ping_interval 3 [seconds] (default) >> pipe_timeout 60.000 [seconds] (default) >> pool_req 10,100,10 (default) >> pool_sess 10,100,10 (default) >> pool_vbo 10,100,10 (default) >> prefer_ipv6 off [bool] (default) >> rush_exponent 3 [requests per request] (default) >> send_timeout 600.000 [seconds] (default) >> session_max 100000 [sessions] (default) >> shm_reclen 255b [bytes] (default) >> shortlived 10.000 [seconds] (default) >> sigsegv_handler on [bool] (default) >> syslog_cli_traffic on [bool] (default) >> tcp_fastopen off [bool] (default) >> tcp_keepalive_intvl 75.000 [seconds] (default) >> tcp_keepalive_probes 9 [probes] (default) >> tcp_keepalive_time 7200.000 [seconds] (default) >> thread_pool_add_delay 0.000 [seconds] (default) >> thread_pool_destroy_delay 1.000 [seconds] (default) >> thread_pool_fail_delay 0.200 [seconds] (default) >> thread_pool_max 5000 [threads] (default) >> thread_pool_min 100 [threads] (default) >> thread_pool_reserve 0 [threads] (default) >> thread_pool_stack 48k [bytes] (default) >> thread_pool_timeout 300.000 [seconds] (default) >> thread_pools 2 [pools] (default) >> thread_queue_limit 20 (default) >> thread_stats_rate 10 [requests] (default) >> timeout_idle 5.000 [seconds] (default) >> timeout_linger 0.050 [seconds] (default) >> vcc_allow_inline_c off [bool] (default) >> vcc_err_unref on [bool] (default) >> vcc_unsafe_path on [bool] (default) >> vcl_cooldown 600.000 [seconds] (default) >> vcl_dir /etc/varnish (default) >> vmod_dir /usr/lib64/varnish/vmods (default) >> vsl_buffer 4k [bytes] (default) >> vsl_mask -VCL_trace,-WorkThread,-Hash,-VfpAcct (default) >> vsl_reclen 255b [bytes] (default) >> vsl_space 80M [bytes] (default) >> vsm_free_cooldown 60.000 [seconds] (default) >> vsm_space 1M [bytes] (default) >> workspace_backend 64k [bytes] (default) >> workspace_client 64k [bytes] (default) >> workspace_session 0.50k [bytes] (default) >> workspace_thread 2k [bytes] (default) >> >> >> -------- >> Regards, >> Junaid >> >> >> On Thu, Sep 20, 2018 at 3:37 PM Dridi Boukelmoune wrote: >> >>> On Thu, Sep 20, 2018 at 4:14 PM Junaid Mukhtar >>> wrote: >>> > >>> > Hi >>> > >>> > we are in middle of upgrading from varnish 3.0.7 to varnish 4.1.10; >>> but unfortunately all of the response times in the performance test are >>> indicating an increase of at least 100% >>> > >>> > We have analyzed the logs and everything but can't get around this; >>> the issue is not only for the non-cached pages but it's also impacting the >>> cached pages. We are also struggling to analyze it further as well, any >>> guidenace as to what we can do in terms of putting debug logging in would >>> be helpful. >>> > >>> > One of the areas we are considering is to output response times, but >>> it's proving difficult for me. Any suggestions >>> >>> Hello, >>> >>> What is the output of `varnishadm param.show` ? >>> >>> Dridi >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Fri Sep 21 15:14:51 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Fri, 21 Sep 2018 17:14:51 +0200 Subject: Varnish 4 Performance Issues In-Reply-To: References: Message-ID: would you be able to share your vcl? -- Guillaume Quintard On Fri, Sep 21, 2018 at 4:54 PM Junaid Mukhtar wrote: > Any help guys; this is really getting to us..... > > -------- > Regards, > Junaid > > > On Thu, Sep 20, 2018 at 3:57 PM Junaid Mukhtar > wrote: > >> >> This is what we get in varnishhist >> >> 280_ >> >> >> >> | >> 230_ | >> | >> | >> | >> | >> 180_ | >> || >> || >> ||| >> ||| >> 130_ ||| >> ||| >> |||||| >> |||||| >> |||||| >> 80_ |||||| >> >> |||||| # >> >> |||||| ### >> >> ||||||| # ### >> >> ||||||||| ##### # >> 30_ >> ||||||||| ######## >> >> ||||||||| >> ######### >> >> |||||||||| >> ########### >> >> |||||||||||| ## # >> ############# # >> >> +---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+--------------------- >> |1e-6 |1e-5 |1e-4 >> |1e-3 |1e-2 |1e-1 >> |1e0 |1e1 |1e2 >> -------- >> Regards, >> Junaid >> >> >> On Thu, Sep 20, 2018 at 3:54 PM Junaid Mukhtar >> wrote: >> >>> Below is the output; all of them are default except i was trying to up >>> the threads_pool to 4 but didn't feel any imporvement in performance >>> degradation >>> >>> accept_filter off [bool] (default) >>> acceptor_sleep_decay 0.9 (default) >>> acceptor_sleep_incr 0.000 [seconds] (default) >>> acceptor_sleep_max 0.050 [seconds] (default) >>> auto_restart on [bool] (default) >>> backend_idle_timeout 60.000 [seconds] (default) >>> ban_dups on [bool] (default) >>> ban_lurker_age 60.000 [seconds] (default) >>> ban_lurker_batch 1000 (default) >>> ban_lurker_sleep 0.010 [seconds] (default) >>> between_bytes_timeout 60.000 [seconds] (default) >>> cc_command "exec gcc -std=gnu99 -O2 -g >>> -Wp,-D_FORTIFY_SOURCE=0 -Wall -Werror -pthread -fpic -shared -Wl,-x -o %o >>> %s" (default) >>> cli_buffer 8k [bytes] (default) >>> cli_limit 48k [bytes] (default) >>> cli_timeout 60.000 [seconds] (default) >>> clock_skew 10 [seconds] (default) >>> clock_step 1.000 [seconds] (default) >>> connect_timeout 3.500 [seconds] (default) >>> critbit_cooloff 180.000 [seconds] (default) >>> debug none (default) >>> default_grace 10.000 [seconds] (default) >>> default_keep 0.000 [seconds] (default) >>> default_ttl 120.000 [seconds] (default) >>> feature none (default) >>> fetch_chunksize 16k [bytes] (default) >>> fetch_maxchunksize 0.25G [bytes] (default) >>> first_byte_timeout 60.000 [seconds] (default) >>> gzip_buffer 32k [bytes] (default) >>> gzip_level 6 (default) >>> gzip_memlevel 8 (default) >>> http_gzip_support on [bool] (default) >>> http_max_hdr 64 [header lines] (default) >>> http_range_support on [bool] (default) >>> http_req_hdr_len 8k [bytes] (default) >>> http_req_size 32k [bytes] (default) >>> http_resp_hdr_len 8k [bytes] (default) >>> http_resp_size 32k [bytes] (default) >>> idle_send_timeout 60.000 [seconds] (default) >>> listen_depth 1024 [connections] (default) >>> lru_interval 2.000 [seconds] (default) >>> max_esi_depth 5 [levels] (default) >>> max_restarts 4 [restarts] (default) >>> max_retries 4 [retries] (default) >>> nuke_limit 50 [allocations] (default) >>> pcre_match_limit 10000 (default) >>> pcre_match_limit_recursion 20 (default) >>> ping_interval 3 [seconds] (default) >>> pipe_timeout 60.000 [seconds] (default) >>> pool_req 10,100,10 (default) >>> pool_sess 10,100,10 (default) >>> pool_vbo 10,100,10 (default) >>> prefer_ipv6 off [bool] (default) >>> rush_exponent 3 [requests per request] (default) >>> send_timeout 600.000 [seconds] (default) >>> session_max 100000 [sessions] (default) >>> shm_reclen 255b [bytes] (default) >>> shortlived 10.000 [seconds] (default) >>> sigsegv_handler on [bool] (default) >>> syslog_cli_traffic on [bool] (default) >>> tcp_fastopen off [bool] (default) >>> tcp_keepalive_intvl 75.000 [seconds] (default) >>> tcp_keepalive_probes 9 [probes] (default) >>> tcp_keepalive_time 7200.000 [seconds] (default) >>> thread_pool_add_delay 0.000 [seconds] (default) >>> thread_pool_destroy_delay 1.000 [seconds] (default) >>> thread_pool_fail_delay 0.200 [seconds] (default) >>> thread_pool_max 5000 [threads] (default) >>> thread_pool_min 100 [threads] (default) >>> thread_pool_reserve 0 [threads] (default) >>> thread_pool_stack 48k [bytes] (default) >>> thread_pool_timeout 300.000 [seconds] (default) >>> thread_pools 2 [pools] (default) >>> thread_queue_limit 20 (default) >>> thread_stats_rate 10 [requests] (default) >>> timeout_idle 5.000 [seconds] (default) >>> timeout_linger 0.050 [seconds] (default) >>> vcc_allow_inline_c off [bool] (default) >>> vcc_err_unref on [bool] (default) >>> vcc_unsafe_path on [bool] (default) >>> vcl_cooldown 600.000 [seconds] (default) >>> vcl_dir /etc/varnish (default) >>> vmod_dir /usr/lib64/varnish/vmods (default) >>> vsl_buffer 4k [bytes] (default) >>> vsl_mask -VCL_trace,-WorkThread,-Hash,-VfpAcct >>> (default) >>> vsl_reclen 255b [bytes] (default) >>> vsl_space 80M [bytes] (default) >>> vsm_free_cooldown 60.000 [seconds] (default) >>> vsm_space 1M [bytes] (default) >>> workspace_backend 64k [bytes] (default) >>> workspace_client 64k [bytes] (default) >>> workspace_session 0.50k [bytes] (default) >>> workspace_thread 2k [bytes] (default) >>> >>> >>> -------- >>> Regards, >>> Junaid >>> >>> >>> On Thu, Sep 20, 2018 at 3:37 PM Dridi Boukelmoune >>> wrote: >>> >>>> On Thu, Sep 20, 2018 at 4:14 PM Junaid Mukhtar < >>>> junaid.mukhtar at gmail.com> wrote: >>>> > >>>> > Hi >>>> > >>>> > we are in middle of upgrading from varnish 3.0.7 to varnish 4.1.10; >>>> but unfortunately all of the response times in the performance test are >>>> indicating an increase of at least 100% >>>> > >>>> > We have analyzed the logs and everything but can't get around this; >>>> the issue is not only for the non-cached pages but it's also impacting the >>>> cached pages. We are also struggling to analyze it further as well, any >>>> guidenace as to what we can do in terms of putting debug logging in would >>>> be helpful. >>>> > >>>> > One of the areas we are considering is to output response times, but >>>> it's proving difficult for me. Any suggestions >>>> >>>> Hello, >>>> >>>> What is the output of `varnishadm param.show` ? >>>> >>>> Dridi >>>> >>> _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From junaid.mukhtar at gmail.com Mon Sep 24 08:34:28 2018 From: junaid.mukhtar at gmail.com (Junaid Mukhtar) Date: Mon, 24 Sep 2018 09:34:28 +0100 Subject: Varnish 4 Performance Issues In-Reply-To: References: Message-ID: Hi Apologies but I am not allowed to share the VCL. If there's any other thing like stats etc that might be helpful then do let me know -------- Regards, Junaid On Fri, Sep 21, 2018 at 4:15 PM Guillaume Quintard < guillaume at varnish-software.com> wrote: > would you be able to share your vcl? > -- > Guillaume Quintard > > > On Fri, Sep 21, 2018 at 4:54 PM Junaid Mukhtar > wrote: > >> Any help guys; this is really getting to us..... >> >> -------- >> Regards, >> Junaid >> >> >> On Thu, Sep 20, 2018 at 3:57 PM Junaid Mukhtar >> wrote: >> >>> >>> This is what we get in varnishhist >>> >>> 280_ >>> >>> >>> >>> | >>> 230_ | >>> | >>> | >>> | >>> | >>> 180_ | >>> || >>> || >>> ||| >>> ||| >>> 130_ ||| >>> ||| >>> |||||| >>> |||||| >>> |||||| >>> 80_ |||||| >>> >>> |||||| # >>> >>> |||||| ### >>> >>> ||||||| # ### >>> >>> ||||||||| ##### # >>> 30_ >>> ||||||||| ######## >>> >>> ||||||||| >>> ######### >>> >>> |||||||||| >>> ########### >>> >>> |||||||||||| ## # >>> ############# # >>> >>> +---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+--------------------- >>> |1e-6 |1e-5 |1e-4 >>> |1e-3 |1e-2 |1e-1 >>> |1e0 |1e1 |1e2 >>> -------- >>> Regards, >>> Junaid >>> >>> >>> On Thu, Sep 20, 2018 at 3:54 PM Junaid Mukhtar >>> wrote: >>> >>>> Below is the output; all of them are default except i was trying to up >>>> the threads_pool to 4 but didn't feel any imporvement in performance >>>> degradation >>>> >>>> accept_filter off [bool] (default) >>>> acceptor_sleep_decay 0.9 (default) >>>> acceptor_sleep_incr 0.000 [seconds] (default) >>>> acceptor_sleep_max 0.050 [seconds] (default) >>>> auto_restart on [bool] (default) >>>> backend_idle_timeout 60.000 [seconds] (default) >>>> ban_dups on [bool] (default) >>>> ban_lurker_age 60.000 [seconds] (default) >>>> ban_lurker_batch 1000 (default) >>>> ban_lurker_sleep 0.010 [seconds] (default) >>>> between_bytes_timeout 60.000 [seconds] (default) >>>> cc_command "exec gcc -std=gnu99 -O2 -g >>>> -Wp,-D_FORTIFY_SOURCE=0 -Wall -Werror -pthread -fpic -shared -Wl,-x -o %o >>>> %s" (default) >>>> cli_buffer 8k [bytes] (default) >>>> cli_limit 48k [bytes] (default) >>>> cli_timeout 60.000 [seconds] (default) >>>> clock_skew 10 [seconds] (default) >>>> clock_step 1.000 [seconds] (default) >>>> connect_timeout 3.500 [seconds] (default) >>>> critbit_cooloff 180.000 [seconds] (default) >>>> debug none (default) >>>> default_grace 10.000 [seconds] (default) >>>> default_keep 0.000 [seconds] (default) >>>> default_ttl 120.000 [seconds] (default) >>>> feature none (default) >>>> fetch_chunksize 16k [bytes] (default) >>>> fetch_maxchunksize 0.25G [bytes] (default) >>>> first_byte_timeout 60.000 [seconds] (default) >>>> gzip_buffer 32k [bytes] (default) >>>> gzip_level 6 (default) >>>> gzip_memlevel 8 (default) >>>> http_gzip_support on [bool] (default) >>>> http_max_hdr 64 [header lines] (default) >>>> http_range_support on [bool] (default) >>>> http_req_hdr_len 8k [bytes] (default) >>>> http_req_size 32k [bytes] (default) >>>> http_resp_hdr_len 8k [bytes] (default) >>>> http_resp_size 32k [bytes] (default) >>>> idle_send_timeout 60.000 [seconds] (default) >>>> listen_depth 1024 [connections] (default) >>>> lru_interval 2.000 [seconds] (default) >>>> max_esi_depth 5 [levels] (default) >>>> max_restarts 4 [restarts] (default) >>>> max_retries 4 [retries] (default) >>>> nuke_limit 50 [allocations] (default) >>>> pcre_match_limit 10000 (default) >>>> pcre_match_limit_recursion 20 (default) >>>> ping_interval 3 [seconds] (default) >>>> pipe_timeout 60.000 [seconds] (default) >>>> pool_req 10,100,10 (default) >>>> pool_sess 10,100,10 (default) >>>> pool_vbo 10,100,10 (default) >>>> prefer_ipv6 off [bool] (default) >>>> rush_exponent 3 [requests per request] (default) >>>> send_timeout 600.000 [seconds] (default) >>>> session_max 100000 [sessions] (default) >>>> shm_reclen 255b [bytes] (default) >>>> shortlived 10.000 [seconds] (default) >>>> sigsegv_handler on [bool] (default) >>>> syslog_cli_traffic on [bool] (default) >>>> tcp_fastopen off [bool] (default) >>>> tcp_keepalive_intvl 75.000 [seconds] (default) >>>> tcp_keepalive_probes 9 [probes] (default) >>>> tcp_keepalive_time 7200.000 [seconds] (default) >>>> thread_pool_add_delay 0.000 [seconds] (default) >>>> thread_pool_destroy_delay 1.000 [seconds] (default) >>>> thread_pool_fail_delay 0.200 [seconds] (default) >>>> thread_pool_max 5000 [threads] (default) >>>> thread_pool_min 100 [threads] (default) >>>> thread_pool_reserve 0 [threads] (default) >>>> thread_pool_stack 48k [bytes] (default) >>>> thread_pool_timeout 300.000 [seconds] (default) >>>> thread_pools 2 [pools] (default) >>>> thread_queue_limit 20 (default) >>>> thread_stats_rate 10 [requests] (default) >>>> timeout_idle 5.000 [seconds] (default) >>>> timeout_linger 0.050 [seconds] (default) >>>> vcc_allow_inline_c off [bool] (default) >>>> vcc_err_unref on [bool] (default) >>>> vcc_unsafe_path on [bool] (default) >>>> vcl_cooldown 600.000 [seconds] (default) >>>> vcl_dir /etc/varnish (default) >>>> vmod_dir /usr/lib64/varnish/vmods (default) >>>> vsl_buffer 4k [bytes] (default) >>>> vsl_mask -VCL_trace,-WorkThread,-Hash,-VfpAcct >>>> (default) >>>> vsl_reclen 255b [bytes] (default) >>>> vsl_space 80M [bytes] (default) >>>> vsm_free_cooldown 60.000 [seconds] (default) >>>> vsm_space 1M [bytes] (default) >>>> workspace_backend 64k [bytes] (default) >>>> workspace_client 64k [bytes] (default) >>>> workspace_session 0.50k [bytes] (default) >>>> workspace_thread 2k [bytes] (default) >>>> >>>> >>>> -------- >>>> Regards, >>>> Junaid >>>> >>>> >>>> On Thu, Sep 20, 2018 at 3:37 PM Dridi Boukelmoune >>>> wrote: >>>> >>>>> On Thu, Sep 20, 2018 at 4:14 PM Junaid Mukhtar < >>>>> junaid.mukhtar at gmail.com> wrote: >>>>> > >>>>> > Hi >>>>> > >>>>> > we are in middle of upgrading from varnish 3.0.7 to varnish 4.1.10; >>>>> but unfortunately all of the response times in the performance test are >>>>> indicating an increase of at least 100% >>>>> > >>>>> > We have analyzed the logs and everything but can't get around this; >>>>> the issue is not only for the non-cached pages but it's also impacting the >>>>> cached pages. We are also struggling to analyze it further as well, any >>>>> guidenace as to what we can do in terms of putting debug logging in would >>>>> be helpful. >>>>> > >>>>> > One of the areas we are considering is to output response times, but >>>>> it's proving difficult for me. Any suggestions >>>>> >>>>> Hello, >>>>> >>>>> What is the output of `varnishadm param.show` ? >>>>> >>>>> Dridi >>>>> >>>> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Mon Sep 24 12:14:50 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 24 Sep 2018 14:14:50 +0200 Subject: Varnish VMods for Version 6.1? In-Reply-To: <8A76B2DB46634944BEEBB218DAE22BC265522A698A@exchange-srv2.office.radioteleffh.de> References: <8A76B2DB46634944BEEBB218DAE22BC265522A6968@exchange-srv2.office.radioteleffh.de> <8A76B2DB46634944BEEBB218DAE22BC265522A6971@exchange-srv2.office.radioteleffh.de> <8A76B2DB46634944BEEBB218DAE22BC265522A6975@exchange-srv2.office.radioteleffh.de> <8A76B2DB46634944BEEBB218DAE22BC265522A697B@exchange-srv2.office.radioteleffh.de> <8A76B2DB46634944BEEBB218DAE22BC265522A698A@exchange-srv2.office.radioteleffh.de> Message-ID: On Fri, Sep 21, 2018 at 1:50 PM Winkelmann, Thomas (RADIO TELE FFH - Online) wrote: > > I again did some testing by settings the params to > > -p thread_pools=4 > -p thread_pool_min=2000 > -p thread_pool_max=5000 > -p thread_pool_reserve=95 Please note that min/max/reserve is per pool. > The document says https://varnish-cache.org/docs/trunk/reference/varnishd.html that thread_pool_reserve is a value from 0 to 95. It's working fine on one of our two servers which have less load (300 req/s) than the primaray one (> 800req/s). You really need to read the full description for this one: Default is 0 to auto-tune (currently 5% of thread_pool_min). Minimum is 1 otherwise, maximum is 95% of thread_pool_min. In other words, if you only bump thread_pool_min, you automatically bump thread_pool_reserve. In my previous suggestion my mighty arithmetic powers had failed me. 5% of 2000 is obviously 100, so a thread_pool_reserve of 500 was no longer in the same ballpark. > On the primary one you can see decreasing the client.req in varnishstat two seconds after restarting hitch with H/2 enabled. It jumps from 100req/s to 1700req/s from one second to the next. Like I said, until that deadlock is fixed: That would postpone the deadlock to higher concurrent traffic. Tweaking the worker pools won't fix the problem, and in that regard h2 traffic is even more peaky (per user) than h1 because browsers will typically ask resources immediately (where h1 TCP pooling would prevent too much concurrency) so seeing such peaks is not really surprising. Thanks for sharing your experience nevertheless :) Dridi From olivier.hanesse at gmail.com Mon Sep 24 16:00:59 2018 From: olivier.hanesse at gmail.com (Olivier Hanesse) Date: Mon, 24 Sep 2018 18:00:59 +0200 Subject: Limits in xkey module usage ? Message-ID: Hello, I'm in the process of testing xkey module, and I didn't find any "limits" in the documentation. Is there a limit of setting the same key on a very large number of objects ? Is there a limit on the number of key an object can be linked to ? I think, it is mainly a memory / pointer size issue, are those limits tunable ? Regards Olivier -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlos.abalde at gmail.com Mon Sep 24 16:11:50 2018 From: carlos.abalde at gmail.com (Carlos Abalde) Date: Mon, 24 Sep 2018 18:11:50 +0200 Subject: Limits in xkey module usage ? In-Reply-To: References: Message-ID: <380D9B6F-A99C-4F6E-8D3C-C5A99B6165A0@gmail.com> Hi Oliver, I'll let the creators of the VMOD talk here, but I think the only limit is the space needed in order to keep the in-memory data structure indexing the cached contents. If you link an object to a lot of keys you might hit varnishd limits like max number of headers or the max length of a header. On the other hand, if you link a huge number of objects to the same key, that's not a problem at all, but you might experience some thread contention during invalidations of that key due to the associated locking required by the VMOD. Best, -- Carlos Abalde > On 24 Sep 2018, at 18:00, Olivier Hanesse wrote: > > Hello, > > I'm in the process of testing xkey module, and I didn't find any "limits" in the documentation. > > Is there a limit of setting the same key on a very large number of objects ? > Is there a limit on the number of key an object can be linked to ? > > I think, it is mainly a memory / pointer size issue, are those limits tunable ? > > Regards > > Olivier > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From guillaume at varnish-software.com Mon Sep 24 16:58:23 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 24 Sep 2018 18:58:23 +0200 Subject: CLI start with multiple VCL on Varnish 6 Failed. In-Reply-To: <8f57e251-a4aa-b6a4-71d5-deb9b5d7407c@smile.fr> References: <8f57e251-a4aa-b6a4-71d5-deb9b5d7407c@smile.fr> Message-ID: Confirmed using a Dockerfile (weirdly it worked the first time when I did it manually), looks like we have a bug. I'll look into it -- Guillaume Quintard On Tue, Sep 18, 2018 at 11:36 AM BIGOT Adrien wrote: > Hello, > > Based on this blog article : > https://info.varnish-software.com/blog/one-vcl-per-domain I've been > using multiple VCL with Varnish 5.2 without any problem. > However, I don't manage to get it working with Varnish 6. > > I tried with the latest stable version on packagecloud.io and even with > the latest weekly-build. > Tested on Debian Strecth, Ubuntu 16.04 and Centos 7. > > > [root at ip-172-31-137-19 varnish]# varnishd -I start.cli -F -a :80 -f '' > Debug: Version: varnish-6.0.1 revision > 8d54bec5330c29304979ebf2c425ae14ab80493c > Debug: Platform: > Linux,3.10.0-862.3.2.el7.x86_64,x86_64,-junix,-sdefault,-sdefault,-hcritbit > BEGIN of -I file processing > 200 326 > ----------------------------- > Varnish Cache CLI 1.0 > ----------------------------- > Linux,3.10.0-862.3.2.el7.x86_64,x86_64,-junix,-sdefault,-sdefault,-hcritbit > varnish-6.0.1 revision 8d54bec5330c29304979ebf2c425ae14ab80493c > > Type 'help' for command list. > Type 'quit' to close CLI session. > Type 'start' to launch worker process. > > > vcl.load vcl-api-orig /etc/varnish/api.vcl > 200 14 > VCL compiled. > > > vcl.load vcl-catz-orig /etc/varnish/catz.vcl > 200 14 > VCL compiled. > > > vcl.label label-api vcl-api-orig > 200 0 > > > vcl.label label-catz vcl-catz-orig > 200 0 > > > vcl.load vcl-root-orig /etc/varnish/root.vcl > 200 14 > VCL compiled. > > > vcl.use vcl-root-orig > 200 30 > VCL 'vcl-root-orig' now active > END of -I file processing > CLI result = 400 > > > > In the logs : > > On Centos > sept. 18 08:35:37 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx su[11820]: > pam_unix(su:session): session opened for user root by ec2-user(uid=0) > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: > Version: varnish-6.0.1 revision 8d54bec5330c29304979ebf2c425ae14ab80493c > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: > Platform: > Linux,3.10.0-862.3.2.el7.x86_64,x86_64,-junix,-sdefault,-sdefault,-hcritbit > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: > CLI -I file Rd vcl.load vcl-api-orig /etc/varnish/api.vcl > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: > CLI -I file Wr 200 VCL compiled. > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: > CLI -I file Rd vcl.load vcl-catz-orig /etc/varnish/catz.vcl > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: > CLI -I file Wr 200 VCL compiled. > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: > CLI -I file Rd vcl.label label-api vcl-api-orig > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: > CLI -I file Wr 200 > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: > CLI -I file Rd vcl.label label-catz vcl-catz-orig > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: > CLI -I file Wr 200 > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: > CLI -I file Rd vcl.load vcl-root-orig /etc/varnish/root.vcl > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: > CLI -I file Wr 200 VCL compiled. > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: > CLI -I file Rd vcl.use vcl-root-orig > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxxm varnishd[11846]: > CLI -I file Wr 200 VCL 'vcl-root-orig' now active > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: > Child (11870) Started > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: > Child (11870) not responding to CLI, killed it. > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: > Child (11870) Pushing vcls failed: > CLI communication error > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: > Stopping Child > sept. 18 08:36:35 ip-172-31-137-19.xxxxxxxxxxxxxxxxxxm varnishd[11846]: > Child (11870) died signal=3 > sept. 18 08:36:35 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: > Child cleanup complete > > On Debian : > Sep 18 10:07:01 ip-172-31-137-84 varnishd[6783]: Child (6809) said Child > starts > Sep 18 10:07:01 ip-172-31-137-84 varnishd[6783]: Child cleanup complete > Sep 18 10:07:02 ip-172-31-137-84 varnishd[6816]: Version: varnish-6.0.1 > revision 8d54bec5330c29304979ebf2c425ae14ab80493c > Sep 18 10:07:02 ip-172-31-137-84 varnishd[6816]: Platform: > Linux,4.9.0-8-amd64,x86_64,-junix,-sdefault,-sdefault,-hcritbit > Sep 18 10:07:02 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.load > vcl-api-orig /etc/varnish/api.vcl > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 VCL > compiled. > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.load > vcl-catz-orig /etc/varnish/catz.vcl > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 VCL > compiled. > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd > vcl.label label-api vcl-api-orig > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd > vcl.label label-catz vcl-catz-orig > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.load > vcl-root-orig /etc/varnish/root.vcl > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 VCL > compiled. > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.use > vcl-root-orig > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 VCL > 'vcl-root-orig' now active > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: Child (6842) Started > Sep 18 10:08:03 ip-172-31-137-84 varnishd[6816]: Child (6842) not > responding to CLI, killed it. > Sep 18 10:08:03 ip-172-31-137-84 varnishd[6816]: Child (6842) Pushing > vcls failed: > CLI communication > error (hdr) > Sep 18 10:08:03 ip-172-31-137-84 varnishd[6816]: Stopping Child > Sep 18 10:08:04 ip-172-31-137-84 varnishd[6816]: Child (6842) died signal=3 > > > Did anyone manage to get it working with varnish 6 ? > > Thank you. > > Best regards, > > -- > > Bigot Adrien > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adrien.bigot at smile.fr Mon Sep 24 17:37:28 2018 From: adrien.bigot at smile.fr (BIGOT Adrien) Date: Mon, 24 Sep 2018 19:37:28 +0200 Subject: CLI start with multiple VCL on Varnish 6 Failed. In-Reply-To: References: <8f57e251-a4aa-b6a4-71d5-deb9b5d7407c@smile.fr> Message-ID: <109bd2d8-500b-5c4c-51e8-bc86f38c093e@smile.fr> Hi, Yesterday, I also tried manualy with varnish 6.1 and the first time I believed it was working. But after 1 or 2 minute (I think to a timeout) I had the same error : "Child not responding to CLI, killed it." Thanks Guillaume for your help ! Regards, -- Adrien Bigot Le 24/09/2018 ? 18:58, Guillaume Quintard a ?crit?: > Confirmed using a Dockerfile (weirdly it worked the first time when I > did it manually), looks like we have a bug. I'll look into it > -- > Guillaume Quintard > > > On Tue, Sep 18, 2018 at 11:36 AM BIGOT Adrien > wrote: > > Hello, > > Based on this blog article : > https://info.varnish-software.com/blog/one-vcl-per-domain I've been > using multiple VCL with Varnish 5.2 without any problem. > However, I don't manage to get it working with Varnish 6. > > I tried with the latest stable version on packagecloud.io > and even with > the latest weekly-build. > Tested on Debian Strecth, Ubuntu 16.04 and Centos 7. > > > [root at ip-172-31-137-19 varnish]# varnishd -I start.cli -F -a :80 -f '' > Debug: Version: varnish-6.0.1 revision > 8d54bec5330c29304979ebf2c425ae14ab80493c > Debug: Platform: > Linux,3.10.0-862.3.2.el7.x86_64,x86_64,-junix,-sdefault,-sdefault,-hcritbit > BEGIN of -I file processing > 200 326 > ----------------------------- > Varnish Cache CLI 1.0 > ----------------------------- > Linux,3.10.0-862.3.2.el7.x86_64,x86_64,-junix,-sdefault,-sdefault,-hcritbit > varnish-6.0.1 revision 8d54bec5330c29304979ebf2c425ae14ab80493c > > Type 'help' for command list. > Type 'quit' to close CLI session. > Type 'start' to launch worker process. > > ?> vcl.load vcl-api-orig /etc/varnish/api.vcl > 200 14 > VCL compiled. > > ?> vcl.load vcl-catz-orig /etc/varnish/catz.vcl > 200 14 > VCL compiled. > > ?> vcl.label label-api vcl-api-orig > 200 0 > > ?> vcl.label label-catz vcl-catz-orig > 200 0 > > ?> vcl.load vcl-root-orig /etc/varnish/root.vcl > 200 14 > VCL compiled. > > ?> vcl.use vcl-root-orig > 200 30 > VCL 'vcl-root-orig' now active > END of -I file processing > CLI result = 400 > > > > In the logs : > > On Centos > sept. 18 08:35:37 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx su[11820]: > pam_unix(su:session): session opened for user root by ec2-user(uid=0) > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx > varnishd[11846]: > Version: varnish-6.0.1 revision > 8d54bec5330c29304979ebf2c425ae14ab80493c > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx > varnishd[11846]: > Platform: > Linux,3.10.0-862.3.2.el7.x86_64,x86_64,-junix,-sdefault,-sdefault,-hcritbit > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx > varnishd[11846]: > CLI -I file Rd vcl.load vcl-api-orig /etc/varnish/api.vcl > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx > varnishd[11846]: > CLI -I file Wr 200 VCL compiled. > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx > varnishd[11846]: > CLI -I file Rd vcl.load vcl-catz-orig /etc/varnish/catz.vcl > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx > varnishd[11846]: > CLI -I file Wr 200 VCL compiled. > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx > varnishd[11846]: > CLI -I file Rd vcl.label label-api vcl-api-orig > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx > varnishd[11846]: > CLI -I file Wr 200 > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx > varnishd[11846]: > CLI -I file Rd vcl.label label-catz vcl-catz-orig > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx > varnishd[11846]: > CLI -I file Wr 200 > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx > varnishd[11846]: > CLI -I file Rd vcl.load vcl-root-orig /etc/varnish/root.vcl > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx > varnishd[11846]: > CLI -I file Wr 200 VCL compiled. > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx > varnishd[11846]: > CLI -I file Rd vcl.use vcl-root-orig > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxxm > varnishd[11846]: > CLI -I file Wr 200 VCL 'vcl-root-orig' now active > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx > varnishd[11846]: > Child (11870) Started > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx > varnishd[11846]: > Child (11870) not responding to CLI, killed it. > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx > varnishd[11846]: > Child (11870) Pushing vcls failed: > CLI communication error > sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx > varnishd[11846]: > Stopping Child > sept. 18 08:36:35 ip-172-31-137-19.xxxxxxxxxxxxxxxxxxm > varnishd[11846]: > Child (11870) died signal=3 > sept. 18 08:36:35 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx > varnishd[11846]: > Child cleanup complete > > On Debian : > Sep 18 10:07:01 ip-172-31-137-84 varnishd[6783]: Child (6809) said > Child > starts > Sep 18 10:07:01 ip-172-31-137-84 varnishd[6783]: Child cleanup > complete > Sep 18 10:07:02 ip-172-31-137-84 varnishd[6816]: Version: > varnish-6.0.1 > revision 8d54bec5330c29304979ebf2c425ae14ab80493c > Sep 18 10:07:02 ip-172-31-137-84 varnishd[6816]: Platform: > Linux,4.9.0-8-amd64,x86_64,-junix,-sdefault,-sdefault,-hcritbit > Sep 18 10:07:02 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd > vcl.load > vcl-api-orig /etc/varnish/api.vcl > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr > 200 VCL > compiled. > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd > vcl.load > vcl-catz-orig /etc/varnish/catz.vcl > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr > 200 VCL > compiled. > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd > vcl.label label-api vcl-api-orig > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd > vcl.label label-catz vcl-catz-orig > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd > vcl.load > vcl-root-orig /etc/varnish/root.vcl > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr > 200 VCL > compiled. > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd > vcl.use > vcl-root-orig > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr > 200 VCL > 'vcl-root-orig' now active > Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: Child (6842) Started > Sep 18 10:08:03 ip-172-31-137-84 varnishd[6816]: Child (6842) not > responding to CLI, killed it. > Sep 18 10:08:03 ip-172-31-137-84 varnishd[6816]: Child (6842) Pushing > vcls failed: > ????????????????????????????????????????????????? CLI communication > error (hdr) > Sep 18 10:08:03 ip-172-31-137-84 varnishd[6816]: Stopping Child > Sep 18 10:08:04 ip-172-31-137-84 varnishd[6816]: Child (6842) died > signal=3 > > > Did anyone manage to get it working with varnish 6 ? > > Thank you. > > Best regards, > > -- > > Bigot Adrien > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Mon Sep 24 20:57:18 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 24 Sep 2018 22:57:18 +0200 Subject: CLI start with multiple VCL on Varnish 6 Failed. In-Reply-To: <109bd2d8-500b-5c4c-51e8-bc86f38c093e@smile.fr> References: <8f57e251-a4aa-b6a4-71d5-deb9b5d7407c@smile.fr> <109bd2d8-500b-5c4c-51e8-bc86f38c093e@smile.fr> Message-ID: test streamlined, commit bisected, issue created: https://github.com/varnishcache/varnish-cache/issues/2782 I'll try to see what I can find once I have more time, unless someone looks into it first -- Guillaume Quintard On Mon, Sep 24, 2018 at 7:37 PM BIGOT Adrien wrote: > Hi, > > Yesterday, I also tried manualy with varnish 6.1 and the first time I > believed it was working. But after 1 or 2 minute (I think to a timeout) I > had the same error : "Child not responding to CLI, killed it." > > Thanks Guillaume for your help ! > > Regards, > > -- > > Adrien Bigot > > Le 24/09/2018 ? 18:58, Guillaume Quintard a ?crit : > > Confirmed using a Dockerfile (weirdly it worked the first time when I did > it manually), looks like we have a bug. I'll look into it > -- > Guillaume Quintard > > > On Tue, Sep 18, 2018 at 11:36 AM BIGOT Adrien > wrote: > >> Hello, >> >> Based on this blog article : >> https://info.varnish-software.com/blog/one-vcl-per-domain I've been >> using multiple VCL with Varnish 5.2 without any problem. >> However, I don't manage to get it working with Varnish 6. >> >> I tried with the latest stable version on packagecloud.io and even with >> the latest weekly-build. >> Tested on Debian Strecth, Ubuntu 16.04 and Centos 7. >> >> >> [root at ip-172-31-137-19 varnish]# varnishd -I start.cli -F -a :80 -f '' >> Debug: Version: varnish-6.0.1 revision >> 8d54bec5330c29304979ebf2c425ae14ab80493c >> Debug: Platform: >> >> Linux,3.10.0-862.3.2.el7.x86_64,x86_64,-junix,-sdefault,-sdefault,-hcritbit >> BEGIN of -I file processing >> 200 326 >> ----------------------------- >> Varnish Cache CLI 1.0 >> ----------------------------- >> >> Linux,3.10.0-862.3.2.el7.x86_64,x86_64,-junix,-sdefault,-sdefault,-hcritbit >> varnish-6.0.1 revision 8d54bec5330c29304979ebf2c425ae14ab80493c >> >> Type 'help' for command list. >> Type 'quit' to close CLI session. >> Type 'start' to launch worker process. >> >> > vcl.load vcl-api-orig /etc/varnish/api.vcl >> 200 14 >> VCL compiled. >> >> > vcl.load vcl-catz-orig /etc/varnish/catz.vcl >> 200 14 >> VCL compiled. >> >> > vcl.label label-api vcl-api-orig >> 200 0 >> >> > vcl.label label-catz vcl-catz-orig >> 200 0 >> >> > vcl.load vcl-root-orig /etc/varnish/root.vcl >> 200 14 >> VCL compiled. >> >> > vcl.use vcl-root-orig >> 200 30 >> VCL 'vcl-root-orig' now active >> END of -I file processing >> CLI result = 400 >> >> >> >> In the logs : >> >> On Centos >> sept. 18 08:35:37 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx su[11820]: >> pam_unix(su:session): session opened for user root by ec2-user(uid=0) >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: >> Version: varnish-6.0.1 revision 8d54bec5330c29304979ebf2c425ae14ab80493c >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: >> Platform: >> >> Linux,3.10.0-862.3.2.el7.x86_64,x86_64,-junix,-sdefault,-sdefault,-hcritbit >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: >> CLI -I file Rd vcl.load vcl-api-orig /etc/varnish/api.vcl >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: >> CLI -I file Wr 200 VCL compiled. >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: >> CLI -I file Rd vcl.load vcl-catz-orig /etc/varnish/catz.vcl >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: >> CLI -I file Wr 200 VCL compiled. >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: >> CLI -I file Rd vcl.label label-api vcl-api-orig >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: >> CLI -I file Wr 200 >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: >> CLI -I file Rd vcl.label label-catz vcl-catz-orig >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: >> CLI -I file Wr 200 >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: >> CLI -I file Rd vcl.load vcl-root-orig /etc/varnish/root.vcl >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: >> CLI -I file Wr 200 VCL compiled. >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: >> CLI -I file Rd vcl.use vcl-root-orig >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxxm varnishd[11846]: >> CLI -I file Wr 200 VCL 'vcl-root-orig' now active >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: >> Child (11870) Started >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: >> Child (11870) not responding to CLI, killed it. >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: >> Child (11870) Pushing vcls failed: >> CLI communication error >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: >> Stopping Child >> sept. 18 08:36:35 ip-172-31-137-19.xxxxxxxxxxxxxxxxxxm varnishd[11846]: >> Child (11870) died signal=3 >> sept. 18 08:36:35 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx varnishd[11846]: >> Child cleanup complete >> >> On Debian : >> Sep 18 10:07:01 ip-172-31-137-84 varnishd[6783]: Child (6809) said Child >> starts >> Sep 18 10:07:01 ip-172-31-137-84 varnishd[6783]: Child cleanup complete >> Sep 18 10:07:02 ip-172-31-137-84 varnishd[6816]: Version: varnish-6.0.1 >> revision 8d54bec5330c29304979ebf2c425ae14ab80493c >> Sep 18 10:07:02 ip-172-31-137-84 varnishd[6816]: Platform: >> Linux,4.9.0-8-amd64,x86_64,-junix,-sdefault,-sdefault,-hcritbit >> Sep 18 10:07:02 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.load >> vcl-api-orig /etc/varnish/api.vcl >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 VCL >> compiled. >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.load >> vcl-catz-orig /etc/varnish/catz.vcl >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 VCL >> compiled. >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd >> vcl.label label-api vcl-api-orig >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd >> vcl.label label-catz vcl-catz-orig >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.load >> vcl-root-orig /etc/varnish/root.vcl >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 VCL >> compiled. >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd vcl.use >> vcl-root-orig >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Wr 200 VCL >> 'vcl-root-orig' now active >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: Child (6842) Started >> Sep 18 10:08:03 ip-172-31-137-84 varnishd[6816]: Child (6842) not >> responding to CLI, killed it. >> Sep 18 10:08:03 ip-172-31-137-84 varnishd[6816]: Child (6842) Pushing >> vcls failed: >> CLI communication >> error (hdr) >> Sep 18 10:08:03 ip-172-31-137-84 varnishd[6816]: Stopping Child >> Sep 18 10:08:04 ip-172-31-137-84 varnishd[6816]: Child (6842) died >> signal=3 >> >> >> Did anyone manage to get it working with varnish 6 ? >> >> Thank you. >> >> Best regards, >> >> -- >> >> Bigot Adrien >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adrien.bigot at smile.fr Mon Sep 24 21:33:09 2018 From: adrien.bigot at smile.fr (BIGOT Adrien) Date: Mon, 24 Sep 2018 23:33:09 +0200 Subject: CLI start with multiple VCL on Varnish 6 Failed. In-Reply-To: References: <8f57e251-a4aa-b6a4-71d5-deb9b5d7407c@smile.fr> <109bd2d8-500b-5c4c-51e8-bc86f38c093e@smile.fr> Message-ID: <06380396-694b-85ae-b008-68361314be2f@smile.fr> Thank you very much Guillaume. For now, I'll postpone our migration to Varnish 6. -- Adrien Bigot Le 24/09/2018 ? 22:57, Guillaume Quintard a ?crit?: > test streamlined, commit bisected, issue created: > https://github.com/varnishcache/varnish-cache/issues/2782 > > I'll try to see what I can find once I have more time, unless someone > looks into it first > > -- > Guillaume Quintard > > > On Mon, Sep 24, 2018 at 7:37 PM BIGOT Adrien > wrote: > > Hi, > > Yesterday, I also tried manualy with varnish 6.1 and the first > time I believed it was working. But after 1 or 2 minute (I think > to a timeout) I had the same error : "Child not responding to CLI, > killed it." > > Thanks Guillaume for your help ! > > Regards, > > -- > > Adrien Bigot > > Le 24/09/2018 ? 18:58, Guillaume Quintard a ?crit?: >> Confirmed using a Dockerfile (weirdly it worked the first time >> when I did it manually), looks like we have a bug. I'll look into it >> -- >> Guillaume Quintard >> >> >> On Tue, Sep 18, 2018 at 11:36 AM BIGOT Adrien >> > wrote: >> >> Hello, >> >> Based on this blog article : >> https://info.varnish-software.com/blog/one-vcl-per-domain >> I've been >> using multiple VCL with Varnish 5.2 without any problem. >> However, I don't manage to get it working with Varnish 6. >> >> I tried with the latest stable version on packagecloud.io >> and even with >> the latest weekly-build. >> Tested on Debian Strecth, Ubuntu 16.04 and Centos 7. >> >> >> [root at ip-172-31-137-19 varnish]# varnishd -I start.cli -F -a >> :80 -f '' >> Debug: Version: varnish-6.0.1 revision >> 8d54bec5330c29304979ebf2c425ae14ab80493c >> Debug: Platform: >> Linux,3.10.0-862.3.2.el7.x86_64,x86_64,-junix,-sdefault,-sdefault,-hcritbit >> BEGIN of -I file processing >> 200 326 >> ----------------------------- >> Varnish Cache CLI 1.0 >> ----------------------------- >> Linux,3.10.0-862.3.2.el7.x86_64,x86_64,-junix,-sdefault,-sdefault,-hcritbit >> varnish-6.0.1 revision 8d54bec5330c29304979ebf2c425ae14ab80493c >> >> Type 'help' for command list. >> Type 'quit' to close CLI session. >> Type 'start' to launch worker process. >> >> ?> vcl.load vcl-api-orig /etc/varnish/api.vcl >> 200 14 >> VCL compiled. >> >> ?> vcl.load vcl-catz-orig /etc/varnish/catz.vcl >> 200 14 >> VCL compiled. >> >> ?> vcl.label label-api vcl-api-orig >> 200 0 >> >> ?> vcl.label label-catz vcl-catz-orig >> 200 0 >> >> ?> vcl.load vcl-root-orig /etc/varnish/root.vcl >> 200 14 >> VCL compiled. >> >> ?> vcl.use vcl-root-orig >> 200 30 >> VCL 'vcl-root-orig' now active >> END of -I file processing >> CLI result = 400 >> >> >> >> In the logs : >> >> On Centos >> sept. 18 08:35:37 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx su[11820]: >> pam_unix(su:session): session opened for user root by >> ec2-user(uid=0) >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx >> varnishd[11846]: >> Version: varnish-6.0.1 revision >> 8d54bec5330c29304979ebf2c425ae14ab80493c >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx >> varnishd[11846]: >> Platform: >> Linux,3.10.0-862.3.2.el7.x86_64,x86_64,-junix,-sdefault,-sdefault,-hcritbit >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx >> varnishd[11846]: >> CLI -I file Rd vcl.load vcl-api-orig /etc/varnish/api.vcl >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx >> varnishd[11846]: >> CLI -I file Wr 200 VCL compiled. >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx >> varnishd[11846]: >> CLI -I file Rd vcl.load vcl-catz-orig /etc/varnish/catz.vcl >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx >> varnishd[11846]: >> CLI -I file Wr 200 VCL compiled. >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx >> varnishd[11846]: >> CLI -I file Rd vcl.label label-api vcl-api-orig >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx >> varnishd[11846]: >> CLI -I file Wr 200 >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx >> varnishd[11846]: >> CLI -I file Rd vcl.label label-catz vcl-catz-orig >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx >> varnishd[11846]: >> CLI -I file Wr 200 >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx >> varnishd[11846]: >> CLI -I file Rd vcl.load vcl-root-orig /etc/varnish/root.vcl >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx >> varnishd[11846]: >> CLI -I file Wr 200 VCL compiled. >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx >> varnishd[11846]: >> CLI -I file Rd vcl.use vcl-root-orig >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxxm >> varnishd[11846]: >> CLI -I file Wr 200 VCL 'vcl-root-orig' now active >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx >> varnishd[11846]: >> Child (11870) Started >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx >> varnishd[11846]: >> Child (11870) not responding to CLI, killed it. >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx >> varnishd[11846]: >> Child (11870) Pushing vcls failed: >> CLI communication error >> sept. 18 08:36:34 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx >> varnishd[11846]: >> Stopping Child >> sept. 18 08:36:35 ip-172-31-137-19.xxxxxxxxxxxxxxxxxxm >> varnishd[11846]: >> Child (11870) died signal=3 >> sept. 18 08:36:35 ip-172-31-137-19.xxxxxxxxxxxxxxxxxx >> varnishd[11846]: >> Child cleanup complete >> >> On Debian : >> Sep 18 10:07:01 ip-172-31-137-84 varnishd[6783]: Child (6809) >> said Child >> starts >> Sep 18 10:07:01 ip-172-31-137-84 varnishd[6783]: Child >> cleanup complete >> Sep 18 10:07:02 ip-172-31-137-84 varnishd[6816]: Version: >> varnish-6.0.1 >> revision 8d54bec5330c29304979ebf2c425ae14ab80493c >> Sep 18 10:07:02 ip-172-31-137-84 varnishd[6816]: Platform: >> Linux,4.9.0-8-amd64,x86_64,-junix,-sdefault,-sdefault,-hcritbit >> Sep 18 10:07:02 ip-172-31-137-84 varnishd[6816]: CLI -I file >> Rd vcl.load >> vcl-api-orig /etc/varnish/api.vcl >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file >> Wr 200 VCL >> compiled. >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file >> Rd vcl.load >> vcl-catz-orig /etc/varnish/catz.vcl >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file >> Wr 200 VCL >> compiled. >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd >> vcl.label label-api vcl-api-orig >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file >> Wr 200 >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file Rd >> vcl.label label-catz vcl-catz-orig >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file >> Wr 200 >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file >> Rd vcl.load >> vcl-root-orig /etc/varnish/root.vcl >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file >> Wr 200 VCL >> compiled. >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file >> Rd vcl.use >> vcl-root-orig >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: CLI -I file >> Wr 200 VCL >> 'vcl-root-orig' now active >> Sep 18 10:07:03 ip-172-31-137-84 varnishd[6816]: Child (6842) >> Started >> Sep 18 10:08:03 ip-172-31-137-84 varnishd[6816]: Child (6842) >> not >> responding to CLI, killed it. >> Sep 18 10:08:03 ip-172-31-137-84 varnishd[6816]: Child (6842) >> Pushing >> vcls failed: >> ????????????????????????????????????????????????? CLI >> communication >> error (hdr) >> Sep 18 10:08:03 ip-172-31-137-84 varnishd[6816]: Stopping Child >> Sep 18 10:08:04 ip-172-31-137-84 varnishd[6816]: Child (6842) >> died signal=3 >> >> >> Did anyone manage to get it working with varnish 6 ? >> >> Thank you. >> >> Best regards, >> >> -- >> >> Bigot Adrien >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Tue Sep 25 09:15:23 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 25 Sep 2018 11:15:23 +0200 Subject: Assert error in ban_lurker_getfirst() In-Reply-To: References: Message-ID: Found this unread thread in my inbox, I will reply here for completeness. > Any idea about what could have happened? Reported on github and fixed in 6.0.1 and 6.1.0, should be fixed as part of the next 4.1 release: https://github.com/varnishcache/varnish-cache/issues/2681 Thanks for reporting! From olivier.hanesse at gmail.com Tue Sep 25 09:27:43 2018 From: olivier.hanesse at gmail.com (Olivier Hanesse) Date: Tue, 25 Sep 2018 11:27:43 +0200 Subject: Limits in xkey module usage ? In-Reply-To: <380D9B6F-A99C-4F6E-8D3C-C5A99B6165A0@gmail.com> References: <380D9B6F-A99C-4F6E-8D3C-C5A99B6165A0@gmail.com> Message-ID: Hello Carlos, Thanks for your answer. About thread contention during invalidation, can you be more specific ? At what scale can we encounter such contentions ? Hundreds of objects linked to a key ? or Thousand ? ten thousand ? Regards Olivier Le lun. 24 sept. 2018 ? 18:11, Carlos Abalde a ?crit : > Hi Oliver, > > I'll let the creators of the VMOD talk here, but I think the only limit is > the space needed in order to keep the in-memory data structure indexing the > cached contents. > > If you link an object to a lot of keys you might hit varnishd limits like > max number of headers or the max length of a header. On the other hand, if > you link a huge number of objects to the same key, that's not a problem at > all, but you might experience some thread contention during invalidations > of that key due to the associated locking required by the VMOD. > > Best, > > -- > Carlos Abalde > > > On 24 Sep 2018, at 18:00, Olivier Hanesse > wrote: > > > > Hello, > > > > I'm in the process of testing xkey module, and I didn't find any > "limits" in the documentation. > > > > Is there a limit of setting the same key on a very large number of > objects ? > > Is there a limit on the number of key an object can be linked to ? > > > > I think, it is mainly a memory / pointer size issue, are those limits > tunable ? > > > > Regards > > > > Olivier > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlos.abalde at gmail.com Tue Sep 25 10:29:33 2018 From: carlos.abalde at gmail.com (Carlos Abalde) Date: Tue, 25 Sep 2018 12:29:33 +0200 Subject: Limits in xkey module usage ? In-Reply-To: References: <380D9B6F-A99C-4F6E-8D3C-C5A99B6165A0@gmail.com> Message-ID: <575CD6C1-8332-49DA-85BD-6461EB94254C@gmail.com> > On 25 Sep 2018, at 11:27, Olivier Hanesse wrote: > > Hello Carlos, > > Thanks for your answer. About thread contention during invalidation, can you be more specific ? > At what scale can we encounter such contentions ? > Hundreds of objects linked to a key ? or Thousand ? ten thousand ? Hi Olivier, Basically I'm talking about thread contention generated by this loop [1]: when a key is purged, if that key is linked to a huge amount of objects, the lock is going to be held during a noticeable amount of time. That will block other threads executing callbacks [2] in order to to add / remove object to / from the xkey state. What would be too many objects linked to a single key? To be honest, I don't know. For an answer we'll need to invoke the VMOD authors :) [1] https://github.com/varnish/varnish-modules/blob/0.15.0/src/vmod_xkey.c#L531-L579 [2] https://github.com/varnish/varnish-modules/blob/0.15.0/src/vmod_xkey.c#L423-L465 Best, -- Carlos Abalde -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Tue Sep 25 15:03:04 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 25 Sep 2018 17:03:04 +0200 Subject: Limits in xkey module usage ? In-Reply-To: <575CD6C1-8332-49DA-85BD-6461EB94254C@gmail.com> References: <380D9B6F-A99C-4F6E-8D3C-C5A99B6165A0@gmail.com> <575CD6C1-8332-49DA-85BD-6461EB94254C@gmail.com> Message-ID: > What would be too many objects linked to a single key? To be honest, I don't know. For an answer we'll need to invoke the VMOD authors :) Keys are just tags, so you could really tag responses in many ways. For example, on a server delivering contents for multiple hosts, you could use the normalized host name as a key and decide that any release on the backend side should invalidate the cache of the associated host. Another common example is tagging responses by type, for example add an "article" key. Once you change your articles template, you have a way to soft-purge all the cached articles. No, there are no implicit limits in vmod-xkey. Yes, there are related limits as pointed out by Carlos, namely number and size of response headers. vmod-xkey allows criteria-based invalidation like bans (but not as versatile) at a purge-like scale, and besides the additional memory footprint it can exhibit contention when the set of purged keys invalidate too many objects. I'm afraid I don't have any numbers to share regarding when things go out of hand. Varnish has a "cutoff" parameter for bans but vmod-xkey has nothing of the sort to keep everything under tight control. Dridi PS. not the vmod-xkey author From guillaume at varnish-software.com Tue Sep 25 16:17:37 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Tue, 25 Sep 2018 18:17:37 +0200 Subject: Varnish 4 Performance Issues In-Reply-To: References: Message-ID: without the vcl (and/or logs, but I'm assuming those are even more confidential), that's a tough cookie to debug -- Guillaume Quintard On Mon, Sep 24, 2018 at 10:35 AM Junaid Mukhtar wrote: > Hi > > Apologies but I am not allowed to share the VCL. If there's any other > thing like stats etc that might be helpful then do let me know > > -------- > Regards, > Junaid > > > On Fri, Sep 21, 2018 at 4:15 PM Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> would you be able to share your vcl? >> -- >> Guillaume Quintard >> >> >> On Fri, Sep 21, 2018 at 4:54 PM Junaid Mukhtar >> wrote: >> >>> Any help guys; this is really getting to us..... >>> >>> -------- >>> Regards, >>> Junaid >>> >>> >>> On Thu, Sep 20, 2018 at 3:57 PM Junaid Mukhtar >>> wrote: >>> >>>> >>>> This is what we get in varnishhist >>>> >>>> 280_ >>>> >>>> >>>> >>>> | >>>> 230_ | >>>> | >>>> | >>>> | >>>> | >>>> 180_ | >>>> || >>>> || >>>> ||| >>>> ||| >>>> 130_ ||| >>>> ||| >>>> |||||| >>>> |||||| >>>> |||||| >>>> 80_ |||||| >>>> >>>> |||||| # >>>> >>>> |||||| ### >>>> >>>> ||||||| # ### >>>> >>>> ||||||||| ##### # >>>> 30_ >>>> ||||||||| ######## >>>> >>>> ||||||||| >>>> ######### >>>> >>>> |||||||||| >>>> ########### >>>> >>>> |||||||||||| ## # >>>> ############# # >>>> >>>> +---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+--------------------- >>>> |1e-6 |1e-5 |1e-4 >>>> |1e-3 |1e-2 |1e-1 >>>> |1e0 |1e1 |1e2 >>>> -------- >>>> Regards, >>>> Junaid >>>> >>>> >>>> On Thu, Sep 20, 2018 at 3:54 PM Junaid Mukhtar < >>>> junaid.mukhtar at gmail.com> wrote: >>>> >>>>> Below is the output; all of them are default except i was trying to up >>>>> the threads_pool to 4 but didn't feel any imporvement in performance >>>>> degradation >>>>> >>>>> accept_filter off [bool] (default) >>>>> acceptor_sleep_decay 0.9 (default) >>>>> acceptor_sleep_incr 0.000 [seconds] (default) >>>>> acceptor_sleep_max 0.050 [seconds] (default) >>>>> auto_restart on [bool] (default) >>>>> backend_idle_timeout 60.000 [seconds] (default) >>>>> ban_dups on [bool] (default) >>>>> ban_lurker_age 60.000 [seconds] (default) >>>>> ban_lurker_batch 1000 (default) >>>>> ban_lurker_sleep 0.010 [seconds] (default) >>>>> between_bytes_timeout 60.000 [seconds] (default) >>>>> cc_command "exec gcc -std=gnu99 -O2 -g >>>>> -Wp,-D_FORTIFY_SOURCE=0 -Wall -Werror -pthread -fpic -shared -Wl,-x -o %o >>>>> %s" (default) >>>>> cli_buffer 8k [bytes] (default) >>>>> cli_limit 48k [bytes] (default) >>>>> cli_timeout 60.000 [seconds] (default) >>>>> clock_skew 10 [seconds] (default) >>>>> clock_step 1.000 [seconds] (default) >>>>> connect_timeout 3.500 [seconds] (default) >>>>> critbit_cooloff 180.000 [seconds] (default) >>>>> debug none (default) >>>>> default_grace 10.000 [seconds] (default) >>>>> default_keep 0.000 [seconds] (default) >>>>> default_ttl 120.000 [seconds] (default) >>>>> feature none (default) >>>>> fetch_chunksize 16k [bytes] (default) >>>>> fetch_maxchunksize 0.25G [bytes] (default) >>>>> first_byte_timeout 60.000 [seconds] (default) >>>>> gzip_buffer 32k [bytes] (default) >>>>> gzip_level 6 (default) >>>>> gzip_memlevel 8 (default) >>>>> http_gzip_support on [bool] (default) >>>>> http_max_hdr 64 [header lines] (default) >>>>> http_range_support on [bool] (default) >>>>> http_req_hdr_len 8k [bytes] (default) >>>>> http_req_size 32k [bytes] (default) >>>>> http_resp_hdr_len 8k [bytes] (default) >>>>> http_resp_size 32k [bytes] (default) >>>>> idle_send_timeout 60.000 [seconds] (default) >>>>> listen_depth 1024 [connections] (default) >>>>> lru_interval 2.000 [seconds] (default) >>>>> max_esi_depth 5 [levels] (default) >>>>> max_restarts 4 [restarts] (default) >>>>> max_retries 4 [retries] (default) >>>>> nuke_limit 50 [allocations] (default) >>>>> pcre_match_limit 10000 (default) >>>>> pcre_match_limit_recursion 20 (default) >>>>> ping_interval 3 [seconds] (default) >>>>> pipe_timeout 60.000 [seconds] (default) >>>>> pool_req 10,100,10 (default) >>>>> pool_sess 10,100,10 (default) >>>>> pool_vbo 10,100,10 (default) >>>>> prefer_ipv6 off [bool] (default) >>>>> rush_exponent 3 [requests per request] (default) >>>>> send_timeout 600.000 [seconds] (default) >>>>> session_max 100000 [sessions] (default) >>>>> shm_reclen 255b [bytes] (default) >>>>> shortlived 10.000 [seconds] (default) >>>>> sigsegv_handler on [bool] (default) >>>>> syslog_cli_traffic on [bool] (default) >>>>> tcp_fastopen off [bool] (default) >>>>> tcp_keepalive_intvl 75.000 [seconds] (default) >>>>> tcp_keepalive_probes 9 [probes] (default) >>>>> tcp_keepalive_time 7200.000 [seconds] (default) >>>>> thread_pool_add_delay 0.000 [seconds] (default) >>>>> thread_pool_destroy_delay 1.000 [seconds] (default) >>>>> thread_pool_fail_delay 0.200 [seconds] (default) >>>>> thread_pool_max 5000 [threads] (default) >>>>> thread_pool_min 100 [threads] (default) >>>>> thread_pool_reserve 0 [threads] (default) >>>>> thread_pool_stack 48k [bytes] (default) >>>>> thread_pool_timeout 300.000 [seconds] (default) >>>>> thread_pools 2 [pools] (default) >>>>> thread_queue_limit 20 (default) >>>>> thread_stats_rate 10 [requests] (default) >>>>> timeout_idle 5.000 [seconds] (default) >>>>> timeout_linger 0.050 [seconds] (default) >>>>> vcc_allow_inline_c off [bool] (default) >>>>> vcc_err_unref on [bool] (default) >>>>> vcc_unsafe_path on [bool] (default) >>>>> vcl_cooldown 600.000 [seconds] (default) >>>>> vcl_dir /etc/varnish (default) >>>>> vmod_dir /usr/lib64/varnish/vmods (default) >>>>> vsl_buffer 4k [bytes] (default) >>>>> vsl_mask -VCL_trace,-WorkThread,-Hash,-VfpAcct >>>>> (default) >>>>> vsl_reclen 255b [bytes] (default) >>>>> vsl_space 80M [bytes] (default) >>>>> vsm_free_cooldown 60.000 [seconds] (default) >>>>> vsm_space 1M [bytes] (default) >>>>> workspace_backend 64k [bytes] (default) >>>>> workspace_client 64k [bytes] (default) >>>>> workspace_session 0.50k [bytes] (default) >>>>> workspace_thread 2k [bytes] (default) >>>>> >>>>> >>>>> -------- >>>>> Regards, >>>>> Junaid >>>>> >>>>> >>>>> On Thu, Sep 20, 2018 at 3:37 PM Dridi Boukelmoune >>>>> wrote: >>>>> >>>>>> On Thu, Sep 20, 2018 at 4:14 PM Junaid Mukhtar < >>>>>> junaid.mukhtar at gmail.com> wrote: >>>>>> > >>>>>> > Hi >>>>>> > >>>>>> > we are in middle of upgrading from varnish 3.0.7 to varnish 4.1.10; >>>>>> but unfortunately all of the response times in the performance test are >>>>>> indicating an increase of at least 100% >>>>>> > >>>>>> > We have analyzed the logs and everything but can't get around this; >>>>>> the issue is not only for the non-cached pages but it's also impacting the >>>>>> cached pages. We are also struggling to analyze it further as well, any >>>>>> guidenace as to what we can do in terms of putting debug logging in would >>>>>> be helpful. >>>>>> > >>>>>> > One of the areas we are considering is to output response times, >>>>>> but it's proving difficult for me. Any suggestions >>>>>> >>>>>> Hello, >>>>>> >>>>>> What is the output of `varnishadm param.show` ? >>>>>> >>>>>> Dridi >>>>>> >>>>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From twbecker at gmail.com Wed Sep 26 00:41:55 2018 From: twbecker at gmail.com (Tommy Becker) Date: Tue, 25 Sep 2018 20:41:55 -0400 Subject: Varnish swallowing 4xx responses from POSTs References: Message-ID: <6DBBA006-52E8-4927-8E64-663B9678A514@gmail.com> We have an application that we front with Varnish 4.0.5. Recently, after an application upgrade in which we migrated from Jetty 9.2 to 9.4, we began noticing a lot of 503s being returned from Varnish on POST requests. We have an endpoint that takes a payload of a potentially large amount of JSON, and validates it as it?s being read. What we have discovered is that if there is a problem with the content, we correctly return a 400 Bad Request from Jetty. Notably, this can happen before the entire content is received. When this happens, Varnish continues to send the remainder of data, despite having already seen the response. Now after our upgrade, Jetty's behavior is to send a TCP RST when this happens (since the data is unwanted by the application). Unfortunately, Varnish interprets the RST as a backend error, and goes to vcl_backend_error, having never sent the original response returned from Jetty to the client. So instead of seeing a 400 Bad Request with a helpful message, they simply get 503 Service Unavailable. I found this issue which seems similar: https://github.com/varnishcache/varnish-cache/issues/2332 Can someone help here? Is there anyway to work around this behavior? -------------- next part -------------- An HTML attachment was scrubbed... URL: From junaid.mukhtar at gmail.com Wed Sep 26 08:31:33 2018 From: junaid.mukhtar at gmail.com (Junaid Mukhtar) Date: Wed, 26 Sep 2018 09:31:33 +0100 Subject: Varnish 4 Performance Issues In-Reply-To: References: Message-ID: I totally agree about that. what we have figured out is that during the steady periods of performance tests it run absolutely fine; but as soon as we spike up the traffic (double the normal peak) performance degrades dramatically. -------- Regards, Junaid On Tue, Sep 25, 2018 at 5:17 PM Guillaume Quintard < guillaume at varnish-software.com> wrote: > without the vcl (and/or logs, but I'm assuming those are even more > confidential), that's a tough cookie to debug > -- > Guillaume Quintard > > > On Mon, Sep 24, 2018 at 10:35 AM Junaid Mukhtar > wrote: > >> Hi >> >> Apologies but I am not allowed to share the VCL. If there's any other >> thing like stats etc that might be helpful then do let me know >> >> -------- >> Regards, >> Junaid >> >> >> On Fri, Sep 21, 2018 at 4:15 PM Guillaume Quintard < >> guillaume at varnish-software.com> wrote: >> >>> would you be able to share your vcl? >>> -- >>> Guillaume Quintard >>> >>> >>> On Fri, Sep 21, 2018 at 4:54 PM Junaid Mukhtar >>> wrote: >>> >>>> Any help guys; this is really getting to us..... >>>> >>>> -------- >>>> Regards, >>>> Junaid >>>> >>>> >>>> On Thu, Sep 20, 2018 at 3:57 PM Junaid Mukhtar < >>>> junaid.mukhtar at gmail.com> wrote: >>>> >>>>> >>>>> This is what we get in varnishhist >>>>> >>>>> 280_ >>>>> >>>>> >>>>> >>>>> | >>>>> 230_ | >>>>> | >>>>> | >>>>> | >>>>> | >>>>> 180_ | >>>>> || >>>>> || >>>>> ||| >>>>> ||| >>>>> 130_ ||| >>>>> ||| >>>>> |||||| >>>>> |||||| >>>>> |||||| >>>>> 80_ |||||| >>>>> >>>>> |||||| # >>>>> >>>>> |||||| ### >>>>> >>>>> ||||||| # ### >>>>> >>>>> ||||||||| ##### # >>>>> 30_ >>>>> ||||||||| ######## >>>>> >>>>> ||||||||| >>>>> ######### >>>>> >>>>> |||||||||| >>>>> ########### >>>>> >>>>> |||||||||||| ## # >>>>> ############# # >>>>> >>>>> +---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+--------------------- >>>>> |1e-6 |1e-5 |1e-4 >>>>> |1e-3 |1e-2 |1e-1 >>>>> |1e0 |1e1 |1e2 >>>>> -------- >>>>> Regards, >>>>> Junaid >>>>> >>>>> >>>>> On Thu, Sep 20, 2018 at 3:54 PM Junaid Mukhtar < >>>>> junaid.mukhtar at gmail.com> wrote: >>>>> >>>>>> Below is the output; all of them are default except i was trying to >>>>>> up the threads_pool to 4 but didn't feel any imporvement in performance >>>>>> degradation >>>>>> >>>>>> accept_filter off [bool] (default) >>>>>> acceptor_sleep_decay 0.9 (default) >>>>>> acceptor_sleep_incr 0.000 [seconds] (default) >>>>>> acceptor_sleep_max 0.050 [seconds] (default) >>>>>> auto_restart on [bool] (default) >>>>>> backend_idle_timeout 60.000 [seconds] (default) >>>>>> ban_dups on [bool] (default) >>>>>> ban_lurker_age 60.000 [seconds] (default) >>>>>> ban_lurker_batch 1000 (default) >>>>>> ban_lurker_sleep 0.010 [seconds] (default) >>>>>> between_bytes_timeout 60.000 [seconds] (default) >>>>>> cc_command "exec gcc -std=gnu99 -O2 -g >>>>>> -Wp,-D_FORTIFY_SOURCE=0 -Wall -Werror -pthread -fpic -shared -Wl,-x -o %o >>>>>> %s" (default) >>>>>> cli_buffer 8k [bytes] (default) >>>>>> cli_limit 48k [bytes] (default) >>>>>> cli_timeout 60.000 [seconds] (default) >>>>>> clock_skew 10 [seconds] (default) >>>>>> clock_step 1.000 [seconds] (default) >>>>>> connect_timeout 3.500 [seconds] (default) >>>>>> critbit_cooloff 180.000 [seconds] (default) >>>>>> debug none (default) >>>>>> default_grace 10.000 [seconds] (default) >>>>>> default_keep 0.000 [seconds] (default) >>>>>> default_ttl 120.000 [seconds] (default) >>>>>> feature none (default) >>>>>> fetch_chunksize 16k [bytes] (default) >>>>>> fetch_maxchunksize 0.25G [bytes] (default) >>>>>> first_byte_timeout 60.000 [seconds] (default) >>>>>> gzip_buffer 32k [bytes] (default) >>>>>> gzip_level 6 (default) >>>>>> gzip_memlevel 8 (default) >>>>>> http_gzip_support on [bool] (default) >>>>>> http_max_hdr 64 [header lines] (default) >>>>>> http_range_support on [bool] (default) >>>>>> http_req_hdr_len 8k [bytes] (default) >>>>>> http_req_size 32k [bytes] (default) >>>>>> http_resp_hdr_len 8k [bytes] (default) >>>>>> http_resp_size 32k [bytes] (default) >>>>>> idle_send_timeout 60.000 [seconds] (default) >>>>>> listen_depth 1024 [connections] (default) >>>>>> lru_interval 2.000 [seconds] (default) >>>>>> max_esi_depth 5 [levels] (default) >>>>>> max_restarts 4 [restarts] (default) >>>>>> max_retries 4 [retries] (default) >>>>>> nuke_limit 50 [allocations] (default) >>>>>> pcre_match_limit 10000 (default) >>>>>> pcre_match_limit_recursion 20 (default) >>>>>> ping_interval 3 [seconds] (default) >>>>>> pipe_timeout 60.000 [seconds] (default) >>>>>> pool_req 10,100,10 (default) >>>>>> pool_sess 10,100,10 (default) >>>>>> pool_vbo 10,100,10 (default) >>>>>> prefer_ipv6 off [bool] (default) >>>>>> rush_exponent 3 [requests per request] (default) >>>>>> send_timeout 600.000 [seconds] (default) >>>>>> session_max 100000 [sessions] (default) >>>>>> shm_reclen 255b [bytes] (default) >>>>>> shortlived 10.000 [seconds] (default) >>>>>> sigsegv_handler on [bool] (default) >>>>>> syslog_cli_traffic on [bool] (default) >>>>>> tcp_fastopen off [bool] (default) >>>>>> tcp_keepalive_intvl 75.000 [seconds] (default) >>>>>> tcp_keepalive_probes 9 [probes] (default) >>>>>> tcp_keepalive_time 7200.000 [seconds] (default) >>>>>> thread_pool_add_delay 0.000 [seconds] (default) >>>>>> thread_pool_destroy_delay 1.000 [seconds] (default) >>>>>> thread_pool_fail_delay 0.200 [seconds] (default) >>>>>> thread_pool_max 5000 [threads] (default) >>>>>> thread_pool_min 100 [threads] (default) >>>>>> thread_pool_reserve 0 [threads] (default) >>>>>> thread_pool_stack 48k [bytes] (default) >>>>>> thread_pool_timeout 300.000 [seconds] (default) >>>>>> thread_pools 2 [pools] (default) >>>>>> thread_queue_limit 20 (default) >>>>>> thread_stats_rate 10 [requests] (default) >>>>>> timeout_idle 5.000 [seconds] (default) >>>>>> timeout_linger 0.050 [seconds] (default) >>>>>> vcc_allow_inline_c off [bool] (default) >>>>>> vcc_err_unref on [bool] (default) >>>>>> vcc_unsafe_path on [bool] (default) >>>>>> vcl_cooldown 600.000 [seconds] (default) >>>>>> vcl_dir /etc/varnish (default) >>>>>> vmod_dir /usr/lib64/varnish/vmods (default) >>>>>> vsl_buffer 4k [bytes] (default) >>>>>> vsl_mask -VCL_trace,-WorkThread,-Hash,-VfpAcct >>>>>> (default) >>>>>> vsl_reclen 255b [bytes] (default) >>>>>> vsl_space 80M [bytes] (default) >>>>>> vsm_free_cooldown 60.000 [seconds] (default) >>>>>> vsm_space 1M [bytes] (default) >>>>>> workspace_backend 64k [bytes] (default) >>>>>> workspace_client 64k [bytes] (default) >>>>>> workspace_session 0.50k [bytes] (default) >>>>>> workspace_thread 2k [bytes] (default) >>>>>> >>>>>> >>>>>> -------- >>>>>> Regards, >>>>>> Junaid >>>>>> >>>>>> >>>>>> On Thu, Sep 20, 2018 at 3:37 PM Dridi Boukelmoune >>>>>> wrote: >>>>>> >>>>>>> On Thu, Sep 20, 2018 at 4:14 PM Junaid Mukhtar < >>>>>>> junaid.mukhtar at gmail.com> wrote: >>>>>>> > >>>>>>> > Hi >>>>>>> > >>>>>>> > we are in middle of upgrading from varnish 3.0.7 to varnish >>>>>>> 4.1.10; but unfortunately all of the response times in the performance test >>>>>>> are indicating an increase of at least 100% >>>>>>> > >>>>>>> > We have analyzed the logs and everything but can't get around >>>>>>> this; the issue is not only for the non-cached pages but it's also >>>>>>> impacting the cached pages. We are also struggling to analyze it further as >>>>>>> well, any guidenace as to what we can do in terms of putting debug logging >>>>>>> in would be helpful. >>>>>>> > >>>>>>> > One of the areas we are considering is to output response times, >>>>>>> but it's proving difficult for me. Any suggestions >>>>>>> >>>>>>> Hello, >>>>>>> >>>>>>> What is the output of `varnishadm param.show` ? >>>>>>> >>>>>>> Dridi >>>>>>> >>>>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From junaid.mukhtar at gmail.com Wed Sep 26 14:55:21 2018 From: junaid.mukhtar at gmail.com (Junaid Mukhtar) Date: Wed, 26 Sep 2018 15:55:21 +0100 Subject: Varnish 4 Performance Issues In-Reply-To: References: Message-ID: Interestingly we are seeing a lot of time in WaitingList before going to VCL Fetch - VCL_return hash - VCL_call HASH - VCL_return lookup - Timestamp Waitinglist: 1537973412.501378 2.410131 2.409963 - VCL_call MISS - VCL_return fetch - Link bereq 22354816 fetch - Timestamp Fetch: 1537973412.505995 2.414748 0.004617 -------- Regards, Junaid On Wed, Sep 26, 2018 at 9:31 AM Junaid Mukhtar wrote: > I totally agree about that. > > what we have figured out is that during the steady periods of performance > tests it run absolutely fine; but as soon as we spike up the traffic > (double the normal peak) performance degrades dramatically. > > > -------- > Regards, > Junaid > > > On Tue, Sep 25, 2018 at 5:17 PM Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> without the vcl (and/or logs, but I'm assuming those are even more >> confidential), that's a tough cookie to debug >> -- >> Guillaume Quintard >> >> >> On Mon, Sep 24, 2018 at 10:35 AM Junaid Mukhtar >> wrote: >> >>> Hi >>> >>> Apologies but I am not allowed to share the VCL. If there's any other >>> thing like stats etc that might be helpful then do let me know >>> >>> -------- >>> Regards, >>> Junaid >>> >>> >>> On Fri, Sep 21, 2018 at 4:15 PM Guillaume Quintard < >>> guillaume at varnish-software.com> wrote: >>> >>>> would you be able to share your vcl? >>>> -- >>>> Guillaume Quintard >>>> >>>> >>>> On Fri, Sep 21, 2018 at 4:54 PM Junaid Mukhtar < >>>> junaid.mukhtar at gmail.com> wrote: >>>> >>>>> Any help guys; this is really getting to us..... >>>>> >>>>> -------- >>>>> Regards, >>>>> Junaid >>>>> >>>>> >>>>> On Thu, Sep 20, 2018 at 3:57 PM Junaid Mukhtar < >>>>> junaid.mukhtar at gmail.com> wrote: >>>>> >>>>>> >>>>>> This is what we get in varnishhist >>>>>> >>>>>> 280_ >>>>>> >>>>>> >>>>>> >>>>>> | >>>>>> 230_ | >>>>>> | >>>>>> | >>>>>> | >>>>>> | >>>>>> 180_ | >>>>>> || >>>>>> || >>>>>> ||| >>>>>> ||| >>>>>> 130_ ||| >>>>>> ||| >>>>>> |||||| >>>>>> |||||| >>>>>> |||||| >>>>>> 80_ |||||| >>>>>> >>>>>> |||||| # >>>>>> >>>>>> |||||| ### >>>>>> >>>>>> ||||||| # ### >>>>>> >>>>>> ||||||||| ##### # >>>>>> 30_ >>>>>> ||||||||| ######## >>>>>> >>>>>> ||||||||| >>>>>> ######### >>>>>> >>>>>> |||||||||| >>>>>> ########### >>>>>> >>>>>> |||||||||||| ## # >>>>>> ############# # >>>>>> >>>>>> +---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+--------------------- >>>>>> |1e-6 |1e-5 |1e-4 >>>>>> |1e-3 |1e-2 |1e-1 >>>>>> |1e0 |1e1 |1e2 >>>>>> -------- >>>>>> Regards, >>>>>> Junaid >>>>>> >>>>>> >>>>>> On Thu, Sep 20, 2018 at 3:54 PM Junaid Mukhtar < >>>>>> junaid.mukhtar at gmail.com> wrote: >>>>>> >>>>>>> Below is the output; all of them are default except i was trying to >>>>>>> up the threads_pool to 4 but didn't feel any imporvement in performance >>>>>>> degradation >>>>>>> >>>>>>> accept_filter off [bool] (default) >>>>>>> acceptor_sleep_decay 0.9 (default) >>>>>>> acceptor_sleep_incr 0.000 [seconds] (default) >>>>>>> acceptor_sleep_max 0.050 [seconds] (default) >>>>>>> auto_restart on [bool] (default) >>>>>>> backend_idle_timeout 60.000 [seconds] (default) >>>>>>> ban_dups on [bool] (default) >>>>>>> ban_lurker_age 60.000 [seconds] (default) >>>>>>> ban_lurker_batch 1000 (default) >>>>>>> ban_lurker_sleep 0.010 [seconds] (default) >>>>>>> between_bytes_timeout 60.000 [seconds] (default) >>>>>>> cc_command "exec gcc -std=gnu99 -O2 -g >>>>>>> -Wp,-D_FORTIFY_SOURCE=0 -Wall -Werror -pthread -fpic -shared -Wl,-x -o %o >>>>>>> %s" (default) >>>>>>> cli_buffer 8k [bytes] (default) >>>>>>> cli_limit 48k [bytes] (default) >>>>>>> cli_timeout 60.000 [seconds] (default) >>>>>>> clock_skew 10 [seconds] (default) >>>>>>> clock_step 1.000 [seconds] (default) >>>>>>> connect_timeout 3.500 [seconds] (default) >>>>>>> critbit_cooloff 180.000 [seconds] (default) >>>>>>> debug none (default) >>>>>>> default_grace 10.000 [seconds] (default) >>>>>>> default_keep 0.000 [seconds] (default) >>>>>>> default_ttl 120.000 [seconds] (default) >>>>>>> feature none (default) >>>>>>> fetch_chunksize 16k [bytes] (default) >>>>>>> fetch_maxchunksize 0.25G [bytes] (default) >>>>>>> first_byte_timeout 60.000 [seconds] (default) >>>>>>> gzip_buffer 32k [bytes] (default) >>>>>>> gzip_level 6 (default) >>>>>>> gzip_memlevel 8 (default) >>>>>>> http_gzip_support on [bool] (default) >>>>>>> http_max_hdr 64 [header lines] (default) >>>>>>> http_range_support on [bool] (default) >>>>>>> http_req_hdr_len 8k [bytes] (default) >>>>>>> http_req_size 32k [bytes] (default) >>>>>>> http_resp_hdr_len 8k [bytes] (default) >>>>>>> http_resp_size 32k [bytes] (default) >>>>>>> idle_send_timeout 60.000 [seconds] (default) >>>>>>> listen_depth 1024 [connections] (default) >>>>>>> lru_interval 2.000 [seconds] (default) >>>>>>> max_esi_depth 5 [levels] (default) >>>>>>> max_restarts 4 [restarts] (default) >>>>>>> max_retries 4 [retries] (default) >>>>>>> nuke_limit 50 [allocations] (default) >>>>>>> pcre_match_limit 10000 (default) >>>>>>> pcre_match_limit_recursion 20 (default) >>>>>>> ping_interval 3 [seconds] (default) >>>>>>> pipe_timeout 60.000 [seconds] (default) >>>>>>> pool_req 10,100,10 (default) >>>>>>> pool_sess 10,100,10 (default) >>>>>>> pool_vbo 10,100,10 (default) >>>>>>> prefer_ipv6 off [bool] (default) >>>>>>> rush_exponent 3 [requests per request] (default) >>>>>>> send_timeout 600.000 [seconds] (default) >>>>>>> session_max 100000 [sessions] (default) >>>>>>> shm_reclen 255b [bytes] (default) >>>>>>> shortlived 10.000 [seconds] (default) >>>>>>> sigsegv_handler on [bool] (default) >>>>>>> syslog_cli_traffic on [bool] (default) >>>>>>> tcp_fastopen off [bool] (default) >>>>>>> tcp_keepalive_intvl 75.000 [seconds] (default) >>>>>>> tcp_keepalive_probes 9 [probes] (default) >>>>>>> tcp_keepalive_time 7200.000 [seconds] (default) >>>>>>> thread_pool_add_delay 0.000 [seconds] (default) >>>>>>> thread_pool_destroy_delay 1.000 [seconds] (default) >>>>>>> thread_pool_fail_delay 0.200 [seconds] (default) >>>>>>> thread_pool_max 5000 [threads] (default) >>>>>>> thread_pool_min 100 [threads] (default) >>>>>>> thread_pool_reserve 0 [threads] (default) >>>>>>> thread_pool_stack 48k [bytes] (default) >>>>>>> thread_pool_timeout 300.000 [seconds] (default) >>>>>>> thread_pools 2 [pools] (default) >>>>>>> thread_queue_limit 20 (default) >>>>>>> thread_stats_rate 10 [requests] (default) >>>>>>> timeout_idle 5.000 [seconds] (default) >>>>>>> timeout_linger 0.050 [seconds] (default) >>>>>>> vcc_allow_inline_c off [bool] (default) >>>>>>> vcc_err_unref on [bool] (default) >>>>>>> vcc_unsafe_path on [bool] (default) >>>>>>> vcl_cooldown 600.000 [seconds] (default) >>>>>>> vcl_dir /etc/varnish (default) >>>>>>> vmod_dir /usr/lib64/varnish/vmods (default) >>>>>>> vsl_buffer 4k [bytes] (default) >>>>>>> vsl_mask -VCL_trace,-WorkThread,-Hash,-VfpAcct >>>>>>> (default) >>>>>>> vsl_reclen 255b [bytes] (default) >>>>>>> vsl_space 80M [bytes] (default) >>>>>>> vsm_free_cooldown 60.000 [seconds] (default) >>>>>>> vsm_space 1M [bytes] (default) >>>>>>> workspace_backend 64k [bytes] (default) >>>>>>> workspace_client 64k [bytes] (default) >>>>>>> workspace_session 0.50k [bytes] (default) >>>>>>> workspace_thread 2k [bytes] (default) >>>>>>> >>>>>>> >>>>>>> -------- >>>>>>> Regards, >>>>>>> Junaid >>>>>>> >>>>>>> >>>>>>> On Thu, Sep 20, 2018 at 3:37 PM Dridi Boukelmoune >>>>>>> wrote: >>>>>>> >>>>>>>> On Thu, Sep 20, 2018 at 4:14 PM Junaid Mukhtar < >>>>>>>> junaid.mukhtar at gmail.com> wrote: >>>>>>>> > >>>>>>>> > Hi >>>>>>>> > >>>>>>>> > we are in middle of upgrading from varnish 3.0.7 to varnish >>>>>>>> 4.1.10; but unfortunately all of the response times in the performance test >>>>>>>> are indicating an increase of at least 100% >>>>>>>> > >>>>>>>> > We have analyzed the logs and everything but can't get around >>>>>>>> this; the issue is not only for the non-cached pages but it's also >>>>>>>> impacting the cached pages. We are also struggling to analyze it further as >>>>>>>> well, any guidenace as to what we can do in terms of putting debug logging >>>>>>>> in would be helpful. >>>>>>>> > >>>>>>>> > One of the areas we are considering is to output response times, >>>>>>>> but it's proving difficult for me. Any suggestions >>>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> What is the output of `varnishadm param.show` ? >>>>>>>> >>>>>>>> Dridi >>>>>>>> >>>>>>> _______________________________________________ >>>>> varnish-misc mailing list >>>>> varnish-misc at varnish-cache.org >>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Wed Sep 26 15:16:16 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 26 Sep 2018 17:16:16 +0200 Subject: Varnish 4 Performance Issues In-Reply-To: References: Message-ID: Getting a lot of Hit-for-Pass? Are you doing something like setting the ttl to 0s by any chance? -- Guillaume Quintard On Wed, Sep 26, 2018 at 4:55 PM Junaid Mukhtar wrote: > Interestingly we are seeing a lot of time in WaitingList before going to > VCL Fetch > > - VCL_return hash > - VCL_call HASH > - VCL_return lookup > - Timestamp Waitinglist: 1537973412.501378 2.410131 2.409963 > - VCL_call MISS > - VCL_return fetch > - Link bereq 22354816 fetch > - Timestamp Fetch: 1537973412.505995 2.414748 0.004617 > > > -------- > Regards, > Junaid > > > On Wed, Sep 26, 2018 at 9:31 AM Junaid Mukhtar > wrote: > >> I totally agree about that. >> >> what we have figured out is that during the steady periods of performance >> tests it run absolutely fine; but as soon as we spike up the traffic >> (double the normal peak) performance degrades dramatically. >> >> >> -------- >> Regards, >> Junaid >> >> >> On Tue, Sep 25, 2018 at 5:17 PM Guillaume Quintard < >> guillaume at varnish-software.com> wrote: >> >>> without the vcl (and/or logs, but I'm assuming those are even more >>> confidential), that's a tough cookie to debug >>> -- >>> Guillaume Quintard >>> >>> >>> On Mon, Sep 24, 2018 at 10:35 AM Junaid Mukhtar < >>> junaid.mukhtar at gmail.com> wrote: >>> >>>> Hi >>>> >>>> Apologies but I am not allowed to share the VCL. If there's any other >>>> thing like stats etc that might be helpful then do let me know >>>> >>>> -------- >>>> Regards, >>>> Junaid >>>> >>>> >>>> On Fri, Sep 21, 2018 at 4:15 PM Guillaume Quintard < >>>> guillaume at varnish-software.com> wrote: >>>> >>>>> would you be able to share your vcl? >>>>> -- >>>>> Guillaume Quintard >>>>> >>>>> >>>>> On Fri, Sep 21, 2018 at 4:54 PM Junaid Mukhtar < >>>>> junaid.mukhtar at gmail.com> wrote: >>>>> >>>>>> Any help guys; this is really getting to us..... >>>>>> >>>>>> -------- >>>>>> Regards, >>>>>> Junaid >>>>>> >>>>>> >>>>>> On Thu, Sep 20, 2018 at 3:57 PM Junaid Mukhtar < >>>>>> junaid.mukhtar at gmail.com> wrote: >>>>>> >>>>>>> >>>>>>> This is what we get in varnishhist >>>>>>> >>>>>>> 280_ >>>>>>> >>>>>>> >>>>>>> >>>>>>> | >>>>>>> 230_ | >>>>>>> | >>>>>>> | >>>>>>> | >>>>>>> | >>>>>>> 180_ | >>>>>>> || >>>>>>> || >>>>>>> ||| >>>>>>> ||| >>>>>>> 130_ ||| >>>>>>> ||| >>>>>>> |||||| >>>>>>> |||||| >>>>>>> |||||| >>>>>>> 80_ |||||| >>>>>>> >>>>>>> |||||| # >>>>>>> >>>>>>> |||||| ### >>>>>>> >>>>>>> ||||||| # ### >>>>>>> >>>>>>> ||||||||| ##### # >>>>>>> 30_ >>>>>>> ||||||||| ######## >>>>>>> >>>>>>> ||||||||| >>>>>>> ######### >>>>>>> >>>>>>> |||||||||| >>>>>>> ########### >>>>>>> >>>>>>> |||||||||||| ## # >>>>>>> ############# # >>>>>>> >>>>>>> +---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+--------------------- >>>>>>> |1e-6 |1e-5 |1e-4 >>>>>>> |1e-3 |1e-2 |1e-1 >>>>>>> |1e0 |1e1 |1e2 >>>>>>> -------- >>>>>>> Regards, >>>>>>> Junaid >>>>>>> >>>>>>> >>>>>>> On Thu, Sep 20, 2018 at 3:54 PM Junaid Mukhtar < >>>>>>> junaid.mukhtar at gmail.com> wrote: >>>>>>> >>>>>>>> Below is the output; all of them are default except i was trying to >>>>>>>> up the threads_pool to 4 but didn't feel any imporvement in performance >>>>>>>> degradation >>>>>>>> >>>>>>>> accept_filter off [bool] (default) >>>>>>>> acceptor_sleep_decay 0.9 (default) >>>>>>>> acceptor_sleep_incr 0.000 [seconds] (default) >>>>>>>> acceptor_sleep_max 0.050 [seconds] (default) >>>>>>>> auto_restart on [bool] (default) >>>>>>>> backend_idle_timeout 60.000 [seconds] (default) >>>>>>>> ban_dups on [bool] (default) >>>>>>>> ban_lurker_age 60.000 [seconds] (default) >>>>>>>> ban_lurker_batch 1000 (default) >>>>>>>> ban_lurker_sleep 0.010 [seconds] (default) >>>>>>>> between_bytes_timeout 60.000 [seconds] (default) >>>>>>>> cc_command "exec gcc -std=gnu99 -O2 -g >>>>>>>> -Wp,-D_FORTIFY_SOURCE=0 -Wall -Werror -pthread -fpic -shared -Wl,-x -o %o >>>>>>>> %s" (default) >>>>>>>> cli_buffer 8k [bytes] (default) >>>>>>>> cli_limit 48k [bytes] (default) >>>>>>>> cli_timeout 60.000 [seconds] (default) >>>>>>>> clock_skew 10 [seconds] (default) >>>>>>>> clock_step 1.000 [seconds] (default) >>>>>>>> connect_timeout 3.500 [seconds] (default) >>>>>>>> critbit_cooloff 180.000 [seconds] (default) >>>>>>>> debug none (default) >>>>>>>> default_grace 10.000 [seconds] (default) >>>>>>>> default_keep 0.000 [seconds] (default) >>>>>>>> default_ttl 120.000 [seconds] (default) >>>>>>>> feature none (default) >>>>>>>> fetch_chunksize 16k [bytes] (default) >>>>>>>> fetch_maxchunksize 0.25G [bytes] (default) >>>>>>>> first_byte_timeout 60.000 [seconds] (default) >>>>>>>> gzip_buffer 32k [bytes] (default) >>>>>>>> gzip_level 6 (default) >>>>>>>> gzip_memlevel 8 (default) >>>>>>>> http_gzip_support on [bool] (default) >>>>>>>> http_max_hdr 64 [header lines] (default) >>>>>>>> http_range_support on [bool] (default) >>>>>>>> http_req_hdr_len 8k [bytes] (default) >>>>>>>> http_req_size 32k [bytes] (default) >>>>>>>> http_resp_hdr_len 8k [bytes] (default) >>>>>>>> http_resp_size 32k [bytes] (default) >>>>>>>> idle_send_timeout 60.000 [seconds] (default) >>>>>>>> listen_depth 1024 [connections] (default) >>>>>>>> lru_interval 2.000 [seconds] (default) >>>>>>>> max_esi_depth 5 [levels] (default) >>>>>>>> max_restarts 4 [restarts] (default) >>>>>>>> max_retries 4 [retries] (default) >>>>>>>> nuke_limit 50 [allocations] (default) >>>>>>>> pcre_match_limit 10000 (default) >>>>>>>> pcre_match_limit_recursion 20 (default) >>>>>>>> ping_interval 3 [seconds] (default) >>>>>>>> pipe_timeout 60.000 [seconds] (default) >>>>>>>> pool_req 10,100,10 (default) >>>>>>>> pool_sess 10,100,10 (default) >>>>>>>> pool_vbo 10,100,10 (default) >>>>>>>> prefer_ipv6 off [bool] (default) >>>>>>>> rush_exponent 3 [requests per request] (default) >>>>>>>> send_timeout 600.000 [seconds] (default) >>>>>>>> session_max 100000 [sessions] (default) >>>>>>>> shm_reclen 255b [bytes] (default) >>>>>>>> shortlived 10.000 [seconds] (default) >>>>>>>> sigsegv_handler on [bool] (default) >>>>>>>> syslog_cli_traffic on [bool] (default) >>>>>>>> tcp_fastopen off [bool] (default) >>>>>>>> tcp_keepalive_intvl 75.000 [seconds] (default) >>>>>>>> tcp_keepalive_probes 9 [probes] (default) >>>>>>>> tcp_keepalive_time 7200.000 [seconds] (default) >>>>>>>> thread_pool_add_delay 0.000 [seconds] (default) >>>>>>>> thread_pool_destroy_delay 1.000 [seconds] (default) >>>>>>>> thread_pool_fail_delay 0.200 [seconds] (default) >>>>>>>> thread_pool_max 5000 [threads] (default) >>>>>>>> thread_pool_min 100 [threads] (default) >>>>>>>> thread_pool_reserve 0 [threads] (default) >>>>>>>> thread_pool_stack 48k [bytes] (default) >>>>>>>> thread_pool_timeout 300.000 [seconds] (default) >>>>>>>> thread_pools 2 [pools] (default) >>>>>>>> thread_queue_limit 20 (default) >>>>>>>> thread_stats_rate 10 [requests] (default) >>>>>>>> timeout_idle 5.000 [seconds] (default) >>>>>>>> timeout_linger 0.050 [seconds] (default) >>>>>>>> vcc_allow_inline_c off [bool] (default) >>>>>>>> vcc_err_unref on [bool] (default) >>>>>>>> vcc_unsafe_path on [bool] (default) >>>>>>>> vcl_cooldown 600.000 [seconds] (default) >>>>>>>> vcl_dir /etc/varnish (default) >>>>>>>> vmod_dir /usr/lib64/varnish/vmods (default) >>>>>>>> vsl_buffer 4k [bytes] (default) >>>>>>>> vsl_mask -VCL_trace,-WorkThread,-Hash,-VfpAcct >>>>>>>> (default) >>>>>>>> vsl_reclen 255b [bytes] (default) >>>>>>>> vsl_space 80M [bytes] (default) >>>>>>>> vsm_free_cooldown 60.000 [seconds] (default) >>>>>>>> vsm_space 1M [bytes] (default) >>>>>>>> workspace_backend 64k [bytes] (default) >>>>>>>> workspace_client 64k [bytes] (default) >>>>>>>> workspace_session 0.50k [bytes] (default) >>>>>>>> workspace_thread 2k [bytes] (default) >>>>>>>> >>>>>>>> >>>>>>>> -------- >>>>>>>> Regards, >>>>>>>> Junaid >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Sep 20, 2018 at 3:37 PM Dridi Boukelmoune >>>>>>>> wrote: >>>>>>>> >>>>>>>>> On Thu, Sep 20, 2018 at 4:14 PM Junaid Mukhtar < >>>>>>>>> junaid.mukhtar at gmail.com> wrote: >>>>>>>>> > >>>>>>>>> > Hi >>>>>>>>> > >>>>>>>>> > we are in middle of upgrading from varnish 3.0.7 to varnish >>>>>>>>> 4.1.10; but unfortunately all of the response times in the performance test >>>>>>>>> are indicating an increase of at least 100% >>>>>>>>> > >>>>>>>>> > We have analyzed the logs and everything but can't get around >>>>>>>>> this; the issue is not only for the non-cached pages but it's also >>>>>>>>> impacting the cached pages. We are also struggling to analyze it further as >>>>>>>>> well, any guidenace as to what we can do in terms of putting debug logging >>>>>>>>> in would be helpful. >>>>>>>>> > >>>>>>>>> > One of the areas we are considering is to output response times, >>>>>>>>> but it's proving difficult for me. Any suggestions >>>>>>>>> >>>>>>>>> Hello, >>>>>>>>> >>>>>>>>> What is the output of `varnishadm param.show` ? >>>>>>>>> >>>>>>>>> Dridi >>>>>>>>> >>>>>>>> _______________________________________________ >>>>>> varnish-misc mailing list >>>>>> varnish-misc at varnish-cache.org >>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Thu Sep 27 13:40:03 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 27 Sep 2018 15:40:03 +0200 Subject: Varnish swallowing 4xx responses from POSTs In-Reply-To: <6DBBA006-52E8-4927-8E64-663B9678A514@gmail.com> References: <6DBBA006-52E8-4927-8E64-663B9678A514@gmail.com> Message-ID: Hello, On Wed, Sep 26, 2018 at 2:44 AM Tommy Becker wrote: > > We have an application that we front with Varnish 4.0.5. Recently, after an application upgrade in which we migrated from Jetty 9.2 to 9.4, we began noticing a lot of 503s being returned from Varnish on POST requests. We have an endpoint that takes a payload of a potentially large amount of JSON, and validates it as it?s being read. What we have discovered is that if there is a problem with the content, we correctly return a 400 Bad Request from Jetty. Notably, this can happen before the entire content is received. When this happens, Varnish continues to send the remainder of data, despite having already seen the response. Now after our upgrade, Jetty's behavior is to send a TCP RST when this happens (since the data is unwanted by the application). Unfortunately, Varnish interprets the RST as a backend error, and goes to vcl_backend_error, having never sent the original response returned from Jetty to the client. So instead of seeing a 400 Bad Request with a helpful message, they simply get 503 Service Unavailable. I'm pretty certain that this optimization does not comply with the HTTP/1 specs. Even though Jetty is trying to improve the latency by replying early, as far as Varnish is concerned it failed to send the full request and won't bother reading the response. > I found this issue which seems similar: https://github.com/varnishcache/varnish-cache/issues/2332 Can someone help here? Is there anyway to work around this behavior? In this case that's different because the backend is inspecting the body as it is coming and not rejecting the request based on the size only. So I'm afraid there's no way to work around this behavior. As this is not a bug, we could introduce either a feature flag or a VCL variable turned off by default to tolerate an early reset of the request side of an HTTP/1 socket. You could join next bugwash on Monday to bring this to the team's attention. Dridi From junaid.mukhtar at gmail.com Thu Sep 27 13:57:14 2018 From: junaid.mukhtar at gmail.com (Junaid Mukhtar) Date: Thu, 27 Sep 2018 14:57:14 +0100 Subject: Varnish 4 Performance Issues In-Reply-To: References: Message-ID: We found the issue, it was Hit-for-Pass :( a lot of the non-cacheable URLs were being queued in VCL_Hash and that was causing varnish to slow down. We forced them to pass directly and this caused masive performance improvements and cache hit ratio jumped up as well -------- Regards, Junaid On Wed, Sep 26, 2018 at 4:16 PM Guillaume Quintard < guillaume at varnish-software.com> wrote: > Getting a lot of Hit-for-Pass? > > Are you doing something like setting the ttl to 0s by any chance? > -- > Guillaume Quintard > > > On Wed, Sep 26, 2018 at 4:55 PM Junaid Mukhtar > wrote: > >> Interestingly we are seeing a lot of time in WaitingList before going to >> VCL Fetch >> >> - VCL_return hash >> - VCL_call HASH >> - VCL_return lookup >> - Timestamp Waitinglist: 1537973412.501378 2.410131 2.409963 >> - VCL_call MISS >> - VCL_return fetch >> - Link bereq 22354816 fetch >> - Timestamp Fetch: 1537973412.505995 2.414748 0.004617 >> >> >> -------- >> Regards, >> Junaid >> >> >> On Wed, Sep 26, 2018 at 9:31 AM Junaid Mukhtar >> wrote: >> >>> I totally agree about that. >>> >>> what we have figured out is that during the steady periods of >>> performance tests it run absolutely fine; but as soon as we spike up the >>> traffic (double the normal peak) performance degrades dramatically. >>> >>> >>> -------- >>> Regards, >>> Junaid >>> >>> >>> On Tue, Sep 25, 2018 at 5:17 PM Guillaume Quintard < >>> guillaume at varnish-software.com> wrote: >>> >>>> without the vcl (and/or logs, but I'm assuming those are even more >>>> confidential), that's a tough cookie to debug >>>> -- >>>> Guillaume Quintard >>>> >>>> >>>> On Mon, Sep 24, 2018 at 10:35 AM Junaid Mukhtar < >>>> junaid.mukhtar at gmail.com> wrote: >>>> >>>>> Hi >>>>> >>>>> Apologies but I am not allowed to share the VCL. If there's any other >>>>> thing like stats etc that might be helpful then do let me know >>>>> >>>>> -------- >>>>> Regards, >>>>> Junaid >>>>> >>>>> >>>>> On Fri, Sep 21, 2018 at 4:15 PM Guillaume Quintard < >>>>> guillaume at varnish-software.com> wrote: >>>>> >>>>>> would you be able to share your vcl? >>>>>> -- >>>>>> Guillaume Quintard >>>>>> >>>>>> >>>>>> On Fri, Sep 21, 2018 at 4:54 PM Junaid Mukhtar < >>>>>> junaid.mukhtar at gmail.com> wrote: >>>>>> >>>>>>> Any help guys; this is really getting to us..... >>>>>>> >>>>>>> -------- >>>>>>> Regards, >>>>>>> Junaid >>>>>>> >>>>>>> >>>>>>> On Thu, Sep 20, 2018 at 3:57 PM Junaid Mukhtar < >>>>>>> junaid.mukhtar at gmail.com> wrote: >>>>>>> >>>>>>>> >>>>>>>> This is what we get in varnishhist >>>>>>>> >>>>>>>> 280_ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> | >>>>>>>> 230_ | >>>>>>>> | >>>>>>>> | >>>>>>>> | >>>>>>>> | >>>>>>>> 180_ | >>>>>>>> || >>>>>>>> || >>>>>>>> ||| >>>>>>>> ||| >>>>>>>> 130_ ||| >>>>>>>> ||| >>>>>>>> |||||| >>>>>>>> |||||| >>>>>>>> |||||| >>>>>>>> 80_ |||||| >>>>>>>> >>>>>>>> |||||| # >>>>>>>> >>>>>>>> |||||| ### >>>>>>>> >>>>>>>> ||||||| # ### >>>>>>>> >>>>>>>> ||||||||| ##### # >>>>>>>> 30_ >>>>>>>> ||||||||| ######## >>>>>>>> >>>>>>>> ||||||||| >>>>>>>> ######### >>>>>>>> >>>>>>>> |||||||||| >>>>>>>> ########### >>>>>>>> >>>>>>>> |||||||||||| ## # >>>>>>>> ############# # >>>>>>>> >>>>>>>> +---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+--------------------- >>>>>>>> |1e-6 |1e-5 |1e-4 >>>>>>>> |1e-3 |1e-2 |1e-1 >>>>>>>> |1e0 |1e1 |1e2 >>>>>>>> -------- >>>>>>>> Regards, >>>>>>>> Junaid >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Sep 20, 2018 at 3:54 PM Junaid Mukhtar < >>>>>>>> junaid.mukhtar at gmail.com> wrote: >>>>>>>> >>>>>>>>> Below is the output; all of them are default except i was trying >>>>>>>>> to up the threads_pool to 4 but didn't feel any imporvement in performance >>>>>>>>> degradation >>>>>>>>> >>>>>>>>> accept_filter off [bool] (default) >>>>>>>>> acceptor_sleep_decay 0.9 (default) >>>>>>>>> acceptor_sleep_incr 0.000 [seconds] (default) >>>>>>>>> acceptor_sleep_max 0.050 [seconds] (default) >>>>>>>>> auto_restart on [bool] (default) >>>>>>>>> backend_idle_timeout 60.000 [seconds] (default) >>>>>>>>> ban_dups on [bool] (default) >>>>>>>>> ban_lurker_age 60.000 [seconds] (default) >>>>>>>>> ban_lurker_batch 1000 (default) >>>>>>>>> ban_lurker_sleep 0.010 [seconds] (default) >>>>>>>>> between_bytes_timeout 60.000 [seconds] (default) >>>>>>>>> cc_command "exec gcc -std=gnu99 -O2 -g >>>>>>>>> -Wp,-D_FORTIFY_SOURCE=0 -Wall -Werror -pthread -fpic -shared -Wl,-x -o %o >>>>>>>>> %s" (default) >>>>>>>>> cli_buffer 8k [bytes] (default) >>>>>>>>> cli_limit 48k [bytes] (default) >>>>>>>>> cli_timeout 60.000 [seconds] (default) >>>>>>>>> clock_skew 10 [seconds] (default) >>>>>>>>> clock_step 1.000 [seconds] (default) >>>>>>>>> connect_timeout 3.500 [seconds] (default) >>>>>>>>> critbit_cooloff 180.000 [seconds] (default) >>>>>>>>> debug none (default) >>>>>>>>> default_grace 10.000 [seconds] (default) >>>>>>>>> default_keep 0.000 [seconds] (default) >>>>>>>>> default_ttl 120.000 [seconds] (default) >>>>>>>>> feature none (default) >>>>>>>>> fetch_chunksize 16k [bytes] (default) >>>>>>>>> fetch_maxchunksize 0.25G [bytes] (default) >>>>>>>>> first_byte_timeout 60.000 [seconds] (default) >>>>>>>>> gzip_buffer 32k [bytes] (default) >>>>>>>>> gzip_level 6 (default) >>>>>>>>> gzip_memlevel 8 (default) >>>>>>>>> http_gzip_support on [bool] (default) >>>>>>>>> http_max_hdr 64 [header lines] (default) >>>>>>>>> http_range_support on [bool] (default) >>>>>>>>> http_req_hdr_len 8k [bytes] (default) >>>>>>>>> http_req_size 32k [bytes] (default) >>>>>>>>> http_resp_hdr_len 8k [bytes] (default) >>>>>>>>> http_resp_size 32k [bytes] (default) >>>>>>>>> idle_send_timeout 60.000 [seconds] (default) >>>>>>>>> listen_depth 1024 [connections] (default) >>>>>>>>> lru_interval 2.000 [seconds] (default) >>>>>>>>> max_esi_depth 5 [levels] (default) >>>>>>>>> max_restarts 4 [restarts] (default) >>>>>>>>> max_retries 4 [retries] (default) >>>>>>>>> nuke_limit 50 [allocations] (default) >>>>>>>>> pcre_match_limit 10000 (default) >>>>>>>>> pcre_match_limit_recursion 20 (default) >>>>>>>>> ping_interval 3 [seconds] (default) >>>>>>>>> pipe_timeout 60.000 [seconds] (default) >>>>>>>>> pool_req 10,100,10 (default) >>>>>>>>> pool_sess 10,100,10 (default) >>>>>>>>> pool_vbo 10,100,10 (default) >>>>>>>>> prefer_ipv6 off [bool] (default) >>>>>>>>> rush_exponent 3 [requests per request] (default) >>>>>>>>> send_timeout 600.000 [seconds] (default) >>>>>>>>> session_max 100000 [sessions] (default) >>>>>>>>> shm_reclen 255b [bytes] (default) >>>>>>>>> shortlived 10.000 [seconds] (default) >>>>>>>>> sigsegv_handler on [bool] (default) >>>>>>>>> syslog_cli_traffic on [bool] (default) >>>>>>>>> tcp_fastopen off [bool] (default) >>>>>>>>> tcp_keepalive_intvl 75.000 [seconds] (default) >>>>>>>>> tcp_keepalive_probes 9 [probes] (default) >>>>>>>>> tcp_keepalive_time 7200.000 [seconds] (default) >>>>>>>>> thread_pool_add_delay 0.000 [seconds] (default) >>>>>>>>> thread_pool_destroy_delay 1.000 [seconds] (default) >>>>>>>>> thread_pool_fail_delay 0.200 [seconds] (default) >>>>>>>>> thread_pool_max 5000 [threads] (default) >>>>>>>>> thread_pool_min 100 [threads] (default) >>>>>>>>> thread_pool_reserve 0 [threads] (default) >>>>>>>>> thread_pool_stack 48k [bytes] (default) >>>>>>>>> thread_pool_timeout 300.000 [seconds] (default) >>>>>>>>> thread_pools 2 [pools] (default) >>>>>>>>> thread_queue_limit 20 (default) >>>>>>>>> thread_stats_rate 10 [requests] (default) >>>>>>>>> timeout_idle 5.000 [seconds] (default) >>>>>>>>> timeout_linger 0.050 [seconds] (default) >>>>>>>>> vcc_allow_inline_c off [bool] (default) >>>>>>>>> vcc_err_unref on [bool] (default) >>>>>>>>> vcc_unsafe_path on [bool] (default) >>>>>>>>> vcl_cooldown 600.000 [seconds] (default) >>>>>>>>> vcl_dir /etc/varnish (default) >>>>>>>>> vmod_dir /usr/lib64/varnish/vmods (default) >>>>>>>>> vsl_buffer 4k [bytes] (default) >>>>>>>>> vsl_mask -VCL_trace,-WorkThread,-Hash,-VfpAcct >>>>>>>>> (default) >>>>>>>>> vsl_reclen 255b [bytes] (default) >>>>>>>>> vsl_space 80M [bytes] (default) >>>>>>>>> vsm_free_cooldown 60.000 [seconds] (default) >>>>>>>>> vsm_space 1M [bytes] (default) >>>>>>>>> workspace_backend 64k [bytes] (default) >>>>>>>>> workspace_client 64k [bytes] (default) >>>>>>>>> workspace_session 0.50k [bytes] (default) >>>>>>>>> workspace_thread 2k [bytes] (default) >>>>>>>>> >>>>>>>>> >>>>>>>>> -------- >>>>>>>>> Regards, >>>>>>>>> Junaid >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Sep 20, 2018 at 3:37 PM Dridi Boukelmoune >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> On Thu, Sep 20, 2018 at 4:14 PM Junaid Mukhtar < >>>>>>>>>> junaid.mukhtar at gmail.com> wrote: >>>>>>>>>> > >>>>>>>>>> > Hi >>>>>>>>>> > >>>>>>>>>> > we are in middle of upgrading from varnish 3.0.7 to varnish >>>>>>>>>> 4.1.10; but unfortunately all of the response times in the performance test >>>>>>>>>> are indicating an increase of at least 100% >>>>>>>>>> > >>>>>>>>>> > We have analyzed the logs and everything but can't get around >>>>>>>>>> this; the issue is not only for the non-cached pages but it's also >>>>>>>>>> impacting the cached pages. We are also struggling to analyze it further as >>>>>>>>>> well, any guidenace as to what we can do in terms of putting debug logging >>>>>>>>>> in would be helpful. >>>>>>>>>> > >>>>>>>>>> > One of the areas we are considering is to output response >>>>>>>>>> times, but it's proving difficult for me. Any suggestions >>>>>>>>>> >>>>>>>>>> Hello, >>>>>>>>>> >>>>>>>>>> What is the output of `varnishadm param.show` ? >>>>>>>>>> >>>>>>>>>> Dridi >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>> varnish-misc mailing list >>>>>>> varnish-misc at varnish-cache.org >>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>>> >>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Sep 27 14:53:11 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 27 Sep 2018 16:53:11 +0200 Subject: Varnish 4 Performance Issues In-Reply-To: References: Message-ID: But hit-for-pass, by default marks the object as uncacheable for two minutes, so you should have issue for too long. The problem you describe really looks like requests serialization stemming from a lack of hit-for-pass. But glad to read that it's over. On Thu, Sep 27, 2018, 15:57 Junaid Mukhtar wrote: > We found the issue, it was Hit-for-Pass :( > > a lot of the non-cacheable URLs were being queued in VCL_Hash and that was > causing varnish to slow down. We forced them to pass directly and this > caused masive performance improvements and cache hit ratio jumped up as well > > -------- > Regards, > Junaid > > > On Wed, Sep 26, 2018 at 4:16 PM Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> Getting a lot of Hit-for-Pass? >> >> Are you doing something like setting the ttl to 0s by any chance? >> -- >> Guillaume Quintard >> >> >> On Wed, Sep 26, 2018 at 4:55 PM Junaid Mukhtar >> wrote: >> >>> Interestingly we are seeing a lot of time in WaitingList before going to >>> VCL Fetch >>> >>> - VCL_return hash >>> - VCL_call HASH >>> - VCL_return lookup >>> - Timestamp Waitinglist: 1537973412.501378 2.410131 2.409963 >>> - VCL_call MISS >>> - VCL_return fetch >>> - Link bereq 22354816 fetch >>> - Timestamp Fetch: 1537973412.505995 2.414748 0.004617 >>> >>> >>> -------- >>> Regards, >>> Junaid >>> >>> >>> On Wed, Sep 26, 2018 at 9:31 AM Junaid Mukhtar >>> wrote: >>> >>>> I totally agree about that. >>>> >>>> what we have figured out is that during the steady periods of >>>> performance tests it run absolutely fine; but as soon as we spike up the >>>> traffic (double the normal peak) performance degrades dramatically. >>>> >>>> >>>> -------- >>>> Regards, >>>> Junaid >>>> >>>> >>>> On Tue, Sep 25, 2018 at 5:17 PM Guillaume Quintard < >>>> guillaume at varnish-software.com> wrote: >>>> >>>>> without the vcl (and/or logs, but I'm assuming those are even more >>>>> confidential), that's a tough cookie to debug >>>>> -- >>>>> Guillaume Quintard >>>>> >>>>> >>>>> On Mon, Sep 24, 2018 at 10:35 AM Junaid Mukhtar < >>>>> junaid.mukhtar at gmail.com> wrote: >>>>> >>>>>> Hi >>>>>> >>>>>> Apologies but I am not allowed to share the VCL. If there's any other >>>>>> thing like stats etc that might be helpful then do let me know >>>>>> >>>>>> -------- >>>>>> Regards, >>>>>> Junaid >>>>>> >>>>>> >>>>>> On Fri, Sep 21, 2018 at 4:15 PM Guillaume Quintard < >>>>>> guillaume at varnish-software.com> wrote: >>>>>> >>>>>>> would you be able to share your vcl? >>>>>>> -- >>>>>>> Guillaume Quintard >>>>>>> >>>>>>> >>>>>>> On Fri, Sep 21, 2018 at 4:54 PM Junaid Mukhtar < >>>>>>> junaid.mukhtar at gmail.com> wrote: >>>>>>> >>>>>>>> Any help guys; this is really getting to us..... >>>>>>>> >>>>>>>> -------- >>>>>>>> Regards, >>>>>>>> Junaid >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Sep 20, 2018 at 3:57 PM Junaid Mukhtar < >>>>>>>> junaid.mukhtar at gmail.com> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> This is what we get in varnishhist >>>>>>>>> >>>>>>>>> 280_ >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> | >>>>>>>>> 230_ | >>>>>>>>> | >>>>>>>>> | >>>>>>>>> | >>>>>>>>> | >>>>>>>>> 180_ | >>>>>>>>> || >>>>>>>>> || >>>>>>>>> ||| >>>>>>>>> ||| >>>>>>>>> 130_ ||| >>>>>>>>> ||| >>>>>>>>> |||||| >>>>>>>>> |||||| >>>>>>>>> |||||| >>>>>>>>> 80_ |||||| >>>>>>>>> >>>>>>>>> |||||| # >>>>>>>>> >>>>>>>>> |||||| ### >>>>>>>>> >>>>>>>>> ||||||| # ### >>>>>>>>> >>>>>>>>> ||||||||| ##### # >>>>>>>>> 30_ >>>>>>>>> ||||||||| ######## >>>>>>>>> >>>>>>>>> ||||||||| >>>>>>>>> ######### >>>>>>>>> >>>>>>>>> |||||||||| >>>>>>>>> ########### >>>>>>>>> >>>>>>>>> |||||||||||| ## # >>>>>>>>> ############# # >>>>>>>>> >>>>>>>>> +---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+---------------------+--------------------- >>>>>>>>> |1e-6 |1e-5 |1e-4 >>>>>>>>> |1e-3 |1e-2 |1e-1 >>>>>>>>> |1e0 |1e1 |1e2 >>>>>>>>> -------- >>>>>>>>> Regards, >>>>>>>>> Junaid >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Sep 20, 2018 at 3:54 PM Junaid Mukhtar < >>>>>>>>> junaid.mukhtar at gmail.com> wrote: >>>>>>>>> >>>>>>>>>> Below is the output; all of them are default except i was trying >>>>>>>>>> to up the threads_pool to 4 but didn't feel any imporvement in performance >>>>>>>>>> degradation >>>>>>>>>> >>>>>>>>>> accept_filter off [bool] (default) >>>>>>>>>> acceptor_sleep_decay 0.9 (default) >>>>>>>>>> acceptor_sleep_incr 0.000 [seconds] (default) >>>>>>>>>> acceptor_sleep_max 0.050 [seconds] (default) >>>>>>>>>> auto_restart on [bool] (default) >>>>>>>>>> backend_idle_timeout 60.000 [seconds] (default) >>>>>>>>>> ban_dups on [bool] (default) >>>>>>>>>> ban_lurker_age 60.000 [seconds] (default) >>>>>>>>>> ban_lurker_batch 1000 (default) >>>>>>>>>> ban_lurker_sleep 0.010 [seconds] (default) >>>>>>>>>> between_bytes_timeout 60.000 [seconds] (default) >>>>>>>>>> cc_command "exec gcc -std=gnu99 -O2 -g >>>>>>>>>> -Wp,-D_FORTIFY_SOURCE=0 -Wall -Werror -pthread -fpic -shared -Wl,-x -o %o >>>>>>>>>> %s" (default) >>>>>>>>>> cli_buffer 8k [bytes] (default) >>>>>>>>>> cli_limit 48k [bytes] (default) >>>>>>>>>> cli_timeout 60.000 [seconds] (default) >>>>>>>>>> clock_skew 10 [seconds] (default) >>>>>>>>>> clock_step 1.000 [seconds] (default) >>>>>>>>>> connect_timeout 3.500 [seconds] (default) >>>>>>>>>> critbit_cooloff 180.000 [seconds] (default) >>>>>>>>>> debug none (default) >>>>>>>>>> default_grace 10.000 [seconds] (default) >>>>>>>>>> default_keep 0.000 [seconds] (default) >>>>>>>>>> default_ttl 120.000 [seconds] (default) >>>>>>>>>> feature none (default) >>>>>>>>>> fetch_chunksize 16k [bytes] (default) >>>>>>>>>> fetch_maxchunksize 0.25G [bytes] (default) >>>>>>>>>> first_byte_timeout 60.000 [seconds] (default) >>>>>>>>>> gzip_buffer 32k [bytes] (default) >>>>>>>>>> gzip_level 6 (default) >>>>>>>>>> gzip_memlevel 8 (default) >>>>>>>>>> http_gzip_support on [bool] (default) >>>>>>>>>> http_max_hdr 64 [header lines] (default) >>>>>>>>>> http_range_support on [bool] (default) >>>>>>>>>> http_req_hdr_len 8k [bytes] (default) >>>>>>>>>> http_req_size 32k [bytes] (default) >>>>>>>>>> http_resp_hdr_len 8k [bytes] (default) >>>>>>>>>> http_resp_size 32k [bytes] (default) >>>>>>>>>> idle_send_timeout 60.000 [seconds] (default) >>>>>>>>>> listen_depth 1024 [connections] (default) >>>>>>>>>> lru_interval 2.000 [seconds] (default) >>>>>>>>>> max_esi_depth 5 [levels] (default) >>>>>>>>>> max_restarts 4 [restarts] (default) >>>>>>>>>> max_retries 4 [retries] (default) >>>>>>>>>> nuke_limit 50 [allocations] (default) >>>>>>>>>> pcre_match_limit 10000 (default) >>>>>>>>>> pcre_match_limit_recursion 20 (default) >>>>>>>>>> ping_interval 3 [seconds] (default) >>>>>>>>>> pipe_timeout 60.000 [seconds] (default) >>>>>>>>>> pool_req 10,100,10 (default) >>>>>>>>>> pool_sess 10,100,10 (default) >>>>>>>>>> pool_vbo 10,100,10 (default) >>>>>>>>>> prefer_ipv6 off [bool] (default) >>>>>>>>>> rush_exponent 3 [requests per request] (default) >>>>>>>>>> send_timeout 600.000 [seconds] (default) >>>>>>>>>> session_max 100000 [sessions] (default) >>>>>>>>>> shm_reclen 255b [bytes] (default) >>>>>>>>>> shortlived 10.000 [seconds] (default) >>>>>>>>>> sigsegv_handler on [bool] (default) >>>>>>>>>> syslog_cli_traffic on [bool] (default) >>>>>>>>>> tcp_fastopen off [bool] (default) >>>>>>>>>> tcp_keepalive_intvl 75.000 [seconds] (default) >>>>>>>>>> tcp_keepalive_probes 9 [probes] (default) >>>>>>>>>> tcp_keepalive_time 7200.000 [seconds] (default) >>>>>>>>>> thread_pool_add_delay 0.000 [seconds] (default) >>>>>>>>>> thread_pool_destroy_delay 1.000 [seconds] (default) >>>>>>>>>> thread_pool_fail_delay 0.200 [seconds] (default) >>>>>>>>>> thread_pool_max 5000 [threads] (default) >>>>>>>>>> thread_pool_min 100 [threads] (default) >>>>>>>>>> thread_pool_reserve 0 [threads] (default) >>>>>>>>>> thread_pool_stack 48k [bytes] (default) >>>>>>>>>> thread_pool_timeout 300.000 [seconds] (default) >>>>>>>>>> thread_pools 2 [pools] (default) >>>>>>>>>> thread_queue_limit 20 (default) >>>>>>>>>> thread_stats_rate 10 [requests] (default) >>>>>>>>>> timeout_idle 5.000 [seconds] (default) >>>>>>>>>> timeout_linger 0.050 [seconds] (default) >>>>>>>>>> vcc_allow_inline_c off [bool] (default) >>>>>>>>>> vcc_err_unref on [bool] (default) >>>>>>>>>> vcc_unsafe_path on [bool] (default) >>>>>>>>>> vcl_cooldown 600.000 [seconds] (default) >>>>>>>>>> vcl_dir /etc/varnish (default) >>>>>>>>>> vmod_dir /usr/lib64/varnish/vmods (default) >>>>>>>>>> vsl_buffer 4k [bytes] (default) >>>>>>>>>> vsl_mask -VCL_trace,-WorkThread,-Hash,-VfpAcct >>>>>>>>>> (default) >>>>>>>>>> vsl_reclen 255b [bytes] (default) >>>>>>>>>> vsl_space 80M [bytes] (default) >>>>>>>>>> vsm_free_cooldown 60.000 [seconds] (default) >>>>>>>>>> vsm_space 1M [bytes] (default) >>>>>>>>>> workspace_backend 64k [bytes] (default) >>>>>>>>>> workspace_client 64k [bytes] (default) >>>>>>>>>> workspace_session 0.50k [bytes] (default) >>>>>>>>>> workspace_thread 2k [bytes] (default) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -------- >>>>>>>>>> Regards, >>>>>>>>>> Junaid >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Sep 20, 2018 at 3:37 PM Dridi Boukelmoune >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> On Thu, Sep 20, 2018 at 4:14 PM Junaid Mukhtar < >>>>>>>>>>> junaid.mukhtar at gmail.com> wrote: >>>>>>>>>>> > >>>>>>>>>>> > Hi >>>>>>>>>>> > >>>>>>>>>>> > we are in middle of upgrading from varnish 3.0.7 to varnish >>>>>>>>>>> 4.1.10; but unfortunately all of the response times in the performance test >>>>>>>>>>> are indicating an increase of at least 100% >>>>>>>>>>> > >>>>>>>>>>> > We have analyzed the logs and everything but can't get around >>>>>>>>>>> this; the issue is not only for the non-cached pages but it's also >>>>>>>>>>> impacting the cached pages. We are also struggling to analyze it further as >>>>>>>>>>> well, any guidenace as to what we can do in terms of putting debug logging >>>>>>>>>>> in would be helpful. >>>>>>>>>>> > >>>>>>>>>>> > One of the areas we are considering is to output response >>>>>>>>>>> times, but it's proving difficult for me. Any suggestions >>>>>>>>>>> >>>>>>>>>>> Hello, >>>>>>>>>>> >>>>>>>>>>> What is the output of `varnishadm param.show` ? >>>>>>>>>>> >>>>>>>>>>> Dridi >>>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>> varnish-misc mailing list >>>>>>>> varnish-misc at varnish-cache.org >>>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>>>> >>>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From twbecker at gmail.com Fri Sep 28 01:08:00 2018 From: twbecker at gmail.com (Tommy Becker) Date: Thu, 27 Sep 2018 21:08:00 -0400 Subject: Varnish swallowing 4xx responses from POSTs In-Reply-To: References: <6DBBA006-52E8-4927-8E64-663B9678A514@gmail.com> Message-ID: <37E704A2-8925-40DC-829A-F5E0110F8905@gmail.com> Hi Dridi, Thanks for the response. I?m curious what specifically you believe to be in violation of the spec here. There?s a lot of ambiguity to be had by my read, but the option to send responses at any point seems pretty clear. From RFC 7230 Section 6.5 A client sending a message body SHOULD monitor the network connection for an error response while it is transmitting the request. If the client sees a response that indicates the server does not wish to receive the message body and is closing the connection, the client SHOULD immediately cease transmitting the body and close its side of the connection. I should point out I initiated a thread on the Jetty mailing list on this same topic prior to this one, and they (perhaps unsurprisingly) defend this behavior. Greg Wilkins of the Jetty team asked me to relay this message in particular: https://www.eclipse.org/lists/jetty-users/msg08611.html As I mentioned in that thread, I have no horse in this race and just want to solve my problem and perhaps spare others from this same issue, which was rather tough to debug. Tommy > On Sep 27, 2018, at 9:40 AM, Dridi Boukelmoune wrote: > > Hello, > > On Wed, Sep 26, 2018 at 2:44 AM Tommy Becker wrote: >> >> We have an application that we front with Varnish 4.0.5. Recently, after an application upgrade in which we migrated from Jetty 9.2 to 9.4, we began noticing a lot of 503s being returned from Varnish on POST requests. We have an endpoint that takes a payload of a potentially large amount of JSON, and validates it as it?s being read. What we have discovered is that if there is a problem with the content, we correctly return a 400 Bad Request from Jetty. Notably, this can happen before the entire content is received. When this happens, Varnish continues to send the remainder of data, despite having already seen the response. Now after our upgrade, Jetty's behavior is to send a TCP RST when this happens (since the data is unwanted by the application). Unfortunately, Varnish interprets the RST as a backend error, and goes to vcl_backend_error, having never sent the original response returned from Jetty to the client. So instead of seeing a 400 Bad Request with a helpful message, they simply get 503 Service Unavailable. > > I'm pretty certain that this optimization does not comply with the > HTTP/1 specs. Even though Jetty is trying to improve the latency by > replying early, as far as Varnish is concerned it failed to send the > full request and won't bother reading the response. > >> I found this issue which seems similar: https://github.com/varnishcache/varnish-cache/issues/2332 Can someone help here? Is there anyway to work around this behavior? > > In this case that's different because the backend is inspecting the > body as it is coming and not rejecting the request based on the size > only. So I'm afraid there's no way to work around this behavior. > > As this is not a bug, we could introduce either a feature flag or a > VCL variable turned off by default to tolerate an early reset of the > request side of an HTTP/1 socket. > > You could join next bugwash on Monday to bring this to the team's > attention. > > Dridi -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Fri Sep 28 10:12:23 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 28 Sep 2018 12:12:23 +0200 Subject: Varnish swallowing 4xx responses from POSTs In-Reply-To: <37E704A2-8925-40DC-829A-F5E0110F8905@gmail.com> References: <6DBBA006-52E8-4927-8E64-663B9678A514@gmail.com> <37E704A2-8925-40DC-829A-F5E0110F8905@gmail.com> Message-ID: On Fri, Sep 28, 2018 at 3:08 AM Tommy Becker wrote: > > Hi Dridi, > Thanks for the response. I?m curious what specifically you believe to be in violation of the spec here. There?s a lot of ambiguity to be had by my read, but the option to send responses at any point seems pretty clear. From RFC 7230 Section 6.5 > > A client sending a message body SHOULD monitor the network connection > for an error response while it is transmitting the request. If the > client sees a response that indicates the server does not wish to > receive the message body and is closing the connection, the client > SHOULD immediately cease transmitting the body and close its side of > the connection. That's a SHOULD I didn't remember, that's why ;) > I should point out I initiated a thread on the Jetty mailing list on this same topic prior to this one, and they (perhaps unsurprisingly) defend this behavior. Greg Wilkins of the Jetty team asked me to relay this message in particular: https://www.eclipse.org/lists/jetty-users/msg08611.html Thank you for relaying it. In the github issue you mentioned earlier I already suggested 100-continue support (mentioned by Greg Wilkins) in Varnish, but I wasn't aware/didn't remember the section you pasted. > As I mentioned in that thread, I have no horse in this race and just want to solve my problem and perhaps spare others from this same issue, which was rather tough to debug. Yes, and thank you for posting to the list. So this is neither a bug in Varnish or Jetty, and we don't use Github issues for feature requests. I will try to bring this up on Monday for the bugwash since this feature would be a latency win. Dridi From twbecker at gmail.com Fri Sep 28 11:45:19 2018 From: twbecker at gmail.com (Tommy Becker) Date: Fri, 28 Sep 2018 07:45:19 -0400 Subject: Varnish swallowing 4xx responses from POSTs In-Reply-To: References: <6DBBA006-52E8-4927-8E64-663B9678A514@gmail.com> <37E704A2-8925-40DC-829A-F5E0110F8905@gmail.com> Message-ID: Thanks! One last thing I will point out, is that in our case, 100-continue doesn't help since we're responding due to a problem with the body itself. My understanding of 100-continue is that it essentially okays the sender to transmit the body. On Fri, Sep 28, 2018 at 6:13 AM Dridi Boukelmoune wrote: > On Fri, Sep 28, 2018 at 3:08 AM Tommy Becker wrote: > > > > Hi Dridi, > > Thanks for the response. I?m curious what specifically you believe to be > in violation of the spec here. There?s a lot of ambiguity to be had by my > read, but the option to send responses at any point seems pretty clear. > From RFC 7230 Section 6.5 > > > > A client sending a message body SHOULD monitor the network connection > > for an error response while it is transmitting the request. If the > > client sees a response that indicates the server does not wish to > > receive the message body and is closing the connection, the client > > SHOULD immediately cease transmitting the body and close its side of > > the connection. > > That's a SHOULD I didn't remember, that's why ;) > > > I should point out I initiated a thread on the Jetty mailing list on > this same topic prior to this one, and they (perhaps unsurprisingly) defend > this behavior. Greg Wilkins of the Jetty team asked me to relay this > message in particular: > https://www.eclipse.org/lists/jetty-users/msg08611.html > > Thank you for relaying it. In the github issue you mentioned earlier I > already suggested 100-continue support (mentioned by Greg Wilkins) in > Varnish, but I wasn't aware/didn't remember the section you pasted. > > > As I mentioned in that thread, I have no horse in this race and just > want to solve my problem and perhaps spare others from this same issue, > which was rather tough to debug. > > Yes, and thank you for posting to the list. So this is neither a bug > in Varnish or Jetty, and we don't use Github issues for feature > requests. I will try to bring this up on Monday for the bugwash since > this feature would be a latency win. > > Dridi > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Fri Sep 28 12:51:50 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 28 Sep 2018 14:51:50 +0200 Subject: Varnish swallowing 4xx responses from POSTs In-Reply-To: References: <6DBBA006-52E8-4927-8E64-663B9678A514@gmail.com> <37E704A2-8925-40DC-829A-F5E0110F8905@gmail.com> Message-ID: On Fri, Sep 28, 2018 at 1:45 PM Tommy Becker wrote: > > Thanks! One last thing I will point out, is that in our case, 100-continue doesn't help since we're responding due to a problem with the body itself. My understanding of 100-continue is that it essentially okays the sender to transmit the body. Correct, this is what I said in my first answer: > In this case that's different because the backend is inspecting the > body as it is coming and not rejecting the request based on the size > only. So I'm afraid there's no way to work around this behavior. Dridi