From nils.goroll at uplex.de Tue Sep 1 10:42:07 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 1 Sep 2020 10:42:07 +0000 (UTC) Subject: [master] 6127079b5 changelog tlc and a minor addition to the API log Message-ID: <20200901104207.A6BDDAA1FC@lists.varnish-cache.org> commit 6127079b50d2debd966ba9332ba9877d5c0071f2 Author: Nils Goroll Date: Tue Sep 1 12:13:01 2020 +0200 changelog tlc and a minor addition to the API log diff --git a/doc/changes.rst b/doc/changes.rst index a6ecf1d96..5a00c1fde 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -30,6 +30,11 @@ release process. NEXT (scheduled 2020-09-15) =========================== +[ABI] marks potentially breaking changes to binary compatibility. + +[API] marks potentially breaking changes to source compatibility +(implies [ABI]). + * ``varnishstat`` now has a help screen, available via the ``h`` key in curses mode @@ -46,7 +51,7 @@ NEXT (scheduled 2020-09-15) * ``varnishstat`` JSON format (``-j`` option) has been changed: * on the top level, a ``version`` identifier has been introduced, - which will be used to mark breaking changes to the JSON + which will be used to mark future breaking changes to the JSON formatting. It will not be used to mark changes to the counters themselves. @@ -59,7 +64,7 @@ NEXT (scheduled 2020-09-15) build a ``struct suckaddr`` * Depending on the setting of the new ``vcc_acl_pedantic`` parameter, - VCC now either emits or warning or fails if network numbers used in + VCC now either emits a warning or fails if network numbers used in ACLs do not have an all-zero host part. For ``vcc_acl_pedantic`` off, the host part is fixed to all-zero and @@ -69,7 +74,7 @@ NEXT (scheduled 2020-09-15) ``vcl_backend_response`` (3273_) * ``obj.can_esi`` has been added to identify if the response can be - ESI processed (See 3002_) + ESI processed (3002_) * ``resp.filters`` now contains a correct value when the auto-determined filter list is read (3002_) @@ -78,6 +83,35 @@ NEXT (scheduled 2020-09-15) ``beresp.do_`` fields which determine the filter list after setting ``resp.filters`` and ``beresp.filters``, respectively +* Behavior for 304 responses was changed not to update + the ``Content-Encoding`` response header of the stored object. + +* [ABI] ``struct vfp_entry`` and ``struct vdp_ctx`` changed + +* [API] VSB_QUOTE_GLOB, which was prematurely added to 6.4, has been + removed again. + +* [API] Add ``VDP_END`` action for delivery processors, which has to + be sent with or after the last buffer. + +* Respect the administrative health for "real" (VBE) backends (3299_) + +* Fixed handling of illegal (internal) four-digit response codes and + with HTTP/2 (3301_) + +* Fixed backend connection pooling of closed connections (3266_) + +* Added the ``.resolve`` method for the ``BACKEND`` type to resolve + (determine the "real" backend) a director. + +* Improved ``vmodtool`` support for out-of-tree builds + +* Added ``VJ_unlink()`` and ``VJ_rmdir()`` jail functions + +* Fixed workdir cleanup (3307_) + +* Added ``JAIL_MASTER_SYSTEM`` jail level + * The Varnish Jail (least privileges) code for Solaris has been largely rewritten. It now reduces privileges even further and thus should improve the security of Varnish on Solaris even more. @@ -89,12 +123,105 @@ NEXT (scheduled 2020-09-15) * The shard director and shard director parameter objects should now work in ``vcl_pipe {}`` like in ``vcl_backend_* {}`` subs. +* For a failure in ``vcl_recv {}``, the VCL state engine now returns + right after return from that subroutine. (3303_) + +* The shard director now supports weights by scaling the number of + replicas of each backend on the consistent hashing ring + +* Fixed a race in the cache expiry code which could lead to a panic (2999_) + +* Added ``VRE_quote()`` to facilitate building literal string matches + with regular expressions. + +* The ``BackendReuse`` VSL (log) tag has been retired and replaced + with ``BackendClose``, which has been changed to contain either + ``close`` or ``recycle`` to signify whether the connection was + closed or returned to a pool for later reuse. + +* ``BackendOpen`` VSL entries have been changed to contain ``reuse`` + or ``connect`` in the last column to signify whether the connection + was reused from a pool or newly opened. + +* ``std.rollback()`` of backend requests with ``return(retry)`` has + been fixed (3353_) + +* ``FetchError`` logs now differentiate between ``No backend`` and + "none resolved" as ``Director %s returned no backend`` + +* Added ``VRT_DirectorResolve()`` to resolve a director + +* Improved VCC handling of symbols and, in particular, type methods + +* Fixed use of the shard director from ``vcl_pipe {}`` (3361_) + +* Handle recursive use of vcl ``include`` (3360_) + +* VCL: Added native support for BLOBs in structured fields notation + (``::``) + +* Fixed handling of the ``Connection:`` header when multiple instances + of the named headers existed. + +* Added support for naming ``PRIV_`` arguments to vmod methods/functions + +* The varnish binary heap implementation has been renamed to use the + ``VBH_`` prefix, complemented with a destructor and added to header + files for use with vmods (via include of ``vbh.h``). + +* A bug in ``vmod_blob`` for base64 decoding with a ``length`` + argument and non-padding decoding has been fixed (3378_) + +* Added ``VRT_BLOB_string()`` to ``vrt.h`` + +* VSB support for dynamic vs. static allocations has been changed: + + For dynamic allocations use:: + + VSB_new_auto() + VSB_destroy() + + For preexisting buffers use:: + + VSB_init() + VSB_fini() + + ``VSB_new()`` + ``VSB_delete()`` are now deprecated. + +* ``std.blobread()`` has been added + +* New ``main.beresp_uncacheable`` and ``main.beresp_shortlived`` + counters have been added. + +* The ``I``, ``X`` and ``R`` arguments have been added to the VSC API + and ``varnishstat`` for inclusion, exclusion and required glob + patterns on the statistic field names. (3394_) + +* [API] overhaul of the workspace API + + * The previously deprecated ``WS_Reserve()`` has been removed + * The signature of ``WS_Printf()`` has been changed + * Add ``WS_ReservationSize()`` + * ``WS_Front()`` replaced by ``WS_Reservation()`` + * Add ``WS_Id()`` + +* Handle a workspace overflow in ``VRY_Validate()`` (3319_) + .. _2990: https://github.com/varnishcache/varnish-cache/issues/2990 +.. _2999: https://github.com/varnishcache/varnish-cache/issues/2999 .. _3002: https://github.com/varnishcache/varnish-cache/issues/3002 .. _3241: https://github.com/varnishcache/varnish-cache/issues/3241 .. _3253: https://github.com/varnishcache/varnish-cache/issues/3253 +.. _3266: https://github.com/varnishcache/varnish-cache/issues/3266 .. _3273: https://github.com/varnishcache/varnish-cache/issues/3273 - +.. _3299: https://github.com/varnishcache/varnish-cache/issues/3299 +.. _3301: https://github.com/varnishcache/varnish-cache/issues/3301 +.. _3303: https://github.com/varnishcache/varnish-cache/issues/3303 +.. _3307: https://github.com/varnishcache/varnish-cache/issues/3307 +.. _3319: https://github.com/varnishcache/varnish-cache/issues/3319 +.. _3353: https://github.com/varnishcache/varnish-cache/issues/3353 +.. _3360: https://github.com/varnishcache/varnish-cache/issues/3360 +.. _3361: https://github.com/varnishcache/varnish-cache/issues/3361 +.. _3378: https://github.com/varnishcache/varnish-cache/issues/3378 +.. _3394: https://github.com/varnishcache/varnish-cache/issues/3394 ================================ Varnish Cache 6.4.0 (2020-03-16) diff --git a/include/vrt.h b/include/vrt.h index a99798c95..1095d6ab0 100644 --- a/include/vrt.h +++ b/include/vrt.h @@ -59,6 +59,7 @@ * [cache.h] WS_Printf() changed * [cache.h] WS_ReservationSize() added * [cache.h] WS_Front() replaced by WS_Reservation() + * [cache.h] WS_Id() added * 11.0 (2020-03-16) * Changed type of vsa_suckaddr_len from int to size_t * New prefix_{ptr|len} fields in vrt_backend From nils.goroll at uplex.de Tue Sep 1 10:42:07 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 1 Sep 2020 10:42:07 +0000 (UTC) Subject: [master] 60db0bd69 add back WS_Front() as deprecated for now Message-ID: <20200901104207.BF289AA1FF@lists.varnish-cache.org> commit 60db0bd69d7e932a82d3f31bcde9da4b9907bc7d Author: Nils Goroll Date: Tue Sep 1 12:40:30 2020 +0200 add back WS_Front() as deprecated for now diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index b3ae30c99..bd849d621 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -800,6 +800,15 @@ void WS_Id(const struct ws *ws, char *id); void WS_VSB_new(struct vsb *, struct ws *); char *WS_VSB_finish(struct vsb *, struct ws *, size_t *); +/* REL_20210315 remove */ +static inline char* WS_Front(const struct ws *) v_deprecated_; +static inline char* +WS_Front(const struct ws *ws) +{ + AN(ws->r); + return ws->f; +} + static inline void * WS_Reservation(const struct ws *ws) { diff --git a/doc/changes.rst b/doc/changes.rst index 5a00c1fde..1ede0021e 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -195,12 +195,14 @@ NEXT (scheduled 2020-09-15) and ``varnishstat`` for inclusion, exclusion and required glob patterns on the statistic field names. (3394_) -* [API] overhaul of the workspace API +* Overhaul of the workspace API * The previously deprecated ``WS_Reserve()`` has been removed - * The signature of ``WS_Printf()`` has been changed + * The signature of ``WS_Printf()`` has been changed to return + ``const char *`` instead of ``void *`` (we do not consider this a + breaking change). * Add ``WS_ReservationSize()`` - * ``WS_Front()`` replaced by ``WS_Reservation()`` + * ``WS_Front()`` is now deprecated and replaced by ``WS_Reservation()`` * Add ``WS_Id()`` * Handle a workspace overflow in ``VRY_Validate()`` (3319_) diff --git a/include/vrt.h b/include/vrt.h index 1095d6ab0..2e5cc2d7d 100644 --- a/include/vrt.h +++ b/include/vrt.h @@ -58,7 +58,8 @@ * [cache.h] WS_Reserve() removed * [cache.h] WS_Printf() changed * [cache.h] WS_ReservationSize() added - * [cache.h] WS_Front() replaced by WS_Reservation() + * [cache.h] WS_Front() deprecated + * [cache.h] WS_Reservation() added * [cache.h] WS_Id() added * 11.0 (2020-03-16) * Changed type of vsa_suckaddr_len from int to size_t From dridi at varni.sh Tue Sep 1 11:11:14 2020 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 1 Sep 2020 11:11:14 +0000 Subject: [master] 6127079b5 changelog tlc and a minor addition to the API log In-Reply-To: <20200901104207.A6BDDAA1FC@lists.varnish-cache.org> References: <20200901104207.A6BDDAA1FC@lists.varnish-cache.org> Message-ID: On Tue, Sep 1, 2020 at 10:42 AM Nils Goroll wrote: > > > commit 6127079b50d2debd966ba9332ba9877d5c0071f2 > Author: Nils Goroll > Date: Tue Sep 1 12:13:01 2020 +0200 > > changelog tlc and a minor addition to the API log > > diff --git a/doc/changes.rst b/doc/changes.rst > index a6ecf1d96..5a00c1fde 100644 > --- a/doc/changes.rst > +++ b/doc/changes.rst > @@ -30,6 +30,11 @@ release process. > NEXT (scheduled 2020-09-15) > =========================== > > +[ABI] marks potentially breaking changes to binary compatibility. > + > +[API] marks potentially breaking changes to source compatibility > +(implies [ABI]). Not necessarily, if you rename a field you break the API but not the ABI for example. If you add fields to a struct that is not supposed to be stored, and only pointed to, without relying on the struct size, appending the new fields may not break. > * ``varnishstat`` now has a help screen, available via the ``h`` key > in curses mode > > @@ -46,7 +51,7 @@ NEXT (scheduled 2020-09-15) > * ``varnishstat`` JSON format (``-j`` option) has been changed: > > * on the top level, a ``version`` identifier has been introduced, > - which will be used to mark breaking changes to the JSON > + which will be used to mark future breaking changes to the JSON > formatting. It will not be used to mark changes to the counters > themselves. > > @@ -59,7 +64,7 @@ NEXT (scheduled 2020-09-15) > build a ``struct suckaddr`` > > * Depending on the setting of the new ``vcc_acl_pedantic`` parameter, > - VCC now either emits or warning or fails if network numbers used in > + VCC now either emits a warning or fails if network numbers used in > ACLs do not have an all-zero host part. > > For ``vcc_acl_pedantic`` off, the host part is fixed to all-zero and > @@ -69,7 +74,7 @@ NEXT (scheduled 2020-09-15) > ``vcl_backend_response`` (3273_) > > * ``obj.can_esi`` has been added to identify if the response can be > - ESI processed (See 3002_) > + ESI processed (3002_) > > * ``resp.filters`` now contains a correct value when the > auto-determined filter list is read (3002_) > @@ -78,6 +83,35 @@ NEXT (scheduled 2020-09-15) > ``beresp.do_`` fields which determine the filter list after setting > ``resp.filters`` and ``beresp.filters``, respectively > > +* Behavior for 304 responses was changed not to update > + the ``Content-Encoding`` response header of the stored object. > + > +* [ABI] ``struct vfp_entry`` and ``struct vdp_ctx`` changed > + > +* [API] VSB_QUOTE_GLOB, which was prematurely added to 6.4, has been > + removed again. > + > +* [API] Add ``VDP_END`` action for delivery processors, which has to > + be sent with or after the last buffer. > + > +* Respect the administrative health for "real" (VBE) backends (3299_) > + > +* Fixed handling of illegal (internal) four-digit response codes and > + with HTTP/2 (3301_) > + > +* Fixed backend connection pooling of closed connections (3266_) > + > +* Added the ``.resolve`` method for the ``BACKEND`` type to resolve > + (determine the "real" backend) a director. > + > +* Improved ``vmodtool`` support for out-of-tree builds > + > +* Added ``VJ_unlink()`` and ``VJ_rmdir()`` jail functions > + > +* Fixed workdir cleanup (3307_) > + > +* Added ``JAIL_MASTER_SYSTEM`` jail level > + > * The Varnish Jail (least privileges) code for Solaris has been > largely rewritten. It now reduces privileges even further and thus > should improve the security of Varnish on Solaris even more. > @@ -89,12 +123,105 @@ NEXT (scheduled 2020-09-15) > * The shard director and shard director parameter objects should now > work in ``vcl_pipe {}`` like in ``vcl_backend_* {}`` subs. > > +* For a failure in ``vcl_recv {}``, the VCL state engine now returns > + right after return from that subroutine. (3303_) > + > +* The shard director now supports weights by scaling the number of > + replicas of each backend on the consistent hashing ring > + > +* Fixed a race in the cache expiry code which could lead to a panic (2999_) > + > +* Added ``VRE_quote()`` to facilitate building literal string matches > + with regular expressions. > + > +* The ``BackendReuse`` VSL (log) tag has been retired and replaced > + with ``BackendClose``, which has been changed to contain either > + ``close`` or ``recycle`` to signify whether the connection was > + closed or returned to a pool for later reuse. > + > +* ``BackendOpen`` VSL entries have been changed to contain ``reuse`` > + or ``connect`` in the last column to signify whether the connection > + was reused from a pool or newly opened. > + > +* ``std.rollback()`` of backend requests with ``return(retry)`` has > + been fixed (3353_) > + > +* ``FetchError`` logs now differentiate between ``No backend`` and > + "none resolved" as ``Director %s returned no backend`` > + > +* Added ``VRT_DirectorResolve()`` to resolve a director > + > +* Improved VCC handling of symbols and, in particular, type methods > + > +* Fixed use of the shard director from ``vcl_pipe {}`` (3361_) > + > +* Handle recursive use of vcl ``include`` (3360_) > + > +* VCL: Added native support for BLOBs in structured fields notation > + (``::``) > + > +* Fixed handling of the ``Connection:`` header when multiple instances > + of the named headers existed. > + > +* Added support for naming ``PRIV_`` arguments to vmod methods/functions > + > +* The varnish binary heap implementation has been renamed to use the > + ``VBH_`` prefix, complemented with a destructor and added to header > + files for use with vmods (via include of ``vbh.h``). > + > +* A bug in ``vmod_blob`` for base64 decoding with a ``length`` > + argument and non-padding decoding has been fixed (3378_) > + > +* Added ``VRT_BLOB_string()`` to ``vrt.h`` > + > +* VSB support for dynamic vs. static allocations has been changed: > + > + For dynamic allocations use:: > + > + VSB_new_auto() + VSB_destroy() > + > + For preexisting buffers use:: > + > + VSB_init() + VSB_fini() > + > + ``VSB_new()`` + ``VSB_delete()`` are now deprecated. > + > +* ``std.blobread()`` has been added > + > +* New ``main.beresp_uncacheable`` and ``main.beresp_shortlived`` > + counters have been added. > + > +* The ``I``, ``X`` and ``R`` arguments have been added to the VSC API > + and ``varnishstat`` for inclusion, exclusion and required glob > + patterns on the statistic field names. (3394_) > + > +* [API] overhaul of the workspace API > + > + * The previously deprecated ``WS_Reserve()`` has been removed > + * The signature of ``WS_Printf()`` has been changed > + * Add ``WS_ReservationSize()`` > + * ``WS_Front()`` replaced by ``WS_Reservation()`` > + * Add ``WS_Id()`` > + > +* Handle a workspace overflow in ``VRY_Validate()`` (3319_) > + > .. _2990: https://github.com/varnishcache/varnish-cache/issues/2990 > +.. _2999: https://github.com/varnishcache/varnish-cache/issues/2999 > .. _3002: https://github.com/varnishcache/varnish-cache/issues/3002 > .. _3241: https://github.com/varnishcache/varnish-cache/issues/3241 > .. _3253: https://github.com/varnishcache/varnish-cache/issues/3253 > +.. _3266: https://github.com/varnishcache/varnish-cache/issues/3266 > .. _3273: https://github.com/varnishcache/varnish-cache/issues/3273 > - > +.. _3299: https://github.com/varnishcache/varnish-cache/issues/3299 > +.. _3301: https://github.com/varnishcache/varnish-cache/issues/3301 > +.. _3303: https://github.com/varnishcache/varnish-cache/issues/3303 > +.. _3307: https://github.com/varnishcache/varnish-cache/issues/3307 > +.. _3319: https://github.com/varnishcache/varnish-cache/issues/3319 > +.. _3353: https://github.com/varnishcache/varnish-cache/issues/3353 > +.. _3360: https://github.com/varnishcache/varnish-cache/issues/3360 > +.. _3361: https://github.com/varnishcache/varnish-cache/issues/3361 > +.. _3378: https://github.com/varnishcache/varnish-cache/issues/3378 > +.. _3394: https://github.com/varnishcache/varnish-cache/issues/3394 > > ================================ > Varnish Cache 6.4.0 (2020-03-16) > diff --git a/include/vrt.h b/include/vrt.h > index a99798c95..1095d6ab0 100644 > --- a/include/vrt.h > +++ b/include/vrt.h > @@ -59,6 +59,7 @@ > * [cache.h] WS_Printf() changed > * [cache.h] WS_ReservationSize() added > * [cache.h] WS_Front() replaced by WS_Reservation() > + * [cache.h] WS_Id() added > * 11.0 (2020-03-16) > * Changed type of vsa_suckaddr_len from int to size_t > * New prefix_{ptr|len} fields in vrt_backend > _______________________________________________ > varnish-commit mailing list > varnish-commit at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-commit From nils.goroll at uplex.de Tue Sep 1 11:15:45 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 1 Sep 2020 13:15:45 +0200 Subject: [master] 6127079b5 changelog tlc and a minor addition to the API log In-Reply-To: References: <20200901104207.A6BDDAA1FC@lists.varnish-cache.org> Message-ID: <47b9435c-db2f-3e01-6dcb-fe2cce9accd2@uplex.de> On 01/09/2020 13:11, Dridi Boukelmoune wrote: > Not necessarily, if you rename a field you break the API but not the > ABI for example. If you add fields to a struct that is not supposed to > be stored, and only pointed to, without relying on the struct size, > appending the new fields may not break. I agree. But I checked the two API cases which I have documented and think that they also break ABI, so I simplified. -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg tel +49 40 28805731 mob +49 170 2723133 fax +49 40 42949753 xmpp://slink at jabber.int.uplex.de/ http://uplex.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From dridi.boukelmoune at gmail.com Tue Sep 1 15:40:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 1 Sep 2020 15:40:06 +0000 (UTC) Subject: [master] 703e57e99 config.h must be the first include Message-ID: <20200901154006.EDB0E5FC2@lists.varnish-cache.org> commit 703e57e99ced24ddf5f2e868358d30e96d9a28ac Author: Dridi Boukelmoune Date: Tue Sep 1 17:35:37 2020 +0200 config.h must be the first include It may define macros recognized by other includes. diff --git a/bin/varnishtest/vtc_sess.c b/bin/varnishtest/vtc_sess.c index ae2284687..11328c23c 100644 --- a/bin/varnishtest/vtc_sess.c +++ b/bin/varnishtest/vtc_sess.c @@ -1,11 +1,11 @@ +#include "config.h" + #include #include #include -#include "config.h" - #include "vtc.h" struct vtc_sess { From dridi.boukelmoune at gmail.com Tue Sep 1 15:40:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 1 Sep 2020 15:40:07 +0000 (UTC) Subject: [master] 27aa059db Missing copyright notice Message-ID: <20200901154007.17D395FC5@lists.varnish-cache.org> commit 27aa059db90c6756b14c655935682c577b0c7a58 Author: Dridi Boukelmoune Date: Tue Sep 1 17:38:58 2020 +0200 Missing copyright notice diff --git a/bin/varnishtest/vtc_sess.c b/bin/varnishtest/vtc_sess.c index 11328c23c..1b06b81dd 100644 --- a/bin/varnishtest/vtc_sess.c +++ b/bin/varnishtest/vtc_sess.c @@ -1,3 +1,33 @@ +/*- + * Copyright (c) 2020 Varnish Software AS + * All rights reserved. + * + * Author: Poul-Henning Kamp + * + * SPDX-License-Identifier: BSD-2-Clause + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ #include "config.h" From dridi.boukelmoune at gmail.com Tue Sep 1 20:05:09 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 1 Sep 2020 20:05:09 +0000 (UTC) Subject: [master] d5c072da9 Fake WS_Assert() in esi_parse_fuzzer.c Message-ID: <20200901200509.68D1363A68@lists.varnish-cache.org> commit d5c072da9f37dd929dc1ba8d8698d84442fb2ffb Author: Dridi Boukelmoune Date: Tue Sep 1 22:03:27 2020 +0200 Fake WS_Assert() in esi_parse_fuzzer.c diff --git a/bin/varnishd/fuzzers/esi_parse_fuzzer.c b/bin/varnishd/fuzzers/esi_parse_fuzzer.c index 0a1b9cc76..b971280e2 100644 --- a/bin/varnishd/fuzzers/esi_parse_fuzzer.c +++ b/bin/varnishd/fuzzers/esi_parse_fuzzer.c @@ -68,6 +68,12 @@ VSLb_ts(struct vsl_log *l, const char *event, vtim_real first, vtim_real *pprev, (void)now; } +void +WS_Assert(const struct ws *ws) +{ + (void)ws; +} + void * WS_Alloc(struct ws *ws, unsigned bytes) { From nils.goroll at uplex.de Sat Sep 5 15:56:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Sat, 5 Sep 2020 15:56:06 +0000 (UTC) Subject: [master] 7872e8179 Document correctly vcl_hit built-in VCL Message-ID: <20200905155606.80A36A9647@lists.varnish-cache.org> commit 7872e81793317246d52b79760b64b5001c537ad6 Author: P?l Hermunn Johansen Date: Fri Sep 4 21:22:27 2020 +0200 Document correctly vcl_hit built-in VCL It has been a while since the built-in vcl_hit changed, and this updates the user guide. Since 6.0.1 we have had the corresponding change in TTL, grace, keep change, and this is also mentioned. diff --git a/doc/sphinx/users-guide/vcl-grace.rst b/doc/sphinx/users-guide/vcl-grace.rst index fcaa301bb..afaac2ce4 100644 --- a/doc/sphinx/users-guide/vcl-grace.rst +++ b/doc/sphinx/users-guide/vcl-grace.rst @@ -90,31 +90,19 @@ new object is in it will replace the one we've already got. If you do not define your own ``sub vcl_hit``, then the default one is used. It looks like this:: - sub vcl_hit { - if (obj.ttl >= 0s) { - // A pure unadulterated hit, deliver it - return (deliver); - } - if (obj.ttl + obj.grace > 0s) { - // Object is in grace, deliver it - // Automatically triggers a background fetch - return (deliver); - } - // fetch & deliver once we get the result - return (miss); - } - -The effect of the built-in VCL is in fact equivalent to the following:: - sub vcl_hit { return (deliver); } -This is because ``obj.ttl + obj.grace > 0s`` always will evaluate to -true. However, the the VCL is as it is to show users how to -differentiate between a pure hit and a `grace` hit. With the next -major version of Varnish, the default VCL is planned to change to the -latter, shorter version. +Note that the condition ``obj.ttl + obj.grace > 0s`` will (in ``sub +vcl_hit``) always evaluate to true. In earlier versions (6.0.0 and +earlier), this was not the case, and a test in the builtin VCL was +necessary to make sure that "keep objects" (objects in the cache where +both TTL and grace had run out) would not be delivered to the clients. + +In the current version, when there are only "keep objects" available, +``sub vcl_miss`` will be called, and a fetch for a new object will be +initiated. Misbehaving servers ~~~~~~~~~~~~~~~~~~~ From phk at FreeBSD.org Mon Sep 7 06:22:08 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 7 Sep 2020 06:22:08 +0000 (UTC) Subject: [master] fe8602b42 Return Flexelint to sanity wrt. Reserved WSs. Message-ID: <20200907062208.18450625D5@lists.varnish-cache.org> commit fe8602b42d98f56379b53eed6ab55ff53984ce10 Author: Poul-Henning Kamp Date: Mon Sep 7 06:17:36 2020 +0000 Return Flexelint to sanity wrt. Reserved WSs. Give WS_Reservation() the same "always return non-NULL or assert" semantics WS_Front() had because literally every single caller which uses the ->f pointer fails to check for NULL. Introduce a new internal WS_IsReserved() for use in the asserts which check if a reservation is active, but which doesn't otherwise care for the ->f pointer, and the single instance of non-WS code which (possibly) legitimatly does variant processing depending on the reservation state. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index bd849d621..32c4972fa 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -814,7 +814,9 @@ WS_Reservation(const struct ws *ws) { WS_Assert(ws); - return (ws->r != NULL ? ws->f : NULL); + AN(ws->r); + AN(ws->f); + return (ws->f); } static inline unsigned diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index 6284b9f6b..c6e4218f4 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -1127,6 +1127,6 @@ CNT_Request(struct req *req) VRB_Free(req); req->wrk = NULL; } - assert(nxt == REQ_FSM_DISEMBARK || WS_Reservation(req->ws) == NULL); + assert(nxt == REQ_FSM_DISEMBARK || !WS_IsReserved(req->ws)); return (nxt); } diff --git a/bin/varnishd/cache/cache_varnishd.h b/bin/varnishd/cache/cache_varnishd.h index 9d98778d3..ececc96aa 100644 --- a/bin/varnishd/cache/cache_varnishd.h +++ b/bin/varnishd/cache/cache_varnishd.h @@ -465,6 +465,12 @@ void WRK_Init(void); /* cache_ws.c */ void WS_Panic(const struct ws *ws, struct vsb *vsb); +static inline int +WS_IsReserved(const struct ws *ws) +{ + + return (ws->r != NULL); +} void WS_Rollback(struct ws *, uintptr_t); void *WS_AtOffset(const struct ws *ws, unsigned off, unsigned len); diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c index 4e09b59ae..338eac7f8 100644 --- a/bin/varnishd/cache/cache_ws.c +++ b/bin/varnishd/cache/cache_ws.c @@ -411,7 +411,7 @@ WS_VSB_finish(struct vsb *vsb, struct ws *ws, size_t *szp) WS_Assert(ws); if (!VSB_finish(vsb)) { p = VSB_data(vsb); - if (p == WS_Reservation(ws)) { + if (p == ws->f) { WS_Release(ws, VSB_len(vsb) + 1); if (szp != NULL) *szp = VSB_len(vsb); diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c index 3bb6345e0..9625bed21 100644 --- a/bin/varnishd/http1/cache_http1_fsm.c +++ b/bin/varnishd/http1/cache_http1_fsm.c @@ -84,7 +84,7 @@ http1_req(struct worker *wrk, void *arg) THR_SetRequest(req); req->transport = &HTTP1_transport; - AZ(WS_Reservation(wrk->aws)); + assert(!WS_IsReserved(wrk->aws)); HTTP1_Session(wrk, req); AZ(wrk->v1l); WS_Assert(wrk->aws); @@ -319,7 +319,7 @@ HTTP1_Session(struct worker *wrk, struct req *req) sp->t_idle + SESS_TMO(sp, timeout_idle), NAN, cache_param->http_req_size); - AZ(WS_Reservation(req->htc->ws)); + assert(!WS_IsReserved(req->htc->ws)); if (hs < HTC_S_EMPTY) { req->acct.req_hdrbytes += req->htc->rxbuf_e - req->htc->rxbuf_b; @@ -354,8 +354,8 @@ HTTP1_Session(struct worker *wrk, struct req *req) if (H2_prism_complete(req->htc) == HTC_S_COMPLETE) { if (!FEATURE(FEATURE_HTTP2)) { SES_Close(req->sp, SC_REQ_HTTP20); - AZ(WS_Reservation(req->ws)); - AZ(WS_Reservation(wrk->aws)); + assert(!WS_IsReserved(req->ws)); + assert(!WS_IsReserved(wrk->aws)); http1_setstate(sp, H1CLEANUP); continue; } @@ -370,8 +370,8 @@ HTTP1_Session(struct worker *wrk, struct req *req) if (i) { assert(req->doclose > 0); SES_Close(req->sp, req->doclose); - AZ(WS_Reservation(req->ws)); - AZ(WS_Reservation(wrk->aws)); + assert(!WS_IsReserved(req->ws)); + assert(!WS_IsReserved(wrk->aws)); http1_setstate(sp, H1CLEANUP); continue; } @@ -405,13 +405,13 @@ HTTP1_Session(struct worker *wrk, struct req *req) AZ(req->top->vcl0); req->task->func = NULL; req->task->priv = NULL; - AZ(WS_Reservation(req->ws)); - AZ(WS_Reservation(wrk->aws)); + assert(!WS_IsReserved(req->ws)); + assert(!WS_IsReserved(wrk->aws)); http1_setstate(sp, H1CLEANUP); } else if (st == H1CLEANUP) { - AZ(WS_Reservation(wrk->aws)); - AZ(WS_Reservation(req->ws)); + assert(!WS_IsReserved(wrk->aws)); + assert(!WS_IsReserved(req->ws)); if (sp->fd >= 0 && req->doclose != SC_NULL) SES_Close(sp, req->doclose); diff --git a/bin/varnishd/http1/cache_http1_line.c b/bin/varnishd/http1/cache_http1_line.c index 7ff71c151..7b595ca02 100644 --- a/bin/varnishd/http1/cache_http1_line.c +++ b/bin/varnishd/http1/cache_http1_line.c @@ -137,7 +137,7 @@ V1L_Close(struct worker *wrk, uint64_t *cnt) wrk->v1l = NULL; CHECK_OBJ_NOTNULL(v1l, V1L_MAGIC); *cnt = v1l->cnt; - if (WS_Reservation(v1l->ws)) + if (WS_IsReserved(v1l->ws)) WS_Release(v1l->ws, 0); WS_Rollback(v1l->ws, v1l->res); ZERO_OBJ(v1l, sizeof *v1l); diff --git a/bin/varnishd/http2/cache_http2_proto.c b/bin/varnishd/http2/cache_http2_proto.c index edcf64c04..c079656b1 100644 --- a/bin/varnishd/http2/cache_http2_proto.c +++ b/bin/varnishd/http2/cache_http2_proto.c @@ -184,7 +184,7 @@ h2_del_req(struct worker *wrk, const struct h2_req *r2) /* XXX: PRIORITY reshuffle */ VTAILQ_REMOVE(&h2->streams, r2, list); Lck_Unlock(&sp->mtx); - AZ(WS_Reservation(r2->req->ws)); + assert(!WS_IsReserved(r2->req->ws)); Req_Cleanup(sp, wrk, r2->req); Req_Release(r2->req); } @@ -532,7 +532,7 @@ h2_do_req(struct worker *wrk, void *priv) wrk->stats->client_req++; if (CNT_Request(req) != REQ_FSM_DISEMBARK) { - AZ(WS_Reservation(req->ws)); + assert(!WS_IsReserved(req->ws)); AZ(req->top->vcl0); h2 = r2->h2sess; CHECK_OBJ_NOTNULL(h2, H2_SESS_MAGIC); @@ -563,7 +563,7 @@ h2_end_headers(struct worker *wrk, struct h2_sess *h2, Lck_Lock(&h2->sess->mtx); VSLb(h2->vsl, SLT_Debug, "HPACK/FINI %s", h2e->name); Lck_Unlock(&h2->sess->mtx); - AZ(WS_Reservation(r2->req->ws)); + assert(!WS_IsReserved(r2->req->ws)); h2_del_req(wrk, r2); return (h2e); } @@ -685,7 +685,7 @@ h2_rx_headers(struct worker *wrk, struct h2_sess *h2, struct h2_req *r2) VSLb(h2->vsl, SLT_Debug, "HPACK(hdr) %s", h2e->name); Lck_Unlock(&h2->sess->mtx); (void)h2h_decode_fini(h2); - AZ(WS_Reservation(r2->req->ws)); + assert(!WS_IsReserved(r2->req->ws)); h2_del_req(wrk, r2); return (h2e); } @@ -720,7 +720,7 @@ h2_rx_continuation(struct worker *wrk, struct h2_sess *h2, struct h2_req *r2) VSLb(h2->vsl, SLT_Debug, "HPACK(cont) %s", h2e->name); Lck_Unlock(&h2->sess->mtx); (void)h2h_decode_fini(h2); - AZ(WS_Reservation(r2->req->ws)); + assert(!WS_IsReserved(r2->req->ws)); h2_del_req(wrk, r2); return (h2e); } diff --git a/bin/varnishd/http2/cache_http2_session.c b/bin/varnishd/http2/cache_http2_session.c index 15dd17b27..4b216f85c 100644 --- a/bin/varnishd/http2/cache_http2_session.c +++ b/bin/varnishd/http2/cache_http2_session.c @@ -148,7 +148,7 @@ h2_del_sess(struct worker *wrk, struct h2_sess *h2, enum sess_close reason) VHT_Fini(h2->dectbl); AZ(pthread_cond_destroy(h2->winupd_cond)); TAKE_OBJ_NOTNULL(req, &h2->srq, REQ_MAGIC); - AZ(WS_Reservation(req->ws)); + assert(!WS_IsReserved(req->ws)); sp = h2->sess; Req_Cleanup(sp, wrk, req); Req_Release(req); From phk at FreeBSD.org Mon Sep 7 13:05:07 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 7 Sep 2020 13:05:07 +0000 (UTC) Subject: [master] ba98c0935 Constify Message-ID: <20200907130507.6A5269BC8A@lists.varnish-cache.org> commit ba98c0935a437a543622799a9188135fce4ca995 Author: Poul-Henning Kamp Date: Mon Sep 7 13:04:37 2020 +0000 Constify diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c index f10dfc387..445c55b79 100644 --- a/lib/libvarnishapi/vsc.c +++ b/lib/libvarnishapi/vsc.c @@ -60,9 +60,9 @@ struct vsc_sf_mode { unsigned append; }; -static struct vsc_sf_mode VSC_SF_INCLUDE[1] = {{"include", 1, 1, 1}}; -static struct vsc_sf_mode VSC_SF_EXCLUDE[1] = {{"exclude", 0, 0, 1}}; -static struct vsc_sf_mode VSC_SF_REQUIRE[1] = {{"require", 1, 0, 0}}; +static const struct vsc_sf_mode VSC_SF_INCLUDE[1] = {{"include", 1, 1, 1}}; +static const struct vsc_sf_mode VSC_SF_EXCLUDE[1] = {{"exclude", 0, 0, 1}}; +static const struct vsc_sf_mode VSC_SF_REQUIRE[1] = {{"require", 1, 0, 0}}; struct vsc_sf { unsigned magic; From dridi.boukelmoune at gmail.com Mon Sep 7 14:56:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 7 Sep 2020 14:56:06 +0000 (UTC) Subject: [master] 4e7d66338 vbe: Update the poll timeout before polling-reading in backend probes Message-ID: <20200907145606.870F09ECEF@lists.varnish-cache.org> commit 4e7d663383393c73241fda56189879448e1dfc54 Author: delthas Date: Mon Sep 7 11:48:49 2020 +0200 vbe: Update the poll timeout before polling-reading in backend probes Varnish uses a global per-probe timeout for backend probes. When reading the backend response, Varnish tries to poll and read in a loop, until a poll timeouts, the streams EOFs or there is an error. The poll is supposed to timeout when the per-probe timeout ends. This is currently setup so that `t_end` is the deadline for the probe, set when the function starts, then the poll waits until `t_end`. Previously, the poll timeout was never updated, and was always set to `t_end - t_now` without updating `t_now`, which means that it was effectively a between-bytes timeout instead of a proper per-probe timeout. This fixes this issue by updating `t_now` before updating the `t_end - t_now` timeout so that the timeout passed to poll effectively corresponds to a deadline of `t_end`. See the issue fixed by this commit for more details. Fixes: #3402 diff --git a/bin/varnishd/cache/cache_backend_probe.c b/bin/varnishd/cache/cache_backend_probe.c index 020ee1e75..bf41b952a 100644 --- a/bin/varnishd/cache/cache_backend_probe.c +++ b/bin/varnishd/cache/cache_backend_probe.c @@ -348,6 +348,7 @@ vbp_poke(struct vbp_target *vt) while (1) { pfd->events = POLLIN; pfd->revents = 0; + t_now = VTIM_real(); tmo = (int)round((t_end - t_now) * 1e3); if (tmo <= 0) { bprintf(vt->resp_buf, From dridi.boukelmoune at gmail.com Mon Sep 7 14:56:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 7 Sep 2020 14:56:06 +0000 (UTC) Subject: [master] 9d8cfbee2 vtc: Test case for #3402 Message-ID: <20200907145606.9E8089ECF3@lists.varnish-cache.org> commit 9d8cfbee2f391d74c2020c2c3f5f93932e27257c Author: Dridi Boukelmoune Date: Mon Sep 7 16:53:59 2020 +0200 vtc: Test case for #3402 diff --git a/bin/varnishtest/tests/r03402.vtc b/bin/varnishtest/tests/r03402.vtc new file mode 100644 index 000000000..6596247f9 --- /dev/null +++ b/bin/varnishtest/tests/r03402.vtc @@ -0,0 +1,24 @@ +varnishtest "probe dripping reads timeout" + +server s0 { + rxreq + delay 1 + send "HTTP/1.1 200 OK\r\n" + delay 0.5 + send "Server: s0\r\n" + delay 1 + send "\r\n" +} -dispatch + +varnish v1 -vcl+backend { + probe default { + .window = 1; + .threshold = 1; + .timeout = 2s; + .interval = 0.1s; + } +} -start + +delay 5 + +varnish v1 -cliexpect sick backend.list From dridi.boukelmoune at gmail.com Mon Sep 7 18:33:08 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 7 Sep 2020 18:33:08 +0000 (UTC) Subject: [master] c31d071ed cli: Document more heredoc pitfalls Message-ID: <20200907183308.C6E82A56CB@lists.varnish-cache.org> commit c31d071edaaeaad08556a7ab721cf0f5ce8e14fa Author: Dridi Boukelmoune Date: Mon Sep 7 20:30:42 2020 +0200 cli: Document more heredoc pitfalls diff --git a/doc/sphinx/reference/varnish-cli.rst b/doc/sphinx/reference/varnish-cli.rst index ad8fe016e..b8575c740 100644 --- a/doc/sphinx/reference/varnish-cli.rst +++ b/doc/sphinx/reference/varnish-cli.rst @@ -134,6 +134,20 @@ can be loaded:: 200 14 VCL compiled. +A big difference with a shell here document is the handling of the ``<<`` +token. Just like command names can be quoted, the here document token keeps +its meaning, even quoted:: + + vcl.inline test "<<" EOF + vcl 4.0; + + backend be { + .host = "localhost"; + } + EOF + 200 14 + VCL compiled. + When using a front-end to the Varnish-CLI like ``varnishadm``, one must take into account the double expansion happening. First in the shell launching the ``varnishadm`` command and then in the Varnish CLI itself. @@ -166,6 +180,43 @@ multi-line argument:: EOF' VCL compiled. +Another difference with a shell here document is that only one here document +can be used on a single command line. For example, it is possible to do this +in a shell script:: + + #!/bin/sh + + cat << EOF1 ; cat << EOF2 + hello + EOF1 + world + EOF2 + +The expected output is:: + + hello + world + +With the Varnish CLI, only the last parameter may use the here document form, +which greatly restricts the number of commands that can effectively use them. +Trying to use multiple here documents only takes the last one into account. + +For example:: + + command argument << EOF1 << EOF2 + heredoc1 + EOF1 + heredoc2 + EOF2 + +This conceptually results in the following command line: + +- ``"command"`` +- ``"argument"`` +- ``"<<"`` +- ``"EOF1"`` +- ``"heredoc1\nEOF1\nheredoc2\n"`` + Other pitfalls include variable expansion of the shell invoking ``varnishadm`` but this is not directly related to the Varnish CLI. If you get the quoting right you should be fine even with complex commands. From dridi.boukelmoune at gmail.com Tue Sep 8 09:13:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 8 Sep 2020 09:13:06 +0000 (UTC) Subject: [master] 4688edd83 doc: Document changes to the workspace API Message-ID: <20200908091306.A63DA6291C@lists.varnish-cache.org> commit 4688edd8362e13ccafadebd1f6e90a30ec346228 Author: Dridi Boukelmoune Date: Tue Sep 8 11:09:06 2020 +0200 doc: Document changes to the workspace API In addition, soften the WS_Front() deprecation and keep the new WS_Id() function internal. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 32c4972fa..40fafc766 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -795,13 +795,11 @@ int WS_Overflowed(const struct ws *ws); const char *WS_Printf(struct ws *ws, const char *fmt, ...) v_printflike_(2, 3); int WS_Inside(const struct ws *, const void *, const void *); void WS_Assert_Allocated(const struct ws *ws, const void *ptr, ssize_t len); -void WS_Id(const struct ws *ws, char *id); void WS_VSB_new(struct vsb *, struct ws *); char *WS_VSB_finish(struct vsb *, struct ws *, size_t *); /* REL_20210315 remove */ -static inline char* WS_Front(const struct ws *) v_deprecated_; static inline char* WS_Front(const struct ws *ws) { diff --git a/bin/varnishd/cache/cache_varnishd.h b/bin/varnishd/cache/cache_varnishd.h index ececc96aa..9a68f4229 100644 --- a/bin/varnishd/cache/cache_varnishd.h +++ b/bin/varnishd/cache/cache_varnishd.h @@ -464,6 +464,7 @@ void VMOD_Panic(struct vsb *); void WRK_Init(void); /* cache_ws.c */ +void WS_Id(const struct ws *ws, char *id); void WS_Panic(const struct ws *ws, struct vsb *vsb); static inline int WS_IsReserved(const struct ws *ws) diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index fbda9d6b9..afbbd0bb2 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -67,6 +67,23 @@ varnishtest Changes for developers and VMOD authors ======================================= +The workspace API saw a number of changes in anticipation of a future +inclusion in VRT. The deprecated ``WS_Reserve()`` function was finally +removed, the functions ``WS_ReserveSize()`` and ``WS_ReserveAll()`` were +introduced as a replacement. + +On the topic of workspace reservation, the ``WS_Front()`` function is +now deprecated in favor of ``WS_Reservation()``. The two functions +behave similarly, but the latter ensures that it is only ever called +during a reservation. There was no legitimate reason to access the +workspace's front outside of a reservation. + +In a scenario where a reservation is made in a part of the code, but +used somewhere else, it is possible to later query the size with the +new ``WS_ReservationSize()`` function. + +The return value for ``WS_Printf()`` is now a constant string. + **XXX changes concerning VRT, the public APIs, source code organization, builds etc.** diff --git a/include/vrt.h b/include/vrt.h index 2e5cc2d7d..f80d5fa47 100644 --- a/include/vrt.h +++ b/include/vrt.h @@ -60,7 +60,6 @@ * [cache.h] WS_ReservationSize() added * [cache.h] WS_Front() deprecated * [cache.h] WS_Reservation() added - * [cache.h] WS_Id() added * 11.0 (2020-03-16) * Changed type of vsa_suckaddr_len from int to size_t * New prefix_{ptr|len} fields in vrt_backend From nils.goroll at uplex.de Tue Sep 8 10:48:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 8 Sep 2020 10:48:06 +0000 (UTC) Subject: [master] 449710e1a changelog Message-ID: <20200908104806.F150F64FF7@lists.varnish-cache.org> commit 449710e1a8dcf665616edb5ba821ad0217d2833a Author: Nils Goroll Date: Tue Sep 8 12:46:45 2020 +0200 changelog diff --git a/doc/changes.rst b/doc/changes.rst index 1ede0021e..367327c8d 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -207,6 +207,8 @@ NEXT (scheduled 2020-09-15) * Handle a workspace overflow in ``VRY_Validate()`` (3319_) +* Fixed the backend probe ``.timeout`` handling for "dripping" responses (3402_) + .. _2990: https://github.com/varnishcache/varnish-cache/issues/2990 .. _2999: https://github.com/varnishcache/varnish-cache/issues/2999 .. _3002: https://github.com/varnishcache/varnish-cache/issues/3002 @@ -224,6 +226,7 @@ NEXT (scheduled 2020-09-15) .. _3361: https://github.com/varnishcache/varnish-cache/issues/3361 .. _3378: https://github.com/varnishcache/varnish-cache/issues/3378 .. _3394: https://github.com/varnishcache/varnish-cache/issues/3394 +.. _3402: https://github.com/varnishcache/varnish-cache/issues/3402 ================================ Varnish Cache 6.4.0 (2020-03-16) From nils.goroll at uplex.de Wed Sep 9 13:08:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 9 Sep 2020 13:08:06 +0000 (UTC) Subject: [master] a7055b351 gc a duplicate initialization Message-ID: <20200909130806.AB74262FDC@lists.varnish-cache.org> commit a7055b351d8d2d49364397f7fd6fe2e16aeda292 Author: Nils Goroll Date: Wed Sep 9 15:03:48 2020 +0200 gc a duplicate initialization diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c index 144736ae0..aef672e27 100644 --- a/bin/varnishd/cache/cache_vrt.c +++ b/bin/varnishd/cache/cache_vrt.c @@ -561,7 +561,6 @@ VRT_UpperLowerStrands(VRT_CTX, VCL_STRANDS s, int up) continue; if (q != NULL) copy = 1; - p = q = s->p[i]; for(p = q = s->p[i]; *p != '\0'; p++) { if ((up && vct_islower(*p)) || (!up && vct_isupper(*p))) { From dridi.boukelmoune at gmail.com Wed Sep 9 14:08:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 9 Sep 2020 14:08:06 +0000 (UTC) Subject: [master] a45319c50 doc: Release notes for varnishstat Message-ID: <20200909140806.3D7D264900@lists.varnish-cache.org> commit a45319c50786d11056a140e9f874c00e522fbd13 Author: Dridi Boukelmoune Date: Wed Sep 9 16:04:23 2020 +0200 doc: Release notes for varnishstat diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index afbbd0bb2..e1b2bb440 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -57,7 +57,23 @@ varnishadm varnishstat =========== -**XXX changes concerning varnishstat(1) and/or varnish-counters(7)** +A help screen is now available in interactive mode via the ``h`` key. + +Again in interactive mode, verbosity is increased from the default value +during startup when the filtering of counters would otherwise display +nothing. + +Filtering using the ``-f`` option is now deprecated in favor of ``-I`` and +``-X`` options that are treated in order. While still present, the ``-f`` +option now also work in order instead of exclusive filters first and then +inclusive filters. It was also wrongly documented as inclusive first. + +The JSON output slightly changed to more easily be consumed with programming +languages that may map JSON objects to types. See upgrade notes for more +details. + +There are two new ``MAIN.beresp_uncacheable`` and ``MAIN.beresp_shortlived`` +counters. varnishtest =========== diff --git a/doc/sphinx/whats-new/upgrading-trunk.rst b/doc/sphinx/whats-new/upgrading-trunk.rst index 6143fde99..a38b67457 100644 --- a/doc/sphinx/whats-new/upgrading-trunk.rst +++ b/doc/sphinx/whats-new/upgrading-trunk.rst @@ -30,4 +30,35 @@ to:** * Changes in public APIs that may require changes in VMODs or VAPI/VUT clients. +varnishstat +=========== + +The JSON output (``-j`` option) changed to avoid having the ``timestamp`` +field mixed with the counters fields. As such the schema version was bumped +from 0 to 1, and a ``version`` top-level field was added to keep track of +future schema changes. Counters are in a new ``counters`` top-level field. + +Before:: + + { + "timestamp": "YYYY-mm-ddTHH:MM:SS", + "MGT.uptime": { + ... + }, + ... + } + +After:: + + { + "version": 1, + "timestamp": "YYYY-mm-ddTHH:MM:SS", + "counters": { + "MGT.uptime": { + ... + }, + ... + } + } + *eof* From dridi.boukelmoune at gmail.com Wed Sep 9 14:08:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 9 Sep 2020 14:08:06 +0000 (UTC) Subject: [master] 7c3f01736 doc: Polish changelog Message-ID: <20200909140806.57D1764903@lists.varnish-cache.org> commit 7c3f01736ab6fc58bbe5fa0851a85b5ac4606d07 Author: Dridi Boukelmoune Date: Wed Sep 9 16:06:58 2020 +0200 doc: Polish changelog diff --git a/doc/changes.rst b/doc/changes.rst index 367327c8d..46c2b4aad 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -79,8 +79,8 @@ NEXT (scheduled 2020-09-15) * ``resp.filters`` now contains a correct value when the auto-determined filter list is read (3002_) -* It is now a VCL (runtime) error to write to ``resp.do`` and - ``beresp.do_`` fields which determine the filter list after setting +* It is now a VCL (runtime) error to write to ``resp.do_*`` and + ``beresp.do_*`` fields which determine the filter list after setting ``resp.filters`` and ``beresp.filters``, respectively * Behavior for 304 responses was changed not to update @@ -203,7 +203,6 @@ NEXT (scheduled 2020-09-15) breaking change). * Add ``WS_ReservationSize()`` * ``WS_Front()`` is now deprecated and replaced by ``WS_Reservation()`` - * Add ``WS_Id()`` * Handle a workspace overflow in ``VRY_Validate()`` (3319_) From dridi.boukelmoune at gmail.com Thu Sep 10 13:30:08 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 10 Sep 2020 13:30:08 +0000 (UTC) Subject: [master] a3cc90d3b doc: Mention VSL changes Message-ID: <20200910133008.68D2D7F8A@lists.varnish-cache.org> commit a3cc90d3ba15ea9c790354946a63bb53ca3100af Author: Dridi Boukelmoune Date: Thu Sep 10 15:28:42 2020 +0200 doc: Mention VSL changes diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index e1b2bb440..b34158668 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -47,7 +47,13 @@ VMODs varnishlog ========== -**XXX changes concerning varnishlog(1) and/or vsl(7)** +The ``BackendReuse`` log record has been retired. It was inconsistently named +compared to other places like stat counters where we use the words reuse and +recycle (it should have been named ``BackendRecycle`` if anything). + +The ``BackendOpen`` record can now tell whether the connection to the backend +was opened or reused from the pool, and the ``BackendClose`` record will tell +whether the connection was effectively closed or recycled into the pool. varnishadm ========== diff --git a/doc/sphinx/whats-new/upgrading-trunk.rst b/doc/sphinx/whats-new/upgrading-trunk.rst index a38b67457..9922355b4 100644 --- a/doc/sphinx/whats-new/upgrading-trunk.rst +++ b/doc/sphinx/whats-new/upgrading-trunk.rst @@ -61,4 +61,16 @@ After:: } } +VSL +=== + +If you need to build VSL queries that depend on ``BackendReuse`` you can +now rely on ``BackendClose``, for example:: + + varnishlog -q 'BackendReuse[2] ~ www' + +The new query would be:: + + varnishlog -q 'BackendClose[2] ~ www and BackendClose[3] eq recycle' + *eof* From dridi.boukelmoune at gmail.com Thu Sep 10 14:41:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 10 Sep 2020 14:41:06 +0000 (UTC) Subject: [master] 555b074ba doc: Mention non-VCL varnishd changes Message-ID: <20200910144106.D61DF60E26@lists.varnish-cache.org> commit 555b074ba622e09a4de356cd440beb90dffabd5c Author: Dridi Boukelmoune Date: Thu Sep 10 16:34:16 2020 +0200 doc: Mention non-VCL varnishd changes Which isn't that many when you omit VCL. diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index b34158668..339914d81 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -23,11 +23,15 @@ varnishd Parameters ~~~~~~~~~~ -**XXX changes in -p parameters** +A new ``vcc_acl_pedantic`` parameter will turn warnings into errors for the +case where an ACL entry includes a network prefix, but host bits aren't all +zeroes. Other changes in varnishd ~~~~~~~~~~~~~~~~~~~~~~~~~ +Some error messages improved in the VCL compiler. + Changes to VCL ============== From dridi.boukelmoune at gmail.com Thu Sep 10 14:41:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 10 Sep 2020 14:41:06 +0000 (UTC) Subject: [master] 9f6170b80 doc: More VSC-related changes Message-ID: <20200910144106.EEEB560E29@lists.varnish-cache.org> commit 9f6170b807741f203565114a98a5a07116f1b7ad Author: Dridi Boukelmoune Date: Thu Sep 10 16:35:45 2020 +0200 doc: More VSC-related changes diff --git a/doc/changes.rst b/doc/changes.rst index 46c2b4aad..29091fb66 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -195,6 +195,11 @@ NEXT (scheduled 2020-09-15) and ``varnishstat`` for inclusion, exclusion and required glob patterns on the statistic field names. (3394_) + * Added the missing ``VSC_OPT_f`` macro and the new ``VSC_OPT_I`` and + ``VSC_OPT_X`` to libvarnishapi headers. + + * Added ``-I`` and ``-X`` options to ``varnishstat``. + * Overhaul of the workspace API * The previously deprecated ``WS_Reserve()`` has been removed diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index 339914d81..f980df0b7 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -75,7 +75,7 @@ nothing. Filtering using the ``-f`` option is now deprecated in favor of ``-I`` and ``-X`` options that are treated in order. While still present, the ``-f`` -option now also work in order instead of exclusive filters first and then +option now also works in order instead of exclusive filters first and then inclusive filters. It was also wrongly documented as inclusive first. The JSON output slightly changed to more easily be consumed with programming From dridi.boukelmoune at gmail.com Thu Sep 10 14:41:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 10 Sep 2020 14:41:07 +0000 (UTC) Subject: [master] 310bce0d0 doc: Mention varnish.m4 changes Message-ID: <20200910144107.1619E60E2D@lists.varnish-cache.org> commit 310bce0d02408c58addcf332ab69f95713b2662d Author: Dridi Boukelmoune Date: Thu Sep 10 16:39:25 2020 +0200 doc: Mention varnish.m4 changes And amend the 7.0.0 mis-prediction. diff --git a/doc/changes.rst b/doc/changes.rst index 29091fb66..3982fb699 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -213,6 +213,8 @@ NEXT (scheduled 2020-09-15) * Fixed the backend probe ``.timeout`` handling for "dripping" responses (3402_) +* New ``VARNISH_VMODS_GENERATED()`` macro in ``varnish.m4``. + .. _2990: https://github.com/varnishcache/varnish-cache/issues/2990 .. _2999: https://github.com/varnishcache/varnish-cache/issues/2999 .. _3002: https://github.com/varnishcache/varnish-cache/issues/3002 diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index f980df0b7..4985fb7a1 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -110,6 +110,10 @@ new ``WS_ReservationSize()`` function. The return value for ``WS_Printf()`` is now a constant string. +VMOD authors who would like to generate VCC files can now use the +``VARNISH_VMODS_GENERATED()`` macro from ``varnish.m4`` for autotools +builds. + **XXX changes concerning VRT, the public APIs, source code organization, builds etc.** diff --git a/varnish.m4 b/varnish.m4 index f86885a9a..abd8dfc78 100644 --- a/varnish.m4 +++ b/varnish.m4 @@ -31,7 +31,7 @@ # OF THE POSSIBILITY OF SUCH DAMAGE. # varnish.m4 - Macros to build against Varnish. -*- Autoconf -*- -# serial 11 (varnish-7.0.0) +# serial 11 (varnish-6.5.0) # # This collection of macros helps create VMODs or tools interacting with # Varnish Cache using the GNU build system (autotools). In order to work @@ -380,9 +380,9 @@ AC_DEFUN([VARNISH_VMODS], [ # VARNISH_VMODS_GENERATED(NAMES) # ------------------------------ -# Since: Varnish 7.0.0 +# Since: Varnish 6.5.0 # -# Varnish 7 adds the possibility to transparently work with a generated VCC +# Varnish 6.5 adds the possibility to transparently work with a generated VCC # file. The VCC file would then be created in the build directory, which is # incompatible with how the VARNISH_VMODS macro operates. # From dridi.boukelmoune at gmail.com Fri Sep 11 08:37:08 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 11 Sep 2020 08:37:08 +0000 (UTC) Subject: [master] 5b21cb888 vbf: Prevent pooling of a Connection:close beresp Message-ID: <20200911083708.431CAABCE2@lists.varnish-cache.org> commit 5b21cb8885c987d9812e046c56e871f6793f6c13 Author: Dridi Boukelmoune Date: Tue Sep 8 11:36:52 2020 +0200 vbf: Prevent pooling of a Connection:close beresp Whether the header was set by the backend or directly in VCL, it is now possible to signal that a backend connection should not be added back to the pool after a successful fetch with a Connection:close header. Pooling such a connection would be counter-productive if closing the session was requested by the backend itself, because it would then be likely that reusing the connection would result in busting the extra chance. Setting the Connection:close directly in VCL can help mitigating against a misbehaving backend. Refs #3400 diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index 1c71e08b4..c09ecafe7 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -452,6 +452,10 @@ vbf_stp_startfetch(struct worker *wrk, struct busyobj *bo) VCL_backend_response_method(bo->vcl, wrk, NULL, bo, NULL); + if (bo->htc != NULL && bo->htc->doclose == SC_NULL && + http_GetHdrField(bo->beresp, H_Connection, "close", NULL)) + bo->htc->doclose = SC_RESP_CLOSE; + if (wrk->handling == VCL_RET_ABANDON || wrk->handling == VCL_RET_FAIL || wrk->handling == VCL_RET_ERROR) { /* do not count deliberately ending the backend connection as diff --git a/bin/varnishtest/tests/b00073.vtc b/bin/varnishtest/tests/b00073.vtc new file mode 100644 index 000000000..6cae03450 --- /dev/null +++ b/bin/varnishtest/tests/b00073.vtc @@ -0,0 +1,39 @@ +varnishtest "backend connection close" + +server s1 { + rxreq + expect req.http.beresp-connection ~ close + txresp + expect_close + + accept + rxreq + expect req.http.beresp-connection !~ close + txresp -hdr "connection: close" + expect_close +} -start + +varnish v1 -vcl+backend { + sub vcl_recv { + return (pass); + } + sub vcl_backend_response { + # NB: this overrides unconditionally on purpose + set beresp.http.connection = bereq.http.beresp-connection; + } +} -start + +client c1 { + txreq -hdr "beresp-connection: close, x-varnish" + rxresp + expect resp.status == 200 + + txreq + rxresp + expect resp.status == 200 +} -run + +server s1 -wait + +varnish v1 -expect MAIN.backend_recycle == 0 +varnish v1 -expect VBE.vcl1.s1.conn == 0 From dridi.boukelmoune at gmail.com Fri Sep 11 08:37:08 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 11 Sep 2020 08:37:08 +0000 (UTC) Subject: [master] ba78ebeb2 doc: Mention the effect of Connection:close in beresp Message-ID: <20200911083708.5876FABCE5@lists.varnish-cache.org> commit ba78ebeb28105290bef56008f52bf63e58d5a894 Author: Dridi Boukelmoune Date: Tue Sep 8 11:51:06 2020 +0200 doc: Mention the effect of Connection:close in beresp diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index 4985fb7a1..38cb23e16 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -38,6 +38,25 @@ Changes to VCL VCL variables ~~~~~~~~~~~~~ +It is now possible to manually set a ``Connection: close`` header in +``beresp`` to signal that the backend connection shouldn't be recycled. +This might help dealing with backends that would under certain circumstances +have trouble managing their end of the connection, for example for certain +kinds of resources. + +Care should be taken to preserve other headers listed in the connection +header:: + + sub vcl_backend_response { + if (beresp.backend == faulty_backend) { + if (beresp.http.Connection) { + set beresp.http.Connection += ", close"; + } else { + set beresp.http.Connection = "close"; + } + } + } + **XXX new, deprecated or removed variables, or changed semantics** Other changes to VCL From dridi.boukelmoune at gmail.com Fri Sep 11 09:43:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 11 Sep 2020 09:43:06 +0000 (UTC) Subject: [master] c9a95a1d8 doc: Mention VMOD changes Message-ID: <20200911094306.88E42AD759@lists.varnish-cache.org> commit c9a95a1d80f71d6fd5b07d113486ec7bdf862add Author: Dridi Boukelmoune Date: Fri Sep 11 11:39:24 2020 +0200 doc: Mention VMOD changes diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index 38cb23e16..5a9f92467 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -65,7 +65,16 @@ Other changes to VCL VMODs ===== -**XXX changes in the bundled VMODs** +A new ``std.blobread()`` function similar to ``std.fileread()`` was added to +work with binary files. + +The shard director's ``.add_backend()`` method has a new optional ``weight`` +parameter. An error when a backend is added or removed now fails the +transaction (or the ``vcl.load`` command in ``vcl_init``) but an invalid +weight does not result into a hard failure. + +The shard director no longer outputs the (unused) ``canon_point`` property +in ``backend.list`` commands. varnishlog ========== From dridi.boukelmoune at gmail.com Fri Sep 11 09:54:09 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 11 Sep 2020 09:54:09 +0000 (UTC) Subject: [master] 89940496a doc: Pretend we had a change in varnishadm Message-ID: <20200911095409.26AF0ADCF5@lists.varnish-cache.org> commit 89940496a2365a4c45d256b4de1f79441e63d025 Author: Dridi Boukelmoune Date: Fri Sep 11 11:50:53 2020 +0200 doc: Pretend we had a change in varnishadm I thought the slight change of behavior was worth mentioning, and that in the varnishadm/CLI section it would stand out. diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index 5a9f92467..4b51473ac 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -90,7 +90,12 @@ whether the connection was effectively closed or recycled into the pool. varnishadm ========== -**XXX changes concerning varnishadm(1) and/or varnish-cli(7)** +The ``backend.set_health`` command can be used to force a specific state +between sick and healthy or restore the automatic behavior, which depends on +the presence of a probe. While forcing a backend to be sick would prevent it +from being selected by a director, a straight selection of the backend from +VCL would still attempt a connection. This has been fixed, and the command's +documentation was clarified. varnishstat =========== From dridi.boukelmoune at gmail.com Fri Sep 11 10:06:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 11 Sep 2020 10:06:06 +0000 (UTC) Subject: [master] 25c7cc5f9 doc: Mention the varnishtest change Message-ID: <20200911100606.4C15BAE2E6@lists.varnish-cache.org> commit 25c7cc5f9ac41383fc718153514f957cbf023241 Author: Dridi Boukelmoune Date: Fri Sep 11 12:04:45 2020 +0200 doc: Mention the varnishtest change The only user-visible change for this release. diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index 4b51473ac..da9782274 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -121,7 +121,9 @@ counters. varnishtest =========== -**XXX changes concerning varnishtest(1) and/or vtc(7)** +The ``process -expect-text`` command will wait an order of magnitude longer +for the text to appear. It used to be too sensitive to any kind of timing +disruption. Changes for developers and VMOD authors ======================================= From dridi.boukelmoune at gmail.com Fri Sep 11 14:20:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 11 Sep 2020 14:20:06 +0000 (UTC) Subject: [master] d109b419c doc: Mention remaining VCL changes Message-ID: <20200911142006.C16B55430@lists.varnish-cache.org> commit d109b419cda9dbf7fc6c12eb50d80d5d2452e9aa Author: Dridi Boukelmoune Date: Fri Sep 11 16:09:15 2020 +0200 doc: Mention remaining VCL changes Leaving out any change that is solely a bug fix on purpose. diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index da9782274..8a0e637f2 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -38,6 +38,21 @@ Changes to VCL VCL variables ~~~~~~~~~~~~~ +A new ``obj.can_esi`` variable has been added to identify whether the response +can be ESI processed. + +Once ``resp.filters`` is explicitly set, trying to set a ``resp.do_*`` field +results in a VCL failure. The same rule applies to ``beresp.filters`` and +``beresp.do_*`` fields. + +The ``BACKEND`` VCL type now has a ``.resolve()`` method to find the effective +backend directly from VCL. When a director is selected, the resolution would +otherwise be delayed until after returning from ``vcl_backend_fetch`` or +``vcl_pipe``:: + + # eager backend selection + set bereq.backend = bereq.backend.resolve(); + It is now possible to manually set a ``Connection: close`` header in ``beresp`` to signal that the backend connection shouldn't be recycled. This might help dealing with backends that would under certain circumstances @@ -57,11 +72,16 @@ header:: } } -**XXX new, deprecated or removed variables, or changed semantics** - Other changes to VCL ~~~~~~~~~~~~~~~~~~~~ +A failure in ``vcl_recv`` could resume execution in ``vcl_hash`` before +effectively ending the transaction, this has been corrected. A failure in +``vcl_recv`` is now definitive. + +There is a new syntax for ``BLOB`` literals: ``::``. This syntax is +also used to automatically cast a blob into a string. + VMODs ===== From dridi.boukelmoune at gmail.com Fri Sep 11 14:20:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 11 Sep 2020 14:20:06 +0000 (UTC) Subject: [master] 377bb58dc doc: Mention the solaris jail change Message-ID: <20200911142006.D496F5437@lists.varnish-cache.org> commit 377bb58dc65642d4d6429e65a214942232fe4eb6 Author: Dridi Boukelmoune Date: Fri Sep 11 16:13:47 2020 +0200 doc: Mention the solaris jail change Yes, I missed at least one varnishd change in my first pass. diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index 8a0e637f2..0f150d59c 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -27,6 +27,10 @@ A new ``vcc_acl_pedantic`` parameter will turn warnings into errors for the case where an ACL entry includes a network prefix, but host bits aren't all zeroes. +The ``solaris`` jail has been improved and can reduce privileges even further. +There is now a new optional ``-j solaris,worker=...`` argument which allows to +extend the effective privilege set of the worker (cache) process. + Other changes in varnishd ~~~~~~~~~~~~~~~~~~~~~~~~~ From dridi.boukelmoune at gmail.com Fri Sep 11 20:05:10 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 11 Sep 2020 20:05:10 +0000 (UTC) Subject: [master] 4d6ca8c5f doc: A word on VSC_Arg() Message-ID: <20200911200510.1A43664E72@lists.varnish-cache.org> commit 4d6ca8c5f426a8277add3ca34f5dc1f282177442 Author: Dridi Boukelmoune Date: Fri Sep 11 22:02:22 2020 +0200 doc: A word on VSC_Arg() And a quick note on the workspace API plus homework for later. diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index 0f150d59c..29ccd5a5b 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -152,6 +152,9 @@ disruption. Changes for developers and VMOD authors ======================================= +VMODs +~~~~~ + The workspace API saw a number of changes in anticipation of a future inclusion in VRT. The deprecated ``WS_Reserve()`` function was finally removed, the functions ``WS_ReserveSize()`` and ``WS_ReserveAll()`` were @@ -169,10 +172,31 @@ new ``WS_ReservationSize()`` function. The return value for ``WS_Printf()`` is now a constant string. +In general, accessing any field of ``struct ws`` is strongly discouraged +and if the workspace API doesn't satisfy all your needs please bring +that to our attention. + VMOD authors who would like to generate VCC files can now use the ``VARNISH_VMODS_GENERATED()`` macro from ``varnish.m4`` for autotools builds. +libvarnishapi +~~~~~~~~~~~~~ + +There are three new VSC arguments that can be set with the ``VSC_Arg()`` +function: + +- ``'I'`` to include counters matching a glob pattern +- ``'X'`` to exclude counters matching a glob pattern +- ``'R'`` to include required counters regardless of ``'I'`` and ``'X'`` + +The ``'f'`` argument is now deprecated and emulated with ``'I'`` and ``'X'``. +Filtering with ``'f'`` used to check exclusions first and then inclusions, +they are all tested in order and the first to match determines the outcome. + +The ``'R'`` argument takes precedence over regular filtering and can be used +to ensure that some counters are present regardless of user configuration. + **XXX changes concerning VRT, the public APIs, source code organization, builds etc.** diff --git a/doc/sphinx/whats-new/upgrading-trunk.rst b/doc/sphinx/whats-new/upgrading-trunk.rst index 9922355b4..27cb67c94 100644 --- a/doc/sphinx/whats-new/upgrading-trunk.rst +++ b/doc/sphinx/whats-new/upgrading-trunk.rst @@ -61,6 +61,9 @@ After:: } } +TODO: a word on ``varnishstat -f`` being deprecated and ``-{I,X}`` being +order-sensitive. + VSL === @@ -73,4 +76,9 @@ The new query would be:: varnishlog -q 'BackendClose[2] ~ www and BackendClose[3] eq recycle' +libvarnishapi +============= + +TODO: a similar note as ``varnishstat -f`` for ``VSC_Arg('f')``. + *eof* From dridi at varni.sh Fri Sep 11 20:08:58 2020 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 11 Sep 2020 20:08:58 +0000 Subject: [master] 4d6ca8c5f doc: A word on VSC_Arg() In-Reply-To: <20200911200510.1A43664E72@lists.varnish-cache.org> References: <20200911200510.1A43664E72@lists.varnish-cache.org> Message-ID: On Fri, Sep 11, 2020 at 8:05 PM Dridi Boukelmoune wrote: > > > commit 4d6ca8c5f426a8277add3ca34f5dc1f282177442 > Author: Dridi Boukelmoune > Date: Fri Sep 11 22:02:22 2020 +0200 > > doc: A word on VSC_Arg() > > And a quick note on the workspace API plus homework for later. I got side-tracked so this is a dump of my meager progress earlier today. > +TODO: a word on ``varnishstat -f`` being deprecated and ``-{I,X}`` being > +order-sensitive. > + > +libvarnishapi > +============= > + > +TODO: a similar note as ``varnishstat -f`` for ``VSC_Arg('f')``. To anyone feeling like making progress before Monday, don't feel obliged to leave the homework to me. I will however pick it up on Monday if it's still present. Cheers From nils.goroll at uplex.de Mon Sep 14 07:39:07 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 14 Sep 2020 07:39:07 +0000 (UTC) Subject: [master] d7c349e08 changes.rst #3400 Message-ID: <20200914073908.03A9BA02FA@lists.varnish-cache.org> commit d7c349e08c35f70ad6d9c28b3938a0a31e5ea2c9 Author: Nils Goroll Date: Mon Sep 14 09:38:08 2020 +0200 changes.rst #3400 diff --git a/doc/changes.rst b/doc/changes.rst index 3982fb699..315c84514 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -215,6 +215,12 @@ NEXT (scheduled 2020-09-15) * New ``VARNISH_VMODS_GENERATED()`` macro in ``varnish.m4``. +* Prevent pooling of a ``Connection: close`` backend response. + + When this header is present, be it sent by the backend or added in + ``vcl_backend_response {}``, varnish closes the connection after the + current request. (3400_) + .. _2990: https://github.com/varnishcache/varnish-cache/issues/2990 .. _2999: https://github.com/varnishcache/varnish-cache/issues/2999 .. _3002: https://github.com/varnishcache/varnish-cache/issues/3002 @@ -232,6 +238,7 @@ NEXT (scheduled 2020-09-15) .. _3361: https://github.com/varnishcache/varnish-cache/issues/3361 .. _3378: https://github.com/varnishcache/varnish-cache/issues/3378 .. _3394: https://github.com/varnishcache/varnish-cache/issues/3394 +.. _3400: https://github.com/varnishcache/varnish-cache/issues/3400 .. _3402: https://github.com/varnishcache/varnish-cache/issues/3402 ================================ From nils.goroll at uplex.de Mon Sep 14 07:52:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 14 Sep 2020 07:52:06 +0000 (UTC) Subject: [master] 5af986947 docs: be a bit more verbose about vcc_acl_pedantic and related Message-ID: <20200914075206.20064A0A60@lists.varnish-cache.org> commit 5af98694788289485563dab34984255364eacd8c Author: Nils Goroll Date: Mon Sep 14 09:49:59 2020 +0200 docs: be a bit more verbose about vcc_acl_pedantic and related We actually changed behavior also for vcc_acl_pedantic=false diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index 29ccd5a5b..d40314b1e 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -20,12 +20,20 @@ merged, may be found in the `change log`_. varnishd ======== +Access Control Lists (ACLs) +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The VCL compiler now emits warnings if network numbers used in ACLs do +not have an all-zero host part (as, for example, +``"192.168.42.42"/24``). By default, such ACL entries are fixed to +all-zero and that fact logged with the ``ACL`` VSL tag. + Parameters ~~~~~~~~~~ -A new ``vcc_acl_pedantic`` parameter will turn warnings into errors for the -case where an ACL entry includes a network prefix, but host bits aren't all -zeroes. +A new ``vcc_acl_pedantic`` parameter, when set to ``true``, turns the +ACL warnings documented above into errors for the case where an ACL +entry includes a network prefix, but host bits aren't all zeroes. The ``solaris`` jail has been improved and can reduce privileges even further. There is now a new optional ``-j solaris,worker=...`` argument which allows to From nils.goroll at uplex.de Mon Sep 14 07:52:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 14 Sep 2020 07:52:06 +0000 (UTC) Subject: [master] 37d25b233 doc: polish Message-ID: <20200914075206.3521DA0A63@lists.varnish-cache.org> commit 37d25b23397a6e338ab925c69cde953ad29aba47 Author: Nils Goroll Date: Mon Sep 14 09:51:06 2020 +0200 doc: polish diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index d40314b1e..54a256c37 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -102,8 +102,8 @@ work with binary files. The shard director's ``.add_backend()`` method has a new optional ``weight`` parameter. An error when a backend is added or removed now fails the -transaction (or the ``vcl.load`` command in ``vcl_init``) but an invalid -weight does not result into a hard failure. +transaction (or the ``vcl.load`` command in ``vcl_init``), but an invalid +weight does not result in a hard failure. The shard director no longer outputs the (unused) ``canon_point`` property in ``backend.list`` commands. From nils.goroll at uplex.de Mon Sep 14 07:59:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 14 Sep 2020 07:59:06 +0000 (UTC) Subject: [master] fd394906b doc: polish Message-ID: <20200914075906.CFE03A1030@lists.varnish-cache.org> commit fd394906bde97490c864496f2231fcfdb557a82b Author: Nils Goroll Date: Mon Sep 14 09:57:28 2020 +0200 doc: polish diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index 54a256c37..d57a02e2b 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -111,9 +111,10 @@ in ``backend.list`` commands. varnishlog ========== -The ``BackendReuse`` log record has been retired. It was inconsistently named -compared to other places like stat counters where we use the words reuse and -recycle (it should have been named ``BackendRecycle`` if anything). +The ``BackendReuse`` log record has been retired. It was named +inconsistently with other places like stat counters where we use the +words reuse and recycle (it should have been named ``BackendRecycle`` +if anything). The ``BackendOpen`` record can now tell whether the connection to the backend was opened or reused from the pool, and the ``BackendClose`` record will tell From nils.goroll at uplex.de Mon Sep 14 07:59:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 14 Sep 2020 07:59:06 +0000 (UTC) Subject: [master] 7786bd8d1 doc: attempt to rephrase for clarity Message-ID: <20200914075906.E4F46A1034@lists.varnish-cache.org> commit 7786bd8d1396f3a60c23057091c5b34b81e3e275 Author: Nils Goroll Date: Mon Sep 14 09:57:56 2020 +0200 doc: attempt to rephrase for clarity diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index d57a02e2b..0b1c9dd45 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -135,9 +135,9 @@ varnishstat A help screen is now available in interactive mode via the ``h`` key. -Again in interactive mode, verbosity is increased from the default value -during startup when the filtering of counters would otherwise display -nothing. +Again in interactive mode, the initial verbosity is now chosen such +that fields selected via the ``-f`` or ``-I`` options are actually +displayed without manually increasing the verbosity level. Filtering using the ``-f`` option is now deprecated in favor of ``-I`` and ``-X`` options that are treated in order. While still present, the ``-f`` From nils.goroll at uplex.de Mon Sep 14 08:03:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 14 Sep 2020 08:03:06 +0000 (UTC) Subject: [master] a1c7b5e92 doc: polish Message-ID: <20200914080306.52851A152C@lists.varnish-cache.org> commit a1c7b5e92c768bf9c12295d2de39fa19d6e30a07 Author: Nils Goroll Date: Mon Sep 14 10:02:13 2020 +0200 doc: polish diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index 0b1c9dd45..9297f219e 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -166,8 +166,8 @@ VMODs The workspace API saw a number of changes in anticipation of a future inclusion in VRT. The deprecated ``WS_Reserve()`` function was finally -removed, the functions ``WS_ReserveSize()`` and ``WS_ReserveAll()`` were -introduced as a replacement. +removed, after the functions ``WS_ReserveSize()`` and +``WS_ReserveAll()`` were introduced in Varnish Cache 6.3.0. On the topic of workspace reservation, the ``WS_Front()`` function is now deprecated in favor of ``WS_Reservation()``. The two functions From nils.goroll at uplex.de Mon Sep 14 08:17:07 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 14 Sep 2020 08:17:07 +0000 (UTC) Subject: [master] fde6d55e5 doc: libvarnish changes and other additions, minor reordering Message-ID: <20200914081707.76461A1C85@lists.varnish-cache.org> commit fde6d55e5b8c97b5f311478d92eb879c08bedef6 Author: Nils Goroll Date: Mon Sep 14 10:16:26 2020 +0200 doc: libvarnish changes and other additions, minor reordering diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index 9297f219e..57dcb87b3 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -94,6 +94,9 @@ effectively ending the transaction, this has been corrected. A failure in There is a new syntax for ``BLOB`` literals: ``::``. This syntax is also used to automatically cast a blob into a string. +Behavior for 304 responses was changed not to update the +``Content-Encoding`` response header of the stored object. + VMODs ===== @@ -161,8 +164,15 @@ disruption. Changes for developers and VMOD authors ======================================= -VMODs -~~~~~ +Build System +~~~~~~~~~~~~ + +VMOD authors who would like to generate VCC files can now use the +``VARNISH_VMODS_GENERATED()`` macro from ``varnish.m4`` for autotools +builds. + +Workspace API +~~~~~~~~~~~~~ The workspace API saw a number of changes in anticipation of a future inclusion in VRT. The deprecated ``WS_Reserve()`` function was finally @@ -185,9 +195,12 @@ In general, accessing any field of ``struct ws`` is strongly discouraged and if the workspace API doesn't satisfy all your needs please bring that to our attention. -VMOD authors who would like to generate VCC files can now use the -``VARNISH_VMODS_GENERATED()`` macro from ``varnish.m4`` for autotools -builds. +Other VRT / cache.h changes +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +* Added ``VRT_DirectorResolve()`` to resolve a director + +* Added ``VRT_BLOB_string()`` for the default BLOB folding documented above libvarnishapi ~~~~~~~~~~~~~ @@ -206,7 +219,29 @@ they are all tested in order and the first to match determines the outcome. The ``'R'`` argument takes precedence over regular filtering and can be used to ensure that some counters are present regardless of user configuration. -**XXX changes concerning VRT, the public APIs, source code organization, -builds etc.** +libvarnish +~~~~~~~~~~ + +* ``VSA_BuildFAP()`` has been added as a convenience function to + build a ``struct suckaddr`` (aka ``VCL_IP``). + +* Added ``VRE_quote()`` to facilitate building literal string matches + with regular expressions. + +* The varnish binary heap implementation has been added + ``VBH_`` prefix, complemented with a destructor and added to header + files for use with vmods (via include of ``vbh.h``). + +* VSB support for dynamic vs. static allocations has been changed: + + For dynamic allocations use:: + + VSB_new_auto() + VSB_destroy() + + For preexisting buffers use:: + + VSB_init() + VSB_fini() + + ``VSB_new()`` + ``VSB_delete()`` are now deprecated. *eof* From dridi.boukelmoune at gmail.com Mon Sep 14 08:37:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 14 Sep 2020 08:37:06 +0000 (UTC) Subject: [master] 82cebdabe doc: Polish Message-ID: <20200914083706.7A529A2590@lists.varnish-cache.org> commit 82cebdabec9f1e11e07cdabe9ba06b5142da101c Author: Dridi Boukelmoune Date: Mon Sep 14 10:36:12 2020 +0200 doc: Polish diff --git a/doc/changes.rst b/doc/changes.rst index 315c84514..ffd2009cd 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -188,7 +188,7 @@ NEXT (scheduled 2020-09-15) * ``std.blobread()`` has been added -* New ``main.beresp_uncacheable`` and ``main.beresp_shortlived`` +* New ``MAIN.beresp_uncacheable`` and ``MAIN.beresp_shortlived`` counters have been added. * The ``I``, ``X`` and ``R`` arguments have been added to the VSC API From nils.goroll at uplex.de Mon Sep 14 08:43:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 14 Sep 2020 08:43:06 +0000 (UTC) Subject: [master] f7869e8e9 doc: upgrading Message-ID: <20200914084307.05A22A297A@lists.varnish-cache.org> commit f7869e8e98167ceae3189d027f42ccb421cb5e72 Author: Nils Goroll Date: Mon Sep 14 10:40:20 2020 +0200 doc: upgrading diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index 57dcb87b3..04bf7938d 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -171,6 +171,8 @@ VMOD authors who would like to generate VCC files can now use the ``VARNISH_VMODS_GENERATED()`` macro from ``varnish.m4`` for autotools builds. +.. _whatsnew_changes_CURRENT_workspace: + Workspace API ~~~~~~~~~~~~~ @@ -202,6 +204,8 @@ Other VRT / cache.h changes * Added ``VRT_BLOB_string()`` for the default BLOB folding documented above +.. _whatsnew_changes_CURRENT_vsc: + libvarnishapi ~~~~~~~~~~~~~ @@ -219,6 +223,8 @@ they are all tested in order and the first to match determines the outcome. The ``'R'`` argument takes precedence over regular filtering and can be used to ensure that some counters are present regardless of user configuration. +.. _whatsnew_changes_CURRENT_libvarnish: + libvarnish ~~~~~~~~~~ diff --git a/doc/sphinx/whats-new/upgrading-trunk.rst b/doc/sphinx/whats-new/upgrading-trunk.rst index 27cb67c94..b92602c12 100644 --- a/doc/sphinx/whats-new/upgrading-trunk.rst +++ b/doc/sphinx/whats-new/upgrading-trunk.rst @@ -61,8 +61,10 @@ After:: } } -TODO: a word on ``varnishstat -f`` being deprecated and ``-{I,X}`` being -order-sensitive. +The filter option ``-f`` is now deprecated in favor of the ``-I`` and +``-X`` options for field inclusions and exclusions, +respectively. Tools using ``varnishstat`` should prepare for future +removal and be changed accordingly. VSL === @@ -76,9 +78,18 @@ The new query would be:: varnishlog -q 'BackendClose[2] ~ www and BackendClose[3] eq recycle' -libvarnishapi -============= +Changes relevant for VMODs and/or VAPI/VUT clients +================================================== -TODO: a similar note as ``varnishstat -f`` for ``VSC_Arg('f')``. +* VMODs using the Workspace API might need minor adjustments, see + :ref:`whatsnew_changes_CURRENT_workspace`. + +* ``VSC_Arg('f')`` is now deprecated and should be rewritten to use + ``VSC_Arg('I')`` / ``VSC_Arg('X')``, see above note on + `varnishstat`_ and :ref:`whatsnew_changes_CURRENT_vsc`. + +* VSB support for dynamic vs. static allocations has been changed and + code using VSBs will need to be adjusted, see + :ref:`whatsnew_changes_CURRENT_libvarnish`. *eof* From nils.goroll at uplex.de Mon Sep 14 08:46:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 14 Sep 2020 08:46:06 +0000 (UTC) Subject: [master] 61a058415 doc: fix glitch Message-ID: <20200914084606.7BC7EA2C13@lists.varnish-cache.org> commit 61a058415a67d6bb33b0d486c86882fa9d6177c8 Author: Nils Goroll Date: Mon Sep 14 10:45:28 2020 +0200 doc: fix glitch diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index 04bf7938d..2b7d1098a 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -234,9 +234,8 @@ libvarnish * Added ``VRE_quote()`` to facilitate building literal string matches with regular expressions. -* The varnish binary heap implementation has been added - ``VBH_`` prefix, complemented with a destructor and added to header - files for use with vmods (via include of ``vbh.h``). +* The varnish binary heap implementation has been added with the + ``VBH_`` prefix for use with vmods (via include of ``vbh.h``). * VSB support for dynamic vs. static allocations has been changed: From dridi.boukelmoune at gmail.com Mon Sep 14 10:26:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 14 Sep 2020 10:26:06 +0000 (UTC) Subject: [master] f29ee2c7f doc: Elaborate a bit more on release notes Message-ID: <20200914102606.A1BD8A57D8@lists.varnish-cache.org> commit f29ee2c7ff8fbdcbad6ad7359016575760ed1576 Author: Dridi Boukelmoune Date: Mon Sep 14 12:22:49 2020 +0200 doc: Elaborate a bit more on release notes And slightly move things around. There are interesting details worth mentioning. Better diff with the --word-diff --ignore-all-space options. diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst index 2b7d1098a..637b63ad8 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -193,10 +193,6 @@ new ``WS_ReservationSize()`` function. The return value for ``WS_Printf()`` is now a constant string. -In general, accessing any field of ``struct ws`` is strongly discouraged -and if the workspace API doesn't satisfy all your needs please bring -that to our attention. - Other VRT / cache.h changes ~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -228,25 +224,28 @@ to ensure that some counters are present regardless of user configuration. libvarnish ~~~~~~~~~~ -* ``VSA_BuildFAP()`` has been added as a convenience function to - build a ``struct suckaddr`` (aka ``VCL_IP``). +A ``VSA_BuildFAP()`` function has been added as a convenience to build a +``struct suckaddr`` (aka ``VCL_IP``) from a Family, Address and Protocol +components. -* Added ``VRE_quote()`` to facilitate building literal string matches - with regular expressions. +We added ``VRE_quote()`` to facilitate building literal string matches +with regular expressions. It can be used to ensure that a user-defined +string literal put inside a regular expression may not accidentally +change the behavior of the overall expression. -* The varnish binary heap implementation has been added with the - ``VBH_`` prefix for use with vmods (via include of ``vbh.h``). +The varnish binary heap implementation has been added with the +``VBH_`` prefix for use with VMODs (via include of ``vbh.h``). -* VSB support for dynamic vs. static allocations has been changed: +VSB support for dynamic vs. static allocations has been changed: - For dynamic allocations use:: +For dynamic allocations use:: - VSB_new_auto() + VSB_destroy() + VSB_new_auto() + VSB_destroy() - For preexisting buffers use:: +For preexisting buffers use:: - VSB_init() + VSB_fini() + VSB_init() + VSB_fini() - ``VSB_new()`` + ``VSB_delete()`` are now deprecated. +``VSB_new()`` + ``VSB_delete()`` are now deprecated. *eof* diff --git a/doc/sphinx/whats-new/upgrading-trunk.rst b/doc/sphinx/whats-new/upgrading-trunk.rst index b92602c12..c907a66db 100644 --- a/doc/sphinx/whats-new/upgrading-trunk.rst +++ b/doc/sphinx/whats-new/upgrading-trunk.rst @@ -8,28 +8,6 @@ released versions of Varnish, see:** :ref:`whats-new-index` Upgrading to Varnish **$NEXT_RELEASE** %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -**XXX: how to upgrade from previous deployments to this -version. Limited to work that has to be done for an upgrade, new -features are listed in "Changes". Explicitly mention what does *not* -have to be changed, especially in VCL. May include, but is not limited -to:** - -* Elements of VCL that have been removed or are deprecated, or whose - semantics have changed. - -* -p parameters that have been removed or are deprecated, or whose - semantics have changed. - -* Changes in the CLI. - -* Changes in the output or interpretation of stats or the log, including - changes affecting varnishncsa/-hist/-top. - -* Changes that may be necessary in VTCs or in the use of varnishtest. - -* Changes in public APIs that may require changes in VMODs or VAPI/VUT - clients. - varnishstat =========== @@ -62,9 +40,9 @@ After:: } The filter option ``-f`` is now deprecated in favor of the ``-I`` and -``-X`` options for field inclusions and exclusions, -respectively. Tools using ``varnishstat`` should prepare for future -removal and be changed accordingly. +``-X`` options for field inclusions and exclusions, respectively. Tools +using ``varnishstat`` should prepare for future removal and be changed +accordingly. VSL === @@ -78,18 +56,44 @@ The new query would be:: varnishlog -q 'BackendClose[2] ~ www and BackendClose[3] eq recycle' -Changes relevant for VMODs and/or VAPI/VUT clients -================================================== +Changes for developers and VMOD authors +======================================= + +VSB +~~~ + +VSB support for dynamic vs. static allocations has been changed and +code using VSBs will need to be adjusted, see +:ref:`whatsnew_changes_CURRENT_libvarnish`. + +It should be noted that the VSB itself and the string buffer must be either +both dynamic or both static. It is no longer possible for example to have +a static ``struct`` with a dynamic buffer with the new API. + +Workspace API +~~~~~~~~~~~~~ + +VMODs using the Workspace API might need minor adjustments, see +:ref:`whatsnew_changes_CURRENT_workspace`. + +In general, accessing any field of ``struct ws`` is strongly discouraged +and if the workspace API doesn't satisfy all your needs please bring +that to our attention. + +VSC +~~~ -* VMODs using the Workspace API might need minor adjustments, see - :ref:`whatsnew_changes_CURRENT_workspace`. +The ``'f'`` argument for ``VSC_Arg()`` is now deprecated as mentioned in +the above note on `varnishstat`_ and :ref:`whatsnew_changes_CURRENT_vsc`. -* ``VSC_Arg('f')`` is now deprecated and should be rewritten to use - ``VSC_Arg('I')`` / ``VSC_Arg('X')``, see above note on - `varnishstat`_ and :ref:`whatsnew_changes_CURRENT_vsc`. +Otherwise you can use the ``'I'`` ans ``'X'`` arguments to respectively +include or exclude counters, they work in a first-match fashion. Since +``'f'`` is now emulated using the new arguments, its filtering behavior +slightly changed from exclusions first to first match. -* VSB support for dynamic vs. static allocations has been changed and - code using VSBs will need to be adjusted, see - :ref:`whatsnew_changes_CURRENT_libvarnish`. +If like ``varnishstat`` in curses mode, you have a utility that always +needs some counters to be present the ``'R'`` argument takes a glob of +required fields. Such counters are not affected by filtering from other +``VSC_Arg()`` arguments. *eof* From dridi.boukelmoune at gmail.com Mon Sep 14 10:26:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 14 Sep 2020 10:26:06 +0000 (UTC) Subject: [master] 5f13f6285 vsc: Deprecation notice Message-ID: <20200914102606.BA1C0A57DB@lists.varnish-cache.org> commit 5f13f6285eab72d15cbeb387ac76a8e9e65be621 Author: Dridi Boukelmoune Date: Mon Sep 14 12:24:46 2020 +0200 vsc: Deprecation notice diff --git a/include/vapi/vsc.h b/include/vapi/vsc.h index ec6c2b92d..c09d6b150 100644 --- a/include/vapi/vsc.h +++ b/include/vapi/vsc.h @@ -117,7 +117,7 @@ int VSC_Arg(struct vsc *, char arg, const char *opt); * 'I' - field inclusion glob * 'X' - field exclusion glob * 'R' - required field glob - * 'f' - legacy field filter glob + * 'f' - legacy field filter glob (deprecated) * * Return: * -1 error, VSM_Error() returns diagnostic string From dridi.boukelmoune at gmail.com Mon Sep 14 13:49:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 14 Sep 2020 13:49:06 +0000 (UTC) Subject: [master] bdd62b036 Code style OCD Message-ID: <20200914134906.3ED80AA813@lists.varnish-cache.org> commit bdd62b036c7bc81151c1688ef75432e7c6111d2f Author: Dridi Boukelmoune Date: Mon Sep 14 15:47:53 2020 +0200 Code style OCD diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index c09ecafe7..b7243235a 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -108,7 +108,8 @@ vbf_cleanup(struct busyobj *bo) VDI_Finish(bo); } -void Bereq_Rollback(struct busyobj *bo) +void +Bereq_Rollback(struct busyobj *bo) { CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); From dridi.boukelmoune at gmail.com Mon Sep 14 15:27:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 14 Sep 2020 15:27:06 +0000 (UTC) Subject: [master] f4c763a8a git: Ignore vbh_test Message-ID: <20200914152706.5CB6CACBED@lists.varnish-cache.org> commit f4c763a8ae91099055d5d25a131577fe7fa1b637 Author: Dridi Boukelmoune Date: Mon Sep 14 17:26:20 2020 +0200 git: Ignore vbh_test diff --git a/.gitignore b/.gitignore index 4d0db9281..0c5a0624e 100644 --- a/.gitignore +++ b/.gitignore @@ -117,7 +117,7 @@ cscope.*out /include/_vrt_test /include/vrt_test /include/vbm_test -/lib/libvarnish/binheap +/lib/libvarnish/vbh_test /lib/libvarnish/vjsn_test /lib/libvarnish/vnum_c_test /lib/libvarnish/vsb_test From reza at naghibi.com Mon Sep 14 18:13:09 2020 From: reza at naghibi.com (Reza Naghibi) Date: Mon, 14 Sep 2020 18:13:09 +0000 (UTC) Subject: [6.0] 729a842b9 CHECK_OBJ before deferencing pointer Message-ID: <20200914181309.85AAC2DE1@lists.varnish-cache.org> commit 729a842b9570b1991d2353a6807ea4aea064e4ed Author: Reza Naghibi Date: Mon Sep 14 14:12:14 2020 -0400 CHECK_OBJ before deferencing pointer diff --git a/bin/varnishd/cache/cache_vrt_priv.c b/bin/varnishd/cache/cache_vrt_priv.c index 05aae3f35..f7cf37455 100644 --- a/bin/varnishd/cache/cache_vrt_priv.c +++ b/bin/varnishd/cache/cache_vrt_priv.c @@ -203,8 +203,8 @@ VCL_TaskLeave(const struct vcl *vcl, struct vrt_privs *privs) AN(vcl); CHECK_OBJ_NOTNULL(privs, VRT_PRIVS_MAGIC); VTAILQ_FOREACH_SAFE(vp, &privs->privs, list, vp1) { - VTAILQ_REMOVE(&privs->privs, vp, list); CHECK_OBJ_NOTNULL(vp, VRT_PRIV_MAGIC); + VTAILQ_REMOVE(&privs->privs, vp, list); VRT_priv_fini(vp->priv); ZERO_OBJ(vp, sizeof *vp); } From dridi.boukelmoune at gmail.com Tue Sep 15 06:55:08 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 15 Sep 2020 06:55:08 +0000 (UTC) Subject: [master] 05f75a15b doc: Move the release notes to 6.5 Message-ID: <20200915065508.23EE99B37E@lists.varnish-cache.org> commit 05f75a15bbc75c2ac440bc913d58b6ce7aa20069 Author: Dridi Boukelmoune Date: Tue Sep 15 08:53:41 2020 +0200 doc: Move the release notes to 6.5 diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-6.5.rst similarity index 94% rename from doc/sphinx/whats-new/changes-trunk.rst rename to doc/sphinx/whats-new/changes-6.5.rst index 637b63ad8..0dbe26afa 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-6.5.rst @@ -1,15 +1,11 @@ -**Note: This is a working document for a future release, with running -updates for changes in the development branch. For changes in the -released versions of Varnish, see:** :ref:`whats-new-index` +.. _whatsnew_changes_6.5: -.. _whatsnew_changes_CURRENT: - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -Changes in Varnish **$NEXT_RELEASE** -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%% +Changes in Varnish 6.5.0 +%%%%%%%%%%%%%%%%%%%%%%%% For information about updating your current Varnish deployment to the -new version, see :ref:`whatsnew_upgrading_CURRENT`. +new version, see :ref:`whatsnew_upgrading_6.5`. A more detailed and technical account of changes in Varnish, with links to issues that have been fixed and pull requests that have been @@ -171,7 +167,7 @@ VMOD authors who would like to generate VCC files can now use the ``VARNISH_VMODS_GENERATED()`` macro from ``varnish.m4`` for autotools builds. -.. _whatsnew_changes_CURRENT_workspace: +.. _whatsnew_changes_6.5_workspace: Workspace API ~~~~~~~~~~~~~ @@ -200,7 +196,7 @@ Other VRT / cache.h changes * Added ``VRT_BLOB_string()`` for the default BLOB folding documented above -.. _whatsnew_changes_CURRENT_vsc: +.. _whatsnew_changes_6.5_vsc: libvarnishapi ~~~~~~~~~~~~~ @@ -219,7 +215,7 @@ they are all tested in order and the first to match determines the outcome. The ``'R'`` argument takes precedence over regular filtering and can be used to ensure that some counters are present regardless of user configuration. -.. _whatsnew_changes_CURRENT_libvarnish: +.. _whatsnew_changes_6.5_libvarnish: libvarnish ~~~~~~~~~~ diff --git a/doc/sphinx/whats-new/index.rst b/doc/sphinx/whats-new/index.rst index 29dc31d84..4c2e31a20 100644 --- a/doc/sphinx/whats-new/index.rst +++ b/doc/sphinx/whats-new/index.rst @@ -8,18 +8,14 @@ This section describes the changes and improvements between different versions of Varnish, and what upgrading between the different versions entail. -Varnish **$NEXT_RELEASE** -------------------------- - -**Note: These are working documents for a future release, with running -updates for changes in the development branch. For changes in the -released versions of Varnish, see the chapters listed below.** +Varnish 6.5 +----------- .. toctree:: :maxdepth: 2 - changes-trunk - upgrading-trunk + changes-6.5 + upgrading-6.5 Varnish 6.4 ----------- diff --git a/doc/sphinx/whats-new/upgrading-trunk.rst b/doc/sphinx/whats-new/upgrading-6.5.rst similarity index 82% rename from doc/sphinx/whats-new/upgrading-trunk.rst rename to doc/sphinx/whats-new/upgrading-6.5.rst index c907a66db..6a1771036 100644 --- a/doc/sphinx/whats-new/upgrading-trunk.rst +++ b/doc/sphinx/whats-new/upgrading-6.5.rst @@ -1,12 +1,8 @@ -**Note: This is a working document for a future release, with running -updates for changes in the development branch. For changes in the -released versions of Varnish, see:** :ref:`whats-new-index` +.. _whatsnew_upgrading_6.5: -.. _whatsnew_upgrading_CURRENT: - -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -Upgrading to Varnish **$NEXT_RELEASE** -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%%%% +Upgrading to Varnish 6.5.0 +%%%%%%%%%%%%%%%%%%%%%%%%%% varnishstat =========== @@ -64,7 +60,7 @@ VSB VSB support for dynamic vs. static allocations has been changed and code using VSBs will need to be adjusted, see -:ref:`whatsnew_changes_CURRENT_libvarnish`. +:ref:`whatsnew_changes_6.5_libvarnish`. It should be noted that the VSB itself and the string buffer must be either both dynamic or both static. It is no longer possible for example to have @@ -74,7 +70,7 @@ Workspace API ~~~~~~~~~~~~~ VMODs using the Workspace API might need minor adjustments, see -:ref:`whatsnew_changes_CURRENT_workspace`. +:ref:`whatsnew_changes_6.5_workspace`. In general, accessing any field of ``struct ws`` is strongly discouraged and if the workspace API doesn't satisfy all your needs please bring @@ -84,7 +80,7 @@ VSC ~~~ The ``'f'`` argument for ``VSC_Arg()`` is now deprecated as mentioned in -the above note on `varnishstat`_ and :ref:`whatsnew_changes_CURRENT_vsc`. +the above note on `varnishstat`_ and :ref:`whatsnew_changes_6.5_vsc`. Otherwise you can use the ``'I'`` ans ``'X'`` arguments to respectively include or exclude counters, they work in a first-match fashion. Since From dridi.boukelmoune at gmail.com Tue Sep 15 07:04:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 15 Sep 2020 07:04:06 +0000 (UTC) Subject: [master] 62925c477 http: Don't expect an overflowed workspace on failure Message-ID: <20200915070406.CFD149B85C@lists.varnish-cache.org> commit 62925c477b8bef8d89a564a792e1d32945e38ab5 Author: Dridi Boukelmoune Date: Tue Sep 15 09:01:09 2020 +0200 http: Don't expect an overflowed workspace on failure If we lose a header from http_SetHeader() we don't formally run out of workspace prior to calling http_fail(). Refs #3410 diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c index 9760f79a1..9fa4c1f64 100644 --- a/bin/varnishd/cache/cache_http.c +++ b/bin/varnishd/cache/cache_http.c @@ -117,7 +117,7 @@ http_fail(const struct http *hp) VSC_C_main->losthdr++; WS_Id(hp->ws, id); VSLb(hp->vsl, SLT_Error, "out of workspace (%s)", id); - assert(WS_Overflowed(hp->ws)); + WS_MarkOverflow(hp->ws); } /*-------------------------------------------------------------------- From dridi.boukelmoune at gmail.com Tue Sep 15 08:10:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 15 Sep 2020 08:10:06 +0000 (UTC) Subject: [master] e6f934e1b http: Lost header workspace overflow coverage Message-ID: <20200915081006.D9B359D338@lists.varnish-cache.org> commit e6f934e1b61974958e0f2ce3ca1e4d763ca0a4a6 Author: Dridi Boukelmoune Date: Tue Sep 15 10:08:01 2020 +0200 http: Lost header workspace overflow coverage Closes #3410 diff --git a/bin/varnishtest/tests/r03410.vtc b/bin/varnishtest/tests/r03410.vtc new file mode 100644 index 000000000..627667bfc --- /dev/null +++ b/bin/varnishtest/tests/r03410.vtc @@ -0,0 +1,48 @@ +varnishtest "lost header workspace overflow" + +server s1 { + rxreq + send "HTTP/1.1 200 OK\r\n" + send "Extra: 1\r\n" + send "Extra: 2\r\n" + send "Extra: 3\r\n" + send "Extra: 4\r\n" + send "Extra: 5\r\n" + send "Extra: 6\r\n" + send "Extra: 7\r\n" + send "Extra: 8\r\n" + send "Extra: 9\r\n" + send "Extra: 10\r\n" + send "Extra: 11\r\n" + send "Extra: 12\r\n" + send "Extra: 13\r\n" + send "Extra: 14\r\n" + send "Extra: 15\r\n" + send "Extra: 16\r\n" + send "Extra: 17\r\n" + send "Extra: 18\r\n" + send "Extra: 19\r\n" + send "Extra: 20\r\n" + send "Extra: 21\r\n" + send "Extra: 22\r\n" + send "Extra: 23\r\n" + send "Extra: 24\r\n" + send "Extra: 25\r\n" + send "Extra: 26\r\n" + # The start line consumes 5 header slots + send "\r\n" +} -start + +varnish v1 -cliok "param.set http_max_hdr 32" +varnish v1 -vcl+backend { + sub vcl_backend_response { + # This is one header too many + set beresp.http.Lost = "header"; + } +} -start + +client c1 { + txreq + rxresp + expect resp.status == 500 +} -run From dridi.boukelmoune at gmail.com Tue Sep 15 15:07:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 15 Sep 2020 15:07:06 +0000 (UTC) Subject: [master] b79e940e7 Prepare for 6.5.0 Message-ID: <20200915150706.CFCD6A8E76@lists.varnish-cache.org> commit b79e940e716474ab5186336db2a86461b32f2413 Author: Dridi Boukelmoune Date: Tue Sep 15 15:43:21 2020 +0200 Prepare for 6.5.0 diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index c6e4218f4..919a34d15 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -141,7 +141,7 @@ Resp_Setup_Deliver(struct req *req) http_PrintfHeader(h, "Age: %.0f", floor(fmax(0., req->t_prev - oc->t_origin))); - http_SetHeader(h, "Via: 1.1 varnish (Varnish/6.4)"); + http_SetHeader(h, "Via: 1.1 varnish (Varnish/6.5)"); if (cache_param->http_gzip_support && ObjCheckFlag(req->wrk, oc, OF_GZIPED) && diff --git a/configure.ac b/configure.ac index b63840512..70aa3aed7 100644 --- a/configure.ac +++ b/configure.ac @@ -2,7 +2,7 @@ AC_PREREQ(2.59) AC_COPYRIGHT([Copyright (c) 2006 Verdens Gang AS Copyright (c) 2006-2020 Varnish Software]) AC_REVISION([$Id$]) -AC_INIT([Varnish], [trunk], [varnish-dev at varnish-cache.org]) +AC_INIT([Varnish], [6.5.0], [varnish-dev at varnish-cache.org]) AC_CONFIG_SRCDIR(include/miniobj.h) AC_CONFIG_HEADERS([config.h]) AC_CONFIG_MACRO_DIR([m4]) diff --git a/doc/sphinx/index.rst b/doc/sphinx/index.rst index e5c33b1e4..83f0cdc79 100644 --- a/doc/sphinx/index.rst +++ b/doc/sphinx/index.rst @@ -37,7 +37,7 @@ Longer listings like example command output and VCL look like this:: $ /opt/varnish/sbin/varnishd -V varnishd (varnish-trunk revision 199de9b) Copyright (c) 2006 Verdens Gang AS - Copyright (c) 2006-2020 Varnish Software AS + Copyright (c) 2006-2020 Varnish Software .. For maintainers: diff --git a/include/vrt.h b/include/vrt.h index f80d5fa47..c4858cd8b 100644 --- a/include/vrt.h +++ b/include/vrt.h @@ -52,7 +52,7 @@ * Whenever something is deleted or changed in a way which is not * binary/load-time compatible, increment MAJOR version * - * NEXT (2020-09-15) + * 12.0 (2020-09-15) * Added VRT_DirectorResolve() * Added VCL_STRING VRT_BLOB_string(VRT_CTX, VCL_BLOB) * [cache.h] WS_Reserve() removed diff --git a/lib/libvarnish/version.c b/lib/libvarnish/version.c index f54e78407..9bc77a378 100644 --- a/lib/libvarnish/version.c +++ b/lib/libvarnish/version.c @@ -74,7 +74,7 @@ VCS_String(const char *which) ")" "\n" "Copyright (c) 2006 Verdens Gang AS\n" - "Copyright (c) 2006-2020 Varnish Software AS\n" + "Copyright (c) 2006-2020 Varnish Software\n" ); default: WRONG("Wrong argument to VCS_String"); From nils.goroll at uplex.de Fri Sep 18 11:43:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 18 Sep 2020 11:43:06 +0000 (UTC) Subject: [master] afd4b8331 Typo: client.ip equal remote.ip, not local.ip Message-ID: <20200918114306.D8EA76E19D@lists.varnish-cache.org> commit afd4b83310067fe53a4b9a61178545487e0abe30 Author: ThijsFeryn Date: Fri Sep 18 13:05:21 2020 +0200 Typo: client.ip equal remote.ip, not local.ip diff --git a/doc/sphinx/reference/vcl_var.rst b/doc/sphinx/reference/vcl_var.rst index 30aef93cf..a6e67d2d2 100644 --- a/doc/sphinx/reference/vcl_var.rst +++ b/doc/sphinx/reference/vcl_var.rst @@ -110,7 +110,7 @@ client.ip Readable from: client, backend - The client's IP address, either the same as ``local.ip`` + The client's IP address, either the same as ``remote.ip`` or what the PROXY protocol told us. client.identity From fgsch at lodoss.net Sat Sep 19 13:15:11 2020 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Sat, 19 Sep 2020 13:15:11 +0000 (UTC) Subject: [master] 191ca4b5d Remove Message-ID: <20200919131511.54D5263535@lists.varnish-cache.org> commit 191ca4b5d17f29430544c3616da48a465392935b Author: Federico G. Schwindt Date: Sat Sep 19 14:12:57 2020 +0100 Remove If direnv is installed you will get a warning otherwise. diff --git a/.envrc b/.envrc deleted file mode 100644 index 79286502e..000000000 --- a/.envrc +++ /dev/null @@ -1,5 +0,0 @@ -DETECT_LEAKS=$(test $(uname) = "Darwin"; echo $?) -export ASAN_OPTIONS=abort_on_error=1,detect_odr_violation=1,detect_leaks=$DETECT_LEAKS,detect_stack_use_after_return=1,detect_invalid_pointer_pairs=1,handle_segv=0,handle_sigbus=0,use_sigaltstack=0,disable_coredump=0 -export LSAN_OPTIONS=abort_on_error=1,use_sigaltstack=0,suppressions=$(pwd)/tools/lsan.suppr -export TSAN_OPTIONS=abort_on_error=1,halt_on_error=1,use_sigaltstack=0,suppressions=$(pwd)/tools/tsan.suppr -export UBSAN_OPTIONS=halt_on_error=1,print_stacktrace=1,use_sigaltstack=0,suppressions=$(pwd)/tools/ubsan.suppr From nils.goroll at uplex.de Thu Sep 24 12:08:04 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 24 Sep 2020 12:08:04 +0000 (UTC) Subject: [6.5] afd4b8331 Typo: client.ip equal remote.ip, not local.ip Message-ID: <20200924120804.BA52EA32DB@lists.varnish-cache.org> commit afd4b83310067fe53a4b9a61178545487e0abe30 Author: ThijsFeryn Date: Fri Sep 18 13:05:21 2020 +0200 Typo: client.ip equal remote.ip, not local.ip diff --git a/doc/sphinx/reference/vcl_var.rst b/doc/sphinx/reference/vcl_var.rst index 30aef93cf..a6e67d2d2 100644 --- a/doc/sphinx/reference/vcl_var.rst +++ b/doc/sphinx/reference/vcl_var.rst @@ -110,7 +110,7 @@ client.ip Readable from: client, backend - The client's IP address, either the same as ``local.ip`` + The client's IP address, either the same as ``remote.ip`` or what the PROXY protocol told us. client.identity From fgsch at lodoss.net Thu Sep 24 12:08:04 2020 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Thu, 24 Sep 2020 12:08:04 +0000 (UTC) Subject: [6.5] 191ca4b5d Remove Message-ID: <20200924120804.D0E20A32DF@lists.varnish-cache.org> commit 191ca4b5d17f29430544c3616da48a465392935b Author: Federico G. Schwindt Date: Sat Sep 19 14:12:57 2020 +0100 Remove If direnv is installed you will get a warning otherwise. diff --git a/.envrc b/.envrc deleted file mode 100644 index 79286502e..000000000 --- a/.envrc +++ /dev/null @@ -1,5 +0,0 @@ -DETECT_LEAKS=$(test $(uname) = "Darwin"; echo $?) -export ASAN_OPTIONS=abort_on_error=1,detect_odr_violation=1,detect_leaks=$DETECT_LEAKS,detect_stack_use_after_return=1,detect_invalid_pointer_pairs=1,handle_segv=0,handle_sigbus=0,use_sigaltstack=0,disable_coredump=0 -export LSAN_OPTIONS=abort_on_error=1,use_sigaltstack=0,suppressions=$(pwd)/tools/lsan.suppr -export TSAN_OPTIONS=abort_on_error=1,halt_on_error=1,use_sigaltstack=0,suppressions=$(pwd)/tools/tsan.suppr -export UBSAN_OPTIONS=halt_on_error=1,print_stacktrace=1,use_sigaltstack=0,suppressions=$(pwd)/tools/ubsan.suppr From martin at varnish-software.com Fri Sep 25 09:14:07 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 25 Sep 2020 09:14:07 +0000 (UTC) Subject: [master] f3614262f Bump the VRT major version Message-ID: <20200925091407.1295F96576@lists.varnish-cache.org> commit f3614262f130aedaae4f8ed975b3801607ddce5d Author: Martin Blix Grydeland Date: Wed Sep 23 11:46:09 2020 +0200 Bump the VRT major version In the Varnish Cache 6.5.0 release we forgot to bump the VRT major version. Do it now. diff --git a/include/vrt.h b/include/vrt.h index c4858cd8b..5b16c6261 100644 --- a/include/vrt.h +++ b/include/vrt.h @@ -166,7 +166,7 @@ * vrt_acl type added */ -#define VRT_MAJOR_VERSION 11U +#define VRT_MAJOR_VERSION 12U #define VRT_MINOR_VERSION 0U From martin at varnish-software.com Fri Sep 25 09:14:07 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 25 Sep 2020 09:14:07 +0000 (UTC) Subject: [master] e7d12ed54 Fixup changes.rst heading for 6.5.0 release Message-ID: <20200925091407.2A3D09657E@lists.varnish-cache.org> commit e7d12ed5401fded27775188dff757bd4ef0002ca Author: Martin Blix Grydeland Date: Thu Sep 24 14:08:48 2020 +0200 Fixup changes.rst heading for 6.5.0 release Retroactively fix up the heading for the 6.5.0 release in the changelog. diff --git a/doc/changes.rst b/doc/changes.rst index ffd2009cd..964f8c2d7 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -26,9 +26,9 @@ http://varnish-cache.org/docs/trunk/whats-new/index.html and via individual releases. These documents are updated as part of the release process. -=========================== -NEXT (scheduled 2020-09-15) -=========================== +================================ +Varnish Cache 6.5.0 (2020-09-15) +================================ [ABI] marks potentially breaking changes to binary compatibility. From martin at varnish-software.com Fri Sep 25 09:14:07 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 25 Sep 2020 09:14:07 +0000 (UTC) Subject: [master] 9595e7c7a Add the VRT major bump to changes.rst Message-ID: <20200925091407.47D2496583@lists.varnish-cache.org> commit 9595e7c7a909e28fe6030f4a86bff13f0d4c29ea Author: Martin Blix Grydeland Date: Thu Sep 24 14:12:29 2020 +0200 Add the VRT major bump to changes.rst This starts a 6.5.1 changes entry, with only the major bump present. diff --git a/doc/changes.rst b/doc/changes.rst index 964f8c2d7..3b9a91d50 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -26,6 +26,14 @@ http://varnish-cache.org/docs/trunk/whats-new/index.html and via individual releases. These documents are updated as part of the release process. +================================ +Varnish Cache 6.5.1 (2020-09-25) +================================ + +* Bump the VRT_MAJOR_VERSION from 11 to 12, to reflect the API changes + that went into the 6.5.0 release. This step was forgotten for that + release. + ================================ Varnish Cache 6.5.0 (2020-09-15) ================================ From martin at varnish-software.com Fri Sep 25 09:14:07 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 25 Sep 2020 09:14:07 +0000 (UTC) Subject: [master] ad6afbfa4 Add two packaging related changes to upgrading-6.5.rst Message-ID: <20200925091407.662E596587@lists.varnish-cache.org> commit ad6afbfa482278c0e24a23e6fabc4de3f4f30ce1 Author: Martin Blix Grydeland Date: Thu Sep 24 14:22:00 2020 +0200 Add two packaging related changes to upgrading-6.5.rst diff --git a/doc/sphinx/whats-new/upgrading-6.5.rst b/doc/sphinx/whats-new/upgrading-6.5.rst index 6a1771036..8d4d15951 100644 --- a/doc/sphinx/whats-new/upgrading-6.5.rst +++ b/doc/sphinx/whats-new/upgrading-6.5.rst @@ -92,4 +92,18 @@ needs some counters to be present the ``'R'`` argument takes a glob of required fields. Such counters are not affected by filtering from other ``VSC_Arg()`` arguments. +Official Packages related changes +================================= + +* The default systemd `varnish.service` unit file now sets `varnishd` to + listen for PROXY protocol connections on port 8443. This corresponds + with the Hitch default configuration, making it easier to set up Varnish + using TLS. + +* The default systemd `varnish.service` unit file now enables the HTTP/2 + feature of `varnishd`. This corresponds with the default ALPN token + advertisement in the Hitch default configuration, making it easier to + enable HTTP/2 in Varnish setups. + + *eof* From martin at varnish-software.com Fri Sep 25 09:14:07 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 25 Sep 2020 09:14:07 +0000 (UTC) Subject: [master] 1dae23376 Prepare for 6.5.1 Message-ID: <20200925091407.888419658B@lists.varnish-cache.org> commit 1dae23376bb5ea7a6b8e9e4b9ed95cdc9469fb64 Author: Martin Blix Grydeland Date: Thu Sep 24 15:43:03 2020 +0200 Prepare for 6.5.1 diff --git a/configure.ac b/configure.ac index 70aa3aed7..419e63b0a 100644 --- a/configure.ac +++ b/configure.ac @@ -2,7 +2,7 @@ AC_PREREQ(2.59) AC_COPYRIGHT([Copyright (c) 2006 Verdens Gang AS Copyright (c) 2006-2020 Varnish Software]) AC_REVISION([$Id$]) -AC_INIT([Varnish], [6.5.0], [varnish-dev at varnish-cache.org]) +AC_INIT([Varnish], [6.5.1], [varnish-dev at varnish-cache.org]) AC_CONFIG_SRCDIR(include/miniobj.h) AC_CONFIG_HEADERS([config.h]) AC_CONFIG_MACRO_DIR([m4]) From martin at varnish-software.com Fri Sep 25 09:31:02 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 25 Sep 2020 09:31:02 +0000 (UTC) Subject: [6.5] f3614262f Bump the VRT major version Message-ID: <20200925093102.857AD9737D@lists.varnish-cache.org> commit f3614262f130aedaae4f8ed975b3801607ddce5d Author: Martin Blix Grydeland Date: Wed Sep 23 11:46:09 2020 +0200 Bump the VRT major version In the Varnish Cache 6.5.0 release we forgot to bump the VRT major version. Do it now. diff --git a/include/vrt.h b/include/vrt.h index c4858cd8b..5b16c6261 100644 --- a/include/vrt.h +++ b/include/vrt.h @@ -166,7 +166,7 @@ * vrt_acl type added */ -#define VRT_MAJOR_VERSION 11U +#define VRT_MAJOR_VERSION 12U #define VRT_MINOR_VERSION 0U From martin at varnish-software.com Fri Sep 25 09:31:02 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 25 Sep 2020 09:31:02 +0000 (UTC) Subject: [6.5] e7d12ed54 Fixup changes.rst heading for 6.5.0 release Message-ID: <20200925093102.9B33997380@lists.varnish-cache.org> commit e7d12ed5401fded27775188dff757bd4ef0002ca Author: Martin Blix Grydeland Date: Thu Sep 24 14:08:48 2020 +0200 Fixup changes.rst heading for 6.5.0 release Retroactively fix up the heading for the 6.5.0 release in the changelog. diff --git a/doc/changes.rst b/doc/changes.rst index ffd2009cd..964f8c2d7 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -26,9 +26,9 @@ http://varnish-cache.org/docs/trunk/whats-new/index.html and via individual releases. These documents are updated as part of the release process. -=========================== -NEXT (scheduled 2020-09-15) -=========================== +================================ +Varnish Cache 6.5.0 (2020-09-15) +================================ [ABI] marks potentially breaking changes to binary compatibility. From martin at varnish-software.com Fri Sep 25 09:31:02 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 25 Sep 2020 09:31:02 +0000 (UTC) Subject: [6.5] 9595e7c7a Add the VRT major bump to changes.rst Message-ID: <20200925093102.B7FC697383@lists.varnish-cache.org> commit 9595e7c7a909e28fe6030f4a86bff13f0d4c29ea Author: Martin Blix Grydeland Date: Thu Sep 24 14:12:29 2020 +0200 Add the VRT major bump to changes.rst This starts a 6.5.1 changes entry, with only the major bump present. diff --git a/doc/changes.rst b/doc/changes.rst index 964f8c2d7..3b9a91d50 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -26,6 +26,14 @@ http://varnish-cache.org/docs/trunk/whats-new/index.html and via individual releases. These documents are updated as part of the release process. +================================ +Varnish Cache 6.5.1 (2020-09-25) +================================ + +* Bump the VRT_MAJOR_VERSION from 11 to 12, to reflect the API changes + that went into the 6.5.0 release. This step was forgotten for that + release. + ================================ Varnish Cache 6.5.0 (2020-09-15) ================================ From martin at varnish-software.com Fri Sep 25 09:31:02 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 25 Sep 2020 09:31:02 +0000 (UTC) Subject: [6.5] ad6afbfa4 Add two packaging related changes to upgrading-6.5.rst Message-ID: <20200925093102.E32A89738A@lists.varnish-cache.org> commit ad6afbfa482278c0e24a23e6fabc4de3f4f30ce1 Author: Martin Blix Grydeland Date: Thu Sep 24 14:22:00 2020 +0200 Add two packaging related changes to upgrading-6.5.rst diff --git a/doc/sphinx/whats-new/upgrading-6.5.rst b/doc/sphinx/whats-new/upgrading-6.5.rst index 6a1771036..8d4d15951 100644 --- a/doc/sphinx/whats-new/upgrading-6.5.rst +++ b/doc/sphinx/whats-new/upgrading-6.5.rst @@ -92,4 +92,18 @@ needs some counters to be present the ``'R'`` argument takes a glob of required fields. Such counters are not affected by filtering from other ``VSC_Arg()`` arguments. +Official Packages related changes +================================= + +* The default systemd `varnish.service` unit file now sets `varnishd` to + listen for PROXY protocol connections on port 8443. This corresponds + with the Hitch default configuration, making it easier to set up Varnish + using TLS. + +* The default systemd `varnish.service` unit file now enables the HTTP/2 + feature of `varnishd`. This corresponds with the default ALPN token + advertisement in the Hitch default configuration, making it easier to + enable HTTP/2 in Varnish setups. + + *eof* From martin at varnish-software.com Fri Sep 25 09:31:03 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 25 Sep 2020 09:31:03 +0000 (UTC) Subject: [6.5] 1dae23376 Prepare for 6.5.1 Message-ID: <20200925093103.0C9BB9738E@lists.varnish-cache.org> commit 1dae23376bb5ea7a6b8e9e4b9ed95cdc9469fb64 Author: Martin Blix Grydeland Date: Thu Sep 24 15:43:03 2020 +0200 Prepare for 6.5.1 diff --git a/configure.ac b/configure.ac index 70aa3aed7..419e63b0a 100644 --- a/configure.ac +++ b/configure.ac @@ -2,7 +2,7 @@ AC_PREREQ(2.59) AC_COPYRIGHT([Copyright (c) 2006 Verdens Gang AS Copyright (c) 2006-2020 Varnish Software]) AC_REVISION([$Id$]) -AC_INIT([Varnish], [6.5.0], [varnish-dev at varnish-cache.org]) +AC_INIT([Varnish], [6.5.1], [varnish-dev at varnish-cache.org]) AC_CONFIG_SRCDIR(include/miniobj.h) AC_CONFIG_HEADERS([config.h]) AC_CONFIG_MACRO_DIR([m4]) From martin at varnish-software.com Fri Sep 25 09:37:06 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 25 Sep 2020 09:37:06 +0000 (UTC) Subject: [6.5] c708da07a Make the nightly CCI workflow the default Message-ID: <20200925093706.A0B3197D67@lists.varnish-cache.org> commit c708da07af7a34f8da80eb719e1eb96a29cddc69 Author: Martin Blix Grydeland Date: Wed Sep 23 15:15:25 2020 +0200 Make the nightly CCI workflow the default Upgrade the CCI script to build release packages using the nightly packages build steps. diff --git a/.circleci/config.yml b/.circleci/config.yml index 7b600094a..bbbba717f 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -12,7 +12,7 @@ parameters: default: "HEAD" pkg-commit: type: string - default: "master" + default: "6.5" jobs: dist: description: Builds varnish-x.y.z.tar.gz that is used later for the packaging jobs @@ -252,43 +252,7 @@ jobs: workflows: version: 2 - commit: - jobs: - - dist - - distcheck: - name: distcheck_centos_7 - dist: centos - release: "7" - requires: - - dist - - distcheck: - name: distcheck_centos_8 - dist: centos - release: "8" - requires: - - dist - - distcheck: - name: distcheck_debian_buster - dist: debian - release: buster - extra_conf: --enable-asan --enable-ubsan - requires: - - dist - - distcheck: - name: distcheck_alpine - dist: alpine - release: "latest" - #extra_conf: --without-jemalloc - requires: - - dist - nightly: - triggers: - - schedule: - cron: "0 4 * * *" - filters: - branches: - only: - - master + release: jobs: - dist - tar_pkg_tools From martin at varnish-software.com Fri Sep 25 15:52:03 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Fri, 25 Sep 2020 15:52:03 +0000 (UTC) Subject: [6.5] 34d11938a Make CircleCI script download dist tarball Message-ID: <20200925155203.236DBA6BFC@lists.varnish-cache.org> commit 34d11938a9d195bf87f03c48759d14b2550b7400 Author: Martin Blix Grydeland Date: Fri Sep 25 15:37:27 2020 +0200 Make CircleCI script download dist tarball If given a build parameter called 'dist-url', the build script downloads a tarball from the given URL instead of doing a 'make dist' step. diff --git a/.circleci/config.yml b/.circleci/config.yml index bbbba717f..756233c08 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -13,6 +13,10 @@ parameters: pkg-commit: type: string default: "6.5" + dist-url: + type: string + default: "" + jobs: dist: description: Builds varnish-x.y.z.tar.gz that is used later for the packaging jobs @@ -33,19 +37,36 @@ jobs: make \ pcre-devel \ python3 \ - python-sphinx + python-sphinx \ + curl - checkout - - run: - name: Create the dist tarball - command: | - git checkout << pipeline.parameters.vc-commit >> - # if version is "trunk", it's a weekly tarball, override the version - if grep 'AC_INIT.*trunk.*' ./configure.ac; then - sed -i -e "s/^AC_INIT.*trunk.*/AC_INIT([Varnish], [$(date +%Y%m%d)], [varnish-dev at varnish-cache.org])/" ./configure.ac - touch .is_weekly - fi - ./autogen.des --quiet - make dist -j 16 + - when: + condition: << pipeline.parameters.dist-url >> + steps: + - run: + name: Download the dist tarball + command: | + curl -s << pipeline.parameters.dist-url >> -o varnish-dist.tar.gz + mkdir parse-version-tmp + cd parse-version-tmp + tar xzf ../varnish-dist.tar.gz + VERSION=$(varnish-*/configure --version | awk 'NR == 1 {print $NF}') + cd .. + mv varnish-dist.tar.gz varnish-${VERSION}.tar.gz + - unless: + condition: << pipeline.parameters.dist-url >> + steps: + - run: + name: Create the dist tarball + command: | + git checkout << pipeline.parameters.vc-commit >> + # if version is "trunk", it's a weekly tarball, override the version + if grep 'AC_INIT.*trunk.*' ./configure.ac; then + sed -i -e "s/^AC_INIT.*trunk.*/AC_INIT([Varnish], [$(date +%Y%m%d)], [varnish-dev at varnish-cache.org])/" ./configure.ac + touch .is_weekly + fi + ./autogen.des --quiet + make dist -j 16 - persist_to_workspace: root: . paths: From phk at FreeBSD.org Tue Sep 29 09:43:06 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 29 Sep 2020 09:43:06 +0000 (UTC) Subject: [master] 8ce440a39 Always report bodybytes as #bytes processed by bottom VFP. Message-ID: <20200929094306.69C9594E4B@lists.varnish-cache.org> commit 8ce440a392c8874131ecf42a7fa2566ff28fec92 Author: Poul-Henning Kamp Date: Tue Sep 29 09:42:09 2020 +0000 Always report bodybytes as #bytes processed by bottom VFP. Fixes #3354 diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index b7243235a..065043871 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -101,7 +101,7 @@ vbf_cleanup(struct busyobj *bo) vfc = bo->vfc; CHECK_OBJ_NOTNULL(vfc, VFP_CTX_MAGIC); - VFP_Close(vfc); + bo->acct.beresp_bodybytes += VFP_Close(vfc); bo->filter_list = NULL; if (bo->director_state != DIR_S_NULL) @@ -558,7 +558,6 @@ vbf_stp_fetchbody(struct worker *wrk, struct busyobj *bo) AZ(vfc->failed); vfps = VFP_Suck(vfc, ptr, &l); if (l > 0 && vfps != VFP_ERROR) { - bo->acct.beresp_bodybytes += l; VFP_Extend(vfc, l); if (est >= l) est -= l; diff --git a/bin/varnishd/cache/cache_fetch_proc.c b/bin/varnishd/cache/cache_fetch_proc.c index fdf4b70f1..760cc04a3 100644 --- a/bin/varnishd/cache/cache_fetch_proc.c +++ b/bin/varnishd/cache/cache_fetch_proc.c @@ -115,20 +115,24 @@ VFP_Setup(struct vfp_ctx *vc, struct worker *wrk) } /********************************************************************** + * Returns the number of bytes processed by the lowest VFP in the stack */ -void +size_t VFP_Close(struct vfp_ctx *vc) { struct vfp_entry *vfe, *tmp; + size_t rv = 0; VTAILQ_FOREACH_SAFE(vfe, &vc->vfp, list, tmp) { if (vfe->vfp->fini != NULL) vfe->vfp->fini(vc, vfe); + rv = vfe->bytes_out; VSLb(vc->wrk->vsl, SLT_VfpAcct, "%s %ju %ju", vfe->vfp->name, - (uintmax_t)vfe->calls, (uintmax_t)vfe->bytes_out); + (uintmax_t)vfe->calls, (uintmax_t)rv); VTAILQ_REMOVE(&vc->vfp, vfe, list); } + return (rv); } int diff --git a/bin/varnishd/cache/cache_req_body.c b/bin/varnishd/cache/cache_req_body.c index 966526e22..0d417ab6c 100644 --- a/bin/varnishd/cache/cache_req_body.c +++ b/bin/varnishd/cache/cache_req_body.c @@ -117,7 +117,6 @@ vrb_pull(struct req *req, ssize_t maxsize, objiterate_f *func, void *priv) vfps = VFP_Suck(vfc, ptr, &l); if (l > 0 && vfps != VFP_ERROR) { req_bodybytes += l; - req->acct.req_bodybytes += l; if (yet >= l) yet -= l; if (func != NULL) { @@ -130,7 +129,7 @@ vrb_pull(struct req *req, ssize_t maxsize, objiterate_f *func, void *priv) } } while (vfps == VFP_OK); - VFP_Close(vfc); + req->acct.req_bodybytes += VFP_Close(vfc); VSLb_ts_req(req, "ReqBody", VTIM_real()); if (func != NULL) { HSH_DerefBoc(req->wrk, req->body_oc); diff --git a/bin/varnishd/cache/cache_varnishd.h b/bin/varnishd/cache/cache_varnishd.h index 9a68f4229..16c4aaaaa 100644 --- a/bin/varnishd/cache/cache_varnishd.h +++ b/bin/varnishd/cache/cache_varnishd.h @@ -251,7 +251,7 @@ enum vfp_status VFP_GetStorage(struct vfp_ctx *, ssize_t *sz, uint8_t **ptr); void VFP_Extend(const struct vfp_ctx *, ssize_t sz); void VFP_Setup(struct vfp_ctx *vc, struct worker *wrk); int VFP_Open(struct vfp_ctx *bo); -void VFP_Close(struct vfp_ctx *bo); +size_t VFP_Close(struct vfp_ctx *bo); extern const struct vfp VFP_gunzip; extern const struct vfp VFP_gzip; diff --git a/bin/varnishtest/tests/r3354.vtc b/bin/varnishtest/tests/r3354.vtc new file mode 100644 index 000000000..985c4b7d7 --- /dev/null +++ b/bin/varnishtest/tests/r3354.vtc @@ -0,0 +1,35 @@ +varnishtest "Bodybytes is #bytes in bottom VFP" + +server s1 { + rxreq + txresp -hdr "Foo: bar" -bodylen 1000 + rxreq + txresp -bodylen 500 +} -start + +varnish v1 -vcl+backend { + sub vcl_backend_response { + set beresp.do_gzip = true; + if (beresp.http.foo == "bar") { + set beresp.do_esi = true; + } + } +} -start + +varnish v1 -cliok "param.set vsl_mask +VfpAcct" + +client c1 { + txreq -url /1 + rxresp + expect resp.bodylen == 1000 +} -run + +varnish v1 -expect VBE.vcl1.s1.beresp_bodybytes == 1000 + +client c1 { + txreq -url /2 + rxresp + expect resp.bodylen == 500 +} -run + +varnish v1 -expect VBE.vcl1.s1.beresp_bodybytes == 1500 From phk at FreeBSD.org Tue Sep 29 10:24:06 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 29 Sep 2020 10:24:06 +0000 (UTC) Subject: [master] c14ed8776 Document the #3354 fix, as it may affect accounting precision. Message-ID: <20200929102406.0CB1C96113@lists.varnish-cache.org> commit c14ed8776632f58de5ae38cc9e1bda3741d488ca Author: Poul-Henning Kamp Date: Tue Sep 29 10:22:39 2020 +0000 Document the #3354 fix, as it may affect accounting precision. diff --git a/doc/changes.rst b/doc/changes.rst index 3b9a91d50..5a81f289e 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -26,6 +26,13 @@ http://varnish-cache.org/docs/trunk/whats-new/index.html and via individual releases. These documents are updated as part of the release process. +================================ +Varnish Cache Next (2021-03-15) +================================ + +* counters MAIN.s_req_bodybytes and VBE.*.tools.beresp_bodybytes + are now always the number of bodybytes moved on the wire. + ================================ Varnish Cache 6.5.1 (2020-09-25) ================================ From dridi.boukelmoune at gmail.com Tue Sep 29 12:23:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 29 Sep 2020 12:23:06 +0000 (UTC) Subject: [master] 7e241844b chmod +x lib/libvcc/vsctool.py Message-ID: <20200929122306.691469C5E9@lists.varnish-cache.org> commit 7e241844bd2a38bb22d0e175bcbebd460ba37b89 Author: Dridi Boukelmoune Date: Tue Sep 29 14:21:55 2020 +0200 chmod +x lib/libvcc/vsctool.py diff --git a/lib/libvcc/vsctool.py b/lib/libvcc/vsctool.py old mode 100644 new mode 100755 From phk at FreeBSD.org Tue Sep 29 13:58:06 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 29 Sep 2020 13:58:06 +0000 (UTC) Subject: [master] 7a1971168 Convert the second request to chunked for stronger test. Message-ID: <20200929135806.E9CF4A2047@lists.varnish-cache.org> commit 7a19711689e0576b6b7f6845a0bd9a7c7e6ee75c Author: Poul-Henning Kamp Date: Tue Sep 29 12:51:55 2020 +0000 Convert the second request to chunked for stronger test. diff --git a/bin/varnishtest/tests/r3354.vtc b/bin/varnishtest/tests/r3354.vtc index 985c4b7d7..b27d85af8 100644 --- a/bin/varnishtest/tests/r3354.vtc +++ b/bin/varnishtest/tests/r3354.vtc @@ -4,7 +4,13 @@ server s1 { rxreq txresp -hdr "Foo: bar" -bodylen 1000 rxreq - txresp -bodylen 500 + txresp -nolen -hdr "Transfer-Encoding: chunked" + chunkedlen 100 + chunkedlen 100 + chunkedlen 100 + chunkedlen 100 + chunkedlen 100 + chunkedlen 0 } -start varnish v1 -vcl+backend { From phk at FreeBSD.org Tue Sep 29 13:58:07 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 29 Sep 2020 13:58:07 +0000 (UTC) Subject: [master] 36e2bfe7d Add per-filter accounting to VDP like we have it for VFP. Message-ID: <20200929135807.0F445A204B@lists.varnish-cache.org> commit 36e2bfe7d34fdbf75ccf67a1263f8f7bac6c4788 Author: Poul-Henning Kamp Date: Tue Sep 29 13:53:10 2020 +0000 Add per-filter accounting to VDP like we have it for VFP. The new VSL record is 'VdpAcct' and masked by default. Fix the name of VDP_close() to VDP_Close() since I'm touching all the lines anyway. diff --git a/bin/varnishd/cache/cache_deliver_proc.c b/bin/varnishd/cache/cache_deliver_proc.c index 85f92b4e3..ea186be28 100644 --- a/bin/varnishd/cache/cache_deliver_proc.c +++ b/bin/varnishd/cache/cache_deliver_proc.c @@ -83,6 +83,8 @@ VDP_bytes(struct req *req, enum vdp_action act, const void *ptr, ssize_t len) /* Call the present layer, while pointing to the next layer down */ vdc->nxt = VTAILQ_NEXT(vdpe, list); + vdpe->calls++; + vdpe->bytes_in += len; retval = vdpe->vdp->bytes(req, act, &vdpe->priv, ptr, len); if (retval && (vdc->retval == 0 || retval < vdc->retval)) vdc->retval = retval; /* Latch error value */ @@ -131,16 +133,20 @@ VDP_Push(struct req *req, const struct vdp *vdp, void *priv) return (vdc->retval); } -void -VDP_close(struct req *req) +uint64_t +VDP_Close(struct req *req) { struct vdp_entry *vdpe; struct vdp_ctx *vdc; + uint64_t rv = 0; CHECK_OBJ_NOTNULL(req, REQ_MAGIC); vdc = req->vdc; while (!VTAILQ_EMPTY(&vdc->vdp)) { vdpe = VTAILQ_FIRST(&vdc->vdp); + rv = vdpe->bytes_in; + VSLb(req->vsl, SLT_VdpAcct, "%s %ju %ju", vdpe->vdp->name, + (uintmax_t)vdpe->calls, (uintmax_t)rv); if (vdc->retval >= 0) AN(vdpe); if (vdpe != NULL) { @@ -152,6 +158,7 @@ VDP_close(struct req *req) } vdc->nxt = VTAILQ_FIRST(&vdc->vdp); } + return (rv); } /*--------------------------------------------------------------------*/ diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c index b22c61f60..9bd59b5c7 100644 --- a/bin/varnishd/cache/cache_esi_deliver.c +++ b/bin/varnishd/cache/cache_esi_deliver.c @@ -871,5 +871,5 @@ ved_deliver(struct req *req, struct boc *boc, int wantbody) if (i && req->doclose == SC_NULL) req->doclose = SC_REM_CLOSE; - VDP_close(req); + (void)VDP_Close(req); } diff --git a/bin/varnishd/cache/cache_fetch_proc.c b/bin/varnishd/cache/cache_fetch_proc.c index 760cc04a3..d9fa10716 100644 --- a/bin/varnishd/cache/cache_fetch_proc.c +++ b/bin/varnishd/cache/cache_fetch_proc.c @@ -118,11 +118,11 @@ VFP_Setup(struct vfp_ctx *vc, struct worker *wrk) * Returns the number of bytes processed by the lowest VFP in the stack */ -size_t +uint64_t VFP_Close(struct vfp_ctx *vc) { struct vfp_entry *vfe, *tmp; - size_t rv = 0; + uint64_t rv = 0; VTAILQ_FOREACH_SAFE(vfe, &vc->vfp, list, tmp) { if (vfe->vfp->fini != NULL) @@ -155,7 +155,7 @@ VFP_Open(struct vfp_ctx *vc) if (vfe->closed != VFP_OK && vfe->closed != VFP_NULL) { (void)VFP_Error(vc, "Fetch filter %s failed to open", vfe->vfp->name); - VFP_Close(vc); + (void)VFP_Close(vc); return (-1); } } diff --git a/bin/varnishd/cache/cache_filter.h b/bin/varnishd/cache/cache_filter.h index 9c3605a1b..483147196 100644 --- a/bin/varnishd/cache/cache_filter.h +++ b/bin/varnishd/cache/cache_filter.h @@ -128,6 +128,8 @@ struct vdp_entry { const struct vdp *vdp; void *priv; VTAILQ_ENTRY(vdp_entry) list; + uint64_t calls; + uint64_t bytes_in; }; VTAILQ_HEAD(vdp_entry_s, vdp_entry); diff --git a/bin/varnishd/cache/cache_varnishd.h b/bin/varnishd/cache/cache_varnishd.h index 16c4aaaaa..c365ce057 100644 --- a/bin/varnishd/cache/cache_varnishd.h +++ b/bin/varnishd/cache/cache_varnishd.h @@ -223,7 +223,7 @@ void CLI_Run(void); void CLI_AddFuncs(struct cli_proto *p); /* cache_deliver_proc.c */ -void VDP_close(struct req *req); +uint64_t VDP_Close(struct req *req); int VDP_DeliverObj(struct req *req); extern const struct vdp VDP_gunzip; @@ -251,7 +251,7 @@ enum vfp_status VFP_GetStorage(struct vfp_ctx *, ssize_t *sz, uint8_t **ptr); void VFP_Extend(const struct vfp_ctx *, ssize_t sz); void VFP_Setup(struct vfp_ctx *vc, struct worker *wrk); int VFP_Open(struct vfp_ctx *bo); -size_t VFP_Close(struct vfp_ctx *bo); +uint64_t VFP_Close(struct vfp_ctx *bo); extern const struct vfp VFP_gunzip; extern const struct vfp VFP_gzip; diff --git a/bin/varnishd/http1/cache_http1_deliver.c b/bin/varnishd/http1/cache_http1_deliver.c index f5f2c9579..97c775dbf 100644 --- a/bin/varnishd/http1/cache_http1_deliver.c +++ b/bin/varnishd/http1/cache_http1_deliver.c @@ -169,5 +169,5 @@ V1D_Deliver(struct req *req, struct boc *boc, int sendbody) sc = SC_REM_CLOSE; if (sc != SC_NULL) Req_Fail(req, sc); - VDP_close(req); + (void)VDP_Close(req); } diff --git a/bin/varnishd/http2/cache_http2_deliver.c b/bin/varnishd/http2/cache_http2_deliver.c index c2ef90eec..190356d5d 100644 --- a/bin/varnishd/http2/cache_http2_deliver.c +++ b/bin/varnishd/http2/cache_http2_deliver.c @@ -331,5 +331,5 @@ h2_deliver(struct req *req, struct boc *boc, int sendbody) } AZ(req->wrk->v1l); - VDP_close(req); + (void)VDP_Close(req); } diff --git a/bin/varnishd/mgt/mgt_param_bits.c b/bin/varnishd/mgt/mgt_param_bits.c index 61edab2e4..cc0812624 100644 --- a/bin/varnishd/mgt/mgt_param_bits.c +++ b/bin/varnishd/mgt/mgt_param_bits.c @@ -128,6 +128,7 @@ tweak_vsl_mask(struct vsb *vsb, const struct parspec *par, const char *arg) (void)bit(mgt_param.vsl_mask, SLT_WorkThread, BSET); (void)bit(mgt_param.vsl_mask, SLT_Hash, BSET); (void)bit(mgt_param.vsl_mask, SLT_VfpAcct, BSET); + (void)bit(mgt_param.vsl_mask, SLT_VdpAcct, BSET); (void)bit(mgt_param.vsl_mask, SLT_H2TxBody, BSET); (void)bit(mgt_param.vsl_mask, SLT_H2TxHdr, BSET); (void)bit(mgt_param.vsl_mask, SLT_H2RxBody, BSET); diff --git a/include/tbl/vsl_tags.h b/include/tbl/vsl_tags.h index ceb7ff917..49dd6ec95 100644 --- a/include/tbl/vsl_tags.h +++ b/include/tbl/vsl_tags.h @@ -696,6 +696,19 @@ SLTM(Notice, 0, "Informational messages about request handling", "\n" ) +SLTM(VdpAcct, 0, "Deliver filter accounting", + "Contains name of VDP and statistics.\n\n" + "The format is::\n\n" + "\t%s %d %d\n" + "\t| | |\n" + "\t| | +- Total bytes produced\n" + "\t| +---- Number of calls made\n" + "\t+------- Name of filter\n" + "\n" + NODEF_NOTICE +) + + #undef NOSUP_NOTICE #undef NODEF_NOTICE #undef SLTM From phk at FreeBSD.org Tue Sep 29 13:58:07 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 29 Sep 2020 13:58:07 +0000 (UTC) Subject: [master] f82d3b5a3 Don't launch varnishncsa to test VSL records, we have logexpect. Message-ID: <20200929135807.2AF14A2051@lists.varnish-cache.org> commit f82d3b5a3fcb5f773782632f075d031be1b8d0fc Author: Poul-Henning Kamp Date: Tue Sep 29 13:57:13 2020 +0000 Don't launch varnishncsa to test VSL records, we have logexpect. diff --git a/bin/varnishtest/tests/t02015.vtc b/bin/varnishtest/tests/t02015.vtc index dc9431b3e..4dc6dec63 100644 --- a/bin/varnishtest/tests/t02015.vtc +++ b/bin/varnishtest/tests/t02015.vtc @@ -8,6 +8,11 @@ server s1 { varnish v1 -cliok "param.set feature +http2" varnish v1 -vcl+backend "" -start +logexpect l1 -v v1 -g request { + expect * 1001 ReqAcct "46 0 46 97 12345 12442" + expect * 1003 ReqAcct "46 0 46 102 1000 1102" +} -start + client c1 { txpri @@ -36,14 +41,4 @@ client c1 { } -run } -run -varnish v1 -vsl_catchup - -shell -match "^12345$" { - varnishncsa -n ${v1_name} -F '%{VSL:ReqAcct[5]}x' -d \ - -q 'ReqHeader:stream == 1' -} - -shell -match "^1000$" { - varnishncsa -n ${v1_name} -F '%{VSL:ReqAcct[5]}x' -d \ - -q 'ReqHeader:stream == 3' -} +logexpect l1 -wait From dridi.boukelmoune at gmail.com Tue Sep 29 14:46:05 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 29 Sep 2020 14:46:05 +0000 (UTC) Subject: [master] 7d90528fc travis: Run verbose witness jobs Message-ID: <20200929144605.EF3CDA4C95@lists.varnish-cache.org> commit 7d90528fc45cf8d3dacf0f10cae79c42fd3ce06a Author: Dridi Boukelmoune Date: Tue Sep 29 16:44:50 2020 +0200 travis: Run verbose witness jobs diff --git a/.travis.yml b/.travis.yml index cf12dcde0..7175105fd 100644 --- a/.travis.yml +++ b/.travis.yml @@ -47,7 +47,7 @@ jobs: - ./configure --enable-maintainer-mode - <<: *test-linux env: WITNESS=1 - script: make -j8 witness + script: make -j8 witness VERBOSE=1 - <<: *test-linux stage: sanitizers addons: