[master] c3729d022 doc: changes.rst since varnish-6.5.1
Nils Goroll
nils.goroll at uplex.de
Sat Mar 13 19:59:07 UTC 2021
commit c3729d022ec9d057f395af88b8a2c7245bd5fcff
Author: Nils Goroll <nils.goroll at uplex.de>
Date: Sat Mar 13 20:56:57 2021 +0100
doc: changes.rst since varnish-6.5.1
The order matches the order of commits, except where I thought it made
more sense to join topics into contiguous sections.
diff --git a/doc/changes.rst b/doc/changes.rst
index 054389186..c7b0a5514 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -35,14 +35,70 @@ release process.
Varnish Cache Next (2021-03-15)
================================
-* counters MAIN.s_req_bodybytes and VBE.*.tools.beresp_bodybytes
- are now always the number of bodybytes moved on the wire.
+* Body bytes accounting has been fixed to always represent the number
+ of bodybytes moved on the wire, exclusive of protocol-specific
+ overhead like HTTP/1 chunked encoding or HTTP/2 framing.
- .. 36e2bfe7d34fdbf75ccf67a1263f8f7bac6c4788
+ This change affects counters like
+ - ``MAIN.s_req_bodybytes``,
+ - ``MAIN.s_resp_bodybytes``,
+ - ``VBE.*.*.bereq_bodybytes`` and
+ - ``VBE.*.*.beresp_bodybytes``
+
+ as well as the VSL records
+ - ``ReqAcct``,
+ - ``PipeAcct`` and
+ - ``BereqAcct``.
+
+* ``VdpAcct`` log records have been added to output delivery filter
+ (VDP) accounting details analogous to the existing ``VfpAcct``. Both
+ tags are masked by default.
+
+* Many filter (VDP/VFP) related signatures have been changed:
+ - ``vdp_init_f``
+ - ``vdp_fini_f``
+ - ``vdp_bytes_f``
+ - ``VDP_bytes()``
+
+ as well as ``struct vdp_entry`` and ``struct vdp_ctx``
+
+ ``VFP_Push()`` and ``VDP_Push()`` are no longer intended for VMOD
+ use and have been removed from the API.
+
+* The VDP code is now more strict about ``VDP_END``, which must be
+ sent down the filter chain at most once.
+
+* Core code has been changed to ensure for most cases that ``VDP_END``
+ gets signaled with the object's last bytes, rather than with an
+ extra zero-data call.
+
+* Reason phrases for more HTTP Status codes have been added to core
+ code.
+
+* Connection pooling behavior has been improved with respect to
+ ``Connection: close`` (3400_, 3405_).
+
+* Handling of the ``Keep-Alive`` HTTP header as hop-by-hop has been
+ fixed (3417_).
+
+* Handling of hop-by-hop headers has been fixed for HTTP/2 (3416_).
+
+* The stevedore API has been changed:
+ - ``OBJ_ITER_FINAL`` has been renamed to ``OBJ_ITER_END``
+ - ``ObjExtend()`` signature has been changed to also cover the
+ ``ObjTrimStore()`` use case and
+ - ``ObjTrimStore()`` has been removed.
+
+* The ``verrno.h`` header file has been removed and merged into
+ ``vas.h``
+
+* The connection close reason has been fixed to properly report
+ ``SC_RESP_CLOSE`` / ``resp_close`` where previously only
+ ``SC_REQ_CLOSE`` / ``req_close`` was reported.
* Unless the new ``validate_headers`` feature is disabled, all newly
set headers are now validated to contain only characters allowed by
- RFC7230. A (runtime) VCL failure is triggered if not.
+ RFC7230. A (runtime) VCL failure is triggered if not (3407_).
* ``VRT_ValidHdr()`` has been added for vmods to conduct the same
check as the ``validate_headers`` feature, for example when headers
@@ -63,6 +119,259 @@ Varnish Cache Next (2021-03-15)
* All shard ``Error`` and ``Notice`` messages now use the unified
prefix ``vmod_directors: shard %s``.
+* In the shard director, use of parameter sets with ``resolve=NOW``
+ has been fixed.
+
+* Performance of log-processing tools like ``varnishlog`` has been
+ improved by using ``mmap()`` if possible when reading from log
+ files.
+
+* An assertion failure has been fixed which could be triggered when a
+ request body was used with restarts (3433_, 3434_).
+
+* A signal handling bug in the Varnish Utility API (VUT) has been
+ fixed which caused log-processing utilities to perform poorly after
+ a signal had been received (3436_).
+
+* The ``client.identity`` variable is now accessible on the backend
+ side.
+
+* Client and backend finite state machine internals (``enum req_step``
+ and ``enum fetch_step``) have been removed from ``cache.h``.
+
+* Three new ``Timestamp`` VSL records have been added to backend
+ request processing:
+
+ - The ``Process`` timestamp after ``return(deliver)`` or
+ ``return(pass(x))`` from ``vcl_backend_response``,
+
+ - the ``Fetch`` timestamp before a backend connection is requested
+ and
+
+ - the ``Connected`` timestamp when a connection to a regular backend
+ (VBE) is established.
+
+* The VRT backend interface has been changed:
+
+ - ``struct vrt_endpoint`` has been added describing a UDS or TCP
+ endpoint for a backend to connect to.
+
+ Endpoints also support a preamble to be sent with every new
+ connection.
+
+ - This structure needs to be passed via the ``endpoint`` member of
+ ``struct vrt_backend`` when creating backends with
+ ``VRT_new_backend()`` or ``VRT_new_backend_clustered()``.
+
+* ``VRT_Endpoint_Clone()`` has been added to facilitate working with
+ endpoints.
+
+* The variables ``bereq.is_hitpass`` and ``bereq.is_hitmiss`` have
+ been added to the backend side matching ``req.is_hitpass`` and
+ ``req.is_hitmiss`` on the client side.
+
+* The ``set_ip_tos()`` function from the bundled ``std`` vmod now sets
+ the IPv6 Taffic Class (TCLASS) when used on an IPv6 connection.
+
+* A bug has been fixed which could lead to varnish failing to start
+ after updates due to outdated content of the ``vmod_cache``
+ directory (3243_).
+
+* An issue has been addressed where using VCL with a high number of
+ literal strings could lead to prolonged c-compiler runtimes since
+ Varnish-Cache 6.3 (3392_).
+
+* The ``MAIN.esi_req`` counter has been added as a statistic of the
+ number of ESI sub requests created.
+
+* The ``vcl.discard`` CLI command can now be used to discard more than
+ one VCL with a single command, which succeeds only if all given VCLs
+ could be discarded (atomic behavior).
+
+* The ``vcl.discard`` CLI command now supports glob patterns for vcl names.
+
+* The ``vcl.deps`` CLI command has been added to output dependencies
+ between VCLs (because of labels and ``return(vcl)`` statements).
+
+* The ``FetchError`` log message ``Timed out reusing backend
+ connection`` has been renamed to ``first byte timeout (reused
+ connection)`` to clarify that it is emit for effectively the same
+ reason as ``first byte timeout``.
+
+* Long strings in VCL can now also be denoted using ``""" ... """`` in
+ addition to the existing ``{" ... "}``.
+
+* The ``pdiff()`` function declaration has been moved from ``cache.h``
+ to ``vas.h``.
+
+* The interface for private pointers in VMODs has been changed:
+
+ - The ``free`` pointer in ``struct vmod_priv`` has been replaced
+ with a pointer to ``struct vmod_priv_methods``, to where the
+ pointer to the former free callback has been moved as the ``fini``
+ member.
+
+ - The former free callback type has been renamed from
+ ``vmod_priv_free_f`` to ``vmod_priv_fini_f`` and as gained a
+ ``VRT_CTX`` argument
+
+* The ``MAIN.s_bgfetch`` counter has been added as a statistic on the
+ number of background fetches issues.
+
+* Various improvements have been made to the ``varnishtest`` facility:
+
+ - the ``loop`` keyword now works everywhere
+
+ - HTTP/2 logging has been improved
+
+ - Default HTTP/2 parameters have been tweaked (3442_)
+
+ - Varnish listen address information is now available by default in
+ the macros ``${vNAME_addr}``, ``${vNAME_port}`` and
+ ``${vNAME_sock}``. Macros by the names ``${vNAME_SOCKET_*}``
+ contain the address information for each listen socket as created
+ with the ``-a`` argument to ``varnishd``.
+
+ - Synchronization points for counters (VSCs) have been added as
+ ``varnish vNAME -expect PATTERN OP PATTERN``
+
+ - varnishtest now also works with IPv6 setups
+
+ - ``feature ipv4`` and ``feature ipv6`` can be used to control
+ execution of test cases which require one or the other protocol.
+
+ - haproxy arguments can now be externally provided through the
+ ``HAPROXY_ARGS`` variable.
+
+ - logexpect now supports alternatives with the ``expect ? ...`` syntax
+ and negative matches with the ``fail add ...`` and ``fail clear``
+ syntax.
+
+ - The overall logexpect match expectation can now be inverted using
+ the ``-err`` argument.
+
+ - Numeric comparisons for HTTP headers have been added: ``-lt``,
+ ``-le``, ``-eq``, ``-ne``, ``-ge``, ``-gt``
+
+ - ``rxdata -some`` has been fixed.
+
+* The ``ban_cutoff`` parameter now refers to the overall length of the
+ ban list, including completed bans, where before only non-completed
+ ("active") bans were counted towards ``ban_cutoff``.
+
+* A race in the round-robin director has been fixed which could lead
+ to backend requests failing when backends in the director were sick
+ (3473_).
+
+* A race in the probe management has been fixed which could lead to a
+ panic when VCLs changed temperature in general and when
+ ``vcl.discard`` was used in particular (3362_).
+
+* A bug has been fixed which lead to counters (VSCs) of backends from
+ cold VCLs being presented (3358_).
+
+* A bug in ``varnishncsa`` has been fixed which could lead to it
+ crashing when header fields were referenced which did not exist in
+ the processed logs (3485_).
+
+* For failing PROXY connections, ``SessClose`` now provides more
+ detailed information on the cause of the failure.
+
+* The ``std.ban()`` and ``std.ban_error()`` functions have been added
+ to the ``std`` vmod, allowing VCL to check for ban errors.
+
+* Use of the ``ban()`` built-in VCL command is now deprecated.
+
+* The source tree has been reorganized with all vmods now moved to a
+ single ``vmod`` directory.
+
+* ``vmodtool.py`` has been improved to simplify Makefiles when many
+ VMODs are built in a single directory.
+
+* The ``VSA_getsockname()`` and ``VSA_getpeername()`` functions have
+ been added to get address information of file descriptors.
+
+* ``varnishd`` now supports the ``-b None`` argument to start with
+ only the builtin VCL and no backend at all (3067_).
+
+* Some corner cases of IPv6 support in ``varnishd`` have been fixed.
+
+* ``vcl_pipe {}``: ``return(synth)`` and vmod private state support
+ have been fixed. Trying to use ``std.rollback()`` from ``vcl_pipe``
+ now results in VCL failure (3329_, 3330_, 3385_).
+
+* The ``bereq.xid`` variable is now also available in ``vcl_pipe {}``
+
+* The ``VRT_priv_task_get()`` and ``VRT_priv_top_get()`` functions
+ have been added to VRT to allow vmods to retrieve existing
+ ``PRIV_TASK`` / ``PRIV_TOP`` private pointers without creating any.
+
+* ``varnishstat`` now avoids display errors of gauges which previously
+ could underflow to negative values, being displayed as extremely
+ high positive values.
+
+ The ``-r`` option and the ``r`` key binding have been added to
+ return to the previous behavior. When raw mode is active in
+ ``varnishstat`` interactive (curses) mode, the word ``RAW`` is
+ displayed at the right hand side in the lower status line.
+
+* The ``VSC_IsRaw()`` function has been added to ``libvarnishapi`` to
+ query if a gauge is being returned raw or adjusted.
+
+* The ``busy_stats_rate`` feature flag has been added to ensure
+ statistics updates (as configured using the ``thread_stats_rate``
+ parameter) even on a fully loaded system, which would otherwise
+ delay statistics updates in order to reduce lock contention.
+
+* ``ExpKill`` log (VSL) records are now masked by default. See the
+ ``vsl_mask`` parameter.
+
+* A bug has been fixed which could lead to panics when ESI was used
+ with ESI-aware VMODs were used because ``PRIV_TOP`` vmod private
+ state was created on a wrong workspace (3496_).
+
+* The ``VCL_REGEX`` data type is now supported for VMODs, allowing
+ them to use regular expression literals checked and compiled by the
+ VCL compiler infrastructure.
+
+ Consequently, the ``VRT_re_init()`` and ``VRT_re_fini()`` functions
+ have been removed, because they are not required and their use was
+ probably wrong anyway.
+
+* The ``%{X}T`` format has been added to ``varnishncsa``, which
+ generalizes ``%D`` and ``%T``, but also support milliseconds
+ (``ms``) output.
+
+* Error handling has been fixed when vmod functions/methods with
+ ``PRIV_TASK`` arguments were wrongly called from the backend side
+ (3498_).
+
+* The ``varnishncsa`` ``-E`` argument to show ESI requests has been
+ changed to imply ``-c`` (client mode).
+
+* Error handling and performance of the VSL (shared log) client code
+ in ``libvarnishapi`` have been improved (3501_).
+
+* ``varnishlog`` now supports the ``-u`` option to write to a file
+ specified with ``-w`` unbuffered.
+
+* Comparisons of numbers in VSL queries have been improved to match
+ better the behavior which is likely expected by users who have not
+ read the documentation in all detail (3463_).
+
+* A bug in the ESI code has been fixed which could trigger a panic
+ when no storage space was available (3502_).
+
+* The ``resp.proto`` variable is now read-only as it should have been
+ for long.
+
+* ``VTCP_open()`` has been fixed to try all possible addresses from
+ the resolver before giving up (3509_). This bug could cause
+ confusing error messages (3510_).
+
+* ``VRT_synth_blob()`` and ``VRT_synth_strands()`` have been
+ added. The latter should now be used instead of ``VRT_synth_page()``.
+
* The ``VCL_SUB`` data type is now supported for VMODs to save
references to subroutines to be called later using
``VRT_call()``. Calls from a wrong context (e.g. calling a
@@ -73,6 +382,9 @@ Varnish Cache Next (2021-03-15)
.. _VMOD - Varnish Modules: https://varnish-cache.org/docs/trunk/reference/vmod.html
+ VMOD functions can also return the ``VCL_SUB`` data type for calls
+ from VCL as in ``call vmod.returning_sub();``.
+
* ``VRT_check_call()`` can be used to check if a ``VRT_call()`` would
succeed in order to avoid the potential VCL failure in case it would
not.
@@ -80,6 +392,49 @@ Varnish Cache Next (2021-03-15)
It returns ``NULL`` if ``VRT_call()`` would make the call or an
error string why not.
+* ``VRT_handled()`` has been added, which is now to be used instead of
+ access to the ``handling`` member of ``VRT_CTX``.
+
+* The session close reason logging/statistics for HTTP/2 connections
+ have been improved (3393_)
+
+* ``varnishadm`` now has the ``-p`` option to disable readline support
+ for use in scripts and as a generic CLI connector.
+
+* A log (VSL) ``Notice`` record is now emitted whenever more than
+ ``vary_notice`` variants are encountered in the cache for a specific
+ hash. The new ``vary_notice`` parameter defaults to 10.
+
+* The modulus operator ``%`` has been added to VCL.
+
+* ``return(retry)`` from ``vcl_backend_error {}`` now correctly resets
+ ``beresp.status`` and ``beresp.reason`` (3525_).
+
+* Handling of the ``gunzip`` filter with ESI has been fixed (3529_).
+
+* A bug where the ``threads_limited`` counter could be increased
+ without reason has been fixed (3531_).
+
+* All varnish tools using the VUT library utilities for argument
+ processing now support the ``--optstring`` argument to return a
+ string suitable for use with ``getopts`` from shell scripts.
+
+* An issue with high CPU consumption when the maximum number of
+ threads was reached has been fixed (2942_, 3531_)
+
+* HTTP/2 streams are now reset for filter chain (VDP) errors.
+
+* The task priority of incoming connections has been fixed.
+
+* An issue has been addressed where the watchdog facility could
+ misfire when tasks are queued.
+
+* The builtin VCL has been reworked: VCL code has been split into
+ small subroutines, which custom VCL can prepend custom code to.
+
+ This allows for better integration of custom VCL and the built-in
+ VCL and better reuse.
+
================================
Varnish Cache 6.5.1 (2020-09-25)
================================
More information about the varnish-commit
mailing list