[master] 00da70680 whats-new: changes
Nils Goroll
nils.goroll at uplex.de
Sun Mar 14 17:03:04 UTC 2021
commit 00da70680f08eee5186f622764ee1cf7cd8d6542
Author: Nils Goroll <nils.goroll at uplex.de>
Date: Sun Mar 14 18:02:11 2021 +0100
whats-new: changes
diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst
index d0e90aed8..af4edb196 100644
--- a/doc/sphinx/whats-new/changes-trunk.rst
+++ b/doc/sphinx/whats-new/changes-trunk.rst
@@ -25,54 +25,468 @@ merged, may be found in the `change log`_.
varnishd
========
+Arguments
+~~~~~~~~~
+
+* ``varnishd`` now supports the ``-b None`` argument to start with
+ only the builtin VCL and no backend at all.
+
Parameters
~~~~~~~~~~
-**XXX changes in -p parameters**
+* The ``validate_headers`` parameter has been added to control
+ `header validation <whatsnew_changes_CURRENT_header_validation_>`_.
+
+* The ``ban_cutoff`` parameter now refers to the overall length of the
+ ban list, including completed bans, where before only non-completed
+ ("active") bans were counted towards ``ban_cutoff``.
+
+* The ``vary_notice`` parameter has been added to control the
+ threshold for the new `Vary Notice
+ <whatsnew_changes_CURRENT_vary_notice_>`_.
+
+``feature`` Flags
+~~~~~~~~~~~~~~~~~
+
+* The ``busy_stats_rate`` feature flag has been added to ensure
+ statistics updates (as configured using the ``thread_stats_rate``
+ parameter) even on a fully loaded system, which would otherwise
+ delay statistics updates in order to reduce lock contention.
+
+.. _whatsnew_changes_CURRENT_accounting:
+
+Accounting
+~~~~~~~~~~
+
+Body bytes accounting has been fixed to always represent the number of
+bodybytes moved on the wire, exclusive of protocol-specific overhead
+like HTTP/1 chunked encoding or HTTP/2 framing.
+
+This change affects counters like
+
+- ``MAIN.s_req_bodybytes``,
+
+- ``MAIN.s_resp_bodybytes``,
+
+- ``VBE.*.*.bereq_bodybytes`` and
+
+- ``VBE.*.*.beresp_bodybytes``
+
+as well as the VSL records
+
+- ``ReqAcct``,
+
+- ``PipeAcct`` and
-Other changes in varnishd
-~~~~~~~~~~~~~~~~~~~~~~~~~
+- ``BereqAcct``.
+
+.. _whatsnew_changes_CURRENT_sc_close:
+
+Session Close Reasons
+~~~~~~~~~~~~~~~~~~~~~
+
+The connection close reason has been fixed to properly report
+``SC_RESP_CLOSE`` / ``resp_close`` where previously only
+``SC_REQ_CLOSE`` / ``req_close`` was reported.
+
+For failing PROXY connections, ``SessClose`` now provides more
+detailed information on the cause of the failure.
+
+The session close reason logging/statistics for HTTP/2 connections
+have been improved.
+
+.. _whatsnew_changes_CURRENT_vary_notice:
+
+Vary Notice
+~~~~~~~~~~~
+
+A log (VSL) ``Notice`` record is now emitted whenever more than
+``vary_notice`` variants are encountered in the cache for a specific
+hash. The new ``vary_notice`` parameter defaults to 10.
Changes to VCL
==============
+.. _whatsnew_changes_CURRENT_header_validation:
+
+Header Validation
+~~~~~~~~~~~~~~~~~
+
+Unless the new ``validate_headers`` feature is disabled, all newly set
+headers are now validated to contain only characters allowed by
+RFC7230. A (runtime) VCL failure is triggered if not.
+
VCL variables
~~~~~~~~~~~~~
-**XXX new, deprecated or removed variables, or changed semantics**
+* The ``client.identity`` variable is now accessible on the backend
+ side.
+
+* The variables ``bereq.is_hitpass`` and ``bereq.is_hitmiss`` have
+ been added to the backend side matching ``req.is_hitpass`` and
+ ``req.is_hitmiss`` on the client side.
+
+* The ``bereq.xid`` variable is now also available in ``vcl_pipe {}``
+
+* The ``resp.proto`` variable is now read-only as it should have been
+ for long.
Other changes to VCL
~~~~~~~~~~~~~~~~~~~~
+* Long strings in VCL can now also be denoted using ``""" ... """`` in
+ addition to the existing ``{" ... "}``.
+
+* The ``ban()`` builtin is now deprecated and should be replaced with
+ `std.ban() <whatsnew_changes_CURRENT_ban_>`_.
+
+* Trying to use ``std.rollback()`` from ``vcl_pipe`` now results in
+ VCL failure.
+
+* The modulus operator ``%`` has been added to VCL.
+
+* ``return(retry)`` from ``vcl_backend_error {}`` now correctly resets
+ ``beresp.status`` and ``beresp.reason``.
+
+* The builtin VCL has been reworked: VCL code has been split into
+ small subroutines, which custom VCL can prepend custom code to.
+
+ This allows for better integration of custom VCL and the built-in
+ VCL and better reuse.
+
VMODs
=====
-**XXX changes in the bundled VMODs**
+``directors.shard()``
+~~~~~~~~~~~~~~~~~~~~~
+
+* The shard director now supports reconfiguration (adding/removing
+ backends) of several instances without any special ordering
+ requirement.
+
+* Calling the shard director ``.reconfigure()`` method is now
+ optional. If not called explicitly, any shard director backend
+ changes are applied at the end of the current task.
+
+* Shard director ``Error`` log messages with ``(notice)`` have been
+ turned into ``Notice`` log messages.
+
+* All shard ``Error`` and ``Notice`` messages now use the unified
+ prefix ``vmod_directors: shard %s``.
+
+``std.set_ip_tos()``
+~~~~~~~~~~~~~~~~~~~~
+
+The ``set_ip_tos()`` function from the bundled ``std`` vmod now sets
+the IPv6 Taffic Class (TCLASS) when used on an IPv6 connection.
+
+.. _whatsnew_changes_CURRENT_ban:
+
+``std.ban()`` and ``std.ban_error()``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``std.ban()`` and ``std.ban_error()`` functions have been added to
+the ``std`` vmod, allowing VCL to check for ban errors. A typical
+usage pattern with the new interface is::
+
+ if (std.ban(...)) {
+ return(synth(200, "Ban added"));
+ } else {
+ return(synth(400, std.ban_error()));
+ }
+
+``cookie`` functions
+~~~~~~~~~~~~~~~~~~~~
+
+The ``filter_re``, ``keep_re`` and ``get_re`` functions from the
+bundled ``cookie`` vmod have been changed to take the ``VCL_REGEX``
+type. This implies that their regular expression arguments now need to
+be literal, whereas before they could be taken from some other
+variable or function returning ``VCL_STRING``.
+
+Note that these functions never actually handled *dynamic* regexen,
+the string passed with the first call was compiled to a regex, which
+was then used for the lifetime of the respective VCL.
+
varnishlog
==========
-**XXX changes concerning varnishlog(1) and/or vsl(7)**
+* See `Accounting <whatsnew_changes_CURRENT_accounting_>`_ for changes
+ to accounting-related VSL records.
+
+* See `Session Close Reasons <whatsnew_changes_CURRENT_sc_close_>`_
+ for a change affecting ``SessClose``.
+
+* Three new ``Timestamp`` VSL records have been added to backend
+ request processing:
+
+ - The ``Process`` timestamp after ``return(deliver)`` or
+ ``return(pass(x))`` from ``vcl_backend_response``,
+
+ - the ``Fetch`` timestamp before a backend connection is requested
+ and
+
+ - the ``Connected`` timestamp when a connection to a regular backend
+ (VBE) is established.
+
+* The ``FetchError`` log message ``Timed out reusing backend
+ connection`` has been renamed to ``first byte timeout (reused
+ connection)`` to clarify that it is emit for effectively the same
+ reason as ``first byte timeout``.
+
+* ``ExpKill`` log (VSL) records are now masked by default. See the
+ ``vsl_mask`` parameter.
+
+* Comparisons of numbers in VSL queries have been improved to match
+ better the behavior which is likely expected by users who have not
+ read the documentation in all detail.
+
+* See `Vary Notice <whatsnew_changes_CURRENT_vary_notice_>`_ for
+ information on a newly added ``Notice`` log (VSL) record.
+
+varnishncsa
+===========
+
+* The ``%{X}T`` format has been added to ``varnishncsa``, which
+ generalizes ``%D`` and ``%T``, but also support milliseconds
+ (``ms``) output.
+
+* The ``varnishncsa`` ``-E`` argument to show ESI requests has been
+ changed to imply ``-c`` (client mode).
+
varnishadm
==========
-**XXX changes concerning varnishadm(1) and/or varnish-cli(7)**
+* The ``vcl.discard`` CLI command can now be used to discard more than
+ one VCL with a single command, which succeeds only if all given VCLs
+ could be discarded (atomic behavior).
+
+* The ``vcl.discard`` CLI command now supports glob patterns for vcl names.
+
+* The ``vcl.deps`` CLI command has been added to output dependencies
+ between VCLs (because of labels and ``return(vcl)`` statements).
+
+* ``varnishadm`` now has the ``-p`` option to disable readline support
+ for use in scripts and as a generic CLI connector.
varnishstat
===========
-**XXX changes concerning varnishstat(1) and/or varnish-counters(7)**
+* See `Accounting <whatsnew_changes_CURRENT_accounting_>`_ for changes
+ to accounting-related counters.
+
+* See `Session Close Reasons <whatsnew_changes_CURRENT_sc_close_>`_
+ for a change affecting ``MAIN.sc_*`` counters.
+
+* The ``MAIN.esi_req`` counter has been added as a statistic of the
+ number of ESI sub requests created.
+
+* The ``MAIN.s_bgfetch`` counter has been added as a statistic on the
+ number of background fetches issued.
+
+.. _whatsnew_changes_CURRENT_varnishstat_raw:
+
+* ``varnishstat`` now avoids display errors of gauges which previously
+ could underflow to negative values, being displayed as extremely
+ high positive values.
+
+ The ``-r`` option and the ``r`` key binding have been added to
+ return to the previous behavior. When raw mode is active in
+ ``varnishstat`` interactive (curses) mode, the word ``RAW`` is
+ displayed at the right hand side in the lower status line.
varnishtest
===========
-**XXX changes concerning varnishtest(1) and/or vtc(7)**
+Various improvements have been made to the ``varnishtest`` facility:
+
+- the ``loop`` keyword now works everywhere
+
+- HTTP/2 logging has been improved
+
+- Default HTTP/2 parameters have been tweaked
+
+- Varnish listen address information is now available by default in
+ the macros ``${vNAME_addr}``, ``${vNAME_port}`` and
+ ``${vNAME_sock}``. Macros by the names ``${vNAME_SOCKET_*}`` contain
+ the address information for each listen socket as created with the
+ ``-a`` argument to ``varnishd``.
+
+- Synchronization points for counters (VSCs) have been added as
+ ``varnish vNAME -expect PATTERN OP PATTERN``
+
+- varnishtest now also works with IPv6 setups
+
+- ``feature ipqv4`` and ``feature ipv6`` can be used to control
+ execution of test cases which require one or the other protocol.
+
+- haproxy arguments can now be externally provided through the
+ ``HAPROXY_ARGS`` variable.
+
+- logexpect now supports alternatives with the ``expect ? ...`` syntax
+ and negative matches with the ``fail add ...`` and ``fail clear``
+ syntax.
+
+- The overall logexpect match expectation can now be inverted using
+ the ``-err`` argument.
+
+- Numeric comparisons for HTTP headers have been added: ``-lt``,
+ ``-le``, ``-eq``, ``-ne``, ``-ge``, ``-gt``
+
+- ``rxdata -some`` has been fixed.
+
+Other Changes to Varnish Utilities
+==================================
+
+All varnish tools using the VUT library utilities for argument
+processing now support the ``--optstring`` argument to return a string
+suitable for use with ``getopts`` from shell scripts.
+
+
+Developer: Changes for VMOD authors
+===================================
+
+VMOD/VCL interface
+~~~~~~~~~~~~~~~~~~
+
+* The ``VCL_REGEX`` data type is now supported for VMODs, allowing
+ them to use regular expression literals checked and compiled by the
+ VCL compiler infrastructure.
+
+ Consequently, the ``VRT_re_init()`` and ``VRT_re_fini()`` functions
+ have been removed, because they are not required and their use was
+ probably wrong anyway.
+
+* The ``VCL_SUB`` data type is now supported for VMODs to save
+ references to subroutines to be called later using
+ ``VRT_call()``. Calls from a wrong context (e.g. calling a
+ subroutine accessing ``req`` from the backend side) and recursive
+ calls fail the VCL.
+
+ See `VMOD - Varnish Modules`_ in the Reference Manual.
+
+.. _VMOD - Varnish Modules: https://varnish-cache.org/docs/trunk/reference/vmod.html
+
+ VMOD functions can also return the ``VCL_SUB`` data type for calls
+ from VCL as in ``call vmod.returning_sub();``.
+
+* ``VRT_check_call()`` can be used to check if a ``VRT_call()`` would
+ succeed in order to avoid the potential VCL failure in case it would
+ not.
+
+ It returns ``NULL`` if ``VRT_call()`` would make the call or an
+ error string why not.
+
+* ``VRT_handled()`` has been added, which is now to be used instead of
+ access to the ``handling`` member of ``VRT_CTX``.
+
+* ``vmodtool.py`` has been improved to simplify Makefiles when many
+ VMODs are built in a single directory.
-Changes for developers and VMOD authors
-=======================================
+General API
+~~~~~~~~~~~
+
+* ``VRT_ValidHdr()`` has been added for vmods to conduct the same
+ check as the `whatsnew_changes_CURRENT_header_validation`_ feature,
+ for example when headers are set by vmods using the ``cache_http.c``
+ Functions like ``http_ForceHeader()`` from untrusted input.
+
+* Client and backend finite state machine internals (``enum req_step``
+ and ``enum fetch_step``) have been removed from ``cache.h``.
+
+* The ``verrno.h`` header file has been removed and merged into
+ ``vas.h``
+
+* The ``pdiff()`` function declaration has been moved from ``cache.h``
+ to ``vas.h``.
+
+VSA
+~~~
+
+* The ``VSA_getsockname()`` and ``VSA_getpeername()`` functions have
+ been added to get address information of file descriptors.
+
+Private Pointers
+~~~~~~~~~~~~~~~~
+
+* The interface for private pointers in VMODs has been changed:
+
+ - The ``free`` pointer in ``struct vmod_priv`` has been replaced
+ with a pointer to ``struct vmod_priv_methods``, to where the
+ pointer to the former free callback has been moved as the ``fini``
+ member.
+
+ - The former free callback type has been renamed from
+ ``vmod_priv_free_f`` to ``vmod_priv_fini_f`` and as gained a
+ ``VRT_CTX`` argument
+
+* The ``VRT_priv_task_get()`` and ``VRT_priv_top_get()`` functions
+ have been added to VRT to allow vmods to retrieve existing
+ ``PRIV_TASK`` / ``PRIV_TOP`` private pointers without creating any.
+
+Backends
+~~~~~~~~
+
+* The VRT backend interface has been changed:
+
+ - ``struct vrt_endpoint`` has been added describing a UDS or TCP
+ endpoint for a backend to connect to.
+
+ Endpoints also support a preamble to be sent with every new
+ connection.
+
+ - This structure needs to be passed via the ``endpoint`` member of
+ ``struct vrt_backend`` when creating backends with
+ ``VRT_new_backend()`` or ``VRT_new_backend_clustered()``.
+
+* ``VRT_Endpoint_Clone()`` has been added to facilitate working with
+ endpoints.
+
+Filters (VDP/VFP)
+~~~~~~~~~~~~~~~~~
+
+* Many filter (VDP/VFP) related signatures have been changed:
+
+ - ``vdp_init_f``
+
+ - ``vdp_fini_f``
+
+ - ``vdp_bytes_f``
+
+ - ``VDP_bytes()``
+
+ as well as ``struct vdp_entry`` and ``struct vdp_ctx``
+
+ ``VFP_Push()`` and ``VDP_Push()`` are no longer intended for VMOD
+ use and have been removed from the API.
+
+* The VDP code is now more strict about ``VDP_END``, which must be
+ sent down the filter chain at most once. Care should be taken to
+ send ``VDP_END`` together with the last payload bytes whenever
+ possible.
+
+Stevedore API
+~~~~~~~~~~~~~
+
+* The stevedore API has been changed:
+
+ - ``OBJ_ITER_FINAL`` has been renamed to ``OBJ_ITER_END``
+
+ - ``ObjExtend()`` signature has been changed to also cover the
+ ``ObjTrimStore()`` use case and
+
+ - ``ObjTrimStore()`` has been removed.
+
+Developer: Changes for Authors of Varnish Utilities
+===================================================
+
+libvarnishapi
+~~~~~~~~~~~~~
-**XXX changes concerning VRT, the public APIs, source code organization,
-builds etc.**
+* The ``VSC_IsRaw()`` function has been added to ``libvarnishapi`` to
+ query if a gauge is being returned raw or adjusted (see
+ `varnishstat -r option <whatsnew_changes_CURRENT_varnishstat_raw_>`_).
*eof*
More information about the varnish-commit
mailing list