.. Copyright 2021 UPLEX Nils Goroll Systemoptimierung SPDX-License-Identifier: BSD-2-Clause See LICENSE file for full text of license .. _whatsnew_changes_6.6: %%%%%%%%%%%%%%%%%%%%%% Changes in Varnish 6.6 %%%%%%%%%%%%%%%%%%%%%% For information about updating your current Varnish deployment to the new version, see :ref:`whatsnew_upgrading_6.6`. A more detailed and technical account of changes in Varnish, with links to issues that have been fixed and pull requests that have been merged, may be found in the `change log`_. .. _change log: https://github.com/varnishcache/varnish-cache/blob/master/doc/changes.rst varnishd ======== Arguments ~~~~~~~~~ * ``varnishd`` now supports the ``-b none`` argument to start with only the builtin VCL and no backend at all. Parameters ~~~~~~~~~~ * The ``validate_headers`` parameter has been added to control `header validation `_. * The ``ban_cutoff`` parameter now refers to the overall length of the ban list, including completed bans, where before only non-completed ("active") bans were counted towards ``ban_cutoff``. * The ``vary_notice`` parameter has been added to control the threshold for the new `Vary Notice `_. ``feature`` Flags ~~~~~~~~~~~~~~~~~ * The ``busy_stats_rate`` feature flag has been added to ensure statistics updates (as configured using the ``thread_stats_rate`` parameter) even in scenarios where worker threads never run out of tasks and may remain forever busy. .. _whatsnew_changes_6.6_accounting: Accounting ~~~~~~~~~~ Body bytes accounting has been fixed to always represent the number of body bytes moved on the wire, exclusive of protocol-specific overhead like HTTP/1 chunked encoding or HTTP/2 framing. This change affects counters like - ``MAIN.s_req_bodybytes``, - ``MAIN.s_resp_bodybytes``, - ``VBE.*.*.bereq_bodybytes`` and - ``VBE.*.*.beresp_bodybytes`` as well as the VSL records - ``ReqAcct``, - ``PipeAcct`` and - ``BereqAcct``. .. _whatsnew_changes_6.6_sc_close: Session Close Reasons ~~~~~~~~~~~~~~~~~~~~~ The connection close reason has been fixed to properly report ``SC_RESP_CLOSE`` / ``resp_close`` where previously only ``SC_REQ_CLOSE`` / ``req_close`` was reported. For failing PROXY connections, ``SessClose`` now provides more detailed information on the cause of the failure. The session close reason logging/statistics for HTTP/2 connections have been improved. .. _whatsnew_changes_6.6_vary_notice: Vary Notice ~~~~~~~~~~~ A log (VSL) ``Notice`` record is now emitted whenever more than ``vary_notice`` variants are encountered in the cache for a specific hash. The new ``vary_notice`` parameter defaults to 10. Changes to VCL ============== .. _whatsnew_changes_6.6_header_validation: Header Validation ~~~~~~~~~~~~~~~~~ Unless the new ``validate_headers`` feature is disabled, all newly set headers are now validated to contain only characters allowed by RFC7230. A (runtime) VCL failure is triggered if not. VCL variables ~~~~~~~~~~~~~ * The ``client.identity`` variable is now accessible on the backend side. * The variables ``bereq.is_hitpass`` and ``bereq.is_hitmiss`` have been added to the backend side matching ``req.is_hitpass`` and ``req.is_hitmiss`` on the client side. * The ``bereq.xid`` variable is now also available in ``vcl_pipe {}`` * The ``resp.proto`` variable is now read-only as it should have been for long, like the other ``*.proto`` variables. Other changes to VCL ~~~~~~~~~~~~~~~~~~~~ * Long strings in VCL can now also be denoted using ``""" ... """`` in addition to the existing ``{" ... "}``. * The ``ban()`` builtin is now deprecated and should be replaced with `std.ban() `_. * Trying to use ``std.rollback()`` from ``vcl_pipe`` now results in VCL failure. * The modulus operator ``%`` has been added to VCL. * ``return(retry)`` from ``vcl_backend_error {}`` now correctly resets ``beresp.status`` and ``beresp.reason``. * The builtin VCL has been reworked: VCL code has been split into small subroutines, which custom VCL can prepend custom code to. This allows for better integration of custom VCL and the built-in VCL and better reuse. VMODs ===== ``directors.shard()`` ~~~~~~~~~~~~~~~~~~~~~ * The shard director now supports reconfiguration (adding/removing backends) of several instances without any special ordering requirement. * Calling the shard director ``.reconfigure()`` method is now optional. If not called explicitly, any shard director backend changes are applied at the end of the current task. * Shard director ``Error`` log messages with ``(notice)`` have been turned into ``Notice`` log messages. * All shard ``Error`` and ``Notice`` messages now use the unified prefix ``vmod_directors: shard %s``. ``std.set_ip_tos()`` ~~~~~~~~~~~~~~~~~~~~ The ``set_ip_tos()`` function from the bundled ``std`` vmod now sets the IPv6 Taffic Class (TCLASS) when used on an IPv6 connection. .. _whatsnew_changes_6.6_ban: ``std.ban()`` and ``std.ban_error()`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``std.ban()`` and ``std.ban_error()`` functions have been added to the ``std`` vmod, allowing VCL to check for ban errors. A typical usage pattern with the new interface is:: if (std.ban(...)) { return(synth(200, "Ban added")); } else { return(synth(400, std.ban_error())); } .. _whatsnew_changes_6.6_cookie: ``cookie`` functions ~~~~~~~~~~~~~~~~~~~~ The ``filter_re``, ``keep_re`` and ``get_re`` functions from the bundled ``cookie`` vmod have been changed to take the ``VCL_REGEX`` type. This implies that their regular expression arguments now need to be literal, whereas before they could be taken from some other variable or function returning ``VCL_STRING``. Note that these functions never actually handled *dynamic* regexen, the string passed with the first call was compiled to a regex, which was then used for the lifetime of the respective VCL. varnishlog ========== * See `Accounting `_ for changes to accounting-related VSL records. * See `Session Close Reasons `_ for a change affecting ``SessClose``. * Three new ``Timestamp`` VSL records have been added to backend request processing: - The ``Process`` timestamp after ``return(deliver)`` or ``return(pass(x))`` from ``vcl_backend_response``, - the ``Fetch`` timestamp before a backend connection is requested and - the ``Connected`` timestamp when a connection to a regular backend (VBE) is established, or when a recycled connection was selected for reuse. * The ``FetchError`` log message ``Timed out reusing backend connection`` has been renamed to ``first byte timeout (reused connection)`` to clarify that it is emit for effectively the same reason as ``first byte timeout``. * ``ExpKill`` log (VSL) records are now masked by default. See the ``vsl_mask`` parameter. * Comparisons of numbers in VSL queries have been improved to match better the behavior which is likely expected by users who have not read the documentation in all detail. * See `Vary Notice `_ for information on a newly added ``Notice`` log (VSL) record. varnishncsa =========== * The ``%{X}T`` format has been added to ``varnishncsa``, which generalizes ``%D`` and ``%T``, but also support milliseconds (``ms``) output. * The ``varnishncsa`` ``-E`` argument to show ESI requests has been changed to imply ``-c`` (client mode). This behavior is now shared by all log utilities, and ``-c`` no longer includes ESI requests. varnishadm ========== * The ``vcl.discard`` CLI command can now be used to discard more than one VCL with a single command, which succeeds only if all given VCLs could be discarded (atomic behavior). * The ``vcl.discard`` CLI command now supports glob patterns for vcl names. * The ``vcl.deps`` CLI command has been added to output dependencies between VCLs (because of labels and ``return(vcl)`` statements). * ``varnishadm`` now has the ``-p`` option to disable readline support for use in scripts and as a generic CLI connector. varnishstat =========== * See `Accounting `_ for changes to accounting-related counters. * See `Session Close Reasons `_ for a change affecting ``MAIN.sc_*`` counters. * The ``MAIN.esi_req`` counter has been added as a statistic of the number of ESI sub requests created. * The ``MAIN.s_bgfetch`` counter has been added as a statistic on the number of background fetches issued. .. _whatsnew_changes_6.6_varnishstat_raw: * ``varnishstat`` now avoids display errors of gauges which previously could underflow to negative values, being displayed as extremely high positive values. The ``-r`` option and the ``r`` key binding have been added to return to the previous behavior. When raw mode is active in ``varnishstat`` interactive (curses) mode, the word ``RAW`` is displayed at the right hand side in the lower status line. varnishtest =========== Various improvements have been made to the ``varnishtest`` facility: - the ``loop`` keyword now works everywhere - HTTP/2 logging has been improved - Default HTTP/2 parameters have been tweaked - Varnish listen address information is now available by default in the macros ``${vNAME_addr}``, ``${vNAME_port}`` and ``${vNAME_sock}``. Macros by the names ``${vNAME_SOCKET_*}`` contain the address information for each listen socket as created with the ``-a`` argument to ``varnishd``. - Synchronization points for counters (VSCs) have been added as ``varnish vNAME -expect PATTERN OP PATTERN`` - varnishtest now also works with IPv6 setups - ``feature ipv4`` and ``feature ipv6`` can be used to control execution of test cases which require one or the other protocol. - haproxy arguments can now be externally provided through the ``HAPROXY_ARGS`` variable. - logexpect now supports alternatives with the ``expect ? ...`` syntax and negative matches with the ``fail add ...`` and ``fail clear`` syntax. - The overall logexpect match expectation can now be inverted using the ``-err`` argument. - Numeric comparisons for HTTP headers have been added: ``-lt``, ``-le``, ``-eq``, ``-ne``, ``-ge``, ``-gt`` - ``rxdata -some`` has been fixed. Other Changes to Varnish Utilities ================================== All varnish tools using the VUT library utilities for argument processing now support the ``--optstring`` argument to return a string suitable for use with ``getopts`` from shell scripts. .. _whatsnew_changes_6.6_vmod: Developer: Changes for VMOD authors =================================== VMOD/VCL interface ~~~~~~~~~~~~~~~~~~ * The ``VCL_REGEX`` data type is now supported for VMODs, allowing them to use regular expression literals checked and compiled by the VCL compiler infrastructure. Consequently, the ``VRT_re_init()`` and ``VRT_re_fini()`` functions have been removed, because they are not required and their use was probably wrong anyway. * The ``VCL_SUB`` data type is now supported for VMODs to save references to subroutines to be called later using ``VRT_call()``. Calls from a wrong context (e.g. calling a subroutine accessing ``req`` from the backend side) and recursive calls fail the VCL. See `VMOD - Varnish Modules`_ in the Reference Manual. .. _VMOD - Varnish Modules: https://varnish-cache.org/docs/trunk/reference/vmod.html VMOD functions can also return the ``VCL_SUB`` data type for calls from VCL as in ``call vmod.returning_sub();``. * ``VRT_check_call()`` can be used to check if a ``VRT_call()`` would succeed in order to avoid the potential VCL failure in case it would not. It returns ``NULL`` if ``VRT_call()`` would make the call or an error string why not. * ``VRT_handled()`` has been added, which is now to be used instead of access to the ``handling`` member of ``VRT_CTX``. * ``vmodtool.py`` has been improved to simplify Makefiles when many VMODs are built in a single directory. General API ~~~~~~~~~~~ * ``VRT_ValidHdr()`` has been added for VMODs to conduct the same check as the `whatsnew_changes_6.6_header_validation`_ feature, for example when headers are set by VMODs using the ``cache_http.c`` Functions like ``http_ForceHeader()`` from untrusted input. * Client and backend finite state machine internals (``enum req_step`` and ``enum fetch_step``) have been removed from ``cache.h``. * The ``verrno.h`` header file has been removed and merged into ``vas.h`` * The ``pdiff()`` function declaration has been moved from ``cache.h`` to ``vas.h``. VSA ~~~ * The ``VSA_getsockname()`` and ``VSA_getpeername()`` functions have been added to get address information of file descriptors. Private Pointers ~~~~~~~~~~~~~~~~ * The interface for private pointers in VMODs has been changed: - The ``free`` pointer in ``struct vmod_priv`` has been replaced with a pointer to ``struct vmod_priv_methods``, to where the pointer to the former free callback has been moved as the ``fini`` member. - The former free callback type has been renamed from ``vmod_priv_free_f`` to ``vmod_priv_fini_f`` and as gained a ``VRT_CTX`` argument * The ``VRT_priv_task_get()`` and ``VRT_priv_top_get()`` functions have been added to VRT to allow vmods to retrieve existing ``PRIV_TASK`` / ``PRIV_TOP`` private pointers without creating any. Backends ~~~~~~~~ * The VRT backend interface has been changed: - ``struct vrt_endpoint`` has been added describing a UDS or TCP endpoint for a backend to connect to. Endpoints also support a preamble to be sent with every new connection. - This structure needs to be passed via the ``endpoint`` member of ``struct vrt_backend`` when creating backends with ``VRT_new_backend()`` or ``VRT_new_backend_clustered()``. * ``VRT_Endpoint_Clone()`` has been added to facilitate working with endpoints. Filters (VDP/VFP) ~~~~~~~~~~~~~~~~~ * Many filter (VDP/VFP) related signatures have been changed: - ``vdp_init_f`` - ``vdp_fini_f`` - ``vdp_bytes_f`` - ``VDP_bytes()`` as well as ``struct vdp_entry`` and ``struct vdp_ctx`` ``VFP_Push()`` and ``VDP_Push()`` are no longer intended for VMOD use and have been removed from the API. * The VDP code is now more strict about ``VDP_END``, which must be sent down the filter chain at most once. Care should be taken to send ``VDP_END`` together with the last payload bytes whenever possible. Stevedore API ~~~~~~~~~~~~~ * The stevedore API has been changed: - ``OBJ_ITER_FINAL`` has been renamed to ``OBJ_ITER_END`` - ``ObjExtend()`` signature has been changed to also cover the ``ObjTrimStore()`` use case and - ``ObjTrimStore()`` has been removed. Developer: Changes for Authors of Varnish Utilities =================================================== libvarnishapi ~~~~~~~~~~~~~ * The ``VSC_IsRaw()`` function has been added to ``libvarnishapi`` to query if a gauge is being returned raw or adjusted (see `varnishstat -r option `_). *eof*