Changes in Varnish 6.6

For information about updating your current Varnish deployment to the new version, see Upgrading to Varnish 6.6.

A more detailed and technical account of changes in Varnish, with links to issues that have been fixed and pull requests that have been merged, may be found in the change log.

varnishd

Arguments

• varnishd now supports the -b none argument to start with only the builtin VCL and no backend at all.

Parameters

• The validate_headers parameter has been added to control header validation.

• The ban_cutoff parameter now refers to the overall length of the ban list, including completed bans, where before only non-completed (“active”) bans were counted towards ban_cutoff.

• The vary_notice parameter has been added to control the threshold for the new Vary Notice.

feature Flags

• The busy_stats_rate feature flag has been added to ensure statistics updates (as configured using the thread_stats_rate parameter) even in scenarios where worker threads never run out of tasks and may remain forever busy.

Accounting

Body bytes accounting has been fixed to always represent the number of body bytes moved on the wire, exclusive of protocol-specific overhead like HTTP/1 chunked encoding or HTTP/2 framing.

This change affects counters like

• MAIN.s_req_bodybytes,

• MAIN.s_resp_bodybytes,

• VBE.*.*.bereq_bodybytes and

• VBE.*.*.beresp_bodybytes

as well as the VSL records

• ReqAcct,

• PipeAcct and

• BereqAcct.

Session Close Reasons

The connection close reason has been fixed to properly report SC_RESP_CLOSE / resp_close where previously only SC_REQ_CLOSE / req_close was reported.

For failing PROXY connections, SessClose now provides more detailed information on the cause of the failure.

The session close reason logging/statistics for HTTP/2 connections have been improved.

Vary Notice

A log (VSL) Notice record is now emitted whenever more than vary_notice variants are encountered in the cache for a specific hash. The new vary_notice parameter defaults to 10.

Changes to VCL

Header Validation

Unless the new validate_headers feature is disabled, all newly set headers are now validated to contain only characters allowed by RFC7230. A (runtime) VCL failure is triggered if not.

VCL variables

• The client.identity variable is now accessible on the backend side.

• The variables bereq.is_hitpass and bereq.is_hitmiss have been added to the backend side matching req.is_hitpass and req.is_hitmiss on the client side.

• The bereq.xid variable is now also available in vcl_pipe {}

• The resp.proto variable is now read-only as it should have been for long, like the other *.proto variables.

Other changes to VCL

• Long strings in VCL can now also be denoted using """ ... """ in addition to the existing {" ... "}.

• The ban() builtin is now deprecated and should be replaced with std.ban().

• Trying to use std.rollback() from vcl_pipe now results in VCL failure.

• The modulus operator % has been added to VCL.

• return(retry) from vcl_backend_error {} now correctly resets beresp.status and beresp.reason.

• The builtin VCL has been reworked: VCL code has been split into small subroutines, which custom VCL can prepend custom code to.

  This allows for better integration of custom VCL and the built-in VCL and better reuse.

VMODs

directors.shard()

• The shard director now supports reconfiguration (adding/removing backends) of several instances without any special ordering requirement.

• Calling the shard director .reconfigure() method is now optional. If not called explicitly, any shard director backend changes are applied at the end of the current task.

• Shard director Error log messages with (notice) have been turned into Notice log messages.

• All shard Error and Notice messages now use the unified prefix vmod_directors: shard %s.

std.set_ip_tos()

The set_ip_tos() function from the bundled std vmod now sets the IPv6 Traffic Class (TCLASS) when used on an IPv6 connection.

std.ban() and std.ban_error()

The std.ban() and std.ban_error() functions have been added to the std vmod, allowing VCL to check for ban errors. A typical usage pattern with the new interface is:

if (std.ban(...)) {
  return(synth(200, "Ban added"));
} else {
  return(synth(400, std.ban_error()));
}

cookie functions

The filter_re, keep_re and get_re functions from the bundled cookie vmod have been changed to take the VCL_REGEX type. This implies that their regular expression arguments now need to be literal, whereas before they could be taken from some other variable or function returning VCL_STRING.

Note that these functions never actually handled dynamic regexen, the string passed with the first call was compiled to a regex, which was then used for the lifetime of the respective VCL.

varnishlog

• See Accounting for changes to accounting-related VSL records.

• See Session Close Reasons for a change affecting SessClose.

• Three new Timestamp VSL records have been added to backend request processing:

  • The Process timestamp after return(deliver) or return(pass(x)) from vcl_backend_response,

  • the Fetch timestamp before a backend connection is requested and

  • the Connected timestamp when a connection to a regular backend (VBE) is established, or when a recycled connection was selected for reuse.

• The FetchError log message Timed out reusing backend connection has been renamed to first byte timeout (reused connection) to clarify that it is emit for effectively the same reason as first byte timeout.

• ExpKill log (VSL) records are now masked by default. See the vsl_mask parameter.

• Comparisons of numbers in VSL queries have been improved to match better the behavior which is likely expected by users who have not read the documentation in all detail.

• See Vary Notice for information on a newly added Notice log (VSL) record.

varnishncsa

• The %{X}T format has been added to varnishncsa, which generalizes %D and %T, but also support milliseconds (ms) output.

• The varnishncsa -E argument to show ESI requests has been changed to imply -c (client mode). This behavior is now shared by all log utilities, and -c no longer includes ESI requests.

varnishadm

• The vcl.discard CLI command can now be used to discard more than one VCL with a single command, which succeeds only if all given VCLs could be discarded (atomic behavior).

• The vcl.discard CLI command now supports glob patterns for vcl names.

• The vcl.deps CLI command has been added to output dependencies between VCLs (because of labels and return(vcl) statements).

• varnishadm now has the -p option to disable readline support for use in scripts and as a generic CLI connector.

varnishstat

• See Accounting for changes to accounting-related counters.

• See Session Close Reasons for a change affecting MAIN.sc_* counters.

• The MAIN.esi_req counter has been added as a statistic of the number of ESI sub requests created.

• The MAIN.s_bgfetch counter has been added as a statistic on the number of background fetches issued.

• varnishstat now avoids display errors of gauges which previously could underflow to negative values, being displayed as extremely high positive values.

  The -r option and the r key binding have been added to return to the previous behavior. When raw mode is active in varnishstat interactive (curses) mode, the word RAW is displayed at the right hand side in the lower status line.

varnishtest

Various improvements have been made to the varnishtest facility:

• the loop keyword now works everywhere

• HTTP/2 logging has been improved

• Default HTTP/2 parameters have been tweaked

• Varnish listen address information is now available by default in the macros ${vNAME_addr}, ${vNAME_port} and ${vNAME_sock}. Macros by the names ${vNAME_SOCKET_*} contain the address information for each listen socket as created with the -a argument to varnishd.

• Synchronization points for counters (VSCs) have been added as varnish vNAME -expect PATTERN OP PATTERN

• varnishtest now also works with IPv6 setups

• feature ipv4 and feature ipv6 can be used to control execution of test cases which require one or the other protocol.

• haproxy arguments can now be externally provided through the HAPROXY_ARGS variable.

• logexpect now supports alternatives with the expect ? ... syntax and negative matches with the fail add ... and fail clear syntax.

• The overall logexpect match expectation can now be inverted using the -err argument.

• Numeric comparisons for HTTP headers have been added: -lt, -le, -eq, -ne, -ge, -gt

• rxdata -some has been fixed.

Other Changes to Varnish Utilities

All varnish tools using the VUT library utilities for argument processing now support the --optstring argument to return a string suitable for use with getopts from shell scripts.

Developer: Changes for VMOD authors

VMOD/VCL interface

• The VCL_REGEX data type is now supported for VMODs, allowing them to use regular expression literals checked and compiled by the VCL compiler infrastructure.

  Consequently, the VRT_re_init() and VRT_re_fini() functions have been removed, because they are not required and their use was probably wrong anyway.

• The VCL_SUB data type is now supported for VMODs to save references to subroutines to be called later using VRT_call(). Calls from a wrong context (e.g. calling a subroutine accessing req from the backend side) and recursive calls fail the VCL.

  See VMOD - Varnish Modules in the Reference Manual.

VMOD functions can also return the VCL_SUB data type for calls from VCL as in call vmod.returning_sub();.

• VRT_check_call() can be used to check if a VRT_call() would succeed in order to avoid the potential VCL failure in case it would not.

  It returns NULL if VRT_call() would make the call or an error string why not.

• VRT_handled() has been added, which is now to be used instead of access to the handling member of VRT_CTX.

• vmodtool.py has been improved to simplify Makefiles when many VMODs are built in a single directory.

General API

• VRT_ValidHdr() has been added for VMODs to conduct the same check as the whatsnew_changes_6.6_header_validation feature, for example when headers are set by VMODs using the cache_http.c Functions like http_ForceHeader() from untrusted input.

• Client and backend finite state machine internals (enum req_step and enum fetch_step) have been removed from cache.h.

• The verrno.h header file has been removed and merged into vas.h

• The pdiff() function declaration has been moved from cache.h to vas.h.

VSA

• The VSA_getsockname() and VSA_getpeername() functions have been added to get address information of file descriptors.

Private Pointers

• The interface for private pointers in VMODs has been changed:

  • The free pointer in struct vmod_priv has been replaced with a pointer to struct vmod_priv_methods, to where the pointer to the former free callback has been moved as the fini member.

  • The former free callback type has been renamed from vmod_priv_free_f to vmod_priv_fini_f and as gained a VRT_CTX argument

• The VRT_priv_task_get() and VRT_priv_top_get() functions have been added to VRT to allow vmods to retrieve existing PRIV_TASK / PRIV_TOP private pointers without creating any.

Backends

• The VRT backend interface has been changed:

  • struct vrt_endpoint has been added describing a UDS or TCP endpoint for a backend to connect to.

    Endpoints also support a preamble to be sent with every new connection.

  • This structure needs to be passed via the endpoint member of struct vrt_backend when creating backends with VRT_new_backend() or VRT_new_backend_clustered().

• VRT_Endpoint_Clone() has been added to facilitate working with endpoints.

Filters (VDP/VFP)

• Many filter (VDP/VFP) related signatures have been changed:

  • vdp_init_f

  • vdp_fini_f

  • vdp_bytes_f

  • VDP_bytes()

  as well as struct vdp_entry and struct vdp_ctx

  VFP_Push() and VDP_Push() are no longer intended for VMOD use and have been removed from the API.

• The VDP code is now more strict about VDP_END, which must be sent down the filter chain at most once. Care should be taken to send VDP_END together with the last payload bytes whenever possible.

Stevedore API

• The stevedore API has been changed:

  • OBJ_ITER_FINAL has been renamed to OBJ_ITER_END

  • ObjExtend() signature has been changed to also cover the ObjTrimStore() use case and

  • ObjTrimStore() has been removed.

Developer: Changes for Authors of Varnish Utilities

libvarnishapi

• The VSC_IsRaw() function has been added to libvarnishapi to query if a gauge is being returned raw or adjusted (see varnishstat -r option).

eof