Changes in Varnish 7.0

For information about updating your current Varnish deployment to the new version, see Upgrading to Varnish 7.0.

A more detailed and technical account of changes in Varnish, with links to issues that have been fixed and pull requests that have been merged, may be found in the change log.

PCRE2

One major change for this release is the migration of the regular expression engine from PCRE to PCRE2. This change should be mostly transparent anywhere regular expressions are used, like VCL, ban expressions, VSL queries etc.

There were some parameters changes, see the upgrade notes for more details.

RFC8941 - Structured Fields

It will come as no surprise to VCL writers that HTTP headers use what can charitably be described as “heterogenous syntax”.

In 2016, on the train back from the HTTP Workshop in Stockholm, and in response to proposals to allow JSON in HTTP headers, I started an effort which culminated with the publication of RFC 8941 Structured Fields.

The syntax in Structured Fields is distilled from current standardized headers, which means that the vast majority of existing HTTP headers are already covered.

There are unfortunate exceptions, most notably the Cookie headers.

Starting with this release, we are gently migrating VCL towards the Structured Field semantics.

The first change is that it is now possible to include BLOBs in VCL, by using the RFC 8941 syntax of:

':' + <base64> + ':'

For example, the VCL string "helloworld" can be represented as the BLOB literal :aGVsbG93b3JsZA==: without involving a VMOD.

See VMOD blob - Utilities for the VCL blob type, encoding and decoding.

The second and likely more significant change is numbers in VCL now conform to RFC8941 as well: Up to 15 digits and at most 3 decimal places, and “scientific notation” is no longer allowed.

(These restrictions were chosen after much careful deliberation, to ensure that no overflows would occur, even when HTTP headers are processed in languages where numbers are represented by IEEE-754 64 binary floating point variables.)

varnishd

Parameters

There were changes to the parameters:

  • new pcre2_jit_compilation boolean defaulting to on

  • the default value increased to 16kB for vsl_buffer

  • the default value increased to 96kB for workspace_client

  • the default value increased to 96kB for workspace_backend

  • the minimum value increased to 384B for workspace_session

  • the minimum value increased to 65535B for h2_initial_window_size

  • the default value increased to 80kB for thread_pool_stack

  • the default value increased to 64kB for thread_pool_stack on 32bit systems

  • vcc_acl_pedantic was removed, see upgrade notes for more details.

  • pcre_match_limit was renamed to pcre2_match_limit

  • pcre_match_limit_recursion was renamed to pcre2_depth_limit

  • new h2_rxbuf_storage defaulting to Transient, see upgrade notes for more details.

Other changes in varnishd

For pass transactions, varnishd no longer strips Range headers from client requests or Accept-Range and Content-Range headers from backend responses to allow partial delivery directly from the backend.

When http_range_support is on (the default) a consistency check is performed on the backend response and malformed or inconsistent responses are treated as fetch errors.

There is a new buffer for HTTP/2 request bodies allocated from storage.

See upgrade notes for more details on both topics.

Changes to VCL

VCL variables

A new req.hash_ignore_vary flag allows to skip vary header checks during a lookup. This can be useful when only the freshness of a resource is relevant, and not a slight difference in representation.

For interoperability purposes, it is now possible to quote header names that aren’t valid VCL symbols, but valid HTTP header names, for example:

req.http."dotted.name"

This is rarely observed and should only be needed to better integrate with the specific needs of certain clients or servers.

Some global VCL symbols can be referenced before their declaration, this was extended to all global VCL symbols for the following keywords:

  • acl

  • backend

  • probe

  • sub

Consider the following example:

sub vcl_recv {
    set req.backend_hint = b;
}

backend b {
    .host = "example.org";
}

It used to fail the VCL compilation with “Symbol not found: ‘b’” in vcl_recv, and is now supported.

Bit flags

There is a new bit flag syntax for certain VCL keywords:

keyword +flag -other ...

Similarly to bit flag varnishd parameters debug, feature and vsl_mask, a + prefix means that a flag is raised and a - prefix that a flag is cleared.

The acl keyword supports the following flags:

  • log

  • pedantic (enabled by default)

  • table

For example:

acl <name> +log -pedantic { ... }

See Access Control List (ACL).

The include keyword supports a glob flag.

For example:

include +glob "example.org/*.vcl";

See Include statement.

See upgrade notes for more details.

VMODs

New BASE64CF encoding scheme in vmod_blob. It is similar to BASE64URL, with the following changes to BASE64:

  • + replaced with -

  • / replaced with ~

  • _ as the padding character

It is used by a certain CDN provider who also inspired the name.

See the vmod_blob manual (BASE64*).

varnishlog

If a cache hit occurs on a streaming object, an object that is still being fetched, Hit records contain progress of the fetch task. This should help troubleshooting when cache hits appear to be slow, whether or not the backend is still serving the response.

See VCL.

By default VCL_acl records are no longer emitted. They can be brought back by adding a +log flag to the ACL declaration.

See Access Control List (ACL).

varnishncsa

New %{...}t time formats:

  • sec

  • msec

  • usec

  • msec_frac

  • usec_frac

See the varnishncsa manual (FORMAT) for more information.

varnishadm

The -t option sets up the timeout for both attaching to a running varnishd instance and individual commands sent to that instance.

Command completion should be more accurate in interactive mode.

varnishtest

Test cases should be generally more reactive, whether it is detecting a varnishd startup failure, waiting for varnishd to stop, or when tests fail and there are barriers waiting for a synchronization.

Clients and servers can have up to 64 headers in requests and responses.

The feature command allows to skip gracefully test cases that are missing specific requirements. It is now possible to skip a test based on the presence of a feature.

For example, for test cases targeting 32bit environment with a working DNS setup:

feature dns !64bit

There are new feature checks:

  • coverage

  • asan

  • msan

  • tsan

  • ubsan

  • sanitizer

  • workspace_emulator

The undocumented pcre_jit feature check is gone.

See the VTC manual (feature) for more details.

There is a new tunnel command that acts as a proxy between two peers. A tunnel can pause and control how much data goes in each direction, and can be used to trigger socket timeouts, possibly in the middle of protocol frames, without having to change how the peers are implemented.

See the VTC manual (tunnel) for more details.

There is a new dynamic macro ${string,repeat,<uint>,<string>} to avoid very long lines or potential mistakes when maintained by hand. For example, the two following strings are equivalent:

"AAA"
"${string,repeat,3,A}"

See the VTC manual (MACROS) for more details.

There were also various improvements to HTTP/2 testing, and more should be expected.

Changes for developers and VMOD authors

Varnish now comes with a second workspace implementation called the workspace emulator. It needs to be enabled during the build with the configure flag --enable-workspace-emulator.

The workspace emulator performs sparse allocations and can help VMOD authors interested in fuzzing, especially when the Address Sanitizer is enabled as well.

In order to make the emulator possible, some adjustments were needed for the workspace API. Deprecated functions WS_Front() and WS_Inside() were removed independently of the emulator.

The STRING_LIST type is gone in favor of STRANDS. All the VRT symbols related to STRING_LIST are either gone or changed.

Convenience constants vrt_null_strands and vrt_null_blob were added.

The migration to PCRE2 also brought many changes to the VRE API. The VRT functions handling REGEX arguments didn’t change.

The VNUM API also changed substantially for structured field number parsing.

The deprecated functions VSB_new() and VSB_delete() were removed.

See upgrade notes for more information.

eof