Upgrading to Varnish 5.2

Varnish statistics and logging

There are extensive changes under the hood with respect to statistics counters, but these should all be transparent at the user-level.

varnishd parameters

The vsm_space and cli_buffer parameters are now deprecated and ignored. They will be removed in a future major release.

The updated shared memory implementation manages space automatically, so it no longer needs vsm_space. Memory for the CLI command buffer is now dynamically allocated.

We have updated the documentation for send_timeout, idle_send_timeout, timeout_idle and ban_cutoff.

Added the debug bit vmod_so_keep, see debug and the notes about changes for developers below.

Changes to VCL

We have added a few new variables and clarified some matters. VCL written for Varnish 5.1 should run without changes on 5.2.

Consistent symbol names

VCL symbols originate from various parts of Varnish: there are built-in variables, subroutines, functions, and the free-form headers. Symbols may live in a namespace denoted by the '.' (dot) character as in req.http.Cache-Control. When you create a VCL label, a new symbol becomes available, named after the label. Storage backends always have a name, even if you don’t specify one, and they can also be accessed in VCL: for example storage.Transient.

Because headers and VCL names could contain dashes, while subroutines or VMOD objects couldn’t, this created an inconsistency. All symbols follow the same rules now and must follow the same (case-insensitive) pattern: [a-z][a-z0-9_-]*.

You can now write code like:

sub my-sub {
    new my-obj = my_vmod.my_constuctor(storage.my-store);
}

sub vcl_init {
    call my-sub;
}

As you may notice in the example above, it is not possible yet to have dashes in a vmod symbol.

Long storage backend names used to be truncated due to a limitation in the VSC subsystem, this is no longer the case.

VCL variables

req.hash and bereq.hash

Added req.hash and bereq.hash, which contain the hash value computed by Varnish for cache lookup in the current transaction, to be used in client or backend context, respectively. Their data type is BLOB, and they contain the raw binary hash.

You can use VMOD blob - Utilities for the VCL blob type, encoding and decoding to work with the hashes:

import blob;

sub vcl_backend_fetch {
    # Send the transaction hash to the backend as a hex string
    set bereq.http.Hash = blob.encode(HEX, blob=bereq.hash);
}

sub vcl_deliver {
    # Send the hash in a response header as a base64 string
    set resp.http.Hash = blob.encode(BASE64, blob=req.hash);
}

server.identity

If the -i option is not set in the invocation of varnishd, then server.identity is set to the host name (as returned by gethostname(3)). Previously, server.identity defaulted to the value of the -n option (or the default instance name if -n was not set). See varnishd.

bereq.is_bgfetch

Added bereq.is_bgfetch, which is readable in backend contexts, and is true if the fetch takes place in the background. That is, it is true if Varnish found a response in the cache whose TTL was expired, but was still in grace time. Varnish returns the stale cached response to the client, and initiates the background fetch to refresh the cache object.

req.backend_hint

We have clarified what happens to req.backend_hint on a client restart – it gets reset to the default backend. So you might want to make sure that the backend hint gets set the way you want in that situation.

vmod_std

Added BOOL file_exists(STRING path).

New VMODs in the standard distribution

See VMOD blob - Utilities for the VCL blob type, encoding and decoding, VMOD purge - Varnish Purge Module and VMOD vtc - Utility module for varnishtest. Read about them in New VMODs in the standard distribution.

Bans

We have clarified the interpretation of a ban when a comparison in the ban expression is attempted against an unset field, see ban(STRING) in VCL.

Other changes

  • varnishd(1):

    • The total size of the shared memory space for logs and counters no longer needs to be configured explicitly and therefore the second subargument to -l is now ignored.

    • The default value of server.identity when the -i option is not set has been changed as noted above.

    • Also, -i no longer determines the ident field used by syslog(3); now Varnish is always identified by the string varnishd in the syslog.

    • On a system that supports setproctitle(3), the Varnish management process will appear in the output of ps(1) as Varnish-Mgt, and the child process as Varnish-Child. If the -i option has been set, then these strings in the ps output are followed by -i and the identity string set by the option.

    • The -f option for a VCL source file now honors the vcl_path parameter if a relative file name is used, see varnishd and vcl_path.

    • The -a option can now take a name, for example -a admin=127.0.0.1:88 to identify an address used for administrative requests but not regular client traffic. Otherwise, a default name is selected for the listen address (a0, a1 and so forth). Endpoint names appear in the log output, as noted below, and may become accessible in VCL in the future.

  • varnishstat(1):

    • In curses mode, the top two lines showing uptimes for the management and child processes show the text Not Running if one or both of the processes are down.

    • The interpretation of multiple -f options in the command line has changed slightly, see varnishstat.

    • The type and ident fields have been removed from the XML and JSON output formats, see varnishstat.

    • The MAIN.s_req statistic has been removed, as it was identical to MAIN.client_req.

    • Added the counter req_dropped. Similar to sess_dropped, this is the number of times an HTTP/2 stream was refused because the internal queue is full. See varnish-counters and thread_queue_limit.

  • varnishlog(1):

    • The Hit, HitMiss and HitPass log records grew an additional field with the remaining TTL of the object at the time of the lookup. While this should greatly help troubleshooting, it might break tools relying on those records to get the VXID of the object hit during lookup.

      Instead of using Hit, such tools should now use Hit[1], and the same applies to HitMiss and HitPass.

      The Hit record also grew two more fields for the grace and keep periods. This should again be useful for troubleshooting.

      See VSL.

    • The SessOpen log record displays the name of the listen address instead of the endpoint in its 3rd field.

      See VSL.

    • The output format of VCL_trace log records, which appear if you have switched on the VCL_trace flag in the VSL mask, has changed to include the VCL configuration name. See VSL and vsl_mask.

  • varnishtest(1) and vtc(7):

    • When varnishtest is invoked with -L or -l, Varnish instances started by a test do not clean up their copies of VMOD shared objects when they stop. See the note about vmod_so_keep below.

    • Added the feature switch ignore_unknown_macro for test cases, see VTC.

  • varnishncsa(1)

    • Field specifiers (such as the 1 in Hit[1]) are now limited to to 255, see varnishncsa.

  • The -N command-line option, which was previously available for varnishlog(1), varnishstat(1), varnishncsa(1) and varnishhist(1), is not compatible with the changed internal logging API, and has been retired.

  • Changes for developers:

    • The VSM and VSC APIs for shared memory and statistics have changed, and may necessitate changes in client applications, see VSM/VSC API changes.

    • Added the $ABI directive for VMOD vcc declarations, see $ABI [strict|vrt].

    • There have been some minor changes in the VRT API, which may be used for VMODs and client apps, see VRT API changes.

    • The VUT API (for Varnish UTilities), which facilitates the development of client apps, is now publicly available, see Added VUT API.

    • The debug bit vmod_so_keep instructs Varnish not to clean up its copies of VMOD shared objects when it stops. This makes it possible for VMOD authors to load their code into a debugger after a varnishd crash. See debug.

eof