vmod_std

Varnish Standard Module

Manual section:

3

SYNOPSIS

import std [from "path"] ;

STRING toupper(STRING s)

STRING tolower(STRING s)

VOID set_ip_tos(INT tos)

REAL random(REAL lo, REAL hi)

VOID log(STRING s)

VOID syslog(INT priority, STRING s)

STRING fileread(STRING)

BOOL file_exists(STRING path)

VOID collect(HEADER hdr, STRING sep)

DURATION duration(STRING s, DURATION fallback)

INT integer(STRING s, INT fallback)

IP ip(STRING s, IP fallback, BOOL resolve, STRING p)

REAL real(STRING s, REAL fallback)

INT real2integer(REAL r, INT fallback)

TIME real2time(REAL r, TIME fallback)

INT time2integer(TIME t, INT fallback)

REAL time2real(TIME t, REAL fallback)

BOOL healthy(BACKEND be)

INT port(IP ip)

VOID rollback(HTTP h)

VOID timestamp(STRING s)

STRING querysort(STRING)

BOOL cache_req_body(BYTES size)

STRING strstr(STRING s1, STRING s2)

TIME time(STRING s, TIME fallback)

STRING getenv(STRING name)

VOID late_100_continue(BOOL late)

BOOL syntax(REAL)

BOOL fnmatch(STRING pattern, STRING subject, BOOL pathname, BOOL noescape, BOOL period)

CONTENTS

DESCRIPTION

vmod_std contains basic functions which are part and parcel of Varnish, but which for reasons of architecture fit better in a VMOD.

One particular class of functions in vmod_std is the conversions functions which all have the form:

TYPE type(STRING, TYPE)

These functions attempt to convert STRING to the TYPE, and if that fails, they return the second argument, which must have the given TYPE.

STRING toupper(STRING s)

Description

Converts the string s to uppercase.

Example

set beresp.http.scream = std.toupper(“yes!”);

STRING tolower(STRING s)

Description

Converts the string s to lowercase.

Example

set beresp.http.nice = std.tolower(“VerY”);

VOID set_ip_tos(INT tos)

Description

Sets the IP type-of-service (TOS) field for the current session to tos. Silently ignored if the listen address is a Unix domain socket. Please note that the TOS field is not removed by the end of the request so probably want to set it on every request should you utilize it.

Example
if (req.url ~ “^/slow/”) {
std.set_ip_tos(0);
}

REAL random(REAL lo, REAL hi)

Description

Returns a random real number between lo and hi. This function uses the “testable” random generator in varnishd which enables determinstic tests to be run (See m00002.vtc). This function should not be used for cryptographic applications.

Example

set beresp.http.random-number = std.random(1, 100);

VOID log(STRING s)

Description

Logs the string s to the shared memory log, using VSL tag SLT_VCL_Log.

Example

std.log(“Something fishy is going on with the vhost “ + req.http.host);

VOID syslog(INT priority, STRING s)

Description

Logs the string s to syslog tagged with priority. priority is formed by ORing the facility and level values. See your system’s syslog.h file for possible values.

Notice: Unlike VCL and other functions in the std vmod, this function will not fail VCL processing for workspace overflows: For an out of workspace condition, the syslog() function has no effect.

Example

std.syslog(9, “Something is wrong”);

This will send a message to syslog using LOG_USER | LOG_ALERT.

STRING fileread(STRING)

Description

Reads a file and returns a string with the content. The result is cached indefinitely per filename.

Example

synthetic(“Response was served by “ + std.fileread(“/etc/hostname”));

Consider that the entire contents of the file appear in the string that is returned, including newlines that may result in invalid headers if std.fileread() is used to form a header. In that case, you may need to modify the string, for example with regsub():

set beresp.http.served-by = regsub(std.fileread("/etc/hostname"), "\R$", "");

BOOL file_exists(STRING path)

Description

Returns true if path or the file pointed to by path exists, false otherwise.

Example
if (std.file_exists(“/etc/return_503”)) {
return (synth(503, “Varnish is in maintenance”));
}

VOID collect(HEADER hdr, STRING sep=”, “)

Description

Collapses multiple hdr headers into one long header. The default separator sep is the standard comma separator to use when collapsing headers, with an additional whitespace for pretty printing.

Care should be taken when collapsing headers. In particular collapsing Set-Cookie will lead to unexpected results on the browser side.

Examples
std.collect(req.http.accept);
std.collect(req.http.cookie, “; “);

DURATION duration(STRING s, DURATION fallback)

Description

Converts the string s to seconds. s must be quantified with ms (milliseconds), s (seconds), m (minutes), h (hours), d (days), w (weeks) or y (years) units. If conversion fails, fallback will be returned.

Example

set beresp.ttl = std.duration(“1w”, 3600s);

INT integer(STRING s, INT fallback)

Description

Converts the string s to an integer. If conversion fails, fallback will be returned.

Example
if (std.integer(req.http.foo, 0) > 5) {
…
}

IP ip(STRING s, IP fallback, BOOL resolve=1, STRING p=”80”)

Description

Converts the string s to the first IP number returned by the system library function getaddrinfo(3). If conversion fails, fallback will be returned or VCL failure will happen.

The IP address includes a port number that can be found with std.port() that defaults to 80. The default port can be set to a different value with the p argument. It will be overriden if s contains both an IP address and a port number or service name.

When s contains both, the syntax is either address:port or address port. If the address is a numerical IPv6 address it must be enclosed between brackets, for example [::1] 80 or [::1]:http. The fallback may also contain both an address and a port, but its default port is always 80.

If resolve is false, getaddrinfo(3) is called using AI_NUMERICHOST and AI_NUMERICSERV to avoid network lookups depending on the system’s getaddrinfo(3) or nsswitch configuration. This makes “numerical” IP strings and services cheaper to convert.

Example
if (std.ip(req.http.X-forwarded-for, “0.0.0.0”) ~ my_acl) {
…
}

REAL real(STRING s, REAL fallback)

Description

Converts the string s to a real. If conversion fails, fallback will be returned.

Example
if (std.real(req.http.foo, 0.0) > 5.5) {
…
}

INT real2integer(REAL r, INT fallback)

Description

Rounds the real r to the nearest integer, but round halfway cases away from zero (see round(3)). If conversion fails, fallback will be returned.

Example

set req.http.integer = std.real2integer(1140618699.00, 0); set req.http.posone = real2integer( 0.5, 0); # = 1.0 set req.http.negone = real2integer(-0.5, 0); # = -1.0

TIME real2time(REAL r, TIME fallback)

Description

Rounds the real r to the nearest integer (see func_real2integer) and returns the corresponding time when interpreted as a unix epoch. If conversion fails, fallback will be returned.

Example

set req.http.time = std.real2time(1140618699.00, now);

INT time2integer(TIME t, INT fallback)

Description

Converts the time t to a integer. If conversion fails, fallback will be returned.

Example

set req.http.int = std.time2integer(now, 0);

REAL time2real(TIME t, REAL fallback)

Description

Converts the time t to a real. If conversion fails, fallback will be returned.

Example

set req.http.real = std.time2real(now, 1.0);

BOOL healthy(BACKEND be)

Description

Returns true if the backend be is healthy.

INT port(IP ip)

Description

Returns the port number of the IP address ip. Always returns 0 for a *.ip variable whose value is 0.0.0.0 because the listen address is a Unix domain socket.

VOID rollback(HTTP h)

Description

Restores the h HTTP headers to their original state.

Example

std.rollback(bereq);

VOID timestamp(STRING s)

Description

Introduces a timestamp in the log with the current time, using the string s as the label. This is useful to time the execution of lengthy VCL procedures, and makes the timestamps inserted automatically by Varnish more accurate.

Example

std.timestamp(“curl-request”);

STRING querysort(STRING)

Description

Sorts the query string for cache normalization purposes.

Example

set req.url = std.querysort(req.url);

BOOL cache_req_body(BYTES size)

Description

Caches the request body if it is smaller than size. Returns true if the body was cached, false otherwise.

Normally the request body is not available after sending it to the backend. By caching it is possible to retry pass operations, e.g. POST and PUT.

Example
if (std.cache_req_body(1KB)) {
…
}

STRING strstr(STRING s1, STRING s2)

Description

Returns a string beginning at the first occurrence of the string s2 in the string s1, or an empty string if s2 is not found.

Note that the comparison is case sensitive.

Example
if (std.strstr(req.url, req.http.restrict)) {
…
}

This will check if the content of req.http.restrict occurs anywhere in req.url.

TIME time(STRING s, TIME fallback)

Description

Converts the string s to a time. If conversion fails, fallback will be returned.

Supported formats:

“Sun, 06 Nov 1994 08:49:37 GMT”
“Sunday, 06-Nov-94 08:49:37 GMT”
“Sun Nov 6 08:49:37 1994”
“1994-11-06T08:49:37”
“784111777.00”
“784111777”
Example
if (std.time(resp.http.last-modified, now) < now - 1w) {
…
}

STRING getenv(STRING name)

Description

Return environment variable name or the empty string.

See getenv(3)

Example
set req.http.My-Env = std.getenv(“MY_ENV”);

VOID late_100_continue(BOOL late)

Description

Controls when varnish reacts to an Expect: 100-continue client request header.

Varnish always generates a 100 Continue response if requested by the client trough the Expect: 100-continue header when waiting for request body data.

But, by default, the 100 Continue response is already generated immediately after vcl_recv returns to reduce latencies under the assumption that the request body will be read eventually.

Calling std.late_100_continue(true) in vcl_recv will cause the 100 Continue response to only be sent when needed. This may cause additional latencies for processing request bodies, but is the correct behavior by strict interpretation of RFC7231.

This function has no effect outside vcl_recv and after calling std.cache_req_body() or any other function consuming the request body.

Example
vcl_recv {
std.late_100_continue(true);

if (req.method == “POST”) {
std.late_100_continue(false);
return (pass);
}
…
}

BOOL syntax(REAL)

Description

Returns the true if VCL version is at least REAL.

fnmatch(…)

BOOL fnmatch(
   STRING pattern,
   STRING subject,
   BOOL pathname=1,
   BOOL noescape=0,
   BOOL period=0
)
Description

Shell-style pattern matching; returns true if subject matches pattern, where pattern may contain wildcard characters such as * or ?.

The match is executed by the implementation of fnmatch(3) on your system. The rules for pattern matching on most systems include the following:

  • * matches any sequence of characters

  • ? matches a single character

  • a bracket expression such as [abc] or [!0-9] is interpreted as a character class according to the rules of basic regular expressions (not PCRE regexen), except that ! is used for character class negation instead of ^.

If pathname is true, then the forward slash character / is only matched literally, and never matches *, ? or a bracket expression. Otherwise, / may match one of those patterns. By default, pathname is true.

If noescape is true, then the backslash character \ is matched as an ordinary character. Otherwise, \ is an escape character, and matches the character that follows it in the pattern. For example, \\ matches \ when noescape is true, and \\ when false. By default, noescape is false.

If period is true, then a leading period character . only matches literally, and never matches *, ? or a bracket expression. A period is leading if it is the first character in subject; if pathname is also true, then a period that immediately follows a / is also leading (as in “/.”). By default, period is false.

fnmatch() invokes VCL failure and returns false if either of pattern or subject is NULL – for example, if an unset header is specified.

Examples
# Matches URLs such as /foo/bar and /foo/baz
if (std.fnmatch(“/foo/*”, req.url)) { … }

# Matches URLs such as /foo/bar/baz and /foo/baz/quux
if (std.fnmatch(“/foo/*/*”, bereq.url)) { … }

# Matches /foo/bar/quux, but not /foo/bar/baz/quux
if (std.fnmatch(“/foo/*/quux”, req.url)) { … }

# Matches /foo/bar/quux and /foo/bar/baz/quux
if (std.fnmatch(“/foo/*/quux”, req.url, pathname=false)) { … }

# Matches /foo/bar, /foo/car and /foo/far
if (std.fnmatch(“/foo/?ar”, req.url)) { … }

# Matches /foo/ followed by a non-digit
if (std.fnmatch(“/foo/[!0-9]”, req.url)) { … }

SEE ALSO

vmod_directors

Varnish Directors Module

Manual section:

3

SYNOPSIS

import directors [from "path"] ;

new xround_robin = directors.round_robin()

   VOID xround_robin.add_backend(BACKEND)

   VOID xround_robin.remove_backend(BACKEND)

   BACKEND xround_robin.backend()

new xfallback = directors.fallback(BOOL sticky)

   VOID xfallback.add_backend(BACKEND)

   VOID xfallback.remove_backend(BACKEND)

   BACKEND xfallback.backend()

new xrandom = directors.random()

   VOID xrandom.add_backend(BACKEND, REAL)

   VOID xrandom.remove_backend(BACKEND)

   BACKEND xrandom.backend()

new xhash = directors.hash()

   VOID xhash.add_backend(BACKEND, REAL)

   VOID xhash.remove_backend(BACKEND)

   BACKEND xhash.backend(STRING)

new xshard = directors.shard()

   VOID xshard.set_warmup(REAL probability)

   VOID xshard.set_rampup(DURATION duration)

   VOID xshard.associate(BLOB param)

   BOOL xshard.add_backend(BACKEND backend, [STRING ident], [DURATION rampup], [REAL weight])

   BOOL xshard.remove_backend([BACKEND backend], [STRING ident])

   BOOL xshard.clear()

   BOOL xshard.reconfigure(INT replicas)

   INT xshard.key(STRING)

   BACKEND xshard.backend([ENUM by], [INT key], [BLOB key_blob], [INT alt], [REAL warmup], [BOOL rampup], [ENUM healthy], [BLOB param], [ENUM resolve])

   VOID xshard.debug(INT)

new xshard_param = directors.shard_param()

   VOID xshard_param.clear()

   VOID xshard_param.set([ENUM by], [INT key], [BLOB key_blob], [INT alt], [REAL warmup], [BOOL rampup], [ENUM healthy])

   STRING xshard_param.get_by()

   INT xshard_param.get_key()

   INT xshard_param.get_alt()

   REAL xshard_param.get_warmup()

   BOOL xshard_param.get_rampup()

   STRING xshard_param.get_healthy()

   BLOB xshard_param.use()

CONTENTS

DESCRIPTION

vmod_directors enables backend load balancing in Varnish.

The module implements load balancing techniques, and also serves as an example on how one could extend the load balancing capabilities of Varnish.

To enable load balancing you must import this vmod (directors).

Then you define your backends. Once you have the backends declared you can add them to a director. This happens in executed VCL code. If you want to emulate the previous behavior of Varnish 3.0 you can just initialize the directors in vcl_init, like this:

sub vcl_init {
    new vdir = directors.round_robin();
    vdir.add_backend(backend1);
    vdir.add_backend(backend2);
}

As you can see there is nothing keeping you from manipulating the directors elsewhere in VCL. So, you could have VCL code that would add more backends to a director when a certain URL is called.

Note that directors can use other directors as backends.

new xround_robin = directors.round_robin()

Description

Create a round robin director.

This director will pick backends in a round robin fashion.

Example

new vdir = directors.round_robin();

VOID xround_robin.add_backend(BACKEND)

Description

Add a backend to the round-robin director.

Example

vdir.add_backend(backend1);

VOID xround_robin.remove_backend(BACKEND)

Description

Remove a backend from the round-robin director.

Example

vdir.remove_backend(backend1);

BACKEND xround_robin.backend()

Description

Pick a backend from the director.

Example

set req.backend_hint = vdir.backend();

new xfallback = directors.fallback(BOOL sticky=0)

Description

Create a fallback director.

A fallback director will try each of the added backends in turn, and return the first one that is healthy.

If sticky is set to true, the director will keep using the healthy backend, even if a higher-priority backend becomes available. Once the whole backend list is exhausted, it’ll start over at the beginning.

Example

new vdir = directors.fallback();

VOID xfallback.add_backend(BACKEND)

Description

Add a backend to the director.

Note that the order in which this is done matters for the fallback director.

Example

vdir.add_backend(backend1);

VOID xfallback.remove_backend(BACKEND)

Description

Remove a backend from the director.

Example

vdir.remove_backend(backend1);

BACKEND xfallback.backend()

Description

Pick a backend from the director.

Example

set req.backend_hint = vdir.backend();

new xrandom = directors.random()

Description

Create a random backend director.

The random director distributes load over the backends using a weighted random probability distribution. The “testable” random generator in varnishd is used, which enables deterministic tests to be run (See: d00004.vtc).

Example

new vdir = directors.random();

VOID xrandom.add_backend(BACKEND, REAL)

Description

Add a backend to the director with a given weight.

Each backend will receive approximately 100 * (weight / (sum(all_added_weights))) per cent of the traffic sent to this director.

Example
# 2/3 to backend1, 1/3 to backend2.
vdir.add_backend(backend1, 10.0);
vdir.add_backend(backend2, 5.0);

VOID xrandom.remove_backend(BACKEND)

Description

Remove a backend from the director.

Example

vdir.remove_backend(backend1);

BACKEND xrandom.backend()

Description

Pick a backend from the director.

Example

set req.backend_hint = vdir.backend();

new xhash = directors.hash()

Description

Create a hashing backend director.

The director chooses the backend server by computing a hash/digest of the string given to .backend().

Commonly used with client.ip or a session cookie to get sticky sessions.

Example

new vdir = directors.hash();

VOID xhash.add_backend(BACKEND, REAL)

Description

Add a backend to the director with a certain weight.

Weight is used as in the random director. Recommended value is 1.0 unless you have special needs.

Example

vdir.add_backend(backend1, 1.0);

VOID xhash.remove_backend(BACKEND)

Description

Remove a backend from the director.

Example

vdir.remove_backend(backend1);

BACKEND xhash.backend(STRING)

Description

Pick a backend from the backend director.

Use the string or list of strings provided to pick the backend.

Example
# pick a backend based on the cookie header from the client
set req.backend_hint = vdir.backend(req.http.cookie);

new xshard = directors.shard()

Create a shard director.

Note that the shard director needs to be configured using at least one shard.add_backend() call(s) followed by a shard.reconfigure() call before it can hand out backends.

_Note_ that due to various restrictions (documented below), it is recommended to use the shard director on the backend side.

Introduction

The shard director selects backends by a key, which can be provided directly or derived from strings. For the same key, the shard director will always return the same backend, unless the backend configuration or health state changes. Conversely, for differing keys, the shard director will likely choose different backends. In the default configuration, unhealthy backends are not selected.

The shard director resembles the hash director, but its main advantage is that, when the backend configuration or health states change, the association of keys to backends remains as stable as possible.

In addition, the rampup and warmup features can help to further improve user-perceived response times.

Sharding

This basic technique allows for numerous applications like optimizing backend server cache efficiency, Varnish clustering or persisting sessions to servers without keeping any state, and, in particular, without the need to synchronize state between nodes of a cluster of Varnish servers:

  • Many applications use caches for data objects, so, in a cluster of application servers, requesting similar objects from the same server may help to optimize efficiency of such caches.

    For example, sharding by URL or some id component of the url has been shown to drastically improve the efficiency of many content management systems.

  • As special case of the previous example, in clusters of Varnish servers without additional request distribution logic, each cache will need store all hot objects, so the effective cache size is approximately the smallest cache size of any server in the cluster.

    Sharding allows to segregate objects within the cluster such that each object is only cached on one of the servers (or on one primary and one backup, on a primary for long and others for short etc…). Effectively, this will lead to a cache size in the order of the sum of all individual caches, with the potential to drastically increase efficiency (scales by the number of servers).

  • Another application is to implement persistence of backend requests, such that all requests sharing a certain criterion (such as an IP address or session ID) get forwarded to the same backend server.

When used with clusters of varnish servers, the shard director will, if otherwise configured equally, make the same decision on all servers. In other words, requests sharing a common criterion used as the shard key will be balanced onto the same backend server(s) no matter which Varnish server handles the request.

The drawbacks are:

  • the distribution of requests depends on the number of requests per key and the uniformity of the distribution of key values. In short, while this technique may lead to much better efficiency overall, it may also lead to less good load balancing for specific cases.

  • When a backend server becomes unavailable, every persistence technique has to reselect a new backend server, but this technique will also switch back to the preferred server once it becomes healthy again, so when used for persistence, it is generally less stable compared to stateful techniques (which would continue to use a selected server for as long as possible (or dictated by a TTL)).

Method

When .reconfigure() is called, a consistent hashing circular data structure gets built from the last 32 bits of SHA256 hash values of <ident><n> (default ident being the backend name) for each backend and for a running number n from 1 to replicas. Hashing creates the seemingly random order for placement of backends on the consistent hashing ring. When .add_backend() is called with a weight argument, replicas is scaled by that weight to add proportionally more copies of the that backend on the ring.

When .backend() is called, a load balancing key gets generated unless provided. The smallest hash value in the circle is looked up that is larger than the key (searching clockwise and wrapping around as necessary). The backend for this hash value is the preferred backend for the given key.

If a healthy backend is requested, the search is continued linearly on the ring as long as backends found are unhealthy or all backends have been checked. The order of these “alternative backends” on the ring is likely to differ for different keys. Alternative backends can also be selected explicitly.

On consistent hashing see:

Error Reporting

Failing methods should report errors to VSL with the Error tag, so when configuring the shard director, you are advised to check:

varnishlog -I Error:^shard

VOID xshard.set_warmup(REAL probability=0.0)

Set the default warmup probability. See the warmup parameter of shard.backend(). If probability is 0.0 (default), warmup is disabled.

VOID xshard.set_rampup(DURATION duration=0)

Set the default rampup duration. See rampup parameter of shard.backend(). If duration is 0 (default), rampup is disabled.

VOID xshard.associate(BLOB param=0)

Associate a default obj_shard_param object or clear an association.

The value of the param argument must be a call to the func_shard_param.use method. No argument clears the association.

The association can be changed per backend request using the param argument of func_shard.backend.

shard.add_backend(…)

BOOL xshard.add_backend(
      BACKEND backend,
      [STRING ident],
      [DURATION rampup],
      [REAL weight]
)

Add a backend backend to the director.

ident: Optionally specify an identification string for this backend, which will be hashed by shard.reconfigure() to construct the consistent hashing ring. The identification string defaults to the backend name.

ident allows to add multiple instances of the same backend.

rampup: Optionally specify a specific rampup time for this backend. Otherwise, the per-director rampup time is used (see VOID xshard.set_rampup(DURATION duration=0)).

weight: Optionally specify a weight to scale the shard.reconfigure() replicas parameter. weight is limited to at least 1. Values above 10 probably do not make much sense. The effect of weight is also capped such that the total number of replicas does not exceed UINT32_MAX.

NOTE: Backend changes need to be finalized with shard.reconfigure() and are only supported on one shard director at a time.

shard.remove_backend(…)

BOOL xshard.remove_backend(
      [BACKEND backend=0],
      [STRING ident=0]
)

Remove backend(s) from the director. Either backend or ident must be specified. ident removes a specific instance. If backend is given without ident, all instances of this backend are removed.

NOTE: Backend changes need to be finalized with shard.reconfigure() and are only supported on one shard director at a time.

BOOL xshard.clear()

Remove all backends from the director.

NOTE: Backend changes need to be finalized with shard.reconfigure() and are only supported on one shard director at a time.

BOOL xshard.reconfigure(INT replicas=67)

Reconfigure the consistent hashing ring to reflect backend changes.

This method must be called at least once before the director can be used.

INT xshard.key(STRING)

Convenience method to generate a sharding key for use with the key argument to the shard.backend() method by hashing the given string with SHA256.

To generate sharding keys using other hashes, use a custom vmod like vmod blobdigest with the key_blob argument of the shard.backend() method.

shard.backend(…)

BACKEND xshard.backend(
      [ENUM {HASH, URL, KEY, BLOB} by=HASH],
      [INT key],
      [BLOB key_blob],
      [INT alt=0],
      [REAL warmup=-1],
      [BOOL rampup=1],
      [ENUM {CHOSEN, IGNORE, ALL} healthy=CHOSEN],
      [BLOB param],
      [ENUM {NOW, LAZY} resolve]
)

Lookup a backend on the consistent hashing ring.

This documentation uses the notion of an order of backends for a particular shard key. This order is deterministic but seemingly random as determined by the consistent hashing algorithm and is likely to differ for different keys, depending on the number of backends and the number of replicas. In particular, the backend order referred to here is _not_ the order given when backends are added.

  • by how to determine the sharding key

    • HASH:

      • when called in backend context and in vcl_pipe {}: Use the varnish hash value as set by vcl_hash{}

      • when called in client context other than vcl_pipe {}: hash req.url

    • URL: hash req.url / bereq.url

    • KEY: use the key argument

    • BLOB: use the key_blob argument

  • key lookup key with by=KEY

    the shard.key() function may come handy to generate a sharding key from custom strings.

  • key_blob lookup key with by=BLOB

    Currently, this uses the first 4 bytes from the given blob in network byte order (big endian), left-padded with zeros for blobs smaller than 4 bytes.

  • alt alternative backend selection

    Select the alt-th alternative backend for the given key.

    This is particularly useful for retries / restarts due to backend errors: By setting alt=req.restarts or alt=bereq.retries with healthy=ALL, another server gets selected.

    The rampup and warmup features are only active for alt==0

  • rampup slow start for servers which just went healthy

    If alt==0 and the chosen backend is in its rampup period, with a probability proportional to the fraction of time since the backup became healthy to the rampup period, return the next alternative backend, unless this is also in its rampup period.

    The default rampup interval can be set per shard director using the set_rampup() method or specifically per backend with the set_backend() method.

  • warmup probabilistic alternative server selection

    possible values: -1, 0..1

    -1: use the warmup probability from the director definition

    Only used for alt==0: Sets the ratio of requests (0.0 to 1.0) that goes to the next alternate backend to warm it up when the preferred backend is healthy. Not active if any of the preferred or alternative backend are in rampup.

    warmup=0.5 is a convenient way to spread the load for each key over two backends under normal operating conditions.

  • healthy

    • CHOSEN: Return a healthy backend if possible.

      For alt==0, return the first healthy backend or none.

      For alt > 0, ignore the health state of backends skipped for alternative backend selection, then return the next healthy backend. If this does not exist, return the last healthy backend of those skipped or none.

    • IGNORE: Completely ignore backend health state

      Just return the first or alt-th alternative backend, ignoring health state. Ignore rampup and warmup.

    • ALL: Check health state also for alternative backend selection

      For alt > 0, return the alt-th alternative backend of all those healthy, the last healthy backend found or none.

  • resolve

    default: LAZY in vcl_init{}, NOW otherwise

    • NOW: look up a backend and return it.

      Can not be used in vcl_init{}.

    • LAZY: return an instance of this director for later backend resolution.

      LAZY mode is required for referencing shard director instances, for example as backends for other directors (director layering).

      In vcl_init{} and on the client side, LAZY mode can not be used with any other argument.

      On the backend side and in vcl_pipe {}, parameters from arguments or an associated parameter set affect the shard director instance for the backend request irrespective of where it is referenced.

  • param

    Use or associate a parameter set. The value of the param argument must be a call to the func_shard_param.use method.

    default: as set by func_shard.associate or unset.

    • for resolve=NOW take parameter defaults from the obj_shard_param parameter set

    • for resolve=LAZY associate the obj_shard_param parameter set for this backend request

      Implementation notes for use of parameter sets with resolve=LAZY:

      • A param argument remains associated and any changes to the associated parameter set affect the sharding decision once the director resolves to an actual backend.

      • If other parameter arguments are also given, they have preference and are kept even if the parameter set given by the param argument is subsequently changed within the same backend request.

      • Each call to func_shard.backend overrides any previous call.

VOID xshard.debug(INT)

intentionally undocumented

new xshard_param = directors.shard_param()

Create a shard parameter set.

A parameter set allows for re-use of func_shard.backend arguments across many shard director instances and simplifies advanced use cases (e.g. shard director with custom parameters layered below other directors).

Parameter sets have two scopes:

  • per-VCL scope defined in vcl_init{}

  • per backend request scope

The per-VCL scope defines defaults for the per backend scope. Any changes to a parameter set in backend context and in vcl_pipe {} only affect the respective backend request.

Parameter sets can not be used in client context except for vcl_pipe {}.

VOID xshard_param.clear()

Reset the parameter set to default values as documented for func_shard.backend.

  • in vcl_init{}, resets the parameter set default for this VCL in

  • backend context and in vcl_pipe {}, resets the parameter set for this backend request to the VCL defaults

This method may not be used in client context other than vcl_pipe {}.

shard_param.set(…)

VOID xshard_param.set(
      [ENUM {HASH, URL, KEY, BLOB} by],
      [INT key],
      [BLOB key_blob],
      [INT alt],
      [REAL warmup],
      [BOOL rampup],
      [ENUM {CHOSEN, IGNORE, ALL} healthy]
)

Change the given parameters of a parameter set as documented for func_shard.backend.

  • in vcl_init{}, changes the parameter set default for this VCL

  • in backend context and in vcl_pipe {}, changes the parameter set for this backend request, keeping the defaults set for this VCL for unspecified arguments.

This method may not be used in client context other than vcl_pipe {}.

STRING xshard_param.get_by()

Get a string representation of the by enum argument which denotes how a shard director using this parameter object would derive the shard key. See func_shard.backend.

INT xshard_param.get_key()

Get the key which a shard director using this parameter object would use. See func_shard.backend.

INT xshard_param.get_alt()

Get the alt parameter which a shard director using this parameter object would use. See func_shard.backend.

REAL xshard_param.get_warmup()

Get the warmup parameter which a shard director using this parameter object would use. See func_shard.backend.

BOOL xshard_param.get_rampup()

Get the rampup parameter which a shard director using this parameter object would use. See func_shard.backend.

STRING xshard_param.get_healthy()

Get a string representation of the healthy enum argument which a shard director using this parameter object would use. See func_shard.backend.

BLOB xshard_param.use()

This method may only be used in backend context and in vcl_pipe {}.

For use with the param argument of func_shard.backend to associate this shard parameter set with a shard director.

ACKNOWLEDGEMENTS

Development of a previous version of the shard director was partly sponsored by Deutsche Telekom AG - Products & Innovation.

Development of a previous version of the shard director was partly sponsored by BILD GmbH & Co KG.

COPYRIGHT

This document is licensed under the same licence as Varnish
itself. See LICENCE for details.

Copyright (c) 2013-2015 Varnish Software AS
Copyright 2009-2018 UPLEX - Nils Goroll Systemoptimierung
All rights reserved.

Authors: Poul-Henning Kamp <phk@FreeBSD.org>
         Julian Wiesener <jw@uplex.de>
         Nils Goroll <slink@uplex.de>
         Geoffrey Simmons <geoff@uplex.de>

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.

vmod_vtc

Utility module for varnishtest

Manual section:

3

SYNOPSIS

import vtc [from "path"] ;

VOID barrier_sync(STRING addr, DURATION timeout)

BACKEND no_backend()

STEVEDORE no_stevedore()

IP no_ip()

VOID panic(STRING)

VOID sleep(DURATION)

VOID workspace_alloc(ENUM, INT size)

BYTES workspace_reserve(ENUM, INT size)

INT workspace_free(ENUM)

VOID workspace_snapshot(ENUM)

VOID workspace_reset(ENUM)

BOOL workspace_overflowed(ENUM)

VOID workspace_overflow(ENUM)

INT typesize(STRING)

CONTENTS

DESCRIPTION

The goal for this VMOD is to provide VCL users and VMOD authors means to test corner cases or reach certain conditions with varnishtest.

VOID barrier_sync(STRING addr, DURATION timeout=0)

When writing test cases, the most common pattern is to start a mock server instance, a Varnish instance, and spin up a mock client. Those entities run asynchronously, and others exist like background processes (process) or log readers (logexpect). While you can synchronize with individual entities and wait for their completion, you must use a barrier if you need to synchronize two or more entities, or wait until a certain point instead of completion.

Not only is it possible to synchronize between test entities, with the barrier_sync function you can even synchronize VCL code:

sub vcl_recv {
    # wait for some barrier b1 to complete
    vtc.barrier_sync("${b1_sock}");
}

If the function fails to synchronize with the barrier for some reason, or if it reaches the optional timeout, it fails the VCL transaction.

MISCELLANEOUS

BACKEND no_backend()

Fails at backend selection.

STEVEDORE no_stevedore()

Fails at storage selection.

IP no_ip()

Returns a null IP address, not even a bogo_ip.

VOID panic(STRING)

It can be useful to crash the child process in order to test the robustness of a VMOD.

VOID sleep(DURATION)

Block the current worker thread.

WORKSPACES

It can be useful to put a workspace in a given state when testing corner cases like resource exhaustion for a transaction, especially for VMOD development. All functions available allow to pick which workspace you need to tamper with, available values are client, backend, session and thread.

VOID workspace_alloc(ENUM, INT size)

VOID workspace_alloc(
   ENUM {client, backend, session, thread},
   INT size
)

Allocate and zero out memory from a workspace. A negative size will allocate as much as needed to leave that many bytes free. The actual allocation size may be higher to comply with memory alignment requirements of the CPU architecture. A failed allocation fails the transaction.

BYTES workspace_reserve(ENUM, INT size)

BYTES workspace_reserve(
   ENUM {client, backend, session, thread},
   INT size
)

Attempt to reserve size bytes and release the reservation right away. Return the size of the reservation.

See vtc.workspace_alloc() for semantics of the size argument.

INT workspace_free(ENUM {client, backend, session, thread})

Find how much unallocated space there is left in a workspace.

VOID workspace_snapshot(ENUM)

VOID workspace_snapshot(ENUM {client, backend, session, thread})

Snapshot a workspace. Only one snapshot may be active at a time.

VOID workspace_reset(ENUM)

VOID workspace_reset(ENUM {client, backend, session, thread})

Reset to the previous snapshot of a workspace, it must be the same workspace too.

BOOL workspace_overflowed(ENUM)

BOOL workspace_overflowed(ENUM {client, backend, session, thread})

Find whether the workspace overflow mark is set or not.

VOID workspace_overflow(ENUM)

VOID workspace_overflow(ENUM {client, backend, session, thread})

Mark a workspace as overflowed.

INT typesize(STRING)

Returns the size in bytes of a collection of C-datatypes:

  • 'p': pointer

  • 'i': int

  • 'd': double

  • 'f': float

  • 'l': long

  • 's': short

  • 'z': size_t

  • 'o': off_t

  • 'j': intmax_t

This can be useful for VMOD authors in conjunction with workspace operations.

SEE ALSO

COPYRIGHT

Copyright (c) 2017 Varnish Software AS
All rights reserved.

Author: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.

vmod_purge

Varnish Purge Module

Manual section:

3

SYNOPSIS

import purge [from "path"] ;

INT hard()

INT soft(DURATION ttl, DURATION grace, DURATION keep)

CONTENTS

DESCRIPTION

vmod_purge contains functions that offer a finer-grained control than the purge transition in vcl_recv. The functions can only be called from vcl_hit or vcl_miss and they should in general be used in both to ensure that all variants of a same object are taken care of.

EXAMPLE

sub vcl_recv {
    if (req.method == "PURGE") {
        if (client.ip !~ purge_acl) {
            return (synth(405));
        }
        return (hash);
    }
}

sub my_purge {
    set req.http.purged = purge.hard();
    if (req.http.purged == "0") {
        return (synth(404));
    }
    else {
        return (synth(200));
    }
}

sub vcl_hit {
    if (req.method == "PURGE") {
        call my_purge;
    }
}

sub vcl_miss {
    if (req.method == "PURGE") {
        call my_purge;
    }
}

sub vcl_synth {
    if (req.method == "PURGE") {
        if (req.http.purged) {
            set resp.http.purged = req.http.purged;
        }
        return (deliver);
    }
}

INT hard()

Description

This is equivalent to return(purge) but explicitly called from vcl_hit and vcl_miss. It returns the number of purged objects.

Example

set req.http.purged = purge.hard();

INT soft(DURATION ttl, DURATION grace, DURATION keep)

INT soft(DURATION ttl=0, DURATION grace=-1, DURATION keep=-1)
Description

Sets the TTL, grace and keep. By default, TTL is set to 0 with grace and keep periods left untouched. Setting a negative value for grace or keep periods leaves them untouched. Setting all three parameters to 0 is equivalent to a hard purge. It can only be called from vcl_hit or vcl_miss. It returns the number of soft-purged objects.

SEE ALSO

COPYRIGHT

Copyright (c) 2017 Varnish Software AS
All rights reserved.

Author: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.

vmod_blob

Utilities for the VCL blob type, encoding and decoding

Manual section:

3

SYNOPSIS

import blob [from "path"] ;

BLOB decode(ENUM decoding, INT length, STRING encoded)

STRING encode(ENUM encoding, ENUM case, BLOB blob)

STRING transcode(ENUM decoding, ENUM encoding, ENUM case, INT length, STRING encoded)

BOOL same(BLOB, BLOB)

BOOL equal(BLOB, BLOB)

INT length(BLOB)

BLOB sub(BLOB, BYTES length, BYTES offset)

new xblob = blob.blob(ENUM decoding, STRING encoded)

   BLOB xblob.get()

   STRING xblob.encode(ENUM encoding, ENUM case)

CONTENTS

DESCRIPTION

This VMOD provides utility functions and an object for the VCL data type BLOB, which may contain arbitrary data of any length.

Examples:

sub vcl_init {
    # Create blob objects from encodings such as base64 or hex.
    new myblob   = blob.blob(BASE64, "Zm9vYmFy");
    new yourblob = blob.blob(encoded="666F6F", decoding=HEX);
}

sub vcl_deliver {
    # The .get() method retrieves the BLOB from an object.
    set resp.http.MyBlob-As-Hex
        = blob.encode(blob=myblob.get(), encoding=HEX);

    # The .encode() method efficiently retrieves an encoding.
    set resp.http.YourBlob-As-Base64 = yourblob.encode(BASE64);

    # decode() and encode() functions convert blobs to text and
    # vice versa at runtime.
    set resp.http.Base64-Encoded
        = blob.encode(BASE64,
                      blob=blob.decode(HEX,
                                       encoded=req.http.Hex-Encoded));
}

sub vcl_recv {
    # transcode() converts from one encoding to another.
    # case=UPPER specifies upper-case hex digits A-F.
    set req.http.Hex-Encoded
        = blob.transcode(decoding=BASE64, encoding=HEX,
                         case=UPPER, encoded="YmF6");

    # transcode() from URL to IDENTITY effects a URL decode.
    set req.url = blob.transcode(encoded=req.url, decoding=URL);

    # transcode() from IDENTITY to URL effects a URL encode.
    set req.http.url_urlcoded
        = blob.transcode(encoded=req.url, encoding=URL);
}

ENCODING SCHEMES

Binary-to-text encoding schemes are specified by ENUMs in the VMOD’s constructor, methods and functions. Decodings convert a (possibly concatenated) string into a blob, while encodings convert a blob into a string.

ENUM values for an encoding scheme can be one of:

  • IDENTITY

  • BASE64

  • BASE64URL

  • BASE64URLNOPAD

  • HEX

  • URL

Empty strings are decoded into a “null blob” (of length 0), and conversely a null blob is encoded as the empty string.

For encodings with HEX or URL, you may also specify a case ENUM with one of the values LOWER, UPPER or DEFAULT to produce a string with lower- or uppercase hex digits (in [a-f] or [A-F]). The default value for case is DEFAULT, which for HEX and URL means the same as LOWER.

The case ENUM is not relevant for decodings; HEX or URL strings to be decoded as BLOBs may have hex digits in either case, or in mixed case.

The case ENUM MUST be set to DEFAULT for the other encodings (BASE64* and IDENTITY). You cannot, for example, produce an uppercase string by using the IDENTITY scheme with case=UPPER. To change the case of a string, use the toupper or tolower functions from vmod_std.

IDENTITY

The simplest encoding converts between the BLOB and STRING data types, leaving the contents byte-identical.

Note that a BLOB may contain a null byte at any position before its end; if such a BLOB is decoded with IDENTITY, the resulting STRING will have a null byte at that position. Since VCL strings, like C strings, are represented with a terminating null byte, the string will be truncated, appearing to contain less data than the original blob. For example:

# Decode from the hex encoding for "foo\0bar".
# The header will be seen as "foo".
set resp.http.Trunced-Foo1
    = blob.encode(IDENTITY, blob=blob.decode(HEX,
                                             encoded="666f6f00626172"));

IDENTITY is the default encoding and decoding. So the above can also be written as:

# Decode from the hex encoding for "foo\0bar".
# The header will be seen as "foo".
set resp.http.Trunced-Foo2
  = blob.encode(blob=blob.decode(HEX, encoded="666f6f00626172"));

The case ENUM MUST be set to DEFAULT for IDENTITY encodings.

BASE64*

The base64 encoding schemes use 4 characters to encode 3 bytes. There are no newlines or maximal line lengths – whitespace is not permitted.

The BASE64 encoding uses the alphanumeric characters, + and /; and encoded strings are padded with the = character so that their length is always a multiple of four.

The BASE64URL encoding also uses the alphanumeric characters, but - and _ instead of + and /, so that an encoded string can be used safely in a URL. This scheme also uses the padding character =.

The BASE64URLNOPAD encoding uses the same alphabet as BASE6URL, but leaves out the padding. Thus the length of an encoding with this scheme is not necessarily a multiple of four.

The case ENUM MUST be set to DEFAULT for for all of the BASE64* encodings.

HEX

The HEX encoding scheme converts hex strings into blobs and vice versa. For encodings, you may use the case ENUM to specify upper- or lowercase hex digits A through f (default DEFAULT, which means the same as LOWER). A prefix such as 0x is not used for an encoding and is illegal for a decoding.

If a hex string to be decoded has an odd number of digits, it is decoded as if a 0 is prepended to it; that is, the first digit is interpreted as representing the least significant nibble of the first byte. For example:

# The concatenated string is "abcdef0", and is decoded as "0abcdef0".
set resp.http.First = "abc";
set resp.http.Second = "def0";
set resp.http.Hex-Decoded
    = blob.encode(HEX, blob=blob.decode(HEX,
                       encoded=resp.http.First + resp.http.Second));

URL

The URL decoding replaces any %<2-hex-digits> substrings with the binary value of the hexadecimal number after the % sign.

The URL encoding implements “percent encoding” as per RFC3986. The case ENUM determines the case of the hex digits, but does not affect alphabetic characters that are not percent-encoded.

BLOB decode(ENUM decoding, INT length, STRING encoded)

BLOB decode(
   ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} decoding=IDENTITY,
   INT length=0,
   STRING encoded
)

Returns the BLOB derived from the string encoded according to the scheme specified by decoding.

If length > 0, only decode the first length characters of the encoded string. If length <= 0 or greater than the length of the string, then decode the entire string. The default value of length is 0.

decoding defaults to IDENTITY.

Example:

blob.decode(BASE64, encoded="Zm9vYmFyYmF6");

# same with named parameters
blob.decode(encoded="Zm9vYmFyYmF6", decoding=BASE64);

# convert string to blob
blob.decode(encoded="foo");

STRING encode(ENUM encoding, ENUM case, BLOB blob)

STRING encode(
   ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} encoding=IDENTITY,
   ENUM {LOWER, UPPER, DEFAULT} case=DEFAULT,
   BLOB blob
)

Returns a string representation of the BLOB blob as specified by encoding. case determines the case of hex digits for the HEX and URL encodings, and is ignored for the other encodings.

encoding defaults to IDENTITY, and case defaults to DEFAULT. DEFAULT is interpreted as LOWER for the HEX and URL encodings, and is the required value for the other encodings.

Example:

set resp.http.encode1
    = blob.encode(HEX,
                  blob=blob.decode(BASE64, encoded="Zm9vYmFyYmF6"));

# same with named parameters
set resp.http.encode2
    = blob.encode(blob=blob.decode(encoded="Zm9vYmFyYmF6",
                                           decoding=BASE64),
                      encoding=HEX);

# convert blob to string
set resp.http.encode3
    = blob.encode(blob=blob.decode(encoded="foo"));

transcode(…)

STRING transcode(
   ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} decoding=IDENTITY,
   ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} encoding=IDENTITY,
   ENUM {LOWER, UPPER, DEFAULT} case=DEFAULT,
   INT length=0,
   STRING encoded
)

Translates from one encoding to another, by first decoding the string encoded according to the scheme decoding, and then returning the encoding of the resulting blob according to the scheme encoding. case determines the case of hex digits for the HEX and URL encodings, and is ignored for other encodings.

As with decode(): If length > 0, only decode the first length characters of the encoded string, otherwise decode the entire string. The default value of length is 0.

decoding and encoding default to IDENTITY, and case defaults to DEFAULT. DEFAULT is interpreted as LOWER for the HEX and URL encodings, and is the required value for the other encodings.

Example:

set resp.http.Hex2Base64-1
     = blob.transcode(HEX, BASE64, encoded="666f6f");

 # same with named parameters
 set resp.http.Hex2Base64-2
    = blob.transcode(encoded="666f6f",
                          encoding=BASE64, decoding=HEX);

 # URL decode -- recall that IDENTITY is the default encoding.
 set resp.http.urldecoded
    = blob.transcode(encoded="foo%20bar", decoding=URL);

 # URL encode
 set resp.http.urlencoded
     = blob.transcode(encoded="foo bar", encoding=URL);

BOOL same(BLOB, BLOB)

Returns true if and only if the two BLOB arguments are the same object, i.e. they specify exactly the same region of memory, or both are empty.

If the BLOBs are both empty (length is 0 and/or the internal pointer is NULL), then same() returns true. If any non-empty BLOB is compared to an empty BLOB, then same() returns false.

BOOL equal(BLOB, BLOB)

Returns true if and only if the two BLOB arguments have equal contents (possibly in different memory regions).

As with same(): If the BLOBs are both empty, then equal() returns true. If any non-empty BLOB is compared to an empty BLOB, then equal() returns false.

INT length(BLOB)

Returns the length of the BLOB.

BLOB sub(BLOB, BYTES length, BYTES offset=0)

Returns a new BLOB formed from length bytes of the BLOB argument starting at offset bytes from the start of its memory region. The default value of offset is 0B.

sub() fails and returns NULL if the BLOB argument is empty, or if offset + length requires more bytes than are available in the BLOB.

new xblob = blob.blob(ENUM decoding, STRING encoded)

new xblob = blob.blob(
   ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} decoding=IDENTITY,
   STRING encoded
)

Creates an object that contains the BLOB derived from the string encoded according to the scheme decoding.

Example:

new theblob1 = blob.blob(BASE64, encoded="YmxvYg==");

# same with named arguments
new theblob2 = blob.blob(encoded="YmxvYg==", decoding=BASE64);

# string as a blob
new stringblob = blob.blob(encoded="bazz");

BLOB xblob.get()

Returns the BLOB created by the constructor.

Example:

set resp.http.The-Blob1 =
    blob.encode(blob=theblob1.get());

set resp.http.The-Blob2 =
    blob.encode(blob=theblob2.get());

set resp.http.The-Stringblob =
    blob.encode(blob=stringblob.get());

STRING xblob.encode(ENUM encoding, ENUM case)

STRING xblob.encode(
      ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX, URL} encoding=IDENTITY,
      ENUM {LOWER, UPPER, DEFAULT} case=DEFAULT
)

Returns an encoding of BLOB created by the constructor, according to the scheme encoding. case determines the case of hex digits for the HEX and URL encodings, and MUST be set to DEFAULT for the other encodings.

Example:

# blob as text
set resp.http.The-Blob = theblob1.encode();

# blob as base64
set resp.http.The-Blob-b64 = theblob1.encode(BASE64);

For any blob object, encoding ENC and case CASE, encodings via the .encode() method and the encode() function are equal:

# Always true:
blob.encode(ENC, CASE, blob.get()) == blob.encode(ENC, CASE)

But the object method is more efficient – the encoding is computed once and cached (with allocation in heap memory), and the cached encoding is retrieved on every subsequent call. The encode() function computes the encoding on every call, allocating space for the string in Varnish workspaces.

So if the data in a BLOB are fixed at VCL initialization time, so that its encodings will always be the same, it is better to create a blob object. The VMOD’s functions should be used for data that are not known until runtime.

ERRORS

The encoders, decoders and sub() may fail if there is insufficient space to create the new blob or string. Decoders may also fail if the encoded string is an illegal format for the decoding scheme. Encoders will fail for the IDENTITY and BASE64* encoding schemes if the case ENUM is not set to DEFAULT.

If any of the VMOD’s methods, functions or constructor fail, then VCL failure is invoked, just as if return(fail) had been called in the VCL source. This means that:

  • If the blob object constructor fails, or if any methods or functions fail during vcl_init, then the VCL program will fail to load, and the VCC compiler will emit an error message.

  • If a method or function fails in any other VCL subroutine besides vcl_synth, then control is directed to vcl_synth. The response status is set to 503 with the reason string "VCL failed", and an error message will be written to the Varnish log using the tag VCL_Error.

  • If the failure occurs during vcl_synth, then vcl_synth is aborted. The response line "503 VCL failed" is returned, and the VCL_Error message is written to the log.

LIMITATIONS

The VMOD allocates memory in various ways for new blobs and strings. The blob object and its methods allocate memory from the heap, and hence they are only limited by available virtual memory.

The encode(), decode() and transcode() functions allocate Varnish workspace, as does sub() for the newly created BLOB. If these functions are failing, as indicated by “out of space” messages in the Varnish log (with the VCL_Error tag), then you will need to increase the varnishd parameters workspace_client and/or workspace_backend.

The transcode() function also allocates space on the stack for a temporary BLOB. If this function causes stack overflow, you may need to increase the varnishd parameter thread_pool_stack.

SEE ALSO

COPYRIGHT

This document is licensed under the same conditions as Varnish itself.
See LICENSE for details.

Authors: Nils Goroll <nils.goroll@uplex.de>
         Geoffrey Simmons <geoffrey.simmons@uplex.de>

vmod_unix

Utilities for Unix domain sockets

Manual section:

3

SYNOPSIS

import unix [from "path"] ;

STRING user()

STRING group()

INT uid()

INT gid()

CONTENTS

DESCRIPTION

This VMOD provides information about the credentials of the peer process (user and group of the process owner) that is connected to a Varnish listener via a Unix domain socket, if the platform supports it.

Examples:

import unix;

sub vcl_recv {
      # Return "403 Forbidden" if the connected peer is
      # not running as the user "trusteduser".
      if (unix.user() != "trusteduser") {
              return( synth(403) );
      }

      # Require the connected peer to run in the group
      # "trustedgroup".
      if (unix.group() != "trustedgroup") {
              return( synth(403) );
      }

      # Require the connected peer to run under a specific numeric
      # user id.
      if (unix.uid() != 4711) {
              return( synth(403) );
      }

      # Require the connected peer to run under a numeric group id.
      if (unix.gid() != 815) {
              return( synth(403) );
      }
}

Obtaining the peer credentials is possible on a platform that supports one of the following:

  • getpeereid(3) (such as FreeBSD and other BSD-derived systems)

  • the socket option SO_PEERCRED for getsockopt(2) (Linux)

  • getpeerucred(3C) (SunOS and descendants)

On SunOS and friends, the PRIV_PROC_INFO privilege set is added to the Varnish child process while the VMOD is loaded, see setppriv(2).

On most platforms, the value returned is the effective user or group that was valid when the peer process initiated the connection.

STRING user()

Return the user name of the peer process owner.

STRING group()

Return the group name of the peer process owner.

INT uid()

Return the numeric user id of the peer process owner.

INT gid()

Return the numeric group id of the peer process owner.

ERRORS

All functions in this VMOD are subject to the following constraints:

  • None of them may be called in vcl_init or vcl_fini. If one of them is called in vcl_init, then the VCL program will fail to load, with an error message from the VMOD.

  • If called on a platform that is not supported, then VCL failure is invoked. An error message is written to the log (with the VCL_Error tag), and for all VCL subroutines except for vcl_synth, control is directed immediately to vcl_synth, with the response status set to 503 and the reason string set to “VCL failed”.

    If the failure occurs during vcl_synth, then vcl_synth is aborted, and the the response line “503 VCL failed” is sent.

  • If the current listener is not a Unix domain socket, or if the attempt to read credentials fails, then a VCL_Error message is written to the log. The STRING functions (vmod_user and vmod_group) return NULL, while the INT functions (vmod_uid and vmod_gid) return -1.

SEE ALSO

  • varnishd

  • VCL

  • getpeereid(3)

  • getsockopt(2)

  • getpeerucred(3C)

  • setppriv(2)

COPYRIGHT

This document is licensed under the same conditions as Varnish itself.
See LICENSE for details.

Authors: Geoffrey Simmons <geoffrey.simmons@uplex.de>

vmod_proxy

Varnish Module to extract TLV attributes from PROXYv2

Manual section:

3

SYNOPSIS

import proxy [from "path"] ;

STRING alpn()

STRING authority()

BOOL is_ssl()

BOOL client_has_cert_sess()

BOOL client_has_cert_conn()

INT ssl_verify_result()

STRING ssl_version()

STRING client_cert_cn()

STRING ssl_cipher()

STRING cert_sign()

STRING cert_key()

CONTENTS

DESCRIPTION

vmod_proxy contains functions to extract proxy-protocol-v2 TLV attributes as described in https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt.

STRING alpn()

Description

Extract ALPN attribute.

Example

set req.http.alpn = proxy.alpn();

STRING authority()

Description

Extract authority attribute. This corresponds to SNI from a TLS connection.

Example

set req.http.authority = proxy.authority();

BOOL is_ssl()

Description

Report if proxy-protocol-v2 has SSL TLV.

Example
if (proxy.is_ssl()) {
set req.http.ssl-version = proxy.ssl_version();
}

BOOL client_has_cert_sess()

Description

Report if the client provided a certificate at least once over the TLS session this connection belongs to.

BOOL client_has_cert_conn()

Description

Report if the client provided a certificate over the current connection.

INT ssl_verify_result()

Description

Report the SSL_get_verify_result from a TLS session. It only matters if client_has_cert_sess() is true. Per default, value is set to 0 (X509_V_OK).

Example
if (proxy.client_has_cert_sess() && proxy.ssl_verify_result() == 0) {
set req.http.ssl-verify = “ok”;
}

STRING ssl_version()

Description

Extract SSL version attribute.

Example

set req.http.ssl-version = proxy.ssl_version();

STRING client_cert_cn()

Description

Extract the common name attribute of the client certificate’s.

Example

set req.http.cert-cn = proxy.client_cert_cn();

STRING ssl_cipher()

Description

Extract the SSL cipher attribute.

Example

set req.http.ssl-cipher = proxy.ssl_cipher();

STRING cert_sign()

Description

Extract the certificate signature algorithm attribute.

Example

set req.http.cert-sign = proxy.cert_sign();

STRING cert_key()

Description

Extract the certificate key algorithm attribute.

Example

set req.http.cert-key = proxy.cert_key();

SEE ALSO

COPYRIGHT

Copyright (c) 2018 GANDI SAS
All rights reserved.

Author: Emmanuel Hocdet <manu@gandi.net>

Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
   notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
   notice, this list of conditions and the following disclaimer in the
   documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.