From phk at FreeBSD.org  Mon Mar  1 08:40:09 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon,  1 Mar 2021 08:40:09 +0000 (UTC)
Subject: [master] 4d483a2cb Add SPDX identifiers
Message-ID: <20210301084009.81B32101C43@lists.varnish-cache.org>


commit 4d483a2cb922b224469a24165b20856bedf792b2
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 1 08:17:58 2021 +0000

    Add SPDX identifiers

diff --git a/vmod/vmod_blob.vcc b/vmod/vmod_blob.vcc
index e9f3a5a4c..514f6028e 100644
--- a/vmod/vmod_blob.vcc
+++ b/vmod/vmod_blob.vcc
@@ -2,6 +2,8 @@
 # This document is licensed under the same conditions as Varnish itself.
 # See LICENSE for details.
 #
+# SPDX-License-Identifier: BSD-2-Clause
+#
 # Authors: Nils Goroll <nils.goroll at uplex.de>
 #          Geoffrey Simmons <geoffrey.simmons at uplex.de>
 #
diff --git a/vmod/vmod_cookie.vcc b/vmod/vmod_cookie.vcc
index 29888f457..ca13d0c0e 100644
--- a/vmod/vmod_cookie.vcc
+++ b/vmod/vmod_cookie.vcc
@@ -1,3 +1,9 @@
+#-
+# This document is licensed under the same conditions as Varnish itself.
+# See LICENSE for details.
+#
+# SPDX-License-Identifier: BSD-2-Clause
+
 $Module cookie 3 "Varnish Cookie Module"
 
 DESCRIPTION
diff --git a/vmod/vmod_directors.vcc b/vmod/vmod_directors.vcc
index 1ea5ff79d..70474cdc0 100644
--- a/vmod/vmod_directors.vcc
+++ b/vmod/vmod_directors.vcc
@@ -2,6 +2,8 @@
 # This document is licensed under the same licence as Varnish
 # itself. See LICENCE for details.
 #
+# SPDX-License-Identifier: BSD-2-Clause
+#
 # Copyright (c) 2013-2015 Varnish Software AS
 # Copyright 2009-2020 UPLEX - Nils Goroll Systemoptimierung
 # All rights reserved.
diff --git a/vmod/vmod_unix.vcc b/vmod/vmod_unix.vcc
index c7f5bc167..a7e6de913 100644
--- a/vmod/vmod_unix.vcc
+++ b/vmod/vmod_unix.vcc
@@ -2,6 +2,8 @@
 # This document is licensed under the same conditions as Varnish itself.
 # See LICENSE for details.
 #
+# SPDX-License-Identifier: BSD-2-Clause
+#
 # Authors: Geoffrey Simmons <geoffrey.simmons at uplex.de>
 #
 

From phk at FreeBSD.org  Mon Mar  1 13:36:08 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon,  1 Mar 2021 13:36:08 +0000 (UTC)
Subject: [master] c749a196a Dont combine VSB_QUOTE_JSON and VSB_QUOTE_CSTR,
 just emit the double quotes ourselves.
Message-ID: <20210301133608.DB260109141@lists.varnish-cache.org>


commit c749a196a9a7c626b67fc517863edeee4d244984
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 1 13:02:38 2021 +0000

    Dont combine VSB_QUOTE_JSON and VSB_QUOTE_CSTR, just emit the
    double quotes ourselves.

diff --git a/bin/varnishd/mgt/mgt_param_tweak.c b/bin/varnishd/mgt/mgt_param_tweak.c
index b6d97797b..939b2a150 100644
--- a/bin/varnishd/mgt/mgt_param_tweak.c
+++ b/bin/varnishd/mgt/mgt_param_tweak.c
@@ -377,7 +377,9 @@ tweak_string(struct vsb *vsb, const struct parspec *par, const char *arg)
 	if (arg == NULL) {
 		VSB_quote(vsb, *p, -1, 0);
 	} else if (arg == JSON_FMT) {
-		VSB_quote(vsb, *p, -1, VSB_QUOTE_JSON|VSB_QUOTE_CSTR);
+		VSB_putc(vsb, '"');
+		VSB_quote(vsb, *p, -1, VSB_QUOTE_JSON);
+		VSB_putc(vsb, '"');
 	} else {
 		REPLACE(*p, arg);
 	}

From phk at FreeBSD.org  Mon Mar  1 13:36:08 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon,  1 Mar 2021 13:36:08 +0000 (UTC)
Subject: [master] 1c4d04f68 Dont pass empty strings to VSB_quote()
Message-ID: <20210301133608.BDA8710913E@lists.varnish-cache.org>


commit 1c4d04f68c7d5fe817ff327a8ffefcf7a60e3a02
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 1 09:15:06 2021 +0000

    Dont pass empty strings to VSB_quote()

diff --git a/bin/varnishstat/varnishstat_help_gen.c b/bin/varnishstat/varnishstat_help_gen.c
index 1412ea992..bef1ebb6e 100644
--- a/bin/varnishstat/varnishstat_help_gen.c
+++ b/bin/varnishstat/varnishstat_help_gen.c
@@ -69,7 +69,7 @@ main(void)
 	do {
 		p = n + 1;
 		n = strchr(p, '\n');
-		if (n != NULL) {
+		if (n != NULL && n > p) {
 			VSB_putc(vsb, '\t');
 			VSB_quote(vsb, p, (int)(n - p), VSB_QUOTE_CSTR);
 			VSB_cat(vsb, ",\n");

From phk at FreeBSD.org  Mon Mar  1 13:36:09 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon,  1 Mar 2021 13:36:09 +0000 (UTC)
Subject: [master] 0799e0cc0 Sort out VSB quoting,
 add testcases and asserts on usage.
Message-ID: <20210301133609.340EC109144@lists.varnish-cache.org>


commit 0799e0cc0669d42c2f79dd54db9ee5414c6bddae
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 1 13:03:14 2021 +0000

    Sort out VSB quoting, add testcases and asserts on usage.

diff --git a/include/vsb.h b/include/vsb.h
index f67220e96..7a32360e3 100644
--- a/include/vsb.h
+++ b/include/vsb.h
@@ -79,12 +79,52 @@ ssize_t		 VSB_len(const struct vsb *);
 void		 VSB_delete(struct vsb *) v_deprecated_;
 void		 VSB_fini(struct vsb *);
 void		 VSB_destroy(struct vsb **);
-#define VSB_QUOTE_NONL		1
+
+/*
+ * VSB_quote[_pfx] has four major modes, and two modifiers
+ */
+
+#define VSB_QUOTE_PLAIN		0
+	/*
+	 * Basic "show me the string" mode
+	 * All output is a single line
+	 */
 #define VSB_QUOTE_JSON		2
+	/*
+	 * Output suitable for inclusion between "..." in JSON
+	 * Uses JSON \u%04x quoting.
+	 * Anything above 0x7e had better be UTF-8
+	 */
 #define VSB_QUOTE_HEX		4
+	/*
+	 * Hex dump, single line.
+	 * All zero data is compressed to "0x0...0"
+	 */
 #define VSB_QUOTE_CSTR		8
+	/*
+	 * C lanuage source code literal string(s)
+	 * Breaks strings at \n (expecting string concatenation)
+	 */
 #define VSB_QUOTE_UNSAFE	16
+	/*
+	 * For general display applications
+	 * " and \ are not quoted
+	 * Splits output to new line at '\n'
+	 * Implies VSB_QUOTE_NONL
+	 */
+
+#define VSB_QUOTE_NONL		1
+	/*
+	 * If the output does not end in \n, append \n
+	 * Can be combined with all other modes.
+	 */
+
 #define VSB_QUOTE_ESCHEX	32
+	/*
+	 * Use \x%02x instead of \%03o
+	 * Not valid with VSB_QUOTE_JSON and VSB_QUOTE_HEX
+	 */
+
 void		 VSB_quote_pfx(struct vsb *, const char*, const void *,
 		     int len, int how);
 void		 VSB_quote(struct vsb *, const void *, int len, int how);
diff --git a/lib/libvarnish/vsb.c b/lib/libvarnish/vsb.c
index 8857bd26a..5cc76601a 100644
--- a/lib/libvarnish/vsb.c
+++ b/lib/libvarnish/vsb.c
@@ -555,73 +555,102 @@ VSB_destroy(struct vsb **s)
 /*
  * Quote a string
  */
+
+static void
+vsb_quote_hex(struct vsb *s, const uint8_t *u, size_t len)
+{
+	const uint8_t *w;
+
+	VSB_cat(s, "0x");
+	for (w = u; w < u + len; w++)
+		if (*w != 0x00)
+			break;
+	if (w == u + len && len > 4) {
+		VSB_cat(s, "0...0");
+	} else {
+		for (w = u; w < u + len; w++)
+			VSB_printf(s, "%02x", *w);
+	}
+}
+
 void
 VSB_quote_pfx(struct vsb *s, const char *pfx, const void *v, int len, int how)
 {
-	const char *p;
-	const char *q;
+	const uint8_t *p = v;
+	const uint8_t *q;
 	int quote = 0;
-	int nl = 0;
-	const unsigned char *u, *w;
+	int nl;
+
+	nl = how &
+	    (VSB_QUOTE_JSON|VSB_QUOTE_HEX|VSB_QUOTE_CSTR|VSB_QUOTE_UNSAFE);
+	AZ(nl & (nl - 1)); // Only one bit can be set
+
+	if (how & VSB_QUOTE_ESCHEX)
+		AZ(how & (VSB_QUOTE_JSON|VSB_QUOTE_HEX));
+
+	if (how & VSB_QUOTE_UNSAFE)
+		how |= VSB_QUOTE_NONL;
 
-	assert(v != NULL);
+	assert(p != NULL);
 	if (len == -1)
 		len = strlen(v);
 
 	if (len == 0 && (how & VSB_QUOTE_CSTR)) {
 		VSB_printf(s, "%s\"\"", pfx);
-		return;
-	} else if (len == 0)
+		if ((how & VSB_QUOTE_NONL))
+			VSB_putc(s, '\n');
+	}
+
+	if (len == 0)
 		return;
 
 	VSB_cat(s, pfx);
 
 	if (how & VSB_QUOTE_HEX) {
-		u = v;
-		for (w = u; w < u + len; w++)
-			if (*w != 0x00)
-				break;
-		VSB_cat(s, "0x");
-		if (w == u + len && len > 4) {
-			VSB_cat(s, "0...0");
-		} else {
-			for (w = u; w < u + len; w++)
-				VSB_printf(s, "%02x", *w);
-		}
+		vsb_quote_hex(s, v, len);
+		if (how & VSB_QUOTE_NONL)
+			VSB_putc(s, '\n');
 		return;
 	}
-	p = v;
+
+	if (how & VSB_QUOTE_CSTR)
+		VSB_putc(s, '"');
 
 	for (q = p; q < p + len; q++) {
-		if (!isgraph(*q) || *q == '"' || *q == '\\') {
+		if (
+		    *q < 0x20 ||
+		    *q == '"' ||
+		    *q == '\\' ||
+		    (*q == '?' && (how & VSB_QUOTE_CSTR)) ||
+		    (*q > 0x7e && !(how & VSB_QUOTE_JSON))
+		) {
 			quote++;
 			break;
 		}
 	}
-	if (!quote && !(how & (VSB_QUOTE_JSON|VSB_QUOTE_CSTR))) {
-		(void)VSB_bcat(s, p, len);
-		if ((how & (VSB_QUOTE_UNSAFE|VSB_QUOTE_NONL)) &&
+
+	if (!quote) {
+		VSB_bcat(s, p, len);
+		if ((how & VSB_QUOTE_NONL) &&
 		    p[len-1] != '\n')
 			(void)VSB_putc(s, '\n');
+		if (how & VSB_QUOTE_CSTR)
+			VSB_putc(s, '"');
 		return;
 	}
 
-	if (how & VSB_QUOTE_CSTR)
-		(void)VSB_putc(s, '"');
-
+	nl = 0;
 	for (q = p; q < p + len; q++) {
 		if (nl)
 			VSB_cat(s, pfx);
 		nl = 0;
 		switch (*q) {
 		case '?':
-			if (how & VSB_QUOTE_CSTR)
+			/* Avoid C Trigraph insanity */
+			if (how & VSB_QUOTE_CSTR && !(how & VSB_QUOTE_JSON))
 				(void)VSB_putc(s, '\\');
 			(void)VSB_putc(s, *q);
 			break;
-		case ' ':
-			(void)VSB_putc(s, *q);
-			break;
 		case '\\':
 		case '"':
 			if (!(how & VSB_QUOTE_UNSAFE))
@@ -630,38 +659,43 @@ VSB_quote_pfx(struct vsb *s, const char *pfx, const void *v, int len, int how)
 			break;
 		case '\n':
 			if (how & VSB_QUOTE_CSTR) {
-				(void)VSB_printf(s, "\\n\"\n%s\"", pfx);
-			} else if (how & (VSB_QUOTE_NONL|VSB_QUOTE_UNSAFE)) {
-				(void)VSB_printf(s, "\n");
+				VSB_printf(s, "\\n\"\n%s\"", pfx);
+			} else if (how & VSB_QUOTE_JSON) {
+				VSB_printf(s, "\\n");
+			} else if (how & VSB_QUOTE_NONL) {
+				VSB_putc(s, *q);
 				nl = 1;
 			} else {
-				(void)VSB_printf(s, "\\n");
+				VSB_printf(s, "\\n");
 			}
 			break;
 		case '\r':
-			(void)VSB_cat(s, "\\r");
+			VSB_cat(s, "\\r");
 			break;
 		case '\t':
-			(void)VSB_cat(s, "\\t");
+			VSB_cat(s, "\\t");
 			break;
 		case '\v':
-			(void)VSB_cat(s, "\\v");
+			VSB_cat(s, "\\v");
 			break;
 		default:
-			/* XXX: Implement VSB_QUOTE_JSON */
-			if (isgraph(*q))
-				(void)VSB_putc(s, *q);
+			if (0x20 <= *q && *q <= 0x7e)
+				VSB_putc(s, *q);
+			else if (*q > 0x7e && (how & VSB_QUOTE_JSON))
+				VSB_putc(s, *q);
+			else if (how & VSB_QUOTE_JSON)
+				VSB_printf(s, "\\u%04x", *q);
 			else if (how & VSB_QUOTE_ESCHEX)
-				(void)VSB_printf(s, "\\x%02x", *q & 0xff);
+				VSB_printf(s, "\\x%02x", *q);
 			else
-				(void)VSB_printf(s, "\\%03o", *q & 0xff);
+				VSB_printf(s, "\\%03o", *q);
 			break;
 		}
 	}
 	if (how & VSB_QUOTE_CSTR)
-		(void)VSB_putc(s, '"');
-	if ((how & (VSB_QUOTE_NONL|VSB_QUOTE_UNSAFE)) && !nl)
-		(void)VSB_putc(s, '\n');
+		VSB_putc(s, '"');
+	if ((how & VSB_QUOTE_NONL) && !nl)
+		VSB_putc(s, '\n');
 }
 
 void
diff --git a/lib/libvarnish/vsb_test.c b/lib/libvarnish/vsb_test.c
index 8ce277c65..ef09a14c4 100644
--- a/lib/libvarnish/vsb_test.c
+++ b/lib/libvarnish/vsb_test.c
@@ -10,13 +10,79 @@
 
 struct tc {
 	int		how;
+	int		inlen;
 	const char	*in;
 	const char	*out;
 };
 
 static struct tc tcs[] = {
 	{
-	    0, NULL, NULL
+		VSB_QUOTE_HEX,
+		5, "\x00\n\x7e\x7f\xff",
+		"PFX0x000a7e7fff"
+	},
+	{
+		VSB_QUOTE_HEX,
+		5, "\0\0\0\0\0",
+		"PFX0x0...0"
+	},
+	{
+		VSB_QUOTE_HEX | VSB_QUOTE_NONL,
+		5, "\x00\n\x7e\x7f\xff",
+		"PFX0x000a7e7fff\n"
+	},
+	{
+		VSB_QUOTE_ESCHEX,
+		5, "\x00\n\x7e\x7f\xff",
+		"PFX\\x00\\n~\\x7f\\xff",
+	},
+	{
+		0,
+		5, "\x00\n\x7e\x7f\xff",
+		"PFX\\000\\n~\\177\\377",
+	},
+	{
+		VSB_QUOTE_UNSAFE,
+		5, "\x00\n\x7e\x7f\xff",
+		"PFX\\000\nPFX~\\177\\377\n",
+	},
+	{
+		VSB_QUOTE_UNSAFE,
+		-1, "\n\"\\\t",
+		"PFX\nPFX\"\\\\t\n"
+	},
+	{
+		VSB_QUOTE_CSTR | VSB_QUOTE_ESCHEX,
+		5, "\x00\n\x7e\x7f\xff",
+		"PFX\"\\x00\\n\"\nPFX\"~\\x7f\\xff\"",
+	},
+	{
+		VSB_QUOTE_JSON,
+		5, "\x00\n\x7e\x7f\xff",
+		"PFX\\u0000\\n~\x7f\xff",
+	},
+	{
+		VSB_QUOTE_JSON | VSB_QUOTE_NONL,
+		5, "\x00\n\x7e\x7f\xff",
+		"PFX\\u0000\\n~\x7f\xff\n",
+	},
+	{
+		VSB_QUOTE_CSTR,
+		-1, "",
+		"PFX\"\""
+	},
+	{
+		VSB_QUOTE_CSTR,
+		-1, "?",
+		"PFX\"\\?\""
+	},
+	{
+		VSB_QUOTE_NONL,
+		-1, "\n\t",
+		"PFX\nPFX\\t\n"
+	},
+	{
+		0, -1, NULL, NULL
 	}
 };
 
@@ -26,25 +92,58 @@ main(int argc, char *argv[])
 	int err = 0;
 	struct tc *tc;
 	struct vsb *vsb;
+	struct vsb *vsbo;
 
 	(void)argc;
 	(void)argv;
 	vsb = VSB_new_auto();
 	AN(vsb);
+	vsbo = VSB_new_auto();
+	AN(vsbo);
 
 	for (tc = tcs; tc->in; tc++) {
-		VSB_quote(vsb, tc->in, -1, tc->how);
+		VSB_quote_pfx(vsb, "PFX", tc->in, tc->inlen, tc->how);
 		assert(VSB_finish(vsb) == 0);
 
-		printf("%s -> %s", tc->in, VSB_data(vsb));
+		VSB_clear(vsbo);
+		VSB_printf(vsbo, "0x%02x: ", tc->how);
+		VSB_quote(vsbo, tc->in, tc->inlen, VSB_QUOTE_HEX);
+		VSB_printf(vsbo, " -> ");
+		VSB_quote(vsbo, VSB_data(vsb), -1, VSB_QUOTE_HEX);
+		VSB_printf(vsbo, " (");
+		VSB_quote(vsbo, tc->out, -1, VSB_QUOTE_ESCHEX);
+		VSB_printf(vsbo, ")");
 		if (strcmp(VSB_data(vsb), tc->out)) {
-			printf(", but should have been %s",  tc->out);
+			VSB_printf(vsbo, "\nShould have been:\n\t");
+			VSB_quote(vsbo, tc->out, -1, VSB_QUOTE_HEX);
+			VSB_printf(vsbo, "\nThat's:\n\t");
+			VSB_quote(vsbo, VSB_data(vsb), -1, VSB_QUOTE_ESCHEX);
+			VSB_printf(vsbo, "\nvs:\n\t");
+			VSB_quote(vsbo, tc->out, -1, VSB_QUOTE_ESCHEX);
+			VSB_printf(vsbo, "\nFlags 0x%02x = ", tc->how);
+			if (!tc->how)
+				VSB_printf(vsbo, "\n\t0");
+			if (tc->how & VSB_QUOTE_NONL)
+				VSB_printf(vsbo, "\n\tVSB_QUOTE_NONL");
+			if (tc->how & VSB_QUOTE_JSON)
+				VSB_printf(vsbo, "\n\tVSB_QUOTE_JSON");
+			if (tc->how & VSB_QUOTE_HEX)
+				VSB_printf(vsbo, "\n\tVSB_QUOTE_HEX");
+			if (tc->how & VSB_QUOTE_CSTR)
+				VSB_printf(vsbo, "\n\tVSB_QUOTE_CSTR");
+			if (tc->how & VSB_QUOTE_UNSAFE)
+				VSB_printf(vsbo, "\n\tVSB_QUOTE_UNSAFE");
+			if (tc->how & VSB_QUOTE_ESCHEX)
+				VSB_printf(vsbo, "\n\tVSB_QUOTE_ESCHEX");
+			VSB_printf(vsbo, "\n\n");
 			err = 1;
 		}
-		printf("\n");
+		AZ(VSB_finish(vsbo));
+		printf("%s\n", VSB_data(vsbo));
 		VSB_clear(vsb);
 	}
 	VSB_destroy(&vsb);
+	VSB_destroy(&vsbo);
 	printf("error is %i\n", err);
 	return (err);
 }

From phk at FreeBSD.org  Mon Mar  1 13:39:09 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon,  1 Mar 2021 13:39:09 +0000 (UTC)
Subject: [master] 039697423 Include <stdint.h>
Message-ID: <20210301133909.726541096EF@lists.varnish-cache.org>


commit 0396974238f90a23036e7d97e40a63b9a8040004
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 1 13:38:18 2021 +0000

    Include <stdint.h>

diff --git a/lib/libvarnish/vsb.c b/lib/libvarnish/vsb.c
index 5cc76601a..008ad2ce0 100644
--- a/lib/libvarnish/vsb.c
+++ b/lib/libvarnish/vsb.c
@@ -34,6 +34,7 @@ __FBSDID("$FreeBSD: head/sys/kern/subr_vsb.c 222004 2011-05-17 06:36:32Z phk $")
 #include <ctype.h>
 #include <stdarg.h>
 #include <stdio.h>
+#include <stdint.h>
 #include <stdlib.h>
 #include <string.h>
 #include <unistd.h>

From nils.goroll at uplex.de  Mon Mar  1 14:01:08 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon,  1 Mar 2021 14:01:08 +0000 (UTC)
Subject: [master] a6e99d818 Leave a note regarding coccinelle and vgz
Message-ID: <20210301140108.71D1D10D08E@lists.varnish-cache.org>


commit a6e99d81845d62d3a5a49eceaaf218848b6b043a
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Mar 1 14:27:07 2021 +0100

    Leave a note regarding coccinelle and vgz
    
    in a place more likely to be noticed
    
    I also played with other spatch options to replace ``--dir .``, but they
    resulted in more clumsy shell commands.
    
    Ref a98c1489640c8098b8e6e80ebcd9831c0a7f087b

diff --git a/tools/coccinelle/README.rst b/tools/coccinelle/README.rst
index cb92f989b..99831344a 100644
--- a/tools/coccinelle/README.rst
+++ b/tools/coccinelle/README.rst
@@ -18,4 +18,7 @@ Unless noted otherwise, all patches should work when invoked as::
 The ``archive/`` directory contains patches which we used once and
 should not need again, but want to retain for reference.
 
+Do not commit any ``libvgz`` changes, as this code is manually kept in
+sync with upstream.
+
 .. _coccinelle: http://coccinelle.lip6.fr/

From martin at varnish-software.com  Mon Mar  1 17:39:08 2021
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Mon,  1 Mar 2021 17:39:08 +0000 (UTC)
Subject: [master] 18f5acc38 Use the REQ priority for incoming connection tasks
 by the acceptor
Message-ID: <20210301173908.E31AF112B02@lists.varnish-cache.org>


commit 18f5acc38af156f9b77c15a898178da3a92f59bf
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Wed Feb 17 13:39:37 2021 +0100

    Use the REQ priority for incoming connection tasks by the acceptor
    
    When accepting new incoming connections in the acceptor thread, it would
    schedule, they would be registered with the VCA priority. This priority is
    reserved for the acceptor thread itself, and specifically is not included
    in the TASK_QUEUE_CLIENT categorisation. This would interfere with the
    thread reserve pools.
    
    t02011.vtc had to be adjusted to account for the new priority
    categorisation of the initial request.

diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index 471ecc824..ae4da2f84 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -516,7 +516,7 @@ vca_accept_task(struct worker *wrk, void *arg)
 
 		wa.acceptsock = i;
 
-		if (!Pool_Task_Arg(wrk, TASK_QUEUE_VCA,
+		if (!Pool_Task_Arg(wrk, TASK_QUEUE_REQ,
 		    vca_make_session, &wa, sizeof wa)) {
 			/*
 			 * We couldn't get another thread, so we will handle
diff --git a/bin/varnishtest/tests/t02011.vtc b/bin/varnishtest/tests/t02011.vtc
index 6991e1025..88c64d904 100644
--- a/bin/varnishtest/tests/t02011.vtc
+++ b/bin/varnishtest/tests/t02011.vtc
@@ -20,8 +20,8 @@ server s1 {
 # at this point, so when we try to get a second stream, we fail.
 
 varnish v1 -cliok "param.set thread_pools 1"
-varnish v1 -cliok "param.set thread_pool_min 5"
-varnish v1 -cliok "param.set thread_pool_max 5"
+varnish v1 -cliok "param.set thread_pool_min 6"
+varnish v1 -cliok "param.set thread_pool_max 6"
 varnish v1 -cliok "param.set thread_queue_limit 0"
 varnish v1 -cliok "param.set thread_stats_rate 1"
 varnish v1 -cliok "param.set feature +http2"
@@ -66,10 +66,10 @@ client c1 {
 } -run
 
 # trigger an update of the stats
-varnish v1 -cliok "param.set thread_pool_max 6"
-varnish v1 -cliok "param.set thread_pool_min 6"
+varnish v1 -cliok "param.set thread_pool_max 7"
+varnish v1 -cliok "param.set thread_pool_min 7"
 delay 1
-varnish v1 -cliok "param.set thread_pool_min 5"
+varnish v1 -cliok "param.set thread_pool_min 6"
 delay 1
 varnish v1 -vsl_catchup
 varnish v1 -expect sess_dropped == 0

From martin at varnish-software.com  Mon Mar  1 17:43:07 2021
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Mon,  1 Mar 2021 17:43:07 +0000 (UTC)
Subject: [master] e4ea44564 Limit watchdog to highest priority only
Message-ID: <20210301174307.2D2ED112E14@lists.varnish-cache.org>


commit e4ea44564f512d42f99aca833b4f71dcc8b445c7
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Thu Feb 25 15:58:27 2021 +0100

    Limit watchdog to highest priority only
    
    The watchdog mechanism currently triggers when any queueing is happening,
    regardless of the priority. Strictly speaking it is only the backend
    fetches that are critical to get executed, and this prevents the thread
    limits to be used as limits on the amount of work the Varnish instance
    should handle.
    
    This can be especially important for instances with H/2 enabled, as these
    connections will be holding threads for extended periods of time, possibly
    triggering the watchdog in benign situations.
    
    This patch limits the watchdog to only trigger for no queue development
    on the highest priority queue.

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 7fba1e6f2..d80877e76 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -220,6 +220,7 @@ enum task_prio {
 	TASK_QUEUE__END
 };
 
+#define TASK_QUEUE_HIGHEST_PRIORITY TASK_QUEUE_BO
 #define TASK_QUEUE_CLIENT(prio) \
 	(prio == TASK_QUEUE_REQ || prio == TASK_QUEUE_STR)
 
diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c
index 3d31072d0..090ada6c3 100644
--- a/bin/varnishd/cache/cache_wrk.c
+++ b/bin/varnishd/cache/cache_wrk.c
@@ -600,7 +600,9 @@ pool_herder(void *priv)
 		 * Instead we implement a watchdog and kill the worker if
 		 * nothing has been dequeued for that long.
 		 */
-		if (pp->lqueue == 0) {
+		if (VTAILQ_EMPTY(&pp->queues[TASK_QUEUE_HIGHEST_PRIORITY])) {
+			/* Watchdog only applies to no movement on the
+			 * highest priority queue (TASK_QUEUE_BO) */
 			dq = pp->ndequeued + 1;
 		} else if (dq != pp->ndequeued) {
 			dq = pp->ndequeued;
diff --git a/bin/varnishtest/tests/c00104.vtc b/bin/varnishtest/tests/c00104.vtc
new file mode 100644
index 000000000..3af498179
--- /dev/null
+++ b/bin/varnishtest/tests/c00104.vtc
@@ -0,0 +1,32 @@
+varnishtest "Test watchdog only active on queue 0"
+
+server s1 {
+	rxreq
+	txresp
+} -start
+
+varnish v1 -cliok "param.set thread_pools 1"
+varnish v1 -cliok "param.set thread_pool_min 5"
+varnish v1 -cliok "param.set thread_pool_max 5"
+varnish v1 -cliok "param.set thread_pool_watchdog 1"
+varnish v1 -cliok "param.set feature +http2"
+
+varnish v1 -vcl+backend {
+} -start
+
+client c1 {
+	txpri
+	delay 2
+} -start
+
+client c2 {
+	txpri
+	delay 2
+} -start
+
+client c3 {
+	txpri
+	delay 2
+} -start
+
+delay 2

From nils.goroll at uplex.de  Tue Mar  2 13:06:09 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Tue,  2 Mar 2021 13:06:09 +0000 (UTC)
Subject: [master] 81aa74ccb Accept all ENET* ECONN* EHOST* errnos
Message-ID: <20210302130609.406001095E2@lists.varnish-cache.org>


commit 81aa74ccbd6682a69a08141e7058b001b7c47efe
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Tue Mar 2 13:59:50 2021 +0100

    Accept all ENET* ECONN* EHOST* errnos
    
    ... documented on Linux as POSIX.1
    
    The exception here is ECONNREFUSED which so far we only tolerate for
    Solaris and which seems to make sense for connect() only.
    
    To be discussed in #3539

diff --git a/lib/libvarnish/vtcp.c b/lib/libvarnish/vtcp.c
index 9ac383769..4c99beac1 100644
--- a/lib/libvarnish/vtcp.c
+++ b/lib/libvarnish/vtcp.c
@@ -604,6 +604,13 @@ VTCP_Check(ssize_t a)
 	 * some time. */
 	if (errno == ETIMEDOUT)
 		return (1);
+	/* #3539 various errnos documented on linux as POSIX.1 */
+	if (errno == ENETDOWN || errno == ENETUNREACH || errno == ENETRESET ||
+	    errno == ECONNABORTED || /* ECONNRESET see above */
+	    errno == EHOSTUNREACH || errno == EHOSTDOWN) {
+		return (1);
+	}
+
 #if (defined (__SVR4) && defined (__sun))
 	if (errno == ECONNREFUSED)	// in r02702.vtc
 		return (1);

From dridi.boukelmoune at gmail.com  Tue Mar  2 16:11:05 2021
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue,  2 Mar 2021 16:11:05 +0000 (UTC)
Subject: [master] 5ea8940c9 hash: Revert recent hash changes
Message-ID: <20210302161105.DF7CD111175@lists.varnish-cache.org>


commit 5ea8940c9d937697e8563609ac3a921e8755255f
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Mar 2 16:02:48 2021 +0100

    hash: Revert recent hash changes
    
    This reverts the following commits:
    
    - e98e8e6497b9bbcca3e709a5ea0e094f2ec4b930.
      "Documentation updates for changed `vcl_hash{}` / `hash_data()`"
    - 001279ebd6ee83fafbc7fc1d1805bf2224361dd9.
      "Document proper design pattern for using hash_data() in vcl_recv,"
    - e36573e255af2aa1ed40fb367fcb5257ef7e0288.
      "Add a test-case for hash_data() in vcl_recv{}"
    - 03fe0cee1abc3741a7652ba2cf9c1bf822d65ab1.
      "Allow hash_data() in vcl_recv{}"
    - 4ebc3cfec133586cd8c4a715d8de18efb76402f1.
      "Make it possible to override the initial digest, and explain in"
    - d6ad52f5ff95daa3668fdad28b4c97e59b4c49d3
      "Change the way we calculate the hash key for the cache."
    
    Conflicts:
            doc/sphinx/reference/dp_vcl_recv_hash.rst
            doc/sphinx/reference/index.rst
    
    Concerns were raised regarding a change of the way we compute the hash
    key outside of the dot-zero release where we would expect such breaking
    changes (among other things, vmod_shard relies on hash stability).
    
    There is also no definite consensus of how to handle hashing from
    vcl_recv.

diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index cc1e44c26..d94ec2ad5 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -183,6 +183,19 @@ HSH_DeleteObjHead(const struct worker *wrk, struct objhead *oh)
 	FREE_OBJ(oh);
 }
 
+void
+HSH_AddString(struct req *req, void *ctx, const char *str)
+{
+
+	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
+	AN(ctx);
+	if (str != NULL) {
+		VSHA256_Update(ctx, str, strlen(str));
+		VSLb(req->vsl, SLT_Hash, "%s", str);
+	} else
+		VSHA256_Update(ctx, &str, 1);
+}
+
 /*---------------------------------------------------------------------
  * This is a debugging hack to enable testing of boundary conditions
  * in the hash algorithm.
diff --git a/bin/varnishd/cache/cache_objhead.h b/bin/varnishd/cache/cache_objhead.h
index af1797462..bc1782379 100644
--- a/bin/varnishd/cache/cache_objhead.h
+++ b/bin/varnishd/cache/cache_objhead.h
@@ -71,6 +71,7 @@ int HSH_DerefObjCore(struct worker *, struct objcore **, int rushmax);
 
 enum lookup_e HSH_Lookup(struct req *, struct objcore **, struct objcore **);
 void HSH_Ref(struct objcore *o);
+void HSH_AddString(struct req *, void *ctx, const char *str);
 unsigned HSH_Purge(struct worker *, struct objhead *, vtim_real ttl_now,
     vtim_dur ttl, vtim_dur grace, vtim_dur keep);
 struct objcore *HSH_Private(const struct worker *wrk);
diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index ea09689f6..1dc349b99 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -50,6 +50,7 @@
 #include "storage/storage.h"
 #include "common/heritage.h"
 #include "vcl.h"
+#include "vsha256.h"
 #include "vtim.h"
 
 #define REQ_STEPS \
@@ -76,14 +77,6 @@
 REQ_STEPS
 #undef REQ_STEP
 
-/*
- * In this specific context we use SHA256 only as a very good
- * hashing function.  That renders most of the normal concerns
- * about salting & seeding moot.  However, if for some reason
- * you want to salt your hashes, this is where you do it.
- */
-static const uint8_t initial_digest[DIGEST_LEN];
-
 /*--------------------------------------------------------------------
  * Handle "Expect:" and "Connection:" on incoming request
  */
@@ -898,6 +891,7 @@ static enum req_fsm_nxt v_matchproto_(req_state_f)
 cnt_recv(struct worker *wrk, struct req *req)
 {
 	unsigned recv_handling;
+	struct VSHA256Context sha256ctx;
 	const char *ci, *cp, *endpname;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
@@ -927,8 +921,6 @@ cnt_recv(struct worker *wrk, struct req *req)
 		return (REQ_FSM_DONE);
 	}
 
-	memcpy(req->digest, initial_digest, sizeof req->digest);
-
 	VCL_recv_method(req->vcl, wrk, req, NULL, NULL);
 
 	if (wrk->handling == VCL_RET_FAIL) {
@@ -970,13 +962,13 @@ cnt_recv(struct worker *wrk, struct req *req)
 		}
 	}
 
-	if (!memcmp(req->digest, initial_digest, sizeof req->digest)) {
-		VCL_hash_method(req->vcl, wrk, req, NULL, NULL);
-		if (wrk->handling == VCL_RET_FAIL)
-			recv_handling = wrk->handling;
-		else
-			assert(wrk->handling == VCL_RET_LOOKUP);
-	}
+	VSHA256_Init(&sha256ctx);
+	VCL_hash_method(req->vcl, wrk, req, NULL, &sha256ctx);
+	if (wrk->handling == VCL_RET_FAIL)
+		recv_handling = wrk->handling;
+	else
+		assert(wrk->handling == VCL_RET_LOOKUP);
+	VSHA256_Final(req->digest, &sha256ctx);
 
 	switch (recv_handling) {
 	case VCL_RET_VCL:
diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index 6d323a22c..0a729c0e4 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -689,29 +689,19 @@ VRT_fail(VRT_CTX, const char *fmt, ...)
 VCL_VOID
 VRT_hashdata(VRT_CTX, VCL_STRANDS s)
 {
-	struct VSHA256Context sha256ctx;
 	int i;
 
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
 	CHECK_OBJ_NOTNULL(ctx->req, REQ_MAGIC);
-	AZ(ctx->specific);
-	VSHA256_Init(&sha256ctx);
-	VSHA256_Update(&sha256ctx, ctx->req->digest, sizeof ctx->req->digest);
+	AN(ctx->specific);
 	AN(s);
-	for (i = 0; i < s->n; i++) {
-		if (s->p[i] != NULL) {
-			VSHA256_Update(&sha256ctx, s->p[i], strlen(s->p[i]));
-			VSLb(ctx->req->vsl, SLT_Hash, "%s", s->p[i]);
-		} else {
-			VSHA256_Update(&sha256ctx, "", 1);
-		}
-	}
+	for (i = 0; i < s->n; i++)
+		HSH_AddString(ctx->req, ctx->specific, s->p[i]);
 	/*
 	 * Add a 'field-separator' to make it more difficult to
 	 * manipulate the hash.
 	 */
-	VSHA256_Update(&sha256ctx, "", 1);
-	VSHA256_Final(ctx->req->digest, &sha256ctx);
+	HSH_AddString(ctx->req, ctx->specific, NULL);
 }
 
 /*--------------------------------------------------------------------*/
diff --git a/bin/varnishtest/tests/b00051.vtc b/bin/varnishtest/tests/b00051.vtc
index 8119dc6b5..3681ad2b3 100644
--- a/bin/varnishtest/tests/b00051.vtc
+++ b/bin/varnishtest/tests/b00051.vtc
@@ -22,5 +22,5 @@ client c1 {
 	rxresp
 	expect resp.http.req_hash ~ "[[:xdigit:]]{64}"
 	expect resp.http.req_hash == resp.http.bereq_hash
-	expect resp.http.req_hash-sf == ":0jkH41nfmD0PRFsKpM1m7ucOApnxFc62B//mQWxOnmQ=:"
+	expect resp.http.req_hash-sf == ":3k0f0yRKtKt7akzkyNsTGSDOJAZOQowTwKWhu5+kIu0=:"
 } -run
diff --git a/bin/varnishtest/tests/b00076.vtc b/bin/varnishtest/tests/b00076.vtc
deleted file mode 100644
index 120339275..000000000
--- a/bin/varnishtest/tests/b00076.vtc
+++ /dev/null
@@ -1,67 +0,0 @@
-varnishtest "hash_data() in vcl_recv{}"
-
-server s1 {
-	rxreq
-	txresp -hdr "Same: One"
-} -start
-
-varnish v1 -vcl+backend {
-    sub vcl_recv {
-	if (req.url == "/2") {
-		hash_data(req.http.foo);
-	}
-    }
-    sub vcl_hash {
-	hash_data(req.url);
-	return (lookup);
-    }
-} -start
-
-varnish v1 -cliok "param.set vsl_mask +Hash"
-
-client c1 {
-	txreq -url /1
-	rxresp
-	expect resp.status == 200
-	expect resp.http.same == One
-	txreq -url /2 -hdr "foo: /1"
-	rxresp
-	expect resp.status == 200
-	expect resp.http.same == One
-} -run
-
-server s1 {
-	rxreq
-	txresp -hdr "Second: One"
-} -start
-
-varnish v1 -vcl+backend {
-    sub make_hash_key {
-        hash_data("Documented Design Pattern");
-        hash_data(req.url);
-    }
-    
-    sub vcl_hash {
-        call make_hash_key;
-        return (lookup);
-    }
- 
-    sub vcl_recv {
-        if (req.http.early) {
-            call make_hash_key;
-        }
-    }
-}
-
-client c1 {
-	txreq
-	rxresp
-	expect resp.status == 200
-	expect resp.http.second == One
-
-	txreq -hdr "early: yes"
-	rxresp
-	expect resp.status == 200
-	expect resp.http.second == One
-} -run
-
diff --git a/bin/varnishtest/tests/d00020.vtc b/bin/varnishtest/tests/d00020.vtc
index 9f70ff2d3..0c42285d7 100644
--- a/bin/varnishtest/tests/d00020.vtc
+++ b/bin/varnishtest/tests/d00020.vtc
@@ -218,9 +218,9 @@ client c1 {
 client c2 {
 	txreq -url /b/def
 	rxresp
-	expect resp.http.hash		== "2e03d4fee2722154a84036faa3e4b6851e003830eadbe0d2874e3df09c8efe55"
+	expect resp.http.hash		== "93d1c4ad76396c91dd97fa310f7f26445332662c89393dbeeb77fe49f9111ee4"
 	expect resp.http.by		== "HASH"
-	expect resp.http.key		-eq 0x2e03d4fe
+	expect resp.http.key		-eq 0x93d1c4ad
 	expect resp.http.alt		== 0
 	expect resp.http.warmup	== "-1.000"
 	expect resp.http.rampup	== "true"
@@ -228,9 +228,9 @@ client c2 {
 
 	txreq -url /b/hash
 	rxresp
-	expect resp.http.hash		== "49f33e9019891091d3dcf6edab6d9433b756678bdf5202dd41b8a667c89887b1"
+	expect resp.http.hash		== "e47da20ea4db49d4f22acdadc69f02f445002be520a2865cd3351272add62540"
 	expect resp.http.by		== "HASH"
-	expect resp.http.key		-eq 0x49f33e90
+	expect resp.http.key		-eq 0xe47da20e
 	expect resp.http.alt		== 1
 	expect resp.http.warmup	== "-1.000"
 	expect resp.http.rampup	== "true"
@@ -268,9 +268,9 @@ client c2 {
 client c3 {
 	txreq -url /b/c/hash/def
 	rxresp
-	expect resp.http.hash		== "dd70dcbbf385c398ee3b53a849f12d0d846fc21292349fb45d37b2d9d8eca25e"
+	expect resp.http.hash		== "df9a465f8a0455c334b24c1638d3adda0f6e64fbe759029ab83602e3b9138884"
 	expect resp.http.by		== "HASH"
-	expect resp.http.key		-eq 0xdd70dcbb
+	expect resp.http.key		-eq 0xdf9a465f
 	expect resp.http.alt		== 7
 	expect resp.http.warmup	== "-1.000"
 	expect resp.http.rampup	== "true"
@@ -278,9 +278,9 @@ client c3 {
 
 	txreq -url /b/c/hash/hash
 	rxresp
-	expect resp.http.hash		== "41d09b9877cd0ac0eab888359b0ad54f0bf41da0ac03dc1b8ae12aff18465a8d"
+	expect resp.http.hash		== "0eb35bc1fab5aad5902fd1bac86540bd13d43aa31c6c46f54e776b43392e66e6"
 	expect resp.http.by		== "HASH"
-	expect resp.http.key		-eq 0x41d09b98
+	expect resp.http.key		-eq 0x0eb35bc1
 	expect resp.http.alt		== 8
 	expect resp.http.warmup	== "-1.000"
 	expect resp.http.rampup	== "true"
@@ -288,9 +288,9 @@ client c3 {
 
 	txreq -url /b/c/hash/url
 	rxresp
-	expect resp.http.hash		== "dcac849e02b3322f5fd3dddf9b9f5fc26d295733e6f1c51b190dfb7239a56e28"
+	expect resp.http.hash		== "1eb67b701ea07151cac5bea1f11b6267b9de15a3ff83cec995590480cbc2c750"
 	expect resp.http.by		== "HASH"
-	expect resp.http.key		-eq 0xdcac849e
+	expect resp.http.key		-eq 0x1eb67b70
 	expect resp.http.alt		== 9
 	expect resp.http.warmup	== "0.500"
 	expect resp.http.rampup	== "true"
@@ -298,9 +298,9 @@ client c3 {
 
 	txreq -url /b/c/hash/key
 	rxresp
-	expect resp.http.hash		== "112393761506e85f0c700a5d669a48b54001c870eb2e8d95f4d2f6fdccbe80a3"
+	expect resp.http.hash		== "a11b617e21aa7db22b6205d7612002e595b1b00d8c11602017f65456a1be3a35"
 	expect resp.http.by		== "HASH"
-	expect resp.http.key		-eq 0x11239376
+	expect resp.http.key		-eq 0xa11b617e
 	expect resp.http.alt		== 10
 	expect resp.http.warmup	== "-1.000"
 	expect resp.http.rampup	== "false"
@@ -308,9 +308,9 @@ client c3 {
 
 	txreq -url /b/c/hash/blob
 	rxresp
-	expect resp.http.hash		== "5ef050c1185ac02a66d9f79703b8cd5f0636abf3b1f15b9f22e0fe64df985d28"
+	expect resp.http.hash		== "d7eecc0ac83e1727332dcd8c7c8ae9f3114123abb2bf7e3fb15ecea8c84bb239"
 	expect resp.http.by		== "HASH"
-	expect resp.http.key		-eq 0x5ef050c1
+	expect resp.http.key		-eq 0xd7eecc0a
 	expect resp.http.alt		== 11
 	expect resp.http.warmup	== "-1.000"
 	expect resp.http.rampup	== "true"
diff --git a/doc/changes.rst b/doc/changes.rst
index 146ec9f60..c11ba2cf5 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -35,15 +35,6 @@ release process.
 Varnish Cache Next (2021-03-15)
 ================================
 
-* `hash_data()` can be called from `vcl_recv`, in which case
-  `vcl_hash` is not called.  This allows `vcl_recv` and 
-  backends to take the object identity into account, for
-  instance when choosing backend and grace periods.
-
-* `hash_data()` calculates the hash-key differently than previously.
-  This means that persistent storage will be lost, and it may break
-  very specific `*.vtc` test-scripts.
-
 * counters MAIN.s_req_bodybytes and VBE.*.tools.beresp_bodybytes
   are now always the number of bodybytes moved on the wire.
 
diff --git a/doc/sphinx/reference/dp_vcl_recv_hash.rst b/doc/sphinx/reference/dp_vcl_recv_hash.rst
deleted file mode 100644
index e63607e1e..000000000
--- a/doc/sphinx/reference/dp_vcl_recv_hash.rst
+++ /dev/null
@@ -1,29 +0,0 @@
-..
-	Copyright (c) 2021 Varnish Software AS
-	SPDX-License-Identifier: BSD-2-Clause
-	See LICENSE file for full text of license
-
-.. _db_vcl_recv_hash:
-
-Hashing in `vcl_recv{}`
-=======================
-
-Calculating the `hash` used for cache lookup already in `vcl_recv{}`
-makes it possible for certain directors to offer targeted health status.
-
-To ensure consistent hashing, use this design pattern::
-
-    sub make_hash_key {
-        hash_data([?]);
-    }
-    
-    sub vcl_hash {
-        call make_hash_key;
-        return (lookup);
-    }
-
-    sub vcl_recv {
-        [?]
-        call make_hash_key;
-        [?]
-    }
diff --git a/doc/sphinx/reference/index.rst b/doc/sphinx/reference/index.rst
index 11fe4737c..c5c751019 100644
--- a/doc/sphinx/reference/index.rst
+++ b/doc/sphinx/reference/index.rst
@@ -27,7 +27,6 @@ VCL Design Patterns
 .. toctree::
 	:maxdepth: 1
 
-	dp_vcl_recv_hash.rst
 	dp_vcl_resp_status.rst
 
 Bundled VMODs
diff --git a/doc/sphinx/reference/vcl.rst b/doc/sphinx/reference/vcl.rst
index 02e2a64df..3dbae425b 100644
--- a/doc/sphinx/reference/vcl.rst
+++ b/doc/sphinx/reference/vcl.rst
@@ -355,9 +355,7 @@ hash_data(input)
 ~~~~~~~~~~~~~~~~
 
   Adds an input to the hash input. In the built-in VCL ``hash_data()``
-  is called on the host and URL of the request.
-  Available in ``vcl_hash`` or ``vcl_recv``.  If used in ``vcl_recv``
-  ``vcl_hash`` will not be called.
+  is called on the host and URL of the request. Available in ``vcl_hash``.
 
 synthetic(STRING)
 ~~~~~~~~~~~~~~~~~
diff --git a/doc/sphinx/users-guide/vcl-built-in-subs.rst b/doc/sphinx/users-guide/vcl-built-in-subs.rst
index 3c0990910..2f46d8b87 100644
--- a/doc/sphinx/users-guide/vcl-built-in-subs.rst
+++ b/doc/sphinx/users-guide/vcl-built-in-subs.rst
@@ -134,9 +134,8 @@ of the following keywords:
 vcl_hash
 ~~~~~~~~
 
-Called after `vcl_recv` to create a hash value for the request,
-unless `vcl_recv` already did that.
-This is used as the key to store and look up objects in the cache.
+Called after `vcl_recv` to create a hash value for the request. This is
+used as a key to look up the object in Varnish.
 
 The `vcl_hash` subroutine may terminate with calling ``return()`` with one
 of the following keywords:
diff --git a/doc/sphinx/users-guide/vcl-hashing.rst b/doc/sphinx/users-guide/vcl-hashing.rst
index 1f50a5e63..c605f5589 100644
--- a/doc/sphinx/users-guide/vcl-hashing.rst
+++ b/doc/sphinx/users-guide/vcl-hashing.rst
@@ -6,10 +6,12 @@
 Hashing
 -------
 
-Internally, when Varnish stores content in the cache indexed by a hash
-key used to find the object again. In the default setup
-this key is calculated based on `URL`, the `Host:` header, or
-if there is none, the IP address of the server::
+Internally, when Varnish stores content in the cache it stores the object
+together with a hash key to find the object again. In the default setup
+this key is calculated based on the content of the *Host* header or the
+IP address of the server and the URL.
+
+Behold the `default vcl`::
 
     sub vcl_hash {
         hash_data(req.url);
@@ -21,7 +23,7 @@ if there is none, the IP address of the server::
         return (lookup);
     }
 
-As you can see it first hashes `req.url` and then `req.http.host` if
+As you can see it first checks in `req.url` then `req.http.host` if
 it exists. It is worth pointing out that Varnish doesn't lowercase the
 hostname or the URL before hashing it so in theory having "Varnish.org/"
 and "varnish.org/" would result in different cache entries. Browsers
@@ -45,16 +47,7 @@ And then add a `vcl_hash`::
         hash_data(req.http.X-Country-Code);
     }
 
-Because there is no `return(lookup)`, the builtin VCL will take care
-of adding the URL, `Host:` or server IP# to the hash as usual.
-
-If `vcl_hash` did return, ie::
-
-    sub vcl_hash {
-        hash_data(req.http.X-Country-Code);
-        return(lookup);
-    }
-
-then *only* the country-code would matter, and Varnish would return
-seemingly random objects, ignoring the URL, (but they would always
-have the correct `X-Country-Code`).
+As the default VCL will take care of adding the host and URL to the hash
+we don't have to do anything else. Be careful calling ``return (lookup)``
+as this will abort the execution of the default VCL and Varnish can end
+up returning data based on more or less random inputs.
diff --git a/include/vrt.h b/include/vrt.h
index 70eb45179..01ca336c5 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -54,7 +54,6 @@
  * binary/load-time compatible, increment MAJOR version
  *
  * 13.0 (2021-03-15)
- *	VRT_hashdata() produces different hash-keys.
  *	Move VRT_synth_page() to deprecated status
  *	Add VRT_synth_strands() and VRT_synth_blob()
  *	struct vrt_type now produced by generate.py
diff --git a/lib/libvcc/vcc_action.c b/lib/libvcc/vcc_action.c
index 57a2f6daf..60fbdfa7b 100644
--- a/lib/libvcc/vcc_action.c
+++ b/lib/libvcc/vcc_action.c
@@ -452,7 +452,7 @@ vcc_Action_Init(struct vcc *tl)
 	ACT(ban,	vcc_act_ban,	0);
 	ACT(call,	vcc_act_call,	0);
 	ACT(hash_data,	vcc_act_hash_data,
-		VCL_MET_RECV | VCL_MET_HASH);
+		VCL_MET_HASH);
 	ACT(if,		vcc_Act_If,	0);
 	ACT(new,	vcc_Act_New,
 		VCL_MET_INIT);

From dridi.boukelmoune at gmail.com  Tue Mar  2 16:11:05 2021
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue,  2 Mar 2021 16:11:05 +0000 (UTC)
Subject: [master] 75c7f15a6 doc: Bring back changes to vcl-hashing.rst
Message-ID: <20210302161105.F3262111178@lists.varnish-cache.org>


commit 75c7f15a6c061f67e38ba147a577181f1675ceb1
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 2 16:23:05 2021 +0100

    doc: Bring back changes to vcl-hashing.rst
    
    On their own, those changes improve the documentation and don't mention
    the reverted "hash_data() in vcl_recv" feature.
    
    Refs e98e8e6497b9

diff --git a/doc/sphinx/users-guide/vcl-hashing.rst b/doc/sphinx/users-guide/vcl-hashing.rst
index c605f5589..1f50a5e63 100644
--- a/doc/sphinx/users-guide/vcl-hashing.rst
+++ b/doc/sphinx/users-guide/vcl-hashing.rst
@@ -6,12 +6,10 @@
 Hashing
 -------
 
-Internally, when Varnish stores content in the cache it stores the object
-together with a hash key to find the object again. In the default setup
-this key is calculated based on the content of the *Host* header or the
-IP address of the server and the URL.
-
-Behold the `default vcl`::
+Internally, when Varnish stores content in the cache indexed by a hash
+key used to find the object again. In the default setup
+this key is calculated based on `URL`, the `Host:` header, or
+if there is none, the IP address of the server::
 
     sub vcl_hash {
         hash_data(req.url);
@@ -23,7 +21,7 @@ Behold the `default vcl`::
         return (lookup);
     }
 
-As you can see it first checks in `req.url` then `req.http.host` if
+As you can see it first hashes `req.url` and then `req.http.host` if
 it exists. It is worth pointing out that Varnish doesn't lowercase the
 hostname or the URL before hashing it so in theory having "Varnish.org/"
 and "varnish.org/" would result in different cache entries. Browsers
@@ -47,7 +45,16 @@ And then add a `vcl_hash`::
         hash_data(req.http.X-Country-Code);
     }
 
-As the default VCL will take care of adding the host and URL to the hash
-we don't have to do anything else. Be careful calling ``return (lookup)``
-as this will abort the execution of the default VCL and Varnish can end
-up returning data based on more or less random inputs.
+Because there is no `return(lookup)`, the builtin VCL will take care
+of adding the URL, `Host:` or server IP# to the hash as usual.
+
+If `vcl_hash` did return, ie::
+
+    sub vcl_hash {
+        hash_data(req.http.X-Country-Code);
+        return(lookup);
+    }
+
+then *only* the country-code would matter, and Varnish would return
+seemingly random objects, ignoring the URL, (but they would always
+have the correct `X-Country-Code`).

From nils.goroll at uplex.de  Wed Mar  3 10:15:03 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed,  3 Mar 2021 10:15:03 +0000 (UTC)
Subject: [master] 5b89fb4f7 builtin: Standardize all comments to '#'
Message-ID: <20210303101503.DA9E81046FA@lists.varnish-cache.org>


commit 5b89fb4f7f0e3a9dc3a637264f916328ad47de95
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Mon Jan 18 12:25:54 2021 +0100

    builtin: Standardize all comments to '#'
    
    It's a bit all over the place right now and '#' is the most "scripty" of
    the 3 types of comments we have.
    
    Better diff with the --word-diff --word-diff-regex=. options.
    
    Salavaged from #3503

diff --git a/bin/varnishd/builtin.vcl b/bin/varnishd/builtin.vcl
index 329323c90..d0be8021f 100644
--- a/bin/varnishd/builtin.vcl
+++ b/bin/varnishd/builtin.vcl
@@ -1,35 +1,34 @@
-/*-
- * Copyright (c) 2006 Verdens Gang AS
- * Copyright (c) 2006-2015 Varnish Software AS
- * All rights reserved.
- *
- * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
- *
- * SPDX-License-Identifier: BSD-2-Clause
- *
- * Redistribution and use in source and binary forms, with or without
- * modification, are permitted provided that the following conditions
- * are met:
- * 1. Redistributions of source code must retain the above copyright
- *    notice, this list of conditions and the following disclaimer.
- * 2. Redistributions in binary form must reproduce the above copyright
- *    notice, this list of conditions and the following disclaimer in the
- *    documentation and/or other materials provided with the distribution.
- *
- * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
- * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
- * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
- * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
- * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
- * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
- * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
- * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
- * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
- * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
- * SUCH DAMAGE.
- *
- * This is the builtin VCL code
- */
+#-
+# Copyright (c) 2006 Verdens Gang AS
+# Copyright (c) 2006-2015 Varnish Software AS
+# All rights reserved.
+#
+# Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+#
+# SPDX-License-Identifier: BSD-2-Clause
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+# This is the builtin VCL code
 
 vcl 4.0;
 
@@ -41,13 +40,13 @@ sub vcl_recv {
         set req.http.host = req.http.host.lower();
     }
     if (req.method == "PRI") {
-        /* This will never happen in properly formed traffic (see: RFC7540) */
+        # This will never happen in properly formed traffic (see: RFC7540)
         return (synth(405));
     }
     if (!req.http.host &&
       req.esi_level == 0 &&
       req.proto ~ "^(?i)HTTP/1.1") {
-        /* In HTTP/1.1, Host is required. */
+        # In HTTP/1.1, Host is required.
         return (synth(400));
     }
     if (req.method != "GET" &&
@@ -58,16 +57,16 @@ sub vcl_recv {
       req.method != "OPTIONS" &&
       req.method != "DELETE" &&
       req.method != "PATCH") {
-        /* Non-RFC2616 or CONNECT which is weird. */
+        # Non-RFC2616 or CONNECT which is weird.
         return (pipe);
     }
 
     if (req.method != "GET" && req.method != "HEAD") {
-        /* We only deal with GET and HEAD by default */
+        # We only deal with GET and HEAD by default
         return (pass);
     }
     if (req.http.Authorization || req.http.Cookie) {
-        /* Not cacheable by default */
+        # Not cacheable by default
         return (pass);
     }
     return (hash);
@@ -112,9 +111,9 @@ sub vcl_deliver {
     return (deliver);
 }
 
-/*
- * We can come here "invisibly" with the following errors: 500 & 503
- */
+#
+# We can come here "invisibly" with the following errors: 500 & 503
+#
 sub vcl_synth {
     set resp.http.Content-Type = "text/html; charset=utf-8";
     set resp.http.Retry-After = "5";

From nils.goroll at uplex.de  Wed Mar  3 10:15:04 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed,  3 Mar 2021 10:15:04 +0000 (UTC)
Subject: [master] 4646a5bd7 builtin: Adopt our C code style for the builtin VCL
Message-ID: <20210303101504.089071046FD@lists.varnish-cache.org>


commit 4646a5bd77bd21aa330a5598016655f56dc7c38e
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Jan 26 11:04:42 2021 +0100

    builtin: Adopt our C code style for the builtin VCL
    
    I removed the else keyword from vcl_backend_response because it doesn't
    add any value.
    
    Better diff with the --ignore-all-space option.

diff --git a/bin/varnishd/builtin.vcl b/bin/varnishd/builtin.vcl
index d0be8021f..3fee28f52 100644
--- a/bin/varnishd/builtin.vcl
+++ b/bin/varnishd/builtin.vcl
@@ -36,88 +36,88 @@ vcl 4.0;
 # Client side
 
 sub vcl_recv {
-    if (req.http.host) {
-        set req.http.host = req.http.host.lower();
-    }
-    if (req.method == "PRI") {
-        # This will never happen in properly formed traffic (see: RFC7540)
-        return (synth(405));
-    }
-    if (!req.http.host &&
-      req.esi_level == 0 &&
-      req.proto ~ "^(?i)HTTP/1.1") {
-        # In HTTP/1.1, Host is required.
-        return (synth(400));
-    }
-    if (req.method != "GET" &&
-      req.method != "HEAD" &&
-      req.method != "PUT" &&
-      req.method != "POST" &&
-      req.method != "TRACE" &&
-      req.method != "OPTIONS" &&
-      req.method != "DELETE" &&
-      req.method != "PATCH") {
-        # Non-RFC2616 or CONNECT which is weird.
-        return (pipe);
-    }
-
-    if (req.method != "GET" && req.method != "HEAD") {
-        # We only deal with GET and HEAD by default
-        return (pass);
-    }
-    if (req.http.Authorization || req.http.Cookie) {
-        # Not cacheable by default
-        return (pass);
-    }
-    return (hash);
+	if (req.http.host) {
+		set req.http.host = req.http.host.lower();
+	}
+	if (req.method == "PRI") {
+		# This will never happen in properly formed traffic (see: RFC7540)
+		return (synth(405));
+	}
+	if (!req.http.host &&
+	    req.esi_level == 0 &&
+	    req.proto ~ "^(?i)HTTP/1.1") {
+		# In HTTP/1.1, Host is required.
+		return (synth(400));
+	}
+	if (req.method != "GET" &&
+	    req.method != "HEAD" &&
+	    req.method != "PUT" &&
+	    req.method != "POST" &&
+	    req.method != "TRACE" &&
+	    req.method != "OPTIONS" &&
+	    req.method != "DELETE" &&
+	    req.method != "PATCH") {
+		# Non-RFC2616 or CONNECT which is weird.
+		return (pipe);
+	}
+
+	if (req.method != "GET" && req.method != "HEAD") {
+		# We only deal with GET and HEAD by default
+		return (pass);
+	}
+	if (req.http.Authorization || req.http.Cookie) {
+		# Not cacheable by default
+		return (pass);
+	}
+	return (hash);
 }
 
 sub vcl_pipe {
-    # By default Connection: close is set on all piped requests, to stop
-    # connection reuse from sending future requests directly to the
-    # (potentially) wrong backend. If you do want this to happen, you can undo
-    # it here.
-    # unset bereq.http.connection;
-    return (pipe);
+	# By default "Connection: close" is set on all piped requests, to stop
+	# connection reuse from sending future requests directly to the
+	# (potentially) wrong backend. If you do want this to happen, you can
+	# undo it here:
+	# unset bereq.http.connection;
+	return (pipe);
 }
 
 sub vcl_pass {
-    return (fetch);
+	return (fetch);
 }
 
 sub vcl_hash {
-    hash_data(req.url);
-    if (req.http.host) {
-        hash_data(req.http.host);
-    } else {
-        hash_data(server.ip);
-    }
-    return (lookup);
+	hash_data(req.url);
+	if (req.http.host) {
+		hash_data(req.http.host);
+	} else {
+		hash_data(server.ip);
+	}
+	return (lookup);
 }
 
 sub vcl_purge {
-    return (synth(200, "Purged"));
+	return (synth(200, "Purged"));
 }
 
 sub vcl_hit {
-    return (deliver);
+	return (deliver);
 }
 
 sub vcl_miss {
-    return (fetch);
+	return (fetch);
 }
 
 sub vcl_deliver {
-    return (deliver);
+	return (deliver);
 }
 
 #
 # We can come here "invisibly" with the following errors: 500 & 503
 #
 sub vcl_synth {
-    set resp.http.Content-Type = "text/html; charset=utf-8";
-    set resp.http.Retry-After = "5";
-    set resp.body = {"<!DOCTYPE html>
+	set resp.http.Content-Type = "text/html; charset=utf-8";
+	set resp.http.Retry-After = "5";
+	set resp.body = {"<!DOCTYPE html>
 <html>
   <head>
     <title>"} + resp.status + " " + resp.reason + {"</title>
@@ -132,39 +132,40 @@ sub vcl_synth {
   </body>
 </html>
 "};
-    return (deliver);
+	return (deliver);
 }
 
 #######################################################################
 # Backend Fetch
 
 sub vcl_backend_fetch {
-    if (bereq.method == "GET") {
-        unset bereq.body;
-    }
-    return (fetch);
+	if (bereq.method == "GET") {
+		unset bereq.body;
+	}
+	return (fetch);
 }
 
 sub vcl_backend_response {
-    if (bereq.uncacheable) {
-        return (deliver);
-    } else if (beresp.ttl <= 0s ||
-      beresp.http.Set-Cookie ||
-      beresp.http.Surrogate-control ~ "(?i)no-store" ||
-      (!beresp.http.Surrogate-Control &&
-        beresp.http.Cache-Control ~ "(?i:no-cache|no-store|private)") ||
-      beresp.http.Vary == "*") {
-        # Mark as "Hit-For-Miss" for the next 2 minutes
-        set beresp.ttl = 120s;
-        set beresp.uncacheable = true;
-    }
-    return (deliver);
+	if (bereq.uncacheable) {
+		return (deliver);
+	}
+	if (beresp.ttl <= 0s ||
+	    beresp.http.Set-Cookie ||
+	    beresp.http.Surrogate-control ~ "(?i)no-store" ||
+	    (!beresp.http.Surrogate-Control &&
+	      beresp.http.Cache-Control ~ "(?i:no-cache|no-store|private)") ||
+	    beresp.http.Vary == "*") {
+		# Mark as "Hit-For-Miss" for the next 2 minutes
+		set beresp.ttl = 120s;
+		set beresp.uncacheable = true;
+	}
+	return (deliver);
 }
 
 sub vcl_backend_error {
-    set beresp.http.Content-Type = "text/html; charset=utf-8";
-    set beresp.http.Retry-After = "5";
-    set beresp.body = {"<!DOCTYPE html>
+	set beresp.http.Content-Type = "text/html; charset=utf-8";
+	set beresp.http.Retry-After = "5";
+	set beresp.body = {"<!DOCTYPE html>
 <html>
   <head>
     <title>"} + beresp.status + " " + beresp.reason + {"</title>
@@ -179,16 +180,16 @@ sub vcl_backend_error {
   </body>
 </html>
 "};
-    return (deliver);
+	return (deliver);
 }
 
 #######################################################################
 # Housekeeping
 
 sub vcl_init {
-    return (ok);
+	return (ok);
 }
 
 sub vcl_fini {
-    return (ok);
+	return (ok);
 }

From nils.goroll at uplex.de  Wed Mar  3 10:15:04 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed,  3 Mar 2021 10:15:04 +0000 (UTC)
Subject: [master] e45595ec9 vcc: Parse the built-in VCL independently
Message-ID: <20210303101504.31656104700@lists.varnish-cache.org>


commit e45595ec9067475d4d5baac2a6593ac97c0370b5
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Mon Dec 14 06:33:35 2020 +0100

    vcc: Parse the built-in VCL independently
    
    This is a lookahead parsing to learn about built-in subroutines that are
    not tied to a VCL state. Instead of maintaining a mapping of the other
    subroutines the builtin.vcl file itself becomes authoritative.

diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c
index cec471f16..e52c1da4b 100644
--- a/lib/libvcc/vcc_compile.c
+++ b/lib/libvcc/vcc_compile.c
@@ -72,6 +72,8 @@ static const struct method method_tab[] = {
 	{ NULL, 0U, 0}
 };
 
+struct vcc *vcc_builtin;
+
 /*--------------------------------------------------------------------*/
 
 static void
@@ -782,11 +784,14 @@ vcc_CompileSource(struct vcc *tl, struct source *sp, const char *jfile)
 	Fh(tl, 0, "\nextern const struct VCL_conf VCL_conf;\n");
 
 	/* Register and lex the main source */
-	VTAILQ_INSERT_TAIL(&tl->sources, sp, list);
-	sp->idx = tl->nsources++;
-	vcc_Lexer(tl, sp, 0);
-	if (tl->err)
-		return (NULL);
+	if (sp != NULL) {
+		AN(vcc_builtin);
+		VTAILQ_INSERT_TAIL(&tl->sources, sp, list);
+		sp->idx = tl->nsources++;
+		vcc_Lexer(tl, sp, 0);
+		if (tl->err)
+			return (NULL);
+	}
 
 	/* Register and lex the builtin VCL */
 	sp = vcc_new_source(tl->builtin_vcl, "Builtin");
@@ -808,15 +813,6 @@ vcc_CompileSource(struct vcc *tl, struct source *sp, const char *jfile)
 	if (tl->err)
 		return (NULL);
 
-	/* Check if we have any backends at all */
-	if (tl->default_director == NULL) {
-		VSB_cat(tl->sb,
-		    "No backends or directors found in VCL program, "
-		    "at least one is necessary.\n");
-		tl->err = 1;
-		return (NULL);
-	}
-
 	/* Check for orphans */
 	if (vcc_CheckReferences(tl))
 		return (NULL);
@@ -829,6 +825,18 @@ vcc_CompileSource(struct vcc *tl, struct source *sp, const char *jfile)
 	if (vcc_CheckUses(tl) || tl->err)
 		return (NULL);
 
+	if (vcc_builtin == NULL)
+		return (NULL);
+
+	/* Check if we have any backends at all */
+	if (tl->default_director == NULL) {
+		VSB_cat(tl->sb,
+		    "No backends or directors found in VCL program, "
+		    "at least one is necessary.\n");
+		tl->err = 1;
+		return (NULL);
+	}
+
 	/* Tie vcl_init/fini in */
 	ifp = New_IniFin(tl);
 	VSB_cat(ifp->ini, "\tVGC_function_vcl_init(ctx, VSUB_STATIC, NULL);\n");
@@ -881,6 +889,19 @@ vcc_CompileSource(struct vcc *tl, struct source *sp, const char *jfile)
 	return (vsb);
 }
 
+static struct vcc *
+vcc_ParseBuiltin(struct vcc *tl)
+{
+	struct vcc *tl_builtin;
+
+	CHECK_OBJ_NOTNULL(tl, VCC_MAGIC);
+	tl_builtin = VCC_New();
+	AN(tl_builtin);
+	VCC_Builtin_VCL(tl_builtin, tl->builtin_vcl);
+	AZ(vcc_CompileSource(tl_builtin, NULL, NULL));
+	return (tl_builtin);
+}
+
 /*--------------------------------------------------------------------
  * Report the range of VCL language we support
  */
@@ -913,6 +934,16 @@ VCC_Compile(struct vcc *tl, struct vsb **sb,
 	AN(vclsrcfile);
 	AN(ofile);
 	AN(jfile);
+
+	AZ(vcc_builtin);
+	vcc_builtin = vcc_ParseBuiltin(tl);
+	AN(vcc_builtin);
+	if (vcc_builtin->err) {
+		AZ(VSB_finish(vcc_builtin->sb));
+		*sb = vcc_builtin->sb;
+		return (-1);
+	}
+
 	if (vclsrc != NULL)
 		sp = vcc_new_source(vclsrc, vclsrcfile);
 	else
diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h
index 2bdc7e480..d261302b1 100644
--- a/lib/libvcc/vcc_compile.h
+++ b/lib/libvcc/vcc_compile.h
@@ -288,6 +288,8 @@ struct vcc {
 	unsigned		vmod_count;
 };
 
+extern struct vcc *vcc_builtin;
+
 struct method {
 	const char		*name;
 	unsigned		ret_bitmap;

From nils.goroll at uplex.de  Wed Mar  3 10:15:04 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed,  3 Mar 2021 10:15:04 +0000 (UTC)
Subject: [master] 28590eef0 vcc: Allow arbitrary subs in the built-in VCL
Message-ID: <20210303101504.5CD7C104705@lists.varnish-cache.org>


commit 28590eef0439b6df6e16846f3fe5210b7b38363f
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Mon Dec 14 06:39:57 2020 +0100

    vcc: Allow arbitrary subs in the built-in VCL
    
    Now that we know in advance the available suboutines in the built-in VCL
    we can use that as the condition to create an append-able subroutine
    when the vcl_ prefix is encountered.
    
    This work is currently incomplete, at least because non-state built-in
    subroutines are currently not listed when an unknwon subroutine is
    encountered.
    
    Also, instead of proper test coverage this currently uses b00000.vtc as
    a strawman to show how to skip a specific processing.

diff --git a/bin/varnishd/builtin.vcl b/bin/varnishd/builtin.vcl
index 3fee28f52..f975fb4d3 100644
--- a/bin/varnishd/builtin.vcl
+++ b/bin/varnishd/builtin.vcl
@@ -65,13 +65,21 @@ sub vcl_recv {
 		# We only deal with GET and HEAD by default
 		return (pass);
 	}
-	if (req.http.Authorization || req.http.Cookie) {
+	if (req.http.Authorization) {
 		# Not cacheable by default
 		return (pass);
 	}
+	call vcl_req_cookie;
 	return (hash);
 }
 
+sub vcl_req_cookie {
+	if (req.http.Cookie) {
+		# Risky to cache by default
+		return (pass);
+	}
+}
+
 sub vcl_pipe {
 	# By default "Connection: close" is set on all piped requests, to stop
 	# connection reuse from sending future requests directly to the
diff --git a/bin/varnishtest/tests/c00000.vtc b/bin/varnishtest/tests/c00000.vtc
new file mode 100644
index 000000000..9da814c44
--- /dev/null
+++ b/bin/varnishtest/tests/c00000.vtc
@@ -0,0 +1,32 @@
+varnishtest "Built-in split subroutine"
+
+server s1 {
+	rxreq
+	txresp -hdr "age: 12" \
+	    -hdr "cache-control: public, max-age=10, stale-while-revalidate=20"
+} -start
+
+varnish v1 -vcl+backend {
+	sub vcl_req_cookie {
+		return; # trust beresp headers
+	}
+
+	sub vcl_beresp_stale {
+		if (beresp.ttl + beresp.grace > 0s) {
+			return; # cache stale responses
+		}
+	}
+} -start
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.status == 200
+
+	txreq -hdr "cookie: unrelated=analytics"
+	rxresp
+	expect resp.status == 200
+} -run
+
+varnish v1 -expect cache_hit == 1
+varnish v1 -expect cache_hit == cache_hit_grace
diff --git a/lib/libvcc/vcc_parse.c b/lib/libvcc/vcc_parse.c
index dc1d55c5b..41fe8dd03 100644
--- a/lib/libvcc/vcc_parse.c
+++ b/lib/libvcc/vcc_parse.c
@@ -219,7 +219,7 @@ vcc_Compound(struct vcc *tl)
 static void
 vcc_ParseFunction(struct vcc *tl)
 {
-	struct symbol *sym;
+	struct symbol *sym, *bsym;
 	struct token *t;
 	struct proc *p;
 
@@ -231,9 +231,20 @@ vcc_ParseFunction(struct vcc *tl)
 	sym = VCC_SymbolGet(tl, SYM_MAIN, SYM_SUB, SYMTAB_CREATE, XREF_DEF);
 	ERRCHK(tl);
 	AN(sym);
+
+	if (vcc_builtin != NULL) {
+		vcc_builtin->t = t;
+		bsym = VCC_SymbolGet(vcc_builtin, SYM_MAIN, SYM_SUB,
+		    SYMTAB_NOERR, XREF_NONE);
+		AZ(vcc_builtin->err);
+	}
+	else
+		bsym = NULL;
+
 	p = sym->proc;
 	if (p == NULL) {
-		if ((t->b[0] == 'v'|| t->b[0] == 'V') &&
+		if (vcc_builtin != NULL && bsym == NULL &&
+		    (t->b[0] == 'v'|| t->b[0] == 'V') &&
 		    (t->b[1] == 'c'|| t->b[1] == 'C') &&
 		    (t->b[2] == 'l'|| t->b[2] == 'L') &&
 		    (t->b[3] == '_')) {
@@ -241,10 +252,10 @@ vcc_ParseFunction(struct vcc *tl)
 			    " are reserved for subroutines.\n");
 			vcc_ErrWhere(tl, t);
 			VSB_printf(tl->sb, "Valid vcl_* subroutines are:\n");
-			VTAILQ_FOREACH(p, &tl->procs, list) {
-				if (p->method != NULL)
-					VSB_printf(tl->sb, "\t%s\n",
-					    p->method->name);
+			VTAILQ_FOREACH(p, &vcc_builtin->procs, list) {
+				t = p->name;
+				VSB_printf(tl->sb, "\t%.*s\n",
+				    (int)pdiff(t->b, t->e), t->b);
 			}
 			return;
 		}
@@ -252,7 +263,7 @@ vcc_ParseFunction(struct vcc *tl)
 		p = vcc_NewProc(tl, sym);
 		p->name = t;
 		VSB_printf(p->cname, "%s", sym->lname);
-	} else if (p->method == NULL) {
+	} else if (p->method == NULL && bsym == NULL) {
 		VSB_printf(tl->sb, "Subroutine '%s' redefined\n", sym->name);
 		vcc_ErrWhere(tl, t);
 		VSB_printf(tl->sb, "Previously defined here:\n");
@@ -260,7 +271,6 @@ vcc_ParseFunction(struct vcc *tl)
 		return;
 	} else {
 		/* Add to VCL sub */
-		AN(p->method);
 		if (p->name == NULL)
 			p->name = t;
 	}

From nils.goroll at uplex.de  Wed Mar  3 10:15:04 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed,  3 Mar 2021 10:15:04 +0000 (UTC)
Subject: [master] 90127a475 builtin: Split vcl_recv in logical chunks
Message-ID: <20210303101504.83D59104711@lists.varnish-cache.org>


commit 90127a47534f3f23eed42a8fe15aca09e38000d0
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Jan 26 10:55:59 2021 +0100

    builtin: Split vcl_recv in logical chunks
    
    There is a very slight breaking change in the sense that the host header
    check is grouped with its normalization which swaps its order with the
    PRI method check.
    
    In practice that only means that if you have both an unattended PRI
    request missing a host header, you'll get a 400 instead of a 405. You
    have to get both wrong in the first place so I don't consider this a
    concern.

diff --git a/bin/varnishd/builtin.vcl b/bin/varnishd/builtin.vcl
index f975fb4d3..96aeb6474 100644
--- a/bin/varnishd/builtin.vcl
+++ b/bin/varnishd/builtin.vcl
@@ -36,19 +36,30 @@ vcl 4.0;
 # Client side
 
 sub vcl_recv {
+	call vcl_req_host;
+	call vcl_req_method;
+	call vcl_req_authorization;
+	call vcl_req_cookie;
+	return (hash);
+}
+
+sub vcl_req_host {
 	if (req.http.host) {
 		set req.http.host = req.http.host.lower();
 	}
-	if (req.method == "PRI") {
-		# This will never happen in properly formed traffic (see: RFC7540)
-		return (synth(405));
-	}
 	if (!req.http.host &&
 	    req.esi_level == 0 &&
 	    req.proto ~ "^(?i)HTTP/1.1") {
 		# In HTTP/1.1, Host is required.
 		return (synth(400));
 	}
+}
+
+sub vcl_req_method {
+	if (req.method == "PRI") {
+		# This will never happen in properly formed traffic.
+		return (synth(405));
+	}
 	if (req.method != "GET" &&
 	    req.method != "HEAD" &&
 	    req.method != "PUT" &&
@@ -60,22 +71,22 @@ sub vcl_recv {
 		# Non-RFC2616 or CONNECT which is weird.
 		return (pipe);
 	}
-
 	if (req.method != "GET" && req.method != "HEAD") {
-		# We only deal with GET and HEAD by default
+		# We only deal with GET and HEAD by default.
 		return (pass);
 	}
+}
+
+sub vcl_req_authorization {
 	if (req.http.Authorization) {
-		# Not cacheable by default
+		# Not cacheable by default.
 		return (pass);
 	}
-	call vcl_req_cookie;
-	return (hash);
 }
 
 sub vcl_req_cookie {
 	if (req.http.Cookie) {
-		# Risky to cache by default
+		# Risky to cache by default.
 		return (pass);
 	}
 }

From nils.goroll at uplex.de  Wed Mar  3 10:15:04 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed,  3 Mar 2021 10:15:04 +0000 (UTC)
Subject: [master] 7d2e14dff builtin: Split vcl_backend_response too
Message-ID: <20210303101504.B04EC104715@lists.varnish-cache.org>


commit 7d2e14dff7dc3761e596113c78e6551a721ef7de
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Jan 26 11:32:12 2021 +0100

    builtin: Split vcl_backend_response too
    
    It's one more indirection than vcl_recv because the same logic applies
    to multiple cases, so we first have subs for the different conditions
    and they all call the shared vcl_beresp_hitmiss sub.
    
    The vcl_beresp_hitmiss sub returns on purpose, that doesn't change the
    default VCL behavior and makes it more usable from VCL code:
    
        if (some user condition applies) {
                call vcl_beresp_hitmiss;
        }
    
    No need to force all call sites to return(deliver) when the desired
    outcome is explicit. Extracting this sub also enables VCL authors to
    only tweak the hitmiss TTL in one place.
    
    For better compliance, we might also introduce a vcl_beresp_hitpass sub
    for some of the built-in cases where a hit-for-miss object might be less
    appropriate. The compliance effort will however change some semantics
    and is outside of the scope of this split.

diff --git a/bin/varnishd/builtin.vcl b/bin/varnishd/builtin.vcl
index 96aeb6474..eb7c9272a 100644
--- a/bin/varnishd/builtin.vcl
+++ b/bin/varnishd/builtin.vcl
@@ -168,16 +168,42 @@ sub vcl_backend_response {
 	if (bereq.uncacheable) {
 		return (deliver);
 	}
-	if (beresp.ttl <= 0s ||
-	    beresp.http.Set-Cookie ||
-	    beresp.http.Surrogate-control ~ "(?i)no-store" ||
+	call vcl_beresp_stale;
+	call vcl_beresp_cookie;
+	call vcl_beresp_control;
+	call vcl_beresp_vary;
+	return (deliver);
+}
+
+sub vcl_beresp_stale {
+	if (beresp.ttl <= 0s) {
+		call vcl_beresp_hitmiss;
+	}
+}
+
+sub vcl_beresp_cookie {
+	if (beresp.http.Set-Cookie) {
+		call vcl_beresp_hitmiss;
+	}
+}
+
+sub vcl_beresp_control {
+	if (beresp.http.Surrogate-control ~ "(?i)no-store" ||
 	    (!beresp.http.Surrogate-Control &&
-	      beresp.http.Cache-Control ~ "(?i:no-cache|no-store|private)") ||
-	    beresp.http.Vary == "*") {
-		# Mark as "Hit-For-Miss" for the next 2 minutes
-		set beresp.ttl = 120s;
-		set beresp.uncacheable = true;
+	      beresp.http.Cache-Control ~ "(?i:no-cache|no-store|private)")) {
+		call vcl_beresp_hitmiss;
 	}
+}
+
+sub vcl_beresp_vary {
+	if (beresp.http.Vary == "*") {
+		call vcl_beresp_hitmiss;
+	}
+}
+
+sub vcl_beresp_hitmiss {
+	set beresp.ttl = 120s;
+	set beresp.uncacheable = true;
 	return (deliver);
 }
 

From nils.goroll at uplex.de  Wed Mar  3 10:15:04 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed,  3 Mar 2021 10:15:04 +0000 (UTC)
Subject: [master] 3efdc84bc doc: First stab at built-in VCL split documentation
Message-ID: <20210303101504.ED89010471B@lists.varnish-cache.org>


commit 3efdc84bc923e1d11f57a41577a8e6e0f0a2a7d1
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Fri Jan 29 20:16:47 2021 +0100

    doc: First stab at built-in VCL split documentation

diff --git a/doc/sphinx/users-guide/vcl-built-in-code.rst b/doc/sphinx/users-guide/vcl-built-in-code.rst
new file mode 100644
index 000000000..2bd72ed50
--- /dev/null
+++ b/doc/sphinx/users-guide/vcl-built-in-code.rst
@@ -0,0 +1,87 @@
+.. _vcl-built-in-code:
+
+Built-in VCL
+============
+
+Whenever a VCL program is loaded, the built-in VCL is appended to it. The
+:ref:`vcl-built-in-subs` have a special property, they can appear multiple
+times and the result is concatenation of all built-in subroutines.
+
+For example, let's take the following snippet::
+
+    sub vcl_recv {
+        # loaded code for vcl_recv
+    }
+
+The effective VCL that is supplied to the compiler looks like::
+
+    sub vcl_recv {
+        # loaded code for vcl_recv
+        # built-in code for vcl_recv
+    }
+
+This is how it is guaranteed that all :ref:`reference-states` have at least
+one ``return (<action>)``.
+
+It is generally recommended not to invariably return from loaded code to
+let Varnish execute the built-in code, because the built-in code provides
+essentially a sensible default behavior for an HTTP cache.
+
+Built-in subroutines split
+--------------------------
+
+It might however not always be practical that the built-in VCL rules take
+effect at the very end of a state, so some subroutines like ``vcl_recv``
+are split into multiple calls to other subroutines.
+
+By convention, those assistant subroutines are named after the variable
+they operate on, like ``req`` or ``beresp``. This allows for instance to
+circumvent default behavior.
+
+For example, ``vcl_recv`` in the built-in VCL prevents caching when clients
+have a cookie. If you can trust your backend to always specify whether a
+response is cacheable or not regardless of whether the request contained a
+cookie you can do this::
+
+    sub vcl_req_cookie {
+        return;
+    }
+
+With this, all other default behaviors from the built-in ``vcl_recv`` are
+executed and only cookie handling is affected.
+
+Another example is how the built-in ``vcl_backend_response`` treats a
+negative TTL as a signal not to cache. It's a historical mechanism to mark
+a response as uncacheable, but only if the built-in ``vcl_backend_response``
+is not circumvented by a ``return (<action>)``.
+
+However, in a multi-tier architecture where a backend might be another
+Varnish server, you might want to cache stale responses to allow the
+delivery of graced objects and enable revalidation on the next fetch. This
+can be done with the following snippet::
+
+    sub vcl_beresp_stale {
+        if (beresp.ttl + beresp.grace > 0s) {
+            return;
+        }
+    }
+
+This granularity, and the general goal of the built-in subroutines split
+is to allow to circumvent a specific aspect of the default rules without
+giving the entire logic up.
+
+Built-in VCL reference
+----------------------
+
+A copy of the ``builtin.vcl`` file might be provided with your Varnish
+installation but :ref:`varnishd(1)` is the reference to determine the code
+that is appended to any loaded VCL.
+
+The VCL compilation happens in two passes:
+
+- the first one compiles the built-in VCL only,
+- and the second pass compiles the concatenation of the loaded and built-in
+  VCLs.
+
+Any VCL subroutine present in the built-in VCL can be extended, in which
+case the loaded VCL code will be executed before the built-in code.
diff --git a/doc/sphinx/users-guide/vcl-built-in-subs.rst b/doc/sphinx/users-guide/vcl-built-in-subs.rst
index 2f46d8b87..b614a8428 100644
--- a/doc/sphinx/users-guide/vcl-built-in-subs.rst
+++ b/doc/sphinx/users-guide/vcl-built-in-subs.rst
@@ -5,7 +5,7 @@
 
 .. _vcl-built-in-subs:
 
-Built in subroutines
+Built-in subroutines
 ====================
 
 Various built-in subroutines are called during processing of client-
@@ -25,6 +25,8 @@ Common actions are documented in
 :ref:`user-guide-vcl_actions`. Actions specific to only one or some
 subroutines are documented herein.
 
+A default behavior is provided for all :ref:`reference-states` in the
+:ref:`vcl-built-in-code` code.
 
 client side
 -----------
diff --git a/doc/sphinx/users-guide/vcl-syntax.rst b/doc/sphinx/users-guide/vcl-syntax.rst
index b3334a4b7..2cc6fc6a5 100644
--- a/doc/sphinx/users-guide/vcl-syntax.rst
+++ b/doc/sphinx/users-guide/vcl-syntax.rst
@@ -87,13 +87,15 @@ down for, uhm, examples.
 Built in subroutines
 ~~~~~~~~~~~~~~~~~~~~
 
-Varnish has quite a few built in subroutines that are called for each
-transaction as it flows through Varnish. These builtin subroutines are all
-named ``vcl_*`` and are explained in :ref:`vcl-built-in-subs`.
+Varnish has quite a few built-in subroutines that are called for each
+transaction as it flows through Varnish. These built-in subroutines are
+all named ``vcl_*`` and are explained in :ref:`vcl-built-in-subs`.
 
-Processing in built in subroutines ends with ``return (<action>)``
+Processing in built-in subroutines ends with ``return (<action>)``
 (see :ref:`user-guide-vcl_actions`).
 
+The :ref:`vcl-built-in-code` also contains custom assistant subroutines
+called by the built-in subroutines, also prefixed with ``vcl_``.
 
 Custom subroutines
 ~~~~~~~~~~~~~~~~~~
diff --git a/doc/sphinx/users-guide/vcl.rst b/doc/sphinx/users-guide/vcl.rst
index 57e6f452b..831a24071 100644
--- a/doc/sphinx/users-guide/vcl.rst
+++ b/doc/sphinx/users-guide/vcl.rst
@@ -41,6 +41,7 @@ code commented out in the file `builtin.vcl` that ships with Varnish Cache.
 
    vcl-syntax
    vcl-built-in-subs
+   vcl-built-in-code
    vcl-variables
    vcl-actions
    vcl-backends

From nils.goroll at uplex.de  Wed Mar  3 10:23:04 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed,  3 Mar 2021 10:23:04 +0000 (UTC)
Subject: [master] c4b418a1c builtin: Minor polish
Message-ID: <20210303102304.A81EE105332@lists.varnish-cache.org>


commit c4b418a1c797d38666f72bc0c9734000af5f69d5
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Mar 3 11:21:44 2021 +0100

    builtin: Minor polish
    
    Only copy the host header if needed

diff --git a/bin/varnishd/builtin.vcl b/bin/varnishd/builtin.vcl
index eb7c9272a..8c7d2eb04 100644
--- a/bin/varnishd/builtin.vcl
+++ b/bin/varnishd/builtin.vcl
@@ -44,7 +44,7 @@ sub vcl_recv {
 }
 
 sub vcl_req_host {
-	if (req.http.host) {
+	if (req.http.host ~ "[[:upper:]]") {
 		set req.http.host = req.http.host.lower();
 	}
 	if (!req.http.host &&

From phk at FreeBSD.org  Wed Mar  3 13:00:07 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed,  3 Mar 2021 13:00:07 +0000 (UTC)
Subject: [master] c8ccfa4bb Also increase the stack for GCOVING
Message-ID: <20210303130007.6D3DA109600@lists.varnish-cache.org>


commit c8ccfa4bb213ad404f7a4eb04f01ef255c0798cd
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 3 12:58:09 2021 +0000

    Also increase the stack for GCOVING

diff --git a/bin/varnishd/mgt/mgt_param.c b/bin/varnishd/mgt/mgt_param.c
index cdd9a01af..6ead58a5f 100644
--- a/bin/varnishd/mgt/mgt_param.c
+++ b/bin/varnishd/mgt/mgt_param.c
@@ -683,7 +683,7 @@ MCF_InitParams(struct cli *cli)
 	low = sysconf(_SC_THREAD_STACK_MIN);
 	MCF_ParamConf(MCF_MINIMUM, "thread_pool_stack", "%jdb", (intmax_t)low);
 
-#if defined(__SANITIZER) || __has_feature(address_sanitizer)
+#if defined(__SANITIZER) || __has_feature(address_sanitizer) || defined(GCOVING)
 	def = 92 * 1024;
 #endif
 

From nils.goroll at uplex.de  Sun Mar  7 13:08:07 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sun,  7 Mar 2021 13:08:07 +0000 (UTC)
Subject: [master] da4554344 Double the vtc maximum macro size
Message-ID: <20210307130807.3914610D7B9@lists.varnish-cache.org>


commit da4554344757f4b080c7be092b56963b42048522
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Sun Mar 7 14:05:18 2021 +0100

    Double the vtc maximum macro size
    
    to support more realistic practical test cases like JWTs

diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c
index 017c53950..f6d696471 100644
--- a/bin/varnishtest/vtc.c
+++ b/bin/varnishtest/vtc.c
@@ -102,7 +102,7 @@ static struct macro *
 macro_def_int(const char *name, const char *fmt, va_list ap)
 {
 	struct macro *m;
-	char buf[256];
+	char buf[512];
 
 	VTAILQ_FOREACH(m, &macro_list, list)
 		if (!strcmp(name, m->name))

From nils.goroll at uplex.de  Wed Mar 10 11:07:07 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 10 Mar 2021 11:07:07 +0000 (UTC)
Subject: [master] dc5bddbd3 range: Propagate the VDP error for short ranges
Message-ID: <20210310110707.72D6D105664@lists.varnish-cache.org>


commit dc5bddbd301529b101598b644544b99ccabca12c
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Mon Mar 8 22:28:37 2021 +0100

    range: Propagate the VDP error for short ranges
    
    And fix the h2_req VDP error handling as per the VDP contract.
    
    Test case inspired by Simon. Since this is one of those test cases that
    explicitly mix two features I wasn't sure whether I wanted to make this
    an h2 test case or a range test case. Since this was ultimately a range
    bug I decided to register it in a range test case.
    
    It's not obvious what should have been authoritative here. The range VDP
    was rightfully latching an error via SC_RANGE_SHORT that is defined as an
    error-type session close reason, but VDP_DeliverObj() doesn't take that
    into account. While SC_RANGE_SHORT isn't a session/protocol error for h2
    but rather a stream error it is not obvious what VDP_DeliverObj() should
    do in the absence of a negative retval and the presence of a non-null
    sess_close.
    
    Maybe another way could be to turn enum sess_close into a struct and
    embed http1 and h2 specificities directly in struct fields. We already
    have somewhat structured information in the sess_close.h table.
    
    Refs 03f71c6e6dae

diff --git a/bin/varnishd/cache/cache_range.c b/bin/varnishd/cache/cache_range.c
index 007cf2c74..bbc40cb46 100644
--- a/bin/varnishd/cache/cache_range.c
+++ b/bin/varnishd/cache/cache_range.c
@@ -55,8 +55,10 @@ vrg_range_fini(struct vdp_ctx *vdc, void **priv)
 
 	CHECK_OBJ_NOTNULL(vdc, VDP_CTX_MAGIC);
 	CAST_OBJ_NOTNULL(vrg_priv, *priv, VRG_PRIV_MAGIC);
-	if (vrg_priv->range_off < vrg_priv->range_high)
+	if (vrg_priv->range_off < vrg_priv->range_high) {
 		Req_Fail(vrg_priv->req, SC_RANGE_SHORT);
+		vrg_priv->req->vdc->retval = -1;
+	}
 	*priv = NULL;	/* struct on ws, no need to free */
 	return (0);
 }
diff --git a/bin/varnishd/http2/cache_http2_deliver.c b/bin/varnishd/http2/cache_http2_deliver.c
index a73c9943d..8b0514228 100644
--- a/bin/varnishd/http2/cache_http2_deliver.c
+++ b/bin/varnishd/http2/cache_http2_deliver.c
@@ -95,7 +95,7 @@ h2_fini(struct vdp_ctx *vdc, void **priv)
 	if (r2->error)
 		return (0);
 
-	if (vdc->retval) {
+	if (vdc->retval < 0) {
 		r2->error = H2SE_INTERNAL_ERROR; /* XXX: proper error? */
 		H2_Send_Get(vdc->wrk, r2->h2sess, r2);
 		H2_Send_RST(vdc->wrk, r2->h2sess, r2, r2->stream, r2->error);
diff --git a/bin/varnishtest/tests/c00034.vtc b/bin/varnishtest/tests/c00034.vtc
index 3fbe5fefd..47366287e 100644
--- a/bin/varnishtest/tests/c00034.vtc
+++ b/bin/varnishtest/tests/c00034.vtc
@@ -191,3 +191,23 @@ client c1 {
 	gunzip
 	expect resp.bodylen == 100
 } -run
+
+# Test partial range with http2
+
+server s1 {
+	rxreq
+	txresp -hdr "Content-length: 3" -body "BLA"
+} -start
+
+varnish v1 -cliok "param.set feature +http2"
+varnish v1 -vcl+backend ""
+
+client c2 {
+	stream 1 {
+		txreq -hdr "range" "bytes=0-1"
+		rxresp
+		expect resp.status == 206
+		expect resp.http.content-length == 2
+		expect resp.body == BL
+	} -run
+} -run

From nils.goroll at uplex.de  Wed Mar 10 11:07:07 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 10 Mar 2021 11:07:07 +0000 (UTC)
Subject: [master] f0e979206 SQUASHME: Avoid a cache hit in the test case
Message-ID: <20210310110707.9539B105667@lists.varnish-cache.org>


commit f0e9792067102a8804efc0392656a5e34ceeb59f
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Mon Mar 8 23:06:34 2021 +0100

    SQUASHME: Avoid a cache hit in the test case
    
    When it was in its own file it worked fine...

diff --git a/bin/varnishtest/tests/c00034.vtc b/bin/varnishtest/tests/c00034.vtc
index 47366287e..d5432a430 100644
--- a/bin/varnishtest/tests/c00034.vtc
+++ b/bin/varnishtest/tests/c00034.vtc
@@ -204,7 +204,7 @@ varnish v1 -vcl+backend ""
 
 client c2 {
 	stream 1 {
-		txreq -hdr "range" "bytes=0-1"
+		txreq -url /3 -hdr "range" "bytes=0-1"
 		rxresp
 		expect resp.status == 206
 		expect resp.http.content-length == 2

From dridi.boukelmoune at gmail.com  Wed Mar 10 21:26:12 2021
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Wed, 10 Mar 2021 21:26:12 +0000 (UTC)
Subject: [master] 762a52de6 doc: Fix broken sentence
Message-ID: <20210310212612.69FBD11C837@lists.varnish-cache.org>


commit 762a52de6fd9eb0247a2c08275bf8db4b66941b4
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed Mar 10 22:24:07 2021 +0100

    doc: Fix broken sentence
    
    Better diff with the --word-diff --ignore-all-space options.

diff --git a/doc/sphinx/reference/varnish-cli.rst b/doc/sphinx/reference/varnish-cli.rst
index 80f9145d3..2cb1a9f3b 100644
--- a/doc/sphinx/reference/varnish-cli.rst
+++ b/doc/sphinx/reference/varnish-cli.rst
@@ -312,9 +312,9 @@ active until all its transactions are over.
 
 Over time, if you often refresh your VCL and keep the previous
 versions around, resource consumption will increase, you can't escape
-that. However, most of the time you want only one to pay the price only
-for the active VCL and keep older VCLs in case you'd need to rollback
-to a previous version.
+that. However, most of the time you want to pay the price only for the
+active VCL and keep older VCLs in case you'd need to rollback to a
+previous version.
 
 The VCL temperature allows you to minimize the footprint of inactive
 VCLs. Once a VCL becomes cold, Varnish will release all the resources

From dridi.boukelmoune at gmail.com  Thu Mar 11 17:24:05 2021
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Thu, 11 Mar 2021 17:24:05 +0000 (UTC)
Subject: [master] 3f3d67d9d builtin: Introduce vcl_builtin_* subs
Message-ID: <20210311172405.18080115FE1@lists.varnish-cache.org>


commit 3f3d67d9da491155bda9f1f1d4986b5228db5fd6
Author: Guillaume Quintard <guillaume at varnish-software.com>
Date:   Mon Mar 8 15:12:17 2021 -0800

    builtin: Introduce vcl_builtin_* subs
    
    Provide a set of predictable subroutines that will be called by the
    built-in vcl. Only transactional built-in subroutines are concerned,
    which excludes vcl_init and vcl_fini.
    
    Closes #3548

diff --git a/bin/varnishd/builtin.vcl b/bin/varnishd/builtin.vcl
index 8c7d2eb04..89c4c54fa 100644
--- a/bin/varnishd/builtin.vcl
+++ b/bin/varnishd/builtin.vcl
@@ -36,11 +36,15 @@ vcl 4.0;
 # Client side
 
 sub vcl_recv {
+	call vcl_builtin_recv;
+	return (hash);
+}
+
+sub vcl_builtin_recv {
 	call vcl_req_host;
 	call vcl_req_method;
 	call vcl_req_authorization;
 	call vcl_req_cookie;
-	return (hash);
 }
 
 sub vcl_req_host {
@@ -92,6 +96,7 @@ sub vcl_req_cookie {
 }
 
 sub vcl_pipe {
+	call vcl_builtin_pipe;
 	# By default "Connection: close" is set on all piped requests, to stop
 	# connection reuse from sending future requests directly to the
 	# (potentially) wrong backend. If you do want this to happen, you can
@@ -100,40 +105,72 @@ sub vcl_pipe {
 	return (pipe);
 }
 
+sub vcl_builtin_pipe {
+}
+
 sub vcl_pass {
+	call vcl_builtin_pass;
 	return (fetch);
 }
 
+sub vcl_builtin_pass {
+}
+
 sub vcl_hash {
+	call vcl_builtin_hash;
+	return (lookup);
+}
+
+sub vcl_builtin_hash {
 	hash_data(req.url);
 	if (req.http.host) {
 		hash_data(req.http.host);
 	} else {
 		hash_data(server.ip);
 	}
-	return (lookup);
 }
 
 sub vcl_purge {
+	call vcl_builtin_purge;
 	return (synth(200, "Purged"));
 }
 
+sub vcl_builtin_purge {
+}
+
 sub vcl_hit {
+	call vcl_builtin_hit;
 	return (deliver);
 }
 
+sub vcl_builtin_hit {
+}
+
 sub vcl_miss {
+	call vcl_builtin_miss;
 	return (fetch);
 }
 
+sub vcl_builtin_miss {
+}
+
 sub vcl_deliver {
+	call vcl_builtin_deliver;
 	return (deliver);
 }
 
+sub vcl_builtin_deliver {
+}
+
 #
 # We can come here "invisibly" with the following errors: 500 & 503
 #
 sub vcl_synth {
+	call vcl_builtin_synth;
+	return (deliver);
+}
+
+sub vcl_builtin_synth {
 	set resp.http.Content-Type = "text/html; charset=utf-8";
 	set resp.http.Retry-After = "5";
 	set resp.body = {"<!DOCTYPE html>
@@ -151,20 +188,28 @@ sub vcl_synth {
   </body>
 </html>
 "};
-	return (deliver);
 }
 
 #######################################################################
 # Backend Fetch
 
 sub vcl_backend_fetch {
+	call vcl_builtin_backend_fetch;
+	return (fetch);
+}
+
+sub vcl_builtin_backend_fetch {
 	if (bereq.method == "GET") {
 		unset bereq.body;
 	}
-	return (fetch);
 }
 
 sub vcl_backend_response {
+	call vcl_builtin_backend_response;
+	return (deliver);
+}
+
+sub vcl_builtin_backend_response {
 	if (bereq.uncacheable) {
 		return (deliver);
 	}
@@ -172,7 +217,6 @@ sub vcl_backend_response {
 	call vcl_beresp_cookie;
 	call vcl_beresp_control;
 	call vcl_beresp_vary;
-	return (deliver);
 }
 
 sub vcl_beresp_stale {
@@ -208,6 +252,11 @@ sub vcl_beresp_hitmiss {
 }
 
 sub vcl_backend_error {
+	call vcl_builtin_backend_error;
+	return (deliver);
+}
+
+sub vcl_builtin_backend_error {
 	set beresp.http.Content-Type = "text/html; charset=utf-8";
 	set beresp.http.Retry-After = "5";
 	set beresp.body = {"<!DOCTYPE html>
@@ -225,7 +274,6 @@ sub vcl_backend_error {
   </body>
 </html>
 "};
-	return (deliver);
 }
 
 #######################################################################

From martin at varnish-software.com  Fri Mar 12 14:07:07 2021
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Fri, 12 Mar 2021 14:07:07 +0000 (UTC)
Subject: [master] 86fc6706e Make sure resp.reason is on workspace before using
 it in vcl_synth
Message-ID: <20210312140707.F064D111631@lists.varnish-cache.org>


commit 86fc6706ea0277cd6949ab102f958a17a3d83d93
Author: Reza Naghibi <reza at naghibi.com>
Date:   Mon Mar 8 11:27:16 2021 -0500

    Make sure resp.reason is on workspace before using it in vcl_synth
    
    We can incorrectly reference resp.reason from other sources when
    jumping into vcl_synth. This also covers passing in a reason in
    vcl_backend_error.

diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index 0a729c0e4..a81b6ba41 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -85,6 +85,14 @@ VRT_synth(VRT_CTX, VCL_INT code, VCL_STRING reason)
 		return;
 	}
 
+	if (reason && !WS_Inside(ctx->ws, reason, NULL)) {
+		reason = WS_Copy(ctx->ws, reason, -1);
+		if (!reason) {
+			VRT_fail(ctx, "Workspace overflow");
+			return;
+		}
+	}
+
 	if (ctx->req == NULL) {
 		CHECK_OBJ_NOTNULL(ctx->bo, BUSYOBJ_MAGIC);
 		ctx->bo->err_code = (uint16_t)code;
diff --git a/bin/varnishtest/tests/r03546.vtc b/bin/varnishtest/tests/r03546.vtc
new file mode 100644
index 000000000..20cf8a85b
--- /dev/null
+++ b/bin/varnishtest/tests/r03546.vtc
@@ -0,0 +1,21 @@
+varnishtest "Synth resp.reason race"
+
+varnish v1 -vcl {
+	backend default none;
+
+	sub vcl_backend_error {
+		set beresp.status = 500;
+		set beresp.reason = "VCL";
+	}
+
+	sub vcl_deliver {
+		return (synth(resp.status, resp.reason));
+	}
+} -start
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.status == 500
+	expect resp.reason == "VCL"
+} -run

From nils.goroll at uplex.de  Sat Mar 13 19:59:07 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sat, 13 Mar 2021 19:59:07 +0000 (UTC)
Subject: [master] 3310f8a38 wip
Message-ID: <20210313195907.B87C9119831@lists.varnish-cache.org>


commit 3310f8a3842222cb336dabfa97a76cd1e0476f21
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Fri Mar 12 14:59:51 2021 +0100

    wip

diff --git a/doc/changes.rst b/doc/changes.rst
index c11ba2cf5..054389186 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -38,6 +38,8 @@ Varnish Cache Next (2021-03-15)
 * counters MAIN.s_req_bodybytes and VBE.*.tools.beresp_bodybytes
   are now always the number of bodybytes moved on the wire.
 
+  .. 36e2bfe7d34fdbf75ccf67a1263f8f7bac6c4788
+
 * Unless the new ``validate_headers`` feature is disabled, all newly
   set headers are now validated to contain only characters allowed by
   RFC7230. A (runtime) VCL failure is triggered if not.

From nils.goroll at uplex.de  Sat Mar 13 19:59:07 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sat, 13 Mar 2021 19:59:07 +0000 (UTC)
Subject: [master] c3729d022 doc: changes.rst since varnish-6.5.1
Message-ID: <20210313195907.D0BB9119834@lists.varnish-cache.org>


commit c3729d022ec9d057f395af88b8a2c7245bd5fcff
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Sat Mar 13 20:56:57 2021 +0100

    doc: changes.rst since varnish-6.5.1
    
    The order matches the order of commits, except where I thought it made
    more sense to join topics into contiguous sections.

diff --git a/doc/changes.rst b/doc/changes.rst
index 054389186..c7b0a5514 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -35,14 +35,70 @@ release process.
 Varnish Cache Next (2021-03-15)
 ================================
 
-* counters MAIN.s_req_bodybytes and VBE.*.tools.beresp_bodybytes
-  are now always the number of bodybytes moved on the wire.
+* Body bytes accounting has been fixed to always represent the number
+  of bodybytes moved on the wire, exclusive of protocol-specific
+  overhead like HTTP/1 chunked encoding or HTTP/2 framing.
 
-  .. 36e2bfe7d34fdbf75ccf67a1263f8f7bac6c4788
+  This change affects counters like
+  - ``MAIN.s_req_bodybytes``,
+  - ``MAIN.s_resp_bodybytes``,
+  - ``VBE.*.*.bereq_bodybytes`` and
+  - ``VBE.*.*.beresp_bodybytes``
+
+  as well as the VSL records
+  - ``ReqAcct``,
+  - ``PipeAcct`` and
+  - ``BereqAcct``.
+
+* ``VdpAcct`` log records have been added to output delivery filter
+  (VDP) accounting details analogous to the existing ``VfpAcct``. Both
+  tags are masked by default.
+
+* Many filter (VDP/VFP) related signatures have been changed:
+  - ``vdp_init_f``
+  - ``vdp_fini_f``
+  - ``vdp_bytes_f``
+  - ``VDP_bytes()``
+
+  as well as ``struct vdp_entry`` and ``struct vdp_ctx``
+
+  ``VFP_Push()`` and ``VDP_Push()`` are no longer intended for VMOD
+  use and have been removed from the API.
+
+* The VDP code is now more strict about ``VDP_END``, which must be
+  sent down the filter chain at most once.
+
+* Core code has been changed to ensure for most cases that ``VDP_END``
+  gets signaled with the object's last bytes, rather than with an
+  extra zero-data call.
+
+* Reason phrases for more HTTP Status codes have been added to core
+  code.
+
+* Connection pooling behavior has been improved with respect to
+  ``Connection: close`` (3400_, 3405_).
+
+* Handling of the ``Keep-Alive`` HTTP header as hop-by-hop has been
+  fixed (3417_).
+
+* Handling of hop-by-hop headers has been fixed for HTTP/2 (3416_).
+
+* The stevedore API has been changed:
+  - ``OBJ_ITER_FINAL`` has been renamed to ``OBJ_ITER_END``
+  - ``ObjExtend()`` signature has been changed to also cover the
+    ``ObjTrimStore()`` use case and
+  - ``ObjTrimStore()`` has been removed.
+
+* The ``verrno.h`` header file has been removed and merged into
+  ``vas.h``
+
+* The connection close reason has been fixed to properly report
+  ``SC_RESP_CLOSE`` / ``resp_close`` where previously only
+  ``SC_REQ_CLOSE`` / ``req_close`` was reported.
 
 * Unless the new ``validate_headers`` feature is disabled, all newly
   set headers are now validated to contain only characters allowed by
-  RFC7230. A (runtime) VCL failure is triggered if not.
+  RFC7230. A (runtime) VCL failure is triggered if not (3407_).
 
 * ``VRT_ValidHdr()`` has been added for vmods to conduct the same
   check as the ``validate_headers`` feature, for example when headers
@@ -63,6 +119,259 @@ Varnish Cache Next (2021-03-15)
 * All shard ``Error`` and ``Notice`` messages now use the unified
   prefix ``vmod_directors: shard %s``.
 
+* In the shard director, use of parameter sets with ``resolve=NOW``
+  has been fixed.
+
+* Performance of log-processing tools like ``varnishlog`` has been
+  improved by using ``mmap()`` if possible when reading from log
+  files.
+
+* An assertion failure has been fixed which could be triggered when a
+  request body was used with restarts (3433_, 3434_).
+
+* A signal handling bug in the Varnish Utility API (VUT) has been
+  fixed which caused log-processing utilities to perform poorly after
+  a signal had been received (3436_).
+
+* The ``client.identity`` variable is now accessible on the backend
+  side.
+
+* Client and backend finite state machine internals (``enum req_step``
+  and ``enum fetch_step``) have been removed from ``cache.h``.
+
+* Three new ``Timestamp`` VSL records have been added to backend
+  request processing:
+
+  - The ``Process`` timestamp after ``return(deliver)`` or
+    ``return(pass(x))`` from ``vcl_backend_response``,
+
+  - the ``Fetch`` timestamp before a backend connection is requested
+    and
+
+  - the ``Connected`` timestamp when a connection to a regular backend
+    (VBE) is established.
+
+* The VRT backend interface has been changed:
+
+  - ``struct vrt_endpoint`` has been added describing a UDS or TCP
+    endpoint for a backend to connect to.
+
+    Endpoints also support a preamble to be sent with every new
+    connection.
+
+  - This structure needs to be passed via the ``endpoint`` member of
+    ``struct vrt_backend`` when creating backends with
+    ``VRT_new_backend()`` or ``VRT_new_backend_clustered()``.
+
+* ``VRT_Endpoint_Clone()`` has been added to facilitate working with
+  endpoints.
+
+* The variables ``bereq.is_hitpass`` and ``bereq.is_hitmiss`` have
+  been added to the backend side matching ``req.is_hitpass`` and
+  ``req.is_hitmiss`` on the client side.
+
+* The ``set_ip_tos()`` function from the bundled ``std`` vmod now sets
+  the IPv6 Taffic Class (TCLASS) when used on an IPv6 connection.
+
+* A bug has been fixed which could lead to varnish failing to start
+  after updates due to outdated content of the ``vmod_cache``
+  directory (3243_).
+
+* An issue has been addressed where using VCL with a high number of
+  literal strings could lead to prolonged c-compiler runtimes since
+  Varnish-Cache 6.3 (3392_).
+
+* The ``MAIN.esi_req`` counter has been added as a statistic of the
+  number of ESI sub requests created.
+
+* The ``vcl.discard`` CLI command can now be used to discard more than
+  one VCL with a single command, which succeeds only if all given VCLs
+  could be discarded (atomic behavior).
+
+* The ``vcl.discard`` CLI command now supports glob patterns for vcl names.
+
+* The ``vcl.deps`` CLI command has been added to output dependencies
+  between VCLs (because of labels and ``return(vcl)`` statements).
+
+* The ``FetchError`` log message ``Timed out reusing backend
+  connection`` has been renamed to ``first byte timeout (reused
+  connection)`` to clarify that it is emit for effectively the same
+  reason as ``first byte timeout``.
+
+* Long strings in VCL can now also be denoted using ``""" ... """`` in
+  addition to the existing ``{" ... "}``.
+
+* The ``pdiff()`` function declaration has been moved from ``cache.h``
+  to ``vas.h``.
+
+* The interface for private pointers in VMODs has been changed:
+
+  - The ``free`` pointer in ``struct vmod_priv`` has been replaced
+    with a pointer to ``struct vmod_priv_methods``, to where the
+    pointer to the former free callback has been moved as the ``fini``
+    member.
+
+  - The former free callback type has been renamed from
+    ``vmod_priv_free_f`` to ``vmod_priv_fini_f`` and as gained a
+    ``VRT_CTX`` argument
+
+* The ``MAIN.s_bgfetch`` counter has been added as a statistic on the
+  number of background fetches issues.
+
+* Various improvements have been made to the ``varnishtest`` facility:
+
+  - the ``loop`` keyword now works everywhere
+
+  - HTTP/2 logging has been improved
+
+  - Default HTTP/2 parameters have been tweaked (3442_)
+
+  - Varnish listen address information is now available by default in
+    the macros ``${vNAME_addr}``, ``${vNAME_port}`` and
+    ``${vNAME_sock}``. Macros by the names ``${vNAME_SOCKET_*}``
+    contain the address information for each listen socket as created
+    with the ``-a`` argument to ``varnishd``.
+
+  - Synchronization points for counters (VSCs) have been added as
+    ``varnish vNAME -expect PATTERN OP PATTERN``
+
+  - varnishtest now also works with IPv6 setups
+
+  - ``feature ipv4`` and ``feature ipv6`` can be used to control
+    execution of test cases which require one or the other protocol.
+
+  - haproxy arguments can now be externally provided through the
+    ``HAPROXY_ARGS`` variable.
+
+  - logexpect now supports alternatives with the ``expect ? ...`` syntax
+    and negative matches with the ``fail add ...`` and ``fail clear``
+    syntax.
+
+  - The overall logexpect match expectation can now be inverted using
+    the ``-err`` argument.
+
+  - Numeric comparisons for HTTP headers have been added: ``-lt``,
+    ``-le``, ``-eq``, ``-ne``, ``-ge``, ``-gt``
+
+  - ``rxdata -some`` has been fixed.
+
+* The ``ban_cutoff`` parameter now refers to the overall length of the
+  ban list, including completed bans, where before only non-completed
+  ("active") bans were counted towards ``ban_cutoff``.
+
+* A race in the round-robin director has been fixed which could lead
+  to backend requests failing when backends in the director were sick
+  (3473_).
+
+* A race in the probe management has been fixed which could lead to a
+  panic when VCLs changed temperature in general and when
+  ``vcl.discard`` was used in particular (3362_).
+
+* A bug has been fixed which lead to counters (VSCs) of backends from
+  cold VCLs being presented (3358_).
+
+* A bug in ``varnishncsa`` has been fixed which could lead to it
+  crashing when header fields were referenced which did not exist in
+  the processed logs (3485_).
+
+* For failing PROXY connections, ``SessClose`` now provides more
+  detailed information on the cause of the failure.
+
+* The ``std.ban()`` and ``std.ban_error()`` functions have been added
+  to the ``std`` vmod, allowing VCL to check for ban errors.
+
+* Use of the ``ban()`` built-in VCL command is now deprecated.
+
+* The source tree has been reorganized with all vmods now moved to a
+  single ``vmod`` directory.
+
+* ``vmodtool.py`` has been improved to simplify Makefiles when many
+  VMODs are built in a single directory.
+
+* The ``VSA_getsockname()`` and ``VSA_getpeername()`` functions have
+  been added to get address information of file descriptors.
+
+* ``varnishd`` now supports the ``-b None`` argument to start with
+  only the builtin VCL and no backend at all (3067_).
+
+* Some corner cases of IPv6 support in ``varnishd`` have been fixed.
+
+* ``vcl_pipe {}``: ``return(synth)`` and vmod private state support
+  have been fixed. Trying to use ``std.rollback()`` from ``vcl_pipe``
+  now results in VCL failure (3329_, 3330_, 3385_).
+
+* The ``bereq.xid`` variable is now also available in ``vcl_pipe {}``
+
+* The ``VRT_priv_task_get()`` and ``VRT_priv_top_get()`` functions
+  have been added to VRT to allow vmods to retrieve existing
+  ``PRIV_TASK`` / ``PRIV_TOP`` private pointers without creating any.
+
+* ``varnishstat`` now avoids display errors of gauges which previously
+  could underflow to negative values, being displayed as extremely
+  high positive values.
+
+  The ``-r`` option and the ``r`` key binding have been added to
+  return to the previous behavior. When raw mode is active in
+  ``varnishstat`` interactive (curses) mode, the word ``RAW`` is
+  displayed at the right hand side in the lower status line.
+
+* The ``VSC_IsRaw()`` function has been added to ``libvarnishapi`` to
+  query if a gauge is being returned raw or adjusted.
+
+* The ``busy_stats_rate`` feature flag has been added to ensure
+  statistics updates (as configured using the ``thread_stats_rate``
+  parameter) even on a fully loaded system, which would otherwise
+  delay statistics updates in order to reduce lock contention.
+
+* ``ExpKill`` log (VSL) records are now masked by default. See the
+  ``vsl_mask`` parameter.
+
+* A bug has been fixed which could lead to panics when ESI was used
+  with ESI-aware VMODs were used because ``PRIV_TOP`` vmod private
+  state was created on a wrong workspace (3496_).
+
+* The ``VCL_REGEX`` data type is now supported for VMODs, allowing
+  them to use regular expression literals checked and compiled by the
+  VCL compiler infrastructure.
+
+  Consequently, the ``VRT_re_init()`` and ``VRT_re_fini()`` functions
+  have been removed, because they are not required and their use was
+  probably wrong anyway.
+
+* The ``%{X}T`` format has been added to ``varnishncsa``, which
+  generalizes ``%D`` and ``%T``, but also support milliseconds
+  (``ms``) output.
+
+* Error handling has been fixed when vmod functions/methods with
+  ``PRIV_TASK`` arguments were wrongly called from the backend side
+  (3498_).
+
+* The ``varnishncsa`` ``-E`` argument to show ESI requests has been
+  changed to imply ``-c`` (client mode).
+
+* Error handling and performance of the VSL (shared log) client code
+  in ``libvarnishapi`` have been improved (3501_).
+
+* ``varnishlog`` now supports the ``-u`` option to write to a file
+  specified with ``-w`` unbuffered.
+
+* Comparisons of numbers in VSL queries have been improved to match
+  better the behavior which is likely expected by users who have not
+  read the documentation in all detail (3463_).
+
+* A bug in the ESI code has been fixed which could trigger a panic
+  when no storage space was available (3502_).
+
+* The ``resp.proto`` variable is now read-only as it should have been
+  for long.
+
+* ``VTCP_open()`` has been fixed to try all possible addresses from
+  the resolver before giving up (3509_). This bug could cause
+  confusing error messages (3510_).
+
+* ``VRT_synth_blob()`` and ``VRT_synth_strands()`` have been
+  added. The latter should now be used instead of ``VRT_synth_page()``.
+
 * The ``VCL_SUB`` data type is now supported for VMODs to save
   references to subroutines to be called later using
   ``VRT_call()``. Calls from a wrong context (e.g. calling a
@@ -73,6 +382,9 @@ Varnish Cache Next (2021-03-15)
 
 .. _VMOD - Varnish Modules: https://varnish-cache.org/docs/trunk/reference/vmod.html
 
+  VMOD functions can also return the ``VCL_SUB`` data type for calls
+  from VCL as in ``call vmod.returning_sub();``.
+
 * ``VRT_check_call()`` can be used to check if a ``VRT_call()`` would
   succeed in order to avoid the potential VCL failure in case it would
   not.
@@ -80,6 +392,49 @@ Varnish Cache Next (2021-03-15)
   It returns ``NULL`` if ``VRT_call()`` would make the call or an
   error string why not.
 
+* ``VRT_handled()`` has been added, which is now to be used instead of
+  access to the ``handling`` member of ``VRT_CTX``.
+
+* The session close reason logging/statistics for HTTP/2 connections
+  have been improved (3393_)
+
+* ``varnishadm`` now has the ``-p`` option to disable readline support
+  for use in scripts and as a generic CLI connector.
+
+* A log (VSL) ``Notice`` record is now emitted whenever more than
+  ``vary_notice`` variants are encountered in the cache for a specific
+  hash. The new ``vary_notice`` parameter defaults to 10.
+
+* The modulus operator ``%`` has been added to VCL.
+
+* ``return(retry)`` from ``vcl_backend_error {}`` now correctly resets
+  ``beresp.status`` and ``beresp.reason`` (3525_).
+
+* Handling of the ``gunzip`` filter with ESI has been fixed (3529_).
+
+* A bug where the ``threads_limited`` counter could be increased
+  without reason has been fixed (3531_).
+
+* All varnish tools using the VUT library utilities for argument
+  processing now support the ``--optstring`` argument to return a
+  string suitable for use with ``getopts`` from shell scripts.
+
+* An issue with high CPU consumption when the maximum number of
+  threads was reached has been fixed (2942_, 3531_)
+
+* HTTP/2 streams are now reset for filter chain (VDP) errors.
+
+* The task priority of incoming connections has been fixed.
+
+* An issue has been addressed where the watchdog facility could
+  misfire when tasks are queued.
+
+* The builtin VCL has been reworked: VCL code has been split into
+  small subroutines, which custom VCL can prepend custom code to.
+
+  This allows for better integration of custom VCL and the built-in
+  VCL and better reuse.
+
 ================================
 Varnish Cache 6.5.1 (2020-09-25)
 ================================

From nils.goroll at uplex.de  Sat Mar 13 20:05:12 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sat, 13 Mar 2021 20:05:12 +0000 (UTC)
Subject: [master] 296897206 doc: changes.rst formatting and github references
Message-ID: <20210313200512.2B6DB119D48@lists.varnish-cache.org>


commit 2968972069b78f6b0429e0ced675ed51bd9cc49e
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Sat Mar 13 21:04:31 2021 +0100

    doc: changes.rst formatting and github references

diff --git a/doc/changes.rst b/doc/changes.rst
index c7b0a5514..c85070054 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -84,9 +84,12 @@ Varnish Cache Next (2021-03-15)
 * Handling of hop-by-hop headers has been fixed for HTTP/2 (3416_).
 
 * The stevedore API has been changed:
+
   - ``OBJ_ITER_FINAL`` has been renamed to ``OBJ_ITER_END``
+
   - ``ObjExtend()`` signature has been changed to also cover the
     ``ObjTrimStore()`` use case and
+
   - ``ObjTrimStore()`` has been removed.
 
 * The ``verrno.h`` header file has been removed and merged into
@@ -435,6 +438,38 @@ Varnish Cache Next (2021-03-15)
   This allows for better integration of custom VCL and the built-in
   VCL and better reuse.
 
+.. _2942: https://github.com/varnishcache/varnish-cache/issues/2942
+.. _3067: https://github.com/varnishcache/varnish-cache/issues/3067
+.. _3243: https://github.com/varnishcache/varnish-cache/issues/3243
+.. _3329: https://github.com/varnishcache/varnish-cache/issues/3329
+.. _3330: https://github.com/varnishcache/varnish-cache/issues/3330
+.. _3358: https://github.com/varnishcache/varnish-cache/issues/3358
+.. _3362: https://github.com/varnishcache/varnish-cache/issues/3362
+.. _3385: https://github.com/varnishcache/varnish-cache/issues/3385
+.. _3392: https://github.com/varnishcache/varnish-cache/issues/3392
+.. _3393: https://github.com/varnishcache/varnish-cache/issues/3393
+.. _3400: https://github.com/varnishcache/varnish-cache/issues/3400
+.. _3405: https://github.com/varnishcache/varnish-cache/issues/3405
+.. _3407: https://github.com/varnishcache/varnish-cache/issues/3407
+.. _3416: https://github.com/varnishcache/varnish-cache/issues/3416
+.. _3417: https://github.com/varnishcache/varnish-cache/issues/3417
+.. _3433: https://github.com/varnishcache/varnish-cache/issues/3433
+.. _3434: https://github.com/varnishcache/varnish-cache/issues/3434
+.. _3436: https://github.com/varnishcache/varnish-cache/issues/3436
+.. _3442: https://github.com/varnishcache/varnish-cache/issues/3442
+.. _3463: https://github.com/varnishcache/varnish-cache/issues/3463
+.. _3473: https://github.com/varnishcache/varnish-cache/issues/3473
+.. _3485: https://github.com/varnishcache/varnish-cache/issues/3485
+.. _3496: https://github.com/varnishcache/varnish-cache/issues/3496
+.. _3498: https://github.com/varnishcache/varnish-cache/issues/3498
+.. _3501: https://github.com/varnishcache/varnish-cache/issues/3501
+.. _3502: https://github.com/varnishcache/varnish-cache/issues/3502
+.. _3509: https://github.com/varnishcache/varnish-cache/issues/3509
+.. _3510: https://github.com/varnishcache/varnish-cache/issues/3510
+.. _3525: https://github.com/varnishcache/varnish-cache/issues/3525
+.. _3529: https://github.com/varnishcache/varnish-cache/issues/3529
+.. _3531: https://github.com/varnishcache/varnish-cache/issues/3531
+
 ================================
 Varnish Cache 6.5.1 (2020-09-25)
 ================================

From nils.goroll at uplex.de  Sat Mar 13 20:08:05 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sat, 13 Mar 2021 20:08:05 +0000 (UTC)
Subject: [master] 6f0afebd2 doc: changes.rst formatting
Message-ID: <20210313200805.F1C4811A01D@lists.varnish-cache.org>


commit 6f0afebd2a33c5f556aa3e8c6009d608ea8a4b4f
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Sat Mar 13 21:07:46 2021 +0100

    doc: changes.rst formatting

diff --git a/doc/changes.rst b/doc/changes.rst
index c85070054..05d488f79 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -40,14 +40,21 @@ Varnish Cache Next (2021-03-15)
   overhead like HTTP/1 chunked encoding or HTTP/2 framing.
 
   This change affects counters like
+
   - ``MAIN.s_req_bodybytes``,
+
   - ``MAIN.s_resp_bodybytes``,
+
   - ``VBE.*.*.bereq_bodybytes`` and
+
   - ``VBE.*.*.beresp_bodybytes``
 
   as well as the VSL records
+
   - ``ReqAcct``,
+
   - ``PipeAcct`` and
+
   - ``BereqAcct``.
 
 * ``VdpAcct`` log records have been added to output delivery filter
@@ -55,9 +62,13 @@ Varnish Cache Next (2021-03-15)
   tags are masked by default.
 
 * Many filter (VDP/VFP) related signatures have been changed:
+
   - ``vdp_init_f``
+
   - ``vdp_fini_f``
+
   - ``vdp_bytes_f``
+
   - ``VDP_bytes()``
 
   as well as ``struct vdp_entry`` and ``struct vdp_ctx``

From nils.goroll at uplex.de  Sun Mar 14 15:28:06 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sun, 14 Mar 2021 15:28:06 +0000 (UTC)
Subject: [master] e291bfaa5 Document vmod cookie VCL_REGEX changes
Message-ID: <20210314152806.B0486112365@lists.varnish-cache.org>


commit e291bfaa5250b86cad600479284108bbe7458161
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Sun Mar 14 16:24:21 2021 +0100

    Document vmod cookie VCL_REGEX changes
    
    Noted by Simon Vikstr?m, thank you
    
    Closes #3551

diff --git a/doc/changes.rst b/doc/changes.rst
index 05d488f79..c9a251986 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -352,6 +352,16 @@ Varnish Cache Next (2021-03-15)
   have been removed, because they are not required and their use was
   probably wrong anyway.
 
+* The ``filter_re``, ``keep_re`` and ``get_re`` functions from the
+  bundled ``cookie`` vmod have been changed to take the ``VCL_REGEX``
+  type. This implies that their regular expression arguments now need
+  to be literal, whereas before they could be taken from some other
+  variable or function returning ``VCL_STRING``.
+
+  Note that these functions never actually handled _dynamic_ regexen,
+  the string passed with the first call was compiled to a regex, which
+  was then used for the lifetime of the respective VCL.
+
 * The ``%{X}T`` format has been added to ``varnishncsa``, which
   generalizes ``%D`` and ``%T``, but also support milliseconds
   (``ms``) output.

From nils.goroll at uplex.de  Sun Mar 14 15:39:04 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sun, 14 Mar 2021 15:39:04 +0000 (UTC)
Subject: [master] 06b9f8816 Start skeleton release notes for the next version.
Message-ID: <20210314153904.D13CE112904@lists.varnish-cache.org>


commit 06b9f8816bd16ab92fd54a99f0460a3232689ea8
Author: Geoff Simmons <geoff at uplex.de>
Date:   Tue Sep 25 16:31:17 2018 +0200

    Start skeleton release notes for the next version.
    
    Restructured so that:
    
    * 'Upgrading' is limited to work that has to be done to upgrade from
      a current deployment to the new version.
    
    * 'Changes' is a comprehensive, user-level description of changes and
      new features.
    
    Conflicts:
            doc/sphinx/whats-new/index.rst
    
    Committer Edit:
            Added Copyright header by example of
            fe8cfe8484e500596f634b16b70c1f778b9a7415 and
            934398848a20b846b6861be3843b3df17bce54d8

diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst
new file mode 100644
index 000000000..d0e90aed8
--- /dev/null
+++ b/doc/sphinx/whats-new/changes-trunk.rst
@@ -0,0 +1,78 @@
+..
+	Copyright 2021 UPLEX Nils Goroll Systemoptimierung
+	SPDX-License-Identifier: BSD-2-Clause
+	See LICENSE file for full text of license
+
+**Note: This is a working document for a future release, with running
+updates for changes in the development branch. For changes in the
+released versions of Varnish, see:** :ref:`whats-new-index`
+
+.. _whatsnew_changes_CURRENT:
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+Changes in Varnish **$NEXT_RELEASE**
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+For information about updating your current Varnish deployment to the
+new version, see :ref:`whatsnew_upgrading_CURRENT`.
+
+A more detailed and technical account of changes in Varnish, with
+links to issues that have been fixed and pull requests that have been
+merged, may be found in the `change log`_.
+
+.. _change log: https://github.com/varnishcache/varnish-cache/blob/master/doc/changes.rst
+
+varnishd
+========
+
+Parameters
+~~~~~~~~~~
+
+**XXX changes in -p parameters**
+
+Other changes in varnishd
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Changes to VCL
+==============
+
+VCL variables
+~~~~~~~~~~~~~
+
+**XXX new, deprecated or removed variables, or changed semantics**
+
+Other changes to VCL
+~~~~~~~~~~~~~~~~~~~~
+
+VMODs
+=====
+
+**XXX changes in the bundled VMODs**
+
+varnishlog
+==========
+
+**XXX changes concerning varnishlog(1) and/or vsl(7)**
+
+varnishadm
+==========
+
+**XXX changes concerning varnishadm(1) and/or varnish-cli(7)**
+
+varnishstat
+===========
+
+**XXX changes concerning varnishstat(1) and/or varnish-counters(7)**
+
+varnishtest
+===========
+
+**XXX changes concerning varnishtest(1) and/or vtc(7)**
+
+Changes for developers and VMOD authors
+=======================================
+
+**XXX changes concerning VRT, the public APIs, source code organization,
+builds etc.**
+
+*eof*
diff --git a/doc/sphinx/whats-new/index.rst b/doc/sphinx/whats-new/index.rst
index 9a62721f8..5845a4890 100644
--- a/doc/sphinx/whats-new/index.rst
+++ b/doc/sphinx/whats-new/index.rst
@@ -13,6 +13,19 @@ This section describes the changes and improvements between different
 versions of Varnish, and what upgrading between the different versions
 entail.
 
+Varnish **$NEXT_RELEASE**
+-------------------------
+
+**Note: These are working documents for a future release, with running
+updates for changes in the development branch. For changes in the
+released versions of Varnish, see the chapters listed below.**
+
+.. toctree::
+   :maxdepth: 2
+
+   changes-trunk
+   upgrading-trunk
+
 Varnish 6.5
 -----------
 
diff --git a/doc/sphinx/whats-new/upgrading-trunk.rst b/doc/sphinx/whats-new/upgrading-trunk.rst
new file mode 100644
index 000000000..950a3d3f9
--- /dev/null
+++ b/doc/sphinx/whats-new/upgrading-trunk.rst
@@ -0,0 +1,38 @@
+..
+	Copyright 2021 UPLEX Nils Goroll Systemoptimierung
+	SPDX-License-Identifier: BSD-2-Clause
+	See LICENSE file for full text of license
+
+**Note: This is a working document for a future release, with running
+updates for changes in the development branch. For changes in the
+released versions of Varnish, see:** :ref:`whats-new-index`
+
+.. _whatsnew_upgrading_CURRENT:
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+Upgrading to Varnish **$NEXT_RELEASE**
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+**XXX: how to upgrade from previous deployments to this
+version. Limited to work that has to be done for an upgrade, new
+features are listed in "Changes". Explicitly mention what does *not*
+have to be changed, especially in VCL. May include, but is not limited
+to:**
+
+* Elements of VCL that have been removed or are deprecated, or whose
+  semantics have changed.
+
+* -p parameters that have been removed or are deprecated, or whose
+  semantics have changed.
+
+* Changes in the CLI.
+
+* Changes in the output or interpretation of stats or the log, including
+  changes affecting varnishncsa/-hist/-top.
+
+* Changes that may be necessary in VTCs or in the use of varnishtest.
+
+* Changes in public APIs that may require changes in VMODs or VAPI/VUT
+  clients.
+
+*eof*

From nils.goroll at uplex.de  Sun Mar 14 17:03:04 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sun, 14 Mar 2021 17:03:04 +0000 (UTC)
Subject: [master] 00da70680 whats-new: changes
Message-ID: <20210314170304.7CBD7114E16@lists.varnish-cache.org>


commit 00da70680f08eee5186f622764ee1cf7cd8d6542
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Sun Mar 14 18:02:11 2021 +0100

    whats-new: changes

diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst
index d0e90aed8..af4edb196 100644
--- a/doc/sphinx/whats-new/changes-trunk.rst
+++ b/doc/sphinx/whats-new/changes-trunk.rst
@@ -25,54 +25,468 @@ merged, may be found in the `change log`_.
 varnishd
 ========
 
+Arguments
+~~~~~~~~~
+
+* ``varnishd`` now supports the ``-b None`` argument to start with
+  only the builtin VCL and no backend at all.
+
 Parameters
 ~~~~~~~~~~
 
-**XXX changes in -p parameters**
+* The ``validate_headers`` parameter has been added to control
+  `header validation <whatsnew_changes_CURRENT_header_validation_>`_.
+
+* The ``ban_cutoff`` parameter now refers to the overall length of the
+  ban list, including completed bans, where before only non-completed
+  ("active") bans were counted towards ``ban_cutoff``.
+
+* The ``vary_notice`` parameter has been added to control the
+  threshold for the new `Vary Notice
+  <whatsnew_changes_CURRENT_vary_notice_>`_.
+
+``feature`` Flags
+~~~~~~~~~~~~~~~~~
+
+* The ``busy_stats_rate`` feature flag has been added to ensure
+  statistics updates (as configured using the ``thread_stats_rate``
+  parameter) even on a fully loaded system, which would otherwise
+  delay statistics updates in order to reduce lock contention.
+
+.. _whatsnew_changes_CURRENT_accounting:
+
+Accounting
+~~~~~~~~~~
+
+Body bytes accounting has been fixed to always represent the number of
+bodybytes moved on the wire, exclusive of protocol-specific overhead
+like HTTP/1 chunked encoding or HTTP/2 framing.
+
+This change affects counters like
+
+- ``MAIN.s_req_bodybytes``,
+
+- ``MAIN.s_resp_bodybytes``,
+
+- ``VBE.*.*.bereq_bodybytes`` and
+
+- ``VBE.*.*.beresp_bodybytes``
+
+as well as the VSL records
+
+- ``ReqAcct``,
+
+- ``PipeAcct`` and
 
-Other changes in varnishd
-~~~~~~~~~~~~~~~~~~~~~~~~~
+- ``BereqAcct``.
+
+.. _whatsnew_changes_CURRENT_sc_close:
+
+Session Close Reasons
+~~~~~~~~~~~~~~~~~~~~~
+
+The connection close reason has been fixed to properly report
+``SC_RESP_CLOSE`` / ``resp_close`` where previously only
+``SC_REQ_CLOSE`` / ``req_close`` was reported.
+
+For failing PROXY connections, ``SessClose`` now provides more
+detailed information on the cause of the failure.
+
+The session close reason logging/statistics for HTTP/2 connections
+have been improved.
+
+.. _whatsnew_changes_CURRENT_vary_notice:
+
+Vary Notice
+~~~~~~~~~~~
+
+A log (VSL) ``Notice`` record is now emitted whenever more than
+``vary_notice`` variants are encountered in the cache for a specific
+hash. The new ``vary_notice`` parameter defaults to 10.
 
 Changes to VCL
 ==============
 
+.. _whatsnew_changes_CURRENT_header_validation:
+
+Header Validation
+~~~~~~~~~~~~~~~~~
+
+Unless the new ``validate_headers`` feature is disabled, all newly set
+headers are now validated to contain only characters allowed by
+RFC7230. A (runtime) VCL failure is triggered if not.
+
 VCL variables
 ~~~~~~~~~~~~~
 
-**XXX new, deprecated or removed variables, or changed semantics**
+* The ``client.identity`` variable is now accessible on the backend
+  side.
+
+* The variables ``bereq.is_hitpass`` and ``bereq.is_hitmiss`` have
+  been added to the backend side matching ``req.is_hitpass`` and
+  ``req.is_hitmiss`` on the client side.
+
+* The ``bereq.xid`` variable is now also available in ``vcl_pipe {}``
+
+* The ``resp.proto`` variable is now read-only as it should have been
+  for long.
 
 Other changes to VCL
 ~~~~~~~~~~~~~~~~~~~~
 
+* Long strings in VCL can now also be denoted using ``""" ... """`` in
+  addition to the existing ``{" ... "}``.
+
+* The ``ban()`` builtin is now deprecated and should be replaced with
+  `std.ban() <whatsnew_changes_CURRENT_ban_>`_.
+
+* Trying to use ``std.rollback()`` from ``vcl_pipe`` now results in
+  VCL failure.
+
+* The modulus operator ``%`` has been added to VCL.
+
+* ``return(retry)`` from ``vcl_backend_error {}`` now correctly resets
+  ``beresp.status`` and ``beresp.reason``.
+
+* The builtin VCL has been reworked: VCL code has been split into
+  small subroutines, which custom VCL can prepend custom code to.
+
+  This allows for better integration of custom VCL and the built-in
+  VCL and better reuse.
+
 VMODs
 =====
 
-**XXX changes in the bundled VMODs**
+``directors.shard()``
+~~~~~~~~~~~~~~~~~~~~~
+
+* The shard director now supports reconfiguration (adding/removing
+  backends) of several instances without any special ordering
+  requirement.
+
+* Calling the shard director ``.reconfigure()`` method is now
+  optional. If not called explicitly, any shard director backend
+  changes are applied at the end of the current task.
+
+* Shard director ``Error`` log messages with ``(notice)`` have been
+  turned into ``Notice`` log messages.
+
+* All shard ``Error`` and ``Notice`` messages now use the unified
+  prefix ``vmod_directors: shard %s``.
+
+``std.set_ip_tos()``
+~~~~~~~~~~~~~~~~~~~~
+
+The ``set_ip_tos()`` function from the bundled ``std`` vmod now sets
+the IPv6 Taffic Class (TCLASS) when used on an IPv6 connection.
+
+.. _whatsnew_changes_CURRENT_ban:
+
+``std.ban()`` and ``std.ban_error()``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``std.ban()`` and ``std.ban_error()`` functions have been added to
+the ``std`` vmod, allowing VCL to check for ban errors. A typical
+usage pattern with the new interface is::
+
+  if (std.ban(...)) {
+    return(synth(200, "Ban added"));
+  } else {
+    return(synth(400, std.ban_error()));
+  }
+
+``cookie`` functions
+~~~~~~~~~~~~~~~~~~~~
+
+The ``filter_re``, ``keep_re`` and ``get_re`` functions from the
+bundled ``cookie`` vmod have been changed to take the ``VCL_REGEX``
+type. This implies that their regular expression arguments now need to
+be literal, whereas before they could be taken from some other
+variable or function returning ``VCL_STRING``.
+
+Note that these functions never actually handled *dynamic* regexen,
+the string passed with the first call was compiled to a regex, which
+was then used for the lifetime of the respective VCL.
+
 
 varnishlog
 ==========
 
-**XXX changes concerning varnishlog(1) and/or vsl(7)**
+* See `Accounting <whatsnew_changes_CURRENT_accounting_>`_ for changes
+  to accounting-related VSL records.
+
+* See `Session Close Reasons <whatsnew_changes_CURRENT_sc_close_>`_
+  for a change affecting ``SessClose``.
+
+* Three new ``Timestamp`` VSL records have been added to backend
+  request processing:
+
+  - The ``Process`` timestamp after ``return(deliver)`` or
+    ``return(pass(x))`` from ``vcl_backend_response``,
+
+  - the ``Fetch`` timestamp before a backend connection is requested
+    and
+
+  - the ``Connected`` timestamp when a connection to a regular backend
+    (VBE) is established.
+
+* The ``FetchError`` log message ``Timed out reusing backend
+  connection`` has been renamed to ``first byte timeout (reused
+  connection)`` to clarify that it is emit for effectively the same
+  reason as ``first byte timeout``.
+
+* ``ExpKill`` log (VSL) records are now masked by default. See the
+  ``vsl_mask`` parameter.
+
+* Comparisons of numbers in VSL queries have been improved to match
+  better the behavior which is likely expected by users who have not
+  read the documentation in all detail.
+
+* See `Vary Notice <whatsnew_changes_CURRENT_vary_notice_>`_ for
+  information on a newly added ``Notice`` log (VSL) record.
+
+varnishncsa
+===========
+
+* The ``%{X}T`` format has been added to ``varnishncsa``, which
+  generalizes ``%D`` and ``%T``, but also support milliseconds
+  (``ms``) output.
+
+* The ``varnishncsa`` ``-E`` argument to show ESI requests has been
+  changed to imply ``-c`` (client mode).
+
 
 varnishadm
 ==========
 
-**XXX changes concerning varnishadm(1) and/or varnish-cli(7)**
+* The ``vcl.discard`` CLI command can now be used to discard more than
+  one VCL with a single command, which succeeds only if all given VCLs
+  could be discarded (atomic behavior).
+
+* The ``vcl.discard`` CLI command now supports glob patterns for vcl names.
+
+* The ``vcl.deps`` CLI command has been added to output dependencies
+  between VCLs (because of labels and ``return(vcl)`` statements).
+
+* ``varnishadm`` now has the ``-p`` option to disable readline support
+  for use in scripts and as a generic CLI connector.
 
 varnishstat
 ===========
 
-**XXX changes concerning varnishstat(1) and/or varnish-counters(7)**
+* See `Accounting <whatsnew_changes_CURRENT_accounting_>`_ for changes
+  to accounting-related counters.
+
+* See `Session Close Reasons <whatsnew_changes_CURRENT_sc_close_>`_
+  for a change affecting ``MAIN.sc_*`` counters.
+
+* The ``MAIN.esi_req`` counter has been added as a statistic of the
+  number of ESI sub requests created.
+
+* The ``MAIN.s_bgfetch`` counter has been added as a statistic on the
+  number of background fetches issued.
+
+.. _whatsnew_changes_CURRENT_varnishstat_raw:
+
+* ``varnishstat`` now avoids display errors of gauges which previously
+  could underflow to negative values, being displayed as extremely
+  high positive values.
+
+  The ``-r`` option and the ``r`` key binding have been added to
+  return to the previous behavior. When raw mode is active in
+  ``varnishstat`` interactive (curses) mode, the word ``RAW`` is
+  displayed at the right hand side in the lower status line.
 
 varnishtest
 ===========
 
-**XXX changes concerning varnishtest(1) and/or vtc(7)**
+Various improvements have been made to the ``varnishtest`` facility:
+
+- the ``loop`` keyword now works everywhere
+
+- HTTP/2 logging has been improved
+
+- Default HTTP/2 parameters have been tweaked
+
+- Varnish listen address information is now available by default in
+  the macros ``${vNAME_addr}``, ``${vNAME_port}`` and
+  ``${vNAME_sock}``. Macros by the names ``${vNAME_SOCKET_*}`` contain
+  the address information for each listen socket as created with the
+  ``-a`` argument to ``varnishd``.
+
+- Synchronization points for counters (VSCs) have been added as
+  ``varnish vNAME -expect PATTERN OP PATTERN``
+
+- varnishtest now also works with IPv6 setups
+
+- ``feature ipqv4`` and ``feature ipv6`` can be used to control
+  execution of test cases which require one or the other protocol.
+
+- haproxy arguments can now be externally provided through the
+  ``HAPROXY_ARGS`` variable.
+
+- logexpect now supports alternatives with the ``expect ? ...`` syntax
+  and negative matches with the ``fail add ...`` and ``fail clear``
+  syntax.
+
+- The overall logexpect match expectation can now be inverted using
+  the ``-err`` argument.
+
+- Numeric comparisons for HTTP headers have been added: ``-lt``,
+  ``-le``, ``-eq``, ``-ne``, ``-ge``, ``-gt``
+
+- ``rxdata -some`` has been fixed.
+
+Other Changes to Varnish Utilities
+==================================
+
+All varnish tools using the VUT library utilities for argument
+processing now support the ``--optstring`` argument to return a string
+suitable for use with ``getopts`` from shell scripts.
+
+
+Developer: Changes for VMOD authors
+===================================
+
+VMOD/VCL interface
+~~~~~~~~~~~~~~~~~~
+
+* The ``VCL_REGEX`` data type is now supported for VMODs, allowing
+  them to use regular expression literals checked and compiled by the
+  VCL compiler infrastructure.
+
+  Consequently, the ``VRT_re_init()`` and ``VRT_re_fini()`` functions
+  have been removed, because they are not required and their use was
+  probably wrong anyway.
+
+* The ``VCL_SUB`` data type is now supported for VMODs to save
+  references to subroutines to be called later using
+  ``VRT_call()``. Calls from a wrong context (e.g. calling a
+  subroutine accessing ``req`` from the backend side) and recursive
+  calls fail the VCL.
+
+  See `VMOD - Varnish Modules`_ in the Reference Manual.
+
+.. _VMOD - Varnish Modules: https://varnish-cache.org/docs/trunk/reference/vmod.html
+
+  VMOD functions can also return the ``VCL_SUB`` data type for calls
+  from VCL as in ``call vmod.returning_sub();``.
+
+* ``VRT_check_call()`` can be used to check if a ``VRT_call()`` would
+  succeed in order to avoid the potential VCL failure in case it would
+  not.
+
+  It returns ``NULL`` if ``VRT_call()`` would make the call or an
+  error string why not.
+
+* ``VRT_handled()`` has been added, which is now to be used instead of
+  access to the ``handling`` member of ``VRT_CTX``.
+
+* ``vmodtool.py`` has been improved to simplify Makefiles when many
+  VMODs are built in a single directory.
 
-Changes for developers and VMOD authors
-=======================================
+General API
+~~~~~~~~~~~
+
+* ``VRT_ValidHdr()`` has been added for vmods to conduct the same
+  check as the `whatsnew_changes_CURRENT_header_validation`_ feature,
+  for example when headers are set by vmods using the ``cache_http.c``
+  Functions like ``http_ForceHeader()`` from untrusted input.
+
+* Client and backend finite state machine internals (``enum req_step``
+  and ``enum fetch_step``) have been removed from ``cache.h``.
+
+* The ``verrno.h`` header file has been removed and merged into
+  ``vas.h``
+
+* The ``pdiff()`` function declaration has been moved from ``cache.h``
+  to ``vas.h``.
+
+VSA
+~~~
+
+* The ``VSA_getsockname()`` and ``VSA_getpeername()`` functions have
+  been added to get address information of file descriptors.
+
+Private Pointers
+~~~~~~~~~~~~~~~~
+
+* The interface for private pointers in VMODs has been changed:
+
+  - The ``free`` pointer in ``struct vmod_priv`` has been replaced
+    with a pointer to ``struct vmod_priv_methods``, to where the
+    pointer to the former free callback has been moved as the ``fini``
+    member.
+
+  - The former free callback type has been renamed from
+    ``vmod_priv_free_f`` to ``vmod_priv_fini_f`` and as gained a
+    ``VRT_CTX`` argument
+
+* The ``VRT_priv_task_get()`` and ``VRT_priv_top_get()`` functions
+  have been added to VRT to allow vmods to retrieve existing
+  ``PRIV_TASK`` / ``PRIV_TOP`` private pointers without creating any.
+
+Backends
+~~~~~~~~
+
+* The VRT backend interface has been changed:
+
+  - ``struct vrt_endpoint`` has been added describing a UDS or TCP
+    endpoint for a backend to connect to.
+
+    Endpoints also support a preamble to be sent with every new
+    connection.
+
+  - This structure needs to be passed via the ``endpoint`` member of
+    ``struct vrt_backend`` when creating backends with
+    ``VRT_new_backend()`` or ``VRT_new_backend_clustered()``.
+
+* ``VRT_Endpoint_Clone()`` has been added to facilitate working with
+  endpoints.
+
+Filters (VDP/VFP)
+~~~~~~~~~~~~~~~~~
+
+* Many filter (VDP/VFP) related signatures have been changed:
+
+  - ``vdp_init_f``
+
+  - ``vdp_fini_f``
+
+  - ``vdp_bytes_f``
+
+  - ``VDP_bytes()``
+
+  as well as ``struct vdp_entry`` and ``struct vdp_ctx``
+
+  ``VFP_Push()`` and ``VDP_Push()`` are no longer intended for VMOD
+  use and have been removed from the API.
+
+* The VDP code is now more strict about ``VDP_END``, which must be
+  sent down the filter chain at most once. Care should be taken to
+  send ``VDP_END`` together with the last payload bytes whenever
+  possible.
+
+Stevedore API
+~~~~~~~~~~~~~
+
+* The stevedore API has been changed:
+
+  - ``OBJ_ITER_FINAL`` has been renamed to ``OBJ_ITER_END``
+
+  - ``ObjExtend()`` signature has been changed to also cover the
+    ``ObjTrimStore()`` use case and
+
+  - ``ObjTrimStore()`` has been removed.
+
+Developer: Changes for Authors of Varnish Utilities
+===================================================
+
+libvarnishapi
+~~~~~~~~~~~~~
 
-**XXX changes concerning VRT, the public APIs, source code organization,
-builds etc.**
+* The ``VSC_IsRaw()`` function has been added to ``libvarnishapi`` to
+  query if a gauge is being returned raw or adjusted (see
+  `varnishstat -r option <whatsnew_changes_CURRENT_varnishstat_raw_>`_).
 
 *eof*

From nils.goroll at uplex.de  Sun Mar 14 17:44:05 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sun, 14 Mar 2021 17:44:05 +0000 (UTC)
Subject: [master] 1aa3c8cc3 whats-new: upgrading
Message-ID: <20210314174405.20017116102@lists.varnish-cache.org>


commit 1aa3c8cc3f06b49455321f8cfec283774d020bc1
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Sun Mar 14 18:43:24 2021 +0100

    whats-new: upgrading

diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst
index af4edb196..d547d73ea 100644
--- a/doc/sphinx/whats-new/changes-trunk.rst
+++ b/doc/sphinx/whats-new/changes-trunk.rst
@@ -3,10 +3,6 @@
 	SPDX-License-Identifier: BSD-2-Clause
 	See LICENSE file for full text of license
 
-**Note: This is a working document for a future release, with running
-updates for changes in the development branch. For changes in the
-released versions of Varnish, see:** :ref:`whats-new-index`
-
 .. _whatsnew_changes_CURRENT:
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
@@ -195,6 +191,8 @@ usage pattern with the new interface is::
     return(synth(400, std.ban_error()));
   }
 
+.. _whatsnew_changes_CURRENT_cookie:
+
 ``cookie`` functions
 ~~~~~~~~~~~~~~~~~~~~
 
@@ -344,6 +342,7 @@ All varnish tools using the VUT library utilities for argument
 processing now support the ``--optstring`` argument to return a string
 suitable for use with ``getopts`` from shell scripts.
 
+.. _whatsnew_changes_CURRENT_vmod:
 
 Developer: Changes for VMOD authors
 ===================================
diff --git a/doc/sphinx/whats-new/upgrading-trunk.rst b/doc/sphinx/whats-new/upgrading-trunk.rst
index 950a3d3f9..36c277ab1 100644
--- a/doc/sphinx/whats-new/upgrading-trunk.rst
+++ b/doc/sphinx/whats-new/upgrading-trunk.rst
@@ -3,36 +3,69 @@
 	SPDX-License-Identifier: BSD-2-Clause
 	See LICENSE file for full text of license
 
-**Note: This is a working document for a future release, with running
-updates for changes in the development branch. For changes in the
-released versions of Varnish, see:** :ref:`whats-new-index`
-
 .. _whatsnew_upgrading_CURRENT:
 
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 Upgrading to Varnish **$NEXT_RELEASE**
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-**XXX: how to upgrade from previous deployments to this
-version. Limited to work that has to be done for an upgrade, new
-features are listed in "Changes". Explicitly mention what does *not*
-have to be changed, especially in VCL. May include, but is not limited
-to:**
+In general, this release should not come with relevant incompatibilies
+to the previous release 6.5.
+
+VCL should continue to work as before except when rather exotic,
+partly unintended and/or undocumented features are used.
+
+Header Validation
+=================
+
+Varnish now validates any headers set from VCL to contain only
+characters allowed by RFC7230. A (runtime) VCL failure is triggered if
+not. Such VCL failures, which result in ``503`` responses, should be
+investigated. As a last resort, the ``validate_headers`` parameter can
+be set to ``false`` to avoid these VCL failures.
+
+BAN changes
+===========
+
+* The ``ban_cutoff`` parameter now refers to the overall length of the
+  ban list, including completed bans, where before only non-completed
+  ("active") bans were counted towards ``ban_cutoff``.
+
+* The ``ban()`` VCL builtin is now deprecated and should be replaced
+  with :ref:`whatsnew_changes_CURRENT_ban`
+
+Accounting Changes
+==================
+
+Accounting statistics and Log records have changed. See
+:ref:`whatsnew_changes_CURRENT_accounting` for details.
+
+VMOD ``cookie`` functions
+=========================
+
+The regular expression arguments taken by various functions from the
+``cookie`` VMOD now need to be literal. See
+:ref:`whatsnew_changes_CURRENT_cookie` for details.
+
+
+Other VCL Changes
+=================
 
-* Elements of VCL that have been removed or are deprecated, or whose
-  semantics have changed.
+* The ``resp.proto`` variable is now read-only as it should have been
+  for long.
 
-* -p parameters that have been removed or are deprecated, or whose
-  semantics have changed.
+  Changing the protocol is an error and should not be required.
 
-* Changes in the CLI.
+* Trying to use ``std.rollback()`` from ``vcl_pipe`` now results in
+  VCL failure.
 
-* Changes in the output or interpretation of stats or the log, including
-  changes affecting varnishncsa/-hist/-top.
+* ``return(retry)`` from ``vcl_backend_error {}`` now correctly resets
+  ``beresp.status`` and ``beresp.reason``.
 
-* Changes that may be necessary in VTCs or in the use of varnishtest.
+Changes to VMODs
+================
 
-* Changes in public APIs that may require changes in VMODs or VAPI/VUT
-  clients.
+Many VMODs will need minor adjustments to work with this release. See
+:ref:`whatsnew_changes_CURRENT_vmod` for details.
 
 *eof*

From dridi.boukelmoune at gmail.com  Mon Mar 15 06:55:08 2021
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 15 Mar 2021 06:55:08 +0000 (UTC)
Subject: [master] dd52b3362 doc: Improve 6.6 release notes
Message-ID: <20210315065508.16193101E4A@lists.varnish-cache.org>


commit dd52b33628715ba94df289710289843b4f91577e
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Mon Mar 15 07:51:59 2021 +0100

    doc: Improve 6.6 release notes

diff --git a/doc/changes.rst b/doc/changes.rst
index c9a251986..55b57f88b 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -163,7 +163,8 @@ Varnish Cache Next (2021-03-15)
     and
 
   - the ``Connected`` timestamp when a connection to a regular backend
-    (VBE) is established.
+    (VBE) is established, or when a recycled connection was selected for
+    reuse.
 
 * The VRT backend interface has been changed:
 
@@ -305,7 +306,7 @@ Varnish Cache Next (2021-03-15)
 * The ``VSA_getsockname()`` and ``VSA_getpeername()`` functions have
   been added to get address information of file descriptors.
 
-* ``varnishd`` now supports the ``-b None`` argument to start with
+* ``varnishd`` now supports the ``-b none`` argument to start with
   only the builtin VCL and no backend at all (3067_).
 
 * Some corner cases of IPv6 support in ``varnishd`` have been fixed.
@@ -334,8 +335,8 @@ Varnish Cache Next (2021-03-15)
 
 * The ``busy_stats_rate`` feature flag has been added to ensure
   statistics updates (as configured using the ``thread_stats_rate``
-  parameter) even on a fully loaded system, which would otherwise
-  delay statistics updates in order to reduce lock contention.
+  parameter) even in scenarios where worker threads never run out
+  of tasks and may remain forever busy.
 
 * ``ExpKill`` log (VSL) records are now masked by default. See the
   ``vsl_mask`` parameter.
diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst
index d547d73ea..b6efc7327 100644
--- a/doc/sphinx/whats-new/changes-trunk.rst
+++ b/doc/sphinx/whats-new/changes-trunk.rst
@@ -24,7 +24,7 @@ varnishd
 Arguments
 ~~~~~~~~~
 
-* ``varnishd`` now supports the ``-b None`` argument to start with
+* ``varnishd`` now supports the ``-b none`` argument to start with
   only the builtin VCL and no backend at all.
 
 Parameters
@@ -46,8 +46,8 @@ Parameters
 
 * The ``busy_stats_rate`` feature flag has been added to ensure
   statistics updates (as configured using the ``thread_stats_rate``
-  parameter) even on a fully loaded system, which would otherwise
-  delay statistics updates in order to reduce lock contention.
+  parameter) even in scenarios where worker threads never run out
+  of tasks and may remain forever busy.
 
 .. _whatsnew_changes_CURRENT_accounting:
 
@@ -55,7 +55,7 @@ Accounting
 ~~~~~~~~~~
 
 Body bytes accounting has been fixed to always represent the number of
-bodybytes moved on the wire, exclusive of protocol-specific overhead
+body bytes moved on the wire, exclusive of protocol-specific overhead
 like HTTP/1 chunked encoding or HTTP/2 framing.
 
 This change affects counters like
@@ -125,7 +125,7 @@ VCL variables
 * The ``bereq.xid`` variable is now also available in ``vcl_pipe {}``
 
 * The ``resp.proto`` variable is now read-only as it should have been
-  for long.
+  for long, like the other ``*.proto`` variables.
 
 Other changes to VCL
 ~~~~~~~~~~~~~~~~~~~~
@@ -226,7 +226,8 @@ varnishlog
     and
 
   - the ``Connected`` timestamp when a connection to a regular backend
-    (VBE) is established.
+    (VBE) is established, or when a recycled connection was selected for
+    reuse.
 
 * The ``FetchError`` log message ``Timed out reusing backend
   connection`` has been renamed to ``first byte timeout (reused
@@ -251,7 +252,8 @@ varnishncsa
   (``ms``) output.
 
 * The ``varnishncsa`` ``-E`` argument to show ESI requests has been
-  changed to imply ``-c`` (client mode).
+  changed to imply ``-c`` (client mode). This behavior is now shared
+  by all log utilities, and ``-c`` no longer includes ESI requests.
 
 
 varnishadm
@@ -317,7 +319,7 @@ Various improvements have been made to the ``varnishtest`` facility:
 
 - varnishtest now also works with IPv6 setups
 
-- ``feature ipqv4`` and ``feature ipv6`` can be used to control
+- ``feature ipv4`` and ``feature ipv6`` can be used to control
   execution of test cases which require one or the other protocol.
 
 - haproxy arguments can now be externally provided through the
@@ -387,9 +389,9 @@ VMOD/VCL interface
 General API
 ~~~~~~~~~~~
 
-* ``VRT_ValidHdr()`` has been added for vmods to conduct the same
+* ``VRT_ValidHdr()`` has been added for VMODs to conduct the same
   check as the `whatsnew_changes_CURRENT_header_validation`_ feature,
-  for example when headers are set by vmods using the ``cache_http.c``
+  for example when headers are set by VMODs using the ``cache_http.c``
   Functions like ``http_ForceHeader()`` from untrusted input.
 
 * Client and backend finite state machine internals (``enum req_step``
diff --git a/doc/sphinx/whats-new/upgrading-trunk.rst b/doc/sphinx/whats-new/upgrading-trunk.rst
index 36c277ab1..501a8f49c 100644
--- a/doc/sphinx/whats-new/upgrading-trunk.rst
+++ b/doc/sphinx/whats-new/upgrading-trunk.rst
@@ -52,7 +52,7 @@ Other VCL Changes
 =================
 
 * The ``resp.proto`` variable is now read-only as it should have been
-  for long.
+  for long, like the other ``*.proto`` variables.
 
   Changing the protocol is an error and should not be required.
 

From dridi.boukelmoune at gmail.com  Mon Mar 15 09:41:05 2021
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 15 Mar 2021 09:41:05 +0000 (UTC)
Subject: [master] b90b293d0 whats-new: Upgrade considerations for VSL utilities
Message-ID: <20210315094105.4CB1E105F04@lists.varnish-cache.org>


commit b90b293d094f65fc214a244f296e2dc23057dc9a
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Mon Mar 15 10:39:53 2021 +0100

    whats-new: Upgrade considerations for VSL utilities

diff --git a/doc/sphinx/whats-new/upgrading-trunk.rst b/doc/sphinx/whats-new/upgrading-trunk.rst
index 501a8f49c..b820e9469 100644
--- a/doc/sphinx/whats-new/upgrading-trunk.rst
+++ b/doc/sphinx/whats-new/upgrading-trunk.rst
@@ -40,6 +40,18 @@ Accounting Changes
 Accounting statistics and Log records have changed. See
 :ref:`whatsnew_changes_CURRENT_accounting` for details.
 
+VSL changes
+===========
+
+The ``-c`` option of log utilities no longer includes ESI requests. A
+new ``-E`` option is now available for ESI requests and it implies ``-c``
+too. This brings all log utilities on par with ``varnishncsa`` where the
+``-E`` option was initially introduced.
+
+If you use ``-c`` to collect both client and ESI requests, you should
+use ``-E`` instead. If you use ``-c`` and a VSL query to exclude ESI
+requests, the query should no longer be needed.
+
 VMOD ``cookie`` functions
 =========================
 
@@ -47,7 +59,6 @@ The regular expression arguments taken by various functions from the
 ``cookie`` VMOD now need to be literal. See
 :ref:`whatsnew_changes_CURRENT_cookie` for details.
 
-
 Other VCL Changes
 =================
 

From martin at varnish-software.com  Mon Mar 15 11:52:04 2021
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Mon, 15 Mar 2021 11:52:04 +0000 (UTC)
Subject: [master] ef54768fc Prepare for Varnish Cache 6.6.0
Message-ID: <20210315115205.09174109747@lists.varnish-cache.org>


commit ef54768fc10f5b19556c7cf9866efc88cfbda8ff
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Mon Mar 15 11:28:48 2021 +0100

    Prepare for Varnish Cache 6.6.0

diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index 1dc349b99..24eac15dc 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -165,7 +165,7 @@ Resp_Setup_Deliver(struct req *req)
 	http_PrintfHeader(h, "Age: %.0f",
 	    floor(fmax(0., req->t_prev - oc->t_origin)));
 
-	http_SetHeader(h, "Via: 1.1 varnish (Varnish/6.5)");
+	http_SetHeader(h, "Via: 1.1 varnish (Varnish/6.6)");
 
 	if (cache_param->http_gzip_support &&
 	    ObjCheckFlag(req->wrk, oc, OF_GZIPED) &&
diff --git a/configure.ac b/configure.ac
index 0201b72ec..0b4c32d2d 100644
--- a/configure.ac
+++ b/configure.ac
@@ -1,8 +1,8 @@
 AC_PREREQ(2.59)
 AC_COPYRIGHT([Copyright (c) 2006 Verdens Gang AS
-Copyright (c) 2006-2020 Varnish Software])
+Copyright (c) 2006-2021 Varnish Software])
 AC_REVISION([$Id$])
-AC_INIT([Varnish], [trunk], [varnish-dev at varnish-cache.org])
+AC_INIT([Varnish], [6.6.0], [varnish-dev at varnish-cache.org])
 AC_CONFIG_SRCDIR(include/miniobj.h)
 AC_CONFIG_HEADERS([config.h])
 AC_CONFIG_MACRO_DIR([m4])
diff --git a/doc/changes.rst b/doc/changes.rst
index 55b57f88b..b63db87c8 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -32,7 +32,7 @@ individual releases. These documents are updated as part of the
 release process.
 
 ================================
-Varnish Cache Next (2021-03-15)
+Varnish Cache 6.6.0 (2021-03-15)
 ================================
 
 * Body bytes accounting has been fixed to always represent the number

From martin at varnish-software.com  Mon Mar 15 11:52:05 2021
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Mon, 15 Mar 2021 11:52:05 +0000 (UTC)
Subject: [master] 33dfb498b Merge branch '6.6'
Message-ID: <20210315115205.2096410974A@lists.varnish-cache.org>


commit 33dfb498b6c8f43ce8c9321867a3203e8ad0992c
Merge: b90b293d0 ef54768fc
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Mon Mar 15 12:49:16 2021 +0100

    Merge branch '6.6'

diff --cc configure.ac
index 0201b72ec,0b4c32d2d..59aa57b9e
--- a/configure.ac
+++ b/configure.ac
@@@ -1,8 -1,8 +1,8 @@@
  AC_PREREQ(2.59)
  AC_COPYRIGHT([Copyright (c) 2006 Verdens Gang AS
- Copyright (c) 2006-2020 Varnish Software])
+ Copyright (c) 2006-2021 Varnish Software])
  AC_REVISION([$Id$])
 -AC_INIT([Varnish], [6.6.0], [varnish-dev at varnish-cache.org])
 +AC_INIT([Varnish], [trunk], [varnish-dev at varnish-cache.org])
  AC_CONFIG_SRCDIR(include/miniobj.h)
  AC_CONFIG_HEADERS([config.h])
  AC_CONFIG_MACRO_DIR([m4])

From martin at varnish-software.com  Mon Mar 15 13:49:05 2021
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Mon, 15 Mar 2021 13:49:05 +0000 (UTC)
Subject: [master] 91b5e5c4f Rename and relabel upgrading-trunk.rst to
 upgrading-6.6.rst
Message-ID: <20210315134905.617AF10F9B9@lists.varnish-cache.org>


commit 91b5e5c4ff21cc21ba20fb5e8ec7d1eece3477e6
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Mon Mar 15 14:41:55 2021 +0100

    Rename and relabel upgrading-trunk.rst to upgrading-6.6.rst

diff --git a/doc/sphinx/whats-new/upgrading-trunk.rst b/doc/sphinx/whats-new/upgrading-6.6.rst
similarity index 94%
rename from doc/sphinx/whats-new/upgrading-trunk.rst
rename to doc/sphinx/whats-new/upgrading-6.6.rst
index b820e9469..db577e2e2 100644
--- a/doc/sphinx/whats-new/upgrading-trunk.rst
+++ b/doc/sphinx/whats-new/upgrading-6.6.rst
@@ -3,11 +3,11 @@
 	SPDX-License-Identifier: BSD-2-Clause
 	See LICENSE file for full text of license
 
-.. _whatsnew_upgrading_CURRENT:
+.. _whatsnew_upgrading_6.6:
 
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-Upgrading to Varnish **$NEXT_RELEASE**
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%
+Upgrading to Varnish 6.6
+%%%%%%%%%%%%%%%%%%%%%%%%
 
 In general, this release should not come with relevant incompatibilies
 to the previous release 6.5.

From martin at varnish-software.com  Mon Mar 15 13:49:05 2021
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Mon, 15 Mar 2021 13:49:05 +0000 (UTC)
Subject: [master] 294ea948f Rename and relable changes-trunk.rst to
 changes-6.6.rst
Message-ID: <20210315134905.7340510F9BC@lists.varnish-cache.org>


commit 294ea948f29d49c52614b067d9bd67d1eca4f554
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Mon Mar 15 14:47:43 2021 +0100

    Rename and relable changes-trunk.rst to changes-6.6.rst

diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-6.6.rst
similarity index 100%
rename from doc/sphinx/whats-new/changes-trunk.rst
rename to doc/sphinx/whats-new/changes-6.6.rst

From martin at varnish-software.com  Mon Mar 15 13:49:05 2021
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Mon, 15 Mar 2021 13:49:05 +0000 (UTC)
Subject: [master] 3bd1ccda2 Relabel CURRENT to 6.6 in changes-6.6.rst
Message-ID: <20210315134905.ACC9A10F9C0@lists.varnish-cache.org>


commit 3bd1ccda2cf5b0f7065532ec3ead6f9d4b4f4de2
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Mon Mar 15 14:48:41 2021 +0100

    Relabel CURRENT to 6.6 in changes-6.6.rst

diff --git a/doc/sphinx/whats-new/changes-6.6.rst b/doc/sphinx/whats-new/changes-6.6.rst
index b6efc7327..1505f63bb 100644
--- a/doc/sphinx/whats-new/changes-6.6.rst
+++ b/doc/sphinx/whats-new/changes-6.6.rst
@@ -3,14 +3,14 @@
 	SPDX-License-Identifier: BSD-2-Clause
 	See LICENSE file for full text of license
 
-.. _whatsnew_changes_CURRENT:
+.. _whatsnew_changes_6.6:
 
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-Changes in Varnish **$NEXT_RELEASE**
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%
+Changes in Varnish 6.6
+%%%%%%%%%%%%%%%%%%%%%%
 
 For information about updating your current Varnish deployment to the
-new version, see :ref:`whatsnew_upgrading_CURRENT`.
+new version, see :ref:`whatsnew_upgrading_6.6`.
 
 A more detailed and technical account of changes in Varnish, with
 links to issues that have been fixed and pull requests that have been
@@ -31,7 +31,7 @@ Parameters
 ~~~~~~~~~~
 
 * The ``validate_headers`` parameter has been added to control
-  `header validation <whatsnew_changes_CURRENT_header_validation_>`_.
+  `header validation <whatsnew_changes_6.6_header_validation_>`_.
 
 * The ``ban_cutoff`` parameter now refers to the overall length of the
   ban list, including completed bans, where before only non-completed
@@ -39,7 +39,7 @@ Parameters
 
 * The ``vary_notice`` parameter has been added to control the
   threshold for the new `Vary Notice
-  <whatsnew_changes_CURRENT_vary_notice_>`_.
+  <whatsnew_changes_6.6_vary_notice_>`_.
 
 ``feature`` Flags
 ~~~~~~~~~~~~~~~~~
@@ -49,7 +49,7 @@ Parameters
   parameter) even in scenarios where worker threads never run out
   of tasks and may remain forever busy.
 
-.. _whatsnew_changes_CURRENT_accounting:
+.. _whatsnew_changes_6.6_accounting:
 
 Accounting
 ~~~~~~~~~~
@@ -76,7 +76,7 @@ as well as the VSL records
 
 - ``BereqAcct``.
 
-.. _whatsnew_changes_CURRENT_sc_close:
+.. _whatsnew_changes_6.6_sc_close:
 
 Session Close Reasons
 ~~~~~~~~~~~~~~~~~~~~~
@@ -91,7 +91,7 @@ detailed information on the cause of the failure.
 The session close reason logging/statistics for HTTP/2 connections
 have been improved.
 
-.. _whatsnew_changes_CURRENT_vary_notice:
+.. _whatsnew_changes_6.6_vary_notice:
 
 Vary Notice
 ~~~~~~~~~~~
@@ -103,7 +103,7 @@ hash. The new ``vary_notice`` parameter defaults to 10.
 Changes to VCL
 ==============
 
-.. _whatsnew_changes_CURRENT_header_validation:
+.. _whatsnew_changes_6.6_header_validation:
 
 Header Validation
 ~~~~~~~~~~~~~~~~~
@@ -134,7 +134,7 @@ Other changes to VCL
   addition to the existing ``{" ... "}``.
 
 * The ``ban()`` builtin is now deprecated and should be replaced with
-  `std.ban() <whatsnew_changes_CURRENT_ban_>`_.
+  `std.ban() <whatsnew_changes_6.6_ban_>`_.
 
 * Trying to use ``std.rollback()`` from ``vcl_pipe`` now results in
   VCL failure.
@@ -176,7 +176,7 @@ VMODs
 The ``set_ip_tos()`` function from the bundled ``std`` vmod now sets
 the IPv6 Taffic Class (TCLASS) when used on an IPv6 connection.
 
-.. _whatsnew_changes_CURRENT_ban:
+.. _whatsnew_changes_6.6_ban:
 
 ``std.ban()`` and ``std.ban_error()``
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -191,7 +191,7 @@ usage pattern with the new interface is::
     return(synth(400, std.ban_error()));
   }
 
-.. _whatsnew_changes_CURRENT_cookie:
+.. _whatsnew_changes_6.6_cookie:
 
 ``cookie`` functions
 ~~~~~~~~~~~~~~~~~~~~
@@ -210,10 +210,10 @@ was then used for the lifetime of the respective VCL.
 varnishlog
 ==========
 
-* See `Accounting <whatsnew_changes_CURRENT_accounting_>`_ for changes
+* See `Accounting <whatsnew_changes_6.6_accounting_>`_ for changes
   to accounting-related VSL records.
 
-* See `Session Close Reasons <whatsnew_changes_CURRENT_sc_close_>`_
+* See `Session Close Reasons <whatsnew_changes_6.6_sc_close_>`_
   for a change affecting ``SessClose``.
 
 * Three new ``Timestamp`` VSL records have been added to backend
@@ -241,7 +241,7 @@ varnishlog
   better the behavior which is likely expected by users who have not
   read the documentation in all detail.
 
-* See `Vary Notice <whatsnew_changes_CURRENT_vary_notice_>`_ for
+* See `Vary Notice <whatsnew_changes_6.6_vary_notice_>`_ for
   information on a newly added ``Notice`` log (VSL) record.
 
 varnishncsa
@@ -274,10 +274,10 @@ varnishadm
 varnishstat
 ===========
 
-* See `Accounting <whatsnew_changes_CURRENT_accounting_>`_ for changes
+* See `Accounting <whatsnew_changes_6.6_accounting_>`_ for changes
   to accounting-related counters.
 
-* See `Session Close Reasons <whatsnew_changes_CURRENT_sc_close_>`_
+* See `Session Close Reasons <whatsnew_changes_6.6_sc_close_>`_
   for a change affecting ``MAIN.sc_*`` counters.
 
 * The ``MAIN.esi_req`` counter has been added as a statistic of the
@@ -286,7 +286,7 @@ varnishstat
 * The ``MAIN.s_bgfetch`` counter has been added as a statistic on the
   number of background fetches issued.
 
-.. _whatsnew_changes_CURRENT_varnishstat_raw:
+.. _whatsnew_changes_6.6_varnishstat_raw:
 
 * ``varnishstat`` now avoids display errors of gauges which previously
   could underflow to negative values, being displayed as extremely
@@ -344,7 +344,7 @@ All varnish tools using the VUT library utilities for argument
 processing now support the ``--optstring`` argument to return a string
 suitable for use with ``getopts`` from shell scripts.
 
-.. _whatsnew_changes_CURRENT_vmod:
+.. _whatsnew_changes_6.6_vmod:
 
 Developer: Changes for VMOD authors
 ===================================
@@ -390,7 +390,7 @@ General API
 ~~~~~~~~~~~
 
 * ``VRT_ValidHdr()`` has been added for VMODs to conduct the same
-  check as the `whatsnew_changes_CURRENT_header_validation`_ feature,
+  check as the `whatsnew_changes_6.6_header_validation`_ feature,
   for example when headers are set by VMODs using the ``cache_http.c``
   Functions like ``http_ForceHeader()`` from untrusted input.
 
@@ -488,6 +488,6 @@ libvarnishapi
 
 * The ``VSC_IsRaw()`` function has been added to ``libvarnishapi`` to
   query if a gauge is being returned raw or adjusted (see
-  `varnishstat -r option <whatsnew_changes_CURRENT_varnishstat_raw_>`_).
+  `varnishstat -r option <whatsnew_changes_6.6_varnishstat_raw_>`_).
 
 *eof*

From martin at varnish-software.com  Mon Mar 15 13:51:05 2021
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Mon, 15 Mar 2021 13:51:05 +0000 (UTC)
Subject: [6.6] 53f2b7fe3 Rename and relabel upgrading-trunk.rst to
 upgrading-6.6.rst
Message-ID: <20210315135105.445FF10FE99@lists.varnish-cache.org>


commit 53f2b7fe3cf69e400d2e4c3afabb083a41bb663a
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Mon Mar 15 14:41:55 2021 +0100

    Rename and relabel upgrading-trunk.rst to upgrading-6.6.rst

diff --git a/doc/sphinx/whats-new/upgrading-trunk.rst b/doc/sphinx/whats-new/upgrading-6.6.rst
similarity index 94%
rename from doc/sphinx/whats-new/upgrading-trunk.rst
rename to doc/sphinx/whats-new/upgrading-6.6.rst
index b820e9469..db577e2e2 100644
--- a/doc/sphinx/whats-new/upgrading-trunk.rst
+++ b/doc/sphinx/whats-new/upgrading-6.6.rst
@@ -3,11 +3,11 @@
 	SPDX-License-Identifier: BSD-2-Clause
 	See LICENSE file for full text of license
 
-.. _whatsnew_upgrading_CURRENT:
+.. _whatsnew_upgrading_6.6:
 
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-Upgrading to Varnish **$NEXT_RELEASE**
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%%%
+Upgrading to Varnish 6.6
+%%%%%%%%%%%%%%%%%%%%%%%%
 
 In general, this release should not come with relevant incompatibilies
 to the previous release 6.5.

From martin at varnish-software.com  Mon Mar 15 13:51:05 2021
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Mon, 15 Mar 2021 13:51:05 +0000 (UTC)
Subject: [6.6] 35670e8f8 Rename and relable changes-trunk.rst to
 changes-6.6.rst
Message-ID: <20210315135105.5D7F510FE9C@lists.varnish-cache.org>


commit 35670e8f819e919ea8bdb40a319ad58268e8e355
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Mon Mar 15 14:47:43 2021 +0100

    Rename and relable changes-trunk.rst to changes-6.6.rst

diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-6.6.rst
similarity index 100%
rename from doc/sphinx/whats-new/changes-trunk.rst
rename to doc/sphinx/whats-new/changes-6.6.rst

From martin at varnish-software.com  Mon Mar 15 13:51:05 2021
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Mon, 15 Mar 2021 13:51:05 +0000 (UTC)
Subject: [6.6] 45930f0b8 Relabel CURRENT to 6.6 in changes-6.6.rst
Message-ID: <20210315135105.7E86810FEA0@lists.varnish-cache.org>


commit 45930f0b8f701440d75f5c1071a39041808d6b0b
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Mon Mar 15 14:48:41 2021 +0100

    Relabel CURRENT to 6.6 in changes-6.6.rst

diff --git a/doc/sphinx/whats-new/changes-6.6.rst b/doc/sphinx/whats-new/changes-6.6.rst
index b6efc7327..1505f63bb 100644
--- a/doc/sphinx/whats-new/changes-6.6.rst
+++ b/doc/sphinx/whats-new/changes-6.6.rst
@@ -3,14 +3,14 @@
 	SPDX-License-Identifier: BSD-2-Clause
 	See LICENSE file for full text of license
 
-.. _whatsnew_changes_CURRENT:
+.. _whatsnew_changes_6.6:
 
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
-Changes in Varnish **$NEXT_RELEASE**
-%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+%%%%%%%%%%%%%%%%%%%%%%
+Changes in Varnish 6.6
+%%%%%%%%%%%%%%%%%%%%%%
 
 For information about updating your current Varnish deployment to the
-new version, see :ref:`whatsnew_upgrading_CURRENT`.
+new version, see :ref:`whatsnew_upgrading_6.6`.
 
 A more detailed and technical account of changes in Varnish, with
 links to issues that have been fixed and pull requests that have been
@@ -31,7 +31,7 @@ Parameters
 ~~~~~~~~~~
 
 * The ``validate_headers`` parameter has been added to control
-  `header validation <whatsnew_changes_CURRENT_header_validation_>`_.
+  `header validation <whatsnew_changes_6.6_header_validation_>`_.
 
 * The ``ban_cutoff`` parameter now refers to the overall length of the
   ban list, including completed bans, where before only non-completed
@@ -39,7 +39,7 @@ Parameters
 
 * The ``vary_notice`` parameter has been added to control the
   threshold for the new `Vary Notice
-  <whatsnew_changes_CURRENT_vary_notice_>`_.
+  <whatsnew_changes_6.6_vary_notice_>`_.
 
 ``feature`` Flags
 ~~~~~~~~~~~~~~~~~
@@ -49,7 +49,7 @@ Parameters
   parameter) even in scenarios where worker threads never run out
   of tasks and may remain forever busy.
 
-.. _whatsnew_changes_CURRENT_accounting:
+.. _whatsnew_changes_6.6_accounting:
 
 Accounting
 ~~~~~~~~~~
@@ -76,7 +76,7 @@ as well as the VSL records
 
 - ``BereqAcct``.
 
-.. _whatsnew_changes_CURRENT_sc_close:
+.. _whatsnew_changes_6.6_sc_close:
 
 Session Close Reasons
 ~~~~~~~~~~~~~~~~~~~~~
@@ -91,7 +91,7 @@ detailed information on the cause of the failure.
 The session close reason logging/statistics for HTTP/2 connections
 have been improved.
 
-.. _whatsnew_changes_CURRENT_vary_notice:
+.. _whatsnew_changes_6.6_vary_notice:
 
 Vary Notice
 ~~~~~~~~~~~
@@ -103,7 +103,7 @@ hash. The new ``vary_notice`` parameter defaults to 10.
 Changes to VCL
 ==============
 
-.. _whatsnew_changes_CURRENT_header_validation:
+.. _whatsnew_changes_6.6_header_validation:
 
 Header Validation
 ~~~~~~~~~~~~~~~~~
@@ -134,7 +134,7 @@ Other changes to VCL
   addition to the existing ``{" ... "}``.
 
 * The ``ban()`` builtin is now deprecated and should be replaced with
-  `std.ban() <whatsnew_changes_CURRENT_ban_>`_.
+  `std.ban() <whatsnew_changes_6.6_ban_>`_.
 
 * Trying to use ``std.rollback()`` from ``vcl_pipe`` now results in
   VCL failure.
@@ -176,7 +176,7 @@ VMODs
 The ``set_ip_tos()`` function from the bundled ``std`` vmod now sets
 the IPv6 Taffic Class (TCLASS) when used on an IPv6 connection.
 
-.. _whatsnew_changes_CURRENT_ban:
+.. _whatsnew_changes_6.6_ban:
 
 ``std.ban()`` and ``std.ban_error()``
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -191,7 +191,7 @@ usage pattern with the new interface is::
     return(synth(400, std.ban_error()));
   }
 
-.. _whatsnew_changes_CURRENT_cookie:
+.. _whatsnew_changes_6.6_cookie:
 
 ``cookie`` functions
 ~~~~~~~~~~~~~~~~~~~~
@@ -210,10 +210,10 @@ was then used for the lifetime of the respective VCL.
 varnishlog
 ==========
 
-* See `Accounting <whatsnew_changes_CURRENT_accounting_>`_ for changes
+* See `Accounting <whatsnew_changes_6.6_accounting_>`_ for changes
   to accounting-related VSL records.
 
-* See `Session Close Reasons <whatsnew_changes_CURRENT_sc_close_>`_
+* See `Session Close Reasons <whatsnew_changes_6.6_sc_close_>`_
   for a change affecting ``SessClose``.
 
 * Three new ``Timestamp`` VSL records have been added to backend
@@ -241,7 +241,7 @@ varnishlog
   better the behavior which is likely expected by users who have not
   read the documentation in all detail.
 
-* See `Vary Notice <whatsnew_changes_CURRENT_vary_notice_>`_ for
+* See `Vary Notice <whatsnew_changes_6.6_vary_notice_>`_ for
   information on a newly added ``Notice`` log (VSL) record.
 
 varnishncsa
@@ -274,10 +274,10 @@ varnishadm
 varnishstat
 ===========
 
-* See `Accounting <whatsnew_changes_CURRENT_accounting_>`_ for changes
+* See `Accounting <whatsnew_changes_6.6_accounting_>`_ for changes
   to accounting-related counters.
 
-* See `Session Close Reasons <whatsnew_changes_CURRENT_sc_close_>`_
+* See `Session Close Reasons <whatsnew_changes_6.6_sc_close_>`_
   for a change affecting ``MAIN.sc_*`` counters.
 
 * The ``MAIN.esi_req`` counter has been added as a statistic of the
@@ -286,7 +286,7 @@ varnishstat
 * The ``MAIN.s_bgfetch`` counter has been added as a statistic on the
   number of background fetches issued.
 
-.. _whatsnew_changes_CURRENT_varnishstat_raw:
+.. _whatsnew_changes_6.6_varnishstat_raw:
 
 * ``varnishstat`` now avoids display errors of gauges which previously
   could underflow to negative values, being displayed as extremely
@@ -344,7 +344,7 @@ All varnish tools using the VUT library utilities for argument
 processing now support the ``--optstring`` argument to return a string
 suitable for use with ``getopts`` from shell scripts.
 
-.. _whatsnew_changes_CURRENT_vmod:
+.. _whatsnew_changes_6.6_vmod:
 
 Developer: Changes for VMOD authors
 ===================================
@@ -390,7 +390,7 @@ General API
 ~~~~~~~~~~~
 
 * ``VRT_ValidHdr()`` has been added for VMODs to conduct the same
-  check as the `whatsnew_changes_CURRENT_header_validation`_ feature,
+  check as the `whatsnew_changes_6.6_header_validation`_ feature,
   for example when headers are set by VMODs using the ``cache_http.c``
   Functions like ``http_ForceHeader()`` from untrusted input.
 
@@ -488,6 +488,6 @@ libvarnishapi
 
 * The ``VSC_IsRaw()`` function has been added to ``libvarnishapi`` to
   query if a gauge is being returned raw or adjusted (see
-  `varnishstat -r option <whatsnew_changes_CURRENT_varnishstat_raw_>`_).
+  `varnishstat -r option <whatsnew_changes_6.6_varnishstat_raw_>`_).
 
 *eof*

From martin at varnish-software.com  Mon Mar 15 13:56:05 2021
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Mon, 15 Mar 2021 13:56:05 +0000 (UTC)
Subject: [master] 65616bab6 Point -trunk to 6.6 in whats-new/index.rst
Message-ID: <20210315135606.098E5110470@lists.varnish-cache.org>


commit 65616bab6be85676157005c8089d3137ccf68a95
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Mon Mar 15 14:55:20 2021 +0100

    Point -trunk to 6.6 in whats-new/index.rst

diff --git a/doc/sphinx/whats-new/index.rst b/doc/sphinx/whats-new/index.rst
index 5845a4890..37ad90d28 100644
--- a/doc/sphinx/whats-new/index.rst
+++ b/doc/sphinx/whats-new/index.rst
@@ -13,18 +13,14 @@ This section describes the changes and improvements between different
 versions of Varnish, and what upgrading between the different versions
 entail.
 
-Varnish **$NEXT_RELEASE**
--------------------------
-
-**Note: These are working documents for a future release, with running
-updates for changes in the development branch. For changes in the
-released versions of Varnish, see the chapters listed below.**
+Varnish 6.6
+-----------
 
 .. toctree::
    :maxdepth: 2
 
-   changes-trunk
-   upgrading-trunk
+   changes-6.6
+   upgrading-6.6
 
 Varnish 6.5
 -----------

From martin at varnish-software.com  Mon Mar 15 13:57:04 2021
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Mon, 15 Mar 2021 13:57:04 +0000 (UTC)
Subject: [6.6] 0f1a08a10 Point -trunk to 6.6 in whats-new/index.rst
Message-ID: <20210315135704.83E8111068D@lists.varnish-cache.org>


commit 0f1a08a10a17714933746be536f4741f450ee734
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Mon Mar 15 14:55:20 2021 +0100

    Point -trunk to 6.6 in whats-new/index.rst

diff --git a/doc/sphinx/whats-new/index.rst b/doc/sphinx/whats-new/index.rst
index 5845a4890..37ad90d28 100644
--- a/doc/sphinx/whats-new/index.rst
+++ b/doc/sphinx/whats-new/index.rst
@@ -13,18 +13,14 @@ This section describes the changes and improvements between different
 versions of Varnish, and what upgrading between the different versions
 entail.
 
-Varnish **$NEXT_RELEASE**
--------------------------
-
-**Note: These are working documents for a future release, with running
-updates for changes in the development branch. For changes in the
-released versions of Varnish, see the chapters listed below.**
+Varnish 6.6
+-----------
 
 .. toctree::
    :maxdepth: 2
 
-   changes-trunk
-   upgrading-trunk
+   changes-6.6
+   upgrading-6.6
 
 Varnish 6.5
 -----------

From martin at varnish-software.com  Mon Mar 15 14:19:05 2021
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Mon, 15 Mar 2021 14:19:05 +0000 (UTC)
Subject: [master] 2b9f4ae4d Fix up CURRENT -> 6.6 references in
 upgrading-6.6.rst
Message-ID: <20210315141905.3CC0C111167@lists.varnish-cache.org>


commit 2b9f4ae4d4d27c351a9d7cccb2092d66948cebc8
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Mon Mar 15 15:17:58 2021 +0100

    Fix up CURRENT -> 6.6 references in upgrading-6.6.rst

diff --git a/doc/sphinx/whats-new/upgrading-6.6.rst b/doc/sphinx/whats-new/upgrading-6.6.rst
index db577e2e2..3cafd4a21 100644
--- a/doc/sphinx/whats-new/upgrading-6.6.rst
+++ b/doc/sphinx/whats-new/upgrading-6.6.rst
@@ -32,13 +32,13 @@ BAN changes
   ("active") bans were counted towards ``ban_cutoff``.
 
 * The ``ban()`` VCL builtin is now deprecated and should be replaced
-  with :ref:`whatsnew_changes_CURRENT_ban`
+  with :ref:`whatsnew_changes_6.6_ban`
 
 Accounting Changes
 ==================
 
 Accounting statistics and Log records have changed. See
-:ref:`whatsnew_changes_CURRENT_accounting` for details.
+:ref:`whatsnew_changes_6.6_accounting` for details.
 
 VSL changes
 ===========
@@ -57,7 +57,7 @@ VMOD ``cookie`` functions
 
 The regular expression arguments taken by various functions from the
 ``cookie`` VMOD now need to be literal. See
-:ref:`whatsnew_changes_CURRENT_cookie` for details.
+:ref:`whatsnew_changes_6.6_cookie` for details.
 
 Other VCL Changes
 =================
@@ -77,6 +77,6 @@ Changes to VMODs
 ================
 
 Many VMODs will need minor adjustments to work with this release. See
-:ref:`whatsnew_changes_CURRENT_vmod` for details.
+:ref:`whatsnew_changes_6.6_vmod` for details.
 
 *eof*

From martin at varnish-software.com  Mon Mar 15 14:20:05 2021
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Mon, 15 Mar 2021 14:20:05 +0000 (UTC)
Subject: [6.6] 829208dbd Fix up CURRENT -> 6.6 references in upgrading-6.6.rst
Message-ID: <20210315142005.7D1BC111338@lists.varnish-cache.org>


commit 829208dbd47ee4f950b8b7605f6e1c7d0c200724
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Mon Mar 15 15:17:58 2021 +0100

    Fix up CURRENT -> 6.6 references in upgrading-6.6.rst

diff --git a/doc/sphinx/whats-new/upgrading-6.6.rst b/doc/sphinx/whats-new/upgrading-6.6.rst
index db577e2e2..3cafd4a21 100644
--- a/doc/sphinx/whats-new/upgrading-6.6.rst
+++ b/doc/sphinx/whats-new/upgrading-6.6.rst
@@ -32,13 +32,13 @@ BAN changes
   ("active") bans were counted towards ``ban_cutoff``.
 
 * The ``ban()`` VCL builtin is now deprecated and should be replaced
-  with :ref:`whatsnew_changes_CURRENT_ban`
+  with :ref:`whatsnew_changes_6.6_ban`
 
 Accounting Changes
 ==================
 
 Accounting statistics and Log records have changed. See
-:ref:`whatsnew_changes_CURRENT_accounting` for details.
+:ref:`whatsnew_changes_6.6_accounting` for details.
 
 VSL changes
 ===========
@@ -57,7 +57,7 @@ VMOD ``cookie`` functions
 
 The regular expression arguments taken by various functions from the
 ``cookie`` VMOD now need to be literal. See
-:ref:`whatsnew_changes_CURRENT_cookie` for details.
+:ref:`whatsnew_changes_6.6_cookie` for details.
 
 Other VCL Changes
 =================
@@ -77,6 +77,6 @@ Changes to VMODs
 ================
 
 Many VMODs will need minor adjustments to work with this release. See
-:ref:`whatsnew_changes_CURRENT_vmod` for details.
+:ref:`whatsnew_changes_6.6_vmod` for details.
 
 *eof*

From martin at varnish-software.com  Thu Mar 18 12:01:06 2021
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Thu, 18 Mar 2021 12:01:06 +0000 (UTC)
Subject: [6.6] 2376a2a09 Change the dist job to be build or download
Message-ID: <20210318120106.3C0ACA873E@lists.varnish-cache.org>


commit 2376a2a09b07f85eb191da730346bc746e26e407
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Wed Mar 17 15:53:18 2021 +0100

    Change the dist job to be build or download
    
    Varnish packages are always built using a release tarball. This change
    enables sending build URL for tarball to be downloaded from and SHA256 sum
    to validate against as build parameters.
    
    If a specific commit is used and the tarball is created for the build
    step, the .is_weekly flag file will always be set, causing the packages to
    be dated.

diff --git a/.circleci/config.yml b/.circleci/config.yml
index 09d0f2db1..5cfc6686c 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -13,9 +13,16 @@ parameters:
   pkg-commit:
     type: string
     default: "master"
+  dist-url:
+    type: string
+    default: ""
+  dist-url-sha256:
+    type: string
+    default: ""
+
 jobs:
   dist:
-    description: Builds varnish-x.y.z.tar.gz that is used later for the packaging jobs
+    description: Build or download varnish-x.y.z.tar.gz that is used later for the packaging jobs
     docker:
       - image: centos:7
     steps:
@@ -33,19 +40,43 @@ jobs:
                 make \
                 pcre-devel \
                 python3 \
-                python-sphinx
+                python-sphinx \
+                curl
       - checkout
-      - run:
-          name: Create the dist tarball
-          command: |
-            git checkout << pipeline.parameters.vc-commit >>
-            # if version is "trunk", it's a weekly tarball, override the version
-            if grep 'AC_INIT.*trunk.*' ./configure.ac; then
-                sed -i -e "s/^AC_INIT.*trunk.*/AC_INIT([Varnish], [$(date +%Y%m%d)], [varnish-dev at varnish-cache.org])/" ./configure.ac
-                touch .is_weekly
-            fi
-            ./autogen.des --quiet
-            make dist -j 16
+      - when:
+          condition: << pipeline.parameters.dist-url >>
+          steps:
+            - run:
+                name: Download the dist tarball
+                command: |
+                  curl -s << pipeline.parameters.dist-url >> -o varnish-dist.tar.gz
+            - when:
+                condition: << pipeline.parameters.dist-url-sha256 >>
+                steps:
+                    - run:
+                        name: Verify downloaded tarball
+                        command: |
+                          echo "<< pipeline.parameters.dist-url-sha256 >> varnish-dist.tar.gz" | sha256sum -c
+            - run:
+                name: Rename the dist tarball by parsed version
+                command: |
+                  mkdir parse-version-tmp
+                  cd parse-version-tmp
+                  tar xzf ../varnish-dist.tar.gz
+                  VERSION=$(varnish-*/configure --version | awk 'NR == 1 {print $NF}')
+                  cd ..
+                  mv -v varnish-dist.tar.gz varnish-${VERSION}.tar.gz
+      - unless:
+          condition: << pipeline.parameters.dist-url >>
+          steps:
+            - run:
+                name: Create the dist tarball
+                command: |
+                  git checkout << pipeline.parameters.vc-commit >>
+                  # Locally built tarballs are always weekly - built with date in package name
+                  touch .is_weekly
+                  ./autogen.des --quiet
+                  make dist -j 16
       - persist_to_workspace:
           root: .
           paths:

From martin at varnish-software.com  Thu Mar 18 12:01:06 2021
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Thu, 18 Mar 2021 12:01:06 +0000 (UTC)
Subject: [6.6] 6da4f0240 Make the nightly into the commit workflow
Message-ID: <20210318120106.50B91A8742@lists.varnish-cache.org>


commit 6da4f02402a0ed92b6ea2f91ed962f012b5b2824
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Wed Mar 17 16:24:55 2021 +0100

    Make the nightly into the commit workflow

diff --git a/.circleci/config.yml b/.circleci/config.yml
index 5cfc6686c..7143b68d8 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -308,38 +308,6 @@ jobs:
 workflows:
   version: 2
   commit:
-    jobs:
-      - dist
-      - distcheck:
-          name: distcheck_centos_7
-          dist: centos
-          release: "7"
-      - distcheck:
-          name: distcheck_centos_8
-          dist: centos
-          release: "8"
-      - distcheck:
-          name: distcheck_debian_buster
-          dist: debian
-          release: buster
-          extra_conf: --enable-asan --enable-ubsan
-      - distcheck:
-          name: distcheck_alpine
-          dist: alpine
-          release: "latest"
-          #extra_conf: --without-jemalloc
-      - distcheck:
-          name: distcheck_archlinux
-          dist: archlinux
-          release: "latest"
-  nightly:
-    triggers:
-      - schedule:
-          cron: "0 4 * * *"
-          filters:
-            branches:
-              only:
-                - master
     jobs:
       - dist
       - tar_pkg_tools

From martin at varnish-software.com  Thu Mar 18 12:01:06 2021
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Thu, 18 Mar 2021 12:01:06 +0000 (UTC)
Subject: [6.6] 48db9c98f Change the pkg-varnish-cache branch to 6.6
Message-ID: <20210318120106.68723A8745@lists.varnish-cache.org>


commit 48db9c98f1cacc2da5ceed5713a91df9eb704673
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Thu Mar 18 12:59:51 2021 +0100

    Change the pkg-varnish-cache branch to 6.6

diff --git a/.circleci/config.yml b/.circleci/config.yml
index 7143b68d8..4eb783a6b 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -12,7 +12,7 @@ parameters:
     default: "HEAD"
   pkg-commit:
     type: string
-    default: "master"
+    default: "6.6"
   dist-url:
     type: string
     default: ""

From nils.goroll at uplex.de  Mon Mar 22 15:12:05 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon, 22 Mar 2021 15:12:05 +0000 (UTC)
Subject: [master] e8155dad5 Polish testcase
Message-ID: <20210322151205.93782708B@lists.varnish-cache.org>


commit e8155dad547072d7ae8b25f77ee75c61241c2225
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Mar 15 17:16:43 2021 +0100

    Polish testcase
    
    we only use one passed and one piped requests, so our servers should
    only ever see two requests.
    
    Motivated by a failure on fedora armv7hl, which I do not understand:
    
    **   c1    === txreq -url /1 -hdr "pipe: true"
    **** c1    txreq|GET /1 HTTP/1.1\r
    **** c1    txreq|pipe: true\r
    **** c1    txreq|Host: 127.0.0.1\r
    **** c1    txreq|\r
    **   c1    === rxresp
    **** dT    3.832
    **** c1    rxhdrlen = 0
    ---- c1    HTTP header is incomplete
    **** dT    3.833

diff --git a/bin/varnishtest/tests/d00029.vtc b/bin/varnishtest/tests/d00029.vtc
index 5bacdae45..26fc5b0b5 100644
--- a/bin/varnishtest/tests/d00029.vtc
+++ b/bin/varnishtest/tests/d00029.vtc
@@ -1,16 +1,16 @@
 varnishtest "shard director LAZY - d18.vtc"
 
-server s1 -repeat 3 {
+server s1 -repeat 2 {
 	rxreq
 	txresp -body "ech3Ooj"
 } -start
 
-server s2 -repeat 3 {
+server s2 -repeat 2 {
 	rxreq
 	txresp -body "ieQu2qua"
 } -start
 
-server s3 -repeat 3 {
+server s3 -repeat 2 {
 	rxreq
 	txresp -body "xiuFi3Pe"
 } -start

From nils.goroll at uplex.de  Mon Mar 22 15:13:05 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon, 22 Mar 2021 15:13:05 +0000 (UTC)
Subject: [master] d14018ca6 Add a NULL VCL_STRANDS
Message-ID: <20210322151305.572B27343@lists.varnish-cache.org>


commit d14018ca607c1ff7b1355360094aaea7130d696c
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Mar 10 17:16:12 2021 +0100

    Add a NULL VCL_STRANDS
    
    ... and document it
    
    The size of 1 for the p array would not be required, it is a safety
    measure only.

diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index a81b6ba41..5cdaf2b3f 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -56,6 +56,10 @@
 
 const void * const vrt_magic_string_end = &vrt_magic_string_end;
 const void * const vrt_magic_string_unset = &vrt_magic_string_unset;
+const struct strands *vrt_null_strands = &(struct strands){
+	.n = 0,
+	.p = (const char *[1]){NULL}
+};
 
 /*--------------------------------------------------------------------*/
 
diff --git a/include/vrt.h b/include/vrt.h
index 01ca336c5..021d0613a 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -238,6 +238,11 @@ struct strands {
 	const char	**p;
 };
 
+/*
+ * A VCL_STRANDS return value must never be NULL. Use this instead
+ */
+extern const struct strands *vrt_null_strands;
+
 /***********************************************************************
  * VCL_BLOB:
  *

From nils.goroll at uplex.de  Mon Mar 22 15:16:05 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon, 22 Mar 2021 15:16:05 +0000 (UTC)
Subject: [master] 3514c8179 Update vrt.h
Message-ID: <20210322151605.9B73D7640@lists.varnish-cache.org>


commit 3514c817947eef69ee0d9cdd68e690aebf72da1b
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Mar 22 16:14:52 2021 +0100

    Update vrt.h

diff --git a/include/vrt.h b/include/vrt.h
index 021d0613a..1d15464d7 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -53,6 +53,8 @@
  * Whenever something is deleted or changed in a way which is not
  * binary/load-time compatible, increment MAJOR version
  *
+ * NEXT (2021-09-15)
+	vrt_null_strands added
  * 13.0 (2021-03-15)
  *	Move VRT_synth_page() to deprecated status
  *	Add VRT_synth_strands() and VRT_synth_blob()

From phk at FreeBSD.org  Tue Mar 23 09:39:05 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 23 Mar 2021 09:39:05 +0000 (UTC)
Subject: [master] cd3e4c616 We no longer use "anon" ACL's to compare VCL_IP
Message-ID: <20210323093905.72687ADEB1@lists.varnish-cache.org>


commit cd3e4c61629076474d046d64d779128431978966
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 23 09:37:37 2021 +0000

    We no longer use "anon" ACL's to compare VCL_IP

diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index b8e5d90d5..213dbc37b 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -422,7 +422,7 @@ vcc_acl_entry(struct vcc *tl)
  */
 
 static void
-vcc_acl_emit(struct vcc *tl, const char *name, const char *rname, int anon)
+vcc_acl_emit(struct vcc *tl, const char *name, const char *rname)
 {
 	struct acl_e *ae;
 	int depth, l, m, i;
@@ -433,7 +433,7 @@ vcc_acl_emit(struct vcc *tl, const char *name, const char *rname, int anon)
 
 	func = VSB_new_auto();
 	AN(func);
-	VSB_printf(func, "match_acl_%s_", anon ? "anon" : "named");
+	VSB_printf(func, "match_acl_");
 	VCC_PrintCName(func, name, NULL);
 	AZ(VSB_finish(func));
 
@@ -448,7 +448,7 @@ vcc_acl_emit(struct vcc *tl, const char *name, const char *rname, int anon)
 	Fh(tl, 0, "\t\tVPI_acl_log(ctx, \"NO_FAM %s\");\n", name);
 	Fh(tl, 0, "\t\treturn(0);\n");
 	Fh(tl, 0, "\t}\n\n");
-	if (!tl->err_unref && !anon) {
+	if (!tl->err_unref) {
 		ifp = New_IniFin(tl);
 		VSB_printf(ifp->ini,
 			"\tif (0) %s(0, 0);\n", VSB_data(func));
@@ -498,27 +498,25 @@ vcc_acl_emit(struct vcc *tl, const char *name, const char *rname, int anon)
 
 		i = ((int)ae->mask + 7) / 8;
 
-		if (!anon) {
-			Fh(tl, 0, "\t%*sVPI_acl_log(ctx, \"%sMATCH %s \" ",
-			    -i, "", ae->not ? "NEG_" : "", name);
-			t = ae->t_addr;
-			do {
-				if (t->tok == CSTR) {
-					Fh(tl, 0, " \"\\\"\" ");
-					EncToken(tl->fh, t);
-					Fh(tl, 0, " \"\\\"\" ");
-				} else
-					Fh(tl, 0, " \"%.*s\"", PF(t));
-				if (t == ae->t_mask)
-					break;
-				t = VTAILQ_NEXT(t, list);
-				AN(t);
-			} while (ae->t_mask != NULL);
-			if (ae->fixed)
-				Fh(tl, 0, "\" fixed: %s\"",
-				   ae->fixed);
-			Fh(tl, 0, ");\n");
-		}
+		Fh(tl, 0, "\t%*sVPI_acl_log(ctx, \"%sMATCH %s \" ",
+		    -i, "", ae->not ? "NEG_" : "", name);
+		t = ae->t_addr;
+		do {
+			if (t->tok == CSTR) {
+				Fh(tl, 0, " \"\\\"\" ");
+				EncToken(tl->fh, t);
+				Fh(tl, 0, " \"\\\"\" ");
+			} else
+				Fh(tl, 0, " \"%.*s\"", PF(t));
+			if (t == ae->t_mask)
+				break;
+			t = VTAILQ_NEXT(t, list);
+			AN(t);
+		} while (ae->t_mask != NULL);
+		if (ae->fixed)
+			Fh(tl, 0, "\" fixed: %s\"",
+			   ae->fixed);
+		Fh(tl, 0, ");\n");
 
 		Fh(tl, 0, "\t%*sreturn (%d);\n", -i, "", ae->not ? 0 : 1);
 	}
@@ -528,18 +526,15 @@ vcc_acl_emit(struct vcc *tl, const char *name, const char *rname, int anon)
 		Fh(tl, 0, "\t%*.*s}\n", depth, depth, "");
 
 	/* Deny by default */
-	if (!anon)
-		Fh(tl, 0, "\tVPI_acl_log(ctx, \"NO_MATCH %s\");\n", name);
+	Fh(tl, 0, "\tVPI_acl_log(ctx, \"NO_MATCH %s\");\n", name);
 	Fh(tl, 0, "\treturn (0);\n}\n");
 
-	if (!anon) {
-		/* Emit the struct that will be referenced */
-		Fh(tl, 0, "\nconst struct vrt_acl %s[] = {{\n", rname);
-		Fh(tl, 0, "\t.magic = VRT_ACL_MAGIC,\n");
-		Fh(tl, 0, "\t.match = &%s,\n", VSB_data(func));
-		Fh(tl, 0, "\t.name = \"%s\",\n", name);
-		Fh(tl, 0, "}};\n\n");
-	}
+	/* Emit the struct that will be referenced */
+	Fh(tl, 0, "\nconst struct vrt_acl %s[] = {{\n", rname);
+	Fh(tl, 0, "\t.magic = VRT_ACL_MAGIC,\n");
+	Fh(tl, 0, "\t.match = &%s,\n", VSB_data(func));
+	Fh(tl, 0, "\t.name = \"%s\",\n", name);
+	Fh(tl, 0, "}};\n\n");
 	VSB_destroy(&func);
 }
 
@@ -566,5 +561,5 @@ vcc_ParseAcl(struct vcc *tl)
 	}
 	SkipToken(tl, '}');
 
-	vcc_acl_emit(tl, sym->name, sym->rname, 0);
+	vcc_acl_emit(tl, sym->name, sym->rname);
 }

From phk at FreeBSD.org  Tue Mar 23 09:53:04 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 23 Mar 2021 09:53:04 +0000 (UTC)
Subject: [master] e1ac59335 Update vtree.h from FreeBSD
Message-ID: <20210323095304.8B764AE676@lists.varnish-cache.org>


commit e1ac59335f749b84280722425b355be344cc76e9
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 23 09:47:29 2021 +0000

    Update vtree.h from FreeBSD

diff --git a/flint.lnt b/flint.lnt
index 4b527f423..0bc076fce 100644
--- a/flint.lnt
+++ b/flint.lnt
@@ -202,7 +202,10 @@
 ///////////////////////////////////////////////////////////////////////
 // <vtree.h>
 
--emacro(801, VRBT_*)		// goto considered bad
+// -emacro(801, VRBT_*)		// goto considered bad
+-emacro(527, VRBT_*)		// unreachable code
+-emacro(740, VRBT_*)		// unusual pointer cast
+-emacro(438, VRBT_*)		// last value assigned not used
 -esym(534, *_VRBT_REMOVE)	// ignore retval
 -esym(534, *_VRBT_INSERT)	// ignore retval
 
diff --git a/include/vtree.h b/include/vtree.h
index 647917690..298a9f72d 100644
--- a/include/vtree.h
+++ b/include/vtree.h
@@ -1,6 +1,6 @@
 /*	$NetBSD: tree.h,v 1.8 2004/03/28 19:38:30 provos Exp $	*/
 /*	$OpenBSD: tree.h,v 1.7 2002/10/17 21:51:54 art Exp $	*/
-/* $FreeBSD: release/9.0.0/sys/sys/tree.h 189204 2009-03-01 04:57:23Z bms $ */
+/* $FreeBSD$ */
 
 /*-
  * SPDX-License-Identifier: BSD-2-Clause-FreeBSD
@@ -32,9 +32,10 @@
 #ifndef	_VTREE_H_
 #define	_VTREE_H_
 
+
 /*
  * This file defines data structures for different types of trees:
- * splay trees and red-black trees.
+ * splay trees and rank-balanced trees.
  *
  * A splay tree is a self-organizing data structure.  Every operation
  * on the tree causes a splay to happen.  The splay moves the requested
@@ -48,15 +49,24 @@
  * and n inserts on an initially empty tree as O((m + n)lg n).  The
  * amortized cost for a sequence of m accesses to a splay tree is O(lg n);
  *
- * A red-black tree is a binary search tree with the node color as an
- * extra attribute.  It fulfills a set of conditions:
- *	- every search path from the root to a leaf consists of the
- *	  same number of black nodes,
- *	- each red node (except for the root) has a black parent,
- *	- each leaf node is black.
+ * A rank-balanced tree is a binary search tree with an integer
+ * rank-difference as an attribute of each pointer from parent to child.
+ * The sum of the rank-differences on any path from a node down to null is
+ * the same, and defines the rank of that node. The rank of the null node
+ * is -1.
+ *
+ * Different additional conditions define different sorts of balanced
+ * trees, including "red-black" and "AVL" trees.  The set of conditions
+ * applied here are the "weak-AVL" conditions of Haeupler, Sen and Tarjan:
+ *	- every rank-difference is 1 or 2.
+ *	- the rank of any leaf is 1.
+ *
+ * For historical reasons, rank differences that are even are associated
+ * with the color red (Rank-Even-Difference), and the child that a red edge
+ * points to is called a red child.
  *
- * Every operation on a red-black tree is bounded as O(lg n).
- * The maximum height of a red-black tree is 2lg (n+1).
+ * Every operation on a rank-balanced tree is bounded as O(lg n).
+ * The maximum height of a rank-balanced tree is 2lg (n+1).
  */
 
 #define VSPLAY_HEAD(name, type)						\
@@ -64,7 +74,7 @@ struct name {								\
 	struct type *sph_root; /* root of the tree */			\
 }
 
-#define VSPLAY_INITIALIZER(root)					\
+#define VSPLAY_INITIALIZER(root)						\
 	{ NULL }
 
 #define VSPLAY_INIT(root) do {						\
@@ -84,13 +94,13 @@ struct {								\
 
 /* VSPLAY_ROTATE_{LEFT,RIGHT} expect that tmp hold VSPLAY_{RIGHT,LEFT} */
 #define VSPLAY_ROTATE_RIGHT(head, tmp, field) do {			\
-	VSPLAY_LEFT((head)->sph_root, field) = VSPLAY_RIGHT(tmp, field);\
+	VSPLAY_LEFT((head)->sph_root, field) = VSPLAY_RIGHT(tmp, field);	\
 	VSPLAY_RIGHT(tmp, field) = (head)->sph_root;			\
 	(head)->sph_root = tmp;						\
 } while (/*CONSTCOND*/ 0)
 
 #define VSPLAY_ROTATE_LEFT(head, tmp, field) do {			\
-	VSPLAY_RIGHT((head)->sph_root, field) = VSPLAY_LEFT(tmp, field);\
+	VSPLAY_RIGHT((head)->sph_root, field) = VSPLAY_LEFT(tmp, field);	\
 	VSPLAY_LEFT(tmp, field) = (head)->sph_root;			\
 	(head)->sph_root = tmp;						\
 } while (/*CONSTCOND*/ 0)
@@ -98,7 +108,7 @@ struct {								\
 #define VSPLAY_LINKLEFT(head, tmp, field) do {				\
 	VSPLAY_LEFT(tmp, field) = (head)->sph_root;			\
 	tmp = (head)->sph_root;						\
-	(head)->sph_root = VSPLAY_LEFT((head)->sph_root, field);	\
+	(head)->sph_root = VSPLAY_LEFT((head)->sph_root, field);		\
 } while (/*CONSTCOND*/ 0)
 
 #define VSPLAY_LINKRIGHT(head, tmp, field) do {				\
@@ -108,22 +118,22 @@ struct {								\
 } while (/*CONSTCOND*/ 0)
 
 #define VSPLAY_ASSEMBLE(head, node, left, right, field) do {		\
-	VSPLAY_RIGHT(left, field) = VSPLAY_LEFT((head)->sph_root, field);\
+	VSPLAY_RIGHT(left, field) = VSPLAY_LEFT((head)->sph_root, field);	\
 	VSPLAY_LEFT(right, field) = VSPLAY_RIGHT((head)->sph_root, field);\
-	VSPLAY_LEFT((head)->sph_root, field) = VSPLAY_RIGHT(node, field);\
-	VSPLAY_RIGHT((head)->sph_root, field) = VSPLAY_LEFT(node, field);\
+	VSPLAY_LEFT((head)->sph_root, field) = VSPLAY_RIGHT(node, field);	\
+	VSPLAY_RIGHT((head)->sph_root, field) = VSPLAY_LEFT(node, field);	\
 } while (/*CONSTCOND*/ 0)
 
 /* Generates prototypes and inline functions */
 
-#define VSPLAY_PROTOTYPE(name, type, field, cmp)			\
+#define VSPLAY_PROTOTYPE(name, type, field, cmp)				\
 void name##_VSPLAY(struct name *, struct type *);			\
 void name##_VSPLAY_MINMAX(struct name *, int);				\
-struct type *name##_VSPLAY_INSERT(struct name *, struct type *);	\
-struct type *name##_VSPLAY_REMOVE(struct name *, struct type *);	\
+struct type *name##_VSPLAY_INSERT(struct name *, struct type *);		\
+struct type *name##_VSPLAY_REMOVE(struct name *, struct type *);		\
 									\
 /* Finds the node with the same key as elm */				\
-static __inline struct type *						\
+static v_unused_ __inline struct type *					\
 name##_VSPLAY_FIND(struct name *head, struct type *elm)			\
 {									\
 	if (VSPLAY_EMPTY(head))						\
@@ -134,7 +144,7 @@ name##_VSPLAY_FIND(struct name *head, struct type *elm)			\
 	return (NULL);							\
 }									\
 									\
-static __inline struct type *						\
+static v_unused_ __inline struct type *					\
 name##_VSPLAY_NEXT(struct name *head, struct type *elm)			\
 {									\
 	name##_VSPLAY(head, elm);					\
@@ -148,10 +158,10 @@ name##_VSPLAY_NEXT(struct name *head, struct type *elm)			\
 	return (elm);							\
 }									\
 									\
-static __inline struct type *						\
+static v_unused_ __inline struct type *					\
 name##_VSPLAY_MIN_MAX(struct name *head, int val)			\
 {									\
-	name##_VSPLAY_MINMAX(head, val);				\
+	name##_VSPLAY_MINMAX(head, val);					\
 	return (VSPLAY_ROOT(head));					\
 }
 
@@ -168,10 +178,10 @@ name##_VSPLAY_INSERT(struct name *head, struct type *elm)		\
 	    int __comp;							\
 	    name##_VSPLAY(head, elm);					\
 	    __comp = (cmp)(elm, (head)->sph_root);			\
-	    if (__comp < 0) {						\
+	    if(__comp < 0) {						\
 		    VSPLAY_LEFT(elm, field) = VSPLAY_LEFT((head)->sph_root, field);\
-		    VSPLAY_RIGHT(elm, field) = (head)->sph_root;	\
-		    VSPLAY_LEFT((head)->sph_root, field) = NULL;	\
+		    VSPLAY_RIGHT(elm, field) = (head)->sph_root;		\
+		    VSPLAY_LEFT((head)->sph_root, field) = NULL;		\
 	    } else if (__comp > 0) {					\
 		    VSPLAY_RIGHT(elm, field) = VSPLAY_RIGHT((head)->sph_root, field);\
 		    VSPLAY_LEFT(elm, field) = (head)->sph_root;		\
@@ -219,7 +229,7 @@ name##_VSPLAY(struct name *head, struct type *elm)			\
 			if (__tmp == NULL)				\
 				break;					\
 			if ((cmp)(elm, __tmp) < 0){			\
-				VSPLAY_ROTATE_RIGHT(head, __tmp, field);\
+				VSPLAY_ROTATE_RIGHT(head, __tmp, field);	\
 				if (VSPLAY_LEFT((head)->sph_root, field) == NULL)\
 					break;				\
 			}						\
@@ -255,7 +265,7 @@ void name##_VSPLAY_MINMAX(struct name *head, int __comp) \
 			if (__tmp == NULL)				\
 				break;					\
 			if (__comp < 0){				\
-				VSPLAY_ROTATE_RIGHT(head, __tmp, field);\
+				VSPLAY_ROTATE_RIGHT(head, __tmp, field);	\
 				if (VSPLAY_LEFT((head)->sph_root, field) == NULL)\
 					break;				\
 			}						\
@@ -292,7 +302,7 @@ void name##_VSPLAY_MINMAX(struct name *head, int __comp) \
 	     (x) != NULL;						\
 	     (x) = VSPLAY_NEXT(name, head, x))
 
-/* Macros that define a red-black tree */
+/* Macros that define a rank-balanced tree */
 #define VRBT_HEAD(name, type)						\
 struct name {								\
 	struct type *rbh_root; /* root of the tree */			\
@@ -305,295 +315,326 @@ struct name {								\
 	(root)->rbh_root = NULL;					\
 } while (/*CONSTCOND*/ 0)
 
-#define VRBT_BLACK	0
-#define VRBT_RED		1
-#define VRBT_ENTRY(type)						\
+#define VRBT_ENTRY(type)							\
 struct {								\
 	struct type *rbe_left;		/* left element */		\
 	struct type *rbe_right;		/* right element */		\
 	struct type *rbe_parent;	/* parent element */		\
-	int rbe_color;			/* node color */		\
 }
 
 #define VRBT_LEFT(elm, field)		(elm)->field.rbe_left
 #define VRBT_RIGHT(elm, field)		(elm)->field.rbe_right
-#define VRBT_PARENT(elm, field)		(elm)->field.rbe_parent
-#define VRBT_COLOR(elm, field)		(elm)->field.rbe_color
+
+/*
+ * With the expectation that any object of struct type has an
+ * address that is a multiple of 4, and that therefore the
+ * 2 least significant bits of a pointer to struct type are
+ * always zero, this implementation sets those bits to indicate
+ * that the left or right child of the tree node is "red".
+ */
+#define VRBT_UP(elm, field)		(elm)->field.rbe_parent
+#define VRBT_BITS(elm, field)		(*(__uintptr_t *)&VRBT_UP(elm, field))
+#define VRBT_RED_L			((__uintptr_t)1)
+#define VRBT_RED_R			((__uintptr_t)2)
+#define VRBT_RED_MASK			((__uintptr_t)3)
+#define VRBT_FLIP_LEFT(elm, field)	(VRBT_BITS(elm, field) ^= VRBT_RED_L)
+#define VRBT_FLIP_RIGHT(elm, field)	(VRBT_BITS(elm, field) ^= VRBT_RED_R)
+#define VRBT_RED_LEFT(elm, field)		((VRBT_BITS(elm, field) & VRBT_RED_L) != 0)
+#define VRBT_RED_RIGHT(elm, field)	((VRBT_BITS(elm, field) & VRBT_RED_R) != 0)
+#define VRBT_PARENT(elm, field)		((__typeof(VRBT_UP(elm, field)))	\
+					 (VRBT_BITS(elm, field) & ~VRBT_RED_MASK))
 #define VRBT_ROOT(head)			(head)->rbh_root
-#define VRBT_EMPTY(head)		(VRBT_ROOT(head) == NULL)
+#define VRBT_EMPTY(head)			(VRBT_ROOT(head) == NULL)
 
-#define VRBT_SET(elm, parent, field) do {				\
-	VRBT_PARENT(elm, field) = parent;				\
-	VRBT_LEFT(elm, field) = VRBT_RIGHT(elm, field) = NULL;		\
-	VRBT_COLOR(elm, field) = VRBT_RED;				\
+#define VRBT_SET_PARENT(dst, src, field) do {				\
+	VRBT_BITS(dst, field) &= VRBT_RED_MASK;				\
+	VRBT_BITS(dst, field) |= (__uintptr_t)src;			\
 } while (/*CONSTCOND*/ 0)
 
-#define VRBT_SET_BLACKRED(black, red, field) do {			\
-	VRBT_COLOR(black, field) = VRBT_BLACK;				\
-	VRBT_COLOR(red, field) = VRBT_RED;				\
+#define VRBT_SET(elm, parent, field) do {					\
+	VRBT_UP(elm, field) = parent;					\
+	VRBT_LEFT(elm, field) = VRBT_RIGHT(elm, field) = NULL;		\
 } while (/*CONSTCOND*/ 0)
 
+#define VRBT_COLOR(elm, field)	(VRBT_PARENT(elm, field) == NULL ? 0 :	\
+				VRBT_LEFT(VRBT_PARENT(elm, field), field) == elm ? \
+				VRBT_RED_LEFT(VRBT_PARENT(elm, field), field) : \
+				VRBT_RED_RIGHT(VRBT_PARENT(elm, field), field))
+
+/*
+ * Something to be invoked in a loop at the root of every modified subtree,
+ * from the bottom up to the root, to update augmented node data.
+ */
 #ifndef VRBT_AUGMENT
-#define VRBT_AUGMENT(x)	do {} while (0)
+#define VRBT_AUGMENT(x)	break
 #endif
 
+#define VRBT_SWAP_CHILD(head, out, in, field) do {			\
+	if (VRBT_PARENT(out, field) == NULL)				\
+		VRBT_ROOT(head) = (in);					\
+	else if ((out) == VRBT_LEFT(VRBT_PARENT(out, field), field))	\
+		VRBT_LEFT(VRBT_PARENT(out, field), field) = (in);		\
+	else								\
+		VRBT_RIGHT(VRBT_PARENT(out, field), field) = (in);		\
+} while (/*CONSTCOND*/ 0)
+
 #define VRBT_ROTATE_LEFT(head, elm, tmp, field) do {			\
 	(tmp) = VRBT_RIGHT(elm, field);					\
 	if ((VRBT_RIGHT(elm, field) = VRBT_LEFT(tmp, field)) != NULL) {	\
-		VRBT_PARENT(VRBT_LEFT(tmp, field), field) = (elm);	\
+		VRBT_SET_PARENT(VRBT_RIGHT(elm, field), elm, field);	\
 	}								\
-	VRBT_AUGMENT(elm);						\
-	if ((VRBT_PARENT(tmp, field) = VRBT_PARENT(elm, field)) != NULL) {\
-		if ((elm) == VRBT_LEFT(VRBT_PARENT(elm, field), field))	\
-			VRBT_LEFT(VRBT_PARENT(elm, field), field) = (tmp);\
-		else							\
-			VRBT_RIGHT(VRBT_PARENT(elm, field), field) = (tmp);\
-	} else								\
-		(head)->rbh_root = (tmp);				\
+	VRBT_SET_PARENT(tmp, VRBT_PARENT(elm, field), field);		\
+	VRBT_SWAP_CHILD(head, elm, tmp, field);				\
 	VRBT_LEFT(tmp, field) = (elm);					\
-	VRBT_PARENT(elm, field) = (tmp);				\
-	VRBT_AUGMENT(tmp);						\
-	if ((VRBT_PARENT(tmp, field)))					\
-		VRBT_AUGMENT(VRBT_PARENT(tmp, field));			\
+	VRBT_SET_PARENT(elm, tmp, field);					\
+	VRBT_AUGMENT(elm);						\
 } while (/*CONSTCOND*/ 0)
 
 #define VRBT_ROTATE_RIGHT(head, elm, tmp, field) do {			\
 	(tmp) = VRBT_LEFT(elm, field);					\
 	if ((VRBT_LEFT(elm, field) = VRBT_RIGHT(tmp, field)) != NULL) {	\
-		VRBT_PARENT(VRBT_RIGHT(tmp, field), field) = (elm);	\
+		VRBT_SET_PARENT(VRBT_LEFT(elm, field), elm, field);		\
 	}								\
-	VRBT_AUGMENT(elm);						\
-	if ((VRBT_PARENT(tmp, field) = VRBT_PARENT(elm, field)) != NULL) {\
-		if ((elm) == VRBT_LEFT(VRBT_PARENT(elm, field), field))	\
-			VRBT_LEFT(VRBT_PARENT(elm, field), field) = (tmp);\
-		else							\
-			VRBT_RIGHT(VRBT_PARENT(elm, field), field) = (tmp);\
-	} else								\
-		(head)->rbh_root = (tmp);				\
+	VRBT_SET_PARENT(tmp, VRBT_PARENT(elm, field), field);		\
+	VRBT_SWAP_CHILD(head, elm, tmp, field);				\
 	VRBT_RIGHT(tmp, field) = (elm);					\
-	VRBT_PARENT(elm, field) = (tmp);				\
-	VRBT_AUGMENT(tmp);						\
-	if ((VRBT_PARENT(tmp, field)))					\
-		VRBT_AUGMENT(VRBT_PARENT(tmp, field));			\
+	VRBT_SET_PARENT(elm, tmp, field);					\
+	VRBT_AUGMENT(elm);						\
 } while (/*CONSTCOND*/ 0)
 
 /* Generates prototypes and inline functions */
-#define	VRBT_PROTOTYPE(name, type, field, cmp)			\
+#define	VRBT_PROTOTYPE(name, type, field, cmp)				\
 	VRBT_PROTOTYPE_INTERNAL(name, type, field, cmp,)
-#define	VRBT_PROTOTYPE_STATIC(name, type, field, cmp)		\
+#define	VRBT_PROTOTYPE_STATIC(name, type, field, cmp)			\
 	VRBT_PROTOTYPE_INTERNAL(name, type, field, cmp, v_unused_ static)
 #define VRBT_PROTOTYPE_INTERNAL(name, type, field, cmp, attr)		\
-/*lint -esym(528, name##_VRBT_*) */					\
-attr void name##_VRBT_INSERT_COLOR(struct name *, struct type *);	\
-attr void name##_VRBT_REMOVE_COLOR(struct name *, struct type *, struct type *);\
-attr struct type *name##_VRBT_REMOVE(struct name *, struct type *);	\
-attr struct type *name##_VRBT_INSERT(struct name *, struct type *);	\
-attr struct type *name##_VRBT_FIND(const struct name *, const struct type *);	\
-attr struct type *name##_VRBT_NFIND(const struct name *, const struct type *);	\
-attr struct type *name##_VRBT_NEXT(struct type *);			\
-attr struct type *name##_VRBT_PREV(struct type *);			\
-attr struct type *name##_VRBT_MINMAX(const struct name *, int);		\
-									\
+	VRBT_PROTOTYPE_INSERT_COLOR(name, type, attr);			\
+	VRBT_PROTOTYPE_REMOVE_COLOR(name, type, attr);			\
+	VRBT_PROTOTYPE_INSERT(name, type, attr);				\
+	VRBT_PROTOTYPE_REMOVE(name, type, attr);				\
+	VRBT_PROTOTYPE_FIND(name, type, attr);				\
+	VRBT_PROTOTYPE_NFIND(name, type, attr);				\
+	VRBT_PROTOTYPE_NEXT(name, type, attr);				\
+	VRBT_PROTOTYPE_PREV(name, type, attr);				\
+	VRBT_PROTOTYPE_MINMAX(name, type, attr);				\
+	VRBT_PROTOTYPE_REINSERT(name, type, attr);
+#define VRBT_PROTOTYPE_INSERT_COLOR(name, type, attr)			\
+	attr void name##_VRBT_INSERT_COLOR(struct name *, struct type *)
+#define VRBT_PROTOTYPE_REMOVE_COLOR(name, type, attr)			\
+	attr void name##_VRBT_REMOVE_COLOR(struct name *,			\
+	    struct type *, struct type *)
+#define VRBT_PROTOTYPE_REMOVE(name, type, attr)				\
+	attr struct type *name##_VRBT_REMOVE(struct name *, struct type *)
+#define VRBT_PROTOTYPE_INSERT(name, type, attr)				\
+	attr struct type *name##_VRBT_INSERT(struct name *, struct type *)
+#define VRBT_PROTOTYPE_FIND(name, type, attr)				\
+	attr struct type *name##_VRBT_FIND(const struct name *, const struct type *)
+#define VRBT_PROTOTYPE_NFIND(name, type, attr)				\
+	attr struct type *name##_VRBT_NFIND(const struct name *, const struct type *)
+#define VRBT_PROTOTYPE_NEXT(name, type, attr)				\
+	attr struct type *name##_VRBT_NEXT(struct type *)
+#define VRBT_PROTOTYPE_PREV(name, type, attr)				\
+	attr struct type *name##_VRBT_PREV(struct type *)
+#define VRBT_PROTOTYPE_MINMAX(name, type, attr)				\
+	attr struct type *name##_VRBT_MINMAX(const struct name *, int)
+#define VRBT_PROTOTYPE_REINSERT(name, type, attr)			\
+	attr struct type *name##_VRBT_REINSERT(struct name *, struct type *)
 
 /* Main rb operation.
  * Moves node close to the key of elm to top
  */
-#define	VRBT_GENERATE(name, type, field, cmp)			\
+#define	VRBT_GENERATE(name, type, field, cmp)				\
 	VRBT_GENERATE_INTERNAL(name, type, field, cmp,)
-#define	VRBT_GENERATE_STATIC(name, type, field, cmp)		\
+#define	VRBT_GENERATE_STATIC(name, type, field, cmp)			\
 	VRBT_GENERATE_INTERNAL(name, type, field, cmp, v_unused_ static)
 #define VRBT_GENERATE_INTERNAL(name, type, field, cmp, attr)		\
+	VRBT_GENERATE_INSERT_COLOR(name, type, field, attr)		\
+	VRBT_GENERATE_REMOVE_COLOR(name, type, field, attr)		\
+	VRBT_GENERATE_INSERT(name, type, field, cmp, attr)		\
+	VRBT_GENERATE_REMOVE(name, type, field, attr)			\
+	VRBT_GENERATE_FIND(name, type, field, cmp, attr)			\
+	VRBT_GENERATE_NFIND(name, type, field, cmp, attr)			\
+	VRBT_GENERATE_NEXT(name, type, field, attr)			\
+	VRBT_GENERATE_PREV(name, type, field, attr)			\
+	VRBT_GENERATE_MINMAX(name, type, field, attr)			\
+	VRBT_GENERATE_REINSERT(name, type, field, cmp, attr)
+
+#define VRBT_GENERATE_INSERT_COLOR(name, type, field, attr)		\
 attr void								\
 name##_VRBT_INSERT_COLOR(struct name *head, struct type *elm)		\
 {									\
-	struct type *parent, *gparent, *tmp;				\
-	while ((parent = VRBT_PARENT(elm, field)) != NULL &&		\
-	    VRBT_COLOR(parent, field) == VRBT_RED) {			\
-		gparent = VRBT_PARENT(parent, field);			\
-		if (parent == VRBT_LEFT(gparent, field)) {		\
-			tmp = VRBT_RIGHT(gparent, field);		\
-			if (tmp && VRBT_COLOR(tmp, field) == VRBT_RED) {\
-				VRBT_COLOR(tmp, field) = VRBT_BLACK;	\
-				VRBT_SET_BLACKRED(parent, gparent, field);\
-				elm = gparent;				\
+	struct type *child, *parent;					\
+	while ((parent = VRBT_PARENT(elm, field)) != NULL) {		\
+		if (VRBT_LEFT(parent, field) == elm) {			\
+			if (VRBT_RED_LEFT(parent, field)) {		\
+				VRBT_FLIP_LEFT(parent, field);		\
+				return;					\
+			}						\
+			VRBT_FLIP_RIGHT(parent, field);			\
+			if (VRBT_RED_RIGHT(parent, field)) {		\
+				elm = parent;				\
 				continue;				\
 			}						\
-			if (VRBT_RIGHT(parent, field) == elm) {	\
-				VRBT_ROTATE_LEFT(head, parent, tmp, field);\
-				tmp = parent;				\
-				parent = elm;				\
-				elm = tmp;				\
+			if (!VRBT_RED_RIGHT(elm, field)) {		\
+				VRBT_FLIP_LEFT(elm, field);		\
+				VRBT_ROTATE_LEFT(head, elm, child, field);\
+				if (VRBT_RED_LEFT(child, field))		\
+					VRBT_FLIP_RIGHT(elm, field);	\
+				else if (VRBT_RED_RIGHT(child, field))	\
+					VRBT_FLIP_LEFT(parent, field);	\
+		AN(parent);			\
+				elm = child;				\
 			}						\
-			VRBT_SET_BLACKRED(parent, gparent, field);	\
-			VRBT_ROTATE_RIGHT(head, gparent, tmp, field);	\
+			VRBT_ROTATE_RIGHT(head, parent, elm, field);	\
 		} else {						\
-			tmp = VRBT_LEFT(gparent, field);		\
-			if (tmp && VRBT_COLOR(tmp, field) == VRBT_RED) {\
-				VRBT_COLOR(tmp, field) = VRBT_BLACK;	\
-				VRBT_SET_BLACKRED(parent, gparent, field);\
-				elm = gparent;				\
+			if (VRBT_RED_RIGHT(parent, field)) {		\
+				VRBT_FLIP_RIGHT(parent, field);		\
+				return;					\
+			}						\
+			VRBT_FLIP_LEFT(parent, field);			\
+			if (VRBT_RED_LEFT(parent, field)) {		\
+				elm = parent;				\
 				continue;				\
 			}						\
-			if (VRBT_LEFT(parent, field) == elm) {		\
-				VRBT_ROTATE_RIGHT(head, parent, tmp, field);\
-				tmp = parent;				\
-				parent = elm;				\
-				elm = tmp;				\
+			if (!VRBT_RED_LEFT(elm, field)) {			\
+				VRBT_FLIP_RIGHT(elm, field);		\
+				VRBT_ROTATE_RIGHT(head, elm, child, field);\
+				if (VRBT_RED_RIGHT(child, field))		\
+					VRBT_FLIP_LEFT(elm, field);	\
+				else if (VRBT_RED_LEFT(child, field))	\
+					VRBT_FLIP_RIGHT(parent, field);	\
+				elm = child;				\
 			}						\
-			VRBT_SET_BLACKRED(parent, gparent, field);	\
-			VRBT_ROTATE_LEFT(head, gparent, tmp, field);	\
+			VRBT_ROTATE_LEFT(head, parent, elm, field);	\
 		}							\
+		VRBT_BITS(elm, field) &= ~VRBT_RED_MASK;			\
+		break;							\
 	}								\
-	VRBT_COLOR(head->rbh_root, field) = VRBT_BLACK;			\
-}									\
-									\
+}
+
+#define VRBT_GENERATE_REMOVE_COLOR(name, type, field, attr)		\
 attr void								\
-name##_VRBT_REMOVE_COLOR(struct name *head, struct type *parent, struct type *elm) \
+name##_VRBT_REMOVE_COLOR(struct name *head,				\
+    struct type *parent, struct type *elm)				\
 {									\
-	struct type *tmp;						\
-	while ((elm == NULL || VRBT_COLOR(elm, field) == VRBT_BLACK) &&	\
-	    elm != VRBT_ROOT(head)) {					\
-		AN(parent);						\
+	struct type *sib;						\
+	if (VRBT_LEFT(parent, field) == elm &&				\
+	    VRBT_RIGHT(parent, field) == elm) {				\
+		VRBT_BITS(parent, field) &= ~VRBT_RED_MASK;			\
+		elm = parent;						\
+		parent = VRBT_PARENT(elm, field);				\
+		if (parent == NULL)					\
+			return;						\
+	}								\
+	do  {								\
 		if (VRBT_LEFT(parent, field) == elm) {			\
-			tmp = VRBT_RIGHT(parent, field);		\
-			if (VRBT_COLOR(tmp, field) == VRBT_RED) {	\
-				VRBT_SET_BLACKRED(tmp, parent, field);	\
-				VRBT_ROTATE_LEFT(head, parent, tmp, field);\
-				tmp = VRBT_RIGHT(parent, field);	\
+			if (!VRBT_RED_LEFT(parent, field)) {		\
+				VRBT_FLIP_LEFT(parent, field);		\
+				return;					\
 			}						\
-			if ((VRBT_LEFT(tmp, field) == NULL ||		\
-			    VRBT_COLOR(VRBT_LEFT(tmp, field), field) == VRBT_BLACK) &&\
-			    (VRBT_RIGHT(tmp, field) == NULL ||		\
-			    VRBT_COLOR(VRBT_RIGHT(tmp, field), field) == VRBT_BLACK)) {\
-				VRBT_COLOR(tmp, field) = VRBT_RED;	\
+			if (VRBT_RED_RIGHT(parent, field)) {		\
+				VRBT_FLIP_RIGHT(parent, field);		\
 				elm = parent;				\
-				parent = VRBT_PARENT(elm, field);	\
-			} else {					\
-				if (VRBT_RIGHT(tmp, field) == NULL ||	\
-				    VRBT_COLOR(VRBT_RIGHT(tmp, field), field) == VRBT_BLACK) {\
-					struct type *oleft;		\
-					if ((oleft = VRBT_LEFT(tmp, field)) \
-					    != NULL)			\
-						VRBT_COLOR(oleft, field) = VRBT_BLACK;\
-					VRBT_COLOR(tmp, field) = VRBT_RED;\
-					VRBT_ROTATE_RIGHT(head, tmp, oleft, field);\
-					tmp = VRBT_RIGHT(parent, field);\
-				}					\
-				VRBT_COLOR(tmp, field) = VRBT_COLOR(parent, field);\
-				VRBT_COLOR(parent, field) = VRBT_BLACK;	\
-				if (VRBT_RIGHT(tmp, field))		\
-					VRBT_COLOR(VRBT_RIGHT(tmp, field), field) = VRBT_BLACK;\
-				VRBT_ROTATE_LEFT(head, parent, tmp, field);\
-				elm = VRBT_ROOT(head);			\
-				break;					\
+				continue;				\
 			}						\
+			sib = VRBT_RIGHT(parent, field);			\
+			if ((~VRBT_BITS(sib, field) & VRBT_RED_MASK) == 0) {\
+				VRBT_BITS(sib, field) &= ~VRBT_RED_MASK;	\
+				elm = parent;				\
+				continue;				\
+			}						\
+			VRBT_FLIP_RIGHT(sib, field);			\
+			if (VRBT_RED_LEFT(sib, field))			\
+				VRBT_FLIP_LEFT(parent, field);		\
+			else if (!VRBT_RED_RIGHT(sib, field)) {		\
+				VRBT_FLIP_LEFT(parent, field);		\
+				VRBT_ROTATE_RIGHT(head, sib, elm, field);	\
+				if (VRBT_RED_RIGHT(elm, field))		\
+					VRBT_FLIP_LEFT(sib, field);	\
+				if (VRBT_RED_LEFT(elm, field))		\
+					VRBT_FLIP_RIGHT(parent, field);	\
+				VRBT_BITS(elm, field) |= VRBT_RED_MASK;	\
+				sib = elm;				\
+			}						\
+			VRBT_ROTATE_LEFT(head, parent, sib, field);	\
 		} else {						\
-			tmp = VRBT_LEFT(parent, field);			\
-			if (VRBT_COLOR(tmp, field) == VRBT_RED) {	\
-				VRBT_SET_BLACKRED(tmp, parent, field);	\
-				VRBT_ROTATE_RIGHT(head, parent, tmp, field);\
-				tmp = VRBT_LEFT(parent, field);		\
+			if (!VRBT_RED_RIGHT(parent, field)) {		\
+				VRBT_FLIP_RIGHT(parent, field);		\
+				return;					\
 			}						\
-			if ((VRBT_LEFT(tmp, field) == NULL ||		\
-			    VRBT_COLOR(VRBT_LEFT(tmp, field), field) == VRBT_BLACK) &&\
-			    (VRBT_RIGHT(tmp, field) == NULL ||		\
-			    VRBT_COLOR(VRBT_RIGHT(tmp, field), field) == VRBT_BLACK)) {\
-				VRBT_COLOR(tmp, field) = VRBT_RED;	\
+			if (VRBT_RED_LEFT(parent, field)) {		\
+				VRBT_FLIP_LEFT(parent, field);		\
 				elm = parent;				\
-				parent = VRBT_PARENT(elm, field);	\
-			} else {					\
-				if (VRBT_LEFT(tmp, field) == NULL ||	\
-				    VRBT_COLOR(VRBT_LEFT(tmp, field), field) == VRBT_BLACK) {\
-					struct type *oright;		\
-					if ((oright = VRBT_RIGHT(tmp, field)) \
-					    != NULL)			\
-						VRBT_COLOR(oright, field) = VRBT_BLACK;\
-					VRBT_COLOR(tmp, field) = VRBT_RED;\
-					VRBT_ROTATE_LEFT(head, tmp, oright, field);\
-					tmp = VRBT_LEFT(parent, field);	\
-				}					\
-				VRBT_COLOR(tmp, field) = VRBT_COLOR(parent, field);\
-				VRBT_COLOR(parent, field) = VRBT_BLACK;	\
-				if (VRBT_LEFT(tmp, field))		\
-					VRBT_COLOR(VRBT_LEFT(tmp, field), field) = VRBT_BLACK;\
-				VRBT_ROTATE_RIGHT(head, parent, tmp, field);\
-				elm = VRBT_ROOT(head);			\
-				break;					\
+				continue;				\
 			}						\
+			sib = VRBT_LEFT(parent, field);			\
+			if ((~VRBT_BITS(sib, field) & VRBT_RED_MASK) == 0) {\
+				VRBT_BITS(sib, field) &= ~VRBT_RED_MASK;	\
+				elm = parent;				\
+				continue;				\
+			}						\
+			VRBT_FLIP_LEFT(sib, field);			\
+			if (VRBT_RED_RIGHT(sib, field))			\
+				VRBT_FLIP_RIGHT(parent, field);		\
+			else if (!VRBT_RED_LEFT(sib, field)) {		\
+				VRBT_FLIP_RIGHT(parent, field);		\
+				VRBT_ROTATE_LEFT(head, sib, elm, field);	\
+				if (VRBT_RED_LEFT(elm, field))		\
+					VRBT_FLIP_RIGHT(sib, field);	\
+				if (VRBT_RED_RIGHT(elm, field))		\
+					VRBT_FLIP_LEFT(parent, field);	\
+				VRBT_BITS(elm, field) |= VRBT_RED_MASK;	\
+				sib = elm;				\
+			}						\
+			VRBT_ROTATE_RIGHT(head, parent, sib, field);	\
 		}							\
-	}								\
-	if (elm)							\
-		VRBT_COLOR(elm, field) = VRBT_BLACK;			\
-}									\
-									\
+		break;							\
+	} while ((parent = VRBT_PARENT(elm, field)) != NULL);		\
+}
+
+#define VRBT_GENERATE_REMOVE(name, type, field, attr)			\
 attr struct type *							\
 name##_VRBT_REMOVE(struct name *head, struct type *elm)			\
 {									\
-	struct type *child, *parent, *old = elm;			\
-	int color;							\
+	struct type *child, *old, *parent, *right;			\
+									\
+	old = elm;							\
+	parent = VRBT_PARENT(elm, field);					\
+	right = VRBT_RIGHT(elm, field);					\
 	if (VRBT_LEFT(elm, field) == NULL)				\
-		child = VRBT_RIGHT(elm, field);				\
-	else if (VRBT_RIGHT(elm, field) == NULL)			\
-		child = VRBT_LEFT(elm, field);				\
+		elm = child = right;					\
+	else if (right == NULL)						\
+		elm = child = VRBT_LEFT(elm, field);			\
 	else {								\
-		struct type *left;					\
-		elm = VRBT_RIGHT(elm, field);				\
-		while ((left = VRBT_LEFT(elm, field)) != NULL)		\
-			elm = left;					\
-		child = VRBT_RIGHT(elm, field);				\
-		parent = VRBT_PARENT(elm, field);			\
-		color = VRBT_COLOR(elm, field);				\
-		if (child)						\
-			VRBT_PARENT(child, field) = parent;		\
-		if (parent) {						\
-			if (VRBT_LEFT(parent, field) == elm)		\
-				VRBT_LEFT(parent, field) = child;	\
-			else						\
-				VRBT_RIGHT(parent, field) = child;	\
-			VRBT_AUGMENT(parent);				\
-		} else							\
-			VRBT_ROOT(head) = child;			\
-		if (VRBT_PARENT(elm, field) == old)			\
-			parent = elm;					\
-		(elm)->field = (old)->field;				\
-		if (VRBT_PARENT(old, field)) {				\
-			if (VRBT_LEFT(VRBT_PARENT(old, field), field) == old)\
-				VRBT_LEFT(VRBT_PARENT(old, field), field) = elm;\
-			else						\
-				VRBT_RIGHT(VRBT_PARENT(old, field), field) = elm;\
-			VRBT_AUGMENT(VRBT_PARENT(old, field));		\
-		} else							\
-			VRBT_ROOT(head) = elm;				\
-		VRBT_PARENT(VRBT_LEFT(old, field), field) = elm;	\
-		if (VRBT_RIGHT(old, field))				\
-			VRBT_PARENT(VRBT_RIGHT(old, field), field) = elm;\
-		if (parent) {						\
-			left = parent;					\
-			do {						\
-				VRBT_AUGMENT(left);			\
-			} while ((left = VRBT_PARENT(left, field)) != NULL);\
+		if ((child = VRBT_LEFT(right, field)) == NULL) {		\
+			child = VRBT_RIGHT(right, field);			\
+			VRBT_RIGHT(old, field) = child;			\
+			parent = elm = right;				\
+		} else {						\
+			do						\
+				elm = child;				\
+			while ((child = VRBT_LEFT(elm, field)) != NULL);	\
+			child = VRBT_RIGHT(elm, field);			\
+			parent = VRBT_PARENT(elm, field);			\
+			VRBT_LEFT(parent, field) = child;			\
+			VRBT_SET_PARENT(VRBT_RIGHT(old, field), elm, field);\
 		}							\
-		goto color;						\
+		VRBT_SET_PARENT(VRBT_LEFT(old, field), elm, field);		\
+		elm->field = old->field;				\
 	}								\
-	parent = VRBT_PARENT(elm, field);				\
-	color = VRBT_COLOR(elm, field);					\
-	if (child)							\
-		VRBT_PARENT(child, field) = parent;			\
-	if (parent) {							\
-		if (VRBT_LEFT(parent, field) == elm)			\
-			VRBT_LEFT(parent, field) = child;		\
-		else							\
-			VRBT_RIGHT(parent, field) = child;		\
-		VRBT_AUGMENT(parent);					\
-	} else								\
-		VRBT_ROOT(head) = child;				\
-color:									\
-	if (color == VRBT_BLACK) {					\
+	VRBT_SWAP_CHILD(head, old, elm, field);				\
+	if (child != NULL)						\
+		VRBT_SET_PARENT(child, parent, field);			\
+	if (parent != NULL)						\
 		name##_VRBT_REMOVE_COLOR(head, parent, child);		\
+	while (parent != NULL) {					\
+		VRBT_AUGMENT(parent);					\
+		parent = VRBT_PARENT(parent, field);			\
 	}								\
 	return (old);							\
-}									\
-									\
+}
+
+#define VRBT_GENERATE_INSERT(name, type, field, cmp, attr)		\
 /* Inserts a node into the RB tree */					\
 attr struct type *							\
 name##_VRBT_INSERT(struct name *head, struct type *elm)			\
@@ -613,21 +654,24 @@ name##_VRBT_INSERT(struct name *head, struct type *elm)			\
 			return (tmp);					\
 	}								\
 	VRBT_SET(elm, parent, field);					\
-	if (parent != NULL) {						\
-		if (comp < 0)						\
-			VRBT_LEFT(parent, field) = elm;			\
-		else							\
-			VRBT_RIGHT(parent, field) = elm;		\
-		VRBT_AUGMENT(parent);					\
-	} else								\
+	if (parent == NULL)						\
 		VRBT_ROOT(head) = elm;					\
+	else if (comp < 0)						\
+		VRBT_LEFT(parent, field) = elm;				\
+	else								\
+		VRBT_RIGHT(parent, field) = elm;				\
 	name##_VRBT_INSERT_COLOR(head, elm);				\
+	while (elm != NULL) {						\
+		VRBT_AUGMENT(elm);					\
+		elm = VRBT_PARENT(elm, field);				\
+	}								\
 	return (NULL);							\
-}									\
-									\
+}
+
+#define VRBT_GENERATE_FIND(name, type, field, cmp, attr)			\
 /* Finds the node with the same key as elm */				\
 attr struct type *							\
-name##_VRBT_FIND(const struct name *head, const struct type *elm)	\
+name##_VRBT_FIND(const struct name *head, const struct type *elm)			\
 {									\
 	struct type *tmp = VRBT_ROOT(head);				\
 	int comp;							\
@@ -641,11 +685,12 @@ name##_VRBT_FIND(const struct name *head, const struct type *elm)	\
 			return (tmp);					\
 	}								\
 	return (NULL);							\
-}									\
-									\
+}
+
+#define VRBT_GENERATE_NFIND(name, type, field, cmp, attr)			\
 /* Finds the first node greater than or equal to the search key */	\
 attr struct type *							\
-name##_VRBT_NFIND(const struct name *head, const struct type *elm)	\
+name##_VRBT_NFIND(const struct name *head, const struct type *elm)			\
 {									\
 	struct type *tmp = VRBT_ROOT(head);				\
 	struct type *res = NULL;					\
@@ -662,8 +707,9 @@ name##_VRBT_NFIND(const struct name *head, const struct type *elm)	\
 			return (tmp);					\
 	}								\
 	return (res);							\
-}									\
-									\
+}
+
+#define VRBT_GENERATE_NEXT(name, type, field, attr)			\
 /* ARGSUSED */								\
 attr struct type *							\
 name##_VRBT_NEXT(struct type *elm)					\
@@ -677,15 +723,16 @@ name##_VRBT_NEXT(struct type *elm)					\
 		    (elm == VRBT_LEFT(VRBT_PARENT(elm, field), field)))	\
 			elm = VRBT_PARENT(elm, field);			\
 		else {							\
-			while (VRBT_PARENT(elm, field) &&		\
+			while (VRBT_PARENT(elm, field) &&			\
 			    (elm == VRBT_RIGHT(VRBT_PARENT(elm, field), field)))\
 				elm = VRBT_PARENT(elm, field);		\
 			elm = VRBT_PARENT(elm, field);			\
 		}							\
 	}								\
 	return (elm);							\
-}									\
-									\
+}
+
+#define VRBT_GENERATE_PREV(name, type, field, attr)			\
 /* ARGSUSED */								\
 attr struct type *							\
 name##_VRBT_PREV(struct type *elm)					\
@@ -696,20 +743,21 @@ name##_VRBT_PREV(struct type *elm)					\
 			elm = VRBT_RIGHT(elm, field);			\
 	} else {							\
 		if (VRBT_PARENT(elm, field) &&				\
-		    (elm == VRBT_RIGHT(VRBT_PARENT(elm, field), field)))\
+		    (elm == VRBT_RIGHT(VRBT_PARENT(elm, field), field)))	\
 			elm = VRBT_PARENT(elm, field);			\
 		else {							\
-			while (VRBT_PARENT(elm, field) &&		\
+			while (VRBT_PARENT(elm, field) &&			\
 			    (elm == VRBT_LEFT(VRBT_PARENT(elm, field), field)))\
 				elm = VRBT_PARENT(elm, field);		\
 			elm = VRBT_PARENT(elm, field);			\
 		}							\
 	}								\
 	return (elm);							\
-}									\
-									\
+}
+
+#define VRBT_GENERATE_MINMAX(name, type, field, attr)			\
 attr struct type *							\
-name##_VRBT_MINMAX(const struct name *head, int val)			\
+name##_VRBT_MINMAX(const struct name *head, int val)				\
 {									\
 	struct type *tmp = VRBT_ROOT(head);				\
 	struct type *parent = NULL;					\
@@ -723,6 +771,22 @@ name##_VRBT_MINMAX(const struct name *head, int val)			\
 	return (parent);						\
 }
 
+#define	VRBT_GENERATE_REINSERT(name, type, field, cmp, attr)		\
+attr struct type *							\
+name##_VRBT_REINSERT(struct name *head, struct type *elm)			\
+{									\
+	struct type *cmpelm;						\
+	if (((cmpelm = VRBT_PREV(name, head, elm)) != NULL &&		\
+	    cmp(cmpelm, elm) >= 0) ||					\
+	    ((cmpelm = VRBT_NEXT(name, head, elm)) != NULL &&		\
+	    cmp(elm, cmpelm) >= 0)) {					\
+		/* XXXLAS: Remove/insert is heavy handed. */		\
+		VRBT_REMOVE(name, head, elm);				\
+		return (VRBT_INSERT(name, head, elm));			\
+	}								\
+	return (NULL);							\
+}									\
+
 #define VRBT_NEGINF	-1
 #define VRBT_INF	1
 
@@ -732,11 +796,12 @@ name##_VRBT_MINMAX(const struct name *head, int val)			\
 #define VRBT_NFIND(name, x, y)	name##_VRBT_NFIND(x, y)
 #define VRBT_NEXT(name, x, y)	name##_VRBT_NEXT(y)
 #define VRBT_PREV(name, x, y)	name##_VRBT_PREV(y)
-#define VRBT_MIN(name, x)	name##_VRBT_MINMAX(x, VRBT_NEGINF)
-#define VRBT_MAX(name, x)	name##_VRBT_MINMAX(x, VRBT_INF)
+#define VRBT_MIN(name, x)		name##_VRBT_MINMAX(x, VRBT_NEGINF)
+#define VRBT_MAX(name, x)		name##_VRBT_MINMAX(x, VRBT_INF)
+#define VRBT_REINSERT(name, x, y)	name##_VRBT_REINSERT(x, y)
 
 #define VRBT_FOREACH(x, name, head)					\
-	for ((x) = VRBT_MIN(name, head);				\
+	for ((x) = VRBT_MIN(name, head);					\
 	     (x) != NULL;						\
 	     (x) = name##_VRBT_NEXT(x))
 
@@ -746,12 +811,12 @@ name##_VRBT_MINMAX(const struct name *head, int val)			\
 	     (x) = (y))
 
 #define VRBT_FOREACH_SAFE(x, name, head, y)				\
-	for ((x) = VRBT_MIN(name, head);				\
+	for ((x) = VRBT_MIN(name, head);					\
 	    ((x) != NULL) && ((y) = name##_VRBT_NEXT(x), (x) != NULL);	\
 	     (x) = (y))
 
 #define VRBT_FOREACH_REVERSE(x, name, head)				\
-	for ((x) = VRBT_MAX(name, head);				\
+	for ((x) = VRBT_MAX(name, head);					\
 	     (x) != NULL;						\
 	     (x) = name##_VRBT_PREV(x))
 
@@ -761,7 +826,7 @@ name##_VRBT_MINMAX(const struct name *head, int val)			\
 	     (x) = (y))
 
 #define VRBT_FOREACH_REVERSE_SAFE(x, name, head, y)			\
-	for ((x) = VRBT_MAX(name, head);				\
+	for ((x) = VRBT_MAX(name, head);					\
 	    ((x) != NULL) && ((y) = name##_VRBT_PREV(x), (x) != NULL);	\
 	     (x) = (y))
 
diff --git a/tools/import_vtree_from_freebsd.sh b/tools/import_vtree_from_freebsd.sh
new file mode 100644
index 000000000..ecc1a8435
--- /dev/null
+++ b/tools/import_vtree_from_freebsd.sh
@@ -0,0 +1,34 @@
+#
+# 
+
+if [ ! -f vtree.h ] ; then
+	echo "Run from include subdir"
+	exit 1
+fi
+
+if [ ! -f /usr/src/sys/sys/tree.h ] ; then
+	echo "You need a FreeBSD source tree in /usr/src"
+	exit 1
+fi
+
+git diff vtree.h | git apply -R > /dev/null 2>&1 || true
+
+GR=f6e54eb360a78856dcde930a00d9b2b3627309ab
+(cd /usr/src/ && git show $GR:sys/sys/tree.h ) |
+sed -E '
+485a\
+		AN(parent);			\\
+s/_SYS_TREE_H_/_VTREE_H_/
+s/SPLAY/VSPLAY/g
+s/RB_/VRBT_/g
+/(VRBT_FIND|VRBT_NFIND|VRBT_MINMAX)/{
+s/struct name [*]/const struct name */
+s/, struct type [*]/, const struct type */
+}
+/sys\/cdefs/d
+s/__unused/v_unused_/
+s/^        /	/g
+' > _t
+
+diff -uw _t vtree.h
+mv _t vtree.h

From phk at FreeBSD.org  Tue Mar 23 09:53:04 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 23 Mar 2021 09:53:04 +0000 (UTC)
Subject: [master] 074d164ea Only instantiate the necessary VRBT functions
Message-ID: <20210323095304.A1224AE679@lists.varnish-cache.org>


commit 074d164ea0129cf40456d2ac92b2853e78f2149d
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 23 09:52:24 2021 +0000

    Only instantiate the necessary VRBT functions

diff --git a/bin/varnishd/cache/cache_vrt_priv.c b/bin/varnishd/cache/cache_vrt_priv.c
index 307b3c9cb..03e684834 100644
--- a/bin/varnishd/cache/cache_vrt_priv.c
+++ b/bin/varnishd/cache/cache_vrt_priv.c
@@ -52,7 +52,11 @@ struct vrt_privs cli_task_privs[1];
 static inline int vrt_priv_dyncmp(const struct vrt_priv *,
     const struct vrt_priv *);
 
-VRBT_PROTOTYPE_STATIC(vrt_privs, vrt_priv, entry, vrt_priv_dyncmp)
+VRBT_GENERATE_INSERT_COLOR(vrt_privs, vrt_priv, entry, static)
+VRBT_GENERATE_FIND(vrt_privs, vrt_priv, entry, vrt_priv_dyncmp, static)
+VRBT_GENERATE_INSERT(vrt_privs, vrt_priv, entry, vrt_priv_dyncmp, static)
+VRBT_GENERATE_MINMAX(vrt_privs, vrt_priv, entry, static)
+VRBT_GENERATE_NEXT(vrt_privs, vrt_priv, entry, static)
 
 /*--------------------------------------------------------------------
  */
@@ -113,8 +117,6 @@ vrt_priv_dyncmp(const struct vrt_priv *vp1, const struct vrt_priv *vp2)
 	return (0);
 }
 
-VRBT_GENERATE_STATIC(vrt_privs, vrt_priv, entry, vrt_priv_dyncmp)
-
 static struct vmod_priv *
 vrt_priv_dynamic_get(const struct vrt_privs *privs, uintptr_t vmod_id)
 {
diff --git a/bin/varnishtop/varnishtop.c b/bin/varnishtop/varnishtop.c
index 8d8d572e8..d8e2b6183 100644
--- a/bin/varnishtop/varnishtop.c
+++ b/bin/varnishtop/varnishtop.c
@@ -110,10 +110,18 @@ cmp_order(const struct top *a, const struct top *b)
 	return (cmp_key(a, b));
 }
 
-VRBT_PROTOTYPE_STATIC(t_order, top, e_order, cmp_order)
-VRBT_GENERATE_STATIC(t_order, top, e_order, cmp_order)
-VRBT_PROTOTYPE_STATIC(t_key, top, e_key, cmp_key)
-VRBT_GENERATE_STATIC(t_key, top, e_key, cmp_key)
+VRBT_GENERATE_INSERT_COLOR(t_order, top, e_order, static)
+VRBT_GENERATE_INSERT(t_order, top, e_order, cmp_order, static)
+VRBT_GENERATE_REMOVE_COLOR(t_order, top, e_order, static)
+VRBT_GENERATE_MINMAX(t_order, top, e_order, static)
+VRBT_GENERATE_NEXT(t_order, top, e_order, static)
+VRBT_GENERATE_REMOVE(t_order, top, e_order, static)
+
+VRBT_GENERATE_INSERT_COLOR(t_key, top, e_key, static)
+VRBT_GENERATE_REMOVE_COLOR(t_key, top, e_key, static)
+VRBT_GENERATE_INSERT(t_key, top, e_key, cmp_key, static)
+VRBT_GENERATE_REMOVE(t_key, top, e_key, static)
+VRBT_GENERATE_FIND(t_key, top, e_key, cmp_key, static)
 
 static int v_matchproto_(VSLQ_dispatch_f)
 accumulate(struct VSL_data *vsl, struct VSL_transaction * const pt[],
diff --git a/lib/libvarnishapi/vsl_dispatch.c b/lib/libvarnishapi/vsl_dispatch.c
index 47b100ea4..e68fa4e29 100644
--- a/lib/libvarnishapi/vsl_dispatch.c
+++ b/lib/libvarnishapi/vsl_dispatch.c
@@ -228,8 +228,11 @@ vtx_keycmp(const struct vtx_key *a, const struct vtx_key *b)
 	return (0);
 }
 
-VRBT_PROTOTYPE_STATIC(vtx_tree, vtx_key, entry, vtx_keycmp)
-VRBT_GENERATE_STATIC(vtx_tree, vtx_key, entry, vtx_keycmp)
+VRBT_GENERATE_REMOVE_COLOR(vtx_tree, vtx_key, entry, static)
+VRBT_GENERATE_REMOVE(vtx_tree, vtx_key, entry, static)
+VRBT_GENERATE_INSERT_COLOR(vtx_tree, vtx_key, entry, static)
+VRBT_GENERATE_INSERT(vtx_tree, vtx_key, entry, vtx_keycmp, static)
+VRBT_GENERATE_FIND(vtx_tree, vtx_key, entry, vtx_keycmp, static)
 
 static enum vsl_status v_matchproto_(vslc_next_f)
 vslc_raw_next(const struct VSL_cursor *cursor)

From phk at FreeBSD.org  Tue Mar 23 10:16:05 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 23 Mar 2021 10:16:05 +0000 (UTC)
Subject: [master] 7b64147e9 Add debug.sweep_acl() for ACL testing
Message-ID: <20210323101605.4CD6DAF405@lists.varnish-cache.org>


commit 7b64147e9cdb77361f3b53a58f5a30fdd3813779
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 23 10:11:35 2021 +0000

    Add debug.sweep_acl() for ACL testing

diff --git a/vmod/automake_boilerplate_debug.am b/vmod/automake_boilerplate_debug.am
index 88f89a403..5617c04c9 100644
--- a/vmod/automake_boilerplate_debug.am
+++ b/vmod/automake_boilerplate_debug.am
@@ -4,6 +4,7 @@ vmod_LTLIBRARIES += libvmod_debug.la
 
 libvmod_debug_la_SOURCES = \
 	vmod_debug.c \
+	vmod_debug_acl.c \
 	vmod_debug_dyn.c \
 	vmod_debug_obj.c
 
diff --git a/vmod/vmod_debug.c b/vmod/vmod_debug.c
index 8f123c276..a203af31b 100644
--- a/vmod/vmod_debug.c
+++ b/vmod/vmod_debug.c
@@ -665,17 +665,6 @@ xyzzy_vcl_discard_delay(VRT_CTX, struct vmod_priv *priv, VCL_DURATION delay)
 	priv_vcl->vcl_discard_delay = delay;
 }
 
-VCL_BOOL v_matchproto_(td_debug_match_acl)
-xyzzy_match_acl(VRT_CTX, VCL_ACL acl, VCL_IP ip)
-{
-
-	CHECK_OBJ_ORNULL(ctx, VRT_CTX_MAGIC);
-	AN(acl);
-	assert(VSA_Sane(ip));
-
-	return (VRT_acl_match(ctx, acl, ip));
-}
-
 VCL_VOID v_matchproto_(td_debug_test_probe)
 xyzzy_test_probe(VRT_CTX, VCL_PROBE probe, VCL_PROBE same)
 {
diff --git a/vmod/vmod_debug.vcc b/vmod/vmod_debug.vcc
index c15079630..cdafcc4d3 100644
--- a/vmod/vmod_debug.vcc
+++ b/vmod/vmod_debug.vcc
@@ -178,6 +178,22 @@ $Function BOOL match_acl(ACL acl, IP ip)
 
 Perform an IP match against a named ACL.
 
+$Function BLOB sweep_acl(ACL acl, IP ip0, IP ip1, INT step = 1)
+
+Sweep `acl` through IP#s `ip0` ... `ip1` and return a hash-signature
+of the results.
+
+VSL_Debug lines will be emitted for every 64 addresses (you probably want:
+`-p vsl_mask=+Debug,-VCL_acl`
+
+Only the lower 64 bits of IPv4 addresses are stepped.
+
+$Function DURATION time_acl(ACL acl, IP ip0, IP ip1, INT step = 1, INT rounds = 1000)
+
+Time `rounds` sweeps from `ip0` to `ip1` through `acl`.
+
+Consider: `-p vsl_mask=-VCL_acl`
+
 $Function VOID test_probe(PROBE probe, PROBE same = 0)
 
 Only here to make sure probe definitions are passed properly.
diff --git a/vmod/vmod_debug_acl.c b/vmod/vmod_debug_acl.c
new file mode 100644
index 000000000..cb5fd3424
--- /dev/null
+++ b/vmod/vmod_debug_acl.c
@@ -0,0 +1,250 @@
+/*-
+ * Copyright (c) 2012-2019 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at FreeBSD.org>
+ *
+ * SPDX-License-Identifier: BSD-2-Clause
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
+#include "config.h"
+
+#include <stdlib.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/socket.h>
+#include <unistd.h>
+
+// #include "vdef.h"
+//#include "vas.h"
+#include "cache/cache.h"
+#include "vend.h"
+#include "vsa.h"
+#include "vsb.h"
+#include "vsha256.h"
+#include "vtcp.h"
+#include "vtim.h"
+#include "vcc_debug_if.h"
+
+VCL_BOOL v_matchproto_(td_debug_match_acl)
+xyzzy_match_acl(VRT_CTX, VCL_ACL acl, VCL_IP ip)
+{
+
+	CHECK_OBJ_ORNULL(ctx, VRT_CTX_MAGIC);
+	AN(acl);
+	assert(VSA_Sane(ip));
+
+	return (VRT_acl_match(ctx, acl, ip));
+}
+
+/*
+ * The code below is more intimate with VSA than anything is supposed to.
+ */
+
+struct acl_sweep {
+	int			family;
+	const uint8_t		*ip0_p;
+	const uint8_t		*ip1_p;
+	struct suckaddr		*probe;
+	uint8_t			*probe_p;
+	VCL_INT			step;
+	uint64_t		reset;
+	uint64_t		this;
+	uint64_t		count;
+};
+
+static void
+reset_sweep(struct acl_sweep *asw)
+{
+	asw->this = asw->reset;
+}
+
+static int
+setup_sweep(VRT_CTX, struct acl_sweep *asw, VCL_IP ip0, VCL_IP ip1,
+    VCL_INT step)
+{
+	int fam0, fam1;
+	const uint8_t *ptr;
+
+	AN(asw);
+	memset(asw, 0, sizeof *asw);
+
+	AN(ip0);
+	AN(ip1);
+	fam0 = VSA_GetPtr(ip0, &asw->ip0_p);
+	fam1 = VSA_GetPtr(ip1, &asw->ip1_p);
+	if (fam0 != fam1) {
+		VRT_fail(ctx, "IPs have different families (0x%x vs 0x%x)",
+		    fam0, fam1);
+		return (-1);
+	}
+
+	asw->family = fam0;
+	if (asw->family == PF_INET) {
+		if (memcmp(asw->ip0_p, asw->ip1_p, 4) > 0) {
+			VRT_fail(ctx, "Sweep: ipv4.end < ipv4.start");
+			return (-1);
+		}
+		asw->reset = vbe32dec(asw->ip0_p);
+	} else {
+		if (memcmp(asw->ip0_p, asw->ip1_p, 16) > 0) {
+			VRT_fail(ctx, "Sweep: ipv6.end < ipv6.start");
+			return (-1);
+		}
+		asw->reset = vbe64dec(asw->ip0_p + 8);
+	}
+	asw->this = asw->reset;
+
+	asw->probe = VSA_Clone(ip0);
+	(void)VSA_GetPtr(asw->probe, &ptr);
+	asw->probe_p = TRUST_ME(ptr);
+
+	asw->step = step;
+
+	return (0);
+}
+
+static void
+cleanup_sweep(struct acl_sweep *asw)
+{
+	free(asw->probe);
+	memset(asw, 0, sizeof *asw);
+}
+
+static int
+step_sweep(struct acl_sweep *asw)
+{
+
+	AN(asw);
+	asw->count++;
+	asw->this += asw->step;
+	if (asw->family == PF_INET) {
+		vbe32enc(asw->probe_p, asw->this);
+		return (memcmp(asw->probe_p, asw->ip1_p, 4));
+	} else {
+		vbe64enc(asw->probe_p + 8, asw->this);
+		return (memcmp(asw->probe_p, asw->ip1_p, 16));
+	}
+}
+
+
+VCL_BLOB
+xyzzy_sweep_acl(VRT_CTX, VCL_ACL acl, VCL_IP ip0, VCL_IP ip1, VCL_INT step)
+{
+	struct acl_sweep asw[1];
+	int i, j;
+	struct vsb *vsb;
+	char abuf[VTCP_ADDRBUFSIZE];
+	char pbuf[VTCP_PORTBUFSIZE];
+	unsigned char digest[VSHA256_DIGEST_LENGTH];
+	struct VSHA256Context vsha[1];
+	struct vrt_blob *b;
+	ssize_t sz;
+
+	vsb = VSB_new_auto();
+	AN(vsb);
+
+	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
+	AN(acl);
+	AN(ip0);
+	AN(ip1);
+	assert(step > 0);
+	if (setup_sweep(ctx, asw, ip0, ip1, step))
+		return(NULL);
+	VSHA256_Init(vsha);
+	for (j = 0; ; j++) {
+		if ((j & 0x3f) == 0x00) {
+			VTCP_name(asw->probe, abuf, sizeof abuf,
+			    pbuf, sizeof pbuf);
+			VSB_printf(vsb, "Sweep: %-15s", abuf);
+		}
+		i = VRT_acl_match(ctx, acl, asw->probe);
+		assert(0 <= i && i <= 1);
+		VSB_putc(vsb, "-X"[i]);
+		if ((j & 0x3f) == 0x3f) {
+			AZ(VSB_finish(vsb));
+			VSLb(ctx->vsl, SLT_Debug, "%s", VSB_data(vsb));
+			sz =VSB_len(vsb);
+			assert (sz > 0);
+			VSHA256_Update(vsha, VSB_data(vsb), sz);
+			VSB_clear(vsb);
+		}
+		if (step_sweep(asw) > 0)
+			break;
+	}
+	if (VSB_len(vsb)) {
+		AZ(VSB_finish(vsb));
+		VSLb(ctx->vsl, SLT_Debug, "%s", VSB_data(vsb));
+		sz =VSB_len(vsb);
+		assert (sz > 0);
+		VSHA256_Update(vsha, VSB_data(vsb), sz);
+		VSB_clear(vsb);
+	}
+	VSB_destroy(&vsb);
+
+	VSHA256_Final(digest, vsha);
+	b = WS_Alloc(ctx->ws, sizeof *b + sizeof digest);
+	if (b != NULL) {
+		memcpy(b + 1, digest, sizeof digest);
+		b->blob = b + 1;
+		b->len = sizeof digest;
+	}
+	cleanup_sweep(asw);
+	return (b);
+}
+
+VCL_DURATION
+xyzzy_time_acl(VRT_CTX, VCL_ACL acl, VCL_IP ip0, VCL_IP ip1,
+    VCL_INT step, VCL_INT turnus)
+{
+	struct acl_sweep asw[1];
+	vtim_mono t0, t1;
+	VCL_INT cnt;
+
+	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
+	AN(acl);
+	AN(ip0);
+	AN(ip1);
+	assert(step > 0);
+	assert(turnus > 0);
+
+	if (setup_sweep(ctx, asw, ip0, ip1, step))
+		return(-1);
+	do {
+		(void)VRT_acl_match(ctx, acl, asw->probe);
+	} while (step_sweep(asw) <= 0);
+	asw->count = 0;
+	t0 = VTIM_mono();
+	for (cnt = 0; cnt < turnus; cnt++) {
+		reset_sweep(asw);
+		do {
+			(void)VRT_acl_match(ctx, acl, asw->probe);
+		} while (step_sweep(asw) <= 0);
+	}
+	t1 = VTIM_mono();
+	VSLb(ctx->vsl, SLT_Debug,
+	    "Timed ACL: %.9f -> %.9f = %.9f %.9f/round, %.9f/IP %jd IPs",
+	    t0, t1, t1 - t0, (t1-t0) / turnus, (t1-t0) / asw->count, asw->count);
+	return ((t1 - t0) / asw->count);
+}

From phk at FreeBSD.org  Tue Mar 23 10:18:04 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 23 Mar 2021 10:18:04 +0000 (UTC)
Subject: [master] 333a1a0c4 Eliminate an FreeBSD kernel-ism
Message-ID: <20210323101804.4E764AF6B3@lists.varnish-cache.org>


commit 333a1a0c4a29bd9833faf4cf8508dd446a126b12
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 23 10:17:34 2021 +0000

    Eliminate an FreeBSD kernel-ism

diff --git a/include/vtree.h b/include/vtree.h
index 298a9f72d..a0ff6ff53 100644
--- a/include/vtree.h
+++ b/include/vtree.h
@@ -333,10 +333,10 @@ struct {								\
  * that the left or right child of the tree node is "red".
  */
 #define VRBT_UP(elm, field)		(elm)->field.rbe_parent
-#define VRBT_BITS(elm, field)		(*(__uintptr_t *)&VRBT_UP(elm, field))
-#define VRBT_RED_L			((__uintptr_t)1)
-#define VRBT_RED_R			((__uintptr_t)2)
-#define VRBT_RED_MASK			((__uintptr_t)3)
+#define VRBT_BITS(elm, field)		(*(uintptr_t *)&VRBT_UP(elm, field))
+#define VRBT_RED_L			((uintptr_t)1)
+#define VRBT_RED_R			((uintptr_t)2)
+#define VRBT_RED_MASK			((uintptr_t)3)
 #define VRBT_FLIP_LEFT(elm, field)	(VRBT_BITS(elm, field) ^= VRBT_RED_L)
 #define VRBT_FLIP_RIGHT(elm, field)	(VRBT_BITS(elm, field) ^= VRBT_RED_R)
 #define VRBT_RED_LEFT(elm, field)		((VRBT_BITS(elm, field) & VRBT_RED_L) != 0)
@@ -348,7 +348,7 @@ struct {								\
 
 #define VRBT_SET_PARENT(dst, src, field) do {				\
 	VRBT_BITS(dst, field) &= VRBT_RED_MASK;				\
-	VRBT_BITS(dst, field) |= (__uintptr_t)src;			\
+	VRBT_BITS(dst, field) |= (uintptr_t)src;			\
 } while (/*CONSTCOND*/ 0)
 
 #define VRBT_SET(elm, parent, field) do {					\
diff --git a/tools/import_vtree_from_freebsd.sh b/tools/import_vtree_from_freebsd.sh
index ecc1a8435..dd8fd69c1 100644
--- a/tools/import_vtree_from_freebsd.sh
+++ b/tools/import_vtree_from_freebsd.sh
@@ -19,6 +19,7 @@ sed -E '
 485a\
 		AN(parent);			\\
 s/_SYS_TREE_H_/_VTREE_H_/
+s/__uintptr_t/uintptr_t/g
 s/SPLAY/VSPLAY/g
 s/RB_/VRBT_/g
 /(VRBT_FIND|VRBT_NFIND|VRBT_MINMAX)/{

From phk at FreeBSD.org  Tue Mar 23 11:45:05 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 23 Mar 2021 11:45:05 +0000 (UTC)
Subject: [master] 784756e02 Neuter the VRBT_AUGMENT facility if not used.
Message-ID: <20210323114506.0A662B1E4B@lists.varnish-cache.org>


commit 784756e02f87899a5612f6671fb9a5c3e00af02f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 23 11:43:34 2021 +0000

    Neuter the VRBT_AUGMENT facility if not used.

diff --git a/include/vtree.h b/include/vtree.h
index a0ff6ff53..c652c48d8 100644
--- a/include/vtree.h
+++ b/include/vtree.h
@@ -366,7 +366,16 @@ struct {								\
  * from the bottom up to the root, to update augmented node data.
  */
 #ifndef VRBT_AUGMENT
-#define VRBT_AUGMENT(x)	break
+#define VRBT_AUGMENT(x) do {} while (0)
+#define VRBT_AUGMENT_UP(x) do {} while (0)
+#else
+#define VRBT_AUGMENT_UP(x)						\
+	do {								\
+		while((x) != NULL) {					\
+			VRBT_AUGMENT(x);				\
+			x = VRBT_PARENT(x, field);			\
+		}							\
+	} while (0)
 #endif
 
 #define VRBT_SWAP_CHILD(head, out, in, field) do {			\
@@ -627,10 +636,7 @@ name##_VRBT_REMOVE(struct name *head, struct type *elm)			\
 		VRBT_SET_PARENT(child, parent, field);			\
 	if (parent != NULL)						\
 		name##_VRBT_REMOVE_COLOR(head, parent, child);		\
-	while (parent != NULL) {					\
-		VRBT_AUGMENT(parent);					\
-		parent = VRBT_PARENT(parent, field);			\
-	}								\
+	VRBT_AUGMENT_UP(parent);					\
 	return (old);							\
 }
 
@@ -661,10 +667,7 @@ name##_VRBT_INSERT(struct name *head, struct type *elm)			\
 	else								\
 		VRBT_RIGHT(parent, field) = elm;				\
 	name##_VRBT_INSERT_COLOR(head, elm);				\
-	while (elm != NULL) {						\
-		VRBT_AUGMENT(elm);					\
-		elm = VRBT_PARENT(elm, field);				\
-	}								\
+	VRBT_AUGMENT_UP(elm);						\
 	return (NULL);							\
 }
 

From phk at FreeBSD.org  Tue Mar 23 12:42:04 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 23 Mar 2021 12:42:04 +0000 (UTC)
Subject: [master] 63cdbd33e Detect varnishd startup failure,
 for instance param trouble, right away.
Message-ID: <20210323124204.F410B5B82@lists.varnish-cache.org>


commit 63cdbd33e08bcfa5ce2e071bcd7848783efdcb26
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 23 12:40:58 2021 +0000

    Detect varnishd startup failure, for instance param trouble, right away.

diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c
index b051b8e08..3612aaa9c 100644
--- a/bin/varnishtest/vtc_varnish.c
+++ b/bin/varnishtest/vtc_varnish.c
@@ -327,7 +327,6 @@ varnish_new(const char *name)
 	v->cli_fd = -1;
 	VTAILQ_INSERT_TAIL(&varnishes, v, list);
 
-
 	return (v);
 }
 
@@ -384,19 +383,19 @@ static void
 varnish_launch(struct varnish *v)
 {
 	struct vsb *vsb, *vsb1;
-	int i, nfd;
+	int i, nfd, asock;
 	char abuf[128], pbuf[128];
-	struct pollfd fd[2];
+	struct pollfd fd[3];
 	enum VCLI_status_e u;
 	const char *err;
 	char *r = NULL;
 
 	/* Create listener socket */
-	v->cli_fd = VTCP_listen_on(default_listen_addr, NULL, 1, &err);
+	asock = VTCP_listen_on(default_listen_addr, NULL, 1, &err);
 	if (err != NULL)
 		vtc_fatal(v->vl, "Create CLI listen socket failed: %s", err);
-	assert(v->cli_fd > 0);
-	VTCP_myname(v->cli_fd, abuf, sizeof abuf, pbuf, sizeof pbuf);
+	assert(asock > 0);
+	VTCP_myname(asock, abuf, sizeof abuf, pbuf, sizeof pbuf);
 
 	AZ(VSB_finish(v->args));
 	vtc_log(v->vl, 2, "Launch");
@@ -465,26 +464,26 @@ varnish_launch(struct varnish *v)
 
 	/* Wait for the varnish to call home */
 	memset(fd, 0, sizeof fd);
-	fd[0].fd = v->cli_fd;
+	fd[0].fd = asock;
 	fd[0].events = POLLIN;
 	fd[1].fd = v->fds[1];
 	fd[1].events = POLLIN;
+	fd[2].fd = v->fds[2];
+	fd[2].events = POLLIN;
 	i = poll(fd, 2, vtc_maxdur * 1000 / 3);
-	vtc_log(v->vl, 4, "CLIPOLL %d 0x%x 0x%x",
-	    i, fd[0].revents, fd[1].revents);
+	vtc_log(v->vl, 4, "CLIPOLL %d 0x%x 0x%x 0x%x",
+	    i, fd[0].revents, fd[1].revents, fd[2].revents);
 	if (i == 0)
 		vtc_fatal(v->vl, "FAIL timeout waiting for CLI connection");
 	if (fd[1].revents & POLLHUP)
 		vtc_fatal(v->vl, "FAIL debug pipe closed");
 	if (!(fd[0].revents & POLLIN))
 		vtc_fatal(v->vl, "FAIL CLI connection wait failure");
-	nfd = accept(v->cli_fd, NULL, NULL);
-	if (nfd < 0) {
-		closefd(&v->cli_fd);
+	nfd = accept(asock, NULL, NULL);
+	closefd(&asock);
+	if (nfd < 0)
 		vtc_fatal(v->vl, "FAIL no CLI connection accepted");
-	}
 
-	closefd(&v->cli_fd);
 	v->cli_fd = nfd;
 
 	vtc_log(v->vl, 3, "CLI connection fd = %d", v->cli_fd);

From phk at FreeBSD.org  Wed Mar 24 09:03:07 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 24 Mar 2021 09:03:07 +0000 (UTC)
Subject: [master] ac0b20c48 Fix a couple of old-style functions
Message-ID: <20210324090307.2C63DAEBDD@lists.varnish-cache.org>


commit ac0b20c4821990c4bfdf83fa1d53cc94f81ca922
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 23 23:17:09 2021 +0000

    Fix a couple of old-style functions

diff --git a/bin/varnishd/storage/stevedore.c b/bin/varnishd/storage/stevedore.c
index 93d3f78d4..9edf85774 100644
--- a/bin/varnishd/storage/stevedore.c
+++ b/bin/varnishd/storage/stevedore.c
@@ -50,7 +50,7 @@ static pthread_mutex_t stv_mtx;
  */
 
 const struct stevedore *
-STV_next()
+STV_next(void)
 {
 	static struct stevedore *stv;
 	struct stevedore *r;
diff --git a/lib/libvarnish/vsa.c b/lib/libvarnish/vsa.c
index 57287b60d..66213be76 100644
--- a/lib/libvarnish/vsa.c
+++ b/lib/libvarnish/vsa.c
@@ -188,7 +188,7 @@ static struct suckaddr bogo_ip6_vsa;
 const struct suckaddr *bogo_ip6 = &bogo_ip6_vsa;
 
 void
-VSA_Init()
+VSA_Init(void)
 {
 	AN(VSA_BuildFAP(&bogo_ip_vsa, PF_INET, NULL, 0, NULL, 0));
 	AN(VSA_BuildFAP(&bogo_ip6_vsa, PF_INET6, NULL, 0, NULL, 0));

From phk at FreeBSD.org  Wed Mar 24 09:03:07 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 24 Mar 2021 09:03:07 +0000 (UTC)
Subject: [master] d088c084a Appease some compiler warnings we use (yet).
Message-ID: <20210324090307.462C5AEBE1@lists.varnish-cache.org>


commit d088c084a66f5b89d4437a7c6b2dbde6fcb1c03e
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 24 09:01:42 2021 +0000

    Appease some compiler warnings we use (yet).

diff --git a/bin/varnishd/fuzzers/esi_parse_fuzzer.c b/bin/varnishd/fuzzers/esi_parse_fuzzer.c
index b971280e2..a0ddaad11 100644
--- a/bin/varnishd/fuzzers/esi_parse_fuzzer.c
+++ b/bin/varnishd/fuzzers/esi_parse_fuzzer.c
@@ -46,7 +46,9 @@
 
 int LLVMFuzzerTestOneInput(const uint8_t *, size_t);
 
+extern struct VSC_main *VSC_C_main;
 struct VSC_main *VSC_C_main;
+extern struct params *cache_param;
 struct params *cache_param;
 
 void
diff --git a/bin/varnishtest/tests/c00057.vtc b/bin/varnishtest/tests/c00057.vtc
index 280cc3711..a4d1ae1ce 100644
--- a/bin/varnishtest/tests/c00057.vtc
+++ b/bin/varnishtest/tests/c00057.vtc
@@ -23,7 +23,7 @@ static void _accessor(volatile char *p) {
      p[0] = 'V'; p[1] = '\0';
      fprintf(stderr, "%p %s\n", p, p);
 }
-void (*accessor)(volatile char *p) = _accessor;
+static void (*accessor)(volatile char *p) = _accessor;
 
 }C
 	sub vcl_recv { C{
diff --git a/lib/libvarnish/vbh.c b/lib/libvarnish/vbh.c
index 073de5e0a..511e9e7de 100644
--- a/lib/libvarnish/vbh.c
+++ b/lib/libvarnish/vbh.c
@@ -495,7 +495,7 @@ struct foo {
 #define N 131101	/* Number of items */
 #define R -1		/* Random modulus */
 
-struct foo *ff[N];
+static struct foo *ff[N];
 
 static int v_matchproto_(vbh_cmp_t)
 cmp(void *priv, const void *a, const void *b)
diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index 213dbc37b..e3720ccf9 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -530,11 +530,15 @@ vcc_acl_emit(struct vcc *tl, const char *name, const char *rname)
 	Fh(tl, 0, "\treturn (0);\n}\n");
 
 	/* Emit the struct that will be referenced */
-	Fh(tl, 0, "\nconst struct vrt_acl %s[] = {{\n", rname);
+	Fh(tl, 0, "\nstatic const struct vrt_acl %s[] = {{\n", rname);
 	Fh(tl, 0, "\t.magic = VRT_ACL_MAGIC,\n");
 	Fh(tl, 0, "\t.match = &%s,\n", VSB_data(func));
 	Fh(tl, 0, "\t.name = \"%s\",\n", name);
 	Fh(tl, 0, "}};\n\n");
+	if (!tl->err_unref) {
+		AN(ifp);
+		VSB_printf(ifp->ini, "\t(void)%s;\n", rname);
+	}
 	VSB_destroy(&func);
 }
 
diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c
index e52c1da4b..906567977 100644
--- a/lib/libvcc/vcc_compile.c
+++ b/lib/libvcc/vcc_compile.c
@@ -199,8 +199,9 @@ vcc_EmitProc(struct vcc *tl, struct proc *p)
 	nsub = tl->nsub++;
 
 	Fh(tl, 1, "vcl_func_f %s;\n", VSB_data(p->cname));
-	Fh(tl, 1, "const struct vcl_sub sub_%s[1] = {{\n",
-	   VSB_data(p->cname));
+	Fh(tl, 1, "extern const struct vcl_sub sub_%s[1];\n",
+	    VSB_data(p->cname));
+	Fh(tl, 1, "const struct vcl_sub sub_%s[1] = {{\n", VSB_data(p->cname));
 	Fh(tl, 1, "\t.magic\t\t= VCL_SUB_MAGIC,\n");
 	Fh(tl, 1, "\t.methods\t= 0x%x,\n", p->okmask);
 	Fh(tl, 1, "\t.name\t\t= \"%.*s\",\n", PF(p->name));
diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h
index d261302b1..5ed4737bc 100644
--- a/lib/libvcc/vcc_compile.h
+++ b/lib/libvcc/vcc_compile.h
@@ -461,4 +461,4 @@ const char *vcc_MarkPriv(struct vcc *, struct procprivhead *,
 #define SkipToken(a, b) \
     do { vcc__Expect(a, b, __LINE__); ERRCHK(a); vcc_NextToken(a); } while (0)
 
-#define ACL_SYMBOL_PREFIX "vrt_acl_named"
+#define ACL_SYMBOL_PREFIX "vrt_acl"
diff --git a/lib/libvcc/vcc_token.c b/lib/libvcc/vcc_token.c
index 167b6ed28..9edff6226 100644
--- a/lib/libvcc/vcc_token.c
+++ b/lib/libvcc/vcc_token.c
@@ -541,7 +541,8 @@ vcc_Lexer(struct vcc *tl, const struct source *sp, int eoi)
 			AZ(VSB_finish(vsb));
 
 			bprintf(namebuf, "blob_%u", tl->unique++);
-			Fh(tl, 0, "\nconst unsigned char %s_data[%zd] = {\n",
+			Fh(tl, 0,
+			    "\nstatic const unsigned char %s_data[%zd] = {\n",
 			    namebuf, VSB_len(vsb));
 			for (u = 0; u < VSB_len(vsb); u++) {
 				Fh(tl, 0, "\t0x%02x,", VSB_data(vsb)[u] & 0xff);
@@ -551,7 +552,8 @@ vcc_Lexer(struct vcc *tl, const struct source *sp, int eoi)
 			if ((u & 7) != 7)
 				Fh(tl, 0, "\n");
 			Fh(tl, 0, "};\n");
-			Fh(tl, 0, "\nconst struct vrt_blob %s[1] = {{\n",
+			Fh(tl, 0,
+			    "\nstatic const struct vrt_blob %s[1] = {{\n",
 			    namebuf);
 			Fh(tl, 0, "\t.len =\t%zd,\n", VSB_len(vsb));
 			Fh(tl, 0, "\t.blob =\t%s_data,\n", namebuf);
diff --git a/lib/libvcc/vcc_xref.c b/lib/libvcc/vcc_xref.c
index 105f45698..e3351ec48 100644
--- a/lib/libvcc/vcc_xref.c
+++ b/lib/libvcc/vcc_xref.c
@@ -411,7 +411,7 @@ vcc_instance_info(struct vcc *tl, const struct symbol *sym)
 void
 VCC_InstanceInfo(struct vcc *tl)
 {
-	Fc(tl, 0, "\nconst struct vpi_ii VGC_instance_info[] = {\n");
+	Fc(tl, 0, "\nstatic const struct vpi_ii VGC_instance_info[] = {\n");
 	VCC_WalkSymbols(tl, vcc_instance_info, SYM_MAIN, SYM_INSTANCE);
 	Fc(tl, 0, "\t{ .p = NULL, .name = \"\" }\n");
 	Fc(tl, 0, "};\n");
diff --git a/vmod/vmod_debug.c b/vmod/vmod_debug.c
index a203af31b..8723481eb 100644
--- a/vmod/vmod_debug.c
+++ b/vmod/vmod_debug.c
@@ -1052,12 +1052,14 @@ xyzzy_get_ip(VRT_CTX)
 
 //lint -save -e9075 external symbol '...' defined without a prior declaration
 
+extern const struct vmod_data Vmod_wrong0_Data;
 const struct vmod_data Vmod_wrong0_Data = {
 	.vrt_major =    0,
 	.vrt_minor =    0,
 };
 
 //lint -save -e835 A zero has been given as left argument to operatorp'+'
+extern const struct vmod_data Vmod_wrong1_Data;
 const struct vmod_data Vmod_wrong1_Data = {
 	.vrt_major =    VRT_MAJOR_VERSION,
 	.vrt_minor =    VRT_MINOR_VERSION + 1,
@@ -1068,6 +1070,7 @@ static const struct foo {
 	int bar;
 } foo_struct[1];
 
+extern const struct vmod_data Vmod_wrong2_Data;
 const struct vmod_data Vmod_wrong2_Data = {
 	.vrt_major =    VRT_MAJOR_VERSION,
 	.vrt_minor =    VRT_MINOR_VERSION,
@@ -1078,6 +1081,7 @@ const struct vmod_data Vmod_wrong2_Data = {
 	.proto =	"blablabla",
 };
 
+extern const struct vmod_data Vmod_wrong3_Data;
 const struct vmod_data Vmod_wrong3_Data = {
 	.vrt_major =    VRT_MAJOR_VERSION,
 	.vrt_minor =    VRT_MINOR_VERSION,

From phk at FreeBSD.org  Wed Mar 24 10:06:05 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 24 Mar 2021 10:06:05 +0000 (UTC)
Subject: [master] b037fa8b2 Constify
Message-ID: <20210324100605.39660B0B6D@lists.varnish-cache.org>


commit b037fa8b20896c67b0e90a930cc22996d5e266cd
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 24 09:53:38 2021 +0000

    Constify

diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index e3720ccf9..9ef9a19d3 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -68,9 +68,9 @@ struct acl_e {
 	} while (0)
 
 static int
-vcl_acl_cmp(struct acl_e *ae1, struct acl_e *ae2)
+vcl_acl_cmp(const struct acl_e *ae1, const struct acl_e *ae2)
 {
-	unsigned char *p1, *p2;
+	const unsigned char *p1, *p2;
 	unsigned m;
 
 	p1 = ae1->data;
@@ -428,7 +428,7 @@ vcc_acl_emit(struct vcc *tl, const char *name, const char *rname)
 	int depth, l, m, i;
 	unsigned at[ACL_MAXADDR];
 	struct token *t;
-	struct inifin *ifp;
+	struct inifin *ifp = NULL;
 	struct vsb *func;
 
 	func = VSB_new_auto();

From phk at FreeBSD.org  Wed Mar 24 10:06:05 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 24 Mar 2021 10:06:05 +0000 (UTC)
Subject: [master] c6d9b9781 Move emitting of tokens for VSL record into a
 subroutine
Message-ID: <20210324100605.4EC32B0B70@lists.varnish-cache.org>


commit c6d9b97816d7fcf19f3bc1c44d853d5336b7dceb
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 24 10:02:06 2021 +0000

    Move emitting of tokens for VSL record into a subroutine

diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index 9ef9a19d3..d3f7efbfc 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -417,6 +417,35 @@ vcc_acl_entry(struct vcc *tl)
 	ERRCHK(tl);
 }
 
+/*********************************************************************
+ * Emit the tokens making up an entry as C-strings
+ */
+
+static void
+vcc_acl_emit_tokens(const struct vcc *tl, const struct acl_e *ae)
+{
+	struct token *t;
+	const char *sep = "";
+
+	t = ae->t_addr;
+	do {
+		if (t->tok == CSTR) {
+			Fh(tl, 0, "%s\"\\\"\" ", sep);
+			EncToken(tl->fh, t);
+			Fh(tl, 0, " \"\\\"\"");
+		} else {
+			Fh(tl, 0, "%s\"%.*s\"", sep, PF(t));
+		}
+		if (t == ae->t_mask)
+			break;
+		t = VTAILQ_NEXT(t, list);
+		AN(t);
+		sep = " ";
+	} while (ae->t_mask != NULL);
+	if (ae->fixed)
+		Fh(tl, 0, "\" fixed: %s\"", ae->fixed);
+}
+
 /*********************************************************************
  * Emit a function to match the ACL we have collected
  */
@@ -427,7 +456,6 @@ vcc_acl_emit(struct vcc *tl, const char *name, const char *rname)
 	struct acl_e *ae;
 	int depth, l, m, i;
 	unsigned at[ACL_MAXADDR];
-	struct token *t;
 	struct inifin *ifp = NULL;
 	struct vsb *func;
 
@@ -500,22 +528,7 @@ vcc_acl_emit(struct vcc *tl, const char *name, const char *rname)
 
 		Fh(tl, 0, "\t%*sVPI_acl_log(ctx, \"%sMATCH %s \" ",
 		    -i, "", ae->not ? "NEG_" : "", name);
-		t = ae->t_addr;
-		do {
-			if (t->tok == CSTR) {
-				Fh(tl, 0, " \"\\\"\" ");
-				EncToken(tl->fh, t);
-				Fh(tl, 0, " \"\\\"\" ");
-			} else
-				Fh(tl, 0, " \"%.*s\"", PF(t));
-			if (t == ae->t_mask)
-				break;
-			t = VTAILQ_NEXT(t, list);
-			AN(t);
-		} while (ae->t_mask != NULL);
-		if (ae->fixed)
-			Fh(tl, 0, "\" fixed: %s\"",
-			   ae->fixed);
+		vcc_acl_emit_tokens(tl, ae);
 		Fh(tl, 0, ");\n");
 
 		Fh(tl, 0, "\t%*sreturn (%d);\n", -i, "", ae->not ? 0 : 1);

From phk at FreeBSD.org  Wed Mar 24 10:27:04 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 24 Mar 2021 10:27:04 +0000 (UTC)
Subject: [master] 596c0a4d2 Keep acl-entries in a VRBT instead of a VTAILQ.
Message-ID: <20210324102704.C3250B16D8@lists.varnish-cache.org>


commit 596c0a4d26ab347125c823a89ca86250d4e22c6c
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 24 10:26:08 2021 +0000

    Keep acl-entries in a VRBT instead of a VTAILQ.

diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index d3f7efbfc..46d6d2769 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -46,7 +46,7 @@
 #define ACL_MAXADDR	(sizeof(struct in6_addr) + 1)
 
 struct acl_e {
-	VTAILQ_ENTRY(acl_e)	list;
+	VRBT_ENTRY(acl_e)	branch;
 	unsigned char		data[ACL_MAXADDR];
 	unsigned		mask;
 	unsigned		not;
@@ -94,6 +94,12 @@ vcl_acl_cmp(const struct acl_e *ae1, const struct acl_e *ae2)
 	return (0);
 }
 
+VRBT_GENERATE_INSERT_COLOR(acl_tree, acl_e, branch, static)
+VRBT_GENERATE_FIND(acl_tree, acl_e, branch, vcl_acl_cmp, static)
+VRBT_GENERATE_INSERT(acl_tree, acl_e, branch, vcl_acl_cmp, static)
+VRBT_GENERATE_MINMAX(acl_tree, acl_e, branch, static)
+VRBT_GENERATE_NEXT(acl_tree, acl_e, branch, static)
+
 static char *
 vcc_acl_chk(struct vcc *tl, const struct acl_e *ae, const int l,
     unsigned char *p, int fam)
@@ -154,7 +160,6 @@ vcc_acl_add_entry(struct vcc *tl, const struct acl_e *ae, int l,
     unsigned char *u, int fam)
 {
 	struct acl_e *ae2, *aen;
-	int i;
 
 	if (fam == PF_INET && ae->mask > 32) {
 		VSB_printf(tl->sb,
@@ -187,40 +192,18 @@ vcc_acl_add_entry(struct vcc *tl, const struct acl_e *ae, int l,
 	assert(l + 1UL <= sizeof aen->data);
 	memcpy(aen->data + 1L, u, l);
 
-	VTAILQ_FOREACH(ae2, &tl->acl, list) {
-		i = vcl_acl_cmp(aen, ae2);
-		if (i == 0) {
-			/*
-			 * If the two rules agree, silently ignore it
-			 * XXX: is that counter intuitive ?
-			 */
-			if (aen->not == ae2->not) {
-				free(aen);
-				return;
-			}
+	ae2 = VRBT_FIND(acl_tree, &tl->acl_tree, aen);
+	if (ae2 != NULL) {
+		if (ae2->not != aen->not) {
 			VSB_cat(tl->sb, "Conflicting ACL entries:\n");
 			vcc_ErrWhere(tl, ae2->t_addr);
 			VSB_cat(tl->sb, "vs:\n");
 			vcc_ErrWhere(tl, aen->t_addr);
-			free(aen);
-			return;
-		}
-		/*
-		 * We could eliminate pointless rules here, for instance in:
-		 *	"10.1.0.1";
-		 *	"10.1";
-		 * The first rule is clearly pointless, as the second one
-		 * covers it.
-		 *
-		 * We do not do this however, because the shmlog may
-		 * be used to gather statistics.
-		 */
-		if (i < 0) {
-			VTAILQ_INSERT_BEFORE(ae2, aen, list);
-			return;
 		}
+		free(aen);
+		return;
 	}
-	VTAILQ_INSERT_TAIL(&tl->acl, aen, list);
+	VRBT_INSERT(acl_tree, &tl->acl_tree, aen);
 }
 
 static void
@@ -483,7 +466,7 @@ vcc_acl_emit(struct vcc *tl, const char *name, const char *rname)
 	}
 	depth = -1;
 	at[0] = 256;
-	VTAILQ_FOREACH(ae, &tl->acl, list) {
+	VRBT_FOREACH(ae, acl_tree, &tl->acl_tree) {
 
 		/* Find how much common prefix we have */
 		for (l = 0; l <= depth && l * 8 < (int)ae->mask - 7; l++) {
@@ -561,7 +544,7 @@ vcc_ParseAcl(struct vcc *tl)
 	struct symbol *sym;
 
 	vcc_NextToken(tl);
-	VTAILQ_INIT(&tl->acl);
+	VRBT_INIT(&tl->acl_tree);
 
 	vcc_ExpectVid(tl, "ACL");
 	ERRCHK(tl);
diff --git a/lib/libvcc/vcc_compile.h b/lib/libvcc/vcc_compile.h
index 5ed4737bc..b52c53216 100644
--- a/lib/libvcc/vcc_compile.h
+++ b/lib/libvcc/vcc_compile.h
@@ -39,6 +39,7 @@
 #include "vrt.h"
 #include "vcl.h"
 #include "vqueue.h"
+#include "vtree.h"
 #include "vsb.h"
 
 #include "vcc_token_defs.h"
@@ -233,6 +234,8 @@ struct inifin {
 
 VTAILQ_HEAD(inifinhead, inifin);
 
+VRBT_HEAD(acl_tree, acl_e);
+
 struct vcc {
 	unsigned		magic;
 #define VCC_MAGIC		0x24ad719d
@@ -271,7 +274,7 @@ struct vcc {
 	struct proc		*curproc;
 	VTAILQ_HEAD(, proc)	procs;
 
-	VTAILQ_HEAD(, acl_e)	acl;
+	struct acl_tree		acl_tree;
 
 	int			nprobe;
 

From phk at FreeBSD.org  Wed Mar 24 12:14:04 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 24 Mar 2021 12:14:04 +0000 (UTC)
Subject: [master] a79456563 Thest ACL's more comprehensively.
Message-ID: <20210324121404.7CDD05BCC@lists.varnish-cache.org>


commit a79456563999de96a65b97038692386b7b1d1fb1
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 24 12:12:41 2021 +0000

    Thest ACL's more comprehensively.

diff --git a/bin/varnishtest/tests/r01312.vtc b/bin/varnishtest/tests/r01312.vtc
index 762da43bf..0f3fc3978 100644
--- a/bin/varnishtest/tests/r01312.vtc
+++ b/bin/varnishtest/tests/r01312.vtc
@@ -1,4 +1,4 @@
-varnishtest "acl miscompile"
+varnishtest "acl functional (& historic miscompile)"
 
 server s1 {
 	rxreq
@@ -7,6 +7,7 @@ server s1 {
 
 varnish v1 -vcl+backend {
 	import std;
+	import debug;
 
 	acl foo {
 		"127.0.0.2";
@@ -16,15 +17,71 @@ varnish v1 -vcl+backend {
 		"127.0.1.2";
 		"127.0.1"/19;
 	}
+
+	acl block {
+		# Tests all boundary conditions
+		  "192.168.8.0" / 21;
+		! "192.168.16" / 21;
+		  "192.168.8.0" / 23;
+		! "192.168.14.0" / 23;
+		  "192.168.16.0" / 23;
+		! "192.168.22.0" / 23;
+		! "192.168.8.0" / 24;
+		! "192.168.11.0" / 24;
+		  "192.168.15.0" / 24;
+		! "192.168.16.0" / 24;
+		  "192.168.19.0" / 24;
+		  "192.168.23.0" / 24;
+
+		# Same pattern, but split across bytes
+		  "::0080" / 121;
+		! "::0100" / 121;
+		  "::0080" / 123;
+		! "::00e0" / 123;
+		  "::0100" / 123;
+		! "::0160" / 123;
+		! "::0080" / 124;
+		! "::00b0" / 124;
+		  "::00f0" / 124;
+		! "::0100" / 124;
+		  "::0130" / 124;
+		  "::0170" / 124;
+	}
+
 	sub vcl_deliver {
+		set resp.http.acl4 = debug.sweep_acl(
+		    block,
+		    std.ip("192.168.0.0"),
+		    std.ip("192.168.32.255"),
+		    256
+		);
+		set resp.http.acl6 = debug.sweep_acl(
+		    block,
+		    std.ip("::"),
+		    std.ip("::0200"),
+		    16
+		);
+
 		set resp.http.ACLfoo = std.ip("127.0.0.1", client.ip) ~ foo;
 		set resp.http.ACLbar = std.ip("127.0.0.1", client.ip) ~ bar;
 	}
 } -start
 
+varnish v1 -cliok "param.set vsl_mask +Debug,-VCL_acl"
+varnish v1 -cliok "param.set debug +syncvsl"
+
+logexpect l1 -v v1 -g raw {
+	expect * 1001 Debug {Sweep: 192.168.0.0    ---------XX-XX-X-X-X---X---------}
+	expect * 1001 Debug {Sweep: ::             ---------XX-XX-X-X-X---X---------}
+} -start
+
 client c1 {
 	txreq
 	rxresp
 	expect resp.http.aclfoo == true
 	expect resp.http.aclbar == true
+	expect resp.http.acl4 == ":4thASR0O18ZxnoKtc4zd8KuO25rPvwvMQyAvRfilz6o=:"
+	expect resp.http.acl6 == ":NSi+7wpvQe7XJj8DPbESjpYPGnIzvjOsA5QCyCnW3kc=:"
 } -run
+
+logexpect l1 -wait

From phk at FreeBSD.org  Wed Mar 24 12:14:04 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 24 Mar 2021 12:14:04 +0000 (UTC)
Subject: [master] 1c86cf44c This test no longer serves any purpose.
Message-ID: <20210324121404.9BE015BCF@lists.varnish-cache.org>


commit 1c86cf44ce2b157f7da1b27e2f82c8d6fe9bf713
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 24 12:13:00 2021 +0000

    This test no longer serves any purpose.

diff --git a/bin/varnishtest/tests/r00948.vtc b/bin/varnishtest/tests/r00948.vtc
deleted file mode 100644
index e0d01f2e4..000000000
--- a/bin/varnishtest/tests/r00948.vtc
+++ /dev/null
@@ -1,30 +0,0 @@
-varnishtest "anon acl numbering"
-
-
-server s1 {
-	rxreq
-	expect req.http.foo1 == "Match"
-	expect req.http.foo2 == "Match"
-	txresp -bodylen 40
-} -start
-
-varnish v1 -vcl+backend {
-
-sub vcl_recv {
-	if (client.ip == "${s1_addr}" || client.ip == "${localhost}") {
-		set req.http.foo1 = "Match";
-	}
-	if (client.ip == "${localhost}" || client.ip == "${s1_addr}") {
-		set req.http.foo2 = "Match";
-	}
-}
-
-} -start
-
-client c1 {
-	txreq
-	rxresp
-	expect resp.bodylen == 40
-} -run
-
-

From phk at FreeBSD.org  Wed Mar 24 13:11:04 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 24 Mar 2021 13:11:04 +0000 (UTC)
Subject: [master] 8a913e2be Cover vmod.debug_acl.c better
Message-ID: <20210324131104.CC1C77AEB@lists.varnish-cache.org>


commit 8a913e2bed24e8b376dd210937c1006c7d46f7f5
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Mar 24 13:09:53 2021 +0000

    Cover vmod.debug_acl.c better

diff --git a/bin/varnishtest/tests/r01312.vtc b/bin/varnishtest/tests/r01312.vtc
index 0f3fc3978..583871d84 100644
--- a/bin/varnishtest/tests/r01312.vtc
+++ b/bin/varnishtest/tests/r01312.vtc
@@ -64,6 +64,21 @@ varnish v1 -vcl+backend {
 
 		set resp.http.ACLfoo = std.ip("127.0.0.1", client.ip) ~ foo;
 		set resp.http.ACLbar = std.ip("127.0.0.1", client.ip) ~ bar;
+
+		// For coverage of vmod_debug_acl.c
+		debug.sweep_acl(
+		    block,
+		    std.ip("::"),
+		    std.ip("::02ff"),
+		    8
+		);
+		debug.time_acl(
+		    block,
+		    std.ip("::"),
+		    std.ip("::02ff"),
+		    8,
+		    1000
+		);
 	}
 } -start
 

From nils.goroll at uplex.de  Wed Mar 24 17:38:04 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 24 Mar 2021 17:38:04 +0000 (UTC)
Subject: [master] 777e03ba9 Disable strict aliasing with gcc where vtree.h is
 used
Message-ID: <20210324173804.D872D6EB8B@lists.varnish-cache.org>


commit 777e03ba9774bf635a972fa0561b50ff071908a9
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Mar 24 18:25:24 2021 +0100

    Disable strict aliasing with gcc where vtree.h is used
    
    Please replace this commit with a better solution, if you have one.
    
    e1ac59335f749b84280722425b355be344cc76e9 brought us the VRBT node
    color encoded in the lower two bits of pointers.
    
    gcc strict aliasing rules (as enabled with -O2 and higher) forbid
    aliased lvalue access as in the following macro expanded code:
    
    (*(uintptr_t *)&(parent)->entry.rbe_parent) &= ~((uintptr_t)3);
    
    Until we have a better solution, disable strict aliasing with gcc
    for targets with any source file using vtree.h.
    
    Note that we would want to limit -fno-strict-aliasing to individual
    compliation units, but automake does not offer a simple and clean way
    to achieve this:
    
    https://www.gnu.org/software/automake/manual/html_node/Per_002dObject-Flags.html

diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am
index ce7145efd..76a47ad2c 100644
--- a/bin/varnishd/Makefile.am
+++ b/bin/varnishd/Makefile.am
@@ -159,6 +159,7 @@ nobase_pkginclude_HEADERS = \
 vcldir=$(datarootdir)/$(PACKAGE)/vcl
 
 varnishd_CFLAGS = \
+	@vtree_extra_cflags@ \
 	@PCRE_CFLAGS@ \
 	@SAN_CFLAGS@ \
 	-DNOT_IN_A_VMOD \
diff --git a/bin/varnishtop/Makefile.am b/bin/varnishtop/Makefile.am
index cdb63b5f8..0427e1140 100644
--- a/bin/varnishtop/Makefile.am
+++ b/bin/varnishtop/Makefile.am
@@ -13,6 +13,7 @@ varnishtop_SOURCES = \
 
 
 varnishtop_CFLAGS = \
+	@vtree_extra_cflags@ \
 	@SAN_CFLAGS@
 
 varnishtop_LDADD = \
diff --git a/configure.ac b/configure.ac
index 59aa57b9e..9fcfb5693 100644
--- a/configure.ac
+++ b/configure.ac
@@ -291,6 +291,8 @@ CFLAGS="${save_CFLAGS}"
 if test "x$GCC" = "xyes"; then
 	libvgz_extra_cflags="${libvgz_extra_cflags} -Wno-unknown-warning-option -Wno-implicit-fallthrough"
 	AC_SUBST(libvgz_extra_cflags)
+	vtree_extra_cflags="-fno-strict-aliasing"
+	AC_SUBST(vtree_extra_cflags)
 fi
 
 SAN_CFLAGS=
diff --git a/lib/libvarnishapi/Makefile.am b/lib/libvarnishapi/Makefile.am
index 0cbff3c4d..a2a314581 100644
--- a/lib/libvarnishapi/Makefile.am
+++ b/lib/libvarnishapi/Makefile.am
@@ -37,6 +37,7 @@ endif
 
 libvarnishapi_la_CFLAGS = \
 	-DVARNISH_STATE_DIR='"${VARNISH_STATE_DIR}"' \
+	@vtree_extra_cflags@ \
 	@SAN_CFLAGS@
 
 libvarnishapi_la_LIBADD = \
diff --git a/lib/libvcc/Makefile.am b/lib/libvcc/Makefile.am
index 1e3d6f2d0..965e0b3f4 100644
--- a/lib/libvcc/Makefile.am
+++ b/lib/libvcc/Makefile.am
@@ -9,6 +9,7 @@ AM_CPPFLAGS = \
 noinst_LIBRARIES = libvcc.a
 
 libvcc_a_CFLAGS = \
+	@vtree_extra_cflags@ \
 	@SAN_CFLAGS@
 
 libvcc_a_SOURCES = \

From dridi at varni.sh  Wed Mar 24 17:46:47 2021
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 24 Mar 2021 17:46:47 +0000
Subject: [master] 777e03ba9 Disable strict aliasing with gcc where vtree.h
 is used
In-Reply-To: <20210324173804.D872D6EB8B@lists.varnish-cache.org>
References: <20210324173804.D872D6EB8B@lists.varnish-cache.org>
Message-ID: <CABoVN9B4905u9U5nBk3W1QU=B7qiCyc5hYi-gYvExX6JZHcG7Q@mail.gmail.com>

On Wed, Mar 24, 2021 at 5:38 PM Nils Goroll <nils.goroll at uplex.de> wrote:
>
>
> commit 777e03ba9774bf635a972fa0561b50ff071908a9
> Author: Nils Goroll <nils.goroll at uplex.de>
> Date:   Wed Mar 24 18:25:24 2021 +0100
>
>     Disable strict aliasing with gcc where vtree.h is used
>
>     Please replace this commit with a better solution, if you have one.
>
>     e1ac59335f749b84280722425b355be344cc76e9 brought us the VRBT node
>     color encoded in the lower two bits of pointers.
>
>     gcc strict aliasing rules (as enabled with -O2 and higher) forbid
>     aliased lvalue access as in the following macro expanded code:
>
>     (*(uintptr_t *)&(parent)->entry.rbe_parent) &= ~((uintptr_t)3);

Is culprit code part of generated vtree.h functions? Or macros that we
may call directly?

>     Until we have a better solution, disable strict aliasing with gcc
>     for targets with any source file using vtree.h.
>
>     Note that we would want to limit -fno-strict-aliasing to individual
>     compliation units, but automake does not offer a simple and clean way
>     to achieve this:
>
>     https://www.gnu.org/software/automake/manual/html_node/Per_002dObject-Flags.html

If this is only in generated functions, we may be able to annotate
them with some kind of v_no_strict_aliasing_ attribute.

At first glance it looks like macros we call directly would be subject
to strict aliasing too.

From dridi.boukelmoune at gmail.com  Thu Mar 25 10:23:06 2021
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Thu, 25 Mar 2021 10:23:06 +0000 (UTC)
Subject: [master] afd759284 vbe: Whitespace OCD
Message-ID: <20210325102306.AE58AB1D90@lists.varnish-cache.org>


commit afd759284567b94badfa806f5fd15224b687ac03
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed Mar 24 17:38:28 2021 +0100

    vbe: Whitespace OCD

diff --git a/bin/varnishd/cache/cache_backend.c b/bin/varnishd/cache/cache_backend.c
index 4c5255d37..3e55cb2ec 100644
--- a/bin/varnishd/cache/cache_backend.c
+++ b/bin/varnishd/cache/cache_backend.c
@@ -292,7 +292,7 @@ vbe_dir_gethdrs(VRT_CTX, VCL_BACKEND d)
 			extrachance = 0;
 
 		i = V1F_SendReq(wrk, bo, &bo->acct.bereq_hdrbytes,
-				&bo->acct.bereq_bodybytes);
+		    &bo->acct.bereq_bodybytes);
 
 		if (PFD_State(pfd) != PFD_STATE_USED) {
 			if (VCP_Wait(wrk, pfd, VTIM_real() +

From dridi.boukelmoune at gmail.com  Thu Mar 25 10:23:06 2021
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Thu, 25 Mar 2021 10:23:06 +0000 (UTC)
Subject: [master] 897695814 vbe: Generic sanity check of non-recyclable
 connections
Message-ID: <20210325102306.C51BCB1D93@lists.varnish-cache.org>


commit 897695814fa026dc263ed293caa822262c73a83a
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed Mar 24 17:39:47 2021 +0100

    vbe: Generic sanity check of non-recyclable connections
    
    The reason we expect here can be summarized as: this was a pipe
    transaction or an error occurred. This could be much simpler if
    we replaced enum sess_close with a struct stream_close instead.
    
    Refs dc5bddbd301529b101598b644544b99ccabca12c

diff --git a/bin/varnishd/cache/cache_backend.c b/bin/varnishd/cache/cache_backend.c
index 3e55cb2ec..d16ecbbd6 100644
--- a/bin/varnishd/cache/cache_backend.c
+++ b/bin/varnishd/cache/cache_backend.c
@@ -231,9 +231,15 @@ vbe_dir_finish(VRT_CTX, VCL_BACKEND d)
 	CHECK_OBJ_NOTNULL(bo->htc, HTTP_CONN_MAGIC);
 	pfd = bo->htc->priv;
 	bo->htc->priv = NULL;
-	if (PFD_State(pfd) != PFD_STATE_USED)
-		assert(bo->htc->doclose == SC_TX_PIPE ||
-		    bo->htc->doclose == SC_RX_TIMEOUT);
+	if (PFD_State(pfd) != PFD_STATE_USED) {
+		AN(bo->htc->doclose);
+		if (bo->htc->doclose != SC_TX_PIPE) {
+#define SESS_CLOSE(U, l, err, desc)				\
+			if (bo->htc->doclose == SC_ ## U)	\
+				AN(err);
+#include "tbl/sess_close.h"
+		}
+	}
 	if (bo->htc->doclose != SC_NULL || bp->proxy_header != 0) {
 		VSLb(bo->vsl, SLT_BackendClose, "%d %s close", *PFD_Fd(pfd),
 		    VRT_BACKEND_string(bp->director));

From dridi.boukelmoune at gmail.com  Thu Mar 25 10:23:06 2021
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Thu, 25 Mar 2021 10:23:06 +0000 (UTC)
Subject: [master] b8a375312 vbe: Check failures to send the request earlier
Message-ID: <20210325102306.E1C53B1D97@lists.varnish-cache.org>


commit b8a3753127ba799a6cbb20f8e651ef5a846516b3
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed Mar 24 17:50:23 2021 +0100

    vbe: Check failures to send the request earlier
    
    There's no point waiting for the backend to complain if we weren't able
    to properly send the backend request.
    
    Fixes #3556

diff --git a/bin/varnishd/cache/cache_backend.c b/bin/varnishd/cache/cache_backend.c
index d16ecbbd6..90c7899a5 100644
--- a/bin/varnishd/cache/cache_backend.c
+++ b/bin/varnishd/cache/cache_backend.c
@@ -300,7 +300,7 @@ vbe_dir_gethdrs(VRT_CTX, VCL_BACKEND d)
 		i = V1F_SendReq(wrk, bo, &bo->acct.bereq_hdrbytes,
 		    &bo->acct.bereq_bodybytes);
 
-		if (PFD_State(pfd) != PFD_STATE_USED) {
+		if (i == 0 && PFD_State(pfd) != PFD_STATE_USED) {
 			if (VCP_Wait(wrk, pfd, VTIM_real() +
 			    bo->htc->first_byte_timeout) != 0) {
 				bo->htc->doclose = SC_RX_TIMEOUT;
diff --git a/bin/varnishtest/tests/r03556.vtc b/bin/varnishtest/tests/r03556.vtc
new file mode 100644
index 000000000..d40afa47f
--- /dev/null
+++ b/bin/varnishtest/tests/r03556.vtc
@@ -0,0 +1,34 @@
+varnishtest "#3556"
+
+server s1 {
+	rxreq
+	txresp
+
+	non_fatal
+	rxreq
+} -start
+
+varnish v1 -cliok "param.set first_byte_timeout 10"
+varnish v1 -vcl+backend {} -start
+
+client c1 {
+	txreq
+	rxresp
+} -run
+
+logexpect l2 -v v1 -q "ReqMethod eq POST" {
+	expect * * End
+} -start
+
+client c2 {
+	txreq -req POST \
+	    -hdr "Content-Length: 10" \
+	    -hdr "Content-Type: text/plain"
+	send incompl
+} -run
+
+logexpect l2 -wait
+
+shell -expect POST {
+	exec varnishncsa -d -n ${v1_name} -q 'Timestamp:Process[2] < 10.0'
+}

From nils.goroll at uplex.de  Thu Mar 25 10:36:05 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Thu, 25 Mar 2021 10:36:05 +0000 (UTC)
Subject: [master] 9b8e2b17f Fix comment in the BASE64 decoding table
Message-ID: <20210325103605.431992A90@lists.varnish-cache.org>


commit 9b8e2b17f351e0bcc17d8bc84ef7a073607ee2dd
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Thu Mar 25 11:12:51 2021 +0100

    Fix comment in the BASE64 decoding table
    
    to match the character at position 0x2F '/'

diff --git a/vmod/vmod_blob_base64.c b/vmod/vmod_blob_base64.c
index b77c98e22..e275b963c 100644
--- a/vmod/vmod_blob_base64.c
+++ b/vmod/vmod_blob_base64.c
@@ -50,7 +50,7 @@ static const struct b64_alphabet {
 			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
 			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
 			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
-			ILL, ILL, ILL,  62, ILL, ILL, ILL,  63, /* +, -    */
+			ILL, ILL, ILL,  62, ILL, ILL, ILL,  63, /* +, /    */
 			 52,  53,  54,  55,  56,  57,  58,  59, /* 0 - 7   */
 			 60,  61, ILL, ILL, ILL, PAD, ILL, ILL, /* 8, 9, = */
 			ILL,   0,   1,   2,   3,   4,   5,   6, /* A - G   */

From nils.goroll at uplex.de  Thu Mar 25 10:36:05 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Thu, 25 Mar 2021 10:36:05 +0000 (UTC)
Subject: [master] bd01fccac Rename vmod blob tests
Message-ID: <20210325103605.6E4A32A94@lists.varnish-cache.org>


commit bd01fccac61e55e971b48803ff327be2d094a53b
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Thu Mar 25 11:34:43 2021 +0100

    Rename vmod blob tests

diff --git a/bin/varnishtest/tests/m00033.vtc b/vmod/tests/blob_b00000.vtc
similarity index 100%
rename from bin/varnishtest/tests/m00033.vtc
rename to vmod/tests/blob_b00000.vtc
diff --git a/bin/varnishtest/tests/m00034.vtc b/vmod/tests/blob_b00001.vtc
similarity index 100%
rename from bin/varnishtest/tests/m00034.vtc
rename to vmod/tests/blob_b00001.vtc
diff --git a/bin/varnishtest/tests/m00035.vtc b/vmod/tests/blob_b00002.vtc
similarity index 100%
rename from bin/varnishtest/tests/m00035.vtc
rename to vmod/tests/blob_b00002.vtc
diff --git a/bin/varnishtest/tests/m00036.vtc b/vmod/tests/blob_b00003.vtc
similarity index 100%
rename from bin/varnishtest/tests/m00036.vtc
rename to vmod/tests/blob_b00003.vtc
diff --git a/bin/varnishtest/tests/m00037.vtc b/vmod/tests/blob_b00004.vtc
similarity index 100%
rename from bin/varnishtest/tests/m00037.vtc
rename to vmod/tests/blob_b00004.vtc
diff --git a/bin/varnishtest/tests/m00038.vtc b/vmod/tests/blob_b00005.vtc
similarity index 100%
rename from bin/varnishtest/tests/m00038.vtc
rename to vmod/tests/blob_b00005.vtc
diff --git a/bin/varnishtest/tests/m00039.vtc b/vmod/tests/blob_b00006.vtc
similarity index 100%
rename from bin/varnishtest/tests/m00039.vtc
rename to vmod/tests/blob_b00006.vtc
diff --git a/bin/varnishtest/tests/m00040.vtc b/vmod/tests/blob_b00007.vtc
similarity index 100%
rename from bin/varnishtest/tests/m00040.vtc
rename to vmod/tests/blob_b00007.vtc
diff --git a/bin/varnishtest/tests/m00043.vtc b/vmod/tests/blob_b00008.vtc
similarity index 100%
rename from bin/varnishtest/tests/m00043.vtc
rename to vmod/tests/blob_b00008.vtc
diff --git a/bin/varnishtest/tests/m00044.vtc b/vmod/tests/blob_b00009.vtc
similarity index 100%
rename from bin/varnishtest/tests/m00044.vtc
rename to vmod/tests/blob_b00009.vtc
diff --git a/bin/varnishtest/tests/m00045.vtc b/vmod/tests/blob_b00010.vtc
similarity index 100%
rename from bin/varnishtest/tests/m00045.vtc
rename to vmod/tests/blob_b00010.vtc
diff --git a/bin/varnishtest/tests/m00049.vtc b/vmod/tests/blob_b00011.vtc
similarity index 100%
rename from bin/varnishtest/tests/m00049.vtc
rename to vmod/tests/blob_b00011.vtc

From nils.goroll at uplex.de  Thu Mar 25 10:38:05 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Thu, 25 Mar 2021 10:38:05 +0000 (UTC)
Subject: [master] bd3d4146c Rename vmod blob tests
Message-ID: <20210325103805.0CA762DE1@lists.varnish-cache.org>


commit bd3d4146c8be2ac6b89759d8b7382725e95d2da3
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Thu Mar 25 11:36:48 2021 +0100

    Rename vmod blob tests
    
    No idea why I just overlooked these

diff --git a/bin/varnishtest/tests/m00041.vtc b/vmod/tests/blob_b00012.vtc
similarity index 100%
rename from bin/varnishtest/tests/m00041.vtc
rename to vmod/tests/blob_b00012.vtc
diff --git a/bin/varnishtest/tests/m00042.vtc b/vmod/tests/blob_b00013.vtc
similarity index 100%
rename from bin/varnishtest/tests/m00042.vtc
rename to vmod/tests/blob_b00013.vtc

From nils.goroll at uplex.de  Thu Mar 25 11:01:04 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Thu, 25 Mar 2021 11:01:04 +0000 (UTC)
Subject: [master] df8532818 Add BASE64CF codec
Message-ID: <20210325110104.9C4DE4A72@lists.varnish-cache.org>


commit df85328189b2a0cd626e8c31ab2d20ac31e50d78
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Thu Mar 25 11:37:42 2021 +0100

    Add BASE64CF codec
    
    Each their own variant, *sigh*

diff --git a/vmod/tests/blob_b00004.vtc b/vmod/tests/blob_b00004.vtc
index 3d463ec0e..e735d583b 100644
--- a/vmod/tests/blob_b00004.vtc
+++ b/vmod/tests/blob_b00004.vtc
@@ -96,6 +96,14 @@ varnish v1 -vcl {
 	    set resp.http.b64urlnopadxcode =
 	      blob.transcode(IDENTITY, BASE64URLNOPAD,
 			     encoded=req.url + "Hello world" + req.url);
+
+	    set resp.http.b64cfxcodeout =
+	      blob.transcode(BASE64, BASE64CF,
+			     encoded="+/Y=");
+
+	    set resp.http.b64cfxcodein =
+	      blob.transcode(BASE64CF, BASE64,
+			     encoded="-~Y_");
 	}
 } -start
 
@@ -122,6 +130,8 @@ client c1 {
 	expect resp.http.b64xcode == "L0hlbGxvIHdvcmxkLw=="
 	expect resp.http.b64urlxcode == "L0hlbGxvIHdvcmxkLw=="
 	expect resp.http.b64urlnopadxcode == "L0hlbGxvIHdvcmxkLw"
+	expect resp.http.b64cfxcodeout == "-~Y_"
+	expect resp.http.b64cfxcodein == "+/Y="
 } -run
 
 varnish v1 -vcl {
diff --git a/vmod/vmod_blob.c b/vmod/vmod_blob.c
index 70644f5ad..3ebb97edf 100644
--- a/vmod/vmod_blob.c
+++ b/vmod/vmod_blob.c
@@ -76,6 +76,10 @@ static const struct vmod_blob_fptr {
 		B64_FUNCS,
 		.encode_l	= base64nopad_encode_l
 	},
+	[BASE64CF] = {
+		B64_FUNCS,
+		.encode_l	= base64_encode_l
+	},
 	[HEX] = {
 		.decode_l	= hex_decode_l,
 		.decode		= hex_decode,
diff --git a/vmod/vmod_blob.vcc b/vmod/vmod_blob.vcc
index 514f6028e..b3644c66c 100644
--- a/vmod/vmod_blob.vcc
+++ b/vmod/vmod_blob.vcc
@@ -70,6 +70,7 @@ ENUM values for an encoding scheme can be one of:
 * ``BASE64``
 * ``BASE64URL``
 * ``BASE64URLNOPAD``
+* ``BASE64CF``
 * ``HEX``
 * ``URL``
 
@@ -141,6 +142,11 @@ The ``BASE64URLNOPAD`` encoding uses the same alphabet as
 ``BASE6URL``, but leaves out the padding. Thus the length of an
 encoding with this scheme is not necessarily a multiple of four.
 
+The ``BASE64CF` is similar to ``BASE64URL``, with the following
+changes to ``BASE64``: ``+`` replaced with ``-``, ``/`` replaced with
+``~`` and ``_`` as the padding character. It is used by a certain CDN
+provider who also inspired the name.
+
 The *case* ENUM MUST be set to ``DEFAULT`` for for all of the
 ``BASE64*`` encodings.
 
@@ -176,8 +182,8 @@ The ``URL`` encoding implements "percent encoding" as per RFC3986. The
 affect alphabetic characters that are not percent-encoded.
 
 $Function BLOB decode(ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD,
-			    HEX, URL} decoding="IDENTITY", INT length=0,
-		      STRANDS encoded)
+			    BASE64CF, HEX, URL} decoding="IDENTITY",
+			    INT length=0, STRANDS encoded)
 
 Returns the BLOB derived from the string *encoded* according to the
 scheme specified by *decoding*.
@@ -201,7 +207,7 @@ Example::
 
 
 $Function STRING encode(ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD,
-			      HEX, URL} encoding="IDENTITY",
+			      BASE64CF, HEX, URL} encoding="IDENTITY",
 			ENUM {LOWER, UPPER, DEFAULT} case="DEFAULT", BLOB blob)
 
 Returns a string representation of the BLOB *blob* as specified by
@@ -230,9 +236,9 @@ Example::
 	    = blob.encode(blob=blob.decode(encoded="foo"));
 
 $Function STRING transcode(ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD,
-				 HEX, URL} decoding="IDENTITY",
+				 BASE64CF, HEX, URL} decoding="IDENTITY",
 			   ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD,
-				 HEX, URL} encoding="IDENTITY",
+				 BASE64CF, HEX, URL} encoding="IDENTITY",
 			   ENUM {LOWER, UPPER, DEFAULT} case="DEFAULT",
 			   INT length=0, STRANDS encoded)
 
@@ -303,8 +309,8 @@ default value of *offset* is ``0B``.
 ``offset + length`` requires more bytes than are available in the
 BLOB.
 
-$Object blob(ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX,
-		   URL} decoding="IDENTITY",
+$Object blob(ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, BASE64CF,
+		   HEX, URL} decoding="IDENTITY",
 	     STRANDS encoded)
 
 Creates an object that contains the BLOB derived from the string
@@ -335,8 +341,8 @@ Example::
 	set resp.http.The-Stringblob =
 	    blob.encode(blob=stringblob.get());
 
-$Method STRING .encode(ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD, HEX,
-			     URL} encoding="IDENTITY",
+$Method STRING .encode(ENUM {IDENTITY, BASE64, BASE64URL, BASE64URLNOPAD,
+			     BASE64CF, HEX,URL} encoding="IDENTITY",
 		       ENUM {LOWER, UPPER, DEFAULT} case="DEFAULT")
 
 Returns an encoding of BLOB created by the constructor, according to
diff --git a/vmod/vmod_blob_base64.c b/vmod/vmod_blob_base64.c
index e275b963c..876f6e309 100644
--- a/vmod/vmod_blob_base64.c
+++ b/vmod/vmod_blob_base64.c
@@ -158,6 +158,45 @@ static const struct b64_alphabet {
 		},
 		0
 	},
+	[BASE64CF] = {
+		"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdef"
+		"ghijklmnopqrstuvwxyz0123456789-~",
+		{
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL,  62, ILL, ILL, /* -          */
+			 52,  53,  54,  55,  56,  57,  58,  59, /* 0 - 7      */
+			 60,  61, ILL, ILL, ILL, ILL, ILL, ILL, /* 8, 9       */
+			ILL,   0,   1,   2,   3,   4,   5,   6, /* A - G      */
+			  7,   8,   9,  10,  11,  12,  13,  14, /* H - O      */
+			 15,  16,  17,  18,  19,  20,  21,  22, /* P - W      */
+			 23,  24,  25, ILL, ILL, ILL, ILL, PAD, /* X, Y, Z, _ */
+			ILL,  26,  27,  28,  29,  30,  31,  32, /* a - g      */
+			 33,  34,  35,  36,  37,  38,  39,  40, /* h - o      */
+			 41,  42,  43,  44,  45,  46,  47,  48, /* p - w      */
+			 49,  50,  51, ILL, ILL, ILL,  63, ILL, /* x, y, z,  ~*/
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+			ILL, ILL, ILL, ILL, ILL, ILL, ILL, ILL,
+		},
+		'_'
+	}
 };
 #define base64_l(l)		(((l) << 2) / 3)
 
diff --git a/vmod/vmod_blob_tbl_encodings.h b/vmod/vmod_blob_tbl_encodings.h
index 48765b27c..b61e211d6 100644
--- a/vmod/vmod_blob_tbl_encodings.h
+++ b/vmod/vmod_blob_tbl_encodings.h
@@ -6,6 +6,7 @@
 VMODENUM(BASE64)
 VMODENUM(BASE64URL)
 VMODENUM(BASE64URLNOPAD)
+VMODENUM(BASE64CF)
 VMODENUM(HEX)
 VMODENUM(IDENTITY)
 VMODENUM(URL)

From nils.goroll at uplex.de  Thu Mar 25 16:40:06 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Thu, 25 Mar 2021 16:40:06 +0000 (UTC)
Subject: [master] 9a1c431cc vrt.h changes formatting glitch
Message-ID: <20210325164006.74B2B65C85@lists.varnish-cache.org>


commit 9a1c431cc20d56f1f1a064b69fd6c0740105fe3b
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Thu Mar 25 17:39:20 2021 +0100

    vrt.h changes formatting glitch

diff --git a/include/vrt.h b/include/vrt.h
index 1d15464d7..7d50b3431 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -54,7 +54,7 @@
  * binary/load-time compatible, increment MAJOR version
  *
  * NEXT (2021-09-15)
-	vrt_null_strands added
+ *	vrt_null_strands added
  * 13.0 (2021-03-15)
  *	Move VRT_synth_page() to deprecated status
  *	Add VRT_synth_strands() and VRT_synth_blob()

From nils.goroll at uplex.de  Thu Mar 25 17:21:04 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Thu, 25 Mar 2021 17:21:04 +0000 (UTC)
Subject: [master] 7821d6447 For VCL_acl, log the ACL entry's netmask
Message-ID: <20210325172104.89B4E91329@lists.varnish-cache.org>


commit 7821d6447b888be645235e2dce7875257df69af3
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Fri Dec 13 18:10:21 2019 +0100

    For VCL_acl, log the ACL entry's netmask
    
    As of now, this makes no difference to the vcc-generated code.

diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index 46d6d2769..e03928ba0 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -416,6 +416,8 @@ vcc_acl_emit_tokens(const struct vcc *tl, const struct acl_e *ae)
 			Fh(tl, 0, "%s\"\\\"\" ", sep);
 			EncToken(tl->fh, t);
 			Fh(tl, 0, " \"\\\"\"");
+		} else if (t == ae->t_mask) {
+			Fh(tl, 0, " \"%u\"", ae->mask - 8);
 		} else {
 			Fh(tl, 0, "%s\"%.*s\"", sep, PF(t));
 		}

From nils.goroll at uplex.de  Thu Mar 25 17:21:04 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Thu, 25 Mar 2021 17:21:04 +0000 (UTC)
Subject: [master] bbb5db47e vcc_acl: split out insertion function
Message-ID: <20210325172104.9D6A09132D@lists.varnish-cache.org>


commit bbb5db47e02f431598e5a7ff7106a8ac316db8be
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Fri Dec 13 18:01:20 2019 +0100

    vcc_acl: split out insertion function
    
    Reduces conflicts with a held-back PR

diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index e03928ba0..ed0042a67 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -155,11 +155,30 @@ vcc_acl_chk(struct vcc *tl, const struct acl_e *ae, const int l,
 	return (strdup(t));
 }
 
+static void
+vcc_acl_insert_entry(struct vcc *tl, struct acl_e *aen)
+{
+	struct acl_e *ae2;
+
+	ae2 = VRBT_FIND(acl_tree, &tl->acl_tree, aen);
+	if (ae2 != NULL) {
+		if (ae2->not != aen->not) {
+			VSB_cat(tl->sb, "Conflicting ACL entries:\n");
+			vcc_ErrWhere(tl, ae2->t_addr);
+			VSB_cat(tl->sb, "vs:\n");
+			vcc_ErrWhere(tl, aen->t_addr);
+		}
+		free(aen);
+		return;
+	}
+	VRBT_INSERT(acl_tree, &tl->acl_tree, aen);
+}
+
 static void
 vcc_acl_add_entry(struct vcc *tl, const struct acl_e *ae, int l,
     unsigned char *u, int fam)
 {
-	struct acl_e *ae2, *aen;
+	struct acl_e *aen;
 
 	if (fam == PF_INET && ae->mask > 32) {
 		VSB_printf(tl->sb,
@@ -178,7 +197,7 @@ vcc_acl_add_entry(struct vcc *tl, const struct acl_e *ae, int l,
 	}
 
 	/* Make a copy from the template */
-	aen = TlAlloc(tl, sizeof *ae2);
+	aen = TlAlloc(tl, sizeof *aen);
 	AN(aen);
 	*aen = *ae;
 
@@ -192,18 +211,7 @@ vcc_acl_add_entry(struct vcc *tl, const struct acl_e *ae, int l,
 	assert(l + 1UL <= sizeof aen->data);
 	memcpy(aen->data + 1L, u, l);
 
-	ae2 = VRBT_FIND(acl_tree, &tl->acl_tree, aen);
-	if (ae2 != NULL) {
-		if (ae2->not != aen->not) {
-			VSB_cat(tl->sb, "Conflicting ACL entries:\n");
-			vcc_ErrWhere(tl, ae2->t_addr);
-			VSB_cat(tl->sb, "vs:\n");
-			vcc_ErrWhere(tl, aen->t_addr);
-		}
-		free(aen);
-		return;
-	}
-	VRBT_INSERT(acl_tree, &tl->acl_tree, aen);
+	vcc_acl_insert_entry(tl, aen);
 }
 
 static void

From nils.goroll at uplex.de  Fri Mar 26 14:23:04 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Fri, 26 Mar 2021 14:23:04 +0000 (UTC)
Subject: [master] 15431a544 vcc_acl: an RBT insert includes a find operation
Message-ID: <20210326142304.CC4C4643ED@lists.varnish-cache.org>


commit 15431a544f97dda1c512a7b52bfba72e9865a787
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Fri Mar 26 15:21:39 2021 +0100

    vcc_acl: an RBT insert includes a find operation

diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index ed0042a67..6d0cd6b25 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -95,7 +95,6 @@ vcl_acl_cmp(const struct acl_e *ae1, const struct acl_e *ae2)
 }
 
 VRBT_GENERATE_INSERT_COLOR(acl_tree, acl_e, branch, static)
-VRBT_GENERATE_FIND(acl_tree, acl_e, branch, vcl_acl_cmp, static)
 VRBT_GENERATE_INSERT(acl_tree, acl_e, branch, vcl_acl_cmp, static)
 VRBT_GENERATE_MINMAX(acl_tree, acl_e, branch, static)
 VRBT_GENERATE_NEXT(acl_tree, acl_e, branch, static)
@@ -160,7 +159,7 @@ vcc_acl_insert_entry(struct vcc *tl, struct acl_e *aen)
 {
 	struct acl_e *ae2;
 
-	ae2 = VRBT_FIND(acl_tree, &tl->acl_tree, aen);
+	ae2 = VRBT_INSERT(acl_tree, &tl->acl_tree, aen);
 	if (ae2 != NULL) {
 		if (ae2->not != aen->not) {
 			VSB_cat(tl->sb, "Conflicting ACL entries:\n");
@@ -171,7 +170,6 @@ vcc_acl_insert_entry(struct vcc *tl, struct acl_e *aen)
 		free(aen);
 		return;
 	}
-	VRBT_INSERT(acl_tree, &tl->acl_tree, aen);
 }
 
 static void

From nils.goroll at uplex.de  Sat Mar 27 13:40:07 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sat, 27 Mar 2021 13:40:07 +0000 (UTC)
Subject: [master] aff516b77 vcc_acl: struct acl_e to miniobj
Message-ID: <20210327134007.6D436637BB@lists.varnish-cache.org>


commit aff516b776039c5317b3ee8126e854984b22e2ab
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Sat Mar 27 14:13:02 2021 +0100

    vcc_acl: struct acl_e to miniobj
    
    note: struct acl_e could be compacted, but we prefer clarify over
    efficiency in VCC.

diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index 6d0cd6b25..6f6591d7d 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -46,6 +46,8 @@
 #define ACL_MAXADDR	(sizeof(struct in6_addr) + 1)
 
 struct acl_e {
+	unsigned		magic;
+#define VCC_ACL_E_MAGIC	0xcac81e23
 	VRBT_ENTRY(acl_e)	branch;
 	unsigned char		data[ACL_MAXADDR];
 	unsigned		mask;
@@ -73,6 +75,9 @@ vcl_acl_cmp(const struct acl_e *ae1, const struct acl_e *ae2)
 	const unsigned char *p1, *p2;
 	unsigned m;
 
+	CHECK_OBJ_NOTNULL(ae1, VCC_ACL_E_MAGIC);
+	CHECK_OBJ_NOTNULL(ae2, VCC_ACL_E_MAGIC);
+
 	p1 = ae1->data;
 	p2 = ae2->data;
 	m = ae1->mask;
@@ -159,6 +164,7 @@ vcc_acl_insert_entry(struct vcc *tl, struct acl_e *aen)
 {
 	struct acl_e *ae2;
 
+	CHECK_OBJ_NOTNULL(aen, VCC_ACL_E_MAGIC);
 	ae2 = VRBT_INSERT(acl_tree, &tl->acl_tree, aen);
 	if (ae2 != NULL) {
 		if (ae2->not != aen->not) {
@@ -167,7 +173,7 @@ vcc_acl_insert_entry(struct vcc *tl, struct acl_e *aen)
 			VSB_cat(tl->sb, "vs:\n");
 			vcc_ErrWhere(tl, aen->t_addr);
 		}
-		free(aen);
+		FREE_OBJ(aen);
 		return;
 	}
 }
@@ -195,8 +201,7 @@ vcc_acl_add_entry(struct vcc *tl, const struct acl_e *ae, int l,
 	}
 
 	/* Make a copy from the template */
-	aen = TlAlloc(tl, sizeof *aen);
-	AN(aen);
+	ALLOC_OBJ(aen, VCC_ACL_E_MAGIC);
 	*aen = *ae;
 
 	aen->fixed = vcc_acl_chk(tl, ae, l, u, fam);
@@ -221,6 +226,7 @@ vcc_acl_try_getaddrinfo(struct vcc *tl, struct acl_e *ae)
 	unsigned char *u, i4, i6;
 	int error;
 
+	CHECK_OBJ_NOTNULL(ae, VCC_ACL_E_MAGIC);
 	memset(&hint, 0, sizeof hint);
 	hint.ai_family = PF_UNSPEC;
 	hint.ai_socktype = SOCK_STREAM;
@@ -319,6 +325,7 @@ vcc_acl_try_netnotation(struct vcc *tl, struct acl_e *ae)
 	unsigned u;
 	const char *p;
 
+	CHECK_OBJ_NOTNULL(ae, VCC_ACL_E_MAGIC);
 	memset(b, 0, sizeof b);
 	p = ae->addr;
 	for (i = 0; i < 4; i++) {
@@ -346,8 +353,7 @@ vcc_acl_entry(struct vcc *tl)
 	struct acl_e *ae;
 	char *sl, *e;
 
-	ae = TlAlloc(tl, sizeof *ae);
-	AN(ae);
+	ALLOC_OBJ(ae, VCC_ACL_E_MAGIC);
 
 	if (tl->t->tok == '!') {
 		ae->not = 1;
@@ -379,14 +385,14 @@ vcc_acl_entry(struct vcc *tl)
 		if (*e != '\0') {
 			VSB_cat(tl->sb, ".../mask is not numeric.\n");
 			vcc_ErrWhere(tl, ae->t_addr);
-			free(ae);
+			FREE_OBJ(ae);
 			return;
 		}
 		ae->t_mask = ae->t_addr;
 		if (tl->t->tok == '/') {
 			VSB_cat(tl->sb, "/mask only allowed once.\n");
 			vcc_ErrWhere(tl, tl->t);
-			free(ae);
+			FREE_OBJ(ae);
 			return;
 		}
 	} else if (tl->t->tok == '/') {
@@ -416,6 +422,7 @@ vcc_acl_emit_tokens(const struct vcc *tl, const struct acl_e *ae)
 	struct token *t;
 	const char *sep = "";
 
+	CHECK_OBJ_NOTNULL(ae, VCC_ACL_E_MAGIC);
 	t = ae->t_addr;
 	do {
 		if (t->tok == CSTR) {

From nils.goroll at uplex.de  Sat Mar 27 13:40:07 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sat, 27 Mar 2021 13:40:07 +0000 (UTC)
Subject: [master] f65f7148c vcc_acl: Create template acl_e on the stack
Message-ID: <20210327134007.81E52637BF@lists.varnish-cache.org>


commit f65f7148c6a6d27e973acf62fc9e12480a0e076a
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Sat Mar 27 14:20:56 2021 +0100

    vcc_acl: Create template acl_e on the stack
    
    We only ever insert copies into the tree in vcc_acl_add_entry()

diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index 6f6591d7d..127b92adb 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -350,10 +350,10 @@ vcc_acl_try_netnotation(struct vcc *tl, struct acl_e *ae)
 static void
 vcc_acl_entry(struct vcc *tl)
 {
-	struct acl_e *ae;
+	struct acl_e ae[1];
 	char *sl, *e;
 
-	ALLOC_OBJ(ae, VCC_ACL_E_MAGIC);
+	INIT_OBJ(ae, VCC_ACL_E_MAGIC);
 
 	if (tl->t->tok == '!') {
 		ae->not = 1;
@@ -385,14 +385,12 @@ vcc_acl_entry(struct vcc *tl)
 		if (*e != '\0') {
 			VSB_cat(tl->sb, ".../mask is not numeric.\n");
 			vcc_ErrWhere(tl, ae->t_addr);
-			FREE_OBJ(ae);
 			return;
 		}
 		ae->t_mask = ae->t_addr;
 		if (tl->t->tok == '/') {
 			VSB_cat(tl->sb, "/mask only allowed once.\n");
 			vcc_ErrWhere(tl, tl->t);
-			FREE_OBJ(ae);
 			return;
 		}
 	} else if (tl->t->tok == '/') {

From nils.goroll at uplex.de  Sat Mar 27 13:40:07 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sat, 27 Mar 2021 13:40:07 +0000 (UTC)
Subject: [master] 521fbc45e Stabilize test
Message-ID: <20210327134007.9ED9E637C2@lists.varnish-cache.org>


commit 521fbc45eaa70e2f7a82197ffeefa68d6eabafb6
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Sat Mar 27 14:39:07 2021 +0100

    Stabilize test

diff --git a/bin/varnishtest/tests/c00069.vtc b/bin/varnishtest/tests/c00069.vtc
index fc41422da..55e9557d0 100644
--- a/bin/varnishtest/tests/c00069.vtc
+++ b/bin/varnishtest/tests/c00069.vtc
@@ -1,22 +1,35 @@
 varnishtest "Test resp.is_streaming"
 
+barrier b1 sock 2
+barrier b2 sock 2
+
 server s1 {
 	rxreq
 	txresp -nolen -hdr "Content-Length: 10"
-	delay 1
+	barrier b1 sync
+	barrier b2 sync
 	send "1234567890"
 } -start
 
 varnish v1 -vcl+backend {
+	import vtc;
+
 	sub vcl_recv {
 		if (req.url == "/synth") {
 			return(synth(200, "OK"));
 		}
 	}
+	sub vcl_backend_response {
+		vtc.barrier_sync("${b1_sock}");
+		return (deliver);
+	}
 	sub vcl_synth {
 		set resp.http.streaming = resp.is_streaming;
 	}
 	sub vcl_deliver {
+		if (obj.hits == 0) {
+			vtc.barrier_sync("${b2_sock}");
+		}
 		set resp.http.streaming = resp.is_streaming;
 	}
 } -start
@@ -36,4 +49,3 @@ client c1 {
 	rxresp
 	expect resp.http.streaming == "false"
 } -run
-

From nils.goroll at uplex.de  Sat Mar 27 13:54:04 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sat, 27 Mar 2021 13:54:04 +0000 (UTC)
Subject: [master] a3664eadc update removal marker
Message-ID: <20210327135404.60FE2642E4@lists.varnish-cache.org>


commit a3664eadcede0a34e2291ee061e7de951ac8620a
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Sat Mar 27 14:53:31 2021 +0100

    update removal marker

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index d80877e76..d5b7f458e 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -796,7 +796,7 @@ void WS_Assert_Allocated(const struct ws *ws, const void *ptr, ssize_t len);
 void WS_VSB_new(struct vsb *, struct ws *);
 char *WS_VSB_finish(struct vsb *, struct ws *, size_t *);
 
-/* REL_20210315 remove */
+/* REL_20210915 remove */
 static inline char*
 WS_Front(const struct ws *ws)
 {

From nils.goroll at uplex.de  Sat Mar 27 14:19:04 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sat, 27 Mar 2021 14:19:04 +0000 (UTC)
Subject: [master] bf854df69 vcc_acl: flexelint
Message-ID: <20210327141904.A8E0A64F27@lists.varnish-cache.org>


commit bf854df693f68a8f624eb59ea329db74aa03d7b2
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Sat Mar 27 15:13:51 2021 +0100

    vcc_acl: flexelint
    
    Ref aff516b776039c5317b3ee8126e854984b22e2ab
        f65f7148c6a6d27e973acf62fc9e12480a0e076a

diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index 127b92adb..2b5708a75 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -54,7 +54,7 @@ struct acl_e {
 	unsigned		not;
 	unsigned		para;
 	char			*addr;
-	char			*fixed;
+	const char		*fixed;
 	struct token		*t_addr;
 	struct token		*t_mask;
 };
@@ -69,6 +69,16 @@ struct acl_e {
 			return (1);					\
 	} while (0)
 
+static void
+vcl_acl_free(struct acl_e **aep)
+{
+	AN(aep);
+	CHECK_OBJ_NOTNULL(*aep, VCC_ACL_E_MAGIC);
+	if ((*aep)->addr != NULL)
+		free((*aep)->addr);
+	FREE_OBJ(*aep);
+}
+
 static int
 vcl_acl_cmp(const struct acl_e *ae1, const struct acl_e *ae2)
 {
@@ -159,7 +169,7 @@ vcc_acl_chk(struct vcc *tl, const struct acl_e *ae, const int l,
 	return (strdup(t));
 }
 
-static void
+static struct acl_e *
 vcc_acl_insert_entry(struct vcc *tl, struct acl_e *aen)
 {
 	struct acl_e *ae2;
@@ -173,12 +183,12 @@ vcc_acl_insert_entry(struct vcc *tl, struct acl_e *aen)
 			VSB_cat(tl->sb, "vs:\n");
 			vcc_ErrWhere(tl, aen->t_addr);
 		}
-		FREE_OBJ(aen);
-		return;
+		return (NULL);
 	}
+	return (aen);
 }
 
-static void
+static struct acl_e *
 vcc_acl_add_entry(struct vcc *tl, const struct acl_e *ae, int l,
     unsigned char *u, int fam)
 {
@@ -191,17 +201,18 @@ vcc_acl_add_entry(struct vcc *tl, const struct acl_e *ae, int l,
 			vcc_ErrWhere(tl, ae->t_mask);
 		else
 			vcc_ErrWhere(tl, ae->t_addr);
-		return;
+		return (NULL);
 	}
 	if (fam == PF_INET6 && ae->mask > 128) {
 		VSB_printf(tl->sb,
 		    "Too wide mask (/%u) for IPv6 address\n", ae->mask);
 		vcc_ErrWhere(tl, ae->t_mask);
-		return;
+		return (NULL);
 	}
 
 	/* Make a copy from the template */
 	ALLOC_OBJ(aen, VCC_ACL_E_MAGIC);
+	AN(aen);
 	*aen = *ae;
 
 	aen->fixed = vcc_acl_chk(tl, ae, l, u, fam);
@@ -214,7 +225,9 @@ vcc_acl_add_entry(struct vcc *tl, const struct acl_e *ae, int l,
 	assert(l + 1UL <= sizeof aen->data);
 	memcpy(aen->data + 1L, u, l);
 
-	vcc_acl_insert_entry(tl, aen);
+	if (vcc_acl_insert_entry(tl, aen) == NULL)
+		vcl_acl_free(&aen);
+	return (aen);
 }
 
 static void
@@ -288,7 +301,7 @@ vcc_acl_try_getaddrinfo(struct vcc *tl, struct acl_e *ae)
 			u = (void*)&sin4->sin_addr;
 			if (ae->t_mask == NULL)
 				ae->mask = 32;
-			vcc_acl_add_entry(tl, ae, 4, u, res->ai_family);
+			(void) vcc_acl_add_entry(tl, ae, 4, u, res->ai_family);
 			break;
 		case PF_INET6:
 			assert(PF_INET6 < 256);
@@ -297,7 +310,7 @@ vcc_acl_try_getaddrinfo(struct vcc *tl, struct acl_e *ae)
 			u = (void*)&sin6->sin6_addr;
 			if (ae->t_mask == NULL)
 				ae->mask = 128;
-			vcc_acl_add_entry(tl, ae, 16, u, res->ai_family);
+			(void) vcc_acl_add_entry(tl, ae, 16, u, res->ai_family);
 			break;
 		default:
 			continue;
@@ -343,7 +356,7 @@ vcc_acl_try_netnotation(struct vcc *tl, struct acl_e *ae)
 	}
 	if (ae->t_mask == NULL)
 		ae->mask = 8 + 8 * i;
-	vcc_acl_add_entry(tl, ae, 4, b, AF_INET);
+	(void) vcc_acl_add_entry(tl, ae, 4, b, AF_INET);
 	return (1);
 }
 

From nils.goroll at uplex.de  Sat Mar 27 15:45:05 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sat, 27 Mar 2021 15:45:05 +0000 (UTC)
Subject: [master] 47489373b vcc_acl: flexelint^2
Message-ID: <20210327154505.AF38591772@lists.varnish-cache.org>


commit 47489373b9fa7e1b92b49bb1319fac9b859c6e7e
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Sat Mar 27 16:40:04 2021 +0100

    vcc_acl: flexelint^2
    
    Apologies for the slight back and forth, but I now think this is a
    better way to make clear to flexelint that we either keep an allocated
    acl_e or not.
    
    This partly undoes changes from bf854df693f68a8f624eb59ea329db74aa03d7b2
    
    To review, I recommend diff'ing to a3664eadcede0a34e2291ee061e7de951ac8620a
    
    Ref aff516b776039c5317b3ee8126e854984b22e2ab
        f65f7148c6a6d27e973acf62fc9e12480a0e076a

diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index 2b5708a75..2cd215f3c 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -169,26 +169,26 @@ vcc_acl_chk(struct vcc *tl, const struct acl_e *ae, const int l,
 	return (strdup(t));
 }
 
-static struct acl_e *
-vcc_acl_insert_entry(struct vcc *tl, struct acl_e *aen)
+static void
+vcc_acl_insert_entry(struct vcc *tl, struct acl_e **aenp)
 {
 	struct acl_e *ae2;
 
-	CHECK_OBJ_NOTNULL(aen, VCC_ACL_E_MAGIC);
-	ae2 = VRBT_INSERT(acl_tree, &tl->acl_tree, aen);
+	CHECK_OBJ_NOTNULL(*aenp, VCC_ACL_E_MAGIC);
+	ae2 = VRBT_INSERT(acl_tree, &tl->acl_tree, *aenp);
 	if (ae2 != NULL) {
-		if (ae2->not != aen->not) {
+		if (ae2->not != (*aenp)->not) {
 			VSB_cat(tl->sb, "Conflicting ACL entries:\n");
 			vcc_ErrWhere(tl, ae2->t_addr);
 			VSB_cat(tl->sb, "vs:\n");
-			vcc_ErrWhere(tl, aen->t_addr);
+			vcc_ErrWhere(tl, (*aenp)->t_addr);
 		}
-		return (NULL);
+		return;
 	}
-	return (aen);
+	*aenp = NULL;
 }
 
-static struct acl_e *
+static void
 vcc_acl_add_entry(struct vcc *tl, const struct acl_e *ae, int l,
     unsigned char *u, int fam)
 {
@@ -201,13 +201,13 @@ vcc_acl_add_entry(struct vcc *tl, const struct acl_e *ae, int l,
 			vcc_ErrWhere(tl, ae->t_mask);
 		else
 			vcc_ErrWhere(tl, ae->t_addr);
-		return (NULL);
+		return;
 	}
 	if (fam == PF_INET6 && ae->mask > 128) {
 		VSB_printf(tl->sb,
 		    "Too wide mask (/%u) for IPv6 address\n", ae->mask);
 		vcc_ErrWhere(tl, ae->t_mask);
-		return (NULL);
+		return;
 	}
 
 	/* Make a copy from the template */
@@ -225,9 +225,9 @@ vcc_acl_add_entry(struct vcc *tl, const struct acl_e *ae, int l,
 	assert(l + 1UL <= sizeof aen->data);
 	memcpy(aen->data + 1L, u, l);
 
-	if (vcc_acl_insert_entry(tl, aen) == NULL)
+	vcc_acl_insert_entry(tl, &aen);
+	if (aen != NULL)
 		vcl_acl_free(&aen);
-	return (aen);
 }
 
 static void
@@ -301,7 +301,7 @@ vcc_acl_try_getaddrinfo(struct vcc *tl, struct acl_e *ae)
 			u = (void*)&sin4->sin_addr;
 			if (ae->t_mask == NULL)
 				ae->mask = 32;
-			(void) vcc_acl_add_entry(tl, ae, 4, u, res->ai_family);
+			vcc_acl_add_entry(tl, ae, 4, u, res->ai_family);
 			break;
 		case PF_INET6:
 			assert(PF_INET6 < 256);
@@ -310,7 +310,7 @@ vcc_acl_try_getaddrinfo(struct vcc *tl, struct acl_e *ae)
 			u = (void*)&sin6->sin6_addr;
 			if (ae->t_mask == NULL)
 				ae->mask = 128;
-			(void) vcc_acl_add_entry(tl, ae, 16, u, res->ai_family);
+			vcc_acl_add_entry(tl, ae, 16, u, res->ai_family);
 			break;
 		default:
 			continue;
@@ -356,7 +356,7 @@ vcc_acl_try_netnotation(struct vcc *tl, struct acl_e *ae)
 	}
 	if (ae->t_mask == NULL)
 		ae->mask = 8 + 8 * i;
-	(void) vcc_acl_add_entry(tl, ae, 4, b, AF_INET);
+	vcc_acl_add_entry(tl, ae, 4, b, AF_INET);
 	return (1);
 }
 

From nils.goroll at uplex.de  Sat Mar 27 16:44:04 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sat, 27 Mar 2021 16:44:04 +0000 (UTC)
Subject: [master] 2cf57b2e9 vcc_acl: centralize acl_e copying
Message-ID: <20210327164404.5718293517@lists.varnish-cache.org>


commit 2cf57b2e951a6df2c7d13733f100cd293223500c
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Sat Mar 27 17:39:10 2021 +0100

    vcc_acl: centralize acl_e copying
    
    Move the address strdup() to where the acl_e is allocated.
    
    This plugs an utterly insignificant leak for error returns from
    vcc_acl_entry().

diff --git a/lib/libvcc/vcc_acl.c b/lib/libvcc/vcc_acl.c
index 2cd215f3c..82cd31985 100644
--- a/lib/libvcc/vcc_acl.c
+++ b/lib/libvcc/vcc_acl.c
@@ -214,6 +214,8 @@ vcc_acl_add_entry(struct vcc *tl, const struct acl_e *ae, int l,
 	ALLOC_OBJ(aen, VCC_ACL_E_MAGIC);
 	AN(aen);
 	*aen = *ae;
+	aen->addr = strdup(ae->addr);
+	AN(aen->addr);
 
 	aen->fixed = vcc_acl_chk(tl, ae, l, u, fam);
 
@@ -385,8 +387,7 @@ vcc_acl_entry(struct vcc *tl)
 
 	ExpectErr(tl, CSTR);
 	ae->t_addr = tl->t;
-	ae->addr = strdup(ae->t_addr->dec);
-	AN(ae->addr);
+	ae->addr = ae->t_addr->dec;
 	vcc_NextToken(tl);
 
 	if (strchr(ae->t_addr->dec, '/') != NULL) {

From nils.goroll at uplex.de  Sat Mar 27 18:14:04 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sat, 27 Mar 2021 18:14:04 +0000 (UTC)
Subject: [master] 599b183d3 Silence gcc strict aliasing warnings where vtree.h
 is used
Message-ID: <20210327181404.E8641962DF@lists.varnish-cache.org>


commit 599b183d3f34d5280bddb2abf755ca8b4cfe768b
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Sat Mar 27 19:00:40 2021 +0100

    Silence gcc strict aliasing warnings where vtree.h is used
    
    Please relace the vtree.h #pragma change of this commit with a better
    solution once you have one.
    
    777e03ba9774bf635a972fa0561b50ff071908a9 had the disadvantage that it
    requires all vmods to adjust their build, besides affecting more code
    than necessary.
    
    Under the assumption that the issue with VRBT is not acutally a
    violation of strict aliasing, but rather only a violation of gcc's
    pedantic "compatible lvalue rule", we silence gcc warnings rather than
    turning off strict aliasing.
    
    Note that this is more convenient, but also more dangerous, because we
    fail to catch any actual strict aliasing errors where vtree.h is used.
    
    This reverts commit 777e03ba9774bf635a972fa0561b50ff071908a9.

diff --git a/bin/varnishd/Makefile.am b/bin/varnishd/Makefile.am
index 76a47ad2c..ce7145efd 100644
--- a/bin/varnishd/Makefile.am
+++ b/bin/varnishd/Makefile.am
@@ -159,7 +159,6 @@ nobase_pkginclude_HEADERS = \
 vcldir=$(datarootdir)/$(PACKAGE)/vcl
 
 varnishd_CFLAGS = \
-	@vtree_extra_cflags@ \
 	@PCRE_CFLAGS@ \
 	@SAN_CFLAGS@ \
 	-DNOT_IN_A_VMOD \
diff --git a/bin/varnishtop/Makefile.am b/bin/varnishtop/Makefile.am
index 0427e1140..cdb63b5f8 100644
--- a/bin/varnishtop/Makefile.am
+++ b/bin/varnishtop/Makefile.am
@@ -13,7 +13,6 @@ varnishtop_SOURCES = \
 
 
 varnishtop_CFLAGS = \
-	@vtree_extra_cflags@ \
 	@SAN_CFLAGS@
 
 varnishtop_LDADD = \
diff --git a/configure.ac b/configure.ac
index 9fcfb5693..59aa57b9e 100644
--- a/configure.ac
+++ b/configure.ac
@@ -291,8 +291,6 @@ CFLAGS="${save_CFLAGS}"
 if test "x$GCC" = "xyes"; then
 	libvgz_extra_cflags="${libvgz_extra_cflags} -Wno-unknown-warning-option -Wno-implicit-fallthrough"
 	AC_SUBST(libvgz_extra_cflags)
-	vtree_extra_cflags="-fno-strict-aliasing"
-	AC_SUBST(vtree_extra_cflags)
 fi
 
 SAN_CFLAGS=
diff --git a/include/vtree.h b/include/vtree.h
index c652c48d8..6fb2fdaae 100644
--- a/include/vtree.h
+++ b/include/vtree.h
@@ -32,6 +32,10 @@
 #ifndef	_VTREE_H_
 #define	_VTREE_H_
 
+/* XXX
+ * Enable -Wall with gcc -O2
+ */
+#pragma GCC diagnostic ignored "-Wstrict-aliasing"
 
 /*
  * This file defines data structures for different types of trees:
@@ -832,5 +836,4 @@ name##_VRBT_REINSERT(struct name *head, struct type *elm)			\
 	for ((x) = VRBT_MAX(name, head);					\
 	    ((x) != NULL) && ((y) = name##_VRBT_PREV(x), (x) != NULL);	\
 	     (x) = (y))
-
 #endif	/* _VTREE_H_ */
diff --git a/lib/libvarnishapi/Makefile.am b/lib/libvarnishapi/Makefile.am
index a2a314581..0cbff3c4d 100644
--- a/lib/libvarnishapi/Makefile.am
+++ b/lib/libvarnishapi/Makefile.am
@@ -37,7 +37,6 @@ endif
 
 libvarnishapi_la_CFLAGS = \
 	-DVARNISH_STATE_DIR='"${VARNISH_STATE_DIR}"' \
-	@vtree_extra_cflags@ \
 	@SAN_CFLAGS@
 
 libvarnishapi_la_LIBADD = \
diff --git a/lib/libvcc/Makefile.am b/lib/libvcc/Makefile.am
index 965e0b3f4..1e3d6f2d0 100644
--- a/lib/libvcc/Makefile.am
+++ b/lib/libvcc/Makefile.am
@@ -9,7 +9,6 @@ AM_CPPFLAGS = \
 noinst_LIBRARIES = libvcc.a
 
 libvcc_a_CFLAGS = \
-	@vtree_extra_cflags@ \
 	@SAN_CFLAGS@
 
 libvcc_a_SOURCES = \

From phk at FreeBSD.org  Mon Mar 29 08:27:06 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 29 Mar 2021 08:27:06 +0000 (UTC)
Subject: [master] d8041f13a include <stddef.h> to pick up __GNUC_PREREQ__ if
 possible.
Message-ID: <20210329082706.8ECD245FD@lists.varnish-cache.org>


commit d8041f13ac2255df2040af1ccc626555c39066e1
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 29 08:05:42 2021 +0000

    include <stddef.h> to pick up __GNUC_PREREQ__ if possible.

diff --git a/include/vdef.h b/include/vdef.h
index a9111feb6..5ba66828e 100644
--- a/include/vdef.h
+++ b/include/vdef.h
@@ -37,6 +37,8 @@
  * This file should always be the first non <...> include in a .c file.
  */
 
+#include <stddef.h>
+
 #ifdef VDEF_H_INCLUDED
 #  error "vdef.h included multiple times"
 #endif

From phk at FreeBSD.org  Mon Mar 29 08:27:06 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 29 Mar 2021 08:27:06 +0000 (UTC)
Subject: [master] d054e9acf Use a python-script to screen and document
 DEVELOPER_CFLAGS
Message-ID: <20210329082706.A1C464600@lists.varnish-cache.org>


commit d054e9acf99748697a2f7952d27054eb34035cbe
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 29 08:25:28 2021 +0000

    Use a python-script to screen and document DEVELOPER_CFLAGS

diff --git a/Makefile.am b/Makefile.am
index 843b67f09..c41cddb94 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -23,7 +23,8 @@ EXTRA_DIST = \
 	varnishapi.pc.in \
 	varnish.m4 \
 	varnish-legacy.m4 \
-	vtc.am
+	vtc.am \
+	wflags.py
 
 AM_DISTCHECK_CONFIGURE_FLAGS = \
     --enable-developer-warnings \
diff --git a/configure.ac b/configure.ac
index 59aa57b9e..8bad83f6b 100644
--- a/configure.ac
+++ b/configure.ac
@@ -288,11 +288,6 @@ if test "$ac_cv_have_viz" = no; then
 fi
 CFLAGS="${save_CFLAGS}"
 
-if test "x$GCC" = "xyes"; then
-	libvgz_extra_cflags="${libvgz_extra_cflags} -Wno-unknown-warning-option -Wno-implicit-fallthrough"
-	AC_SUBST(libvgz_extra_cflags)
-fi
-
 SAN_CFLAGS=
 SAN_LDFLAGS=
 UBSAN_CFLAGS=
@@ -723,31 +718,7 @@ AX_CHECK_COMPILE_FLAG([-Werror=unused-result],
       OCFLAGS="${OCFLAGS} -Wno-unused-result"])])
 
 # This corresponds to FreeBSD's WARNS level 6
-DEVELOPER_CFLAGS=" \
- -Werror \
- -Wno-format-y2k \
- -Wstrict-prototypes \
- -Wmissing-prototypes \
- -Wpointer-arith \
- -Wcast-qual \
- -Wwrite-strings \
- -Wshadow \
- -Wunused-parameter \
- -Wcast-align \
- -Wchar-subscripts \
- -Wnested-externs \
- -Wextra \
- -Wno-sign-compare \
-"
-
-# These are not compliable yet
-DEVELOPER_GCC_CFLAGS="-Wold-style-definition -Wredundant-decls "
-#DEVELOPER_CFLAGS="${DEVELOPER_CFLAGS} ${DEVELOPER_GCC_CFLAGS}"
-
-# These are compilable
-DEVELOPER_CLANG_CFLAGS="-Wmissing-variable-declarations -Wno-string-plus-int"
-# -Wno-empty-body
-#DEVELOPER_CFLAGS="${DEVELOPER_CFLAGS} ${DEVELOPER_CLANG_CFLAGS}"
+DEVELOPER_CFLAGS=`python3 wflags.py`
 
 # --enable-stack-protector
 AC_ARG_ENABLE(stack-protector,
diff --git a/wflags.py b/wflags.py
new file mode 100644
index 000000000..1141f05b0
--- /dev/null
+++ b/wflags.py
@@ -0,0 +1,115 @@
+#!/usr/bin/env python3
+#
+# Copyright (c) 2021 Varnish Software AS
+# All rights reserved.
+#
+# Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+#
+# SPDX-License-Identifier: BSD-2-Clause
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions
+# are met:
+# 1. Redistributions of source code must retain the above copyright
+#    notice, this list of conditions and the following disclaimer.
+# 2. Redistributions in binary form must reproduce the above copyright
+#    notice, this list of conditions and the following disclaimer in the
+#    documentation and/or other materials provided with the distribution.
+#
+# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+# ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+# SUCH DAMAGE.
+#
+
+''' Figure out what developer-warnings CC can grok '''
+
+import os
+import sys
+import subprocess
+import tempfile
+
+DESIRABLE_OPTIONS = [
+    "-fstack-protector-strong",
+    "-Werror",
+    "-Wall",
+]
+
+DESIRABLE_WFLAGS = [
+    "-Wcast-align",
+    "-Wcast-qual",
+    "-Wchar-subscripts",
+    "-Wempty-body",
+    "-Wextra",
+    "-Wformat-y2k",
+    "-Wformat-zero-length",
+    "-Wmissing-field-initializers",
+    "-Wmissing-prototypes",
+    "-Wmissing-variable-declarations",
+    "-Wnested-externs",
+    "-Wpointer-arith",
+    "-Wpointer-sign",
+    "-Wredundant-decls",
+    "-Wreturn-type",
+    "-Wshadow",
+    "-Wstrict-aliasing",
+    "-Wstrict-prototypes",
+    "-Wstring-plus-int",
+    "-Wswitch",
+    "-Wsystem-headers",
+    "-Wunused-parameter",
+    "-Wunused-parameters",
+    "-Wunused-result",
+    "-Wwrite-strings",
+]
+
+UNDESIRABLE_WFLAGS = [
+    "-Wno-thread-safety", # Does not understand our mutexs are wrapped
+    "-Wno-old-style-definition", # Does not like vgz
+    "-Wno-sign-compare", # Fixable
+    "-Wno-implicit-fallthrough", # Probably Fixable
+]
+
+def main():
+    compiler = os.environ.get("CC", "cc")
+
+    src_file = tempfile.NamedTemporaryFile(suffix='.c')
+    src_file.write(b'int main(int argc, char **argv) {(void)argc;(void)argv;return(0);}\n')
+    src_file.flush()
+    obj_file = tempfile.NamedTemporaryFile(suffix='.o')
+
+    use_flags = []
+    for i in DESIRABLE_OPTIONS + DESIRABLE_WFLAGS + UNDESIRABLE_WFLAGS:
+        j = subprocess.run(
+            [
+                compiler,
+                "-c",
+                i,
+                "-o", obj_file.name,
+                src_file.name,
+            ],
+            capture_output=True,
+        )
+        if not j.returncode and not j.stdout and not j.stderr:
+            use_flags.append(i)
+        else:
+            sys.stderr.write(compiler + " cannot " + i + '\n')
+            if b'error: unrecognized command line option' in j.stderr:
+                # LLVM
+                pass
+            elif b'warning: unknown warning option' in j.stderr:
+                # GCC
+                pass
+            else:
+                sys.stderr.write("\n\t" + j.stderr.decode('utf8') + '\n')
+    print(" ".join(use_flags))
+
+if __name__ == "__main__":
+    main()

From phk at FreeBSD.org  Mon Mar 29 08:43:04 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 29 Mar 2021 08:43:04 +0000 (UTC)
Subject: [master] 23b154c61 Be compatible with older python3 versions
Message-ID: <20210329084304.9830551AB@lists.varnish-cache.org>


commit 23b154c61c9c24b3c10a171e536932a7c12bc8d2
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 29 08:42:18 2021 +0000

    Be compatible with older python3 versions

diff --git a/wflags.py b/wflags.py
index 1141f05b0..6e51bc44b 100644
--- a/wflags.py
+++ b/wflags.py
@@ -95,7 +95,8 @@ def main():
                 "-o", obj_file.name,
                 src_file.name,
             ],
-            capture_output=True,
+            stderr=subprocess.PIPE,
+            stdout=subprocess.PIPE,
         )
         if not j.returncode and not j.stdout and not j.stderr:
             use_flags.append(i)

From phk at FreeBSD.org  Mon Mar 29 08:47:04 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 29 Mar 2021 08:47:04 +0000 (UTC)
Subject: [master] 128b02a92 Two -Wno-* which complain about Linux include files
Message-ID: <20210329084704.A91F254AF@lists.varnish-cache.org>


commit 128b02a927264f4c248ac7c64f67b15bcefd29e2
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 29 08:45:49 2021 +0000

    Two -Wno-* which complain about Linux include files

diff --git a/wflags.py b/wflags.py
index 6e51bc44b..4aa80275d 100644
--- a/wflags.py
+++ b/wflags.py
@@ -56,7 +56,6 @@ DESIRABLE_WFLAGS = [
     "-Wnested-externs",
     "-Wpointer-arith",
     "-Wpointer-sign",
-    "-Wredundant-decls",
     "-Wreturn-type",
     "-Wshadow",
     "-Wstrict-aliasing",
@@ -75,6 +74,8 @@ UNDESIRABLE_WFLAGS = [
     "-Wno-old-style-definition", # Does not like vgz
     "-Wno-sign-compare", # Fixable
     "-Wno-implicit-fallthrough", # Probably Fixable
+    "-Wno-builtin-requires-header", # Complains about linux::pthread.h
+    "-Wno-redundant-decls", # Complains about centos::stdio.h
 ]
 
 def main():

From phk at FreeBSD.org  Mon Mar 29 09:04:04 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 29 Mar 2021 09:04:04 +0000 (UTC)
Subject: [master] 9a45810d0 More -Wno, but now with explanation
Message-ID: <20210329090404.DEF8E5E56@lists.varnish-cache.org>


commit 9a45810d0f7f437e55726de4c7cd0c65f3493bcf
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 29 09:02:54 2021 +0000

    More -Wno, but now with explanation

diff --git a/wflags.py b/wflags.py
index 4aa80275d..bd9d27e5f 100644
--- a/wflags.py
+++ b/wflags.py
@@ -52,7 +52,6 @@ DESIRABLE_WFLAGS = [
     "-Wformat-zero-length",
     "-Wmissing-field-initializers",
     "-Wmissing-prototypes",
-    "-Wmissing-variable-declarations",
     "-Wnested-externs",
     "-Wpointer-arith",
     "-Wpointer-sign",
@@ -76,6 +75,7 @@ UNDESIRABLE_WFLAGS = [
     "-Wno-implicit-fallthrough", # Probably Fixable
     "-Wno-builtin-requires-header", # Complains about linux::pthread.h
     "-Wno-redundant-decls", # Complains about centos::stdio.h
+    "-Wno-missing-variable-declarations", # Complains about optreset
 ]
 
 def main():

From phk at FreeBSD.org  Mon Mar 29 09:19:04 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 29 Mar 2021 09:19:04 +0000 (UTC)
Subject: [master] bd0f3d277 Remove two unused includes. (Spotted by FlexeLint)
Message-ID: <20210329091904.CF05266D2@lists.varnish-cache.org>


commit bd0f3d2777db7f24b33178654560dbf09609bbd1
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 29 09:18:26 2021 +0000

    Remove two unused includes. (Spotted by FlexeLint)

diff --git a/lib/libvarnish/venc.c b/lib/libvarnish/venc.c
index 61916affa..1d9dd2d98 100644
--- a/lib/libvarnish/venc.c
+++ b/lib/libvarnish/venc.c
@@ -33,7 +33,6 @@
 
 #include <stdint.h>
 #include <string.h>
-#include <limits.h>
 #include <sys/types.h>
 
 #include "vdef.h"
diff --git a/lib/libvarnish/vsa.c b/lib/libvarnish/vsa.c
index 66213be76..613555e3f 100644
--- a/lib/libvarnish/vsa.c
+++ b/lib/libvarnish/vsa.c
@@ -40,7 +40,6 @@
 #include <sys/socket.h>
 #include <sys/types.h>
 #include <netinet/in.h>
-#include <netdb.h>
 
 #include "vdef.h"
 #include "vas.h"

From phk at FreeBSD.org  Mon Mar 29 09:22:04 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 29 Mar 2021 09:22:04 +0000 (UTC)
Subject: [master] 5dd19c0db Eliminate a memory leak in case of parameter
 errors. (Coverity)
Message-ID: <20210329092204.B83716A39@lists.varnish-cache.org>


commit 5dd19c0dbeffc3c82e0963f31e733b061bb4be0c
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 29 09:21:23 2021 +0000

    Eliminate a memory leak in case of parameter errors. (Coverity)

diff --git a/vmod/vmod_debug_acl.c b/vmod/vmod_debug_acl.c
index cb5fd3424..bff2a0ce2 100644
--- a/vmod/vmod_debug_acl.c
+++ b/vmod/vmod_debug_acl.c
@@ -162,9 +162,6 @@ xyzzy_sweep_acl(VRT_CTX, VCL_ACL acl, VCL_IP ip0, VCL_IP ip1, VCL_INT step)
 	struct vrt_blob *b;
 	ssize_t sz;
 
-	vsb = VSB_new_auto();
-	AN(vsb);
-
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
 	AN(acl);
 	AN(ip0);
@@ -172,6 +169,10 @@ xyzzy_sweep_acl(VRT_CTX, VCL_ACL acl, VCL_IP ip0, VCL_IP ip1, VCL_INT step)
 	assert(step > 0);
 	if (setup_sweep(ctx, asw, ip0, ip1, step))
 		return(NULL);
+
+	vsb = VSB_new_auto();
+	AN(vsb);
+
 	VSHA256_Init(vsha);
 	for (j = 0; ; j++) {
 		if ((j & 0x3f) == 0x00) {

From dridi.boukelmoune at gmail.com  Mon Mar 29 13:54:04 2021
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 29 Mar 2021 13:54:04 +0000 (UTC)
Subject: [master] af7ef4718 condfetch: Handle failed stale_oc without a boc
Message-ID: <20210329135404.E85B065CB2@lists.varnish-cache.org>


commit af7ef47182ac26e9b4f9a7b1fffd7e166fd9e1a3
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Mon Mar 22 12:16:39 2021 +0100

    condfetch: Handle failed stale_oc without a boc
    
    The assertion that the stale objcore of a conditional fetch cannot be
    failed unless it was streaming is incorrect. Between the moment when
    we grab the stale objcore in HSH_Lookup and the moment we try to use
    it after vcl_backend_response, the backend fetch may have completed or
    failed.
    
    Instead, we need to treat an ongoing fetch and a failed fetch as
    separate checks since the latter may happen with or without a boc.

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index f68ae7fa2..4bb321b78 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -800,14 +800,18 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo)
 		HSH_DerefBoc(bo->wrk, stale_oc);
 		stale_boc = NULL;
 		if (stale_state != BOS_FINISHED) {
-			(void)VFP_Error(bo->vfc, "Template object failed");
-			vbf_cleanup(bo);
-			wrk->stats->fetch_failed++;
-			return (F_STP_FAIL);
+			assert(stale_state == BOS_FAILED);
+			AN(stale_oc->flags & OC_F_FAILED);
 		}
 	}
+
 	AZ(stale_boc);
-	AZ(stale_oc->flags & OC_F_FAILED);
+	if (stale_oc->flags & OC_F_FAILED) {
+		(void)VFP_Error(bo->vfc, "Template object failed");
+		vbf_cleanup(bo);
+		wrk->stats->fetch_failed++;
+		return (F_STP_FAIL);
+	}
 
 	if (vbf_beresp2obj(bo)) {
 		vbf_cleanup(bo);
diff --git a/bin/varnishtest/tests/c00105.vtc b/bin/varnishtest/tests/c00105.vtc
new file mode 100644
index 000000000..a5588b550
--- /dev/null
+++ b/bin/varnishtest/tests/c00105.vtc
@@ -0,0 +1,60 @@
+varnishtest "Failed post-streaming revalidation"
+
+barrier b1 cond 2
+barrier b2 sock 2
+barrier b3 sock 2
+
+server s1 {
+	rxreq
+	txresp -nolen -hdr {Etag: "abc"} -hdr "Content-Length: 100"
+	barrier b1 sync
+	barrier b2 sync
+} -start
+
+server s2 {
+	rxreq
+	expect req.http.If-None-Match == {"abc"}
+	txresp -status 304 -nolen -hdr {Etag: "abc"} -hdr "Content-Length: 100"
+} -start
+
+varnish v1 -vcl+backend {
+	import directors;
+	import vtc;
+
+	sub vcl_recv {
+		if (req.http.backend == "s2") {
+			set req.backend_hint = s2;
+		}
+	}
+
+	sub vcl_backend_response {
+		if (beresp.was_304) {
+			vtc.barrier_sync("${b2_sock}");
+			vtc.barrier_sync("${b3_sock}");
+		}
+		set beresp.ttl = 1ms;
+	}
+} -start
+
+client c1 {
+	txreq -hdr "backend: s1"
+	rxresphdrs
+	expect resp.status == 200
+	expect_close
+} -start
+
+barrier b1 sync
+
+# ensure stale_oc
+delay 0.01
+
+client c2 {
+	txreq -hdr "backend: s2"
+	rxresphdrs
+	expect resp.status == 200
+	expect_close
+} -start
+
+client c1 -wait
+barrier b3 sync
+client c2 -wait

From nils.goroll at uplex.de  Mon Mar 29 13:56:05 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon, 29 Mar 2021 13:56:05 +0000 (UTC)
Subject: [master] 1df203a75 When streaming,
 send headers if no body data available yet
Message-ID: <20210329135605.2E45265FF6@lists.varnish-cache.org>


commit 1df203a7574b4c6f6853f94c1e80da70a7af7849
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Mar 1 19:09:29 2021 +0100

    When streaming, send headers if no body data available yet
    
    This is a TTFB optimization: I noticed that, when streaming, we would
    only sometimes send headers while waiting for more body data from the
    backend, depending on the amount of body data read ahead when reading
    the backend headers.
    
    The flush_head debug bit would avoid the delay, but it leads to an extra
    flush in all cases, even when we could send some body data down the line
    with the headers.
    
    This trivial patch peeks into the busy object to determine if the
    first ObjWaitExtend() is likely to block and, if so, issues a flush,
    which will cause the headers to be sent in addition to any initial data
    VDPs might have to send.

diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index 314da53fe..62d03f638 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -280,6 +280,8 @@ sml_iterator(struct worker *wrk, struct objcore *oc,
 	l = 0;
 
 	u = 0;
+	if (boc->len_so_far == 0)
+		func(priv, OBJ_ITER_FLUSH, NULL, 0);
 	while (1) {
 		ol = len;
 		nl = ObjWaitExtend(wrk, oc, ol);

From nils.goroll at uplex.de  Mon Mar 29 14:50:05 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon, 29 Mar 2021 14:50:05 +0000 (UTC)
Subject: [master] ef1073f30 flexelint/polish
 1df203a7574b4c6f6853f94c1e80da70a7af7849
Message-ID: <20210329145005.DC61491ADC@lists.varnish-cache.org>


commit ef1073f309bc5ad324e8225d50974f2f8d6b3a4f
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Mar 29 16:49:08 2021 +0200

    flexelint/polish 1df203a7574b4c6f6853f94c1e80da70a7af7849

diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index 62d03f638..1b94bb37e 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -280,8 +280,11 @@ sml_iterator(struct worker *wrk, struct objcore *oc,
 	l = 0;
 
 	u = 0;
-	if (boc->len_so_far == 0)
-		func(priv, OBJ_ITER_FLUSH, NULL, 0);
+	if (boc->len_so_far == 0) {
+		ret = func(priv, OBJ_ITER_FLUSH, NULL, 0);
+		if (ret)
+			return (ret);
+	}
 	while (1) {
 		ol = len;
 		nl = ObjWaitExtend(wrk, oc, ol);

From phk at FreeBSD.org  Mon Mar 29 20:06:12 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 29 Mar 2021 20:06:12 +0000 (UTC)
Subject: [master] cd86bd0ce Rename roundup2() macro to rndup2() to not collide
 with what sys/param.h might have defined.
Message-ID: <20210329200612.A112CA2393@lists.varnish-cache.org>


commit cd86bd0ce46d07b40bf767f6c3e64aaeb554b840
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 29 19:36:57 2021 +0000

    Rename roundup2() macro to rndup2() to not collide with what sys/param.h might have defined.

diff --git a/lib/libvarnish/vsb.c b/lib/libvarnish/vsb.c
index 008ad2ce0..e868cade0 100644
--- a/lib/libvarnish/vsb.c
+++ b/lib/libvarnish/vsb.c
@@ -47,7 +47,7 @@ __FBSDID("$FreeBSD: head/sys/kern/subr_vsb.c 222004 2011-05-17 06:36:32Z phk $")
 #define	SBMALLOC(size)		malloc(size)
 #define	SBFREE(buf)		free(buf)
 
-#define	roundup2(x, y)	(((x)+((y)-1))&(~((y)-1))) /* if y is powers of two */
+#define	rndup2(x, y)	(((x)+((y)-1))&(~((y)-1))) /* if y is powers of two */
 
 /*
  * Predicates
@@ -127,7 +127,7 @@ VSB_extendsize(ssize_t size)
 		while (newsize < size)
 			newsize *= 2;
 	} else {
-		newsize = roundup2(size, VSB_MAXEXTENDINCR);
+		newsize = rndup2(size, VSB_MAXEXTENDINCR);
 	}
 	KASSERT(newsize >= size, ("%s: %d < %d\n", __func__, newsize, size));
 	return (newsize);

From phk at FreeBSD.org  Mon Mar 29 20:06:12 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 29 Mar 2021 20:06:12 +0000 (UTC)
Subject: [master] 2dbcb4d45 Update the comment in this test to reflect we no
 longer have these debug VSL.
Message-ID: <20210329200612.BC78BA2396@lists.varnish-cache.org>


commit 2dbcb4d4546e985d48dd7b0a522ead20d9c936df
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 29 19:48:57 2021 +0000

    Update the comment in this test to reflect we no longer have these debug VSL.
    
    Also compare in the exact resp.body, because now we can.

diff --git a/bin/varnishtest/tests/e00007.vtc b/bin/varnishtest/tests/e00007.vtc
index 038f45f47..6e4e0e9df 100644
--- a/bin/varnishtest/tests/e00007.vtc
+++ b/bin/varnishtest/tests/e00007.vtc
@@ -1,17 +1,18 @@
 varnishtest "ESI spanning storage bits"
 
-# NB!  The layout of the body in the response is very carefully
-# NB!  tuned to give the desired code coverage.
-# NB!  It should look like this in the varnishlog:
-# NB!    7 Debug        c "Fetch 32 byte segments:"
-# NB!    7 Debug        c "%0a%09%09<html>filler%0a%09%09This is before"
-# NB!    7 Debug        c " the test%0a%09%09<![CDATA[foooooooooo"
-# NB!    7 Debug        c "ooooooooobar]]>%0a%09%09filler<esi:rem"
-# NB!    7 Debug        c "ove>%0a%09%09This is a test: Unseen Un"
-# NB!    7 Debug        c "iversity%0a%09%09Department of cruel a"
-# NB!    7 Debug        c "nd unjust geography%0a%09%09</esi:remo"
-# NB!    7 Debug        c "ve>%0a%09%09This is a test: Hello worl"
-# NB!    7 Debug        c "d%0a%09"
+# The layout of the body in the response is very carefully
+# tuned to give the desired code coverage.
+# The objects segments should have the following *precise* content
+#
+#     "Fetch 32 byte segments:"
+#     "%0a%09%09<html>filler%0a%09%09This is before"
+#     " the test%0a%09%09<![CDATA[foooooooooo"
+#     "ooooooooobar]]>%0a%09%09filler<esi:rem"
+#     "ove>%0a%09%09This is a test: Unseen Un"
+#     "iversity%0a%09%09Department of cruel a"
+#     "nd unjust geography%0a%09%09</esi:remo"
+#     "ve>%0a%09%09This is a test: Hello worl"
+#     "d%0a%09"
 
 server s1 {
 	rxreq
@@ -44,6 +45,7 @@ client c1 {
 	rxresp
 	expect resp.status == 200
 	expect resp.bodylen == 120
+        expect resp.body == "\n\t\t<html>filler\n\t\tThis is before the test\n\t\t<![CDATA[fooooooooooooooooooobar]]>\n\t\tfiller\n\t\tThis is a test: Hello world\n\t"
 }
 
 client c1 -run

From phk at FreeBSD.org  Mon Mar 29 20:06:12 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 29 Mar 2021 20:06:12 +0000 (UTC)
Subject: [master] 3fedecadc Reverse the order of storage segments,
 to avoid a race.
Message-ID: <20210329200612.D8394A239A@lists.varnish-cache.org>


commit 3fedecadc83d16cfc4a7ac4097c39bd3e9607a2f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Mar 29 20:00:42 2021 +0000

    Reverse the order of storage segments, to avoid a race.
    
    See Also: #3557

diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c
index 1b94bb37e..70d8d1280 100644
--- a/bin/varnishd/storage/storage_simple.c
+++ b/bin/varnishd/storage/storage_simple.c
@@ -259,9 +259,11 @@ sml_iterator(struct worker *wrk, struct objcore *oc,
 	boc = HSH_RefBoc(oc);
 
 	if (boc == NULL) {
-		VTAILQ_FOREACH_SAFE(st, &obj->list, list, checkpoint) {
+		VTAILQ_FOREACH_REVERSE_SAFE(
+		    st, &obj->list, storagehead, list, checkpoint) {
+
 			u = 0;
-			if (VTAILQ_NEXT(st, list) == NULL)
+			if (VTAILQ_PREV(st, storagehead, list) == NULL)
 				u |= OBJ_ITER_END;
 			if (final)
 				u |= OBJ_ITER_FLUSH;
@@ -300,7 +302,7 @@ sml_iterator(struct worker *wrk, struct objcore *oc,
 		Lck_Lock(&boc->mtx);
 		AZ(VTAILQ_EMPTY(&obj->list));
 		if (checkpoint == NULL) {
-			st = VTAILQ_FIRST(&obj->list);
+			st = VTAILQ_LAST(&obj->list, storagehead);
 			sl = 0;
 		} else {
 			st = checkpoint;
@@ -319,8 +321,8 @@ sml_iterator(struct worker *wrk, struct objcore *oc,
 			nl -= st->len;
 			assert(nl > 0);
 			sl += st->len;
-			st = VTAILQ_NEXT(st, list);
-			if (VTAILQ_NEXT(st, list) != NULL) {
+			st = VTAILQ_PREV(st, storagehead, list);
+			if (VTAILQ_PREV(st, storagehead, list) != NULL) {
 				if (final && checkpoint != NULL) {
 					VTAILQ_REMOVE(&obj->list,
 					    checkpoint, list);
@@ -332,7 +334,7 @@ sml_iterator(struct worker *wrk, struct objcore *oc,
 		}
 		CHECK_OBJ_NOTNULL(obj, OBJECT_MAGIC);
 		CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC);
-		st = VTAILQ_NEXT(st, list);
+		st = VTAILQ_PREV(st, storagehead, list);
 		if (st != NULL && st->len == 0)
 			st = NULL;
 		state = boc->state;
@@ -407,7 +409,7 @@ sml_getspace(struct worker *wrk, struct objcore *oc, ssize_t *sz,
 	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
 	CHECK_OBJ_NOTNULL(oc->boc, BOC_MAGIC);
 
-	st = VTAILQ_LAST(&o->list, storagehead);
+	st = VTAILQ_FIRST(&o->list);
 	if (st != NULL && st->len < st->space) {
 		*sz = st->space - st->len;
 		*ptr = st->ptr + st->len;
@@ -422,7 +424,7 @@ sml_getspace(struct worker *wrk, struct objcore *oc, ssize_t *sz,
 
 	CHECK_OBJ_NOTNULL(oc->boc, BOC_MAGIC);
 	Lck_Lock(&oc->boc->mtx);
-	VTAILQ_INSERT_TAIL(&o->list, st, list);
+	VTAILQ_INSERT_HEAD(&o->list, st, list);
 	Lck_Unlock(&oc->boc->mtx);
 
 	*sz = st->space - st->len;
@@ -442,7 +444,7 @@ sml_extend(struct worker *wrk, struct objcore *oc, ssize_t l)
 
 	o = sml_getobj(wrk, oc);
 	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-	st = VTAILQ_LAST(&o->list, storagehead);
+	st = VTAILQ_FIRST(&o->list);
 	CHECK_OBJ_NOTNULL(st, STORAGE_MAGIC);
 	assert(st->len + l <= st->space);
 	st->len += l;
@@ -467,7 +469,7 @@ sml_trimstore(struct worker *wrk, struct objcore *oc)
 
 	o = sml_getobj(wrk, oc);
 	CHECK_OBJ_NOTNULL(o, OBJECT_MAGIC);
-	st = VTAILQ_LAST(&o->list, storagehead);
+	st = VTAILQ_FIRST(&o->list);
 
 	if (st == NULL)
 		return;
@@ -492,7 +494,7 @@ sml_trimstore(struct worker *wrk, struct objcore *oc)
 	st1->len = st->len;
 	Lck_Lock(&oc->boc->mtx);
 	VTAILQ_REMOVE(&o->list, st, list);
-	VTAILQ_INSERT_TAIL(&o->list, st1, list);
+	VTAILQ_INSERT_HEAD(&o->list, st1, list);
 	Lck_Unlock(&oc->boc->mtx);
 	/* sml_bocdone frees this */
 	AZ(oc->boc->stevedore_priv);

From dridi.boukelmoune at gmail.com  Tue Mar 30 11:58:06 2021
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 30 Mar 2021 11:58:06 +0000 (UTC)
Subject: [master] 1af227233 build: Disable -Wincomplete-setjmp-declaration
Message-ID: <20210330115806.A3227643A0@lists.varnish-cache.org>


commit 1af2272331216a51e2e4b83e08378e2e3c0caecd
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Mar 30 13:45:13 2021 +0200

    build: Disable -Wincomplete-setjmp-declaration

diff --git a/wflags.py b/wflags.py
index bd9d27e5f..d7b0e6060 100644
--- a/wflags.py
+++ b/wflags.py
@@ -74,6 +74,7 @@ UNDESIRABLE_WFLAGS = [
     "-Wno-sign-compare", # Fixable
     "-Wno-implicit-fallthrough", # Probably Fixable
     "-Wno-builtin-requires-header", # Complains about linux::pthread.h
+    "-Wno-incomplete-setjmp-declaration", # Clang complains about glibc pthread.h
     "-Wno-redundant-decls", # Complains about centos::stdio.h
     "-Wno-missing-variable-declarations", # Complains about optreset
 ]

From dridi.boukelmoune at gmail.com  Tue Mar 30 11:58:06 2021
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 30 Mar 2021 11:58:06 +0000 (UTC)
Subject: [master] 21d774fa7 build: Declare wflags.py as a configure dependency
Message-ID: <20210330115806.B0AA7643A2@lists.varnish-cache.org>


commit 21d774fa7054b0cd7405d920dc2485e716d7f444
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Mar 30 13:53:53 2021 +0200

    build: Declare wflags.py as a configure dependency

diff --git a/Makefile.am b/Makefile.am
index c41cddb94..1c3bbfff9 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -26,6 +26,8 @@ EXTRA_DIST = \
 	vtc.am \
 	wflags.py
 
+CONFIGURE_DEPENDENCIES = wflags.py
+
 AM_DISTCHECK_CONFIGURE_FLAGS = \
     --enable-developer-warnings \
     --enable-debugging-symbols \

From dridi.boukelmoune at gmail.com  Tue Mar 30 11:58:06 2021
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 30 Mar 2021 11:58:06 +0000 (UTC)
Subject: [master] b440f850b build: Invoke wflags.py as configured
Message-ID: <20210330115806.CBD11643A7@lists.varnish-cache.org>


commit b440f850be34604ccb2e70b59f529aee45e98b7a
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Mar 30 13:55:22 2021 +0200

    build: Invoke wflags.py as configured

diff --git a/configure.ac b/configure.ac
index 8bad83f6b..de24d8ceb 100644
--- a/configure.ac
+++ b/configure.ac
@@ -718,7 +718,7 @@ AX_CHECK_COMPILE_FLAG([-Werror=unused-result],
       OCFLAGS="${OCFLAGS} -Wno-unused-result"])])
 
 # This corresponds to FreeBSD's WARNS level 6
-DEVELOPER_CFLAGS=`python3 wflags.py`
+DEVELOPER_CFLAGS=`$PYTHON $srcdir/wflags.py`
 
 # --enable-stack-protector
 AC_ARG_ENABLE(stack-protector,

From nils.goroll at uplex.de  Tue Mar 30 14:55:24 2021
From: nils.goroll at uplex.de (Nils Goroll)
Date: Tue, 30 Mar 2021 16:55:24 +0200
Subject: [master] 777e03ba9 Disable strict aliasing with gcc where vtree.h
 is used
In-Reply-To: <CABoVN9B4905u9U5nBk3W1QU=B7qiCyc5hYi-gYvExX6JZHcG7Q@mail.gmail.com>
References: <20210324173804.D872D6EB8B@lists.varnish-cache.org>
 <CABoVN9B4905u9U5nBk3W1QU=B7qiCyc5hYi-gYvExX6JZHcG7Q@mail.gmail.com>
Message-ID: <0c97ecce-515d-b400-78a7-3669b350f23e@uplex.de>

late response to an email which I had overlooked:

On 24/03/2021 18:46, Dridi Boukelmoune wrote:
>>     (*(uintptr_t *)&(parent)->entry.rbe_parent) &= ~((uintptr_t)3);
> Is culprit code part of generated vtree.h functions? Or macros that we
> may call directly?

Yes, this is some of the problematic code that the VRBT macros expand to.

>>     https://www.gnu.org/software/automake/manual/html_node/Per_002dObject-Flags.html
> If this is only in generated functions, we may be able to annotate
> them with some kind of v_no_strict_aliasing_ attribute.

Good idea, we could try __attribute__ ((__may_alias__))

https://gcc.gnu.org/onlinedocs/gcc/Common-Type-Attributes.html#Common-Type-Attributes

I have not looked at how invasive this would be.

Nils
-- 

** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstra?e 32
22301 Hamburg

tel +49 40 28805731
mob +49 170 2723133
fax +49 40 42949753

xmpp://slink at jabber.int.uplex.de/

http://uplex.de/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-commit/attachments/20210330/86e0ba86/attachment.bin>

From phk at FreeBSD.org  Tue Mar 30 21:06:12 2021
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 30 Mar 2021 21:06:12 +0000 (UTC)
Subject: [master] ccd41ff82 Add a rant about getaddrinfo(3)
Message-ID: <20210330210612.5297CA5D9F@lists.varnish-cache.org>


commit ccd41ff823b8bff21fc6f8be4bcd0f3447aa3312
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Mar 30 21:05:07 2021 +0000

    Add a rant about getaddrinfo(3)

diff --git a/doc/sphinx/phk/index.rst b/doc/sphinx/phk/index.rst
index 2b358a031..cbe4ff314 100644
--- a/doc/sphinx/phk/index.rst
+++ b/doc/sphinx/phk/index.rst
@@ -13,6 +13,7 @@ You may or may not want to know what Poul-Henning thinks.
 .. toctree::
 	:maxdepth: 1
 
+	ip_address.rst
 	vdd19q3.rst
 	quic.rst
 	VSV00003.rst
diff --git a/doc/sphinx/phk/ip_address.rst b/doc/sphinx/phk/ip_address.rst
new file mode 100644
index 000000000..0ec02f1ca
--- /dev/null
+++ b/doc/sphinx/phk/ip_address.rst
@@ -0,0 +1,106 @@
+..
+	Copyright (c) 2021 Varnish Software AS
+	SPDX-License-Identifier: BSD-2-Clause
+	See LICENSE file for full text of license
+
+.. _phk_ip_address:
+
+======================================
+IP Addresses - A long expected problem
+======================================
+
+I'm old enough to remember `HOSTS.TXT` and the introduction of the DNS system.
+
+Those were the days when you got a class B network by sending a
+polite letter to California, getting a polite letter back, and then,
+some months later, when
+`RFC1166 INTERNET NUMBERS <https://tools.ietf.org/html/rfc1166>`_ 
+arrived with in semi-annual packet of printed RFCs,
+find out that letter had at typo and you had configured all of
+the European Parliaments 1200 computers on 136.172/16 instead of
+136.173/16.
+
+But things were not simpler, if anything they were far more complex,
+because TCP/IP was not, as today, the only protocol that mattered.
+
+In addition to TCP/IP, there were IBM's SNA, Digitals DecNet,
+ApolloRing, Banyan/VINES, Novell NetWare, X.21, X.25, X.75, and the
+whole CCITT-OSI-"Intelligent Network" telecom disaster that never
+got of the ground.
+
+This is why DNS packets have a `class` field which can be set to
+`Hesiod` or `CHAOS` in addition to `the Internet`:  The idea was
+that all the different protocols would get a number each, and we
+would have "The One Directory To Rule Them All".
+
+Largely because of this, a new and "protocol agnostic" lookup
+functions were designed: `getaddrinfo(3)` and `getnameinfo(3)`,
+but of course for IP they were supposed to be backwards compatible
+because there were *thousands* of users out there already.
+
+This is why `telnet 80.1440` tries to connect to `80.0.5.160`,
+why `ping 0x7f000001` becomes `127.0.0.1` and `0127.0.0.1`
+becomes `87.0.0.1`.
+
+If you read the manual page for `getaddrinfo(3)` you will find
+that it does not tell you that, it merely says it 
+`conforms to IEEE Std 1001`.
+
+But everybody knew what that was back in 1990, and nobody had firewalls
+anyway because Cheswick & Bellowins book
+`Firewalls and Internet Security <http://www.wilyhacker.com/>`_
+was not published until 1994, so no worries ?
+
+As is often the case with 'designed for the future' the `getaddrinfo(3)`
+API instantly fossilized, hit by a freeze-ray in the 'the Unix? wars'.
+
+This is why, when IPv4 numbers started to look like a finite resource,
+and the old A-, B- and C- class networks got dissolved into Classless
+Inter-Domain Routing or "CIDR" netmasks of any random size, getaddrinfo(3)
+did not grow to be able to translate "192.168.61/23" into something useful.
+
+I belive there were also some lilliputian dispute about the fact that
+`192.168.61` would return `192.168.0.61` to stay backwards compatible,
+whereas `192.168.61/23` would return `192.168.61.0 + 255.255.254.0`.
+
+Because of this, Varnish uses `getaddrinfo(3)` everywhere but one single
+place:  Parsing of ACL specifications in VCL.  First we have to use our
+own parser to check if it is a CIDR entry and if not we ask `getaddrinfo(3)`.
+
+The reason for this rant, is that somebody noticed that `ping
+0127.0.0.1` didn't go to `127.0.0.1` as they expected.
+
+That has just become CVE-2021-29418 and CVE-2021-28918 and will
+probably become a dozen more, once the CVE-trophy-hunters go to town.
+
+All IP number strings enter Varnish from trusted points, either
+as command line arguments (`-a`, `-b`, `-M` etc.), 
+in the VCL source (`backend`, `acl` etc.) or as PROXYv1 header
+strings from the TLS-stripper in front of Varnish.
+
+Of course, VCL allows you to do pretty much anything, including::
+
+    if (std.ip(req.http.trustme) ~ important_acl) {
+         ...
+    }
+
+If you do something like that, you may want to a) Consider the wisdom
+of trusting IP#'s from strangers and b) Think about this "critical
+netmask problem".
+
+Otherwise, I do not expect this new "critical netmask problem" to
+result in any source code changes in Varnish.
+
+If and when the various UNIX-oid operating systems, and the smoking
+remains of the "serious UNIX industry", (IEEE ?  The Austin Group
+?  The Open Group ?  Whatever they are called these days) get their
+act together, and renovate the `getaddrinfo(3)` API, Varnish will
+automatically pick that up an use it.
+
+Should they, in a flash of enlightenment, also make `getaddrinfo(3)`
+useful for parsing these newfangled CIDR adresses we got in 1993,
+I will be more than happy to ditch `vcc_acl_try_netnotation()` too.
+
+Until next time,
+
+Poul-Henning, 2021-03-30

From dridi.boukelmoune at gmail.com  Wed Mar 31 05:22:08 2021
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Wed, 31 Mar 2021 05:22:08 +0000 (UTC)
Subject: [master] 61f77f848 build: Ignore -Wparentheses
Message-ID: <20210331052208.49773B2FDC@lists.varnish-cache.org>


commit 61f77f848a952cd7a4194fbd871a893b5118f2d9
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed Mar 31 07:18:46 2021 +0200

    build: Ignore -Wparentheses
    
    Refs #3565

diff --git a/wflags.py b/wflags.py
index d7b0e6060..f5dedf7e6 100644
--- a/wflags.py
+++ b/wflags.py
@@ -74,9 +74,10 @@ UNDESIRABLE_WFLAGS = [
     "-Wno-sign-compare", # Fixable
     "-Wno-implicit-fallthrough", # Probably Fixable
     "-Wno-builtin-requires-header", # Complains about linux::pthread.h
-    "-Wno-incomplete-setjmp-declaration", # Clang complains about glibc pthread.h
+    "-Wno-incomplete-setjmp-declaration", # Clang complains about glibc::pthread.h
     "-Wno-redundant-decls", # Complains about centos::stdio.h
     "-Wno-missing-variable-declarations", # Complains about optreset
+    "-Wno-parentheses", # GCC complains about musl::endian.h
 ]
 
 def main():

From dridi.boukelmoune at gmail.com  Wed Mar 31 06:22:05 2021
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Wed, 31 Mar 2021 06:22:05 +0000 (UTC)
Subject: [master] 13981aec4 doc: Typo
Message-ID: <20210331062205.14CB14B3D@lists.varnish-cache.org>


commit 13981aec4d4937307c7b9a8fe9b3609f05511e80
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed Mar 31 08:20:57 2021 +0200

    doc: Typo

diff --git a/doc/sphinx/phk/ip_address.rst b/doc/sphinx/phk/ip_address.rst
index 0ec02f1ca..840177e09 100644
--- a/doc/sphinx/phk/ip_address.rst
+++ b/doc/sphinx/phk/ip_address.rst
@@ -26,7 +26,7 @@ because TCP/IP was not, as today, the only protocol that mattered.
 In addition to TCP/IP, there were IBM's SNA, Digitals DecNet,
 ApolloRing, Banyan/VINES, Novell NetWare, X.21, X.25, X.75, and the
 whole CCITT-OSI-"Intelligent Network" telecom disaster that never
-got of the ground.
+got off the ground.
 
 This is why DNS packets have a `class` field which can be set to
 `Hesiod` or `CHAOS` in addition to `the Internet`:  The idea was
@@ -59,7 +59,7 @@ and the old A-, B- and C- class networks got dissolved into Classless
 Inter-Domain Routing or "CIDR" netmasks of any random size, getaddrinfo(3)
 did not grow to be able to translate "192.168.61/23" into something useful.
 
-I belive there were also some lilliputian dispute about the fact that
+I believe there were also some lilliputian dispute about the fact that
 `192.168.61` would return `192.168.0.61` to stay backwards compatible,
 whereas `192.168.61/23` would return `192.168.61.0 + 255.255.254.0`.
 
@@ -95,7 +95,7 @@ If and when the various UNIX-oid operating systems, and the smoking
 remains of the "serious UNIX industry", (IEEE ?  The Austin Group
 ?  The Open Group ?  Whatever they are called these days) get their
 act together, and renovate the `getaddrinfo(3)` API, Varnish will
-automatically pick that up an use it.
+automatically pick that up and use it.
 
 Should they, in a flash of enlightenment, also make `getaddrinfo(3)`
 useful for parsing these newfangled CIDR adresses we got in 1993,