From dridi.boukelmoune at gmail.com  Sat May  2 15:38:08 2020
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Sat,  2 May 2020 15:38:08 +0000 (UTC)
Subject: [master] a47763575 Tolerate lock witness leaks
Message-ID: <20200502153808.CA4B6AA06E@lists.varnish-cache.org>


commit a47763575707d62a08fbf1979dcfb0758a586991
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Sat May 2 17:36:24 2020 +0200

    Tolerate lock witness leaks
    
    It's not a gadget for production use.

diff --git a/tools/lsan.suppr b/tools/lsan.suppr
index 4e9892357..8ed7048df 100644
--- a/tools/lsan.suppr
+++ b/tools/lsan.suppr
@@ -16,3 +16,5 @@ leak:mct_callback
 leak:vsmw_newcluster
 #
 leak:smp_thread
+#
+leak:Lck_Witness_Lock

From nils.goroll at uplex.de  Mon May  4 13:42:07 2020
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon,  4 May 2020 13:42:07 +0000 (UTC)
Subject: [master] 15527d5d1 typo
Message-ID: <20200504134207.82A0B95118@lists.varnish-cache.org>


commit 15527d5d1fbd41eb2fe121ecc1efbcedd8bd6184
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon May 4 15:40:57 2020 +0200

    typo

diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index 7a9e7060e..4a770b444 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -29,7 +29,7 @@
  * SUCH DAMAGE.
  *
  * This file contains the request-handling state engine, which is intended to
- * (over time) be(ome) protocol agnostic.
+ * (over time) be(come) protocol agnostic.
  * We already use this now with ESI:includes, which are for all relevant
  * purposes a different "protocol"
  *

From guillaume at varnish-software.com  Mon May  4 20:50:08 2020
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Mon,  4 May 2020 20:50:08 +0000 (UTC)
Subject: [master] 9074a6f54 switch packaging branch to master
Message-ID: <20200504205008.79236A536A@lists.varnish-cache.org>


commit 9074a6f54c2b7205e8c4b37113bfdfa757844587
Author: Guillaume Quintard <guillaume at varnish-software.com>
Date:   Mon May 4 13:36:47 2020 -0700

    switch packaging branch to master

diff --git a/.circleci/config.yml b/.circleci/config.yml
index 62b2155ac..26716acb8 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -6,7 +6,7 @@ parameters:
     default: "HEAD"
   pkg-commit:
     type: string
-    default: "weekly"
+    default: "master"
 jobs:
   dist:
     docker:

From dridi.boukelmoune at gmail.com  Tue May  5 10:26:07 2020
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue,  5 May 2020 10:26:07 +0000 (UTC)
Subject: [master] 6251d83e8 Typo
Message-ID: <20200505102607.4816179C0@lists.varnish-cache.org>


commit 6251d83e83ff037bf6c5d9c5e600989d8e12d2e1
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue May 5 12:21:14 2020 +0200

    Typo

diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c
index b122aa343..4a747f0dd 100644
--- a/bin/varnishtest/vtc_http.c
+++ b/bin/varnishtest/vtc_http.c
@@ -272,7 +272,7 @@ http_count_header(char * const *hh, const char *hdr)
  *         meant as strings OP is a matching operator, either == (exact match)
  *         or ~ (regex match).
  *
- *         varnishtet will first try to resolve STRING1 and STRING2 by looking
+ *         varnishtest will first try to resolve STRING1 and STRING2 by looking
  *         if they have special meanings, in which case, the resolved value is
  *         use for the test. Note that this value can be a string representing a
  *         number, allowing for tests such as::

From martin at varnish-software.com  Tue May  5 11:39:07 2020
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Tue,  5 May 2020 11:39:07 +0000 (UTC)
Subject: [master] ff8d29ae5 Apply the 3 least significant digit rule of status
 code on H/2
Message-ID: <20200505113907.16C6461614@lists.varnish-cache.org>


commit ff8d29ae5a7e3eaeb30a4a7bf21efc89eed9f93a
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Apr 24 16:24:54 2020 +0200

    Apply the 3 least significant digit rule of status code on H/2
    
    In VCL we allow status codes larger then 999, and do modulo 1000 when
    sending it to the client. But the code to apply the modulo was missing in
    H/2.
    
    This issue was uncovered and reported by @Francois-v3.
    
    Fixes first case of: #3301

diff --git a/bin/varnishd/http2/cache_http2_deliver.c b/bin/varnishd/http2/cache_http2_deliver.c
index 6996e879a..675e9bd29 100644
--- a/bin/varnishd/http2/cache_http2_deliver.c
+++ b/bin/varnishd/http2/cache_http2_deliver.c
@@ -232,7 +232,7 @@ h2_build_headers(struct vsb *resp, struct req *req)
 	uint8_t buf[6];
 	ssize_t sz, sz1;
 
-	l = h2_status(buf, req->resp->status);
+	l = h2_status(buf, req->resp->status % 1000);
 	VSB_bcat(resp, buf, l);
 
 	hp = req->resp;

From martin at varnish-software.com  Tue May  5 11:39:07 2020
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Tue,  5 May 2020 11:39:07 +0000 (UTC)
Subject: [master] aba2855cb Check for VRT_fail before setting handling on
 return() in VCL
Message-ID: <20200505113907.2E53361617@lists.varnish-cache.org>


commit aba2855cb947dd5aa9ca1a9680c330264b23925d
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Apr 24 16:28:35 2020 +0200

    Check for VRT_fail before setting handling on return() in VCL
    
    The compound statement doing return() in VCL may have failed for some
    reason, for example running out of workspace while putting together the
    reason field. The fail handling would be overwritten by the forced
    handling at the end of the generated code for a return statement.
    
    This patch adds a test for the failure condition before forcing the
    handling to the result of the return statement.

diff --git a/lib/libvcc/vcc_action.c b/lib/libvcc/vcc_action.c
index 8fa48b37c..40ad52060 100644
--- a/lib/libvcc/vcc_action.c
+++ b/lib/libvcc/vcc_action.c
@@ -366,6 +366,7 @@ vcc_act_return(struct vcc *tl, struct token *t, struct symbol *sym)
 		}
 	}
 	ERRCHK(tl);
+	Fb(tl, 1, "END_;\n");
 	if (hand == VCL_RET_FAIL)
 		Fb(tl, 1, "VRT_fail(ctx, \"Failed from VCL\");\n");
 	else

From martin at varnish-software.com  Tue May  5 11:39:07 2020
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Tue,  5 May 2020 11:39:07 +0000 (UTC)
Subject: [master] c7799008f Check for illegal status codes when executing
 return(synth())
Message-ID: <20200505113907.4B23D6161C@lists.varnish-cache.org>


commit c7799008fe12b3bbd9ee95988700a081867a5859
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Apr 24 16:31:45 2020 +0200

    Check for illegal status codes when executing return(synth())
    
    Some status codes are illegal and will cause VRT_fail() when executed as
    normal set instructions in VCL. But this test is bypassed when status is
    set as a side effect of a `return (synth(code))` statement.
    
    This patch applies the same rules as when executing a set-instruction to
    the return(synth()) handling.
    
    Fixes second part of: #3301

diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index b89efccf4..1d8bac59d 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -62,8 +62,22 @@ VRT_synth(VRT_CTX, VCL_INT code, VCL_STRING reason)
 
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
 	assert(ctx->req != NULL || ctx->bo != NULL);
-	if (code < 100 || code > 65535)
-		code = 503;
+	if (code < 0) {
+		VRT_fail(ctx, "return(synth()) status code (%jd) is negative",
+		    code);
+		return;
+	}
+	if (code > 65535) {
+		VRT_fail(ctx, "return(synth()) status code (%jd) > 65535",
+		    code);
+		return;
+	}
+	if ((code % 1000) < 100) {
+		VRT_fail(ctx,
+		    "illegal return(synth()) status code (%jd) (..0##)",
+		    code);
+		return;
+	}
 
 	if (ctx->req == NULL) {
 		CHECK_OBJ_NOTNULL(ctx->bo, BUSYOBJ_MAGIC);

From martin at varnish-software.com  Tue May  5 11:39:07 2020
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Tue,  5 May 2020 11:39:07 +0000 (UTC)
Subject: [master] 4c1b0345d Add a test case
Message-ID: <20200505113907.6507561620@lists.varnish-cache.org>


commit 4c1b0345d32a58a095768996ef03fab9aa878e5f
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Apr 24 16:34:46 2020 +0200

    Add a test case

diff --git a/bin/varnishtest/tests/r03301.vtc b/bin/varnishtest/tests/r03301.vtc
new file mode 100644
index 000000000..d39f31709
--- /dev/null
+++ b/bin/varnishtest/tests/r03301.vtc
@@ -0,0 +1,73 @@
+varnishtest "Issue 3301: Illegal error codes"
+
+varnish v1 -vcl {
+	backend default none;
+	sub vcl_recv {
+		if (req.url == "/test1") {
+			return (synth(1301));
+		}
+		if (req.url == "/test2") {
+			return (synth(1001));
+		}
+	}
+	sub vcl_backend_fetch {
+		if (bereq.url == "/test3") {
+			return (error(1302));
+		}
+		if (bereq.url == "/test4") {
+			return (error(1000));
+		}
+	}
+} -start
+
+varnish v1 -cliok "param.set feature +http2"
+
+client c1 {
+	stream 1 {
+		txreq -url /test1
+		rxresp
+		expect resp.status == 301
+	} -run
+
+	stream 3 {
+		txreq -url /test2
+		rxresp
+		expect resp.status == 503
+	} -run
+
+	stream 5 {
+		txreq -url /test3
+		rxresp
+		expect resp.status == 302
+	} -run
+
+	stream 7 {
+		txreq -url /test4
+		rxresp
+		expect resp.status == 503
+	} -run
+} -run
+
+client c2 {
+	txreq -url /test1
+	rxresp
+	expect resp.status == 301
+	expect resp.reason == "Moved Permanently"
+
+	txreq -url /test2
+	rxresp
+	expect resp.status == 503
+} -run
+
+client c2 {
+	txreq -url /test3
+	rxresp
+	expect resp.status == 302
+	expect resp.reason == "Found"
+} -run
+
+client c2 {
+	txreq -url /test4
+	rxresp
+	expect resp.status == 503
+} -run

From martin at varnish-software.com  Tue May  5 11:39:07 2020
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Tue,  5 May 2020 11:39:07 +0000 (UTC)
Subject: [master] 9494205f7 Document special meaning of status code also for
 resp.status
Message-ID: <20200505113907.8025D61624@lists.varnish-cache.org>


commit 9494205f7379b598fbbefdb6fe853e7dcce7bba6
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Tue Apr 28 13:03:48 2020 +0200

    Document special meaning of status code also for resp.status
    
    Copy paste the text from beresp.status to resp.status about the special
    handling of status codes.

diff --git a/doc/sphinx/reference/vcl_var.rst b/doc/sphinx/reference/vcl_var.rst
index 306f82eee..300ea0a22 100644
--- a/doc/sphinx/reference/vcl_var.rst
+++ b/doc/sphinx/reference/vcl_var.rst
@@ -1186,6 +1186,10 @@ resp.status
 	Assigning a HTTP standardized code to resp.status will also
 	set resp.reason to the corresponding status message.
 
+	Status codes on the form XXYZZ can be set where
+	XXYZZ is less than 65536 and Y is [1...9].
+	Only YZZ will be sent back to clients.
+
 	resp.status 200 will get changed into 304 by core code after
 	a return(deliver) from vcl_deliver for conditional requests
 	to cached content if validation succeeds.

From martin at varnish-software.com  Tue May  5 11:39:07 2020
From: martin at varnish-software.com (Martin Blix Grydeland)
Date: Tue,  5 May 2020 11:39:07 +0000 (UTC)
Subject: [master] b71ccdeb8 Add an assert that the status is valid on entry to
 h2_build_headers
Message-ID: <20200505113907.9761761628@lists.varnish-cache.org>


commit b71ccdeb8bda20cf37425b0622a0e06a6d7f45f3
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Tue May 5 13:36:22 2020 +0200

    Add an assert that the status is valid on entry to h2_build_headers

diff --git a/bin/varnishd/http2/cache_http2_deliver.c b/bin/varnishd/http2/cache_http2_deliver.c
index 675e9bd29..c2ef90eec 100644
--- a/bin/varnishd/http2/cache_http2_deliver.c
+++ b/bin/varnishd/http2/cache_http2_deliver.c
@@ -232,6 +232,7 @@ h2_build_headers(struct vsb *resp, struct req *req)
 	uint8_t buf[6];
 	ssize_t sz, sz1;
 
+	assert(req->resp->status % 1000 >= 100);
 	l = h2_status(buf, req->resp->status % 1000);
 	VSB_bcat(resp, buf, l);
 

From nils.goroll at uplex.de  Thu May  7 16:25:09 2020
From: nils.goroll at uplex.de (Nils Goroll)
Date: Thu,  7 May 2020 16:25:09 +0000 (UTC)
Subject: [master] f2a800cce sync Makefile for generate.py
Message-ID: <20200507162509.668E85481@lists.varnish-cache.org>


commit f2a800cce0f244e4f33d94c40b042398984b4a44
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Thu May 7 17:35:00 2020 +0200

    sync Makefile for generate.py
    
    - missing dependencies in each of the both places of invocation
    - vrt_obj.h is generated, so it cannot be a dependency at the same time
    
    notice with out-of-tree parallel build

diff --git a/include/Makefile.am b/include/Makefile.am
index 23e355c86..fa6430de7 100644
--- a/include/Makefile.am
+++ b/include/Makefile.am
@@ -111,6 +111,7 @@ nobase_noinst_HEADERS = \
 ## keep in sync with lib/libvcc/Makefile.am
 vcl.h: \
 	    $(top_srcdir)/lib/libvcc/generate.py \
+	    $(top_srcdir)/include/vcc_interface.h \
 	    $(top_srcdir)/include/vdef.h \
 	    $(top_srcdir)/include/vrt.h \
 	    $(top_srcdir)/doc/sphinx/reference/vcl_var.rst
diff --git a/lib/libvcc/Makefile.am b/lib/libvcc/Makefile.am
index 9333284fd..6f023a452 100644
--- a/lib/libvcc/Makefile.am
+++ b/lib/libvcc/Makefile.am
@@ -45,7 +45,7 @@ vcc_obj.c: \
 	    $(top_srcdir)/include/vcc_interface.h \
 	    $(top_srcdir)/include/vdef.h \
 	    $(top_srcdir)/include/vrt.h \
-	    $(top_srcdir)/include/vrt_obj.h
+	    $(top_srcdir)/doc/sphinx/reference/vcl_var.rst
 	mkdir -p $(top_builddir)/include/tbl
 	@PYTHON@ $(top_srcdir)/lib/libvcc/generate.py \
 	    $(top_srcdir) $(top_builddir)

From nils.goroll at uplex.de  Thu May  7 16:25:09 2020
From: nils.goroll at uplex.de (Nils Goroll)
Date: Thu,  7 May 2020 16:25:09 +0000 (UTC)
Subject: [master] cb61f6029 fix make maintainer-clean
Message-ID: <20200507162509.7CB0A5484@lists.varnish-cache.org>


commit cb61f6029c77aa2dcfb9c4bfcf4129cd336772f3
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Thu May 7 17:40:39 2020 +0200

    fix make maintainer-clean
    
    vcc_compile.h is not generated
    EXTRA_DIST is not necessarily generated

diff --git a/doc/sphinx/Makefile.am b/doc/sphinx/Makefile.am
index 82ba04760..7c2a38a37 100644
--- a/doc/sphinx/Makefile.am
+++ b/doc/sphinx/Makefile.am
@@ -249,7 +249,6 @@ include/vmod_proxy.generated.rst: $(top_builddir)/lib/libvmod_proxy/vmod_proxy.r
 BUILT_SOURCES += include/vmod_proxy.generated.rst
 
 EXTRA_DIST += $(BUILT_SOURCES)
-MAINTAINERCLEANFILES = $(EXTRA_DIST)
 CLEANFILES = $(BUILT_SOURCES)
 
 .NOPATH: $(BUILT_SOURCES)
diff --git a/lib/libvcc/Makefile.am b/lib/libvcc/Makefile.am
index 6f023a452..628e321e6 100644
--- a/lib/libvcc/Makefile.am
+++ b/lib/libvcc/Makefile.am
@@ -55,7 +55,6 @@ vcc_obj.c: \
 GEN_H = \
 	vcc_fixed_token.c \
 	vcc_token_defs.h \
-	vcc_compile.h \
 	tbl/vrt_stv_var.h
 
 $(GEN_H): vcc_obj.c

From fgsch at lodoss.net  Sun May 10 19:51:09 2020
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Sun, 10 May 2020 19:51:09 +0000 (UTC)
Subject: [master] 0ea64396b Fix compilation under macos
Message-ID: <20200510195109.2C59EB04AD@lists.varnish-cache.org>


commit 0ea64396b64b1dba6cf0930d9e4c395b5dd6cfbc
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Sun May 10 20:46:15 2020 +0100

    Fix compilation under macos

diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index 1d8bac59d..619ba4d27 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -64,18 +64,18 @@ VRT_synth(VRT_CTX, VCL_INT code, VCL_STRING reason)
 	assert(ctx->req != NULL || ctx->bo != NULL);
 	if (code < 0) {
 		VRT_fail(ctx, "return(synth()) status code (%jd) is negative",
-		    code);
+		    (intmax_t)code);
 		return;
 	}
 	if (code > 65535) {
 		VRT_fail(ctx, "return(synth()) status code (%jd) > 65535",
-		    code);
+		    (intmax_t)code);
 		return;
 	}
 	if ((code % 1000) < 100) {
 		VRT_fail(ctx,
 		    "illegal return(synth()) status code (%jd) (..0##)",
-		    code);
+		    (intmax_t)code);
 		return;
 	}
 

From fgsch at lodoss.net  Sun May 10 19:51:09 2020
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Sun, 10 May 2020 19:51:09 +0000 (UTC)
Subject: [master] fb159a3fd Bump osx image version
Message-ID: <20200510195109.42B4DB04B0@lists.varnish-cache.org>


commit fb159a3fde67af97e7f3cebfcc1029db1718c1db
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Sun May 10 20:46:25 2020 +0100

    Bump osx image version

diff --git a/.travis.yml b/.travis.yml
index a68713459..ac097fe8b 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -74,7 +74,7 @@ jobs:
         - ./configure --with-unwind --enable-developer-warnings --enable-debugging-symbols --disable-stack-protector --with-persistent-storage --enable-asan --enable-ubsan
     - stage: test
       os: osx
-      osx_image: xcode11.3
+      osx_image: xcode11.4
       compiler: clang
       addons:
         homebrew:

From fgsch at lodoss.net  Sun May 10 19:54:06 2020
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Sun, 10 May 2020 19:54:06 +0000 (UTC)
Subject: [master] 1721a0b6e Switch to clang/llvm 9
Message-ID: <20200510195406.AB9B1B0879@lists.varnish-cache.org>


commit 1721a0b6ee0fc022af22b9f530cf04a4297fae5f
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Sun May 10 20:53:34 2020 +0100

    Switch to clang/llvm 9

diff --git a/.travis.yml b/.travis.yml
index ac097fe8b..b3c37356e 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -50,15 +50,16 @@ jobs:
       script: make -j8 witness
     - <<: *test-linux
       stage: sanitizers
-      dist: xenial
       addons:
         apt:
           sources:
-            - llvm-toolchain-xenial-8
+            - sourceline: 'deb http://apt.llvm.org/bionic/ llvm-toolchain-bionic-9 main'
+              key_url: https://apt.llvm.org/llvm-snapshot.gpg.key
+            - ubuntu-toolchain-r-test
           packages:
-            - clang-8
+            - clang-9
             - libunwind-dev
-            - llvm-8
+            - llvm-9
             - nghttp2
             - python3-docutils
             - python3-sphinx
@@ -69,7 +70,7 @@ jobs:
           export LSAN_OPTIONS=abort_on_error=1,use_sigaltstack=0,suppressions=$(pwd)/tools/lsan.suppr
           export TSAN_OPTIONS=abort_on_error=1,halt_on_error=1,use_sigaltstack=0,suppressions=$(pwd)/tools/tsan.suppr
           export UBSAN_OPTIONS=halt_on_error=1,print_stacktrace=1,use_sigaltstack=0,suppressions=$(pwd)/tools/ubsan.suppr
-          export CC=clang-8
+          export CC=clang-9
         - ./autogen.sh
         - ./configure --with-unwind --enable-developer-warnings --enable-debugging-symbols --disable-stack-protector --with-persistent-storage --enable-asan --enable-ubsan
     - stage: test

From phk at FreeBSD.org  Mon May 11 07:55:07 2020
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 11 May 2020 07:55:07 +0000 (UTC)
Subject: [master] b8efd99ab Ensure we don't recycle a closed backend connection
Message-ID: <20200511075507.BF53B93F9D@lists.varnish-cache.org>


commit b8efd99aba079fe9bc92ec7f70a839527fff321a
Author: Dag Haavi Finstad <daghf at varnish-software.com>
Date:   Wed Apr 1 15:52:31 2020 +0200

    Ensure we don't recycle a closed backend connection
    
    Fixes: #3266

diff --git a/bin/varnishd/http1/cache_http1_vfp.c b/bin/varnishd/http1/cache_http1_vfp.c
index 7e4782a15..5e3b5cfa0 100644
--- a/bin/varnishd/http1/cache_http1_vfp.c
+++ b/bin/varnishd/http1/cache_http1_vfp.c
@@ -81,6 +81,9 @@ v1f_read(const struct vfp_ctx *vc, struct http_conn *htc, void *d, ssize_t len)
 			    "%s", vstrerror(errno));
 			return (i);
 		}
+		if (i == 0)
+			htc->doclose = SC_RESP_CLOSE;
+
 	}
 	return (i + l);
 }
diff --git a/bin/varnishtest/tests/r03266.vtc b/bin/varnishtest/tests/r03266.vtc
new file mode 100644
index 000000000..1c4718192
--- /dev/null
+++ b/bin/varnishtest/tests/r03266.vtc
@@ -0,0 +1,19 @@
+varnishtest "Don't recycle a closed backend connection"
+
+# broken origin: sends eof-encoded HTTP/1.1 response
+server s1 {
+	rxreq
+	send "HTTP/1.1 200 OK\r\n\r\n"
+	send "foobar"
+} -start
+
+varnish v1 -vcl+backend {} -start
+
+client c1 {
+	txreq
+	rxresp
+} -run
+
+varnish v1 -expect fetch_failed == 0
+varnish v1 -expect fetch_eof == 1
+varnish v1 -expect backend_recycle == 0

From dridi.boukelmoune at gmail.com  Tue May 12 07:21:10 2020
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 12 May 2020 07:21:10 +0000 (UTC)
Subject: [master] 86032715e Reference the correct transition in VRT_synth()
Message-ID: <20200512072110.3A311623D1@lists.varnish-cache.org>


commit 86032715e3526d56d3324a25972ebe1ac23f003f
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue May 12 08:59:50 2020 +0200

    Reference the correct transition in VRT_synth()

diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index 619ba4d27..4229d8e53 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -59,23 +59,26 @@ const void * const vrt_magic_string_unset = &vrt_magic_string_unset;
 VCL_VOID
 VRT_synth(VRT_CTX, VCL_INT code, VCL_STRING reason)
 {
+	const char *ret;
 
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
 	assert(ctx->req != NULL || ctx->bo != NULL);
+
+	ret = ctx->req == NULL ? "error" : "synth";
 	if (code < 0) {
-		VRT_fail(ctx, "return(synth()) status code (%jd) is negative",
-		    (intmax_t)code);
+		VRT_fail(ctx, "return(%s()) status code (%jd) is negative",
+		    ret, (intmax_t)code);
 		return;
 	}
 	if (code > 65535) {
-		VRT_fail(ctx, "return(synth()) status code (%jd) > 65535",
-		    (intmax_t)code);
+		VRT_fail(ctx, "return(%s()) status code (%jd) > 65535",
+		    ret, (intmax_t)code);
 		return;
 	}
 	if ((code % 1000) < 100) {
 		VRT_fail(ctx,
-		    "illegal return(synth()) status code (%jd) (..0##)",
-		    (intmax_t)code);
+		    "illegal return(%s()) status code (%jd) (..0##)",
+		    ret, (intmax_t)code);
 		return;
 	}
 

From dridi.boukelmoune at gmail.com  Tue May 12 07:21:10 2020
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 12 May 2020 07:21:10 +0000 (UTC)
Subject: [master] 8f3f5b57c Centralize the VCL status code documentation
Message-ID: <20200512072110.4F36E623D7@lists.varnish-cache.org>


commit 8f3f5b57c811f750704e6e1e085706642be08e6c
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue May 12 09:03:38 2020 +0200

    Centralize the VCL status code documentation

diff --git a/doc/sphinx/reference/vcl_var.rst b/doc/sphinx/reference/vcl_var.rst
index 300ea0a22..30aef93cf 100644
--- a/doc/sphinx/reference/vcl_var.rst
+++ b/doc/sphinx/reference/vcl_var.rst
@@ -702,13 +702,7 @@ beresp.status
 
 	The HTTP status code returned by the server.
 
-	Status codes on the form XXYZZ can be set where
-	XXYZZ is less than 65536 and Y is [1...9].
-	Only YZZ will be sent back to clients.
-
-	XX can be therefore be used to pass information
-	around inside VCL, for instance ``return(synth(22404))``
-	from ``vcl_recv{}`` to ``vcl_synth{}``
+	More information in the `HTTP response status`_ section.
 
 beresp.reason
 
@@ -1028,6 +1022,8 @@ obj.status
 
 	The HTTP status code stored in the object.
 
+	More information in the `HTTP response status`_ section.
+
 
 obj.reason
 
@@ -1183,12 +1179,7 @@ resp.status
 
 	The HTTP status code that will be returned.
 
-	Assigning a HTTP standardized code to resp.status will also
-	set resp.reason to the corresponding status message.
-
-	Status codes on the form XXYZZ can be set where
-	XXYZZ is less than 65536 and Y is [1...9].
-	Only YZZ will be sent back to clients.
+	More information in the `HTTP response status`_ section.
 
 	resp.status 200 will get changed into 304 by core code after
 	a return(deliver) from vcl_deliver for conditional requests
@@ -1389,3 +1380,29 @@ storage.<name>.happy
 	Health status for the named stevedore. Not available in any of the
 	current stevedores.
 
+HTTP response status
+--------------------
+
+A status code normally has 3 digits XYZ where X must be between 1 and 5
+included. Since it is not uncommon to see HTTP clients or servers relying
+on non-standard or even invalid status codes Varnish tolerates any status
+between 100 and 999.
+
+With VCL code it is possible to use status codes in the form XXYZZ where the
+overall value is lower than 65536 and the Y digit is between 1 and 9 included.
+Only the YZZ part is sent to the client.
+
+The XXYZZ form of status codes can be set on ``resp.status`` and
+``beresp.status`` or passed via ``return(synth(...))`` and
+``return(error(...))`` transitions.
+
+XX can be therefore be used to pass information around inside VCL, for
+instance ``return(synth(22404))`` from ``vcl_recv{}`` to ``vcl_synth{}``.
+
+The ``obj.status`` variable will inherit the XXYZZ form, but in a ban
+expresion only the YZZ part will be available. The XXYZZ form is strictly
+limited to VCL execution.
+
+Assigning an HTTP standardized code to ``resp.status`` or ``beresp.status``
+will also set ``resp.reason`` or ``beresp.reason``  to the corresponding
+status message.

From nils.goroll at uplex.de  Tue May 12 13:07:09 2020
From: nils.goroll at uplex.de (Nils Goroll)
Date: Tue, 12 May 2020 13:07:09 +0000 (UTC)
Subject: [master] 32530b5c2 Document connection pooling
Message-ID: <20200512130709.33DA794C14@lists.varnish-cache.org>


commit 32530b5c28a24a9ceefa8ca363b0cdd41cccd23f
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Tue May 12 15:05:42 2020 +0200

    Document connection pooling
    
    Closes #3312 (probably)

diff --git a/bin/varnishd/VSC_vbe.vsc b/bin/varnishd/VSC_vbe.vsc
index f513e41ea..c1c01632d 100644
--- a/bin/varnishd/VSC_vbe.vsc
+++ b/bin/varnishd/VSC_vbe.vsc
@@ -71,7 +71,12 @@
 .. varnish_vsc:: conn
 	:type:	gauge
 	:level:	info
-	:oneliner:	Concurrent connections to backend
+	:oneliner:	Concurrent connections used
+
+	The number of currently used connections to the backend. This
+	number is always less or equal to the number of connections to
+	the backend (as, for example shown as ESTABLISHED for TCP
+	onnections in netstat) due to connection pooling.
 
 .. varnish_vsc:: req
 	:type:	counter
diff --git a/doc/sphinx/users-guide/vcl-backends.rst b/doc/sphinx/users-guide/vcl-backends.rst
index d4fe87763..5a640712f 100644
--- a/doc/sphinx/users-guide/vcl-backends.rst
+++ b/doc/sphinx/users-guide/vcl-backends.rst
@@ -219,3 +219,35 @@ VCLs. Varnish will coalesce probes that seem identical - so be careful
 not to change the probe config if you do a lot of VCL loading. Unloading
 the VCL will discard the probes. For more information on how to do this
 please see ref:`reference-vcl-director`.
+
+.. _users-guide-advanced_backend_connection-pooling:
+
+Connection Pooling
+------------------
+
+Opening connections to backends always comes at a cost: Depending on
+the type of connection and backend infrastructure, the overhead for
+opening a new connection ranges from pretty low for a local Unix
+domain socket (see :ref:`backend_definition` ``.path`` attribute) to
+substantial for establishing possibly multiple TCP and/or TLS
+connections over possibly multiple hops and long network
+paths. However relevant the overhead, it certainly always exists.
+
+So because re-using existing connections can generally be considered
+to reduce overhead and latencies, Varnish pools backend connections by
+default: Whenever a backend task is finished, the used connection is
+not closed but rather added to a pool for later reuse. To avoid a
+connection from being reused, the ``Connection: close`` http header
+can be added in :ref:`vcl_backend_fetch`.
+
+While backends are defined per VCL, connection pooling works across
+VCLs and even across backends: By default, the identifier for pooled
+connections is constructed from the ``.host``\ /\ ``.port`` or
+``.path`` attributes of the :ref:`backend_definition` (VMODs can make
+use of custom identifiers). So whenever two backends share the same
+address information, irrespective of which VCLs they are defined in,
+their connections are taken from a common pool.
+
+If not actively closed by the backend, pooled connections are kept
+open by Varnish until the :ref:`ref_param_backend_idle_timeout`
+expires.

From nils.goroll at uplex.de  Tue May 12 13:36:07 2020
From: nils.goroll at uplex.de (Nils Goroll)
Date: Tue, 12 May 2020 13:36:07 +0000 (UTC)
Subject: [master] 05869183f adjust test to changed counter description
Message-ID: <20200512133607.71C5D958DC@lists.varnish-cache.org>


commit 05869183fb8487de8c956919a1730e162e8ad428
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Tue May 12 15:34:55 2020 +0200

    adjust test to changed counter description
    
    Ref #3312

diff --git a/bin/varnishtest/tests/u00008.vtc b/bin/varnishtest/tests/u00008.vtc
index 6a0ce88f9..5e6e827ad 100644
--- a/bin/varnishtest/tests/u00008.vtc
+++ b/bin/varnishtest/tests/u00008.vtc
@@ -40,7 +40,7 @@ process p1 -expect-text 0 0 "DIAG"
 process p1 -screen_dump
 
 process p1 -write {dek}
-process p1 -expect-text 0 1 "Concurrent connections to backend:"
+process p1 -expect-text 0 1 "Concurrent connections used:"
 process p1 -screen_dump
 
 process p1 -write {h}
@@ -53,7 +53,7 @@ process p1 -screen_dump
 
 # the counters screen is preserved
 process p1 -write {h}
-process p1 -expect-text 0 1 "Concurrent connections to backend:"
+process p1 -expect-text 0 1 "Concurrent connections used:"
 process p1 -screen_dump
 
 # the help screen always appears from the top

From reza at naghibi.com  Tue May 12 15:12:07 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Tue, 12 May 2020 15:12:07 +0000 (UTC)
Subject: [6.0] f7854267e Expose the master and worker PIDs via the CLI
Message-ID: <20200512151207.F383397F10@lists.varnish-cache.org>


commit f7854267ef7ce2bf52517a801dc9896bb9336f88
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Mon Dec 30 16:38:05 2019 +0100

    Expose the master and worker PIDs via the CLI
    
    The change in u00011.vtc is the result of having two commands starting
    with "pi", breaking auto-completion. Fortunately "pin\t" still does the
    trick.

diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index fdcdfd830..bc5c00ec2 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -670,6 +670,31 @@ MCH_Running(void)
  * CLI commands
  */
 
+static void v_matchproto_(cli_func_t)
+mch_pid(struct cli *cli, const char * const *av, void *priv)
+{
+
+	(void)av;
+	(void)priv;
+	VCLI_Out(cli, "Master: %10jd\n", (intmax_t)getpid());
+	if (!MCH_Running())
+		return;
+	VCLI_Out(cli, "Worker: %10jd\n", (intmax_t)child_pid);
+}
+
+static void v_matchproto_(cli_func_t)
+mch_pid_json(struct cli *cli, const char * const *av, void *priv)
+{
+
+	(void)priv;
+	VCLI_JSON_begin(cli, 2, av);
+	VCLI_Out(cli, ",\n  {\"master\": %jd", (intmax_t)getpid());
+	if (MCH_Running())
+		VCLI_Out(cli, ", \"worker\": %jd", (intmax_t)child_pid);
+	VCLI_Out(cli, "}");
+	VCLI_JSON_end(cli);
+}
+
 static void v_matchproto_(cli_func_t)
 mch_cli_server_start(struct cli *cli, const char * const *av, void *priv)
 {
@@ -729,6 +754,7 @@ static struct cli_proto cli_mch[] = {
 	{ CLICMD_PANIC_SHOW,		"", mch_cli_panic_show,
 	  mch_cli_panic_show_json },
 	{ CLICMD_PANIC_CLEAR,		"", mch_cli_panic_clear },
+	{ CLICMD_PID,			"", mch_pid, mch_pid_json },
 	{ NULL }
 };
 
diff --git a/bin/varnishtest/tests/b00071.vtc b/bin/varnishtest/tests/b00071.vtc
new file mode 100644
index 000000000..e69a92510
--- /dev/null
+++ b/bin/varnishtest/tests/b00071.vtc
@@ -0,0 +1,9 @@
+varnishtest "varnish-cli pid command"
+
+varnish v1 -cliexpect "^Master: +[0-9]+\n$" pid
+varnish v1 -cliok "pid -j"
+
+varnish v1 -vcl {backend be none;} -start
+
+varnish v1 -cliexpect "^Master: +[0-9]+\nWorker: +[0-9]+\n$" pid
+varnish v1 -cliok "pid -j"
diff --git a/bin/varnishtest/tests/u00011.vtc b/bin/varnishtest/tests/u00011.vtc
index b53a9ba7b..bb9701db4 100644
--- a/bin/varnishtest/tests/u00011.vtc
+++ b/bin/varnishtest/tests/u00011.vtc
@@ -18,7 +18,7 @@ process p1 -log {varnishadm -n ${v1_name}} -start
 
 process p1 -expect-text 0 1 "Type 'quit' to close CLI session."
 
-process p1 -write "pi\t\r"
+process p1 -write "pin\t\r"
 
 process p1 -expect-text 0 1 "PONG"
 
diff --git a/include/tbl/cli_cmds.h b/include/tbl/cli_cmds.h
index 732522736..bf95a7901 100644
--- a/include/tbl/cli_cmds.h
+++ b/include/tbl/cli_cmds.h
@@ -348,6 +348,14 @@ CLI_CMD(STORAGE_LIST,
 	0, 0
 )
 
+CLI_CMD(PID,
+	"pid",
+	"pid [-j]",
+	"Show the pid of the master process, and the worker if it's running.",
+	"  ``-j`` specifies JSON output.",
+	0, 0
+)
+
 #undef CLI_CMD
 
 /*lint -restore */

From reza at naghibi.com  Tue May 12 15:13:07 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Tue, 12 May 2020 15:13:07 +0000 (UTC)
Subject: [6.0] cd7ec513b Fixup private rfc2616_time function
Message-ID: <20200512151307.1D9EF9B18B@lists.varnish-cache.org>


commit cd7ec513b38380a054a23475ca7900316c99360b
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Tue Feb 25 15:22:36 2020 +0100

    Fixup private rfc2616_time function
    
    Change the return value to unsigned, to match with the expected data type
    where it is used.
    
    Handle very large numbers consistently. Currently it was converting from
    unsigned long to int, which would throw away the most significant
    bits. Now overly large integers will be capped at UINT_MAX.
    
    Implement the "allow and ignore decimal point" behaviour that the Age
    header parsing incorporated in rfc2616_time(). This way we will allow a
    decimal points also in max-age and stale-while-revalidate parsing of
    Cache-Control directives.

diff --git a/bin/varnishd/cache/cache_rfc2616.c b/bin/varnishd/cache/cache_rfc2616.c
index e78204782..79317394e 100644
--- a/bin/varnishd/cache/cache_rfc2616.c
+++ b/bin/varnishd/cache/cache_rfc2616.c
@@ -62,17 +62,22 @@
  *
  */
 
-static inline int
+static inline unsigned
 rfc2616_time(const char *p)
 {
 	char *ep;
-	int val;
+	unsigned long val;
 	if (*p == '-')
 		return (0);
 	val = strtoul(p, &ep, 10);
-	for (; *ep != '\0' && vct_issp(*ep); ep++)
-		continue;
-	if (*ep == '\0' || *ep == ',')
+	if (val > UINT_MAX)
+		return (UINT_MAX);
+	while (vct_issp(*ep))
+		ep++;
+	/* We accept ',' as an end character because we may be parsing a
+	 * multi-element Cache-Control part. We accept '.' to be future
+	 * compatble with fractional seconds. */
+	if (*ep == '\0' || *ep == ',' || *ep == '.')
 		return (val);
 	return (0);
 }

From reza at naghibi.com  Tue May 12 15:13:07 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Tue, 12 May 2020 15:13:07 +0000 (UTC)
Subject: [6.0] 3bc22d40c Use rfc2616_time() to parse Age headers
Message-ID: <20200512151307.34FCA9B190@lists.varnish-cache.org>


commit 3bc22d40c762d33bb9a9d98523225a5e87f10603
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Tue Feb 25 15:29:58 2020 +0100

    Use rfc2616_time() to parse Age headers
    
    One time element function to rule and parse them all.

diff --git a/bin/varnishd/cache/cache_rfc2616.c b/bin/varnishd/cache/cache_rfc2616.c
index 79317394e..2a1b476dd 100644
--- a/bin/varnishd/cache/cache_rfc2616.c
+++ b/bin/varnishd/cache/cache_rfc2616.c
@@ -115,12 +115,7 @@ RFC2616_Ttl(struct busyobj *bo, vtim_real now, vtim_real *t_origin,
 	 */
 
 	if (http_GetHdr(hp, H_Age, &p)) {
-		/*
-		 * We deliberately run with partial results, rather than
-		 * reject the Age: header outright.  This will be future
-		 * compatible with fractional seconds.
-		 */
-		age = strtoul(p, NULL, 10);
+		age = rfc2616_time(p);
 		*t_origin -= age;
 	}
 

From reza at naghibi.com  Tue May 12 15:13:07 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Tue, 12 May 2020 15:13:07 +0000 (UTC)
Subject: [6.0] bdbe85cf5 Keep Age information on passes
Message-ID: <20200512151307.6BA959B195@lists.varnish-cache.org>


commit bdbe85cf50b960a6bb4b362e78a251f15c36ffe6
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Feb 21 16:40:38 2020 +0100

    Keep Age information on passes
    
    When doing a pass, we would remove the Age header from the backend, and
    create a new one based on the time the fetch was initiated. This creates
    problems when calculating the time to live in downstream caches (browser
    cache or layered varnishes).
    
    With this patch, the RFC_2616_Ttl calculation routine is run also for
    passes, where the t_origin field of the object is adjusted for an incoming
    Age header. This makes sure that the Age header generated during delivery
    is correct. The rest of the Ttl calculation is skipped for passes,
    including the logging of SLT_TTL "RFC".
    
    Fixes: varnishcache/varnish-cache#3221

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 3ce30ca8a..2395b1b33 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -336,21 +336,12 @@ vbf_stp_startfetch(struct worker *wrk, struct busyobj *bo)
 	http_CollectHdr(bo->beresp, H_Cache_Control);
 	http_CollectHdr(bo->beresp, H_Vary);
 
-	if (bo->fetch_objcore->flags & OC_F_PRIVATE) {
-		/* private objects have negative TTL */
-		bo->fetch_objcore->t_origin = now;
-		bo->fetch_objcore->ttl = -1.;
-		bo->fetch_objcore->grace = 0;
-		bo->fetch_objcore->keep = 0;
-	} else {
-		/* What does RFC2616 think about TTL ? */
-		RFC2616_Ttl(bo, now,
-		    &bo->fetch_objcore->t_origin,
-		    &bo->fetch_objcore->ttl,
-		    &bo->fetch_objcore->grace,
-		    &bo->fetch_objcore->keep
-		    );
-	}
+	/* What does RFC2616 think about TTL ? */
+	RFC2616_Ttl(bo, now,
+	    &bo->fetch_objcore->t_origin,
+	    &bo->fetch_objcore->ttl,
+	    &bo->fetch_objcore->grace,
+	    &bo->fetch_objcore->keep);
 
 	AZ(bo->do_esi);
 	AZ(bo->was_304);
diff --git a/bin/varnishd/cache/cache_rfc2616.c b/bin/varnishd/cache/cache_rfc2616.c
index 2a1b476dd..8b94ec302 100644
--- a/bin/varnishd/cache/cache_rfc2616.c
+++ b/bin/varnishd/cache/cache_rfc2616.c
@@ -92,6 +92,7 @@ RFC2616_Ttl(struct busyobj *bo, vtim_real now, vtim_real *t_origin,
 	const struct http *hp;
 
 	CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC);
+	CHECK_OBJ_NOTNULL(bo->fetch_objcore, OBJCORE_MAGIC);
 	assert(now != 0.0 && !isnan(now));
 	AN(t_origin);
 	AN(ttl);
@@ -119,6 +120,19 @@ RFC2616_Ttl(struct busyobj *bo, vtim_real now, vtim_real *t_origin,
 		*t_origin -= age;
 	}
 
+	if (bo->fetch_objcore->flags & OC_F_PRIVATE) {
+		/* Pass object. Halt the processing here, keeping only the
+		 * parsed value of t_origin, as that will be needed to
+		 * synthesize a correct Age header in delivery. The
+		 * SLT_TTL log tag at the end of this function is
+		 * deliberetaly skipped to avoid confusion when reading
+		 * the log.*/
+		*ttl = -1;
+		*grace = 0;
+		*keep = 0;
+		return;
+	}
+
 	if (http_GetHdr(hp, H_Expires, &p))
 		h_expires = VTIM_parse(p);
 
diff --git a/bin/varnishtest/tests/r03221.vtc b/bin/varnishtest/tests/r03221.vtc
new file mode 100644
index 000000000..8fab00691
--- /dev/null
+++ b/bin/varnishtest/tests/r03221.vtc
@@ -0,0 +1,27 @@
+varnishtest "Handling of Age when return(pass)"
+
+server s1 {
+	rxreq
+	txresp -hdr "cache-control: max-age=2" -hdr "age: 1"
+
+	rxreq
+	txresp -hdr "cache-control: max-age=2" -hdr "age: 1"
+} -start
+
+varnish v1 -vcl+backend {
+	sub vcl_recv {
+		if (req.url == "/pass") {
+			return(pass);
+		}
+	}
+} -start
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.http.age == 1
+
+	txreq -url /pass
+	rxresp
+	expect resp.http.age == 1
+} -run

From reza at naghibi.com  Tue May 12 15:14:07 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Tue, 12 May 2020 15:14:07 +0000 (UTC)
Subject: [6.0] ca5071770 Log using SLT_Notice on condfetch streaming delay
Message-ID: <20200512151407.242479B6BC@lists.varnish-cache.org>


commit ca5071770535bd657c31575c23cad408c859906e
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Mon Nov 25 12:01:42 2019 +0100

    Log using SLT_Notice on condfetch streaming delay
    
    Log a notice message when delaying a conditional fetch and the stale
    template object is still streaming.
    
    Conflicts:
        bin/varnishd/cache/cache_fetch.c
        doc/sphinx/reference/vsl.rst

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 2395b1b33..f85372b10 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -706,6 +706,34 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo)
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC);
+	CHECK_OBJ_NOTNULL(bo->fetch_objcore, OBJCORE_MAGIC);
+	CHECK_OBJ_NOTNULL(bo->stale_oc, OBJCORE_MAGIC);
+
+	stale_boc = HSH_RefBoc(bo->stale_oc);
+	CHECK_OBJ_ORNULL(stale_boc, BOC_MAGIC);
+	if (stale_boc) {
+		/* Wait for the stale object to become fully fetched, so
+		 * that we can catch fetch errors, before we unbusy the
+		 * new object. This serves two purposes. First it helps
+		 * with request coalesching, and stops long chains of
+		 * IMS-updated short-TTL objects all streaming from a
+		 * single slow body fetch. Second it makes sure that all
+		 * the object attributes are complete when we copy them
+		 * (this would be an issue for ie OA_GZIPBITS). */
+		VSLb(bo->vsl, SLT_Notice,
+		    "[core] Conditional fetch wait for streaming object");
+		ObjWaitState(bo->stale_oc, BOS_FINISHED);
+		stale_state = stale_boc->state;
+		HSH_DerefBoc(bo->wrk, bo->stale_oc);
+		stale_boc = NULL;
+		if (stale_state != BOS_FINISHED) {
+			(void)VFP_Error(bo->vfc, "Template object failed");
+			vbf_cleanup(bo);
+			wrk->stats->fetch_failed++;
+			return (F_STP_FAIL);
+		}
+	}
+	AZ(bo->stale_oc->flags & OC_F_FAILED);
 
 	AZ(vbf_beresp2obj(bo));
 
diff --git a/doc/sphinx/reference/vsl.rst b/doc/sphinx/reference/vsl.rst
index 3f39c67b6..4d01f5ba8 100644
--- a/doc/sphinx/reference/vsl.rst
+++ b/doc/sphinx/reference/vsl.rst
@@ -102,6 +102,26 @@ Error
 	Backend request failed to vcl_backend_error.
 
 
+NOTICE MESSAGES
+===============
+
+Notice messages contain informational messages about the handling of a
+request. These can be exceptional circumstances encountered that causes
+deviation from the normal handling. The messages are prefixed with [core]
+for core Varnish generated messages, or with [<VMOD name>] for VMOD
+generated messages. The [core] messages are described in detail below, see
+the individual VMOD documentation for messages from VMODs.
+
+[core] Conditional fetch wait for streaming object
+	The backend answered 304 Not Modified on a conditional fetch using
+	an object that has not yet been fully fetched as the stale
+	template object. This can only happen when the TTL of the object
+	is less than the time it takes to fetch it. The fetch is halted
+	until the stale object is fully fetched, upon which the new object
+	is created as normal. While waiting, any grace time on the stale
+	object will be in effect.
+
+
 HISTORY
 =======
 

From reza at naghibi.com  Tue May 12 15:14:07 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Tue, 12 May 2020 15:14:07 +0000 (UTC)
Subject: [6.0] 812ede3eb Add a SLT_Notice VSL tag
Message-ID: <20200512151407.409969B6C0@lists.varnish-cache.org>


commit 812ede3eb1ec2a69fecce54ac2330563605e0619
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Mon Nov 25 11:41:26 2019 +0100

    Add a SLT_Notice VSL tag
    
    This VSL tag will be used for informational messages related to
    exceptional handling of requests.
    
    Conflicts:
        doc/sphinx/reference/vsl.rst

diff --git a/include/tbl/vsl_tags.h b/include/tbl/vsl_tags.h
index a99fe9fcf..81f759608 100644
--- a/include/tbl/vsl_tags.h
+++ b/include/tbl/vsl_tags.h
@@ -661,6 +661,13 @@ SLTM(VCL_use, 0, "VCL in use",
 	"\n"
 )
 
+SLTM(Notice, 0, "Informational messages about request handling",
+	"Informational log messages on events occured during request"
+	" handling. Lines are prefixed with either [core] or [<VMOD name>]."
+	" See the NOTICE MESSAGES section below or the individual VMOD manual"
+	" pages for detailed information of notice messages.\n"
+)
+
 #undef NOSUP_NOTICE
 #undef NODEF_NOTICE
 #undef SLTM

From reza at naghibi.com  Tue May 12 15:14:07 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Tue, 12 May 2020 15:14:07 +0000 (UTC)
Subject: [6.0] f11faf9e8 Conditional fetch wait for streaming stale object
Message-ID: <20200512151407.5BCB69B6C4@lists.varnish-cache.org>


commit f11faf9e879eb09870edbc20e23dbaf576503728
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Wed Oct 9 14:01:15 2019 +0200

    Conditional fetch wait for streaming stale object
    
    Wait for the stale object to become fully fetched, so that we can catch
    fetch errors, before we unbusy the new object. This serves two
    purposes. First it helps with request coalescing, and stops long chains of
    IMS-updated short-TTL objects all streaming from a single slow body
    fetch. Second it makes sure that all the object attributes are complete
    when we copy them (this would be an issue for ie OA_GZIPBITS).
    
    This patch OBE's r01646.vtc, and slightly patches r01648.vtc to expect a
    503 instead of a 200 and a broken connection on the failing client.
    
    Fixes: #3089
    
    Conflicts:
        bin/varnishd/cache/cache_fetch.c

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index f85372b10..c3ff67e9a 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -703,6 +703,8 @@ vbf_objiterator(void *priv, int flush, const void *ptr, ssize_t len)
 static enum fetch_step
 vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo)
 {
+	struct boc *stale_boc;
+	enum boc_state_e stale_state;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC);
@@ -728,7 +730,7 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo)
 		stale_boc = NULL;
 		if (stale_state != BOS_FINISHED) {
 			(void)VFP_Error(bo->vfc, "Template object failed");
-			vbf_cleanup(bo);
+			VDI_Finish(bo->wrk, bo);
 			wrk->stats->fetch_failed++;
 			return (F_STP_FAIL);
 		}
@@ -755,8 +757,6 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo)
 	if (ObjIterate(wrk, bo->stale_oc, bo, vbf_objiterator, 0))
 		(void)VFP_Error(bo->vfc, "Template object failed");
 
-	if (bo->stale_oc->flags & OC_F_FAILED)
-		(void)VFP_Error(bo->vfc, "Template object failed");
 	if (bo->vfc->failed) {
 		VDI_Finish(bo->wrk, bo);
 		wrk->stats->fetch_failed++;
diff --git a/bin/varnishtest/tests/r01646.vtc b/bin/varnishtest/tests/r01646.vtc
deleted file mode 100644
index 10b640ea5..000000000
--- a/bin/varnishtest/tests/r01646.vtc
+++ /dev/null
@@ -1,57 +0,0 @@
-varnishtest "cond/stream race ?"
-
-barrier b1 cond 3
-barrier b2 cond 2
-
-server s1 {
-	rxreq
-	txresp -nolen -hdr "Transfer-Encoding: chunked" -hdr "Etag: foo"
-	chunkedlen 32
-	barrier b1 sync
-	chunkedlen 32
-	barrier b2 sync
-	chunkedlen 32
-	chunkedlen 0
-} -start
-
-server s2 {
-	rxreq
-	expect req.http.if-none-match == "foo"
-	txresp -status 304
-} -start
-
-varnish v1 -vcl+backend {
-	sub vcl_backend_fetch {
-		if (bereq.http.foo == "s2") {
-			set bereq.backend = s2;
-		}
-	}
-	sub vcl_backend_response {
-		set beresp.ttl = 1s;
-		set beresp.grace = 0s;
-		set beresp.keep = 30s;
-	}
-} -start
-
-varnish v1 -cliok "param.set debug +syncvsl"
-
-client c1 {
-	txreq
-	rxresphdrs
-	barrier b1 sync
-	rxrespbody
-	expect resp.bodylen == 96
-} -start
-
-client c2 {
-	barrier b1 sync
-	delay 2
-	txreq -hdr "foo: s2"
-	rxresphdrs
-	barrier b2 sync
-	rxrespbody
-	expect resp.bodylen == 96
-} -start
-
-client c1 -wait
-client c2 -wait
diff --git a/bin/varnishtest/tests/r01648.vtc b/bin/varnishtest/tests/r01648.vtc
index 2780db864..7e565af00 100644
--- a/bin/varnishtest/tests/r01648.vtc
+++ b/bin/varnishtest/tests/r01648.vtc
@@ -62,8 +62,7 @@ client c2 {
 	txreq -hdr "Client: c2"
 	barrier b1 sync
 	rxresphdrs
-	expect resp.status == 200
-	expect resp.http.transfer-encoding == "chunked"
+	expect resp.status == 503
 } -run
 
 delay 1
@@ -76,4 +75,5 @@ client c3 {
 	expect resp.body == "abcdef"
 } -run
 
+client c1 -wait
 varnish v1 -expect fetch_failed == 2
diff --git a/bin/varnishtest/tests/r03089.vtc b/bin/varnishtest/tests/r03089.vtc
new file mode 100644
index 000000000..15d912e0b
--- /dev/null
+++ b/bin/varnishtest/tests/r03089.vtc
@@ -0,0 +1,69 @@
+varnishtest "r03089 - Assert error corrupt OA_GZIPBITS"
+
+barrier b1 cond 2
+
+server s1 {
+	rxreq
+	expect req.url == /included
+	expect req.http.accept-encoding == "gzip"
+	txresp -nolen -hdr "Content-Length: 49" -hdr "Content-Encoding: gzip" -hdr {Etag: "asdf"}
+	barrier b1 sync
+	delay 1
+	sendhex "1f 8b 08 08 ef 00 22 59  02 03 5f 00 0b 2e cd 53"
+	sendhex "f0 4d ac 54 30 32 04 22  2b 03 13 2b 13 73 85 d0"
+	sendhex "10 67 05 23 03 43 73 2e  00 cf 9b db c0 1d 00 00"
+	sendhex "00"
+} -start
+
+server s2 {
+	rxreq
+	expect req.url == /included
+	expect req.http.If-None-Match == {"asdf"}
+	barrier b1 sync
+	txresp -status 304 -nolen -hdr "Etag: asdf"
+} -start
+
+server s3 {
+	rxreq
+	expect req.url == /
+	txresp -gzipbody {<esi:include src="/included"/>}
+} -start
+
+varnish v1 -arg "-p debug=+syncvsl" -vcl+backend {
+	sub vcl_deliver {
+	}
+	sub vcl_backend_fetch {
+		if (bereq.url == "/") {
+			set bereq.backend = s3;
+		} else if (bereq.http.backend == "s1") {
+			set bereq.backend = s1;
+		} else {
+			set bereq.backend = s2;
+		}
+	}
+	sub vcl_backend_response {
+		if (bereq.url == "/") {
+			set beresp.do_esi = true;
+		} else {
+			set beresp.ttl = 0.1s;
+			set beresp.grace = 0s;
+			set beresp.keep = 1m;
+		}
+	}
+} -start
+
+client c1 {
+	txreq -url / -hdr "backend: s1" -hdr "Accept-encoding: gzip"
+	rxresp
+} -start
+
+delay 1
+
+client c2 {
+	txreq -url / -hdr "backend: s2" -hdr "Accept-encoding: gzip"
+	rxresp
+} -start
+
+client c1 -wait
+client c2 -wait
+

From dridi.boukelmoune at gmail.com  Wed May 13 08:57:06 2020
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Wed, 13 May 2020 08:57:06 +0000 (UTC)
Subject: [master] 0c9c38513 Test case for VSV00004
Message-ID: <20200513085707.01ED89BD0@lists.varnish-cache.org>


commit 0c9c38513bdb7730ac886eba7563f2d87894d734
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Sep 20 16:25:30 2019 +0200

    Test case for VSV00004

diff --git a/bin/varnishtest/tests/f00004.vtc b/bin/varnishtest/tests/f00004.vtc
new file mode 100644
index 000000000..215a2a0ff
--- /dev/null
+++ b/bin/varnishtest/tests/f00004.vtc
@@ -0,0 +1,98 @@
+varnishtest "VSV00004"
+
+server s1 {
+	rxreq
+	expect req.url == /test1
+	txresp
+
+	rxreq
+	expect req.url == /test2
+	send "bogus\r\n\r\n"
+	expect_close
+
+	accept
+	rxreq
+	expect req.url == /test3
+	txresp
+} -start
+
+varnish v1 -arg "-p debug=+syncvsl" -arg "-p max_restarts=0" -vcl+backend {
+	import vtc;
+
+	sub vcl_recv {
+		if (req.url == "/prime") {
+			# Avoid allocations at start of workspace so
+			# that test string is not overwritten
+			vtc.workspace_alloc(client, 1024);
+
+			set req.http.temp = "super";
+			set req.http.secret = req.http.temp + "secret";
+			return (synth(200, req.http.secret));
+		}
+	}
+	sub vcl_deliver {
+		if (req.url == "/test1") {
+			return (restart);
+		}
+	}
+	sub vcl_backend_error {
+		return (abandon);
+	}
+} -start
+
+# Case 1
+client c1 {
+	txreq -url /prime
+	rxresp
+	expect resp.status == 200
+	expect resp.reason == supersecret
+
+	txreq -url /test1
+	rxresp
+	expect resp.status == 503
+	expect resp.reason != supersecret
+	expect resp.reason == "Service Unavailable"
+} -run
+
+# Case 2
+client c2 {
+	txreq -url /prime
+	rxresp
+	expect resp.status == 200
+	expect resp.reason == supersecret
+
+	txreq -url /test2
+	rxresp
+	expect resp.status == 503
+	expect resp.reason != supersecret
+	expect resp.reason == "Service Unavailable"
+} -run
+
+# Case 3
+varnish v1 -cliok "vcl.label label1 vcl1"
+varnish v1 -cliok "param.reset max_restarts"
+varnish v1 -vcl+backend {
+	sub vcl_recv {
+		if (req.url == "/prime") {
+			return (vcl(label1));
+		}
+		if (req.restarts > 0) {
+			return (vcl(label1));
+		}
+	}
+	sub vcl_deliver {
+		return (restart);
+	}
+}
+client c3 {
+	txreq -url /prime
+	rxresp
+	expect resp.status == 200
+	expect resp.reason == supersecret
+
+	txreq -url /test3
+	rxresp
+	expect resp.status == 503
+	expect resp.reason != supersecret
+	expect resp.reason == "Service Unavailable"
+} -run

From nils.goroll at uplex.de  Wed May 13 15:38:06 2020
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 13 May 2020 15:38:06 +0000 (UTC)
Subject: [master] 97e8a9f58 Make .resolve() a type specific method of BACKEND
Message-ID: <20200513153806.A8DAF9678F@lists.varnish-cache.org>


commit 97e8a9f58ac4270215f2c2529d9d0897f7d9b168
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon May 11 08:45:01 2020 +0000

    Make .resolve() a type specific method of BACKEND
    
    This is an alternative implementation of #2680

diff --git a/bin/varnishd/cache/cache_director.c b/bin/varnishd/cache/cache_director.c
index fe5b132b9..aebbc7a71 100644
--- a/bin/varnishd/cache/cache_director.c
+++ b/bin/varnishd/cache/cache_director.c
@@ -80,27 +80,32 @@ VDI_Ahealth(const struct director *d)
 
 /* Resolve director --------------------------------------------------*/
 
+VCL_BACKEND
+VRT_VDI_Resolve(VRT_CTX, VCL_BACKEND d)
+{
+	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
+
+	while (d != NULL) {
+		CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
+		AN(d->vdir);
+		if (d->vdir->methods->resolve == NULL)
+			break;
+		d = d->vdir->methods->resolve(ctx, d);
+		CHECK_OBJ_ORNULL(d, DIRECTOR_MAGIC);
+	}
+        return (d);
+}
+
 static VCL_BACKEND
 VDI_Resolve(VRT_CTX)
 {
 	const struct director *d;
-	const struct director *d2;
 	struct busyobj *bo;
 
 	bo = ctx->bo;
 	CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC);
 	CHECK_OBJ_ORNULL(bo->director_req, DIRECTOR_MAGIC);
-
-	for (d = bo->director_req; d != NULL &&
-	    d->vdir->methods->resolve != NULL; d = d2) {
-		CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
-		AN(d->vdir);
-		d2 = d->vdir->methods->resolve(ctx, d);
-		if (d2 == NULL)
-			VSLb(bo->vsl, SLT_FetchError,
-			    "Director %s returned no backend", d->vcl_name);
-	}
-
+	d = VRT_VDI_Resolve(ctx, bo->director_req);
 	CHECK_OBJ_ORNULL(d, DIRECTOR_MAGIC);
 	if (d == NULL) {
 		VSLb(bo->vsl, SLT_FetchError, "No backend");
diff --git a/bin/varnishtest/tests/d00021.vtc b/bin/varnishtest/tests/d00021.vtc
index 963356188..91930c18b 100644
--- a/bin/varnishtest/tests/d00021.vtc
+++ b/bin/varnishtest/tests/d00021.vtc
@@ -1,16 +1,16 @@
 varnishtest "shard director LAZY and layering"
 
-server s1 -repeat 2 {
+server s1 -repeat 3 {
 	rxreq
 	txresp -body "ech3Ooj"
 } -start
 
-server s2 -repeat 2 {
+server s2 -repeat 3 {
 	rxreq
 	txresp -body "ieQu2qua"
 } -start
 
-server s3 -repeat 2 {
+server s3 -repeat 3 {
 	rxreq
 	txresp -body "xiuFi3Pe"
 } -start
@@ -88,6 +88,9 @@ varnish v1 -vcl+backend {
 	sub vcl_backend_fetch {
 	    if (bereq.http.layered) {
 		call backend_fetch_layered;
+            } else if (bereq.http.resolve) {
+                call backend_fetch_layered;
+                set bereq.backend = bereq.backend.resolve();
 	    } else {
 		call backend_fetch_direct;
 	    }
@@ -148,4 +151,25 @@ client c1 {
 	expect resp.http.healthy == "true"
 	expect resp.http.director == "ll"
 	expect resp.http.backend == "s3"
+
+	txreq -url /1 -hdr "resolve: true"
+	rxresp
+	expect resp.body == "ech3Ooj"
+	expect resp.http.healthy == "true"
+	expect resp.http.director == "s1"
+	expect resp.http.backend == "s1"
+
+	txreq -url /2 -hdr "resolve: true"
+	rxresp
+	expect resp.body == "ieQu2qua"
+	expect resp.http.healthy == "true"
+	expect resp.http.director == "s2"
+	expect resp.http.backend == "s2"
+
+	txreq -url /3 -hdr "resolve: true"
+	rxresp
+	expect resp.body == "xiuFi3Pe"
+	expect resp.http.healthy == "true"
+	expect resp.http.director == "s3"
+	expect resp.http.backend == "s3"
 } -run
diff --git a/include/vrt.h b/include/vrt.h
index 6d5dcdc93..9e8a91191 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -435,6 +435,7 @@ struct vrt_backend_probe {
 	VRT_BACKEND_PROBE_FIELDS(const)
 };
 
+VCL_BACKEND VRT_VDI_Resolve(VRT_CTX, VCL_BACKEND);
 
 /***********************************************************************
  * Implementation details of ACLs
diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c
index dd62b279d..1b0494e11 100644
--- a/lib/libvcc/vcc_expr.c
+++ b/lib/libvcc/vcc_expr.c
@@ -839,6 +839,7 @@ static const struct vcc_methods {
 
 	{ STRINGS, STRING, "upper", "VRT_UpperLowerStrands(ctx, \vT, 1)", 1 },
 	{ STRINGS, STRING, "lower", "VRT_UpperLowerStrands(ctx, \vT, 0)", 1 },
+	{ BACKEND, BACKEND, "resolve", "VRT_VDI_Resolve(ctx, \v1)", 1 },
 
 	{ NULL, NULL,		NULL,		NULL},
 };

From nils.goroll at uplex.de  Wed May 13 15:47:06 2020
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 13 May 2020 15:47:06 +0000 (UTC)
Subject: [master] c9765cf58 polish assertions
Message-ID: <20200513154706.6DF9296CF9@lists.varnish-cache.org>


commit c9765cf5865568107356e8474d1fa265b5b7b83d
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed May 13 17:42:46 2020 +0200

    polish assertions
    
    See comments in #3311, but done slightly differently

diff --git a/bin/varnishd/cache/cache_director.c b/bin/varnishd/cache/cache_director.c
index aebbc7a71..3e6902031 100644
--- a/bin/varnishd/cache/cache_director.c
+++ b/bin/varnishd/cache/cache_director.c
@@ -86,14 +86,18 @@ VRT_VDI_Resolve(VRT_CTX, VCL_BACKEND d)
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
 
 	while (d != NULL) {
-		CHECK_OBJ_NOTNULL(d, DIRECTOR_MAGIC);
+		CHECK_OBJ(d, DIRECTOR_MAGIC);
 		AN(d->vdir);
 		if (d->vdir->methods->resolve == NULL)
 			break;
 		d = d->vdir->methods->resolve(ctx, d);
-		CHECK_OBJ_ORNULL(d, DIRECTOR_MAGIC);
 	}
-        return (d);
+	if (d == NULL)
+		return (NULL);
+
+	CHECK_OBJ(d, DIRECTOR_MAGIC);
+	AN(d->vdir);
+	return (d);
 }
 
 static VCL_BACKEND

From nils.goroll at uplex.de  Wed May 13 15:50:07 2020
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 13 May 2020 15:50:07 +0000 (UTC)
Subject: [master] 051d1efb7 vrt.h changelog tlc
Message-ID: <20200513155007.36CB596FC3@lists.varnish-cache.org>


commit 051d1efb766f5fb09134cb29989fa0859b57b540
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed May 13 17:48:39 2020 +0200

    vrt.h changelog tlc
    
    Ref #3311

diff --git a/include/vrt.h b/include/vrt.h
index 9e8a91191..3b015ff6a 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -53,6 +53,8 @@
  * Whenever something is deleted or changed in a way which is not
  * binary/load-time compatible, increment MAJOR version
  *
+ * NEXT (2020-09-15)
+ *	VRT_VDI_Resolve() added
  * 11.0 (2020-03-16)
  *	Changed type of vsa_suckaddr_len from int to size_t
  *	New prefix_{ptr|len} fields in vrt_backend

From nils.goroll at uplex.de  Thu May 14 09:28:06 2020
From: nils.goroll at uplex.de (Nils Goroll)
Date: Thu, 14 May 2020 09:28:06 +0000 (UTC)
Subject: [master] 4dc2402ae Document VSC files
Message-ID: <20200514092806.93470971F@lists.varnish-cache.org>


commit 4dc2402ae23aa5b83fed418005d034ee4503b858
Author: Jordan Christiansen <Jordan.Christiansen at target.com>
Date:   Thu Apr 30 16:29:10 2020 -0500

    Document VSC files

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index faa5e374d..c48f51f13 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -636,3 +636,65 @@ unless they access VMOD specific global state, shared with other VCLs.
 
 Traffic in other VCLs which also import this VMOD, will be happening
 while housekeeping is going on.
+
+Statistics Counters
+===================
+
+Starting in Varnish 6.0, VMODs can define their own counters that appear
+in *varnishstat*.
+
+If you're using autotools, see the ``VARNISH_COUNTERS`` macro in
+varnish.m4 for documentation on getting your build set up.
+
+Counters are defined in a .vsc file. The ``VARNISH_COUNTERS`` macro
+calls *vsctool.py* to turn a *foo.vsc* file into *VSC_foo.c* and
+*VSC_foo.h* files, just like *vmodtool.py* turns *foo.vcc* into
+*vcc_foo_if.c* and *vcc_foo_if.h* files. Similarly to the VCC files, the
+generated VSC files give you a structure and functions that you can use
+in your VMOD's code to create and destroy the counters your defined. The
+*vsctool.py* tool also generates a *VSC_foo.rst* file that you can
+include in your documentation to describe the counters your VMOD has.
+
+The .vsc file looks like this::
+
+	.. varnish_vsc_begin:: xkey
+		:oneliner:	xkey Counters
+		:order:		70
+
+		Metrics from vmod_xkey
+
+	.. varnish_vsc:: g_keys
+		:type:		gauge
+		:oneliner:	Number of surrogate keys
+
+		Number of surrogate keys in use. Increases after a request that includes a new key in the xkey header. Decreases when a key is purged or when all cache objects associated with a key expire.
+
+	.. varnish_vsc_end:: xkey
+
+Counters can have the following parameters:
+
+type
+	The type of metric this is. Can be one of ``counter``,
+	``gauge``, or ``bitmap``.
+
+ctype
+	The type that this counter will have in the C code. This can
+	only be ``uint64_t`` and does not need to be specified.
+
+level
+	The verbosity level of this counter. *varnishstat* will only
+	show counters with a higher verbosity level than the one
+	currently configured. Can be one of ``info``, ``diag``, or
+	``debug``.
+
+oneliner
+	A short, one line description of the counter.
+
+group
+	I don't know what this does.
+
+format
+	Can be one of ``integer``, ``bytes``, ``bitmap``, or ``duration``.
+
+After these parameters, a counter can have a longer description, though
+this description has to be all on one line in the .vsc file.

From nils.goroll at uplex.de  Thu May 14 09:28:06 2020
From: nils.goroll at uplex.de (Nils Goroll)
Date: Thu, 14 May 2020 09:28:06 +0000 (UTC)
Subject: [master] b654a1e21 Don't highlight this example as Python
Message-ID: <20200514092806.AB4589723@lists.varnish-cache.org>


commit b654a1e21fa47d4730d92d2426940e04b714e076
Author: Jordan Christiansen <Jordan.Christiansen at target.com>
Date:   Fri May 1 09:40:42 2020 -0500

    Don't highlight this example as Python

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index c48f51f13..6204f6684 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -655,7 +655,9 @@ in your VMOD's code to create and destroy the counters your defined. The
 *vsctool.py* tool also generates a *VSC_foo.rst* file that you can
 include in your documentation to describe the counters your VMOD has.
 
-The .vsc file looks like this::
+The .vsc file looks like this:
+
+.. code-block:: none
 
 	.. varnish_vsc_begin:: xkey
 		:oneliner:	xkey Counters

From nils.goroll at uplex.de  Thu May 14 09:28:06 2020
From: nils.goroll at uplex.de (Nils Goroll)
Date: Thu, 14 May 2020 09:28:06 +0000 (UTC)
Subject: [master] 8551058ef Add docs for contents of VSC_*.h
Message-ID: <20200514092806.C512D9729@lists.varnish-cache.org>


commit 8551058efaee216903e861a55b90948c0ce6841f
Author: Jordan Christiansen <Jordan.Christiansen at target.com>
Date:   Mon May 4 14:11:19 2020 -0500

    Add docs for contents of VSC_*.h

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index 6204f6684..ca448bb8e 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -700,3 +700,8 @@ format
 
 After these parameters, a counter can have a longer description, though
 this description has to be all on one line in the .vsc file.
+
+You should call ``VSC_*_New()`` when your VMOD is loaded and
+``VSC_*_Destroy()`` when it is unloaded. See the generated
+``VSC_*.h`` file for the full details about the structure that contains
+your counters.

From dridi.boukelmoune at gmail.com  Fri May 15 09:12:07 2020
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Fri, 15 May 2020 09:12:07 +0000 (UTC)
Subject: [master] 100ccc2b2 Major reorganization of the CircleCI workflows
Message-ID: <20200515091207.1C63F92D7@lists.varnish-cache.org>


commit 100ccc2b23ae096e8d25b094622fd2afdc6c60c1
Author: Martin Tzvetanov Grigorov <mgrigorov at apache.org>
Date:   Thu Mar 26 09:48:02 2020 +0200

    Major reorganization of the CircleCI workflows
    
    In order to build aarch64 packages it relies on the machine executors
    feature based on QEMU. Since the same scripts were needed to build
    packages for different architectures they were extracted in standalone
    shell scripts.
    
    Alpine Linux packages are now built in addition to RPM and Deb packages
    but considering how long emulation takes for aarch64 packages, packaging
    was moved to a nightly workflow.
    
    Some changes happened in the pkg-varnish-cache repository to accomodate
    the CircleCI reorganization, mainly around dependency management.
    
    More information in the CircleCI README file.
    
    Closes #3313
    
    Reviewed-by: Guillaume Quintard <guillaume at varnish-software.com>
    Signed-off-by: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>

diff --git a/.circleci/README.rst b/.circleci/README.rst
new file mode 100644
index 000000000..0c52c4235
--- /dev/null
+++ b/.circleci/README.rst
@@ -0,0 +1,93 @@
+Multiarch building, testing & packaging
+=======================================
+
+Varnish Cache uses CircleCI_ for building, testing and creating packages for
+several Linux distributions for both x86_64 and aarch64 architectures.
+
+Since CircleCI provides only x86_64 VMs the setup uses Docker and QEMU to be
+able to build, test and create packages for aarch64.  This is accomplished by
+registering ``qemu-user-static`` for the CircleCI ``machine`` executor::
+
+    sudo docker run --rm --privileged multiarch/qemu-user-static --reset --credential yes --persistent yes
+
+Note 1: **--credential yes** is needed so that *setuid* flag is working.
+Without it ``sudo`` does not work in the Docker containers with architecture
+different than x86_64.
+
+Note 2: **--persistent yes** is needed so that there is no need to use
+``:register`` tag. This way one can run locally pure foreign arch Docker
+images, like the official ``arm64v8/***`` ones.
+
+With QEMU registered each build step can start a Docker image for any of the
+supported architectures to execute the ``configure``, ``make``, package steps.
+
+Workflows
+---------
+
+There are two CircleCI workflows:
+
+commit
+~~~~~~
+
+It is executed after each push to any branch, including Pull Requests
+
+The ``commit`` workflow runs two jobs:
+
+- ``dist`` - this job creates the source code distribution of Varnish Cache as
+  compressed archive (``varnish-${VERSION}.tar.gz``).
+
+- ``distcheck`` - untars the source code distribution from ``dist`` job and
+  builds (*configure*, *make*) on different Linux distributions
+
+nightly
+~~~~~~~
+
+It is executed once per day at 04:00 AM UTC time.
+
+This workflow also builds binary packages for different Linux distributions
+and CPU architectures (x86_64 & aarch64) and for this reason its run takes
+longer.
+
+It runs the following jobs:
+
+- The first two jobs that run in parallel are:
+
+  - ``tar_pkg_tools`` - this step checks out pkg-varnish-cache_ with the
+    packaging descriptions for Debian, RedHat and Alpine, and stores them in
+    the build workspace for the next steps in the pipeline.
+
+  - ``dist`` - this step creates the source code distribution of Varnish Cache
+    as compressed archive (``varnish-${VERSION}.tar.gz``). This archive is
+    also stored in the build workspace and used later by the packaging steps.
+
+
+- The next job in the workflow is ``package`` - a job  that creates the
+  packages (e.g. .rpm, .deb) for each supported CPU architecture, Linux
+  distribution and its major version (e.g. *x64_centos_7*,
+  *aarch64_ubuntu_bionic*, *x64_alpine_3*, etc.). This step creates a
+  Dockerfile on the fly by using a base Docker image. This custom Docker image
+  executes a Shell script that has the recipe for creating the package for the
+  specific Linux flavor, e.g.  *make-rpm-packages.sh*. The step stores the
+  packages in the build workspace.
+
+- Finally, if the previous jobs are successful, a final step is executed -
+  ``collect_packages``. This step creates an archive with all packages and
+  stores it as an artifact that can be uploaded to PackageCloud_.
+
+
+More
+----
+
+This setup can be easily extended for any CPU architectures supported by QEMU
+and for any Linux distributions which have Docker image. To do this one needs
+to add a new ``package`` job with the proper parameters for it.
+
+At the moment the setup uses *raw* Docker images and installs the required
+Linux distribution dependencies before running the tests/build/packaging code.
+This could be optimized to save some execution time by creating custom Docker
+images that extend the current ones and pre-installs the required
+dependencies.
+
+.. _CircleCI: https://app.circleci.com/pipelines/github/varnishcache/varnish-cache
+.. _pkg-varnish-cache: https://github.com/varnishcache/pkg-varnish-cache
+.. _PackageCloud: https://packagecloud.io/varnishcache/
diff --git a/.circleci/config.yml b/.circleci/config.yml
index 26716acb8..0ede16afa 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -1,5 +1,11 @@
 version: 2.1
 
+aliases:
+  pkg_req: &pkg_req
+    requires:
+      - dist
+      - tar_pkg_tools
+
 parameters:
   vc-commit:
     type: string
@@ -9,6 +15,7 @@ parameters:
     default: "master"
 jobs:
   dist:
+    description: Builds varnish-x.y.z.tar.gz that is used later for the packaging jobs
     docker:
       - image: centos:7
     steps:
@@ -45,7 +52,9 @@ jobs:
             - .is_weekly
             - varnish*.tar.gz
             - tools/*.suppr
+            - .circleci
   tar_pkg_tools:
+    description: Builds archives with the packaging tools from https://github.com/varnishcache/pkg-varnish-cache
     docker:
       - image: centos:7
     steps:
@@ -62,7 +71,7 @@ jobs:
             git clone https://github.com/varnishcache/pkg-varnish-cache.git .
             git checkout << pipeline.parameters.pkg-commit >>
             tar cvzf debian.tar.gz debian --dereference
-            tar cvzf redhat.tar.gz redhat --dereference 
+            tar cvzf redhat.tar.gz redhat --dereference
             tar cvzf alpine.tar.gz alpine --dereference
       - persist_to_workspace:
           root: .
@@ -70,6 +79,53 @@ jobs:
             - debian.tar.gz
             - redhat.tar.gz
             - alpine.tar.gz
+  package:
+    parameters:
+      dist:
+        description: the Linux distribution (debian|ubuntu|centos)
+        type: string
+      release:
+        description: the release name (stretch|buster|xenial|bionic|7|8)
+        type: string
+      ext:
+        description: the package extension (deb|rpm|apk)
+        type: string
+      arch:
+        description: the architecture (x64|aarch64)
+        type: string
+      image:
+        description: the base Docker image for Dockerfile
+        type: string
+    machine:
+      image: ubuntu-1604:201903-01
+    steps:
+      - attach_workspace:
+          at: ~/project
+      - run: ls -la ~/project
+      - run:
+          name: Activate QEMU
+          command: |
+            sudo docker run -it --rm --privileged multiarch/qemu-user-static --reset --credential yes --persistent yes
+      - run:
+          name: Create Dockerfile
+          command: |
+            echo "FROM << parameters.image >>" > Dockerfile
+            echo "ADD make-<< parameters.ext >>-packages.sh /usr/bin/" >> Dockerfile
+            echo 'CMD ["make-<< parameters.ext >>-packages.sh"]' >> Dockerfile
+      - run:
+          name: Build << parameters.dist >> << parameters.release >> << parameters.arch >> << parameters.ext >>
+          command: |
+            mkdir -p packages
+            cp .circleci/make-<< parameters.ext >>-packages.sh .
+            docker build -t varnish-<< parameters.ext >>-package-build:<< parameters.arch >> .
+            docker run --rm -it -e PARAM_DIST=<< parameters.dist >> -e PARAM_RELEASE=<< parameters.release >> -v$(pwd):/varnish-cache varnish-<< parameters.ext >>-package-build:<< parameters.arch >>
+      - run:
+          name: List created packages
+          command: find ./packages -name "*.<< parameters.ext >>"
+      - persist_to_workspace:
+          root: .
+          paths:
+            - "packages"
   distcheck:
     parameters:
       dist:
@@ -87,7 +143,7 @@ jobs:
     working_directory: /workspace
     steps:
       - run:
-          name: Possibly activate centos:8 extra repos
+          name: Possible << parameters.dist >>:<< parameters.release >> extra repos
           command: |
             if [ << parameters.dist >> = centos ]; then
                 if [ << parameters.release >> = 8 ]; then
@@ -150,13 +206,14 @@ jobs:
       - run:
           name: Extract and distcheck
           command: |
-            tar xavf *.tar.gz --strip 1
+            tar xavf varnish-*.tar.gz --strip 1
             if [ << parameters.dist >> = centos ]; then
             	adduser varnish
             else
             	adduser --disabled-password --gecos "" varnish
             fi
-            chown -R varnish:varnish /workspace
+
+            chown -R varnish:varnish .
 
             export ASAN_OPTIONS=abort_on_error=1,detect_odr_violation=1,detect_leaks=1,detect_stack_use_after_return=1,detect_invalid_pointer_pairs=1,handle_segv=0,handle_sigbus=0,use_sigaltstack=0,disable_coredump=0
             export LSAN_OPTIONS=abort_on_error=1,use_sigaltstack=0,suppressions=$(pwd)/tools/lsan.suppr
@@ -174,285 +231,28 @@ jobs:
             sudo -u varnish \
             	--preserve-env=ASAN_OPTIONS,LSAN_OPTIONS,TSAN_OPTIONS,UBSAN_OPTIONS \
             	make distcheck VERBOSE=1 -j 12 -k
-  build_apks:
-    parameters:
-      dist:
-        description: the Linux distribution (alpine)
-        default: alpine
-        type: string
-      release:
-        description: the release version
-        default: latest
-        type: string
-    description: Build alpine apks
-    docker:
-      - image: alpine
-    working_directory: /workspace
-    steps:
-      - run:
-          name: Install certificates to mount the workspace, and tar
-          command: |
-            apk update
-            apk add -q ca-certificates tar
-      - attach_workspace:
-          at: /workspace
-      - run:
-          name: Untar alpine
-          command: |
-            tar xavf alpine.tar.gz --strip 1
-      - run:
-          name: Install sdk, add user
-          command: |
-            apk add alpine-sdk
-            adduser -D builder
-            echo "builder ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers
-            addgroup builder abuild
-            mkdir -p /var/cache/distfiles
-            chmod a+w /var/cache/distfiles
-      - run:
-          name: Generate key
-          command: |
-            su builder -c "abuild-keygen -nai"
-      - run:
-          name: Fix APKBUILD's variables
-          command: |
-            tar xavf varnish-*.tar.gz
-            VERSION=$(varnish-*/configure --version | awk 'NR == 1 {print $NF}')
-            sed -i "s/@VERSION@/$VERSION/" APKBUILD
-            rm -rf varnish-*/
-      - run:
-          name: Fix checksums, build
-          command: |
-            chown builder -R /workspace
-            su builder -c "abuild checksum"
-            su builder -c "abuild -r"
-      - run:
-          name: Fix the APKBUILD's version
-          command: |
-            mkdir apks
-            cp /home/builder/packages/x86_64/*.apk apks
-      - run:
-          name: Import the packages into the workspace
-          command: |
-            mkdir -p packages/<< parameters.dist >>/<< parameters.release >>
-            mv /home/builder/packages/x86_64/*.apk packages/<< parameters.dist >>/<< parameters.release >>
-      - persist_to_workspace:
-          root: .
-          paths:
-            - packages/<< parameters.dist >>/<< parameters.release >>/varnish*
-      - store_artifacts:
-          path: packages/<< parameters.dist >>/<< parameters.release >>
-          destination: ./
-  build_debs:
-    parameters:
-      dist:
-        description: the Linux distribution (debian|ubuntu)
-        type: string
-      release:
-        description: the release name (stretch|buster|xenial|bionic)
-        type: string
-    description: Build << parameters.release >> debs
-    docker:
-      - image: << parameters.dist >>:<< parameters.release >>
-    steps:
-      - run:
-          name: Install packaging tools
-          command: |
-            apt-get update
-            apt-get install -y dpkg-dev ca-certificates debhelper devscripts equivs
-      - attach_workspace:
-          at: ~/project
-      - run:
-          name: Untar debian
-          command: tar xavf debian.tar.gz
-      - run:
-          name: Untar orig
-          command: tar xavf varnish*.tar.gz --strip 1
-      - run:
-          name: Update changelog version
-          command: |
-            if [ -e .is_weekly ]; then
-                WEEKLY='-weekly'
-            else
-                WEEKLY=
-            fi
-            VERSION=$(./configure --version | awk 'NR == 1 {print $NF}')$WEEKLY~<< parameters.release >>
-            sed -i -e "s|@VERSION@|$VERSION-1|"  "debian/changelog"
-      - run:
-          name: Install Build-Depends packages
-          command: |
-            export DEBIAN_FRONTEND=noninteractive
-            export DEBCONF_NONINTERACTIVE_SEEN=true
-            yes | mk-build-deps --install debian/control || true
-      - run:
-          name: Build the packages
-          command: |
-            dpkg-buildpackage -us -uc -j16
-      - run:
-          name: Import the packages into the workspace
-          command: |
-            mkdir -p packages/<< parameters.dist >>/<< parameters.release >>
-            mv ../*.deb ../*.dsc packages/<< parameters.dist >>/<< parameters.release >>
-      - persist_to_workspace:
-          root: .
-          paths:
-            - packages/<< parameters.dist >>/<< parameters.release >>/varnish*
-      - store_artifacts:
-          path: packages/<< parameters.dist >>/<< parameters.release >>
-          destination: ./
-  build_rpms:
-    parameters:
-      dist:
-        description: the Linux distribution (centos|amazonlinux)
-        type: string
-      release:
-        description: the distribution version (7|8|2)
-        type: string
-    description: Build << parameters.dist >>:<< parameters.release >> rpms
-    docker:
-      - image: centos:<< parameters.release >>
-    environment:
-      DIST_DIR: build
-    steps:
-      - run:
-          name: Install packaging tools
-          command: |
-            yum install -y rpm-build yum-utils epel-release
-            # XXX: we should NOT have to do that here, they should be in the
-            # spec as BuildRequires
-            yum install -y make gcc
-      - attach_workspace:
-          at: ~/project
-      - run:
-          name: Create build dir
-          command: mkdir $DIST_DIR
-      - run:
-          name: Untar redhat
-          command: |
-            tar xavf redhat.tar.gz -C build
-      - run:
-          name: Untar orig
-          command: |
-            tar xavf varnish*.tar.gz -C build --strip 1
-      - run:
-          name: Build Packages
-          command: |
-            set -e
-            set -u
-
-            if [ << parameters.release >> = 8 ]; then
-                yum config-manager --set-enabled PowerTools
-            fi
-            # use python3
-            sed -i '1 i\%global __python %{__python3}' "$DIST_DIR"/redhat/varnish.spec
-            if [ -e .is_weekly ]; then
-                WEEKLY='.weekly'
-            else
-                WEEKLY=
-            fi
-            VERSION=$("$DIST_DIR"/configure --version | awk 'NR == 1 {print $NF}')$WEEKLY
 
-            cp -r -L "$DIST_DIR"/redhat/* "$DIST_DIR"/
-            tar zcf "$DIST_DIR.tgz" --exclude "$DIST_DIR/redhat" "$DIST_DIR"/
-
-            RPMVERSION="$VERSION"
-
-            RESULT_DIR="rpms"
-            CUR_DIR="$(pwd)"
-
-            rpmbuild() {
-            	command rpmbuild \
-            		--define "_smp_mflags -j10" \
-                	--define "_sourcedir $CUR_DIR" \
-                	--define "_srcrpmdir $CUR_DIR/${RESULT_DIR}" \
-                	--define "_rpmdir $CUR_DIR/${RESULT_DIR}" \
-            		--define "versiontag ${RPMVERSION}" \
-            		--define "releasetag 0.0" \
-            		--define "srcname $DIST_DIR" \
-            		--define "nocheck 1" \
-            		"$@"
-            }
-            yum-builddep -y "$DIST_DIR"/redhat/varnish.spec
-            rpmbuild -bs "$DIST_DIR"/redhat/varnish.spec
-            rpmbuild --rebuild "$RESULT_DIR"/varnish-*.src.rpm
-      - run:
-          name: Prepare the packages for storage
-          command: |
-            mkdir -p packages/<< parameters.dist >>/<< parameters.release >>
-            mv rpms/*/*.rpm packages/<< parameters.dist >>/<< parameters.release >>
-      - persist_to_workspace:
-          root: .
-          paths:
-            - packages/<< parameters.dist >>/<< parameters.release >>/*.rpm
-      - store_artifacts:
-          path: packages/<< parameters.dist >>/<< parameters.release >>
-          destination: ./
   collect_packages:
     docker:
       - image: centos:7
     steps:
       - attach_workspace:
           at: ~/project
+      - run: ls -la ~/project/
       - run:
           name: Tar the packages
           command: |
-              tar cvzf packages.tar.gz packages 
+              tar cvzf packages.tar.gz packages
       - store_artifacts:
           destination: packages.tar.gz
           path: packages.tar.gz
-pkg_req: &pkg_req
-  requires:
-    - dist
-    - tar_pkg_tools
+
 
 workflows:
   version: 2
-  build:
+  commit:
     jobs:
       - dist
-      - tar_pkg_tools
-      - build_debs:
-          name: build_debian_stretch
-          dist: debian
-          release: stretch
-          <<: *pkg_req
-      - build_debs:
-          name: build_debian_buster
-          dist: debian
-          release: buster
-          <<: *pkg_req
-      - build_debs:
-          name: build_ubuntu_xenial
-          dist: ubuntu
-          release: xenial
-          <<: *pkg_req
-      - build_debs:
-          name: build_ubuntu_bionic
-          dist: ubuntu
-          release: bionic
-          <<: *pkg_req
-      - build_rpms:
-          name: build_centos_7
-          dist: centos
-          release: "7"
-          <<: *pkg_req
-      - build_rpms:
-          name: build_centos_8
-          dist: centos
-          release: "8"
-          <<: *pkg_req
-      - build_apks:
-          name: build_alpine
-          <<: *pkg_req
-      - collect_packages:
-          requires:
-            - build_debian_stretch
-            - build_debian_buster
-            - build_ubuntu_xenial
-            - build_ubuntu_bionic
-            - build_centos_7
-            - build_centos_8
-            - build_alpine
       - distcheck:
           name: distcheck_centos_7
           dist: centos
@@ -479,3 +279,161 @@ workflows:
           #extra_conf: --without-jemalloc
           requires:
             - dist
+  nightly:
+    triggers:
+      - schedule:
+          cron: "0 4 * * *"
+          filters:
+            branches:
+              only:
+                - master
+    jobs:
+      - dist
+      - tar_pkg_tools
+      - package:
+          name: aarch64-ubuntu-bionic
+          dist: ubuntu
+          release: bionic
+          arch: aarch64
+          image: arm64v8/ubuntu:bionic
+          ext: deb
+          <<: *pkg_req
+      - package:
+          name: x64-ubuntu-bionic
+          dist: ubuntu
+          release: bionic
+          arch: x64
+          image: ubuntu:bionic
+          ext: deb
+          <<: *pkg_req
+      - package:
+          name: aarch64-ubuntu-xenial
+          dist: ubuntu
+          release: xenial
+          arch: aarch64
+          image: arm64v8/ubuntu:xenial
+          ext: deb
+          <<: *pkg_req
+      - package:
+          name: x64-ubuntu-xenial
+          dist: ubuntu
+          release: xenial
+          arch: x64
+          image: ubuntu:xenial
+          ext: deb
+          <<: *pkg_req
+      - package:
+          name: aarch64-debian-bullseye
+          dist: debian
+          release: bullseye
+          arch: aarch64
+          image: arm64v8/debian:bullseye-slim
+          ext: deb
+          <<: *pkg_req
+      - package:
+          name: x64-debian-bullseye
+          dist: debian
+          release: bullseye
+          arch: x64
+          image: debian:bullseye-slim
+          ext: deb
+          <<: *pkg_req
+      - package:
+          name: aarch64-debian-buster
+          dist: debian
+          release: buster
+          arch: aarch64
+          image: arm64v8/debian:buster-slim
+          ext: deb
+          <<: *pkg_req
+      - package:
+          name: x64-debian-buster
+          dist: debian
+          release: buster
+          arch: x64
+          image: debian:buster-slim
+          ext: deb
+          <<: *pkg_req
+      - package:
+          name: aarch64-debian-stretch
+          dist: debian
+          release: stretch
+          arch: aarch64
+          image: arm64v8/debian:stretch-slim
+          ext: deb
+          <<: *pkg_req
+      - package:
+          name: x64-debian-stretch
+          dist: debian
+          release: stretch
+          arch: x64
+          image: debian:stretch-slim
+          ext: deb
+          <<: *pkg_req
+      - package:
+          name: aarch64-centos-7
+          dist: centos
+          release: "7"
+          arch: aarch64
+          image: arm64v8/centos:7
+          ext: rpm
+          <<: *pkg_req
+      - package:
+          name: x64-centos-7
+          dist: centos
+          release: "7"
+          arch: x64
+          image: centos:7
+          ext: rpm
+          <<: *pkg_req
+      - package:
+          name: aarch64-centos-8
+          dist: centos
+          release: "8"
+          arch: aarch64
+          image: arm64v8/centos:8
+          ext: rpm
+          <<: *pkg_req
+      - package:
+          name: x64-centos-8
+          dist: centos
+          release: "8"
+          arch: x64
+          image: centos:8
+          ext: rpm
+          <<: *pkg_req
+      - package:
+          name: x64-alpine-3
+          dist: alpine
+          release: "3"
+          arch: x64
+          image: alpine:3
+          ext: apk
+          <<: *pkg_req
+      - package:
+          name: aarch64-alpine-3
+          dist: alpine
+          release: "3"
+          arch: aarch64
+          image: arm64v8/alpine:3
+          ext: apk
+          <<: *pkg_req
+      - collect_packages:
+          requires:
+            - x64-ubuntu-xenial
+            - x64-ubuntu-bionic
+            - aarch64-ubuntu-xenial
+            - aarch64-ubuntu-bionic
+            - x64-debian-bullseye
+            - aarch64-debian-bullseye
+            - x64-debian-buster
+            - aarch64-debian-buster
+            - x64-debian-stretch
+            - aarch64-debian-stretch
+            - x64-centos-7
+            - x64-centos-8
+            - aarch64-centos-7
+            - aarch64-centos-8
+            - x64-alpine-3
+            - aarch64-alpine-3
+
diff --git a/.circleci/make-apk-packages.sh b/.circleci/make-apk-packages.sh
new file mode 100755
index 000000000..26b672c2c
--- /dev/null
+++ b/.circleci/make-apk-packages.sh
@@ -0,0 +1,53 @@
+#!/usr/bin/env sh
+
+set -eux
+
+apk add -q --no-progress --update tar alpine-sdk
+
+echo "PARAM_RELEASE: $PARAM_RELEASE"
+echo "PARAM_DIST: $PARAM_DIST"
+
+if [ -z "$PARAM_RELEASE" ]; then
+    echo "Env variable PARAM_RELEASE is not set! For example PARAM_RELEASE=8, for CentOS 8"
+    exit 1
+elif [ -z "$PARAM_DIST" ]; then
+    echo "Env variable PARAM_DIST is not set! For example PARAM_DIST=centos"
+    exit 1
+fi
+
+cd /varnish-cache
+tar xazf alpine.tar.gz --strip 1
+
+adduser -D builder
+echo "builder ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers
+addgroup builder abuild
+mkdir -p /var/cache/distfiles
+chmod -R a+w /var/cache/distfiles
+
+echo "Generate key"
+su builder -c "abuild-keygen -nai"
+
+echo "Fix APKBUILD's variables"
+tar xavf varnish-*.tar.gz
+VERSION=$(varnish-*/configure --version | awk 'NR == 1 {print $NF}')
+echo "Version: $VERSION"
+sed -i "s/@VERSION@/$VERSION/" APKBUILD
+rm -rf varnish-*/
+
+echo "Change the ownership so that abuild is able to write its logs"
+chown builder -R .
+echo "Fix checksums, build"
+su builder -c "abuild checksum"
+su builder -c "abuild -r"
+
+echo "Fix the APKBUILD's version"
+su builder -c "mkdir apks"
+ARCH=`uname -m`
+su builder -c "cp /home/builder/packages/$ARCH/*.apk apks"
+
+echo "Import the packages into the workspace"
+mkdir -p packages/$PARAM_DIST/$PARAM_RELEASE/$ARCH/
+mv /home/builder/packages/$ARCH/*.apk packages/$PARAM_DIST/$PARAM_RELEASE/$ARCH/
+
+echo "Allow to read the packages by 'circleci' user outside of Docker after 'chown builder -R .' above"
+chmod -R a+rwx .
diff --git a/.circleci/make-deb-packages.sh b/.circleci/make-deb-packages.sh
new file mode 100755
index 000000000..ba4126f93
--- /dev/null
+++ b/.circleci/make-deb-packages.sh
@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+
+set -eux
+
+export DEBIAN_FRONTEND=noninteractive
+export DEBCONF_NONINTERACTIVE_SEEN=true
+apt-get update
+apt-get install -y dpkg-dev debhelper devscripts equivs pkg-config apt-utils
+
+echo "PARAM_RELEASE: $PARAM_RELEASE"
+echo "PARAM_DIST: $PARAM_DIST"
+
+
+if [ -z "$PARAM_RELEASE" ]; then
+    echo "Env variable PARAM_RELEASE is not set! For example PARAM_RELEASE=8, for CentOS 8"
+    exit 1
+elif [ -z "$PARAM_DIST" ]; then
+    echo "Env variable PARAM_DIST is not set! For example PARAM_DIST=centos"
+    exit 1
+fi
+
+cd /varnish-cache
+ls -la
+
+echo "Untar debian..."
+tar xavf debian.tar.gz
+
+echo "Untar orig..."
+tar xavf varnish-*.tar.gz --strip 1
+
+echo "Update changelog version..."
+if [ -e .is_weekly ]; then
+    WEEKLY='-weekly'
+else
+    WEEKLY=
+fi
+VERSION=$(./configure --version | awk 'NR == 1 {print $NF}')$WEEKLY~$PARAM_RELEASE
+sed -i -e "s|@VERSION@|$VERSION-1|"  "debian/changelog"
+
+echo "Install Build-Depends packages..."
+yes | mk-build-deps --install debian/control || true
+
+echo "Build the packages..."
+dpkg-buildpackage -us -uc -j16
+
+echo "Prepare the packages for storage..."
+mkdir -p packages/$PARAM_DIST/$PARAM_RELEASE/
+mv ../*.deb packages/$PARAM_DIST/$PARAM_RELEASE/
diff --git a/.circleci/make-rpm-packages.sh b/.circleci/make-rpm-packages.sh
new file mode 100755
index 000000000..b443c9139
--- /dev/null
+++ b/.circleci/make-rpm-packages.sh
@@ -0,0 +1,75 @@
+#!/usr/bin/env bash
+
+set -eux
+
+echo "PARAM_RELEASE: $PARAM_RELEASE"
+echo "PARAM_DIST: $PARAM_DIST"
+
+if [ -z "$PARAM_RELEASE" ]; then
+    echo "Env variable PARAM_RELEASE is not set! For example PARAM_RELEASE=8, for CentOS 8"
+    exit 1
+elif [ -z "$PARAM_DIST" ]; then
+    echo "Env variable PARAM_DIST is not set! For example PARAM_DIST=centos"
+    exit 1
+fi
+
+yum install -y epel-release
+
+if [ "$PARAM_DIST" = centos ]; then
+  if [ "$PARAM_RELEASE" = 8 ]; then
+      dnf install -y 'dnf-command(config-manager)'
+      yum config-manager --set-enabled PowerTools
+  fi
+fi
+
+yum install -y rpm-build yum-utils
+
+export DIST_DIR=build
+
+cd /varnish-cache
+rm -rf $DIST_DIR
+mkdir $DIST_DIR
+
+
+echo "Untar redhat..."
+tar xavf redhat.tar.gz -C $DIST_DIR
+
+echo "Untar orig..."
+tar xavf varnish-*.tar.gz -C $DIST_DIR --strip 1
+
+echo "Build Packages..."
+if [ -e .is_weekly ]; then
+    WEEKLY='.weekly'
+else
+    WEEKLY=
+fi
+VERSION=$("$DIST_DIR"/configure --version | awk 'NR == 1 {print $NF}')$WEEKLY
+
+cp -r -L "$DIST_DIR"/redhat/* "$DIST_DIR"/
+tar zcf "$DIST_DIR.tgz" --exclude "$DIST_DIR/redhat" "$DIST_DIR"/
+
+RPMVERSION="$VERSION"
+
+RESULT_DIR="rpms"
+CUR_DIR="$(pwd)"
+
+rpmbuild() {
+    command rpmbuild \
+        --define "_smp_mflags -j10" \
+        --define "_sourcedir $CUR_DIR" \
+        --define "_srcrpmdir $CUR_DIR/${RESULT_DIR}" \
+        --define "_rpmdir $CUR_DIR/${RESULT_DIR}" \
+        --define "versiontag ${RPMVERSION}" \
+        --define "releasetag 0.0" \
+        --define "srcname $DIST_DIR" \
+        --define "nocheck 1" \
+        "$@"
+}
+
+yum-builddep -y "$DIST_DIR"/redhat/varnish.spec
+rpmbuild -bs "$DIST_DIR"/redhat/varnish.spec
+rpmbuild --rebuild "$RESULT_DIR"/varnish-*.src.rpm
+
+echo "Prepare the packages for storage..."
+mkdir -p packages/$PARAM_DIST/$PARAM_RELEASE/
+mv rpms/*/*.rpm packages/$PARAM_DIST/$PARAM_RELEASE/

From gquintard at users.noreply.github.com  Fri May 15 13:55:06 2020
From: gquintard at users.noreply.github.com (guillaume quintard)
Date: Fri, 15 May 2020 13:55:06 +0000 (UTC)
Subject: [master] 6d0a7280a Issue pkg-varnish-cache/issues/139 - Add package
 for Ubuntu 20.04
Message-ID: <20200515135506.A82D49159B@lists.varnish-cache.org>


commit 6d0a7280a712e323af102c99838b920a5668ba5a
Author: Martin Tzvetanov Grigorov <mgrigorov at apache.org>
Date:   Fri May 15 13:49:40 2020 +0300

    Issue pkg-varnish-cache/issues/139 - Add package for Ubuntu 20.04

diff --git a/.circleci/config.yml b/.circleci/config.yml
index 0ede16afa..fc6ce85c4 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -290,6 +290,22 @@ workflows:
     jobs:
       - dist
       - tar_pkg_tools
+      - package:
+          name: aarch64-ubuntu-focal
+          dist: ubuntu
+          release: focal
+          arch: aarch64
+          image: arm64v8/ubuntu:focal
+          ext: deb
+          <<: *pkg_req
+      - package:
+          name: x64-ubuntu-focal
+          dist: ubuntu
+          release: focal
+          arch: x64
+          image: ubuntu:focal
+          ext: deb
+          <<: *pkg_req
       - package:
           name: aarch64-ubuntu-bionic
           dist: ubuntu
@@ -422,8 +438,10 @@ workflows:
           requires:
             - x64-ubuntu-xenial
             - x64-ubuntu-bionic
+            - x64-ubuntu-focal
             - aarch64-ubuntu-xenial
             - aarch64-ubuntu-bionic
+            - aarch64-ubuntu-focal
             - x64-debian-bullseye
             - aarch64-debian-bullseye
             - x64-debian-buster

From gquintard at users.noreply.github.com  Fri May 15 13:55:06 2020
From: gquintard at users.noreply.github.com (guillaume quintard)
Date: Fri, 15 May 2020 13:55:06 +0000 (UTC)
Subject: [master] 09532c18b Add some kind of order to the list of required
 jobs for collect_packages
Message-ID: <20200515135506.C7F3D9159E@lists.varnish-cache.org>


commit 09532c18b9a565618aafc56194db4fbf0c3adae1
Author: Martin Tzvetanov Grigorov <mgrigorov at apache.org>
Date:   Fri May 15 15:21:00 2020 +0300

    Add some kind of order to the list of required jobs for collect_packages

diff --git a/.circleci/config.yml b/.circleci/config.yml
index fc6ce85c4..ee4c63b37 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -437,20 +437,20 @@ workflows:
       - collect_packages:
           requires:
             - x64-ubuntu-xenial
-            - x64-ubuntu-bionic
-            - x64-ubuntu-focal
             - aarch64-ubuntu-xenial
+            - x64-ubuntu-bionic
             - aarch64-ubuntu-bionic
+            - x64-ubuntu-focal
             - aarch64-ubuntu-focal
-            - x64-debian-bullseye
-            - aarch64-debian-bullseye
-            - x64-debian-buster
-            - aarch64-debian-buster
             - x64-debian-stretch
             - aarch64-debian-stretch
+            - x64-debian-buster
+            - aarch64-debian-buster
+            - x64-debian-bullseye
+            - aarch64-debian-bullseye
             - x64-centos-7
-            - x64-centos-8
             - aarch64-centos-7
+            - x64-centos-8
             - aarch64-centos-8
             - x64-alpine-3
             - aarch64-alpine-3

From dridi.boukelmoune at gmail.com  Mon May 18 20:23:08 2020
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 18 May 2020 20:23:08 +0000 (UTC)
Subject: [master] 48845c95b Accommodate the musl libc in c3.vtc
Message-ID: <20200518202308.84B5264381@lists.varnish-cache.org>


commit 48845c95bcf30769a6ab8d7eb83850791efd396f
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Mon May 18 22:21:11 2020 +0200

    Accommodate the musl libc in c3.vtc

diff --git a/bin/varnishtest/tests/c00003.vtc b/bin/varnishtest/tests/c00003.vtc
index 1c369bb07..d5a8ef9d4 100644
--- a/bin/varnishtest/tests/c00003.vtc
+++ b/bin/varnishtest/tests/c00003.vtc
@@ -2,11 +2,11 @@ varnishtest "Check that we fail to start with erroneous -a/-b arguments"
 
 # Duplicate -a arguments
 # XXX: this cannot be tested reliably, we tolerate port collision
-shell -err -match "have same address|already in use" {
+shell -err -match "have same address|already in use|Address in use" {
 	varnishd -d -a 127.0.0.1:38484 -a 127.0.0.1:38484 -b localhost:80
 }
 
-shell -err -match "have same address|already in use" {
+shell -err -match "have same address|already in use|Address in use" {
 	varnishd -d -a ${tmpdir}/vtc.sock -a ${tmpdir}/vtc.sock -b localhost:80
 }
 

From nils.goroll at uplex.de  Tue May 19 06:42:06 2020
From: nils.goroll at uplex.de (Nils Goroll)
Date: Tue, 19 May 2020 06:42:06 +0000 (UTC)
Subject: [master] 501fb9567 emacs: tabs mode only applies to c sources
Message-ID: <20200519064206.582B2A378A@lists.varnish-cache.org>


commit 501fb9567980b5e30bbcc844bd7f52640f646b35
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Tue May 19 08:38:17 2020 +0200

    emacs: tabs mode only applies to c sources
    
    our python sources use spaces

diff --git a/.dir-locals.el b/.dir-locals.el
index 143ac8abc..fd85435c6 100644
--- a/.dir-locals.el
+++ b/.dir-locals.el
@@ -1,2 +1,2 @@
-((nil . ((indent-tabs-mode . t)))
- (c-mode . ((c-file-style . "BSD"))))
+((c-mode . ((indent-tabs-mode . t)
+	    (c-file-style . "BSD"))))

From nils.goroll at uplex.de  Tue May 19 07:27:07 2020
From: nils.goroll at uplex.de (Nils Goroll)
Date: Tue, 19 May 2020 07:27:07 +0000 (UTC)
Subject: [master] 134d06333 vmodtool: support multiple possible input files
Message-ID: <20200519072707.4A663A4B19@lists.varnish-cache.org>


commit 134d063337b6f9b4260ec56feec552807c7fd090
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Tue May 19 09:23:23 2020 +0200

    vmodtool: support multiple possible input files
    
    This is to support generated vcc files with out-of-tree vmod builds.

diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py
index 87a9ce4b9..61a695b70 100755
--- a/lib/libvcc/vmodtool.py
+++ b/lib/libvcc/vmodtool.py
@@ -29,11 +29,13 @@
 # SUCH DAMAGE.
 
 """
-Read the vmod.vcc file (inputvcc) and produce:
-    vcc_if.h -- Prototypes for the implementation
-    vcc_if.c -- Magic glue & datastructures to make things a VMOD.
+Read the first existing file from arguments or vmod.vcc and produce:
+    ${prefix}.h -- Prototypes for the implementation
+    ${prefix}.c -- Magic glue & datastructures to make things a VMOD.
     vmod_${name}.rst -- Extracted documentation
     vmod_${name}.man.rst -- Extracted documentation (rst2man input)
+
+    prefix can set via -o and defaults to vcc_if
 """
 
 import copy
@@ -1189,12 +1191,13 @@ if __name__ == "__main__":
     (opts, args) = oparser.parse_args()
 
     i_vcc = None
-    if len(args) == 1 and os.path.exists(args[0]):
-        i_vcc = args[0]
-    elif os.path.exists("vmod.vcc"):
-        if not i_vcc:
-            i_vcc = "vmod.vcc"
-    else:
+    for f in args:
+        if os.path.exists(f):
+            i_vcc = f
+            break
+    if i_vcc is None and os.path.exists("vmod.vcc"):
+        i_vcc = "vmod.vcc"
+    if i_vcc is None:
         print("ERROR: No vmod.vcc file supplied or found.", file=sys.stderr)
         oparser.print_help()
         exit(-1)
diff --git a/varnish.m4 b/varnish.m4
index 2ede52bfc..788dd3973 100644
--- a/varnish.m4
+++ b/varnish.m4
@@ -210,7 +210,8 @@ vmod_$1.lo: \$(nodist_libvmod_$1_la_SOURCES)
 vcc_$1_if.h vmod_$1.rst vmod_$1.man.rst: vcc_$1_if.c
 
 vcc_$1_if.c: vmod_$1.vcc
-	\$(A""M_V_VMODTOOL) \$(PYTHON) \$(VMODTOOL) -o vcc_$1_if \$(srcdir)/vmod_$1.vcc
+	\$(A""M_V_VMODTOOL) \$(PYTHON) \$(VMODTOOL) -o vcc_$1_if \\
+	  \$(builddir)/vmod_$1.vcc \$(srcdir)/vmod_$1.vcc
 
 vmod_$1.3: vmod_$1.man.rst
 	\$(A""M_V_GEN) \$(RST2MAN) vmod_$1.man.rst vmod_$1.3

From dridi.boukelmoune at gmail.com  Tue May 19 10:43:06 2020
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 19 May 2020 10:43:06 +0000 (UTC)
Subject: [master] e6335abff Whitespace OCD
Message-ID: <20200519104306.EF65EA9596@lists.varnish-cache.org>


commit e6335abffd56b1ef73268bd190ece540ff948ea9
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue May 19 10:50:08 2020 +0200

    Whitespace OCD

diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index 4a770b444..2a1903957 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -166,8 +166,7 @@ Resp_Setup_Synth(struct req *req)
 
 	http_TimeHeader(h, "Date: ", W_TIM_real(req->wrk));
 	http_SetHeader(h, "Server: Varnish");
-	http_PrintfHeader(h, "X-Varnish: %u",
-	    VXID(req->vsl->wid));
+	http_PrintfHeader(h, "X-Varnish: %u", VXID(req->vsl->wid));
 
 	/*
 	 * For late 100-continue, we suggest to VCL to close the connection to

From dridi.boukelmoune at gmail.com  Tue May 19 10:43:07 2020
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 19 May 2020 10:43:07 +0000 (UTC)
Subject: [master] 04e1c1744 vmodtool: support only one input file
Message-ID: <20200519104307.0FB91A9599@lists.varnish-cache.org>


commit 04e1c1744893c1765c8e3fe1b46e9b4953241300
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue May 19 12:10:32 2020 +0200

    vmodtool: support only one input file
    
    That is, in the context of varnish.m4, because this changes existing
    behavior in a way that breaks backward compatibility.
    
    Partial revert of 134d063337b6f9b4260ec56feec552807c7fd090.

diff --git a/varnish.m4 b/varnish.m4
index 788dd3973..2ede52bfc 100644
--- a/varnish.m4
+++ b/varnish.m4
@@ -210,8 +210,7 @@ vmod_$1.lo: \$(nodist_libvmod_$1_la_SOURCES)
 vcc_$1_if.h vmod_$1.rst vmod_$1.man.rst: vcc_$1_if.c
 
 vcc_$1_if.c: vmod_$1.vcc
-	\$(A""M_V_VMODTOOL) \$(PYTHON) \$(VMODTOOL) -o vcc_$1_if \\
-	  \$(builddir)/vmod_$1.vcc \$(srcdir)/vmod_$1.vcc
+	\$(A""M_V_VMODTOOL) \$(PYTHON) \$(VMODTOOL) -o vcc_$1_if \$(srcdir)/vmod_$1.vcc
 
 vmod_$1.3: vmod_$1.man.rst
 	\$(A""M_V_GEN) \$(RST2MAN) vmod_$1.man.rst vmod_$1.3

From gquintard at users.noreply.github.com  Tue May 19 16:53:05 2020
From: gquintard at users.noreply.github.com (guillaume quintard)
Date: Tue, 19 May 2020 16:53:05 +0000 (UTC)
Subject: [master] 8b4021956 Use fakeroot-tcp for Debian/Ubuntu
Message-ID: <20200519165305.39062B185E@lists.varnish-cache.org>


commit 8b4021956aefee46e42ca5170bd213314e5d8b55
Author: Martin Tzvetanov Grigorov <mgrigorov at apache.org>
Date:   Tue May 19 12:17:22 2020 +0300

    Use fakeroot-tcp for Debian/Ubuntu
    
    There is a problem with fakeroot-sysv on Ubuntu 20.04 aarch64:
    
    Install Build-Depends packages...
    
    + mk-build-deps --install debian/control
    
    + yes
    
    semop(1): encountered an error: Function not implemented
    
    Error in the build process: exit status 1

diff --git a/.circleci/make-deb-packages.sh b/.circleci/make-deb-packages.sh
index ba4126f93..a2b84f223 100755
--- a/.circleci/make-deb-packages.sh
+++ b/.circleci/make-deb-packages.sh
@@ -5,7 +5,7 @@ set -eux
 export DEBIAN_FRONTEND=noninteractive
 export DEBCONF_NONINTERACTIVE_SEEN=true
 apt-get update
-apt-get install -y dpkg-dev debhelper devscripts equivs pkg-config apt-utils
+apt-get install -y dpkg-dev debhelper devscripts equivs pkg-config apt-utils fakeroot
 
 echo "PARAM_RELEASE: $PARAM_RELEASE"
 echo "PARAM_DIST: $PARAM_DIST"
@@ -19,6 +19,10 @@ elif [ -z "$PARAM_DIST" ]; then
     exit 1
 fi
 
+# Ubuntu 20.04 aarch64 fails when using fakeroot-sysv with:
+#    semop(1): encountered an error: Function not implemented
+update-alternatives --set fakeroot /usr/bin/fakeroot-tcp
+
 cd /varnish-cache
 ls -la
 

From reza at naghibi.com  Tue May 19 19:39:08 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Tue, 19 May 2020 19:39:08 +0000 (UTC)
Subject: [6.0] 84fe7970e Warn about too many (>100) VCLs.
Message-ID: <20200519193908.E996E6562@lists.varnish-cache.org>


commit 84fe7970e75adbaef15671d18777a0bf54b4240f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Aug 15 12:37:22 2018 +0000

    Warn about too many (>100) VCLs.
    
    Two new paramters allow the limit and than handling (ignore, warn, fail)
    to be configured.
    
    Fixes #2713

diff --git a/bin/varnishd/mgt/mgt_vcl.c b/bin/varnishd/mgt/mgt_vcl.c
index 049cee5af..c0fadf475 100644
--- a/bin/varnishd/mgt/mgt_vcl.c
+++ b/bin/varnishd/mgt/mgt_vcl.c
@@ -53,6 +53,7 @@
 #include "tbl/vcl_states.h"
 
 static const char * const VCL_STATE_LABEL = "label";
+static int vcl_count;
 
 struct vclprog;
 struct vmodfile;
@@ -252,6 +253,8 @@ mgt_vcl_add(const char *name, const char *state)
 		vp->warm = 1;
 
 	VTAILQ_INSERT_TAIL(&vclhead, vp, list);
+	if (vp->state != VCL_STATE_LABEL)
+		vcl_count++;
 	return (vp);
 }
 
@@ -269,6 +272,8 @@ mgt_vcl_del(struct vclprog *vp)
 		mgt_vcl_dep_del(VTAILQ_FIRST(&vp->dfrom));
 
 	VTAILQ_REMOVE(&vclhead, vp, list);
+	if (vp->state != VCL_STATE_LABEL)
+		vcl_count--;
 	if (vp->fname != NULL) {
 		if (!MGT_DO_DEBUG(DBG_VCL_KEEP))
 			AZ(unlink(vp->fname));
@@ -566,6 +571,12 @@ mgt_new_vcl(struct cli *cli, const char *vclname, const char *vclsrc,
 	if (C_flag) {
 		bprintf(buf, ".CflagTest.%d", (int)getpid());
 		vclname = buf;
+	} else if (vcl_count >= mgt_param.max_vcl &&
+	    mgt_param.max_vcl_handling == 2) {
+		VCLI_Out(cli, "Too many (%d) VCLs already loaded\n", vcl_count);
+		VCLI_Out(cli, "(See max_vcl and max_vcl_handling parameters)");
+		VCLI_SetResult(cli, CLIS_CANT);
+		return;
 	}
 
 	if (state == NULL)
@@ -592,6 +603,14 @@ mgt_new_vcl(struct cli *cli, const char *vclname, const char *vclsrc,
 	if (active_vcl == NULL)
 		active_vcl = vp;
 
+	if (cli->result == CLIS_OK &&
+	    vcl_count > mgt_param.max_vcl &&
+	    mgt_param.max_vcl_handling == 1) {
+		VCLI_Out(cli, "%d VCLs loaded\n", vcl_count);
+		VCLI_Out(cli, "Remember to vcl.discard the old/unused VCLs.\n");
+		VCLI_Out(cli, "(See max_vcl and max_vcl_handling parameters)");
+	}
+
 	if (!MCH_Running())
 		return;
 
diff --git a/bin/varnishtest/tests/v00025.vtc b/bin/varnishtest/tests/v00025.vtc
index 699137b0f..d3d0cf684 100644
--- a/bin/varnishtest/tests/v00025.vtc
+++ b/bin/varnishtest/tests/v00025.vtc
@@ -7,7 +7,23 @@ server s1 {
 	txresp
 } -start
 
-varnish v1 -syntax 4.0 -arg "-i J.F.Nobody" -vcl+backend {
+# Test max_vcl param
+varnish v1 -arg "-i J.F.Nobody" -vcl+backend { } -start
+
+varnish v1 -cliok "param.set max_vcl 2"
+varnish v1 -cliok "param.set max_vcl_handling 2"
+
+varnish v1 -vcl+backend { }
+varnish v1 -errvcl {Too many (2) VCLs already loaded} {}
+
+varnish v1 -cliok "param.set max_vcl_handling 1"
+
+varnish v1 -cliexpect "Remember to vcl.discard the old/unused VCLs." \
+	{vcl.inline foobar "vcl 4.1; backend b1 {.host=\"${s1_addr}\";}"}
+
+varnish v1 -cliok "param.set max_vcl 100"
+
+varnish v1 -syntax 4.0 -vcl+backend {
 
 	import std;
 	import directors;
@@ -78,7 +94,7 @@ varnish v1 -syntax 4.0 -arg "-i J.F.Nobody" -vcl+backend {
 		set bereq.connect_timeout = 10s;
 	}
 
-} -start
+}
 
 client c1 {
 	txreq
diff --git a/include/tbl/params.h b/include/tbl/params.h
index 71d425195..deecd2051 100644
--- a/include/tbl/params.h
+++ b/include/tbl/params.h
@@ -1537,6 +1537,38 @@ PARAM(
 	/* func */	NULL
 )
 
+PARAM(
+	/* name */	max_vcl_handling,
+	/* typ */	uint,
+	/* min */	"0",
+	/* max */	"2",
+	/* default */	"1",
+	/* units */	NULL,
+	/* flags */	0,
+	/* s-text */
+	"Behaviour when attempting to exceed max_vcl loaded VCL.\n"
+	"\n*  0 - Ignore max_vcl parameter.\n"
+	"\n*  1 - Issue warning.\n"
+	"\n*  2 - Refuse loading VCLs.",
+	/* l-text */	"",
+	/* func */	NULL
+)
+
+PARAM(
+	/* name */	max_vcl,
+	/* typ */	uint,
+	/* min */	"0",
+	/* max */	NULL,
+	/* default */	"100",
+	/* units */	NULL,
+	/* flags */	0,
+	/* s-text */
+	"Threshold of loaded VCL programs.  (VCL labels are not counted.)"
+	"  Parameter max_vcl_handling determines behaviour.",
+	/* l-text */	"",
+	/* func */	NULL
+)
+
 PARAM(
 	/* name */	vsm_free_cooldown,
 	/* typ */	timeout,

From reza at naghibi.com  Tue May 19 19:39:09 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Tue, 19 May 2020 19:39:09 +0000 (UTC)
Subject: [6.0] 87fbff317 Add convenience function VSB_tofile() to write a VSB
 to a fd.
Message-ID: <20200519193909.27A916564@lists.varnish-cache.org>


commit 87fbff3178695c8ae6b6baf4232d44f234742f56
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed May 22 06:55:20 2019 +0000

    Add convenience function VSB_tofile() to write a VSB to a fd.
    
    Conflicts:
        bin/varnishd/common/common_vsmw.c

diff --git a/bin/varnishd/common/common_vsmw.c b/bin/varnishd/common/common_vsmw.c
index 5149b8bfb..dcf83dbd3 100644
--- a/bin/varnishd/common/common_vsmw.c
+++ b/bin/varnishd/common/common_vsmw.c
@@ -198,7 +198,6 @@ static void
 vsmw_append_record(struct vsmw *vsmw, struct vsmwseg *seg, char act)
 {
 	int fd;
-	ssize_t s;
 
 	CHECK_OBJ_NOTNULL(vsmw, VSMW_MAGIC);
 	CHECK_OBJ_NOTNULL(seg, VSMWSEG_MAGIC);
@@ -207,9 +206,8 @@ vsmw_append_record(struct vsmw *vsmw, struct vsmwseg *seg, char act)
 	VSB_clear(vsmw->vsb);
 	vsmw_fmt_index(vsmw, seg, act);
 	AZ(VSB_finish(vsmw->vsb));
-	s = write(fd, VSB_data(vsmw->vsb), VSB_len(vsmw->vsb));
-	assert(s == VSB_len(vsmw->vsb)); // XXX handle ENOSPC? #2764
-	AZ(close(fd));
+	XXXAZ(VSB_tofile(fd, vsmw->vsb)); // XXX handle ENOSPC? #2764
+	closefd(&fd);
 }
 
 /*--------------------------------------------------------------------*/
@@ -230,7 +228,6 @@ static void
 vsmw_delseg(struct vsmw *vsmw, struct vsmwseg *seg)
 {
 	char *t = NULL;
-	ssize_t s;
 	int fd;
 	struct vsmwseg *s2;
 
@@ -256,9 +253,8 @@ vsmw_delseg(struct vsmw *vsmw, struct vsmwseg *seg)
 		VTAILQ_FOREACH(s2, &vsmw->segs, list)
 			vsmw_fmt_index(vsmw, s2, '+');
 		AZ(VSB_finish(vsmw->vsb));
-		s = write(fd, VSB_data(vsmw->vsb), VSB_len(vsmw->vsb));
-		assert(s == VSB_len(vsmw->vsb));
-		AZ(close(fd));
+		XXXAZ(VSB_tofile(fd, vsmw->vsb)); // XXX handle ENOSPC? #2764
+		closefd(&fd);
 		AZ(renameat(vsmw->vdirfd, t, vsmw->vdirfd, vsmw->idx));
 		REPLACE(t, NULL);
 		vsmw->nsubs = 0;
diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c
index 3677eb620..394664939 100644
--- a/bin/varnishd/mgt/mgt_cli.c
+++ b/bin/varnishd/mgt/mgt_cli.c
@@ -145,8 +145,7 @@ mcf_askchild(struct cli *cli, const char * const *av, void *priv)
 	}
 	VSB_putc(vsb, '\n');
 	AZ(VSB_finish(vsb));
-	i = write(cli_o, VSB_data(vsb), VSB_len(vsb));
-	if (i != VSB_len(vsb)) {
+	if (VSB_tofile(cli_o, vsb)) {
 		VSB_destroy(&vsb);
 		VCLI_SetResult(cli, CLIS_COMMS);
 		VCLI_Out(cli, "CLI communication error");
@@ -177,7 +176,7 @@ static struct cli_proto cli_askchild[] = {
 int
 mgt_cli_askchild(unsigned *status, char **resp, const char *fmt, ...)
 {
-	int i, j;
+	int i;
 	va_list ap;
 	unsigned u;
 
@@ -203,8 +202,7 @@ mgt_cli_askchild(unsigned *status, char **resp, const char *fmt, ...)
 	AZ(VSB_finish(cli_buf));
 	i = VSB_len(cli_buf);
 	assert(i > 0 && VSB_data(cli_buf)[i - 1] == '\n');
-	j = write(cli_o, VSB_data(cli_buf), i);
-	if (j != i) {
+	if (VSB_tofile(cli_o, cli_buf)) {
 		if (status != NULL)
 			*status = CLIS_COMMS;
 		if (resp != NULL)
diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c
index 64a3a2b08..ddcc8db77 100644
--- a/bin/varnishd/proxy/cache_proxy_proto.c
+++ b/bin/varnishd/proxy/cache_proxy_proto.c
@@ -667,7 +667,7 @@ VPX_Send_Proxy(int fd, int version, const struct sess *sp)
 		WRONG("Wrong proxy version");
 
 	AZ(VSB_finish(vsb));
-	(void)write(fd, VSB_data(vsb), VSB_len(vsb));
+	(void)VSB_tofile(fd, vsb);	// XXX: Error handling ?
 	if (!DO_DEBUG(DBG_PROTOCOL)) {
 		VSB_delete(vsb);
 		return;
diff --git a/bin/varnishtest/vtc_haproxy.c b/bin/varnishtest/vtc_haproxy.c
index 55e986f57..d6565b834 100644
--- a/bin/varnishtest/vtc_haproxy.c
+++ b/bin/varnishtest/vtc_haproxy.c
@@ -143,7 +143,6 @@ cmd_haproxy_cli_send(CMD_ARGS)
 {
 	struct vsb *vsb;
 	struct haproxy_cli *hc;
-	ssize_t wr;
 
 	(void)cmd;
 	(void)vl;
@@ -174,8 +173,7 @@ cmd_haproxy_cli_send(CMD_ARGS)
 	}
 	vtc_dump(hc->vl, 4, "CLI send", VSB_data(vsb), -1);
 
-	wr = write(hc->sock, VSB_data(vsb), VSB_len(vsb));
-	if (wr != VSB_len(vsb))
+	if (VSB_tofile(hc->sock, vsb))
 		vtc_fatal(hc->vl,
 		    "CLI fd %d send error %s", hc->sock, strerror(errno));
 
diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c
index e17643f8e..20969c27a 100644
--- a/bin/varnishtest/vtc_http.c
+++ b/bin/varnishtest/vtc_http.c
@@ -213,14 +213,12 @@ synth_body(const char *len, int rnd)
 static void
 http_write(const struct http *hp, int lvl, const char *pfx)
 {
-	ssize_t l;
 
 	AZ(VSB_finish(hp->vsb));
 	vtc_dump(hp->vl, lvl, pfx, VSB_data(hp->vsb), VSB_len(hp->vsb));
-	l = write(hp->fd, VSB_data(hp->vsb), VSB_len(hp->vsb));
-	if (l != VSB_len(hp->vsb))
-		vtc_log(hp->vl, hp->fatal, "Write failed: (%zd vs %zd) %s",
-		    l, VSB_len(hp->vsb), strerror(errno));
+	if (VSB_tofile(hp->fd, hp->vsb))
+		vtc_log(hp->vl, hp->fatal, "Write failed: %s",
+		    strerror(errno));
 }
 
 /**********************************************************************
@@ -1417,7 +1415,6 @@ cmd_http_sendhex(CMD_ARGS)
 {
 	struct vsb *vsb;
 	struct http *hp;
-	int j;
 
 	(void)cmd;
 	(void)vl;
@@ -1427,8 +1424,9 @@ cmd_http_sendhex(CMD_ARGS)
 	vsb = vtc_hex_to_bin(hp->vl, av[1]);
 	assert(VSB_len(vsb) >= 0);
 	vtc_hexdump(hp->vl, 4, "sendhex", VSB_data(vsb), VSB_len(vsb));
-	j = write(hp->fd, VSB_data(vsb), VSB_len(vsb));
-	assert(j == VSB_len(vsb));
+	if (VSB_tofile(hp->fd, vsb))
+		vtc_log(hp->vl, hp->fatal, "Write failed: %s",
+		    strerror(errno));
 	VSB_destroy(&vsb);
 }
 
diff --git a/bin/varnishtest/vtc_process.c b/bin/varnishtest/vtc_process.c
index 10d545b38..ae98f53e1 100644
--- a/bin/varnishtest/vtc_process.c
+++ b/bin/varnishtest/vtc_process.c
@@ -786,7 +786,6 @@ static void
 process_write_hex(const struct process *p, const char *text)
 {
 	struct vsb *vsb;
-	int j;
 
 	if (!p->hasthread)
 		vtc_fatal(p->vl, "Cannot write to a non-running process");
@@ -794,8 +793,7 @@ process_write_hex(const struct process *p, const char *text)
 	vsb = vtc_hex_to_bin(p->vl, text);
 	assert(VSB_len(vsb) >= 0);
 	vtc_hexdump(p->vl, 4, "sendhex", VSB_data(vsb), VSB_len(vsb));
-	j = write(p->fd_term, VSB_data(vsb), VSB_len(vsb));
-	assert(j == VSB_len(vsb));
+	AZ(VSB_tofile(p->fd_term, vsb));
 	VSB_destroy(&vsb);
 }
 
diff --git a/bin/varnishtest/vtc_proxy.c b/bin/varnishtest/vtc_proxy.c
index 03e186b57..4477ab056 100644
--- a/bin/varnishtest/vtc_proxy.c
+++ b/bin/varnishtest/vtc_proxy.c
@@ -84,7 +84,7 @@ vtc_send_proxy(int fd, int version, const struct suckaddr *sac,
 	char pc[VTCP_PORTBUFSIZE];
 	char hs[VTCP_ADDRBUFSIZE];
 	char ps[VTCP_PORTBUFSIZE];
-	int i, len;
+	int i;
 	int proto;
 
 	AN(sac);
@@ -126,8 +126,7 @@ vtc_send_proxy(int fd, int version, const struct suckaddr *sac,
 		WRONG("Wrong proxy version");
 
 	AZ(VSB_finish(vsb));
-	len = VSB_len(vsb);
-	i = write(fd, VSB_data(vsb), len);
+	i = VSB_tofile(fd, vsb);
 	VSB_delete(vsb);
-	return (i != len);
+	return (i);
 }
diff --git a/include/vsb.h b/include/vsb.h
index d51088499..10e4c9563 100644
--- a/include/vsb.h
+++ b/include/vsb.h
@@ -85,6 +85,7 @@ void		 VSB_quote_pfx(struct vsb *, const char*, const void *,
 		     int len, int how);
 void		 VSB_quote(struct vsb *, const void *, int len, int how);
 void		 VSB_indent(struct vsb *, int);
+int		 VSB_tofile(int fd, const struct vsb *);
 #ifdef __cplusplus
 };
 #endif
diff --git a/lib/libvarnish/vsb.c b/lib/libvarnish/vsb.c
index 1261ee57d..e6519e98c 100644
--- a/lib/libvarnish/vsb.c
+++ b/lib/libvarnish/vsb.c
@@ -35,6 +35,7 @@ __FBSDID("$FreeBSD: head/sys/kern/subr_vsb.c 222004 2011-05-17 06:36:32Z phk $")
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
+#include <unistd.h>
 
 #include "vdef.h"
 #include "vas.h"	// XXX Flexelint "not used" - but req'ed for assert()
@@ -614,7 +615,7 @@ VSB_quote(struct vsb *s, const void *v, int len, int how)
  */
 
 void
-VSB_indent(struct vsb * s, int i)
+VSB_indent(struct vsb *s, int i)
 {
 
 	assert_VSB_integrity(s);
@@ -623,3 +624,14 @@ VSB_indent(struct vsb * s, int i)
 	else
 		s->s_indent += i;
 }
+
+int
+VSB_tofile(int fd, const struct vsb *s)
+{
+	int sz;
+
+	assert_VSB_integrity(s);
+	assert_VSB_state(s, VSB_FINISHED);
+	sz = write(fd, VSB_data(s), VSB_len(s));
+	return (sz == VSB_len(s) ? 0 : -1);
+}

From reza at naghibi.com  Tue May 19 19:39:09 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Tue, 19 May 2020 19:39:09 +0000 (UTC)
Subject: [6.0] c6406f242 For CLI commands forwarded indirectly to the worker
 (via mgt_cli_askchild() as opposed to directly through mcf_askchild()),
 a truncated CLI response is not a failure.
Message-ID: <20200519193909.4D8506568@lists.varnish-cache.org>


commit c6406f242d0725fbf90b988e4287f2d37378b856
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Dec 2 22:05:30 2019 +0000

    For CLI commands forwarded indirectly to the worker (via
    mgt_cli_askchild() as opposed to directly through mcf_askchild()),
    a truncated CLI response is not a failure.
    
    Fixes #3038

diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c
index 394664939..bbb4a425f 100644
--- a/bin/varnishd/mgt/mgt_cli.c
+++ b/bin/varnishd/mgt/mgt_cli.c
@@ -123,7 +123,6 @@ mcf_askchild(struct cli *cli, const char * const *av, void *priv)
 	int i;
 	char *q;
 	unsigned u;
-	struct vsb *vsb;
 
 	(void)priv;
 	/*
@@ -138,21 +137,19 @@ mcf_askchild(struct cli *cli, const char * const *av, void *priv)
 		    "Type 'help' for more info.");
 		return;
 	}
-	vsb = VSB_new_auto();
+	VSB_clear(cli_buf);
 	for (i = 1; av[i] != NULL; i++) {
-		VSB_quote(vsb, av[i], strlen(av[i]), 0);
-		VSB_putc(vsb, ' ');
+		VSB_quote(cli_buf, av[i], strlen(av[i]), 0);
+		VSB_putc(cli_buf, ' ');
 	}
-	VSB_putc(vsb, '\n');
-	AZ(VSB_finish(vsb));
-	if (VSB_tofile(cli_o, vsb)) {
-		VSB_destroy(&vsb);
+	VSB_putc(cli_buf, '\n');
+	AZ(VSB_finish(cli_buf));
+	if (VSB_tofile(cli_o, cli_buf)) {
 		VCLI_SetResult(cli, CLIS_COMMS);
 		VCLI_Out(cli, "CLI communication error");
 		MCH_Cli_Fail();
 		return;
 	}
-	VSB_destroy(&vsb);
 	if (VCLI_ReadResult(cli_i, &u, &q, mgt_param.cli_timeout))
 		MCH_Cli_Fail();
 	VCLI_SetResult(cli, u);
@@ -180,20 +177,14 @@ mgt_cli_askchild(unsigned *status, char **resp, const char *fmt, ...)
 	va_list ap;
 	unsigned u;
 
-	if (cli_buf == NULL) {
-		cli_buf = VSB_new_auto();
-		AN(cli_buf);
-	} else {
-		VSB_clear(cli_buf);
-	}
+	AN(status);
+	VSB_clear(cli_buf);
 
 	if (resp != NULL)
 		*resp = NULL;
-	if (status != NULL)
-		*status = 0;
+	*status = 0;
 	if (cli_i < 0 || cli_o < 0) {
-		if (status != NULL)
-			*status = CLIS_CANT;
+		*status = CLIS_CANT;
 		return (CLIS_CANT);
 	}
 	va_start(ap, fmt);
@@ -203,8 +194,7 @@ mgt_cli_askchild(unsigned *status, char **resp, const char *fmt, ...)
 	i = VSB_len(cli_buf);
 	assert(i > 0 && VSB_data(cli_buf)[i - 1] == '\n');
 	if (VSB_tofile(cli_o, cli_buf)) {
-		if (status != NULL)
-			*status = CLIS_COMMS;
+		*status = CLIS_COMMS;
 		if (resp != NULL)
 			*resp = strdup("CLI communication error");
 		MCH_Cli_Fail();
@@ -213,9 +203,8 @@ mgt_cli_askchild(unsigned *status, char **resp, const char *fmt, ...)
 
 	if (VCLI_ReadResult(cli_i, &u, resp, mgt_param.cli_timeout))
 		MCH_Cli_Fail();
-	if (status != NULL)
-		*status = u;
-	return (u == CLIS_OK ? 0 : u);
+	*status = u;
+	return (u == CLIS_OK || u == CLIS_TRUNCATED ? 0 : u);
 }
 
 /*--------------------------------------------------------------------*/
@@ -370,6 +359,8 @@ mgt_cli_init_cls(void)
 	VCLS_AddFunc(mgt_cls, MCF_AUTH, cli_proto);
 	VCLS_AddFunc(mgt_cls, MCF_AUTH, cli_debug);
 	VCLS_AddFunc(mgt_cls, MCF_AUTH, cli_askchild);
+	cli_buf = VSB_new_auto();
+	AN(cli_buf);
 }
 
 /*--------------------------------------------------------------------
diff --git a/bin/varnishtest/tests/r03038.vtc b/bin/varnishtest/tests/r03038.vtc
new file mode 100644
index 000000000..fd27faab8
--- /dev/null
+++ b/bin/varnishtest/tests/r03038.vtc
@@ -0,0 +1,25 @@
+varnishtest "vcl.list and cli_limit"
+
+server s1 { 
+} -start
+
+varnish v1 -vcl+backend { } -start
+varnish v1 -vcl+backend { } 
+varnish v1 -vcl+backend { }
+varnish v1 -vcl+backend { }
+varnish v1 -vcl+backend { }
+varnish v1 -vcl+backend { }
+varnish v1 -vcl+backend { }
+varnish v1 -vcl+backend { }
+varnish v1 -vcl+backend { }
+varnish v1 -vcl+backend { }
+
+varnish v1 -expect n_vcl_avail == 10
+
+varnish v1 -cliok "param.set cli_limit 128b"
+
+varnish v1 -clierr 201 "vcl.list"
+
+varnish v1 -stop
+
+varnish v1 -clierr 201 "vcl.list"

From reza at naghibi.com  Tue May 19 19:39:09 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Tue, 19 May 2020 19:39:09 +0000 (UTC)
Subject: [6.0] 6590787f0 for compiling VCLs, truncation is not an error
Message-ID: <20200519193909.71D456573@lists.varnish-cache.org>


commit 6590787f0e935161d3e99fb7b68dfbb4e593f976
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Dec 16 14:57:09 2019 +0100

    for compiling VCLs, truncation is not an error
    
    Conflicts:
        bin/varnishd/mgt/mgt_main.c

diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index 2e0ee14f5..765efd39a 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -175,7 +175,10 @@ usage(void)
 static void
 cli_check(const struct cli *cli)
 {
-	if (cli->result == CLIS_OK) {
+	if (cli->result == CLIS_OK || cli->result == CLIS_TRUNCATED) {
+		AZ(VSB_finish(cli->sb));
+		if (VSB_len(cli->sb) > 0)
+			fprintf(stderr, "Warnings:\n%s\n", VSB_data(cli->sb));
 		VSB_clear(cli->sb);
 		return;
 	}
@@ -798,7 +801,9 @@ main(int argc, char * const *argv)
 			fprintf(stderr, "%s\n", VSB_data(cli->sb));
 			VSB_clear(cli->sb);
 		}
-		exit(cli->result == CLIS_OK ? 0 : 2);
+		if (cli->result == CLIS_OK || cli->result == CLIS_TRUNCATED)
+			exit(0);
+		exit(2);
 	} else {
 		while (!VTAILQ_EMPTY(&f_args)) {
 			fa = VTAILQ_FIRST(&f_args);
diff --git a/bin/varnishd/mgt/mgt_vcl.c b/bin/varnishd/mgt/mgt_vcl.c
index c0fadf475..5d7bab1a7 100644
--- a/bin/varnishd/mgt/mgt_vcl.c
+++ b/bin/varnishd/mgt/mgt_vcl.c
@@ -603,7 +603,7 @@ mgt_new_vcl(struct cli *cli, const char *vclname, const char *vclsrc,
 	if (active_vcl == NULL)
 		active_vcl = vp;
 
-	if (cli->result == CLIS_OK &&
+	if ((cli->result == CLIS_OK || cli->result == CLIS_TRUNCATED) &&
 	    vcl_count > mgt_param.max_vcl &&
 	    mgt_param.max_vcl_handling == 1) {
 		VCLI_Out(cli, "%d VCLs loaded\n", vcl_count);

From reza at naghibi.com  Tue May 19 19:39:09 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Tue, 19 May 2020 19:39:09 +0000 (UTC)
Subject: [6.0] c3ba03ee3 set the cli_limit when the parameter is actually
 available
Message-ID: <20200519193909.8B5B36580@lists.varnish-cache.org>


commit c3ba03ee3614fa053f4c6f58b1d5bc398eb6c7fe
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Dec 16 14:49:54 2019 +0100

    set the cli_limit when the parameter is actually available
    
    it is not before we have parsed parameters

diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c
index bbb4a425f..8db75e33b 100644
--- a/bin/varnishd/mgt/mgt_cli.c
+++ b/bin/varnishd/mgt/mgt_cli.c
@@ -353,7 +353,6 @@ mgt_cli_init_cls(void)
 
 	mgt_cls = VCLS_New(NULL);
 	AN(mgt_cls);
-	VCLS_SetLimit(mgt_cls, &mgt_param.cli_limit);
 	VCLS_SetHooks(mgt_cls, mgt_cli_cb_before, mgt_cli_cb_after);
 	VCLS_AddFunc(mgt_cls, MCF_NOAUTH, cli_auth);
 	VCLS_AddFunc(mgt_cls, MCF_AUTH, cli_proto);
diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index 765efd39a..72a378e1d 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -703,6 +703,8 @@ main(int argc, char * const *argv)
 		    VSB_data(cli[0].sb));
 	}
 
+	VCLS_SetLimit(mgt_cls, &mgt_param.cli_limit);
+
 	assert(d_flag == 0 || F_flag == 0);
 
 	if (C_flag) {

From reza at naghibi.com  Tue May 19 19:39:09 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Tue, 19 May 2020 19:39:09 +0000 (UTC)
Subject: [6.0] 795366627 Formally expose VSB_tofile in libvarnishapi
Message-ID: <20200519193909.AB7BE6588@lists.varnish-cache.org>


commit 795366627ffdb0515c8e897a5f3d15623109cc56
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed Mar 4 10:27:38 2020 +0100

    Formally expose VSB_tofile in libvarnishapi
    
    Closes #3238
    
    Conflicts:
        lib/libvarnishapi/libvarnishapi.map

diff --git a/lib/libvarnishapi/libvarnishapi.map b/lib/libvarnishapi/libvarnishapi.map
index 4cc3b5050..30873af89 100644
--- a/lib/libvarnishapi/libvarnishapi.map
+++ b/lib/libvarnishapi/libvarnishapi.map
@@ -1,5 +1,5 @@
 /*-
- * Copyright (c) 2011-2017 Varnish Software AS
+ * Copyright (c) 2011-2020 Varnish Software AS
  * All rights reserved.
  *
  * Author: Tollef Fog Heen <tfheen at varnish-software.com>
@@ -140,3 +140,11 @@ LIBVARNISHAPI_2.0 {
     local:
 	*;
 };
+
+LIBVARNISHAPI_2.1 {
+    global:
+	# vsb.c
+	VSB_tofile;
+    local:
+	*;
+};
\ No newline at end of file

From reza at naghibi.com  Tue May 19 19:39:09 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Tue, 19 May 2020 19:39:09 +0000 (UTC)
Subject: [6.0] e54544b0c Swap the VSB_tofile() arguments
Message-ID: <20200519193909.D11356590@lists.varnish-cache.org>


commit e54544b0c341b2cfb86bd5c911e04dabbb505f19
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed Mar 11 10:14:01 2020 +0100

    Swap the VSB_tofile() arguments
    
    Automated with Coccinelle, so the semantic patch could be reused in the
    vtest project.
    
    Closes #3238
    
    Conflicts:
        lib/libvcc/vcc_compile.c
        bin/varnishd/proxy/cache_proxy_proto.c

diff --git a/bin/varnishd/common/common_vsmw.c b/bin/varnishd/common/common_vsmw.c
index dcf83dbd3..dcc85fc22 100644
--- a/bin/varnishd/common/common_vsmw.c
+++ b/bin/varnishd/common/common_vsmw.c
@@ -206,7 +206,7 @@ vsmw_append_record(struct vsmw *vsmw, struct vsmwseg *seg, char act)
 	VSB_clear(vsmw->vsb);
 	vsmw_fmt_index(vsmw, seg, act);
 	AZ(VSB_finish(vsmw->vsb));
-	XXXAZ(VSB_tofile(fd, vsmw->vsb)); // XXX handle ENOSPC? #2764
+	XXXAZ(VSB_tofile(vsmw->vsb, fd)); // XXX handle ENOSPC? #2764
 	closefd(&fd);
 }
 
@@ -253,7 +253,7 @@ vsmw_delseg(struct vsmw *vsmw, struct vsmwseg *seg)
 		VTAILQ_FOREACH(s2, &vsmw->segs, list)
 			vsmw_fmt_index(vsmw, s2, '+');
 		AZ(VSB_finish(vsmw->vsb));
-		XXXAZ(VSB_tofile(fd, vsmw->vsb)); // XXX handle ENOSPC? #2764
+		XXXAZ(VSB_tofile(vsmw->vsb, fd)); // XXX handle ENOSPC? #2764
 		closefd(&fd);
 		AZ(renameat(vsmw->vdirfd, t, vsmw->vdirfd, vsmw->idx));
 		REPLACE(t, NULL);
diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c
index 8db75e33b..6f4c2accd 100644
--- a/bin/varnishd/mgt/mgt_cli.c
+++ b/bin/varnishd/mgt/mgt_cli.c
@@ -144,7 +144,7 @@ mcf_askchild(struct cli *cli, const char * const *av, void *priv)
 	}
 	VSB_putc(cli_buf, '\n');
 	AZ(VSB_finish(cli_buf));
-	if (VSB_tofile(cli_o, cli_buf)) {
+	if (VSB_tofile(cli_buf, cli_o)) {
 		VCLI_SetResult(cli, CLIS_COMMS);
 		VCLI_Out(cli, "CLI communication error");
 		MCH_Cli_Fail();
@@ -193,7 +193,7 @@ mgt_cli_askchild(unsigned *status, char **resp, const char *fmt, ...)
 	AZ(VSB_finish(cli_buf));
 	i = VSB_len(cli_buf);
 	assert(i > 0 && VSB_data(cli_buf)[i - 1] == '\n');
-	if (VSB_tofile(cli_o, cli_buf)) {
+	if (VSB_tofile(cli_buf, cli_o)) {
 		*status = CLIS_COMMS;
 		if (resp != NULL)
 			*resp = strdup("CLI communication error");
diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c
index ddcc8db77..64935686d 100644
--- a/bin/varnishd/proxy/cache_proxy_proto.c
+++ b/bin/varnishd/proxy/cache_proxy_proto.c
@@ -667,7 +667,7 @@ VPX_Send_Proxy(int fd, int version, const struct sess *sp)
 		WRONG("Wrong proxy version");
 
 	AZ(VSB_finish(vsb));
-	(void)VSB_tofile(fd, vsb);	// XXX: Error handling ?
+	(void)VSB_tofile(vsb, fd);	// XXX: Error handling ?
 	if (!DO_DEBUG(DBG_PROTOCOL)) {
 		VSB_delete(vsb);
 		return;
diff --git a/bin/varnishtest/vtc_haproxy.c b/bin/varnishtest/vtc_haproxy.c
index d6565b834..04590596e 100644
--- a/bin/varnishtest/vtc_haproxy.c
+++ b/bin/varnishtest/vtc_haproxy.c
@@ -173,7 +173,7 @@ cmd_haproxy_cli_send(CMD_ARGS)
 	}
 	vtc_dump(hc->vl, 4, "CLI send", VSB_data(vsb), -1);
 
-	if (VSB_tofile(hc->sock, vsb))
+	if (VSB_tofile(vsb, hc->sock))
 		vtc_fatal(hc->vl,
 		    "CLI fd %d send error %s", hc->sock, strerror(errno));
 
diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c
index 20969c27a..da4747872 100644
--- a/bin/varnishtest/vtc_http.c
+++ b/bin/varnishtest/vtc_http.c
@@ -216,7 +216,7 @@ http_write(const struct http *hp, int lvl, const char *pfx)
 
 	AZ(VSB_finish(hp->vsb));
 	vtc_dump(hp->vl, lvl, pfx, VSB_data(hp->vsb), VSB_len(hp->vsb));
-	if (VSB_tofile(hp->fd, hp->vsb))
+	if (VSB_tofile(hp->vsb, hp->fd))
 		vtc_log(hp->vl, hp->fatal, "Write failed: %s",
 		    strerror(errno));
 }
@@ -1424,7 +1424,7 @@ cmd_http_sendhex(CMD_ARGS)
 	vsb = vtc_hex_to_bin(hp->vl, av[1]);
 	assert(VSB_len(vsb) >= 0);
 	vtc_hexdump(hp->vl, 4, "sendhex", VSB_data(vsb), VSB_len(vsb));
-	if (VSB_tofile(hp->fd, vsb))
+	if (VSB_tofile(vsb, hp->fd))
 		vtc_log(hp->vl, hp->fatal, "Write failed: %s",
 		    strerror(errno));
 	VSB_destroy(&vsb);
diff --git a/bin/varnishtest/vtc_process.c b/bin/varnishtest/vtc_process.c
index ae98f53e1..b2e0e2659 100644
--- a/bin/varnishtest/vtc_process.c
+++ b/bin/varnishtest/vtc_process.c
@@ -793,7 +793,7 @@ process_write_hex(const struct process *p, const char *text)
 	vsb = vtc_hex_to_bin(p->vl, text);
 	assert(VSB_len(vsb) >= 0);
 	vtc_hexdump(p->vl, 4, "sendhex", VSB_data(vsb), VSB_len(vsb));
-	AZ(VSB_tofile(p->fd_term, vsb));
+	AZ(VSB_tofile(vsb, p->fd_term));
 	VSB_destroy(&vsb);
 }
 
diff --git a/bin/varnishtest/vtc_proxy.c b/bin/varnishtest/vtc_proxy.c
index 4477ab056..a72221573 100644
--- a/bin/varnishtest/vtc_proxy.c
+++ b/bin/varnishtest/vtc_proxy.c
@@ -126,7 +126,7 @@ vtc_send_proxy(int fd, int version, const struct suckaddr *sac,
 		WRONG("Wrong proxy version");
 
 	AZ(VSB_finish(vsb));
-	i = VSB_tofile(fd, vsb);
+	i = VSB_tofile(vsb, fd);
 	VSB_delete(vsb);
 	return (i);
 }
diff --git a/include/vsb.h b/include/vsb.h
index 10e4c9563..086ace255 100644
--- a/include/vsb.h
+++ b/include/vsb.h
@@ -85,7 +85,7 @@ void		 VSB_quote_pfx(struct vsb *, const char*, const void *,
 		     int len, int how);
 void		 VSB_quote(struct vsb *, const void *, int len, int how);
 void		 VSB_indent(struct vsb *, int);
-int		 VSB_tofile(int fd, const struct vsb *);
+int		 VSB_tofile(const struct vsb *, int fd);
 #ifdef __cplusplus
 };
 #endif
diff --git a/lib/libvarnish/vsb.c b/lib/libvarnish/vsb.c
index e6519e98c..3c641730e 100644
--- a/lib/libvarnish/vsb.c
+++ b/lib/libvarnish/vsb.c
@@ -626,7 +626,7 @@ VSB_indent(struct vsb *s, int i)
 }
 
 int
-VSB_tofile(int fd, const struct vsb *s)
+VSB_tofile(const struct vsb *s, int fd)
 {
 	int sz;
 
diff --git a/tools/coccinelle/vsb_tofile.cocci b/tools/coccinelle/vsb_tofile.cocci
new file mode 100644
index 000000000..b8ca8254d
--- /dev/null
+++ b/tools/coccinelle/vsb_tofile.cocci
@@ -0,0 +1,46 @@
+/*
+ * This patch fixes the order of VSB_tofile arguments.
+ */
+
+@@
+idexpression struct vsb *vsb;
+idexpression int fd;
+@@
+
+- VSB_tofile(fd, vsb)
++ VSB_tofile(vsb, fd)
+
+@@
+idexpression struct vsb[] vsb;
+idexpression int fd;
+@@
+
+- VSB_tofile(fd, vsb)
++ VSB_tofile(vsb, fd)
+
+@@
+idexpression struct vsb *vsb;
+expression fd;
+@@
+
+- VSB_tofile(fd, vsb)
++ VSB_tofile(vsb, fd)
+
+/* Opportunistic fallback */
+
+@@
+idexpression int fd;
+expression vsb;
+@@
+
+- VSB_tofile(fd, vsb)
++ VSB_tofile(vsb, fd)
+
+/* Opportunistic last resort */
+
+@@
+expression fd, other;
+@@
+
+- VSB_tofile(fd, other->vsb)
++ VSB_tofile(other->vsb, fd)

From reza at naghibi.com  Tue May 19 19:45:07 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Tue, 19 May 2020 19:45:07 +0000 (UTC)
Subject: [6.0] ee991059c vbf_stp_condfetch: handle storage error instead of
 crashing
Message-ID: <20200519194507.A854A7290@lists.varnish-cache.org>


commit ee991059c92a2ea1c473effcb4f303137f60eacd
Author: Emanuele Rocca <ema at wikimedia.org>
Date:   Fri Apr 3 17:10:22 2020 +0200

    vbf_stp_condfetch: handle storage error instead of crashing
    
    Fixes: #3273

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index c3ff67e9a..f7fd60424 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -737,7 +737,13 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo)
 	}
 	AZ(bo->stale_oc->flags & OC_F_FAILED);
 
-	AZ(vbf_beresp2obj(bo));
+	if (vbf_beresp2obj(bo)) {
+		(void)VFP_Error(bo->vfc, "Could not get storage");
+		bo->htc->doclose = SC_RX_BODY;
+		VFP_Close(bo->vfc);
+		VDI_Finish(bo->wrk, bo);
+		return (F_STP_ERROR);
+	}
 
 	if (ObjHasAttr(bo->wrk, bo->stale_oc, OA_ESIDATA))
 		AZ(ObjCopyAttr(bo->wrk, bo->fetch_objcore, bo->stale_oc,
diff --git a/bin/varnishtest/tests/v00064.vtc b/bin/varnishtest/tests/v00064.vtc
new file mode 100644
index 000000000..8a608e38e
--- /dev/null
+++ b/bin/varnishtest/tests/v00064.vtc
@@ -0,0 +1,66 @@
+varnishtest "vbf_stp_condfetch could not get storage #3273"
+
+server s1 {
+    rxreq
+    expect req.url == "/transient"
+    txresp -bodylen 1048400
+
+    rxreq
+    expect req.url == "/malloc"
+    txresp -hdr "Cache-Control: max-age=2" -hdr "Last-Modified: Fri, 03 Apr 2020 13:00:01 GMT" -bodylen 1048300
+
+    rxreq
+    expect req.http.If-Modified-Since == "Fri, 03 Apr 2020 13:00:01 GMT"
+    expect req.url == "/malloc"
+    txresp -status 304
+} -start
+
+varnish v1 \
+ -arg "-s Transient=default,1m" \
+ -arg "-s malloc,1m" \
+ -arg "-p nuke_limit=0" \
+ -syntax 4.0 \
+ -vcl+backend {
+    sub vcl_backend_response {
+        if (bereq.url == "/transient") {
+            set beresp.storage = storage.Transient;
+            # Unset Date header to not change the object sizes
+            unset beresp.http.Date;
+        }
+    }
+} -start
+
+varnish v1 -cliok "param.set debug +syncvsl"
+
+delay .1
+
+client c1 {
+    # Fill transient
+    txreq -url "/transient"
+    rxresp
+    expect resp.status == 200
+} -run
+
+delay .1
+
+varnish v1 -expect SM?.Transient.g_bytes > 1048000
+varnish v1 -expect SM?.Transient.g_space == 8
+
+client c1 {
+    # Fill malloc
+    txreq -url "/malloc" -hdr "If-Modified-Since: Fri, 03 Apr 2020 12:00:01 GMT"
+    rxresp
+    expect resp.status == 200
+
+    delay 3
+} -run
+
+varnish v1 -expect SM?.s0.g_bytes > 1048000
+varnish v1 -expect SM?.s0.g_space == 4
+
+client c1 {
+    # Check that Varnish is still alive
+    txreq -url "/malloc" -hdr "If-Modified-Since: Fri, 03 Apr 2020 12:00:01 GMT"
+    rxresp
+    expect resp.status == 200
+} -run

From reza at naghibi.com  Wed May 20 13:54:08 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Wed, 20 May 2020 13:54:08 +0000 (UTC)
Subject: [6.0] b501bb064 Ensure we don't recycle a closed backend connection
Message-ID: <20200520135408.A5DAAA9695@lists.varnish-cache.org>


commit b501bb06489a6458a13d17d726a9339a8431b995
Author: Dag Haavi Finstad <daghf at varnish-software.com>
Date:   Wed Apr 1 15:52:31 2020 +0200

    Ensure we don't recycle a closed backend connection
    
    Fixes: #3266

diff --git a/bin/varnishd/http1/cache_http1_vfp.c b/bin/varnishd/http1/cache_http1_vfp.c
index 36228cc44..7b03ca8ce 100644
--- a/bin/varnishd/http1/cache_http1_vfp.c
+++ b/bin/varnishd/http1/cache_http1_vfp.c
@@ -80,6 +80,9 @@ v1f_read(const struct vfp_ctx *vc, struct http_conn *htc, void *d, ssize_t len)
 			    "%s", strerror(errno));
 			return (i);
 		}
+		if (i == 0)
+			htc->doclose = SC_RESP_CLOSE;
+
 	}
 	return (i + l);
 }
diff --git a/bin/varnishtest/tests/r03266.vtc b/bin/varnishtest/tests/r03266.vtc
new file mode 100644
index 000000000..1c4718192
--- /dev/null
+++ b/bin/varnishtest/tests/r03266.vtc
@@ -0,0 +1,19 @@
+varnishtest "Don't recycle a closed backend connection"
+
+# broken origin: sends eof-encoded HTTP/1.1 response
+server s1 {
+	rxreq
+	send "HTTP/1.1 200 OK\r\n\r\n"
+	send "foobar"
+} -start
+
+varnish v1 -vcl+backend {} -start
+
+client c1 {
+	txreq
+	rxresp
+} -run
+
+varnish v1 -expect fetch_failed == 0
+varnish v1 -expect fetch_eof == 1
+varnish v1 -expect backend_recycle == 0

From reza at naghibi.com  Wed May 20 13:55:07 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Wed, 20 May 2020 13:55:07 +0000 (UTC)
Subject: [6.0] 002a27a6b Apply the 3 least significant digit rule of status
 code on H/2
Message-ID: <20200520135507.7EEBDA98E2@lists.varnish-cache.org>


commit 002a27a6bd7a0934af14e5cbab530050ed4a4f6a
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Apr 24 16:24:54 2020 +0200

    Apply the 3 least significant digit rule of status code on H/2
    
    In VCL we allow status codes larger then 999, and do modulo 1000 when
    sending it to the client. But the code to apply the modulo was missing in
    H/2.
    
    This issue was uncovered and reported by @Francois-v3.
    
    Fixes first case of: #3301
    
    Conflicts:
        bin/varnishd/http2/cache_http2_deliver.c

diff --git a/bin/varnishd/http2/cache_http2_deliver.c b/bin/varnishd/http2/cache_http2_deliver.c
index 94018edd7..ee255067e 100644
--- a/bin/varnishd/http2/cache_http2_deliver.c
+++ b/bin/varnishd/http2/cache_http2_deliver.c
@@ -238,7 +238,7 @@ h2_build_headers(struct vsb *resp, struct req *req)
 
 	AN(VSB_new(resp, req->ws->f, l, VSB_FIXEDLEN));
 
-	l = h2_status(buf, req->resp->status);
+	l = h2_status(buf, req->resp->status % 1000);
 	VSB_bcat(resp, buf, l);
 
 	hp = req->resp;

From reza at naghibi.com  Wed May 20 13:55:07 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Wed, 20 May 2020 13:55:07 +0000 (UTC)
Subject: [6.0] 9e6f8a33b Check for VRT_fail before setting handling on
 return() in VCL
Message-ID: <20200520135507.A2AE0A98E5@lists.varnish-cache.org>


commit 9e6f8a33beceba66bfc237a0c1c68182833f72d0
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Apr 24 16:28:35 2020 +0200

    Check for VRT_fail before setting handling on return() in VCL
    
    The compound statement doing return() in VCL may have failed for some
    reason, for example running out of workspace while putting together the
    reason field. The fail handling would be overwritten by the forced
    handling at the end of the generated code for a return statement.
    
    This patch adds a test for the failure condition before forcing the
    handling to the result of the return statement.
    
    Conflicts:
        lib/libvcc/vcc_action.c

diff --git a/lib/libvcc/vcc_action.c b/lib/libvcc/vcc_action.c
index 210101feb..1dd9e88cf 100644
--- a/lib/libvcc/vcc_action.c
+++ b/lib/libvcc/vcc_action.c
@@ -350,6 +350,7 @@ vcc_act_return(struct vcc *tl, struct token *t, struct symbol *sym)
 		}
 	}
 	ERRCHK(tl);
+	Fb(tl, 1, "END_;\n");
 	Fb(tl, 1, "VRT_handling(ctx, VCL_RET_%s);\n", h);
 	SkipToken(tl, ')');
 	SkipToken(tl, ';');

From reza at naghibi.com  Wed May 20 13:55:07 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Wed, 20 May 2020 13:55:07 +0000 (UTC)
Subject: [6.0] ae97f2ccf Check for illegal status codes when executing
 return(synth())
Message-ID: <20200520135507.CCBD2A98E9@lists.varnish-cache.org>


commit ae97f2ccfc029a5883b7c424035f3a8c71f5d34c
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Apr 24 16:31:45 2020 +0200

    Check for illegal status codes when executing return(synth())
    
    Some status codes are illegal and will cause VRT_fail() when executed as
    normal set instructions in VCL. But this test is bypassed when status is
    set as a side effect of a `return (synth(code))` statement.
    
    This patch applies the same rules as when executing a set-instruction to
    the return(synth()) handling.
    
    Fixes second part of: #3301

diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index 343481346..42f2995a1 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -56,8 +56,22 @@ VRT_synth(VRT_CTX, VCL_INT code, VCL_STRING reason)
 
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
 	assert(ctx->req != NULL || ctx->bo != NULL);
-	if (code < 100 || code > 65535)
-		code = 503;
+	if (code < 0) {
+		VRT_fail(ctx, "return(synth()) status code (%jd) is negative",
+		    code);
+		return;
+	}
+	if (code > 65535) {
+		VRT_fail(ctx, "return(synth()) status code (%jd) > 65535",
+		    code);
+		return;
+	}
+	if ((code % 1000) < 100) {
+		VRT_fail(ctx,
+		    "illegal return(synth()) status code (%jd) (..0##)",
+		    code);
+		return;
+	}
 
 	if (ctx->req == NULL) {
 		CHECK_OBJ_NOTNULL(ctx->bo, BUSYOBJ_MAGIC);

From reza at naghibi.com  Wed May 20 13:55:07 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Wed, 20 May 2020 13:55:07 +0000 (UTC)
Subject: [6.0] 7f538cce7 Add a test case
Message-ID: <20200520135507.F24BFA98EE@lists.varnish-cache.org>


commit 7f538cce7e3add26cd9fda93e00b0c4c83951906
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Fri Apr 24 16:34:46 2020 +0200

    Add a test case

diff --git a/bin/varnishtest/tests/r03301.vtc b/bin/varnishtest/tests/r03301.vtc
new file mode 100644
index 000000000..d39f31709
--- /dev/null
+++ b/bin/varnishtest/tests/r03301.vtc
@@ -0,0 +1,73 @@
+varnishtest "Issue 3301: Illegal error codes"
+
+varnish v1 -vcl {
+	backend default none;
+	sub vcl_recv {
+		if (req.url == "/test1") {
+			return (synth(1301));
+		}
+		if (req.url == "/test2") {
+			return (synth(1001));
+		}
+	}
+	sub vcl_backend_fetch {
+		if (bereq.url == "/test3") {
+			return (error(1302));
+		}
+		if (bereq.url == "/test4") {
+			return (error(1000));
+		}
+	}
+} -start
+
+varnish v1 -cliok "param.set feature +http2"
+
+client c1 {
+	stream 1 {
+		txreq -url /test1
+		rxresp
+		expect resp.status == 301
+	} -run
+
+	stream 3 {
+		txreq -url /test2
+		rxresp
+		expect resp.status == 503
+	} -run
+
+	stream 5 {
+		txreq -url /test3
+		rxresp
+		expect resp.status == 302
+	} -run
+
+	stream 7 {
+		txreq -url /test4
+		rxresp
+		expect resp.status == 503
+	} -run
+} -run
+
+client c2 {
+	txreq -url /test1
+	rxresp
+	expect resp.status == 301
+	expect resp.reason == "Moved Permanently"
+
+	txreq -url /test2
+	rxresp
+	expect resp.status == 503
+} -run
+
+client c2 {
+	txreq -url /test3
+	rxresp
+	expect resp.status == 302
+	expect resp.reason == "Found"
+} -run
+
+client c2 {
+	txreq -url /test4
+	rxresp
+	expect resp.status == 503
+} -run

From reza at naghibi.com  Wed May 20 13:55:08 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Wed, 20 May 2020 13:55:08 +0000 (UTC)
Subject: [6.0] f1d13df95 Document special meaning of status code also for
 resp.status
Message-ID: <20200520135508.372F5A98F4@lists.varnish-cache.org>


commit f1d13df953b99041e3ee6e87465038cbef1727f1
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Tue Apr 28 13:03:48 2020 +0200

    Document special meaning of status code also for resp.status
    
    Copy paste the text from beresp.status to resp.status about the special
    handling of status codes.

diff --git a/doc/sphinx/reference/vcl_var.rst b/doc/sphinx/reference/vcl_var.rst
index 16a35106a..99c114f39 100644
--- a/doc/sphinx/reference/vcl_var.rst
+++ b/doc/sphinx/reference/vcl_var.rst
@@ -1099,6 +1099,10 @@ resp.status
 	Assigning a HTTP standardized code to resp.status will also
 	set resp.reason to the corresponding status message.
 
+	Status codes on the form XXYZZ can be set where
+	XXYZZ is less than 65536 and Y is [1...9].
+	Only YZZ will be sent back to clients.
+
 	resp.status 200 will get changed into 304 by core code after
 	a return(deliver) from vcl_deliver for conditional requests
 	to cached content if validation succeeds.

From reza at naghibi.com  Wed May 20 13:55:08 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Wed, 20 May 2020 13:55:08 +0000 (UTC)
Subject: [6.0] 8f8c85c4a Add an assert that the status is valid on entry to
 h2_build_headers
Message-ID: <20200520135508.65019A98FC@lists.varnish-cache.org>


commit 8f8c85c4a9cf485c8013623041511fa8f7e3b0aa
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Tue May 5 13:36:22 2020 +0200

    Add an assert that the status is valid on entry to h2_build_headers
    
    Conflicts:
        bin/varnishd/http2/cache_http2_deliver.c

diff --git a/bin/varnishd/http2/cache_http2_deliver.c b/bin/varnishd/http2/cache_http2_deliver.c
index ee255067e..ee9f40cb2 100644
--- a/bin/varnishd/http2/cache_http2_deliver.c
+++ b/bin/varnishd/http2/cache_http2_deliver.c
@@ -238,6 +238,7 @@ h2_build_headers(struct vsb *resp, struct req *req)
 
 	AN(VSB_new(resp, req->ws->f, l, VSB_FIXEDLEN));
 
+	assert(req->resp->status % 1000 >= 100);
 	l = h2_status(buf, req->resp->status % 1000);
 	VSB_bcat(resp, buf, l);
 

From reza at naghibi.com  Wed May 20 13:55:08 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Wed, 20 May 2020 13:55:08 +0000 (UTC)
Subject: [6.0] b3344f26c Govern thread creation by queue length instead of dry
 signals
Message-ID: <20200520135508.9113FA9908@lists.varnish-cache.org>


commit b3344f26ce49a700220be9aeca1afc30df174964
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Wed Mar 13 14:14:44 2019 +0100

    Govern thread creation by queue length instead of dry signals
    
    When getidleworker signals the pool herder to spawn a thread, it increases
    the dry counter, and the herder resets dry again when spawning a single
    thread. This will in many cases only create a single thread even though
    the herder was signaled dry multiple times, and may cause a situation
    where the cache acceptor is queued and no new thread is created. Together
    with long lasting tasks (ie endless pipelines), and all other tasks having
    higher priority, this will prevent the cache acceptor from being
    rescheduled.
    
    c00096.vtc demonstrates how this can lock up.
    
    To fix this, spawn threads if we have queued tasks and we are below the
    thread maximum level.
    
    Conflicts:
        bin/varnishd/cache/cache_wrk.c

diff --git a/bin/varnishd/cache/cache_pool.h b/bin/varnishd/cache/cache_pool.h
index 9902e32ad..4e668fa0b 100644
--- a/bin/varnishd/cache/cache_pool.h
+++ b/bin/varnishd/cache/cache_pool.h
@@ -48,7 +48,6 @@ struct pool {
 	struct taskhead			idle_queue;
 	struct taskhead			queues[TASK_QUEUE_END];
 	unsigned			nthr;
-	unsigned			dry;
 	unsigned			lqueue;
 	uintmax_t			sdropped;
 	uintmax_t			rdropped;
diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c
index fa3fa366a..044e140c4 100644
--- a/bin/varnishd/cache/cache_wrk.c
+++ b/bin/varnishd/cache/cache_wrk.c
@@ -194,13 +194,8 @@ pool_getidleworker(struct pool *pp, enum task_prio prio)
 			AZ(pp->nidle);
 	}
 
-	if (pt == NULL) {
-		if (pp->nthr < cache_param->wthread_max) {
-			pp->dry++;
-			AZ(pthread_cond_signal(&pp->herder_cond));
-		}
+	if (pt == NULL)
 		return (NULL);
-	}
 	AZ(pt->func);
 	CAST_OBJ_NOTNULL(wrk, pt->priv, WORKER_MAGIC);
 	return (wrk);
@@ -303,6 +298,7 @@ Pool_Task(struct pool *pp, struct pool_task *task, enum task_prio prio)
 		pp->nqueued++;
 		pp->lqueue++;
 		VTAILQ_INSERT_TAIL(&pp->queues[prio], task, list);
+		AZ(pthread_cond_signal(&pp->herder_cond));
 	} else {
 		if (prio == TASK_QUEUE_REQ)
 			pp->sdropped++;
@@ -473,7 +469,6 @@ pool_breed(struct pool *qp)
 		Lck_Unlock(&pool_mtx);
 		VTIM_sleep(cache_param->wthread_fail_delay);
 	} else {
-		qp->dry = 0;
 		qp->nthr++;
 		Lck_Lock(&pool_mtx);
 		VSC_C_main->threads++;
@@ -549,7 +544,7 @@ pool_herder(void *priv)
 
 		/* Make more threads if needed and allowed */
 		if (pp->nthr < wthread_min ||
-		    (pp->dry && pp->nthr < cache_param->wthread_max)) {
+		    (pp->lqueue > 0 && pp->nthr < cache_param->wthread_max)) {
 			pool_breed(pp);
 			continue;
 		}
@@ -610,16 +605,14 @@ pool_herder(void *priv)
 			continue;
 		}
 		Lck_Lock(&pp->mtx);
-		if (!pp->dry) {
+		if (pp->lqueue == 0) {
 			if (DO_DEBUG(DBG_VTC_MODE))
 				delay = 0.5;
 			(void)Lck_CondWait(&pp->herder_cond, &pp->mtx,
 				VTIM_real() + delay);
-		} else {
+		} else
 			/* XXX: unsafe counters */
 			VSC_C_main->threads_limited++;
-			pp->dry = 0;
-		}
 		Lck_Unlock(&pp->mtx);
 	}
 	return (NULL);
diff --git a/bin/varnishtest/tests/c00096.vtc b/bin/varnishtest/tests/c00096.vtc
new file mode 100644
index 000000000..cf80a0f53
--- /dev/null
+++ b/bin/varnishtest/tests/c00096.vtc
@@ -0,0 +1,106 @@
+varnishtest "Test thread creation on acceptor thread queuing"
+
+# This tests that we are able to spawn new threads in the event that the
+# cache acceptor has been queued. It does this by starting 6 long lasting
+# fetches, which will consume 12 threads. That exceeds the initial
+# allotment of 10 threads, giving some probability that the acceptor
+# thread is queued. Then a single quick fetch is done, which should be
+# served since we are well below the maximum number of threads allowed.
+
+# Barrier b1 blocks the slow servers from finishing until the quick fetch
+# is done.
+barrier b1 cond 7
+
+# Barrier b2 blocks the start of the quick fetch until all slow fetches
+# are known to hold captive two threads each.
+barrier b2 cond 7
+
+server s0 {
+	rxreq
+	txresp -nolen -hdr "Content-Length: 10" -hdr "Connection: close"
+	send "123"
+	barrier b1 sync
+	send "4567890"
+	expect_close
+} -dispatch
+
+server stest {
+	rxreq
+	txresp -body "All good"
+} -start
+
+varnish v1 -arg "-p debug=+syncvsl -p debug=+flush_head"
+varnish v1 -arg "-p thread_pools=1 -p thread_pool_min=10"
+varnish v1 -vcl+backend {
+	sub vcl_backend_fetch {
+		if (bereq.url == "/test") {
+			set bereq.backend = stest;
+		} else {
+			set bereq.backend = s0;
+		}
+	}
+} -start
+
+# NB: we might go above 10 threads when early tasks are submitted to
+# the pool since at least one idle thread must be kept in the pool
+# reserve.
+varnish v1 -expect MAIN.threads >= 10
+
+client c1 {
+	txreq -url /1
+	rxresphdrs
+	barrier b2 sync
+	rxrespbody
+} -start
+
+client c2 {
+	txreq -url /2
+	rxresphdrs
+	barrier b2 sync
+	rxrespbody
+} -start
+
+client c3 {
+	txreq -url /3
+	rxresphdrs
+	barrier b2 sync
+	rxrespbody
+} -start
+
+client c4 {
+	txreq -url /4
+	rxresphdrs
+	barrier b2 sync
+	rxrespbody
+} -start
+
+client c5 {
+	txreq -url /5
+	rxresphdrs
+	barrier b2 sync
+	rxrespbody
+} -start
+
+client c6 {
+	txreq -url /6
+	rxresphdrs
+	barrier b2 sync
+	rxrespbody
+} -start
+
+client ctest {
+	barrier b2 sync
+	txreq -url "/test"
+	rxresp
+	expect resp.status == 200
+	expect resp.body == "All good"
+} -run
+
+barrier b1 sync
+
+client c1 -wait
+client c2 -wait
+client c3 -wait
+client c4 -wait
+client c5 -wait
+client c6 -wait

From reza at naghibi.com  Wed May 20 13:55:08 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Wed, 20 May 2020 13:55:08 +0000 (UTC)
Subject: [6.0] 60b17ab54 Fix a small memory leak
Message-ID: <20200520135508.C1ABDA9929@lists.varnish-cache.org>


commit 60b17ab541bc774e46a09247c6c0a7c7665ebb2d
Author: Martin Blix Grydeland <martin at varnish-software.com>
Date:   Wed Mar 13 14:43:01 2019 +0100

    Fix a small memory leak
    
    Fix a small memory leak when failing to spawn a new thread.

diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c
index 044e140c4..15255736a 100644
--- a/bin/varnishd/cache/cache_wrk.c
+++ b/bin/varnishd/cache/cache_wrk.c
@@ -462,6 +462,7 @@ pool_breed(struct pool *qp)
 	pi->qp = qp;
 
 	if (pthread_create(&tp, &tp_attr, pool_thread, pi)) {
+		FREE_OBJ(pi);
 		VSL(SLT_Debug, 0, "Create worker thread failed %d %s",
 		    errno, strerror(errno));
 		Lck_Lock(&pool_mtx);

From reza at naghibi.com  Wed May 20 13:55:08 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Wed, 20 May 2020 13:55:08 +0000 (UTC)
Subject: [6.0] df5ce9937 SQUASHME: c96 stabilization
Message-ID: <20200520135508.E386CA9943@lists.varnish-cache.org>


commit df5ce9937f98f9a709cef4371ec2a64863a3727b
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Sun Oct 27 15:42:06 2019 +0100

    SQUASHME: c96 stabilization

diff --git a/bin/varnishtest/tests/c00096.vtc b/bin/varnishtest/tests/c00096.vtc
index cf80a0f53..6c1ca372a 100644
--- a/bin/varnishtest/tests/c00096.vtc
+++ b/bin/varnishtest/tests/c00096.vtc
@@ -31,6 +31,7 @@ server stest {
 
 varnish v1 -arg "-p debug=+syncvsl -p debug=+flush_head"
 varnish v1 -arg "-p thread_pools=1 -p thread_pool_min=10"
+varnish v1 -arg "-p thread_pool_add_delay=0.01"
 varnish v1 -vcl+backend {
 	sub vcl_backend_fetch {
 		if (bereq.url == "/test") {
@@ -41,10 +42,7 @@ varnish v1 -vcl+backend {
 	}
 } -start
 
-# NB: we might go above 10 threads when early tasks are submitted to
-# the pool since at least one idle thread must be kept in the pool
-# reserve.
-varnish v1 -expect MAIN.threads >= 10
+varnish v1 -expect MAIN.threads == 10
 
 client c1 {
 	txreq -url /1

From nils.goroll at uplex.de  Thu May 21 10:43:08 2020
From: nils.goroll at uplex.de (Nils Goroll)
Date: Thu, 21 May 2020 10:43:08 +0000 (UTC)
Subject: [master] b3ec626b9 test client -> haproxy -> [UDS|TCP],
 proxy-v2 -> varnish
Message-ID: <20200521104308.9FB879C884@lists.varnish-cache.org>


commit b3ec626b975152b607d85d58d1790d3c064436ae
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Thu May 21 12:41:36 2020 +0200

    test client -> haproxy -> [UDS|TCP],proxy-v2 -> varnish
    
    and, in particular, matching client.ip against a vcl
    
    Ref #3334

diff --git a/bin/varnishtest/tests/h00006.vtc b/bin/varnishtest/tests/h00006.vtc
new file mode 100644
index 000000000..1844b7b19
--- /dev/null
+++ b/bin/varnishtest/tests/h00006.vtc
@@ -0,0 +1,55 @@
+varnishtest "haproxy tcp-mode, uds, send-proxy-v2, client ip and acl"
+
+# same as h00007.vtc, but usinguds for haproxy->varnish
+
+feature ignore_unknown_macro
+
+feature cmd {haproxy --version 2>&1 | grep -q 'HA-Proxy version'}
+
+server s1 {
+    rxreq
+    txresp -body "s1 >>> Hello world!"
+} -start
+
+varnish v1 -arg "-a ${tmpdir}/v1.sock,PROXY" -vcl+backend {
+	import std;
+
+	acl localhost {
+		"localhost";
+		"127.0.0.1";
+		"::1";
+	}
+
+	sub vcl_deliver {
+		set resp.http.cip = client.ip ~ localhost;
+		set resp.http.stdip =
+		    std.ip("" + client.ip, resolve = false) ~ localhost;
+
+		set resp.http.notcip = client.ip !~ localhost;
+		set resp.http.notstdip =
+		    std.ip("" + client.ip, resolve = false) !~ localhost;
+	}
+} -start
+
+haproxy h1 -D -conf {
+    defaults
+	mode   tcp
+	timeout connect         5s
+	timeout server          30s
+	timeout client          30s
+
+    listen ssloff
+	bind "fd@${fe1}"
+	server v1 ${tmpdir}/v1.sock send-proxy-v2
+} -start
+
+client c1 -connect ${h1_fe1_sock} {
+    txreq -url "/"
+    rxresp
+    expect resp.status == 200
+    expect resp.http.cip == true
+    expect resp.http.stdip == true
+    expect resp.http.notcip == false
+    expect resp.http.notstdip == false
+    expect resp.body == "s1 >>> Hello world!"
+} -run
diff --git a/bin/varnishtest/tests/h00007.vtc b/bin/varnishtest/tests/h00007.vtc
new file mode 100644
index 000000000..2e42472ad
--- /dev/null
+++ b/bin/varnishtest/tests/h00007.vtc
@@ -0,0 +1,55 @@
+varnishtest "haproxy tcp-mode, tcp, send-proxy-v2, client ip and acl"
+
+# same as h00006.vtc, but using tcp for haproxy->varnish
+
+feature ignore_unknown_macro
+
+feature cmd {haproxy --version 2>&1 | grep -q 'HA-Proxy version'}
+
+server s1 {
+    rxreq
+    txresp -body "s1 >>> Hello world!"
+} -start
+
+varnish v1 -proto "PROXY" -vcl+backend {
+	import std;
+
+	acl localhost {
+		"localhost";
+		"127.0.0.1";
+		"::1";
+	}
+
+	sub vcl_deliver {
+		set resp.http.cip = client.ip ~ localhost;
+		set resp.http.stdip =
+		    std.ip("" + client.ip, resolve = false) ~ localhost;
+
+		set resp.http.notcip = client.ip !~ localhost;
+		set resp.http.notstdip =
+		    std.ip("" + client.ip, resolve = false) !~ localhost;
+	}
+} -start
+
+haproxy h1 -D -conf {
+    defaults
+	mode   tcp
+	timeout connect         5s
+	timeout server          30s
+	timeout client          30s
+
+    listen ssloff
+	bind "fd@${fe1}"
+	server v1 ${v1_addr}:${v1_port} send-proxy-v2
+} -start
+
+client c1 -connect ${h1_fe1_sock} {
+    txreq -url "/"
+    rxresp
+    expect resp.status == 200
+    expect resp.http.cip == true
+    expect resp.http.stdip == true
+    expect resp.http.notcip == false
+    expect resp.http.notstdip == false
+    expect resp.body == "s1 >>> Hello world!"
+} -run

From reza at naghibi.com  Thu May 21 13:07:08 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Thu, 21 May 2020 13:07:08 +0000 (UTC)
Subject: [6.0] b90509232 Rename the obj flag OF_CHGGZIP to OF_CHGCE,
 for "changed Content-Encoding".
Message-ID: <20200521130708.31FF9A327F@lists.varnish-cache.org>


commit b90509232e5668aa79332749d0593718077f50ae
Author: Geoff Simmons <geoff at uplex.de>
Date:   Mon Feb 18 14:41:42 2019 +0100

    Rename the obj flag OF_CHGGZIP to OF_CHGCE, for "changed Content-Encoding".
    
    And update the comment explaining what happens. This is to make it
    apparent that a VFP has changed C-E in the stored object, so C-E
    should not be changed by an IMS fetch, and a new ETag from IMS has
    to be weakened. The flag is appropriate for use by any VFP that
    needs to manipulate C-E this way (not just gzip).
    
    Closes #2910

diff --git a/bin/varnishd/cache/cache_esi_fetch.c b/bin/varnishd/cache/cache_esi_fetch.c
index b1f993dbf..16ff6e749 100644
--- a/bin/varnishd/cache/cache_esi_fetch.c
+++ b/bin/varnishd/cache/cache_esi_fetch.c
@@ -163,7 +163,7 @@ vfp_esi_gzip_init(struct vfp_ctx *vc, struct vfp_entry *vfe)
 	ALLOC_OBJ(vef, VEF_MAGIC);
 	if (vef == NULL)
 		return (VFP_ERROR);
-	vc->obj_flags |= OF_GZIPED | OF_CHGGZIP | OF_ESIPROC;
+	vc->obj_flags |= OF_GZIPED | OF_CHGCE | OF_ESIPROC;
 	vef->vgz = VGZ_NewGzip(vc->wrk->vsl, "G F E");
 	vef->vep = VEP_Init(vc, vc->req, vfp_vep_callback, vef);
 	vef->ibuf_sz = cache_param->gzip_buffer;
diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index f7fd60424..92f6feb70 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -350,11 +350,12 @@ vbf_stp_startfetch(struct worker *wrk, struct busyobj *bo)
 		if (bo->stale_oc != NULL &&
 		    ObjCheckFlag(bo->wrk, bo->stale_oc, OF_IMSCAND)) {
 			AZ(bo->stale_oc->flags & (OC_F_HFM|OC_F_PRIVATE));
-			if (ObjCheckFlag(bo->wrk, bo->stale_oc, OF_CHGGZIP)) {
+			if (ObjCheckFlag(bo->wrk, bo->stale_oc, OF_CHGCE)) {
 				/*
-				 * If we changed the gzip status of the object
-				 * the stored Content_Encoding controls we
-				 * must weaken any new ETag we get.
+				 * If a VFP changed C-E in the stored
+				 * object, then don't overwrite C-E from
+				 * the IMS fetch, and we must weaken any
+				 * new ETag we get.
 				 */
 				http_Unset(bo->beresp, H_Content_Encoding);
 				RFC2616_Weaken_Etag(bo->beresp);
diff --git a/bin/varnishd/cache/cache_gzip.c b/bin/varnishd/cache/cache_gzip.c
index bfe2f6279..6483a6758 100644
--- a/bin/varnishd/cache/cache_gzip.c
+++ b/bin/varnishd/cache/cache_gzip.c
@@ -466,14 +466,14 @@ vfp_gzip_init(struct vfp_ctx *vc, struct vfp_entry *vfe)
 		if (http_GetHdr(vc->resp, H_Content_Encoding, NULL))
 			return (VFP_NULL);
 		vg = VGZ_NewGzip(vc->wrk->vsl, vfe->vfp->priv1);
-		vc->obj_flags |= OF_GZIPED | OF_CHGGZIP;
+		vc->obj_flags |= OF_GZIPED | OF_CHGCE;
 	} else {
 		if (!http_HdrIs(vc->resp, H_Content_Encoding, "gzip"))
 			return (VFP_NULL);
 		if (vfe->vfp == &VFP_gunzip) {
 			vg = VGZ_NewGunzip(vc->wrk->vsl, vfe->vfp->priv1);
 			vc->obj_flags &= ~OF_GZIPED;
-			vc->obj_flags |= OF_CHGGZIP;
+			vc->obj_flags |= OF_CHGCE;
 		} else {
 			vg = VGZ_NewTestGunzip(vc->wrk->vsl, vfe->vfp->priv1);
 			vc->obj_flags |= OF_GZIPED;
diff --git a/include/tbl/obj_attr.h b/include/tbl/obj_attr.h
index b0b6773e1..a695c1efa 100644
--- a/include/tbl/obj_attr.h
+++ b/include/tbl/obj_attr.h
@@ -55,7 +55,7 @@
 #ifdef OBJ_FLAG
 /* upper, lower, val */
   OBJ_FLAG(GZIPED,	gziped,		(1<<1))
-  OBJ_FLAG(CHGGZIP,	chggzip,	(1<<2))
+  OBJ_FLAG(CHGCE,	chgce,		(1<<2))
   OBJ_FLAG(IMSCAND,	imscand,	(1<<3))
   OBJ_FLAG(ESIPROC,	esiproc,	(1<<4))
   #undef OBJ_FLAG

From reza at naghibi.com  Thu May 21 13:07:08 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Thu, 21 May 2020 13:07:08 +0000 (UTC)
Subject: [6.0] 4763c778d Move the 304 logic into a separate function for
 clarity.
Message-ID: <20200521130708.46CF8A3282@lists.varnish-cache.org>


commit 4763c778dad9d54b1ed1bb3bb37ec64fbac0a067
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Jan 13 09:11:06 2020 +0000

    Move the 304 logic into a separate function for clarity.
    
    conflicts:
        bin/varnishd/cache/cache_fetch.c

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 92f6feb70..970a10011 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -250,6 +250,43 @@ vbf_stp_retry(struct worker *wrk, struct busyobj *bo)
 	return (F_STP_STARTFETCH);
 }
 
+/*--------------------------------------------------------------------
+ * 304 setup logic
+ */
+
+static int
+vbf_304_logic(struct busyobj *bo)
+{
+	if (bo->stale_oc != NULL &&
+	    ObjCheckFlag(bo->wrk, bo->stale_oc, OF_IMSCAND)) {
+		AZ(bo->stale_oc->flags & (OC_F_HFM|OC_F_PRIVATE));
+		if (ObjCheckFlag(bo->wrk, bo->stale_oc, OF_CHGCE)) {
+			/*
+			 * If a VFP changed C-E in the stored
+			 * object, then don't overwrite C-E from
+			 * the IMS fetch, and we must weaken any
+			 * new ETag we get.
+			 */
+			http_Unset(bo->beresp, H_Content_Encoding);
+			RFC2616_Weaken_Etag(bo->beresp);
+		}
+		http_Unset(bo->beresp, H_Content_Length);
+		HTTP_Merge(bo->wrk, bo->stale_oc, bo->beresp);
+		assert(http_IsStatus(bo->beresp, 200));
+		bo->was_304 = 1;
+	} else if (!bo->do_pass) {
+		/*
+		 * Backend sent unallowed 304
+		 */
+		VSLb(bo->vsl, SLT_Error,
+		    "304 response but not conditional fetch");
+		bo->htc->doclose = SC_RX_BAD;
+		VDI_Finish(bo->wrk, bo);
+		return (-1);
+	}
+	return (1);
+}
+
 /*--------------------------------------------------------------------
  * Setup bereq from bereq0, run vcl_backend_fetch
  */
@@ -346,35 +383,8 @@ vbf_stp_startfetch(struct worker *wrk, struct busyobj *bo)
 	AZ(bo->do_esi);
 	AZ(bo->was_304);
 
-	if (http_IsStatus(bo->beresp, 304)) {
-		if (bo->stale_oc != NULL &&
-		    ObjCheckFlag(bo->wrk, bo->stale_oc, OF_IMSCAND)) {
-			AZ(bo->stale_oc->flags & (OC_F_HFM|OC_F_PRIVATE));
-			if (ObjCheckFlag(bo->wrk, bo->stale_oc, OF_CHGCE)) {
-				/*
-				 * If a VFP changed C-E in the stored
-				 * object, then don't overwrite C-E from
-				 * the IMS fetch, and we must weaken any
-				 * new ETag we get.
-				 */
-				http_Unset(bo->beresp, H_Content_Encoding);
-				RFC2616_Weaken_Etag(bo->beresp);
-			}
-			http_Unset(bo->beresp, H_Content_Length);
-			HTTP_Merge(bo->wrk, bo->stale_oc, bo->beresp);
-			assert(http_IsStatus(bo->beresp, 200));
-			bo->was_304 = 1;
-		} else if (!bo->do_pass) {
-			/*
-			 * Backend sent unallowed 304
-			 */
-			VSLb(bo->vsl, SLT_Error,
-			    "304 response but not conditional fetch");
-			bo->htc->doclose = SC_RX_BAD;
-			VDI_Finish(bo->wrk, bo);
-			return (F_STP_ERROR);
-		}
-	}
+	if (http_IsStatus(bo->beresp, 304) && vbf_304_logic(bo) < 0)
+		return (F_STP_ERROR);
 
 	VCL_backend_response_method(bo->vcl, wrk, NULL, bo, NULL);
 

From reza at naghibi.com  Thu May 21 13:07:08 2020
From: reza at naghibi.com (Reza Naghibi)
Date: Thu, 21 May 2020 13:07:08 +0000 (UTC)
Subject: [6.0] 66ad64fe6 Ensure we don't override Content-Encoding in a 304
 conditional fetch
Message-ID: <20200521130708.66F0DA3289@lists.varnish-cache.org>


commit 66ad64fe6b9d824429639095e726ffb28a091b92
Author: Dag Haavi Finstad <daghf at varnish-software.com>
Date:   Thu Nov 21 11:50:23 2019 +0100

    Ensure we don't override Content-Encoding in a 304 conditional fetch
    
    Test inspired by Carlos Abalde

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 970a10011..bd02fc822 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -267,9 +267,9 @@ vbf_304_logic(struct busyobj *bo)
 			 * the IMS fetch, and we must weaken any
 			 * new ETag we get.
 			 */
-			http_Unset(bo->beresp, H_Content_Encoding);
 			RFC2616_Weaken_Etag(bo->beresp);
 		}
+		http_Unset(bo->beresp, H_Content_Encoding);
 		http_Unset(bo->beresp, H_Content_Length);
 		HTTP_Merge(bo->wrk, bo->stale_oc, bo->beresp);
 		assert(http_IsStatus(bo->beresp, 200));
diff --git a/bin/varnishtest/tests/r03169.vtc b/bin/varnishtest/tests/r03169.vtc
new file mode 100644
index 000000000..650b15c83
--- /dev/null
+++ b/bin/varnishtest/tests/r03169.vtc
@@ -0,0 +1,61 @@
+varnishtest "Ensure we don't override Content-Encoding on a cond fetch"
+
+server s1 {
+    rxreq
+    txresp -hdr {ETag: "foo"} -body {****}
+
+    rxreq
+    expect req.http.If-None-Match == {"foo"}
+    txresp -status 304 -hdr {ETag: W/"foo"} -hdr "Content-Encoding: gzip"
+
+    rxreq
+    expect req.url == "/1"
+    txresp -hdr {Etag: "bar"} -gzipbody {asdf}
+
+    rxreq
+    expect req.url == "/1"
+    expect req.http.If-None-Match == {"bar"}
+    txresp -status 304
+
+} -start
+
+varnish v1 -vcl+backend {
+    sub vcl_backend_response {
+        set beresp.ttl = 1s;
+        set beresp.grace = 0s;
+        set beresp.keep = 1d;
+    }
+} -start
+
+client c1 {
+    txreq -hdr "Accept-Encoding: gzip"
+    rxresp
+    expect resp.status == 200
+    expect resp.http.ETag == {"foo"}
+    expect resp.http.Content-Length == "4"
+    expect resp.http.Content-Encoding == <undef>
+
+    delay 1.5
+
+    txreq -hdr "Accept-Encoding: gzip"
+    rxresp
+    expect resp.status == 200
+    expect resp.http.ETag == {W/"foo"}
+    expect resp.http.Content-Length == "4"
+    expect resp.http.Content-Encoding == <undef>
+
+    # Also check that we're still OK in the general case
+    txreq -url "/1" -hdr "Accept-Encoding: gzip"
+    rxresp
+    expect resp.status == 200
+    expect resp.http.ETag == {"bar"}
+    expect resp.http.Content-Encoding == "gzip"
+
+    delay 1.5
+
+    txreq -url "/1" -hdr "Accept-Encoding: gzip"
+    rxresp
+    expect resp.status == 200
+    expect resp.http.ETag == {"bar"}
+    expect resp.http.Content-Encoding == "gzip"
+} -run
\ No newline at end of file

From phk at FreeBSD.org  Sat May 23 07:17:08 2020
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Sat, 23 May 2020 07:17:08 +0000 (UTC)
Subject: [master] d64383569 Fix this test for FreeBSD Jail use.
Message-ID: <20200523071708.335E17044@lists.varnish-cache.org>


commit d6438356906c8048a275894d33fa6b4893e1b135
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sat May 23 07:15:59 2020 +0000

    Fix this test for FreeBSD Jail use.

diff --git a/bin/varnishtest/tests/h00006.vtc b/bin/varnishtest/tests/h00006.vtc
index 1844b7b19..556cd6d05 100644
--- a/bin/varnishtest/tests/h00006.vtc
+++ b/bin/varnishtest/tests/h00006.vtc
@@ -1,6 +1,6 @@
 varnishtest "haproxy tcp-mode, uds, send-proxy-v2, client ip and acl"
 
-# same as h00007.vtc, but usinguds for haproxy->varnish
+# same as h00007.vtc, but using uds for haproxy->varnish
 
 feature ignore_unknown_macro
 
@@ -11,6 +11,18 @@ server s1 {
     txresp -body "s1 >>> Hello world!"
 } -start
 
+haproxy h1 -D -conf {
+    defaults
+	mode   tcp
+	timeout connect         5s
+	timeout server          30s
+	timeout client          30s
+
+    listen ssloff
+	bind "fd@${fe1}"
+	server v1 ${tmpdir}/v1.sock send-proxy-v2
+} -start
+
 varnish v1 -arg "-a ${tmpdir}/v1.sock,PROXY" -vcl+backend {
 	import std;
 
@@ -18,6 +30,8 @@ varnish v1 -arg "-a ${tmpdir}/v1.sock,PROXY" -vcl+backend {
 		"localhost";
 		"127.0.0.1";
 		"::1";
+		"${s1_addr}";		// Jails IPv4 address
+		"${h1_fe1_addr}";	// Jails IPv6 address
 	}
 
 	sub vcl_deliver {
@@ -31,18 +45,6 @@ varnish v1 -arg "-a ${tmpdir}/v1.sock,PROXY" -vcl+backend {
 	}
 } -start
 
-haproxy h1 -D -conf {
-    defaults
-	mode   tcp
-	timeout connect         5s
-	timeout server          30s
-	timeout client          30s
-
-    listen ssloff
-	bind "fd@${fe1}"
-	server v1 ${tmpdir}/v1.sock send-proxy-v2
-} -start
-
 client c1 -connect ${h1_fe1_sock} {
     txreq -url "/"
     rxresp

From phk at FreeBSD.org  Sat May 23 13:34:07 2020
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Sat, 23 May 2020 13:34:07 +0000 (UTC)
Subject: [master] 4d006b802 Also fix this test for Jails
Message-ID: <20200523133407.0C41C92773@lists.varnish-cache.org>


commit 4d006b802f8d34455d8d289aac983dcd247c35a8
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sat May 23 13:33:39 2020 +0000

    Also fix this test for Jails

diff --git a/bin/varnishtest/tests/h00007.vtc b/bin/varnishtest/tests/h00007.vtc
index 2e42472ad..237c45cae 100644
--- a/bin/varnishtest/tests/h00007.vtc
+++ b/bin/varnishtest/tests/h00007.vtc
@@ -11,13 +11,29 @@ server s1 {
     txresp -body "s1 >>> Hello world!"
 } -start
 
-varnish v1 -proto "PROXY" -vcl+backend {
+varnish v1 -proto "PROXY" -vcl+backend {} -start
+
+haproxy h1 -D -conf {
+    defaults
+	mode   tcp
+	timeout connect         5s
+	timeout server          30s
+	timeout client          30s
+
+    listen ssloff
+	bind "fd@${fe1}"
+	server v1 ${v1_addr}:${v1_port} send-proxy-v2
+} -start
+
+varnish v1 -vcl+backend {
 	import std;
 
 	acl localhost {
 		"localhost";
 		"127.0.0.1";
 		"::1";
+		"${s1_addr}";		# Jail IPv4 address
+		"${h1_fe1_addr}";	# Jail IPv6 address
 	}
 
 	sub vcl_deliver {
@@ -29,19 +45,7 @@ varnish v1 -proto "PROXY" -vcl+backend {
 		set resp.http.notstdip =
 		    std.ip("" + client.ip, resolve = false) !~ localhost;
 	}
-} -start
-
-haproxy h1 -D -conf {
-    defaults
-	mode   tcp
-	timeout connect         5s
-	timeout server          30s
-	timeout client          30s
-
-    listen ssloff
-	bind "fd@${fe1}"
-	server v1 ${v1_addr}:${v1_port} send-proxy-v2
-} -start
+} 
 
 client c1 -connect ${h1_fe1_sock} {
     txreq -url "/"

From fgsch at lodoss.net  Sat May 23 20:08:08 2020
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Sat, 23 May 2020 20:08:08 +0000 (UTC)
Subject: [master] ebd64fd68 Typo
Message-ID: <20200523200808.ED267A24E2@lists.varnish-cache.org>


commit ebd64fd68c9b8f0805050598eae2581b4c55c7d9
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Sat May 23 21:06:38 2020 +0100

    Typo

diff --git a/bin/varnishd/VSC_vbe.vsc b/bin/varnishd/VSC_vbe.vsc
index c1c01632d..c26207af9 100644
--- a/bin/varnishd/VSC_vbe.vsc
+++ b/bin/varnishd/VSC_vbe.vsc
@@ -76,7 +76,7 @@
 	The number of currently used connections to the backend. This
 	number is always less or equal to the number of connections to
 	the backend (as, for example shown as ESTABLISHED for TCP
-	onnections in netstat) due to connection pooling.
+	connections in netstat) due to connection pooling.
 
 .. varnish_vsc:: req
 	:type:	counter

From phk at FreeBSD.org  Mon May 25 06:13:06 2020
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 25 May 2020 06:13:06 +0000 (UTC)
Subject: [master] 899ca1bd0 Don't expect rm(1) to live in /bin,
 find it with ${PATH}
Message-ID: <20200525061306.53A6FAD27C@lists.varnish-cache.org>


commit 899ca1bd0ea176fe061bd804f8efb6e68f48391b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon May 25 06:11:52 2020 +0000

    Don't expect rm(1) to live in /bin, find it with ${PATH}
    
    Fixes: #3327

diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c
index 6b7ebf0a6..1deea4ab7 100644
--- a/bin/varnishtest/vtc_main.c
+++ b/bin/varnishtest/vtc_main.c
@@ -247,8 +247,8 @@ cleaner_setup(void)
 			pp = fork();
 			assert(pp >= 0);
 			if (pp == 0)
-				exit(execl(
-				    "/bin/rm", "rm", "-rf", buf, (char*)0));
+				exit(execlp(
+				    "rm", "rm", "-rf", buf, (char*)0));
 			assert(waitpid(pp, &st, 0) == pp);
 			AZ(st);
 		}

From phk at FreeBSD.org  Mon May 25 08:17:07 2020
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 25 May 2020 08:17:07 +0000 (UTC)
Subject: [master] e4fb2a943 Refactor pid-file creation into a separate
 function.
Message-ID: <20200525081707.6EDABB27EC@lists.varnish-cache.org>


commit e4fb2a943f75d97d1e8672b49050e13e6870674c
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon May 25 06:35:11 2020 +0000

    Refactor pid-file creation into a separate function.

diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index 0b21c7c61..79335bda4 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -36,6 +36,7 @@
 #include <ctype.h>
 #include <fcntl.h>
 #include <signal.h>
+#include <stdarg.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
@@ -431,6 +432,32 @@ mgt_f_read(const char *fn)
 	VTAILQ_INSERT_TAIL(&f_args, fa, list);
 }
 
+static struct vpf_fh *
+create_pid_file(pid_t *ppid, const char *fmt, ...)
+{
+	struct vsb *vsb;
+	va_list ap;
+	struct vpf_fh *pfh;
+
+	va_start(ap, fmt);
+	vsb = VSB_new_auto();
+	AN(vsb);
+	VSB_vprintf(vsb, fmt, ap);
+	AZ(VSB_finish(vsb));
+	VJ_master(JAIL_MASTER_FILE);
+	pfh = VPF_Open(VSB_data(vsb), 0644, ppid);
+	if (pfh == NULL && errno == EEXIST)
+		ARGV_ERR(
+		    "Varnishd is already running (pid=%jd) (pidfile=%s)\n",
+		    (intmax_t)*ppid, VSB_data(vsb));
+	if (pfh == NULL)
+		ARGV_ERR("Could not open pid-file (%s): %s\n",
+		    VSB_data(vsb), vstrerror(errno));
+	VJ_master(JAIL_MASTER_LOW);
+	VSB_destroy(&vsb);
+	return (pfh);
+}
+
 int
 main(int argc, char * const *argv)
 {
@@ -764,29 +791,10 @@ main(int argc, char * const *argv)
 		    dirname, vstrerror(errno));
 	}
 
-	vsb = VSB_new_auto();
-	AN(vsb);
-	VSB_printf(vsb, "%s/_.pid", dirname);
-	AZ(VSB_finish(vsb));
-	VJ_master(JAIL_MASTER_FILE);
-	pfh1 = VPF_Open(VSB_data(vsb), 0644, &pid);
-	if (pfh1 == NULL && errno == EEXIST)
-		ARGV_ERR("Varnishd is already running (pid=%jd)\n",
-		    (intmax_t)pid);
-	if (pfh1 == NULL)
-		ARGV_ERR("Could not open pid-file (%s): %s\n",
-		    VSB_data(vsb), vstrerror(errno));
-	VSB_destroy(&vsb);
-	if (P_arg) {
-		pfh2 = VPF_Open(P_arg, 0644, &pid);
-		if (pfh2 == NULL && errno == EEXIST)
-			ARGV_ERR("Varnishd is already running (pid=%jd)\n",
-			    (intmax_t)pid);
-		if (pfh2 == NULL)
-			ARGV_ERR("Could not open pid-file (%s): %s\n",
-			    P_arg, vstrerror(errno));
-	}
-	VJ_master(JAIL_MASTER_LOW);
+	pfh1 = create_pid_file(&pid, "%s/_.pid", dirname);
+
+	if (P_arg)
+		pfh2 = create_pid_file(&pid, "%s", P_arg);
 
 	/* If no -s argument specified, process default -s argument */
 	if (!s_arg_given)

From phk at FreeBSD.org  Mon May 25 08:17:07 2020
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 25 May 2020 08:17:07 +0000 (UTC)
Subject: [master] 8de6262f3 Removing files and directories needs Jail attention
Message-ID: <20200525081707.8308FB27EF@lists.varnish-cache.org>


commit 8de6262f3c838aa8f23ab1fbbcd9aa45dfdc1121
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon May 25 08:13:27 2020 +0000

    Removing files and directories needs Jail attention
    
    Add VJ_unlink() and VJ_rmdir() to do so.

diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index 54b42ad80..0456d84b4 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -147,6 +147,8 @@ void VJ_subproc(enum jail_subproc_e);
 int VJ_make_workdir(const char *);
 int VJ_make_subdir(const char *, const char *, struct vsb *);
 void VJ_fix_fd(int, enum jail_fixfd_e);
+void VJ_unlink(const char *);
+void VJ_rmdir(const char *);
 
 extern const struct jail_tech jail_tech_unix;
 extern const struct jail_tech jail_tech_solaris;
diff --git a/bin/varnishd/mgt/mgt_jail.c b/bin/varnishd/mgt/mgt_jail.c
index 36b426ebf..615f6735d 100644
--- a/bin/varnishd/mgt/mgt_jail.c
+++ b/bin/varnishd/mgt/mgt_jail.c
@@ -196,6 +196,28 @@ VJ_make_subdir(const char *dname, const char *what, struct vsb *vsb)
 	return (0);
 }
 
+void
+VJ_unlink(const char *fname)
+{
+	VJ_master(JAIL_MASTER_FILE);
+	if (unlink(fname)) {
+		fprintf(stderr, "Could not delete '%s': %s\n",
+		    fname, strerror(errno));
+	}
+	VJ_master(JAIL_MASTER_LOW);
+}
+
+void
+VJ_rmdir(const char *dname)
+{
+	VJ_master(JAIL_MASTER_FILE);
+	if (rmdir(dname)) {
+		fprintf(stderr, "Could not rmdir '%s': %s\n",
+		    dname, strerror(errno));
+	}
+	VJ_master(JAIL_MASTER_LOW);
+}
+
 void
 VJ_fix_fd(int fd, enum jail_fixfd_e what)
 {

From phk at FreeBSD.org  Mon May 25 08:17:07 2020
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 25 May 2020 08:17:07 +0000 (UTC)
Subject: [master] d5e5d47b4 No need to use bogos vcl names with -C
Message-ID: <20200525081707.9D99DB27F3@lists.varnish-cache.org>


commit d5e5d47b4305cf9f8148419cd2ae3e8ed335528f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon May 25 08:14:51 2020 +0000

    No need to use bogos vcl names with -C

diff --git a/bin/varnishd/mgt/mgt_vcl.c b/bin/varnishd/mgt/mgt_vcl.c
index c15d3879d..359877aa4 100644
--- a/bin/varnishd/mgt/mgt_vcl.c
+++ b/bin/varnishd/mgt/mgt_vcl.c
@@ -424,14 +424,10 @@ mgt_new_vcl(struct cli *cli, const char *vclname, const char *vclsrc,
 	char *lib, *p;
 	struct vclprog *vp;
 	const struct vclstate *vs;
-	char buf[32];
 
 	AN(cli);
 
-	if (C_flag) {
-		bprintf(buf, ".CflagTest.%d", (int)getpid());
-		vclname = buf;
-	} else if (vcl_count >= mgt_param.max_vcl &&
+	if (vcl_count >= mgt_param.max_vcl &&
 	    mgt_param.max_vcl_handling == 2) {
 		VCLI_Out(cli, "Too many (%d) VCLs already loaded\n", vcl_count);
 		VCLI_Out(cli, "(See max_vcl and max_vcl_handling parameters)");

From phk at FreeBSD.org  Mon May 25 08:17:07 2020
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 25 May 2020 08:17:07 +0000 (UTC)
Subject: [master] 1e977a772 Use Jail functions to remove files and directories.
Message-ID: <20200525081707.B736EB27F7@lists.varnish-cache.org>


commit 1e977a772b1fcc56e025e968e2825862e2828801
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon May 25 08:15:21 2020 +0000

    Use Jail functions to remove files and directories.

diff --git a/bin/varnishd/mgt/mgt_vcc.c b/bin/varnishd/mgt/mgt_vcc.c
index e82fa81c5..1d1d86d58 100644
--- a/bin/varnishd/mgt/mgt_vcc.c
+++ b/bin/varnishd/mgt/mgt_vcc.c
@@ -275,12 +275,12 @@ static void
 mgt_vcc_fini_vp(struct vcc_priv *vp, int leave_lib)
 {
 	if (!MGT_DO_DEBUG(DBG_VCL_KEEP)) {
-		(void)unlink(VSB_data(vp->csrcfile));
-		(void)unlink(VSB_data(vp->symfile));
+		VJ_unlink(VSB_data(vp->csrcfile));
+		VJ_unlink(VSB_data(vp->symfile));
 		if (!leave_lib)
-			(void)unlink(VSB_data(vp->libfile));
+			VJ_unlink(VSB_data(vp->libfile));
 	}
-	(void)rmdir(VSB_data(vp->dir));
+	VJ_rmdir(VSB_data(vp->dir));
 	VSB_destroy(&vp->csrcfile);
 	VSB_destroy(&vp->libfile);
 	VSB_destroy(&vp->symfile);

From phk at FreeBSD.org  Mon May 25 08:17:07 2020
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 25 May 2020 08:17:07 +0000 (UTC)
Subject: [master] d4ad4ecd8 Have -C always remove the workdir.
Message-ID: <20200525081707.CFF73B27FB@lists.varnish-cache.org>


commit d4ad4ecd87e4fc6c1cd6e7fa375907a835620178
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon May 25 08:15:47 2020 +0000

    Have -C always remove the workdir.
    
    Previously it only did so if no -n argument was given.
    
    Fixes #3307

diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index 79335bda4..e2fa37ba4 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -71,7 +71,7 @@ int			exit_status = 0;
 struct vsb		*vident;
 struct VSC_mgt		*VSC_C_mgt;
 static int		I_fd = -1;
-static char		*Cn_arg;
+static char		*workdir;
 
 static struct vpf_fh *pfh1 = NULL;
 static struct vpf_fh *pfh2 = NULL;
@@ -250,9 +250,9 @@ mgt_Cflag_atexit(void)
 	/* Only master process */
 	if (getpid() != heritage.mgt_pid)
 		return;
-	(void)rmdir("vmod_cache");
-	(void)unlink("_.pid");
-	(void)rmdir(Cn_arg);
+	VJ_rmdir("vmod_cache");
+	VJ_unlink("_.pid");
+	VJ_rmdir(workdir);
 }
 
 /*--------------------------------------------------------------------*/
@@ -480,7 +480,6 @@ main(int argc, char * const *argv)
 	const char *T_arg = "localhost:0";
 	char *p;
 	struct cli cli[1];
-	char *dirname;
 	char **av;
 	unsigned u;
 	struct sigaction sac;
@@ -747,13 +746,11 @@ main(int argc, char * const *argv)
 				VSB_cat(vsb, "/tmp");
 			VSB_cat(vsb, "/varnishd_C_XXXXXXX");
 			AZ(VSB_finish(vsb));
-			Cn_arg = strdup(VSB_data(vsb));
-			AN(Cn_arg);
+			p = strdup(VSB_data(vsb));
+			AN(p);
 			VSB_destroy(&vsb);
-			AN(mkdtemp(Cn_arg));
-			AZ(chmod(Cn_arg, 0755));
-			AZ(atexit(mgt_Cflag_atexit));
-			n_arg = Cn_arg;
+			AN(mkdtemp(p));
+			n_arg = p;
 		}
 	}
 
@@ -770,7 +767,7 @@ main(int argc, char * const *argv)
 		VJ_master(JAIL_MASTER_LOW);
 	}
 
-	if (VIN_n_Arg(n_arg, &dirname) != 0)
+	if (VIN_n_Arg(n_arg, &workdir) != 0)
 		ARGV_ERR("Invalid instance (-n) name: %s\n", vstrerror(errno));
 
 	if (i_arg == NULL || *i_arg == '\0')
@@ -781,17 +778,20 @@ main(int argc, char * const *argv)
 
 	openlog("varnishd", LOG_PID, LOG_LOCAL0);
 
-	if (VJ_make_workdir(dirname))
+	if (VJ_make_workdir(workdir))
 		ARGV_ERR("Cannot create working directory (%s): %s\n",
-		    dirname, vstrerror(errno));
+		    workdir, vstrerror(errno));
 
 	if (VJ_make_subdir("vmod_cache", "VMOD cache", NULL)) {
 		ARGV_ERR(
 		    "Cannot create vmod directory (%s/vmod_cache): %s\n",
-		    dirname, vstrerror(errno));
+		    workdir, vstrerror(errno));
 	}
 
-	pfh1 = create_pid_file(&pid, "%s/_.pid", dirname);
+	if (C_flag)
+		AZ(atexit(mgt_Cflag_atexit));
+
+	pfh1 = create_pid_file(&pid, "%s/_.pid", workdir);
 
 	if (P_arg)
 		pfh2 = create_pid_file(&pid, "%s", P_arg);
@@ -805,31 +805,29 @@ main(int argc, char * const *argv)
 
 	mgt_vcl_init();
 
-	if (C_flag) {
-		VTAILQ_FOREACH(fa, &f_args, list) {
-			mgt_vcl_startup(cli, fa->src,
-			    VTAILQ_NEXT(fa, list) == NULL ? "boot" : NULL,
-			    fa->farg, 1);
+	u = 0;
+	while (!VTAILQ_EMPTY(&f_args)) {
+		fa = VTAILQ_FIRST(&f_args);
+		VTAILQ_REMOVE(&f_args, fa, list);
+		mgt_vcl_startup(cli, fa->src,
+		    VTAILQ_EMPTY(&f_args) ? "boot" : NULL,
+		    fa->farg, C_flag);
+		if (C_flag) {
+			if (cli->result != CLIS_OK &&
+			    cli->result != CLIS_TRUNCATED)
+				u = 2;
 			AZ(VSB_finish(cli->sb));
 			fprintf(stderr, "%s\n", VSB_data(cli->sb));
 			VSB_clear(cli->sb);
-		}
-		if (cli->result == CLIS_OK || cli->result == CLIS_TRUNCATED)
-			exit(0);
-		exit(2);
-	} else {
-		while (!VTAILQ_EMPTY(&f_args)) {
-			fa = VTAILQ_FIRST(&f_args);
-			VTAILQ_REMOVE(&f_args, fa, list);
-			mgt_vcl_startup(cli, fa->src,
-			    VTAILQ_EMPTY(&f_args) ? "boot" : NULL,
-			    fa->farg, 0);
+		} else {
 			cli_check(cli);
-			free(fa->farg);
-			free(fa->src);
-			FREE_OBJ(fa);
 		}
+		free(fa->farg);
+		free(fa->src);
+		FREE_OBJ(fa);
 	}
+	if (C_flag)
+		exit(u);
 
 	if (VTAILQ_EMPTY(&heritage.socks))
 		MAC_Arg(":80\0");	// XXX: extra NUL for FlexeLint
@@ -853,7 +851,7 @@ main(int argc, char * const *argv)
 	AZ(VSB_finish(vident));
 
 	if (S_arg == NULL)
-		S_arg = make_secret(dirname);
+		S_arg = make_secret(workdir);
 	AN(S_arg);
 
 	VPF_Write(pfh1);

From phk at FreeBSD.org  Mon May 25 09:11:06 2020
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 25 May 2020 09:11:06 +0000 (UTC)
Subject: [master] b7398dbf3 Reset the dir mode to 750 after mkdtemp(3) returns.
Message-ID: <20200525091106.7CC344D7D@lists.varnish-cache.org>


commit b7398dbf360f4ee5a2350dd76a5c0db1db716617
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon May 25 09:09:00 2020 +0000

    Reset the dir mode to 750 after mkdtemp(3) returns.
    
    Mkdtemp(3) creates the directory with mode 0700, and since it
    already exists, we do not change the mode subsequently, which
    causes jailed with uid=vcache to keel over.

diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index e2fa37ba4..cd63ed93b 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -750,6 +750,7 @@ main(int argc, char * const *argv)
 			AN(p);
 			VSB_destroy(&vsb);
 			AN(mkdtemp(p));
+			AZ(chmod(p, 0750));
 			n_arg = p;
 		}
 	}

From dridi.boukelmoune at gmail.com  Mon May 25 14:05:06 2020
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 25 May 2020 14:05:06 +0000 (UTC)
Subject: [master] ce037da73 Add varnishd -C coverage for #3307
Message-ID: <20200525140506.88400A14AF@lists.varnish-cache.org>


commit ce037da7304833c20b58146f1060da41355d83b3
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Mon May 25 16:03:33 2020 +0200

    Add varnishd -C coverage for #3307

diff --git a/bin/varnishtest/tests/u00000.vtc b/bin/varnishtest/tests/u00000.vtc
index 46bc0c013..f7c13fb27 100644
--- a/bin/varnishtest/tests/u00000.vtc
+++ b/bin/varnishtest/tests/u00000.vtc
@@ -1,6 +1,17 @@
 varnishtest "Code coverage of mgt_main, (VCL compiler and RSTdump etc)"
 
 shell "varnishd -b 127.0.0.1:80 -C 2> ${tmpdir}/_.c"
+
+shell {
+	varnishd -n ${tmpdir}/no_keep -C -b 127.0.0.1:80 2> no_keep.c
+	test -s no_keep.c && ! test -d no_keep
+}
+
+shell {
+	varnishd -n ${tmpdir}/keep -p debug=+vcl_keep -C -b 127.0.0.1:80 2> keep.c
+	test -s keep.c && test -d keep
+}
+
 shell -err -expect {VCL version declaration missing} {
 	echo 'bad vcl' > ${tmpdir}/bad.vcl
 	varnishd -f ${tmpdir}/bad.vcl -n ${tmpdir}

From dridi.boukelmoune at gmail.com  Mon May 25 16:14:07 2020
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 25 May 2020 16:14:07 +0000 (UTC)
Subject: [6.0] 009c6753d Fix compilation under macos
Message-ID: <20200525161407.4D9EFA4965@lists.varnish-cache.org>


commit 009c6753d5c35eadd9c49341d9fe70fa8b48245c
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Sun May 10 20:46:15 2020 +0100

    Fix compilation under macos

diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index 42f2995a1..4893c20df 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -58,18 +58,18 @@ VRT_synth(VRT_CTX, VCL_INT code, VCL_STRING reason)
 	assert(ctx->req != NULL || ctx->bo != NULL);
 	if (code < 0) {
 		VRT_fail(ctx, "return(synth()) status code (%jd) is negative",
-		    code);
+		    (intmax_t)code);
 		return;
 	}
 	if (code > 65535) {
 		VRT_fail(ctx, "return(synth()) status code (%jd) > 65535",
-		    code);
+		    (intmax_t)code);
 		return;
 	}
 	if ((code % 1000) < 100) {
 		VRT_fail(ctx,
 		    "illegal return(synth()) status code (%jd) (..0##)",
-		    code);
+		    (intmax_t)code);
 		return;
 	}
 

From dridi.boukelmoune at gmail.com  Mon May 25 16:14:07 2020
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 25 May 2020 16:14:07 +0000 (UTC)
Subject: [6.0] 308ff5da1 Reference the correct transition in VRT_synth()
Message-ID: <20200525161407.64C06A4968@lists.varnish-cache.org>


commit 308ff5da1152a5c3cb500d676fe15f1a717d5a8f
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue May 12 08:59:50 2020 +0200

    Reference the correct transition in VRT_synth()

diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index 4893c20df..d0188492e 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -53,23 +53,26 @@ const void * const vrt_magic_string_unset = &vrt_magic_string_unset;
 VCL_VOID
 VRT_synth(VRT_CTX, VCL_INT code, VCL_STRING reason)
 {
+	const char *ret;
 
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
 	assert(ctx->req != NULL || ctx->bo != NULL);
+
+	ret = ctx->req == NULL ? "error" : "synth";
 	if (code < 0) {
-		VRT_fail(ctx, "return(synth()) status code (%jd) is negative",
-		    (intmax_t)code);
+		VRT_fail(ctx, "return(%s()) status code (%jd) is negative",
+		    ret, (intmax_t)code);
 		return;
 	}
 	if (code > 65535) {
-		VRT_fail(ctx, "return(synth()) status code (%jd) > 65535",
-		    (intmax_t)code);
+		VRT_fail(ctx, "return(%s()) status code (%jd) > 65535",
+		    ret, (intmax_t)code);
 		return;
 	}
 	if ((code % 1000) < 100) {
 		VRT_fail(ctx,
-		    "illegal return(synth()) status code (%jd) (..0##)",
-		    (intmax_t)code);
+		    "illegal return(%s()) status code (%jd) (..0##)",
+		    ret, (intmax_t)code);
 		return;
 	}
 

From dridi.boukelmoune at gmail.com  Mon May 25 16:14:07 2020
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 25 May 2020 16:14:07 +0000 (UTC)
Subject: [6.0] a15a5fcb6 Centralize the VCL status code documentation
Message-ID: <20200525161407.87ABEA496B@lists.varnish-cache.org>


commit a15a5fcb6ec379a3d54eedc412d97a1bdfb6bdcf
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue May 12 09:03:38 2020 +0200

    Centralize the VCL status code documentation
    
    Conflicts:
            doc/sphinx/reference/vcl_var.rst

diff --git a/doc/sphinx/reference/vcl_var.rst b/doc/sphinx/reference/vcl_var.rst
index 99c114f39..55b0defca 100644
--- a/doc/sphinx/reference/vcl_var.rst
+++ b/doc/sphinx/reference/vcl_var.rst
@@ -702,13 +702,7 @@ beresp.status
 
 	The HTTP status code returned by the server.
 
-	Status codes on the form XXYZZ can be set where
-	XXYZZ is less than 65536 and Y is [1...9].
-	Only YZZ will be sent back to clients.
-
-	XX can be therefore be used to pass information
-	around inside VCL, for instance `return(synth(22404))`
-	from `vcl_recv{}` to `vcl_synth{}`
+	More information in the `HTTP response status`_ section.
 
 beresp.reason
 
@@ -953,6 +947,8 @@ obj.status
 
 	The HTTP status code stored in the object.
 
+	More information in the `HTTP response status`_ section.
+
 
 obj.reason
 
@@ -1096,12 +1092,7 @@ resp.status
 
 	The HTTP status code that will be returned.
 
-	Assigning a HTTP standardized code to resp.status will also
-	set resp.reason to the corresponding status message.
-
-	Status codes on the form XXYZZ can be set where
-	XXYZZ is less than 65536 and Y is [1...9].
-	Only YZZ will be sent back to clients.
+	More information in the `HTTP response status`_ section.
 
 	resp.status 200 will get changed into 304 by core code after
 	a return(deliver) from vcl_deliver for conditional requests
@@ -1235,3 +1226,29 @@ storage.<name>.happy
 	Health status for the named stevedore. Not available in any of the
 	current stevedores.
 
+HTTP response status
+--------------------
+
+A status code normally has 3 digits XYZ where X must be between 1 and 5
+included. Since it is not uncommon to see HTTP clients or servers relying
+on non-standard or even invalid status codes Varnish tolerates any status
+between 100 and 999.
+
+With VCL code it is possible to use status codes in the form XXYZZ where the
+overall value is lower than 65536 and the Y digit is between 1 and 9 included.
+Only the YZZ part is sent to the client.
+
+The XXYZZ form of status codes can be set on ``resp.status`` and
+``beresp.status`` or passed via ``return(synth(...))`` and
+``return(error(...))`` transitions.
+
+XX can be therefore be used to pass information around inside VCL, for
+instance ``return(synth(22404))`` from ``vcl_recv{}`` to ``vcl_synth{}``.
+
+The ``obj.status`` variable will inherit the XXYZZ form, but in a ban
+expresion only the YZZ part will be available. The XXYZZ form is strictly
+limited to VCL execution.
+
+Assigning an HTTP standardized code to ``resp.status`` or ``beresp.status``
+will also set ``resp.reason`` or ``beresp.reason``  to the corresponding
+status message.

From guillaume at varnish-software.com  Tue May 26 14:34:08 2020
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Tue, 26 May 2020 14:34:08 +0000 (UTC)
Subject: [master] df3994c3a [cci] get the dsc file too
Message-ID: <20200526143408.13D849583E@lists.varnish-cache.org>


commit df3994c3a2e19f36498c4f71f889fee6bc075776
Author: Guillaume Quintard <guillaume at varnish-software.com>
Date:   Tue May 26 07:32:51 2020 -0700

    [cci] get the dsc file too

diff --git a/.circleci/make-deb-packages.sh b/.circleci/make-deb-packages.sh
index a2b84f223..f4939ec1e 100755
--- a/.circleci/make-deb-packages.sh
+++ b/.circleci/make-deb-packages.sh
@@ -49,4 +49,4 @@ dpkg-buildpackage -us -uc -j16
 
 echo "Prepare the packages for storage..."
 mkdir -p packages/$PARAM_DIST/$PARAM_RELEASE/
-mv ../*.deb packages/$PARAM_DIST/$PARAM_RELEASE/
+mv ../*.deb ../*.dsc packages/$PARAM_DIST/$PARAM_RELEASE/

From dridi.boukelmoune at gmail.com  Wed May 27 11:15:09 2020
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Wed, 27 May 2020 11:15:09 +0000 (UTC)
Subject: [master] 6f1563cf9 vtest: Temporary debugging in u0.vtc
Message-ID: <20200527111509.CBD309ACA@lists.varnish-cache.org>


commit 6f1563cf9b301677f2b2934d7ef97419cd74a7be
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed May 27 13:11:52 2020 +0200

    vtest: Temporary debugging in u0.vtc
    
    For some reason this is failing on some platforms reporting on VTEST.

diff --git a/bin/varnishtest/tests/u00000.vtc b/bin/varnishtest/tests/u00000.vtc
index f7c13fb27..d26626eb2 100644
--- a/bin/varnishtest/tests/u00000.vtc
+++ b/bin/varnishtest/tests/u00000.vtc
@@ -4,11 +4,15 @@ shell "varnishd -b 127.0.0.1:80 -C 2> ${tmpdir}/_.c"
 
 shell {
 	varnishd -n ${tmpdir}/no_keep -C -b 127.0.0.1:80 2> no_keep.c
+	ls -l
+	set -x
 	test -s no_keep.c && ! test -d no_keep
 }
 
 shell {
 	varnishd -n ${tmpdir}/keep -p debug=+vcl_keep -C -b 127.0.0.1:80 2> keep.c
+	ls -l
+	set -x
 	test -s keep.c && test -d keep
 }
 

From dridi.boukelmoune at gmail.com  Wed May 27 13:02:07 2020
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Wed, 27 May 2020 13:02:07 +0000 (UTC)
Subject: [master] b64cb1ec0 Centralize VFP storage error log
Message-ID: <20200527130207.0DAF7637D9@lists.varnish-cache.org>


commit b64cb1ec0c287a894513709061309140dcb42a74
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed May 27 14:54:51 2020 +0200

    Centralize VFP storage error log

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 2924a9448..277a52e4d 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -169,7 +169,7 @@ vbf_beresp2obj(struct busyobj *bo)
 		if (vary != NULL)
 			VSB_destroy(&vary);
 		AZ(vary);
-		return (-1);
+		return (VFP_Error(bo->vfc, "Could not get storage"));
 	}
 
 	if (vary != NULL) {
@@ -617,7 +617,6 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo)
 	}
 
 	if (vbf_beresp2obj(bo)) {
-		(void)VFP_Error(bo->vfc, "Could not get storage");
 		bo->htc->doclose = SC_RX_BODY;
 		vbf_cleanup(bo);
 		return (F_STP_ERROR);
@@ -760,7 +759,6 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo)
 	AZ(stale_oc->flags & OC_F_FAILED);
 
 	if (vbf_beresp2obj(bo)) {
-		(void)VFP_Error(bo->vfc, "Could not get storage in vbf_stp_condfetch");
 		vbf_cleanup(bo);
 		wrk->stats->fetch_failed++;
 		return (F_STP_FAIL);
@@ -897,7 +895,6 @@ vbf_stp_error(struct worker *wrk, struct busyobj *bo)
 	assert(bo->vfc->req == bo->bereq);
 
 	if (vbf_beresp2obj(bo)) {
-		(void)VFP_Error(bo->vfc, "Could not get storage");
 		VSB_destroy(&synth_body);
 		return (F_STP_FAIL);
 	}

From dridi.boukelmoune at gmail.com  Wed May 27 13:39:08 2020
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Wed, 27 May 2020 13:39:08 +0000 (UTC)
Subject: [6.0] fc7749333 Revert "Formally expose VSB_tofile in libvarnishapi"
Message-ID: <20200527133908.E189264730@lists.varnish-cache.org>


commit fc7749333bdcfbe5badb09d0787b351762d37afc
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed May 27 15:34:45 2020 +0200

    Revert "Formally expose VSB_tofile in libvarnishapi"
    
    This reverts commit 795366627ffdb0515c8e897a5f3d15623109cc56.
    
    It shouldn't be exposed in libvarnishapi the way it is in this change
    because it creates two diverging symbol versions (for systems with
    linker support for symbol versioning) without changing the soname,
    which of course we want to avoid in the first place in a stable branch.
    
    If the requirement comes up we can revisit this topic and ensure both
    binary stability and compatility (upgrade path).
    
    Refs #3268

diff --git a/lib/libvarnishapi/libvarnishapi.map b/lib/libvarnishapi/libvarnishapi.map
index 30873af89..4cc3b5050 100644
--- a/lib/libvarnishapi/libvarnishapi.map
+++ b/lib/libvarnishapi/libvarnishapi.map
@@ -1,5 +1,5 @@
 /*-
- * Copyright (c) 2011-2020 Varnish Software AS
+ * Copyright (c) 2011-2017 Varnish Software AS
  * All rights reserved.
  *
  * Author: Tollef Fog Heen <tfheen at varnish-software.com>
@@ -140,11 +140,3 @@ LIBVARNISHAPI_2.0 {
     local:
 	*;
 };
-
-LIBVARNISHAPI_2.1 {
-    global:
-	# vsb.c
-	VSB_tofile;
-    local:
-	*;
-};
\ No newline at end of file

From phk at FreeBSD.org  Thu May 28 06:13:08 2020
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 28 May 2020 06:13:08 +0000 (UTC)
Subject: [master] f43679b69 More debugging
Message-ID: <20200528061308.C5173ADD3F@lists.varnish-cache.org>


commit f43679b69c9dad8258b7a3f5e5ac58b30fbee042
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu May 28 06:12:23 2020 +0000

    More debugging

diff --git a/bin/varnishtest/tests/u00000.vtc b/bin/varnishtest/tests/u00000.vtc
index d26626eb2..166cf3ead 100644
--- a/bin/varnishtest/tests/u00000.vtc
+++ b/bin/varnishtest/tests/u00000.vtc
@@ -4,8 +4,9 @@ shell "varnishd -b 127.0.0.1:80 -C 2> ${tmpdir}/_.c"
 
 shell {
 	varnishd -n ${tmpdir}/no_keep -C -b 127.0.0.1:80 2> no_keep.c
-	ls -l
+	ls -la
 	set -x
+	tail no_keep.c || true
 	test -s no_keep.c && ! test -d no_keep
 }
 

From phk at FreeBSD.org  Thu May 28 06:29:07 2020
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 28 May 2020 06:29:07 +0000 (UTC)
Subject: [master] 8eb42d2cc Even more debugging
Message-ID: <20200528062907.ACD1BAE42E@lists.varnish-cache.org>


commit 8eb42d2cc91179ac56d428d6f1b96fe25d8b3135
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu May 28 06:28:35 2020 +0000

    Even more debugging

diff --git a/bin/varnishtest/tests/u00000.vtc b/bin/varnishtest/tests/u00000.vtc
index 166cf3ead..563caca8a 100644
--- a/bin/varnishtest/tests/u00000.vtc
+++ b/bin/varnishtest/tests/u00000.vtc
@@ -4,7 +4,7 @@ shell "varnishd -b 127.0.0.1:80 -C 2> ${tmpdir}/_.c"
 
 shell {
 	varnishd -n ${tmpdir}/no_keep -C -b 127.0.0.1:80 2> no_keep.c
-	ls -la
+	ls -laR
 	set -x
 	tail no_keep.c || true
 	test -s no_keep.c && ! test -d no_keep

From gquintard at users.noreply.github.com  Fri May 29 04:49:06 2020
From: gquintard at users.noreply.github.com (guillaume quintard)
Date: Fri, 29 May 2020 04:49:06 +0000 (UTC)
Subject: [master] 39156f4ea [cci] Copy the .dsc file to the folder
Message-ID: <20200529044906.D6C03A6B14@lists.varnish-cache.org>


commit 39156f4eab7e7d2f754d56f6511515eb853ca6f1
Author: Martin Tzvetanov Grigorov <mgrigorov at apache.org>
Date:   Thu May 28 22:19:13 2020 +0300

    [cci] Copy the .dsc file to the folder
    
    Add  _amd64|_arm64 before the extension

diff --git a/.circleci/make-deb-packages.sh b/.circleci/make-deb-packages.sh
index f4939ec1e..3b3ba0847 100755
--- a/.circleci/make-deb-packages.sh
+++ b/.circleci/make-deb-packages.sh
@@ -49,4 +49,14 @@ dpkg-buildpackage -us -uc -j16
 
 echo "Prepare the packages for storage..."
 mkdir -p packages/$PARAM_DIST/$PARAM_RELEASE/
-mv ../*.deb ../*.dsc packages/$PARAM_DIST/$PARAM_RELEASE/
+mv ../*.deb packages/$PARAM_DIST/$PARAM_RELEASE/
+
+if [ "`uname -m`" = "x86_64" ]; then
+  ARCH="amd64"
+else
+  ARCH="arm64"
+fi
+
+DSC_FILE=$(ls ../*.dsc)
+DSC_FILE_WO_EXT=$(basename ${DSC_FILE%.*})
+mv $DSC_FILE packages/$PARAM_DIST/$PARAM_RELEASE/${DSC_FILE_WO_EXT}_${ARCH}.dsc

From nils.goroll at uplex.de  Fri May 29 12:54:06 2020
From: nils.goroll at uplex.de (Nils Goroll)
Date: Fri, 29 May 2020 12:54:06 +0000 (UTC)
Subject: [master] d2c1b12b2 jail_subproc_e and jail_master_e to use distinct
 values
Message-ID: <20200529125406.8A38BB1554@lists.varnish-cache.org>


commit d2c1b12b2b81cc16338698efb018c3633e7cadcc
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Fri May 29 13:37:42 2020 +0200

    jail_subproc_e and jail_master_e to use distinct values
    
    this is to allow a follow-up simplification in the solaris to use a
    single array for privileges.

diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index 0456d84b4..a2b3a3d56 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -102,21 +102,23 @@ void mgt_cli_init_cls(void);
 
 /* mgt_jail.c */
 
-enum jail_subproc_e {
-	JAIL_SUBPROC_VCC,
-	JAIL_SUBPROC_CC,
-	JAIL_SUBPROC_VCLLOAD,
-	JAIL_SUBPROC_WORKER,
-};
-
 enum jail_master_e {
-	JAIL_MASTER_LOW,
+	JAIL_MASTER_LOW = 0,
 	JAIL_MASTER_FILE,
 	JAIL_MASTER_STORAGE,
 	JAIL_MASTER_PRIVPORT,
 	JAIL_MASTER_KILL,
 };
 
+enum jail_subproc_e {
+	JAIL_SUBPROC_VCC = JAIL_MASTER_KILL + 1,
+	JAIL_SUBPROC_CC,
+	JAIL_SUBPROC_VCLLOAD,
+	JAIL_SUBPROC_WORKER,
+};
+
+#define JAIL_LIMIT (JAIL_SUBPROC_WORKER + 1)
+
 enum jail_fixfd_e {
 	JAIL_FIXFD_FILE,
 	JAIL_FIXFD_VSMMGT,

From nils.goroll at uplex.de  Fri May 29 16:23:06 2020
From: nils.goroll at uplex.de (Nils Goroll)
Date: Fri, 29 May 2020 16:23:06 +0000 (UTC)
Subject: [master] 1b04343a0 jail enum assertions
Message-ID: <20200529162306.6EFDC4E89@lists.varnish-cache.org>


commit 1b04343a0d1766d94af03a35bb53ca0181a728cb
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Fri May 29 17:41:56 2020 +0200

    jail enum assertions

diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index a2b3a3d56..4973abb4d 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -110,6 +110,11 @@ enum jail_master_e {
 	JAIL_MASTER_KILL,
 };
 
+#define ASSERT_JAIL_MASTER(x) do {		\
+	assert(x >= JAIL_MASTER_LOW);		\
+	assert(x <= JAIL_MASTER_KILL);		\
+	} while (0)
+
 enum jail_subproc_e {
 	JAIL_SUBPROC_VCC = JAIL_MASTER_KILL + 1,
 	JAIL_SUBPROC_CC,
@@ -117,6 +122,11 @@ enum jail_subproc_e {
 	JAIL_SUBPROC_WORKER,
 };
 
+#define ASSERT_JAIL_SUBPROC(x) do {		\
+	assert(x >= JAIL_SUBPROC_VCC);		\
+	assert(x <= JAIL_SUBPROC_WORKER);	\
+	} while (0)
+
 #define JAIL_LIMIT (JAIL_SUBPROC_WORKER + 1)
 
 enum jail_fixfd_e {
diff --git a/bin/varnishd/mgt/mgt_jail_unix.c b/bin/varnishd/mgt/mgt_jail_unix.c
index b34fc04e8..15aed013a 100644
--- a/bin/varnishd/mgt/mgt_jail_unix.c
+++ b/bin/varnishd/mgt/mgt_jail_unix.c
@@ -188,6 +188,7 @@ vju_init(char **args)
 static void v_matchproto_(jail_master_f)
 vju_master(enum jail_master_e jme)
 {
+	ASSERT_JAIL_MASTER(jme);
 	if (jme == JAIL_MASTER_LOW) {
 		AZ(setegid(vju_gid));
 		AZ(seteuid(vju_uid));
@@ -203,6 +204,7 @@ vju_subproc(enum jail_subproc_e jse)
 	int i;
 	gid_t gid_list[NGID];
 
+	ASSERT_JAIL_SUBPROC(jse);
 	AZ(seteuid(0));
 	if (vju_wrkuser != NULL &&
 	    (jse == JAIL_SUBPROC_VCLLOAD || jse == JAIL_SUBPROC_WORKER)) {

From dridi at varni.sh  Fri May 29 16:38:18 2020
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 29 May 2020 16:38:18 +0000
Subject: [master] 1b04343a0 jail enum assertions
In-Reply-To: <20200529162306.6EFDC4E89@lists.varnish-cache.org>
References: <20200529162306.6EFDC4E89@lists.varnish-cache.org>
Message-ID: <CABoVN9DUR6CE7UxJyyPK4eSCuqn2Um3Ywb6vFBbfyN3UphSxZw@mail.gmail.com>

On Fri, May 29, 2020 at 4:23 PM Nils Goroll <nils.goroll at uplex.de> wrote:
>
>
> commit 1b04343a0d1766d94af03a35bb53ca0181a728cb
> Author: Nils Goroll <nils.goroll at uplex.de>
> Date:   Fri May 29 17:41:56 2020 +0200
>
>     jail enum assertions
>
> diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
> index a2b3a3d56..4973abb4d 100644
> --- a/bin/varnishd/mgt/mgt.h
> +++ b/bin/varnishd/mgt/mgt.h
> @@ -110,6 +110,11 @@ enum jail_master_e {
>         JAIL_MASTER_KILL,
>  };
>
> +#define ASSERT_JAIL_MASTER(x) do {             \
> +       assert(x >= JAIL_MASTER_LOW);           \
> +       assert(x <= JAIL_MASTER_KILL);          \
> +       } while (0)
> +

This assertion breaks with targets where enums are unsigned. You
might as well revert the last two commits and turn the enums into
structs right off the bat.

My USD 0.02,
Dridi

From nils.goroll at uplex.de  Sat May 30 11:00:08 2020
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sat, 30 May 2020 11:00:08 +0000 (UTC)
Subject: [master] d13a51f1b avoid >= 0 comparison for enum which may be
 unsigned
Message-ID: <20200530110008.A8FD9A98F7@lists.varnish-cache.org>


commit d13a51f1bb1c16f12fbfdfaaf0a333609528257f
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Sat May 30 12:58:55 2020 +0200

    avoid >= 0 comparison for enum which may be unsigned
    
    depending on the compiler's mood

diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index 4973abb4d..294b10a7f 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -110,10 +110,7 @@ enum jail_master_e {
 	JAIL_MASTER_KILL,
 };
 
-#define ASSERT_JAIL_MASTER(x) do {		\
-	assert(x >= JAIL_MASTER_LOW);		\
-	assert(x <= JAIL_MASTER_KILL);		\
-	} while (0)
+#define ASSERT_JAIL_MASTER(x) assert(x <= JAIL_MASTER_KILL)
 
 enum jail_subproc_e {
 	JAIL_SUBPROC_VCC = JAIL_MASTER_KILL + 1,

From nils.goroll at uplex.de  Sat May 30 14:35:06 2020
From: nils.goroll at uplex.de (Nils Goroll)
Date: Sat, 30 May 2020 14:35:06 +0000 (UTC)
Subject: [master] c40060c3a polish the jail enums some more
Message-ID: <20200530143506.B3AA6AE203@lists.varnish-cache.org>


commit c40060c3ae9f4fb9ef5181218e9c9ab7d0b4ed6e
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Sat May 30 16:26:36 2020 +0200

    polish the jail enums some more
    
    We are also going to need JAIL_SUBPROC to determine the boundary between
    the MASTER and SUBPROC enum values.

diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h
index 294b10a7f..25169384d 100644
--- a/bin/varnishd/mgt/mgt.h
+++ b/bin/varnishd/mgt/mgt.h
@@ -108,24 +108,24 @@ enum jail_master_e {
 	JAIL_MASTER_STORAGE,
 	JAIL_MASTER_PRIVPORT,
 	JAIL_MASTER_KILL,
+#define JAIL_SUBPROC (JAIL_MASTER_KILL + 1)
 };
 
-#define ASSERT_JAIL_MASTER(x) assert(x <= JAIL_MASTER_KILL)
-
 enum jail_subproc_e {
-	JAIL_SUBPROC_VCC = JAIL_MASTER_KILL + 1,
+	JAIL_SUBPROC_VCC = JAIL_SUBPROC,
 	JAIL_SUBPROC_CC,
 	JAIL_SUBPROC_VCLLOAD,
 	JAIL_SUBPROC_WORKER,
+#define JAIL_LIMIT (JAIL_SUBPROC_WORKER + 1)
 };
 
+#define ASSERT_JAIL_MASTER(x) assert((x) < JAIL_SUBPROC)
+
 #define ASSERT_JAIL_SUBPROC(x) do {		\
-	assert(x >= JAIL_SUBPROC_VCC);		\
-	assert(x <= JAIL_SUBPROC_WORKER);	\
+		assert((x) >= JAIL_SUBPROC);	\
+		assert((x) < JAIL_LIMIT);	\
 	} while (0)
 
-#define JAIL_LIMIT (JAIL_SUBPROC_WORKER + 1)
-
 enum jail_fixfd_e {
 	JAIL_FIXFD_FILE,
 	JAIL_FIXFD_VSMMGT,