From phk at FreeBSD.org  Mon Sep  3 06:54:07 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon,  3 Sep 2018 06:54:07 +0000 (UTC)
Subject: [master] d6b3076bd Make the vmod->file_id a hash over the .vcc file.
Message-ID: <20180903065407.BBB4465E67@lists.varnish-cache.org>


commit d6b3076bd0e1bacbdfc5ab869696048ed0d7009b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 3 06:52:32 2018 +0000

    Make the vmod->file_id a hash over the .vcc file.
    
    I belive this makes our builds reproducible.
    
    Fixes:  #2436

diff --git a/include/vrt.h b/include/vrt.h
index 0d76b2be5..88dd5214c 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -59,7 +59,7 @@
  *	VRT_STRANDS_string() removed from vrt.h (never implemented)
  *	VRT_Healthy() changed prototype
  *	VRT_Vmod_Init signature changed
- * 	VRT_Vmod_Fini changed to VRT_Vmod_Unload
+ *	VRT_Vmod_Fini changed to VRT_Vmod_Unload
  * 7.0 (2018-03-15)
  *	lots of stuff moved from cache.h to cache_varnishd.h
  *	   (ie: from "$Abi vrt" to "$Abi strict")
diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py
index 66e99b236..4b25dc623 100755
--- a/lib/libvcc/vmodtool.py
+++ b/lib/libvcc/vmodtool.py
@@ -44,6 +44,7 @@ import unittest
 import random
 import copy
 import json
+import hashlib
 
 AMBOILERPLATE = '''
 # Boilerplate generated by vmodtool.py - changes will be overwritten
@@ -858,6 +859,7 @@ class vcc(object):
     def parse(self):
         global inputline
         a = "\n" + open(self.inputfile, "r").read()
+        self.file_id = hashlib.sha256(a).hexdigest()
         s = a.split("\n$")
         self.copyright = s.pop(0).strip()
         while s:
@@ -983,14 +985,7 @@ class vcc(object):
         fo.write('\t.proto =\tVmod_Proto,\n')
         fo.write('\t.json =\t\tVmod_Json,\n')
         fo.write('\t.abi =\t\tVMOD_ABI_Version,\n')
-        # NB: Sort of hackish:
-        # Fill file_id with random stuff, so we can tell if
-        # VCC and VRT_Vmod_Init() dlopens the same file
-        #
-        fo.write("\t.file_id =\t\"")
-        for i in range(32):
-            fo.write("%c" % random.randint(0x40, 0x5a))
-        fo.write("\",\n")
+        fo.write("\t.file_id =\t\"%s\",\n" % self.file_id)
         fo.write("};\n")
 
     def cfile(self):

From noreply at github.com  Mon Sep  3 10:05:15 2018
From: noreply at github.com (GitHub)
Date: Mon,  3 Sep 2018 10:05:15 +0000 (UTC)
Subject: [master] ff4c40335 Merge pull request #2759 from bmwiedemann/python3
Message-ID: <20180903100516.0198593D01@lists.varnish-cache.org>


commit ff4c40335d98b5d890d8b21e00313274afb2fef4
Merge: d6b3076bd ddaa6e5e7
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 3 10:04:48 2018 +0000

    Merge pull request #2759 from bmwiedemann/python3
    
    Fix vmodtool with python3

From bwiedemann at suse.de  Mon Sep  3 10:05:15 2018
From: bwiedemann at suse.de (Bernhard M. Wiedemann)
Date: Mon,  3 Sep 2018 10:05:15 +0000 (UTC)
Subject: [master] ddaa6e5e7 Fix vmodtool with python3
Message-ID: <20180903100515.E91F093CF9@lists.varnish-cache.org>


commit ddaa6e5e73bfe7198d344fb0d0c4f6510151c4cc
Author: Bernhard M. Wiedemann <bwiedemann at suse.de>
Date:   Mon Sep 3 11:58:14 2018 +0200

    Fix vmodtool with python3
    
    Without this patch, it did
    TypeError: Unicode-objects must be encoded before hashing
    
    Fixes: #2436

diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py
index 4b25dc623..3032f380f 100755
--- a/lib/libvcc/vmodtool.py
+++ b/lib/libvcc/vmodtool.py
@@ -859,7 +859,7 @@ class vcc(object):
     def parse(self):
         global inputline
         a = "\n" + open(self.inputfile, "r").read()
-        self.file_id = hashlib.sha256(a).hexdigest()
+        self.file_id = hashlib.sha256(a.encode('utf-8')).hexdigest()
         s = a.split("\n$")
         self.copyright = s.pop(0).strip()
         while s:

From phk at FreeBSD.org  Mon Sep  3 11:33:14 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon,  3 Sep 2018 11:33:14 +0000 (UTC)
Subject: [master] caa1334c6 Improve long desc a little bit.
Message-ID: <20180903113315.3443995D38@lists.varnish-cache.org>


commit caa1334c68c8d7523b2c9703e9b916bcdb2f9400
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 3 11:32:23 2018 +0000

    Improve long desc a little bit.

diff --git a/bin/varnishd/VSC_main.vsc b/bin/varnishd/VSC_main.vsc
index 4749eb5c2..56ad787b2 100644
--- a/bin/varnishd/VSC_main.vsc
+++ b/bin/varnishd/VSC_main.vsc
@@ -11,7 +11,7 @@
 	:level:	debug
 	:oneliner:	stat summ operations
 
-	Number of times per-thread statistics were added to the
+	Number of times per-thread statistics were summed into the
 	global counters.
 
 .. varnish_vsc:: uptime

From nils.goroll at uplex.de  Mon Sep  3 13:58:11 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon,  3 Sep 2018 13:58:11 +0000 (UTC)
Subject: [master] 0c87f59a5 vtc syntax cleanup
Message-ID: <20180903135811.81DC998D69@lists.varnish-cache.org>


commit 0c87f59a5918b4c299e8f2c9babc6de9f91e34aa
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Sep 3 15:04:35 2018 +0200

    vtc syntax cleanup
    
    return(pass) from vcl_recv does not take a duration argument

diff --git a/bin/varnishtest/tests/c00010.vtc b/bin/varnishtest/tests/c00010.vtc
index a128b73ca..5e8b087da 100644
--- a/bin/varnishtest/tests/c00010.vtc
+++ b/bin/varnishtest/tests/c00010.vtc
@@ -17,7 +17,7 @@ varnish v1 -vcl+backend {
 		if (req.url == "/foo") {
 			return(hash);
 		} else {
-			return(pass(10m));
+			return(pass);
 		}
 	}
 	sub vcl_hit {

From nils.goroll at uplex.de  Mon Sep  3 13:58:11 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon,  3 Sep 2018 13:58:11 +0000 (UTC)
Subject: [master] 9367d8003 fix logexpect syntax in some vtcs
Message-ID: <20180903135811.9552A98D6C@lists.varnish-cache.org>


commit 9367d8003c26955d6fefbaba290a4de141802290
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Sep 3 15:55:38 2018 +0200

    fix logexpect syntax in some vtcs
    
    The syntax is
    
            expect <skip> <vxid> <tag> <regex>
    
    Also remove some bogus .*$ from regexen (waving to @slimhazard)

diff --git a/bin/varnishtest/tests/c00011.vtc b/bin/varnishtest/tests/c00011.vtc
index 24622675f..907355f2f 100644
--- a/bin/varnishtest/tests/c00011.vtc
+++ b/bin/varnishtest/tests/c00011.vtc
@@ -23,7 +23,7 @@ varnish v1 -vcl+backend {
 } -start
 
 logexpect l1 -v v1 -g vxid {
-	expect 1003 *	HitMiss "^1002 119.*$"
+	expect * 1003	HitMiss "^1002 119"
 } -start
 
 client c1 {
diff --git a/bin/varnishtest/tests/c00081.vtc b/bin/varnishtest/tests/c00081.vtc
index 5b7c55166..b08fa3250 100644
--- a/bin/varnishtest/tests/c00081.vtc
+++ b/bin/varnishtest/tests/c00081.vtc
@@ -30,7 +30,7 @@ varnish v1 -vcl+backend {
 } -start
 
 logexpect l1 -v v1 -g vxid {
-	expect 1003 *	HitPass "^1002 1.*$"
+	expect * 1003	HitPass "^1002 1"
 } -start
 
 client c1 {
diff --git a/bin/varnishtest/tests/r01858.vtc b/bin/varnishtest/tests/r01858.vtc
index fe1cbe1de..1fb787f76 100644
--- a/bin/varnishtest/tests/r01858.vtc
+++ b/bin/varnishtest/tests/r01858.vtc
@@ -24,7 +24,7 @@ varnish v1 -vcl+backend {
 
 # Tests logging hit-for-miss on an expired object
 logexpect l1 -v v1 -g vxid {
-	expect 1003 *	HitMiss "^1002 -.*$"
+	expect * 1003	HitMiss "^1002 -"
 } -start
 
 client c1 {

From nils.goroll at uplex.de  Mon Sep  3 16:06:11 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon,  3 Sep 2018 16:06:11 +0000 (UTC)
Subject: [master] caa7a4e07 fix comment
Message-ID: <20180903160612.02DE29CC4B@lists.varnish-cache.org>


commit caa7a4e07ef6c820d7e539312b1a09812066304b
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Sep 3 18:05:09 2018 +0200

    fix comment

diff --git a/include/tbl/boc_state.h b/include/tbl/boc_state.h
index 5297d8af5..a5c4a61ff 100644
--- a/include/tbl/boc_state.h
+++ b/include/tbl/boc_state.h
@@ -29,7 +29,7 @@
 /*lint -save -e525 -e539 */
 
 BOC_STATE(INVALID,	invalid)	/* don't touch (yet) */
-BOC_STATE(REQ_DONE,	req_done)	/* beresp.* can be examined */
+BOC_STATE(REQ_DONE,	req_done)	/* bereq.* can be examined */
 BOC_STATE(PREP_STREAM,	prep_stream)	/* Prepare for streaming */
 BOC_STATE(STREAM,	stream)		/* beresp.* can be examined */
 BOC_STATE(FINISHED,	finished)	/* object is complete */

From nils.goroll at uplex.de  Mon Sep  3 16:16:09 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon,  3 Sep 2018 16:16:09 +0000 (UTC)
Subject: [master] 9804794a0 side effect of staring at code not worth a ticket
Message-ID: <20180903161609.B88859D09F@lists.varnish-cache.org>


commit 9804794a0e876800e4e3ca10e965b568bd37c38e
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Sep 3 18:11:42 2018 +0200

    side effect of staring at code not worth a ticket

diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index c3b652bbe..1b63d906d 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -407,6 +407,8 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp,
 			continue;
 
 		if (oc->boc != NULL && oc->boc->state < BOS_STREAM) {
+			// XXX does it make sense to have the CHECK_OBJ
+			//     if we access boc->state before it?
 			CHECK_OBJ_ORNULL(oc->boc, BOC_MAGIC);
 
 			if (req->hash_ignore_busy)

From dridi at varni.sh  Mon Sep  3 16:20:08 2018
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 3 Sep 2018 18:20:08 +0200
Subject: [master] 9804794a0 side effect of staring at code not worth a
 ticket
In-Reply-To: <20180903161609.B88859D09F@lists.varnish-cache.org>
References: <20180903161609.B88859D09F@lists.varnish-cache.org>
Message-ID: <CABoVN9Bv_yh=wdUHU3v=9YmwMCVTXbCBYoCtD2iS=kWD1nYg+g@mail.gmail.com>

On Mon, Sep 3, 2018 at 6:16 PM, Nils Goroll <nils.goroll at uplex.de> wrote:
>
> commit 9804794a0e876800e4e3ca10e965b568bd37c38e
> Author: Nils Goroll <nils.goroll at uplex.de>
> Date:   Mon Sep 3 18:11:42 2018 +0200
>
>     side effect of staring at code not worth a ticket
>
> diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
> index c3b652bbe..1b63d906d 100644
> --- a/bin/varnishd/cache/cache_hash.c
> +++ b/bin/varnishd/cache/cache_hash.c
> @@ -407,6 +407,8 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp,
>                         continue;
>
>                 if (oc->boc != NULL && oc->boc->state < BOS_STREAM) {
> +                       // XXX does it make sense to have the CHECK_OBJ
> +                       //     if we access boc->state before it?
>                         CHECK_OBJ_ORNULL(oc->boc, BOC_MAGIC);

Looks like a simple mistake, it's an ORNULL check so it belongs before
the if statement.

>
>                         if (req->hash_ignore_busy)
> _______________________________________________
> varnish-commit mailing list
> varnish-commit at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-commit

From nils.goroll at uplex.de  Mon Sep  3 17:50:45 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon, 3 Sep 2018 19:50:45 +0200
Subject: [master] 9804794a0 side effect of staring at code not worth a
 ticket
In-Reply-To: <CABoVN9Bv_yh=wdUHU3v=9YmwMCVTXbCBYoCtD2iS=kWD1nYg+g@mail.gmail.com>
References: <20180903161609.B88859D09F@lists.varnish-cache.org>
 <CABoVN9Bv_yh=wdUHU3v=9YmwMCVTXbCBYoCtD2iS=kWD1nYg+g@mail.gmail.com>
Message-ID: <b62161e0-8399-a48c-3cb5-46d48e4f0264@uplex.de>

On 03/09/18 18:20, Dridi Boukelmoune wrote:
> Looks like a simple mistake, it's an ORNULL check so it belongs before
> the if statement.

my simple mistake was to read NOTNULL where it clearly sais ORNULL.

Thank you

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-commit/attachments/20180903/62962ee8/attachment.bin>

From phk at FreeBSD.org  Tue Sep  4 06:28:10 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue,  4 Sep 2018 06:28:10 +0000 (UTC)
Subject: [master] 5ab14bc3a Return the old hit-for-miss object so we can
 expire it.
Message-ID: <20180904062811.19F224871@lists.varnish-cache.org>


commit 5ab14bc3a1c3a9bc0c768b52ec770b85f0af2bce
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Sep 4 06:26:24 2018 +0000

    Return the old hit-for-miss object so we can expire it.
    
    Partial fix for #2654 and #2754

diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index 1b63d906d..a87462646 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -447,23 +447,25 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp,
 				VSLb(req->vsl, SLT_HitPass, "%u %.6f",
 				    ObjGetXID(wrk, oc), EXP_Dttl(req, oc));
 				oc = NULL;
+				retval = HSH_MISS;
 			} else if (oc->flags & OC_F_PASS) {
 				wrk->stats->cache_hitmiss++;
 				VSLb(req->vsl, SLT_HitMiss, "%u %.6f",
 				    ObjGetXID(wrk, oc), EXP_Dttl(req, oc));
-				oc = NULL;
 				*bocp = hsh_insert_busyobj(wrk, oh);
+				oc->refcnt++;
+				retval = HSH_MISS;
 			} else {
 				oc->refcnt++;
 				if (oc->hits < LONG_MAX)
 					oc->hits++;
+				retval = HSH_HIT;
 			}
 			Lck_Unlock(&oh->mtx);
-			if (oc == NULL)
-				return (HSH_MISS);
-			assert(HSH_DerefObjHead(wrk, &oh));
 			*ocp = oc;
-			return (HSH_HIT);
+			if (retval == HSH_HIT)
+				assert(HSH_DerefObjHead(wrk, &oh));
+			return (retval);
 		}
 		if (EXP_Ttl(NULL, oc) < req->t_req && /* ignore req.ttl */
 		    oc->t_origin > exp_t_origin) {

From dridi.boukelmoune at gmail.com  Tue Sep  4 06:48:07 2018
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue,  4 Sep 2018 06:48:07 +0000 (UTC)
Subject: [master] 621c408ae Move boc check before boc->state is dereferenced
Message-ID: <20180904064808.12FBB4EEC@lists.varnish-cache.org>


commit 621c408ae36ceceae0314a3df8f8ec3c51297bff
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Sep 4 08:44:52 2018 +0200

    Move boc check before boc->state is dereferenced
    
    Refs 9804794a0e876800e4e3ca10e965b568bd37c38e

diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index a87462646..182a7a97d 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -406,11 +406,8 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp,
 		if (oc->flags & OC_F_FAILED)
 			continue;
 
+		CHECK_OBJ_ORNULL(oc->boc, BOC_MAGIC);
 		if (oc->boc != NULL && oc->boc->state < BOS_STREAM) {
-			// XXX does it make sense to have the CHECK_OBJ
-			//     if we access boc->state before it?
-			CHECK_OBJ_ORNULL(oc->boc, BOC_MAGIC);
-
 			if (req->hash_ignore_busy)
 				continue;
 

From phk at FreeBSD.org  Tue Sep  4 07:40:10 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue,  4 Sep 2018 07:40:10 +0000 (UTC)
Subject: [master] d22e7de26 Test more of varnishadms error handling
Message-ID: <20180904074010.DB6305EDA@lists.varnish-cache.org>


commit d22e7de2611bb53bd9d4dc6e318da20c70686862
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Sep 4 07:38:27 2018 +0000

    Test more of varnishadms error handling

diff --git a/bin/varnishtest/tests/b00041.vtc b/bin/varnishtest/tests/b00041.vtc
index 2fd2690a4..ca66c4539 100644
--- a/bin/varnishtest/tests/b00041.vtc
+++ b/bin/varnishtest/tests/b00041.vtc
@@ -2,4 +2,94 @@ varnishtest "Test varnishadm and the Telnet CLI"
 
 varnish v1 -vcl {backend foo { .host = "127.0.0.1"; } } -start
 
+shell -err -expect {Usage: varnishadm} \
+	"varnishadm -7"
+
+shell -err -expect {Could not get hold of varnishd, is it running?} \
+	"varnishadm -n ${v1_name}/nonexistent"
+
+shell -err -expect {Connection failed} \
+	"varnishadm -t 4 -T ${bad_ip}:1 -S ${v1_name}/_.secret"
+
+server s1 {
+	send "FOO\n"
+} -start
+
+shell -err -expect {Rejected 400} \
+	{varnishadm -T ${s1_addr}:${s1_port} -S /etc/group}
+
+server s1 {
+	send "107 59      \n"
+	send "qbvnnftpkgubadqpzznkkazoxlyqbcbj\n\n"
+	send "Authentication required.\n"
+	send "\n"
+} -start
+
+shell -err -expect {Authentication required} \
+	{varnishadm -T ${s1_addr}:${s1_port}}
+
+server s1 {
+	send "107 59      \n"
+	send "qbvnnftpkgubadqpzznkkazoxlyqbcbj\n\n"
+	send "Authentication required.\n"
+	send "\n"
+} -start
+
+shell -err -expect {Cannot open } \
+	{varnishadm -T ${s1_addr}:${s1_port} -S ${v1_name}/_.nonexistent}
+
+server s1 {
+	send "107 59      \n"
+	send "qbvnnftpkgubadqpzznkkazoxlyqbcbj\n\n"
+	send "Authentication required.\n"
+	send "\n"
+
+	recv 70
+	send "599 0       \n"
+	send "\n"
+} -start
+
+shell -err -expect {Rejected 599} \
+	{varnishadm -T ${s1_addr}:${s1_port} -S ${v1_name}/_.secret}
+
+server s1 {
+	send "107 59      \n"
+	send "qbvnnftpkgubadqpzznkkazoxlyqbcbj\n\n"
+	send "Authentication required.\n"
+	send "\n"
+
+	recv 70
+	send "200 0       \n"
+	send "\n"
+
+	recv 5
+} -start
+
+shell -err -expect {No pong received from server} \
+	{varnishadm -T ${s1_addr}:${s1_port} -S ${v1_name}/_.secret}
+
+server s1 {
+	send "107 59      \n"
+	send "qbvnnftpkgubadqpzznkkazoxlyqbcbj\n\n"
+	send "Authentication required.\n"
+	send "\n"
+
+	recv 70
+	send "200 0       \n"
+	send "\n"
+
+	recv 5
+	send "200 8       \n"
+	send "PONG 12\n"
+	send "\n"
+
+	recv 5
+	send "200 7       \n"
+	send "Tested\n"
+	send "\n"
+} -start
+
+shell -expect {Tested} \
+	{varnishadm -T ${s1_addr}:${s1_port} -S ${v1_name}/_.secret test}
+
 shell "varnishadm -n ${v1_name} help > /dev/null"
diff --git a/bin/varnishtest/tests/c00081.vtc b/bin/varnishtest/tests/c00081.vtc
index b08fa3250..419036cfe 100644
--- a/bin/varnishtest/tests/c00081.vtc
+++ b/bin/varnishtest/tests/c00081.vtc
@@ -4,7 +4,11 @@ server s1 {
 	rxreq
 	txresp -hdr "foo: 1"
 	rxreq
-	txresp -hdr "foo: 2"
+	txresp -hdr "foo: 2a"
+	rxreq
+	txresp -hdr "foo: 2b"
+	rxreq
+	txresp -hdr "foo: 2c"
 	rxreq
 	txresp -hdr "foo: 3"
 } -start
@@ -29,8 +33,31 @@ varnish v1 -vcl+backend {
 
 } -start
 
+varnish v1 -cliok "param.set debug +refcnt"
+
 logexpect l1 -v v1 -g vxid {
-	expect * 1003	HitPass "^1002 1"
+	expect * 1001	Debug		"^oh.*refcnt 1$"
+	expect 0 =	VCL_call	"^MISS"
+} -start
+
+logexpect l2 -v v1 -g vxid {
+	expect * 1003	Debug		"^oh.*refcnt 2$"
+	expect 0 =	HitPass	"^1002 1"
+} -start
+
+logexpect l3 -v v1 -g vxid {
+	expect * 1005	Debug		"^oh.*refcnt 2$"
+	expect 0 =	HitPass	"^1002 1"
+} -start
+
+logexpect l4 -v v1 -g vxid {
+	expect * 1007	Debug		"^oh.*refcnt 2$"
+	expect 0 =	HitPass	"^1002 1"
+} -start
+
+logexpect l5 -v v1 -g vxid {
+	expect * 1009	Debug		"^oh.*refcnt 1$"
+	expect 0 =	VCL_call	"^MISS"
 } -start
 
 client c1 {
@@ -38,6 +65,12 @@ client c1 {
 	rxresp
 	expect resp.http.miss == True
 
+	txreq
+	rxresp
+	expect resp.http.pass == True
+	txreq
+	rxresp
+	expect resp.http.pass == True
 	txreq
 	rxresp
 	expect resp.http.pass == True
@@ -50,7 +83,11 @@ client c1 {
 } -run
 
 logexpect l1 -wait
+logexpect l2 -wait
+logexpect l3 -wait
+logexpect l4 -wait
+logexpect l5 -wait
 
-varnish v1 -expect MAIN.cache_hitpass == 1
+varnish v1 -expect MAIN.cache_hitpass == 3
 varnish v1 -expect MAIN.cache_miss == 2
 varnish v1 -expect MAIN.cache_hitmiss == 0

From phk at FreeBSD.org  Tue Sep  4 08:06:10 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue,  4 Sep 2018 08:06:10 +0000 (UTC)
Subject: [master] ae34324de Document -need-bytes and implement an optional '+'
 prefix to require new bytes to be received.
Message-ID: <20180904080610.43B2A671F@lists.varnish-cache.org>


commit ae34324de1f739b303ac5be0c18a0c36d6bc9574
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Sep 4 08:01:59 2018 +0000

    Document -need-bytes and implement an optional '+' prefix to
    require new bytes to be received.

diff --git a/bin/varnishtest/vtc_process.c b/bin/varnishtest/vtc_process.c
index 95f6c0517..5f5cd2a86 100644
--- a/bin/varnishtest/vtc_process.c
+++ b/bin/varnishtest/vtc_process.c
@@ -894,6 +894,10 @@ process_close(struct process *p)
  * \-writehex HEXSTRING
  *	Same as -write but interpreted as hexadecimal bytes.
  *
+ * \-need-bytes [+]NUMBER
+ *	Wait until at least NUMBER bytes have been received in total.
+ *	If '+' is prefixed, NUMBER new bytes must be received.
+ *
  * \-expect-text LIN COL PAT
  *	Wait for PAT to appear at LIN,COL on the virtual screen.
  *	Lines and columns are numbered 1...N
@@ -998,6 +1002,8 @@ cmd_process(CMD_ARGS)
 		}
 		if (!strcmp(*av, "-need-bytes")) {
 			u = strtoumax(av[1], NULL, 0);
+			if (av[1][0] == '+')
+				u += p->stdout_bytes;
 			av++;
 			do {
 				AZ(pthread_mutex_lock(&p->mtx));

From phk at FreeBSD.org  Tue Sep  4 08:06:10 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue,  4 Sep 2018 08:06:10 +0000 (UTC)
Subject: [master] 4eac99e8a Oops, accidentally commited test-patch from #2760
Message-ID: <20180904080610.565806721@lists.varnish-cache.org>


commit 4eac99e8a6c116b0f2ae4a8f4f897ec0542c3e56
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Sep 4 08:04:11 2018 +0000

    Oops, accidentally commited test-patch from #2760

diff --git a/bin/varnishtest/tests/c00081.vtc b/bin/varnishtest/tests/c00081.vtc
index 419036cfe..b08fa3250 100644
--- a/bin/varnishtest/tests/c00081.vtc
+++ b/bin/varnishtest/tests/c00081.vtc
@@ -4,11 +4,7 @@ server s1 {
 	rxreq
 	txresp -hdr "foo: 1"
 	rxreq
-	txresp -hdr "foo: 2a"
-	rxreq
-	txresp -hdr "foo: 2b"
-	rxreq
-	txresp -hdr "foo: 2c"
+	txresp -hdr "foo: 2"
 	rxreq
 	txresp -hdr "foo: 3"
 } -start
@@ -33,31 +29,8 @@ varnish v1 -vcl+backend {
 
 } -start
 
-varnish v1 -cliok "param.set debug +refcnt"
-
 logexpect l1 -v v1 -g vxid {
-	expect * 1001	Debug		"^oh.*refcnt 1$"
-	expect 0 =	VCL_call	"^MISS"
-} -start
-
-logexpect l2 -v v1 -g vxid {
-	expect * 1003	Debug		"^oh.*refcnt 2$"
-	expect 0 =	HitPass	"^1002 1"
-} -start
-
-logexpect l3 -v v1 -g vxid {
-	expect * 1005	Debug		"^oh.*refcnt 2$"
-	expect 0 =	HitPass	"^1002 1"
-} -start
-
-logexpect l4 -v v1 -g vxid {
-	expect * 1007	Debug		"^oh.*refcnt 2$"
-	expect 0 =	HitPass	"^1002 1"
-} -start
-
-logexpect l5 -v v1 -g vxid {
-	expect * 1009	Debug		"^oh.*refcnt 1$"
-	expect 0 =	VCL_call	"^MISS"
+	expect * 1003	HitPass "^1002 1"
 } -start
 
 client c1 {
@@ -65,12 +38,6 @@ client c1 {
 	rxresp
 	expect resp.http.miss == True
 
-	txreq
-	rxresp
-	expect resp.http.pass == True
-	txreq
-	rxresp
-	expect resp.http.pass == True
 	txreq
 	rxresp
 	expect resp.http.pass == True
@@ -83,11 +50,7 @@ client c1 {
 } -run
 
 logexpect l1 -wait
-logexpect l2 -wait
-logexpect l3 -wait
-logexpect l4 -wait
-logexpect l5 -wait
 
-varnish v1 -expect MAIN.cache_hitpass == 3
+varnish v1 -expect MAIN.cache_hitpass == 1
 varnish v1 -expect MAIN.cache_miss == 2
 varnish v1 -expect MAIN.cache_hitmiss == 0

From phk at FreeBSD.org  Tue Sep  4 08:06:10 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue,  4 Sep 2018 08:06:10 +0000 (UTC)
Subject: [master] 6bb9bcec1 Test more of varnishtop
Message-ID: <20180904080610.7CC7A6724@lists.varnish-cache.org>


commit 6bb9bcec1ad7d1d96df0e4eae0ea77384703b621
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Sep 4 08:05:30 2018 +0000

    Test more of varnishtop

diff --git a/bin/varnishtest/tests/u00004.vtc b/bin/varnishtest/tests/u00004.vtc
index 5dfaa3310..9d2be8ed4 100644
--- a/bin/varnishtest/tests/u00004.vtc
+++ b/bin/varnishtest/tests/u00004.vtc
@@ -31,7 +31,12 @@ shell -match "1\\.00 RespHeader Date\\n" {
 
 shell -match "Usage: .*varnishtop <options>" \
 	"varnishtop -h"
+
 shell -expect "Copyright (c) 2006 Verdens Gang AS" \
 	"varnishtop -V"
+
 shell -err -match "Usage: .*varnishtop <options>" \
 	"varnishtop extra"
+
+shell -err -match "is not a number" \
+	"varnishtop -p ABC"
diff --git a/bin/varnishtest/tests/u00010.vtc b/bin/varnishtest/tests/u00010.vtc
index 5eee23ab3..dd37a15d0 100644
--- a/bin/varnishtest/tests/u00010.vtc
+++ b/bin/varnishtest/tests/u00010.vtc
@@ -9,8 +9,6 @@ varnish v1 -vcl+backend {} -start
 
 process p1 -dump {varnishtop -n ${v1_name}} -start
 
-process p1 -winsz 30 80
-
 process p1 -expect-text 1 1 {list length}
 
 delay 1
@@ -20,5 +18,20 @@ client c1 {
 	rxresp
 } -run
 
-process p1 -expect-text 0 0 {Fetch_Body}
+varnish v1 -vsl_catchup
+
+process p1 -expect-text 1 1 {list length 64}
+
+process p1 -writehex 0c
+
+process p1 -need-bytes +1
+
+process p1 -writehex 0b
+
+process p1 -need-bytes +1
+
+process p1 -winsz 30 80
+
+process p1 -need-bytes +1
+
 process p1 -screen_dump -write {q} -wait

From phk at FreeBSD.org  Tue Sep  4 08:52:13 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue,  4 Sep 2018 08:52:13 +0000 (UTC)
Subject: [master] 5fb624078 Test strtoul() result comprehensively.
Message-ID: <20180904085213.EF8B477BC@lists.varnish-cache.org>


commit 5fb62407820256dadedcf4f7ff3b6771cce6e517
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Sep 4 08:49:16 2018 +0000

    Test strtoul() result comprehensively.

diff --git a/bin/varnishtop/varnishtop.c b/bin/varnishtop/varnishtop.c
index 942e25b97..8f02e6022 100644
--- a/bin/varnishtop/varnishtop.c
+++ b/bin/varnishtop/varnishtop.c
@@ -342,6 +342,7 @@ main(int argc, char **argv)
 {
 	int o, once = 0;
 	pthread_t thr;
+	char *e = NULL;
 
 	vut = VUT_InitProg(argc, argv, &vopt_spec);
 	AN(vut);
@@ -360,8 +361,9 @@ main(int argc, char **argv)
 			usage(0);
 		case 'p':
 			errno = 0;
-			period = strtol(optarg, NULL, 0);
-			if (errno != 0)  {
+			e = NULL;
+			period = strtoul(optarg, &e, 0);
+			if (errno != 0 || e == NULL || *e != '\0') {
 				fprintf(stderr,
 				    "Syntax error, %s is not a number", optarg);
 				exit(1);

From phk at FreeBSD.org  Tue Sep  4 08:52:13 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue,  4 Sep 2018 08:52:13 +0000 (UTC)
Subject: [master] 8ec48cf55 Tweak the -p test slightly
Message-ID: <20180904085214.0407C77CD@lists.varnish-cache.org>


commit 8ec48cf552b7b9e0d391d99048dc86f5b3363659
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Sep 4 08:50:37 2018 +0000

    Tweak the -p test slightly

diff --git a/bin/varnishtest/tests/u00004.vtc b/bin/varnishtest/tests/u00004.vtc
index 9d2be8ed4..b6015a061 100644
--- a/bin/varnishtest/tests/u00004.vtc
+++ b/bin/varnishtest/tests/u00004.vtc
@@ -39,4 +39,4 @@ shell -err -match "Usage: .*varnishtop <options>" \
 	"varnishtop extra"
 
 shell -err -match "is not a number" \
-	"varnishtop -p ABC"
+	"varnishtop -p 12ABC"

From phk at FreeBSD.org  Tue Sep  4 10:18:10 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue,  4 Sep 2018 10:18:10 +0000 (UTC)
Subject: [master] 60a6aba69 Deref the objhead if we don't return a busy object.
Message-ID: <20180904101810.950DB60472@lists.varnish-cache.org>


commit 60a6aba69ac02f2ba78a8df0a7df98d77294991f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Sep 4 10:15:52 2018 +0000

    Deref the objhead if we don't return a busy object.
    
    Most of the heavy lifting by:   slink
    
    Fixes: #2760, #2754, #2654

diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index 182a7a97d..bb32d4fc2 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -460,7 +460,7 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp,
 			}
 			Lck_Unlock(&oh->mtx);
 			*ocp = oc;
-			if (retval == HSH_HIT)
+			if (*bocp == NULL)
 				assert(HSH_DerefObjHead(wrk, &oh));
 			return (retval);
 		}

From phk at FreeBSD.org  Tue Sep  4 15:20:12 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue,  4 Sep 2018 15:20:12 +0000 (UTC)
Subject: [master] 61bb5d90c Polish
Message-ID: <20180904152012.647436E0A1@lists.varnish-cache.org>


commit 61bb5d90ced3b6d0bf19c56171df6a4586c7497b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Sep 4 15:19:29 2018 +0000

    Polish

diff --git a/bin/varnishtop/varnishtop.c b/bin/varnishtop/varnishtop.c
index 8f02e6022..a4a77e7c9 100644
--- a/bin/varnishtop/varnishtop.c
+++ b/bin/varnishtop/varnishtop.c
@@ -301,8 +301,8 @@ do_curses(void *arg)
 		case 'Q':
 		case 'q':
 			AZ(raise(SIGINT));
-			AC(endwin());
-			return (NULL);
+			quit = 1;
+			break;
 		default:
 			AC(beep());
 			break;
@@ -388,18 +388,14 @@ main(int argc, char **argv)
 	vut->dispatch_priv = NULL;
 	vut->sighup_f = sighup;
 	if (once) {
-		VUT_Main(vut);
+		(void)VUT_Main(vut);
 		dump();
 	} else {
-		if (pthread_create(&thr, NULL, do_curses, NULL) != 0) {
-			fprintf(stderr, "pthread_create(): %s\n",
-			    strerror(errno));
-			exit(1);
-		}
-		VUT_Main(vut);
+		AZ(pthread_create(&thr, NULL, do_curses, NULL));
+		(void)VUT_Main(vut);
 		end_of_file = 1;
 		AZ(pthread_join(thr, NULL));
 	}
 	VUT_Fini(&vut);
-	exit(0);
+	return (0);
 }

From nils.goroll at uplex.de  Tue Sep  4 18:19:07 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Tue,  4 Sep 2018 18:19:07 +0000 (UTC)
Subject: [master] a19786b38 cleanup
Message-ID: <20180904181907.980C093921@lists.varnish-cache.org>


commit a19786b3881b763be8834d958591e27af19bf1ee
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Tue Sep 4 19:55:34 2018 +0200

    cleanup

diff --git a/lib/libvmod_directors/vmod_shard.c b/lib/libvmod_directors/vmod_shard.c
index 1d366195c..3f4924bc3 100644
--- a/lib/libvmod_directors/vmod_shard.c
+++ b/lib/libvmod_directors/vmod_shard.c
@@ -840,11 +840,6 @@ shard_param_task(VRT_CTX, const void *id,
 	if (task->priv) {
 		CAST_OBJ_NOTNULL(p, task->priv, VMOD_SHARD_SHARD_PARAM_MAGIC);
 		assert(p->scope == SCOPE_TASK);
-		/* XXX
-		VSL(SLT_Debug, 0,
-		    "shard_param_task(id %p, pa %p) = %p (found, ws=%p)",
-		    id, pa, p, ctx->ws);
-		*/
 		return (p);
 	}
 
@@ -863,11 +858,6 @@ shard_param_task(VRT_CTX, const void *id,
 	else
 		p->defaults = shard_param_task(ctx, pa, pa);
 
-	/* XXX
-	VSL(SLT_Debug, 0,
-	    "shard_param_task(id %p, pa %p) = %p (new, defaults = %p, ws=%p)",
-	    id, pa, p, p->defaults, ctx->ws);
-	*/
 	return (p);
 }
 

From phk at FreeBSD.org  Tue Sep  4 18:41:08 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue,  4 Sep 2018 18:41:08 +0000 (UTC)
Subject: [master] 635e95a24 Test a few lines more.
Message-ID: <20180904184108.6BF1294106@lists.varnish-cache.org>


commit 635e95a2449261a34144c401d7380e6001a1d27b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Sep 4 18:40:06 2018 +0000

    Test a few lines more.

diff --git a/bin/varnishtest/tests/u00004.vtc b/bin/varnishtest/tests/u00004.vtc
index b6015a061..5d95d7611 100644
--- a/bin/varnishtest/tests/u00004.vtc
+++ b/bin/varnishtest/tests/u00004.vtc
@@ -32,6 +32,9 @@ shell -match "1\\.00 RespHeader Date\\n" {
 shell -match "Usage: .*varnishtop <options>" \
 	"varnishtop -h"
 
+shell -err -match "illegal option -- K" \
+	"varnishtop -K"
+
 shell -expect "Copyright (c) 2006 Verdens Gang AS" \
 	"varnishtop -V"
 
@@ -39,4 +42,4 @@ shell -err -match "Usage: .*varnishtop <options>" \
 	"varnishtop extra"
 
 shell -err -match "is not a number" \
-	"varnishtop -p 12ABC"
+	"varnishtop -p 60 -p 12ABC"

From phk at FreeBSD.org  Tue Sep  4 19:38:07 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue,  4 Sep 2018 19:38:07 +0000 (UTC)
Subject: [master] 5a57d134d Rely only on error messages we format ourselves.
Message-ID: <20180904193807.F2F07952EA@lists.varnish-cache.org>


commit 5a57d134d497af9f94ddf984542adcb08389fe78
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Sep 4 19:36:38 2018 +0000

    Rely only on error messages we format ourselves.

diff --git a/bin/varnishtest/tests/u00004.vtc b/bin/varnishtest/tests/u00004.vtc
index 5d95d7611..a861ae57a 100644
--- a/bin/varnishtest/tests/u00004.vtc
+++ b/bin/varnishtest/tests/u00004.vtc
@@ -32,7 +32,7 @@ shell -match "1\\.00 RespHeader Date\\n" {
 shell -match "Usage: .*varnishtop <options>" \
 	"varnishtop -h"
 
-shell -err -match "illegal option -- K" \
+shell -err -match "Usage: .*varnishtop <options>" \
 	"varnishtop -K"
 
 shell -expect "Copyright (c) 2006 Verdens Gang AS" \

From phk at FreeBSD.org  Wed Sep  5 06:43:09 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed,  5 Sep 2018 06:43:09 +0000 (UTC)
Subject: [master] f8461d09f Sanitize H2_Send_Frame error handling.
Message-ID: <20180905064309.84CE5A4A61@lists.varnish-cache.org>


commit f8461d09f52735279e05f4ac67b416fd9832bae6
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Sep 5 06:40:01 2018 +0000

    Sanitize H2_Send_Frame error handling.
    
    The only way H2_Send_Frame can fail is if the TCP connection is
    dead, so reach out and set the session errored and be done with it,
    rather than pass an inconvenient error status down through the callers.
    
    While here use writev(2) instead of two write(2) calls.

diff --git a/bin/varnishd/http2/cache_http2.h b/bin/varnishd/http2/cache_http2.h
index 1c28f99ee..70e31d14e 100644
--- a/bin/varnishd/http2/cache_http2.h
+++ b/bin/varnishd/http2/cache_http2.h
@@ -217,7 +217,7 @@ h2_error h2h_decode_bytes(struct h2_sess *h2, const uint8_t *ptr,
 void H2_Send_Get(struct worker *, struct h2_sess *, struct h2_req *);
 void H2_Send_Rel(struct h2_sess *, const struct h2_req *);
 
-h2_error H2_Send_Frame(struct worker *, const struct h2_sess *,
+void H2_Send_Frame(struct worker *, struct h2_sess *,
     h2_frame type, uint8_t flags, uint32_t len, uint32_t stream,
     const void *);
 
diff --git a/bin/varnishd/http2/cache_http2_proto.c b/bin/varnishd/http2/cache_http2_proto.c
index f6ea07eae..56aa27552 100644
--- a/bin/varnishd/http2/cache_http2_proto.c
+++ b/bin/varnishd/http2/cache_http2_proto.c
@@ -132,11 +132,10 @@ h2_connectionerror(uint32_t u)
 
 /**********************************************************************/
 
-static h2_error
+static void
 h2_tx_rst(struct worker *wrk, struct h2_sess *h2, struct h2_req *r2,
     uint32_t stream, h2_error h2e)
 {
-	h2_error ret;
 	char b[4];
 
 	CHECK_OBJ_NOTNULL(h2, H2_SESS_MAGIC);
@@ -148,11 +147,8 @@ h2_tx_rst(struct worker *wrk, struct h2_sess *h2, struct h2_req *r2,
 	vbe32enc(b, h2e->val);
 
 	H2_Send_Get(wrk, h2, r2);
-	ret = H2_Send_Frame(wrk, h2, H2_F_RST_STREAM,
-	    0, sizeof b, stream, b);
+	H2_Send_Frame(wrk, h2, H2_F_RST_STREAM, 0, sizeof b, stream, b);
 	H2_Send_Rel(h2, r2);
-
-	return (ret);
 }
 
 /**********************************************************************
@@ -936,8 +932,9 @@ h2_procframe(struct worker *wrk, struct h2_sess *h2,
 			     "H2: stream %u: Hit maximum number of "
 			     "concurrent streams", h2->rxf_stream);
 			// rfc7540,l,1200,1205
-			return (h2_tx_rst(wrk, h2, h2->req0, h2->rxf_stream,
-				H2SE_REFUSED_STREAM));
+			h2_tx_rst(wrk, h2, h2->req0, h2->rxf_stream,
+				H2SE_REFUSED_STREAM);
+			return (0);
 		}
 		h2->highest_stream = h2->rxf_stream;
 		r2 = h2_new_req(wrk, h2, h2->rxf_stream, NULL);
@@ -954,7 +951,8 @@ h2_procframe(struct worker *wrk, struct h2_sess *h2,
 	if (h2->rxf_stream == 0 || h2e->connection)
 		return (h2e);	// Connection errors one level up
 
-	return (h2_tx_rst(wrk, h2, h2->req0, h2->rxf_stream, h2e));
+	h2_tx_rst(wrk, h2, h2->req0, h2->rxf_stream, h2e);
+	return (0);
 }
 
 static int
@@ -1143,8 +1141,8 @@ h2_rxframe(struct worker *wrk, struct h2_sess *h2)
 		vbe32enc(b, h2->highest_stream);
 		vbe32enc(b + 4, h2e->val);
 		H2_Send_Get(wrk, h2, h2->req0);
-		(void)H2_Send_Frame(wrk, h2, H2_F_GOAWAY, 0, 8, 0, b);
+		H2_Send_Frame(wrk, h2, H2_F_GOAWAY, 0, 8, 0, b);
 		H2_Send_Rel(h2, h2->req0);
 	}
-	return (h2e ? 0 : 1);
+	return (h2->error ? 0 : 1);
 }
diff --git a/bin/varnishd/http2/cache_http2_send.c b/bin/varnishd/http2/cache_http2_send.c
index 054796e4b..41ca146b6 100644
--- a/bin/varnishd/http2/cache_http2_send.c
+++ b/bin/varnishd/http2/cache_http2_send.c
@@ -29,6 +29,8 @@
 
 #include "config.h"
 
+#include <sys/uio.h>
+
 #include "cache/cache_varnishd.h"
 
 #include "cache/cache_transport.h"
@@ -112,13 +114,14 @@ h2_mk_hdr(uint8_t *hdr, h2_frame ftyp, uint8_t flags,
  * the session mtx must be held.
  */
 
-h2_error
-H2_Send_Frame(struct worker *wrk, const struct h2_sess *h2,
+void
+H2_Send_Frame(struct worker *wrk, struct h2_sess *h2,
     h2_frame ftyp, uint8_t flags,
     uint32_t len, uint32_t stream, const void *ptr)
 {
 	uint8_t hdr[9];
 	ssize_t s;
+	struct iovec iov[2];
 
 	(void)wrk;
 
@@ -137,18 +140,24 @@ H2_Send_Frame(struct worker *wrk, const struct h2_sess *h2,
 		h2->srq->acct.resp_bodybytes += len;
 	Lck_Unlock(&h2->sess->mtx);
 
-	s = write(h2->sess->fd, hdr, sizeof hdr);
-	if (s != sizeof hdr)
-		return (H2CE_PROTOCOL_ERROR);		// XXX Need private ?
-	if (len > 0) {
-		s = write(h2->sess->fd, ptr, len);
-		if (s != len)
-			return (H2CE_PROTOCOL_ERROR);	// XXX Need private ?
+	memset(iov, 0, sizeof iov);
+	iov[0].iov_base = hdr;
+	iov[0].iov_len = sizeof hdr;
+	iov[1].iov_base = TRUST_ME(ptr);
+	iov[1].iov_len = len;
+	s = writev(h2->sess->fd, iov, len == 0 ? 1 : 2);
+	if (s != sizeof hdr + len) {
+		/*
+		 * There is no point in being nice here, we will be unable
+		 * to send a GOAWAY once the code unrolls, so go directly
+		 * to the finale and be done with it.
+		 */
+		h2->error = H2CE_PROTOCOL_ERROR;
+	} else if (len > 0) {
 		Lck_Lock(&h2->sess->mtx);
 		VSLb_bin(h2->vsl, SLT_H2TxBody, len, ptr);
 		Lck_Unlock(&h2->sess->mtx);
 	}
-	return (0);
 }
 
 static int64_t
@@ -251,7 +260,6 @@ void
 H2_Send(struct worker *wrk, struct h2_req *r2,
     h2_frame ftyp, uint8_t flags, uint32_t len, const void *ptr)
 {
-	h2_error retval;
 	struct h2_sess *h2;
 	uint32_t mfs, tf;
 	const char *p;
@@ -289,8 +297,7 @@ H2_Send(struct worker *wrk, struct h2_req *r2,
 		tf = mfs;
 
 	if (len <= tf) {
-		(void)H2_Send_Frame(wrk, h2,
-		    ftyp, flags, len, r2->stream, ptr);
+		H2_Send_Frame(wrk, h2, ftyp, flags, len, r2->stream, ptr);
 	} else {
 		AN(ptr);
 		p = ptr;
@@ -308,19 +315,19 @@ H2_Send(struct worker *wrk, struct h2_req *r2,
 				assert(VTAILQ_FIRST(&h2->txqueue) == r2);
 			}
 			if (tf < len) {
-				retval = H2_Send_Frame(wrk, h2, ftyp,
+				H2_Send_Frame(wrk, h2, ftyp,
 				    flags, tf, r2->stream, p);
 			} else {
 				if (ftyp->respect_window)
 					assert(tf == len);
 				tf = len;
-				retval = H2_Send_Frame(wrk, h2, ftyp,
+				H2_Send_Frame(wrk, h2, ftyp,
 				    final_flags, tf, r2->stream, p);
 				flags = 0;
 			}
 			p += tf;
 			len -= tf;
 			ftyp = ftyp->continuation;
-		} while (len > 0 && retval == 0);
+		} while (!h2->error && len > 0);
 	}
 }

From phk at FreeBSD.org  Wed Sep  5 07:29:09 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed,  5 Sep 2018 07:29:09 +0000 (UTC)
Subject: [master] 545290b81 Use UID_MAX instead of -1
Message-ID: <20180905072909.9D8EAA5873@lists.varnish-cache.org>


commit 545290b81f38ad1282eb0c7f7c2034cd4d4c645b
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Sep 5 07:25:30 2018 +0000

    Use UID_MAX instead of -1

diff --git a/bin/varnishd/mgt/mgt_jail_unix.c b/bin/varnishd/mgt/mgt_jail_unix.c
index f91cf24b9..a173726f0 100644
--- a/bin/varnishd/mgt/mgt_jail_unix.c
+++ b/bin/varnishd/mgt/mgt_jail_unix.c
@@ -39,6 +39,7 @@
 #include <string.h>
 #include <unistd.h>
 #include <sys/stat.h>
+#include <sys/limits.h>
 
 #include "mgt/mgt.h"
 #include "common/heritage.h"
@@ -280,7 +281,7 @@ vju_make_workdir(const char *dname, const char *what, struct vsb *vsb)
 		    dname, strerror(errno));
 		return (1);
 	}
-	AZ(chown(dname, -1, vju_gid));
+	AZ(chown(dname, UID_MAX, vju_gid));
 	AZ(seteuid(vju_uid));
 	return (0);
 }

From phk at FreeBSD.org  Wed Sep  5 07:29:09 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed,  5 Sep 2018 07:29:09 +0000 (UTC)
Subject: [master] bbc6e2c60 Flexelinting
Message-ID: <20180905072909.A785BA5875@lists.varnish-cache.org>


commit bbc6e2c60b8f13396299f7f5ab2131aa10c76d92
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Sep 5 07:26:15 2018 +0000

    Flexelinting

diff --git a/bin/varnishd/cache/cache_shmlog.c b/bin/varnishd/cache/cache_shmlog.c
index 9ae48331b..e4db8158f 100644
--- a/bin/varnishd/cache/cache_shmlog.c
+++ b/bin/varnishd/cache/cache_shmlog.c
@@ -260,7 +260,7 @@ VSL_Flush(struct vsl_log *vsl, int overflow)
 	memcpy(p + 2, vsl->wlb, l);
 	p[1] = l;
 	VWMB();
-	p[0] = ((((unsigned)SLT__Batch & 0xff) << 24) | 0);
+	p[0] = ((((unsigned)SLT__Batch & 0xff) << 24));
 	vsl->wlp = vsl->wlb;
 	vsl->wlr = 0;
 }
diff --git a/bin/varnishd/cache/cache_vary.c b/bin/varnishd/cache/cache_vary.c
index 379f7b36f..46d6930fb 100644
--- a/bin/varnishd/cache/cache_vary.c
+++ b/bin/varnishd/cache/cache_vary.c
@@ -335,7 +335,7 @@ VRY_Match(struct req *req, const uint8_t *vary)
 			memcpy(vsp + 2, vary + 2, vary[2] + 2);
 			if (h != NULL)
 				memcpy(vsp + 2 + vsp[2] + 2, h, lh);
-			vsp[ln + 0] = 0xff;
+			vsp[ln] = 0xff;
 			vsp[ln + 1] = 0xff;
 			vsp[ln + 2] = 0;
 			(void)VRY_Validate(vsp);
diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c
index bfd0d80df..7cb3feb17 100644
--- a/bin/varnishd/cache/cache_vrt_var.c
+++ b/bin/varnishd/cache/cache_vrt_var.c
@@ -629,6 +629,8 @@ VRT_r_##which##_##fld(VRT_CTX)					\
 	return (d);						\
 }
 
+/*lint -save -e835 */	// Zero right hand arg to '-'
+
 VRT_DO_EXP_R(obj, ctx->req->objcore, ttl,
     ttl_now(ctx) - ctx->req->objcore->t_origin)
 VRT_DO_EXP_R(obj, ctx->req->objcore, grace, 0)
@@ -643,6 +645,8 @@ VRT_DO_EXP_R(beresp, ctx->bo->fetch_objcore, grace, 0)
 VRT_DO_EXP_L(beresp, ctx->bo->fetch_objcore, keep, 0)
 VRT_DO_EXP_R(beresp, ctx->bo->fetch_objcore, keep, 0)
 
+/*lint -restore */
+
 /*--------------------------------------------------------------------
  */
 

From phk at FreeBSD.org  Wed Sep  5 07:29:09 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed,  5 Sep 2018 07:29:09 +0000 (UTC)
Subject: [master] ae6e9e2bc Polishing
Message-ID: <20180905072909.E63DCA587A@lists.varnish-cache.org>


commit ae6e9e2bc081853d45fca1911a459fe33484e853
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Sep 5 07:28:36 2018 +0000

    Polishing

diff --git a/bin/varnishd/http2/cache_http2_proto.c b/bin/varnishd/http2/cache_http2_proto.c
index 56aa27552..070bf2cad 100644
--- a/bin/varnishd/http2/cache_http2_proto.c
+++ b/bin/varnishd/http2/cache_http2_proto.c
@@ -893,10 +893,9 @@ h2_frame_complete(struct http_conn *htc)
 /**********************************************************************/
 
 static h2_error
-h2_procframe(struct worker *wrk, struct h2_sess *h2,
-    h2_frame h2f)
+h2_procframe(struct worker *wrk, struct h2_sess *h2, h2_frame h2f)
 {
-	struct h2_req *r2 = NULL;
+	struct h2_req *r2;
 	h2_error h2e;
 
 	ASSERT_RXTHR(h2);
diff --git a/bin/varnishd/http2/cache_http2_session.c b/bin/varnishd/http2/cache_http2_session.c
index 5985fe472..ce114d390 100644
--- a/bin/varnishd/http2/cache_http2_session.c
+++ b/bin/varnishd/http2/cache_http2_session.c
@@ -55,18 +55,16 @@ static const char H2_prism[24] = {
 static size_t
 h2_enc_settings(const struct h2_settings *h2s, uint8_t *buf, size_t n)
 {
-	uint8_t *b;
 	size_t len = 0;
 
-	b = buf;
 #define H2_SETTING(U,l,v,d,...)				\
 	if (h2s->l != d) {				\
 		len += 6;				\
 		assert(len <= n);			\
-		vbe16enc(b, v);				\
-		b += 2;					\
-		vbe32enc(b, h2s->l);			\
-		b += 4;					\
+		vbe16enc(buf, v);			\
+		buf += 2;				\
+		vbe32enc(buf, h2s->l);			\
+		buf += 4;				\
 	}
 #include "tbl/h2_settings.h"
 	return (len);

From phk at FreeBSD.org  Wed Sep  5 07:29:09 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed,  5 Sep 2018 07:29:09 +0000 (UTC)
Subject: [master] 198dc16ec Fail object delivery little earlier if the
 session-fd is dead.
Message-ID: <20180905072909.BDB93A5877@lists.varnish-cache.org>


commit 198dc16ec04a34c312f118fe3c2e3602f3aed92a
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Sep 5 07:28:13 2018 +0000

    Fail object delivery little earlier if the session-fd is dead.

diff --git a/bin/varnishd/http2/cache_http2_deliver.c b/bin/varnishd/http2/cache_http2_deliver.c
index aea59e9a8..912eb88bc 100644
--- a/bin/varnishd/http2/cache_http2_deliver.c
+++ b/bin/varnishd/http2/cache_http2_deliver.c
@@ -84,7 +84,7 @@ h2_bytes(struct req *req, enum vdp_action act, void **priv,
 
 	if (act == VDP_INIT)
 		return (0);
-	if (r2->error && act != VDP_FINI)
+	if ((r2->h2sess->error || r2->error) && act != VDP_FINI)
 		return (-1);
 	H2_Send_Get(req->wrk, r2->h2sess, r2);
 	H2_Send(req->wrk, r2,
@@ -189,7 +189,7 @@ h2_deliver(struct req *req, struct boc *boc, int sendbody)
 	struct sess *sp;
 	struct h2_req *r2;
 	struct vsb resp;
-	int i, err;
+	int i;
 	const struct hpack_static *hps;
 
 	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
@@ -264,9 +264,8 @@ h2_deliver(struct req *req, struct boc *boc, int sendbody)
 
 	/* XXX someone into H2 please add appropriate error handling */
 	if (sendbody) {
-		err = VDP_push(req, &h2_vdp, NULL, 1);
-		if (!err)
-			err = VDP_DeliverObj(req);
+		if (!VDP_push(req, &h2_vdp, NULL, 1))
+			(void)VDP_DeliverObj(req);
 	}
 
 	AZ(req->wrk->v1l);

From phk at FreeBSD.org  Wed Sep  5 07:40:09 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed,  5 Sep 2018 07:40:09 +0000 (UTC)
Subject: [master] cddc44128 Revert this last change, I was not awake.
Message-ID: <20180905074009.60E3EA6083@lists.varnish-cache.org>


commit cddc4412820771e4fdf4f0cde46f72241d7c9176
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Sep 5 07:39:29 2018 +0000

    Revert this last change, I was not awake.

diff --git a/bin/varnishd/mgt/mgt_jail_unix.c b/bin/varnishd/mgt/mgt_jail_unix.c
index a173726f0..f91cf24b9 100644
--- a/bin/varnishd/mgt/mgt_jail_unix.c
+++ b/bin/varnishd/mgt/mgt_jail_unix.c
@@ -39,7 +39,6 @@
 #include <string.h>
 #include <unistd.h>
 #include <sys/stat.h>
-#include <sys/limits.h>
 
 #include "mgt/mgt.h"
 #include "common/heritage.h"
@@ -281,7 +280,7 @@ vju_make_workdir(const char *dname, const char *what, struct vsb *vsb)
 		    dname, strerror(errno));
 		return (1);
 	}
-	AZ(chown(dname, UID_MAX, vju_gid));
+	AZ(chown(dname, -1, vju_gid));
 	AZ(seteuid(vju_uid));
 	return (0);
 }

From phk at FreeBSD.org  Wed Sep  5 07:46:07 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed,  5 Sep 2018 07:46:07 +0000 (UTC)
Subject: [master] 5141161f8 Spend a void for GCC's -Werror=pointer-sign
Message-ID: <20180905074608.0FD54A638C@lists.varnish-cache.org>


commit 5141161f86eaec111b5bb6e0d42c14f24e826d88
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Sep 5 07:45:27 2018 +0000

    Spend a void for GCC's -Werror=pointer-sign

diff --git a/bin/varnishd/http2/cache_http2_send.c b/bin/varnishd/http2/cache_http2_send.c
index 41ca146b6..638fea632 100644
--- a/bin/varnishd/http2/cache_http2_send.c
+++ b/bin/varnishd/http2/cache_http2_send.c
@@ -141,7 +141,7 @@ H2_Send_Frame(struct worker *wrk, struct h2_sess *h2,
 	Lck_Unlock(&h2->sess->mtx);
 
 	memset(iov, 0, sizeof iov);
-	iov[0].iov_base = hdr;
+	iov[0].iov_base = (void*)hdr;
 	iov[0].iov_len = sizeof hdr;
 	iov[1].iov_base = TRUST_ME(ptr);
 	iov[1].iov_len = len;

From phk at FreeBSD.org  Wed Sep  5 08:18:10 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed,  5 Sep 2018 08:18:10 +0000 (UTC)
Subject: [master] 4195327ef Use a dedicated short-TTL'ed FQDN for testing if
 DNS works.
Message-ID: <20180905081810.806ACA6DD6@lists.varnish-cache.org>


commit 4195327efd40bb85439632aa416cce73ba1dc5be
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Sep 5 08:16:37 2018 +0000

    Use a dedicated short-TTL'ed FQDN for testing if DNS works.

diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c
index e85ae92b7..d6168a2ae 100644
--- a/bin/varnishtest/vtc_main.c
+++ b/bin/varnishtest/vtc_main.c
@@ -441,7 +441,7 @@ dns_cb(void *priv, const struct suckaddr *sa)
 	int *ret = priv;
 
 	VTCP_name(sa, abuf, sizeof abuf, pbuf, sizeof pbuf);
-	if (strcmp(abuf, "130.225.244.222")) {
+	if (strcmp(abuf, "192.0.2.255")) {
 		fprintf(stderr, "DNS-test: Wrong response: %s\n", abuf);
 		*ret = -1;
 	} else if (*ret == 0)
@@ -455,7 +455,7 @@ dns_works(void)
 	int ret = 0, error;
 	const char *msg;
 
-	error = VSS_resolver("phk.freebsd.dk", NULL, dns_cb, &ret, &msg);
+	error = VSS_resolver("dns-canary.freebsd.dk", NULL, dns_cb, &ret, &msg);
 	if (error || msg != NULL || ret != 1)
 		return (0);
 	return (1);

From phk at FreeBSD.org  Wed Sep  5 09:00:15 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed,  5 Sep 2018 09:00:15 +0000 (UTC)
Subject: [master] fe5584216 Don't attempt to normalize .vcc file encoding,
 just take the bytes
Message-ID: <20180905090016.0B209A7B47@lists.varnish-cache.org>


commit fe558421628f17340fae6a0b1cca477f04d033b1
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Sep 5 08:58:10 2018 +0000

    Don't attempt to normalize .vcc file encoding, just take the bytes
    
    Fixes #2761

diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py
index 3032f380f..7a4e362fc 100755
--- a/lib/libvcc/vmodtool.py
+++ b/lib/libvcc/vmodtool.py
@@ -859,7 +859,7 @@ class vcc(object):
     def parse(self):
         global inputline
         a = "\n" + open(self.inputfile, "r").read()
-        self.file_id = hashlib.sha256(a.encode('utf-8')).hexdigest()
+        self.file_id = hashlib.sha256(bytearray(a)).hexdigest()
         s = a.split("\n$")
         self.copyright = s.pop(0).strip()
         while s:
diff --git a/lib/libvmod_debug/vmod.vcc b/lib/libvmod_debug/vmod.vcc
index 0ab33501f..200b69de2 100644
--- a/lib/libvmod_debug/vmod.vcc
+++ b/lib/libvmod_debug/vmod.vcc
@@ -24,6 +24,8 @@
 # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
 # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
 # SUCH DAMAGE.
+#
+# Test vmodtool.py's UTF-8 handling: ?blefl?sk med koldsk?l eller ?l
 
 $Module debug 3 Development, test and debug
 $ABI strict

From phk at FreeBSD.org  Wed Sep  5 18:08:08 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed,  5 Sep 2018 18:08:08 +0000 (UTC)
Subject: [master] f0397b830 Tighten an assert,
 and replicate it another place with similar constraints.
Message-ID: <20180905180808.D665D90EE@lists.varnish-cache.org>


commit f0397b8306389860cdf01d84e934577ad718524f
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Sep 5 18:07:06 2018 +0000

    Tighten an assert, and replicate it another place with similar
    constraints.
    
    Related to: #2763

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index 57583239a..bedc6dae7 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -195,7 +195,7 @@ vbf_stp_mkbereq(struct worker *wrk, struct busyobj *bo)
 	if (bo->stale_oc != NULL &&
 	    ObjCheckFlag(bo->wrk, bo->stale_oc, OF_IMSCAND) &&
 	    (bo->stale_oc->boc != NULL || ObjGetLen(wrk, bo->stale_oc) != 0)) {
-		AZ(bo->stale_oc->flags & OC_F_PASS);
+		AZ(bo->stale_oc->flags & (OC_F_PASS|OC_F_PRIVATE));
 		q = HTTP_GetHdrPack(bo->wrk, bo->stale_oc, H_Last_Modified);
 		if (q != NULL)
 			http_PrintfHeader(bo->bereq0,
@@ -355,6 +355,7 @@ vbf_stp_startfetch(struct worker *wrk, struct busyobj *bo)
 	if (http_IsStatus(bo->beresp, 304)) {
 		if (bo->stale_oc != NULL &&
 		    ObjCheckFlag(bo->wrk, bo->stale_oc, OF_IMSCAND)) {
+			AZ(bo->stale_oc->flags & (OC_F_PASS|OC_F_PRIVATE));
 			if (ObjCheckFlag(bo->wrk, bo->stale_oc, OF_CHGGZIP)) {
 				/*
 				 * If we changed the gzip status of the object

From phk at FreeBSD.org  Thu Sep  6 07:41:16 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu,  6 Sep 2018 07:41:16 +0000 (UTC)
Subject: [master] ea6b5ec58 Test more of varnishhist
Message-ID: <20180906074116.350249765A@lists.varnish-cache.org>


commit ea6b5ec586f9a035a0f364937406737c735732e8
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Sep 6 07:40:16 2018 +0000

    Test more of varnishhist

diff --git a/bin/varnishtest/tests/u00009.vtc b/bin/varnishtest/tests/u00009.vtc
index 5a3bf928b..bd6286e4f 100644
--- a/bin/varnishtest/tests/u00009.vtc
+++ b/bin/varnishtest/tests/u00009.vtc
@@ -8,8 +8,12 @@ server s1 {
 varnish v1 -vcl+backend {} -start
 
 process p1 -dump {varnishhist -n ${v1_name}} -start
+process p2 -dump {varnishhist -n ${v1_name} -P b:BereqAcct::5:1:8} -start
+process p3 -dump {varnishhist -n ${v1_name} -P BerespBodytime} -start
 
 process p1 -expect-text 24 0 {1e2}
+process p2 -expect-text 24 0 {1e2}
+process p3 -expect-text 24 0 {1e2}
 
 delay 1
 
@@ -24,14 +28,14 @@ process p1 -expect-text 3 1 {20_}
 
 process p1 -screen_dump
 
-process p1 -winsz 23 80
+process p1 -winsz 23 80 -need-bytes +10
 
-delay 5
-
-process p1 -write {0>+-<}
-
-delay 1.5
+process p1 -write {0>+-<} -need-bytes +10
 
 process p1 -expect-text 2 1 {20_}
 
+process p1 -writehex 0c -need-bytes +10
+
 process p1 -screen_dump -write {q} -wait
+process p2 -screen_dump -write {q} -wait
+process p3 -screen_dump -write {q} -wait

From phk at FreeBSD.org  Thu Sep  6 08:59:13 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu,  6 Sep 2018 08:59:13 +0000 (UTC)
Subject: [master] 65c6d4294 Fix a race exposed on fast 32 bit FreeBSD
Message-ID: <20180906085913.CD67C9B10B@lists.varnish-cache.org>


commit 65c6d429449f9ebc711f724802e5a8192de1d484
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Sep 6 08:58:25 2018 +0000

    Fix a race exposed on fast 32 bit FreeBSD

diff --git a/bin/varnishtest/vtc_process.c b/bin/varnishtest/vtc_process.c
index 5f5cd2a86..be943cfaf 100644
--- a/bin/varnishtest/vtc_process.c
+++ b/bin/varnishtest/vtc_process.c
@@ -916,7 +916,7 @@ void
 cmd_process(CMD_ARGS)
 {
 	struct process *p, *p2;
-	uintmax_t u, v;
+	uintmax_t u, v, bsnap;
 	unsigned lin,col;
 	int spec_set = 0;
 
@@ -952,6 +952,8 @@ cmd_process(CMD_ARGS)
 		p = process_new(av[0]);
 	av++;
 
+	bsnap = p->stdout_bytes;
+
 	for (; *av != NULL; av++) {
 		if (vtc_error)
 			break;
@@ -1003,7 +1005,7 @@ cmd_process(CMD_ARGS)
 		if (!strcmp(*av, "-need-bytes")) {
 			u = strtoumax(av[1], NULL, 0);
 			if (av[1][0] == '+')
-				u += p->stdout_bytes;
+				u += bsnap;
 			av++;
 			do {
 				AZ(pthread_mutex_lock(&p->mtx));

From nils.goroll at uplex.de  Thu Sep  6 20:20:11 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Thu,  6 Sep 2018 20:20:11 +0000 (UTC)
Subject: [master] 5e838844a Sync comment with reality
Message-ID: <20180906202011.66E91A8E7E@lists.varnish-cache.org>


commit 5e838844ab3772ad4c961c94292efff68b974dd3
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Thu Sep 6 22:18:14 2018 +0200

    Sync comment with reality
    
    it really is about time we rename OC_F_PASS -> OC_F_HFM, but
    my recent cleanup attempt included too much change before the release

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index bedc6dae7..ee5923db7 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -451,7 +451,7 @@ vbf_stp_fetchbody(struct worker *wrk, struct busyobj *bo)
 		if (vfc->oc->flags & OC_F_ABANDON) {
 			/*
 			 * A pass object and delivery was terminated
-			 * We don't fail the fetch, in order for hit-for-pass
+			 * We don't fail the fetch, in order for HitMiss
 			 * objects to be created.
 			 */
 			AN(vfc->oc->flags & OC_F_PASS);

From phk at FreeBSD.org  Thu Sep  6 21:51:05 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu,  6 Sep 2018 21:51:05 +0000 (UTC)
Subject: [master] 1ed631942 Test varnishhist a little bit more
Message-ID: <20180906215105.DA721AC833@lists.varnish-cache.org>


commit 1ed63194208fda6502df1489c09703c3b3a05a92
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Sep 6 21:50:04 2018 +0000

    Test varnishhist a little bit more

diff --git a/bin/varnishhist/varnishhist.c b/bin/varnishhist/varnishhist.c
index 65f9246be..52bb69a6d 100644
--- a/bin/varnishhist/varnishhist.c
+++ b/bin/varnishhist/varnishhist.c
@@ -422,8 +422,8 @@ do_curses(void *arg)
 		case 'Q':
 		case 'q':
 			AZ(raise(SIGINT));
-			endwin();
-			return (NULL);
+			quit = 1;
+			break;
 		case '0':
 		case '1':
 		case '2':
diff --git a/bin/varnishtest/tests/u00009.vtc b/bin/varnishtest/tests/u00009.vtc
index bd6286e4f..8b962a63d 100644
--- a/bin/varnishtest/tests/u00009.vtc
+++ b/bin/varnishtest/tests/u00009.vtc
@@ -24,6 +24,13 @@ client c1 {
 
 process p1 -expect-text 22 0 {#}
 
+client c1 {
+	txreq
+	rxresp
+} -run
+
+process p1 -expect-text 22 0 { | }
+
 process p1 -expect-text 3 1 {20_}
 
 process p1 -screen_dump

From nils.goroll at uplex.de  Fri Sep  7 10:15:12 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Fri,  7 Sep 2018 10:15:12 +0000 (UTC)
Subject: [master] addea755c Clear the IMS object attribute when copying from a
 stale object
Message-ID: <20180907101512.F23E364126@lists.varnish-cache.org>


commit addea755c1d41e0431730da666363f76ad00990a
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Thu Sep 6 23:10:17 2018 +0200

    Clear the IMS object attribute when copying from a stale object
    
    The fact that the previous object was IMS/INM eligable does not make
    one created from it eligable as well.
    
    Also test for the miss + hfm case with IMS/INM
    
    Fixes #2763

diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c
index ee5923db7..04617f2f7 100644
--- a/bin/varnishd/cache/cache_fetch.c
+++ b/bin/varnishd/cache/cache_fetch.c
@@ -740,6 +740,7 @@ vbf_stp_condfetch(struct worker *wrk, struct busyobj *bo)
 		    OA_ESIDATA));
 
 	AZ(ObjCopyAttr(bo->wrk, bo->fetch_objcore, bo->stale_oc, OA_FLAGS));
+	ObjSetFlag(bo->wrk, bo->fetch_objcore, OF_IMSCAND, 0);
 	AZ(ObjCopyAttr(bo->wrk, bo->fetch_objcore, bo->stale_oc, OA_GZIPBITS));
 
 	if (bo->do_stream) {
diff --git a/bin/varnishtest/tests/r02763.vtc b/bin/varnishtest/tests/r02763.vtc
new file mode 100644
index 000000000..3488d7c95
--- /dev/null
+++ b/bin/varnishtest/tests/r02763.vtc
@@ -0,0 +1,68 @@
+varnishtest "Cacheable IMS replaced by HFM object"
+
+server s1 {
+	rxreq
+	expect req.url == "/etag"
+	txresp -hdr "ETag: foo" -bodylen 7
+
+	rxreq
+	expect req.url == "/etag"
+	expect req.http.If-None-Match == "foo"
+	txresp -status 304 -hdr "ETag: foo"
+
+	rxreq
+	expect req.url == "/etag"
+	txresp -hdr "ETag: foo" -bodylen 7
+} -start
+
+varnish v1 -vcl+backend {
+
+	sub vcl_miss {
+		set req.http.X-Cache = "MISS";
+	}
+
+	sub vcl_pass {
+		set req.http.X-Cache = "PASS";
+	}
+
+	sub vcl_backend_response {
+		if (bereq.http.HFM) {
+			set beresp.uncacheable = true;
+		} else {
+			set beresp.ttl = 0.0001s;
+			set beresp.grace = 0s;
+			set beresp.keep = 1m;
+		}
+		return (deliver);
+	}
+
+	sub vcl_deliver {
+		set resp.http.X-Cache = req.http.X-Cache;
+	}
+
+} -start
+
+client c1 {
+	txreq -url "/etag"
+	rxresp
+	expect resp.status == 200
+	expect resp.bodylen == 7
+	expect resp.http.ETag == "foo"
+	expect resp.http.X-Cache == "MISS"
+
+	delay 0.1
+
+	txreq -url "/etag" -hdr "HFM: true"
+	rxresp
+	expect resp.status == 200
+	expect resp.bodylen == 7
+	expect resp.http.ETag == "foo"
+	expect resp.http.X-Cache == "MISS"
+
+	txreq -url "/etag"
+	rxresp
+	expect resp.status == 200
+	expect resp.bodylen == 7
+	expect resp.http.ETag == "foo"
+	expect resp.http.X-Cache == "MISS"
+} -run

From phk at FreeBSD.org  Sun Sep  9 12:42:10 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Sun,  9 Sep 2018 12:42:10 +0000 (UTC)
Subject: [master] 0cb0fe9fc Ignore values non-positive values.
Message-ID: <20180909124211.0FC02A2841@lists.varnish-cache.org>


commit 0cb0fe9fc1f4c3bb39b66c98efb40b629564ec9c
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Sep 9 12:41:18 2018 +0000

    Ignore values non-positive values.
    
    Fixes:  #2773

diff --git a/bin/varnishhist/varnishhist.c b/bin/varnishhist/varnishhist.c
index 52bb69a6d..1ef81f15f 100644
--- a/bin/varnishhist/varnishhist.c
+++ b/bin/varnishhist/varnishhist.c
@@ -295,7 +295,7 @@ accumulate(struct VSL_data *vsl, struct VSL_transaction * const pt[],
 			}
 		}
 
-		if (skip || !match)
+		if (skip || !match || value <= 0)
 			continue;
 
 		/* select bucket */

From phk at FreeBSD.org  Sun Sep  9 14:55:15 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Sun,  9 Sep 2018 14:55:15 +0000 (UTC)
Subject: [master] 88d648f81 Test more distant corners of varnishhist
Message-ID: <20180909145515.CBB98A5194@lists.varnish-cache.org>


commit 88d648f81d898f66d14fadefb6dfa2257853f9bb
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Sep 9 14:54:03 2018 +0000

    Test more distant corners of varnishhist

diff --git a/bin/varnishhist/varnishhist.c b/bin/varnishhist/varnishhist.c
index 1ef81f15f..48bfce265 100644
--- a/bin/varnishhist/varnishhist.c
+++ b/bin/varnishhist/varnishhist.c
@@ -657,8 +657,7 @@ main(int argc, char **argv)
 		ident = VSM_Dup(vut->vsm, "Arg", "-i");
 	else
 		ident = strdup("");
-	if (pthread_create(&thr, NULL, do_curses, NULL) != 0)
-		VUT_Error(vut, 1, "pthread_create(): %s", strerror(errno));
+	AZ(pthread_create(&thr, NULL, do_curses, NULL));
 	vut->dispatch_f = accumulate;
 	vut->dispatch_priv = NULL;
 	vut->sighup_f = sighup;
diff --git a/bin/varnishtest/tests/u00007.vtc b/bin/varnishtest/tests/u00007.vtc
index 64e9dc7c5..fb3a81a26 100644
--- a/bin/varnishtest/tests/u00007.vtc
+++ b/bin/varnishtest/tests/u00007.vtc
@@ -11,12 +11,14 @@ shell -match "Usage: .*varnishhist <options>" \
 	"varnishhist -h"
 shell -expect "Copyright (c) 2006 Verdens Gang AS" \
 	"varnishhist -V"
+shell -err -match "Usage: .*varnishhist <options>" \
+	"varnishhist -K"
 shell -err -match "Usage: .*varnishhist <options>" \
 	"varnishhist extra"
 shell -err -expect "-p: invalid '0'" \
 	"varnishhist -p 0"
 shell -err -expect "-B: being able to bend time does not mean we can stop it" \
-	"varnishhist -B 0"
+	"varnishhist -p 2.0 -B 0"
 shell -err -expect "-B: being able to bend time does not mean we can make it go backwards" \
 	"varnishhist -B -1"
 shell -err -expect "Invalid grouping mode: raw" \
@@ -25,5 +27,13 @@ shell -err -expect "-P: No such profile 'foo'" \
 	"varnishhist -P foo"
 shell -err -expect "-P: 'Timestamp:' is not a valid profile name or definition" \
 	"varnishhist -P Timestamp::"
-shell -err -expect "-P: 'foo::0:0:0' is not a valid tag name" \
-	"varnishhist -P foo::0:0:0"
+shell -err -expect "-P: 'b:' is not a valid profile name or definition" \
+	"varnishhist -P b:"
+shell -err -expect "-P: 'foo:' is not a valid tag name" \
+	"varnishhist -P foo:"
+shell -err -expect "-P: 'b:Debug:' is an unsafe or binary record" \
+	"varnishhist -P b:Debug:"
+shell -err -expect "-P: 'b:BereqAcct:x' is not a valid profile name or definition" \
+	"varnishhist -P b:BereqAcct:x"
+shell -err -expect "-P: 'b:BereqAcct:x' is not a valid profile name or definition" \
+	"varnishhist -P b:BereqAcct:x:"
diff --git a/bin/varnishtest/tests/u00009.vtc b/bin/varnishtest/tests/u00009.vtc
index 8b962a63d..df3551cef 100644
--- a/bin/varnishtest/tests/u00009.vtc
+++ b/bin/varnishtest/tests/u00009.vtc
@@ -9,7 +9,7 @@ varnish v1 -vcl+backend {} -start
 
 process p1 -dump {varnishhist -n ${v1_name}} -start
 process p2 -dump {varnishhist -n ${v1_name} -P b:BereqAcct::5:1:8} -start
-process p3 -dump {varnishhist -n ${v1_name} -P BerespBodytime} -start
+process p3 -dump {varnishhist -n ${v1_name} -P BerespBodytime -B 2} -start
 
 process p1 -expect-text 24 0 {1e2}
 process p2 -expect-text 24 0 {1e2}
@@ -39,6 +39,8 @@ process p1 -winsz 23 80 -need-bytes +10
 
 process p1 -write {0>+-<} -need-bytes +10
 
+process p3 -write {0>+-<x} -need-bytes +10
+
 process p1 -expect-text 2 1 {20_}
 
 process p1 -writehex 0c -need-bytes +10

From phk at FreeBSD.org  Sun Sep  9 18:33:08 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Sun,  9 Sep 2018 18:33:08 +0000 (UTC)
Subject: [master] 594c737d7 Test even more obscure corners of varnishhist
Message-ID: <20180909183308.797D0A8DBD@lists.varnish-cache.org>


commit 594c737d7e0629b1691f2c2c33b02501247e7214
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Sun Sep 9 18:31:44 2018 +0000

    Test even more obscure corners of varnishhist

diff --git a/bin/varnishtest/tests/u00007.vtc b/bin/varnishtest/tests/u00007.vtc
index fb3a81a26..161643130 100644
--- a/bin/varnishtest/tests/u00007.vtc
+++ b/bin/varnishtest/tests/u00007.vtc
@@ -37,3 +37,7 @@ shell -err -expect "-P: 'b:BereqAcct:x' is not a valid profile name or definitio
 	"varnishhist -P b:BereqAcct:x"
 shell -err -expect "-P: 'b:BereqAcct:x' is not a valid profile name or definition" \
 	"varnishhist -P b:BereqAcct:x:"
+shell -err -expect "-P: 'b:BereqAcct:' is not a valid profile name or definition" \
+	"varnishhist -P b:BereqAcct::5:1:a"
+shell -err -expect "-p: invalid '0'" \
+	"varnishhist -P b:BereqAcct::5 -p 0"
diff --git a/bin/varnishtest/tests/u00009.vtc b/bin/varnishtest/tests/u00009.vtc
index df3551cef..3f62e34ab 100644
--- a/bin/varnishtest/tests/u00009.vtc
+++ b/bin/varnishtest/tests/u00009.vtc
@@ -2,17 +2,17 @@ varnishtest	"trivial run of varnishhist in curses mode"
 
 server s1 {
 	rxreq
-	txresp
+	txresp -bodylen 32
 } -start
 
 varnish v1 -vcl+backend {} -start
 
 process p1 -dump {varnishhist -n ${v1_name}} -start
-process p2 -dump {varnishhist -n ${v1_name} -P b:BereqAcct::5:1:8} -start
+process p2 -dump {varnishhist -n ${v1_name} -P b:BereqAcct::5:-1:1} -start
 process p3 -dump {varnishhist -n ${v1_name} -P BerespBodytime -B 2} -start
 
 process p1 -expect-text 24 0 {1e2}
-process p2 -expect-text 24 0 {1e2}
+process p2 -expect-text 24 0 {1e-1}
 process p3 -expect-text 24 0 {1e2}
 
 delay 1
@@ -22,13 +22,18 @@ client c1 {
 	rxresp
 } -run
 
+varnish v1 -vsl_catchup
+
 process p1 -expect-text 22 0 {#}
+process p2 -expect-text 22 80 {#}
 
 client c1 {
 	txreq
 	rxresp
 } -run
 
+varnish v1 -vsl_catchup
+
 process p1 -expect-text 22 0 { | }
 
 process p1 -expect-text 3 1 {20_}
@@ -39,12 +44,12 @@ process p1 -winsz 23 80 -need-bytes +10
 
 process p1 -write {0>+-<} -need-bytes +10
 
-process p3 -write {0>+-<x} -need-bytes +10
+process p3 -write {0++++++++++>+-<x} -need-bytes +10
 
 process p1 -expect-text 2 1 {20_}
 
 process p1 -writehex 0c -need-bytes +10
 
 process p1 -screen_dump -write {q} -wait
-process p2 -screen_dump -write {q} -wait
+process p2 -screen_dump -kill HUP -wait
 process p3 -screen_dump -write {q} -wait

From geoff at uplex.de  Sun Sep  9 21:04:07 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Sun,  9 Sep 2018 21:04:07 +0000 (UTC)
Subject: [master] 83abe5fde First wave of release notes for 2018-09 in
 'Upgrading'.
Message-ID: <20180909210407.CF1BEAD5D9@lists.varnish-cache.org>


commit 83abe5fde0738abd810328664cd81aeb54ab69dc
Author: Geoff Simmons <geoff at uplex.de>
Date:   Sun Sep 9 23:03:05 2018 +0200

    First wave of release notes for 2018-09 in 'Upgrading'.

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 40bf30224..f91e6a188 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -19,7 +19,21 @@ Upgrading to Varnish 6.1
 varnishd parameters
 ===================
 
-**XXX**
+We have added the :ref:`ref_param_max_vcl` parameter to set a
+threshold for the number of loaded VCL programs, since it is a common
+error to let previous VCL instances accumulate without discarding
+them. The default threshold is 100, and VCL labels are not counted
+against the total. The :ref:`ref_param_max_vcl_handling` parameter
+controls what happens when you reach the limit. By default you just
+get a warning from the VCL compiler, but you can set it to refuse to
+load more VCLs, or to ignore the threshold.
+
+Added the :ref:`ref_param_backend_local_error_holddown` and
+:ref:`ref_param_backend_remote_error_holddown` parameters. These define
+delays for new attempts to connect to backends when certain classes of
+errors have been encountered, for which immediate re-connect attempts
+are likely to be counter-productive. See the parameter documentation
+for details.
 
 Changes to VCL
 ==============
@@ -32,25 +46,77 @@ Changes to VCL
 VCL variables
 ~~~~~~~~~~~~~
 
-``var1``
---------
-
-**XXX**
+``req.ttl``, ``req.grace`` and keep
+-----------------------------------
+
+``req.ttl`` had been previously listed as deprecated, but it is now
+fully supported, since there are use cases that cannot be solved
+without it.
+
+``req.ttl`` and ``req.grace`` set upper bounds on the TTL and grace
+times that are permitted for the current request -- if these variables
+are set and the TTL/grace of a cache object is longer than their
+settings, then a new response is fetched from the backend, despite the
+presence of the response in the cache.
+
+A common application is to set shorter TTLs when the backend is known
+to be healthy, so that responses are fresher when all is well. But if
+the backend is unhealthy, then use cached responses with longer TTLs
+to relieve load on the troubled backend::
+
+  sub vcl_recv {
+	# ...
+	if (std.healthy(req.backend_hint)) {
+		# Get responses no older than 70s for healthy backends
+		set req.ttl = 60s;
+		set req.grace = 10s;
+	}
+	# ...
+  }
+
+The evaluation of the ``beresp.keep`` timer has changed a
+bit. ``keep`` sets a lifetime in the cache in addition to TTL for
+objects that can be validated by a 304 "Not Modified" response from
+the backend to a conditional request (with ``If-None-Match`` or
+``If-Modified-Since``). If an expired object is also out of grace
+time, it is no longer possible to deliver a "keep" object from
+``vcl_hit``. It is possible to validate a 304 candidate from
+``vcl_miss``.
+
+The documentation in :ref:`users-guide-handling_misbehaving_servers`
+has been expanded to discuss these matters in greater depth, look
+there for more details.
 
 ``var2``
 --------
 
 **XXX**
 
-**etc**
-~~~~~~~
+Other changes
+~~~~~~~~~~~~~
 
-**XXX**
+You can now provide a string argument to ``return(fail("Foo!"))``,
+which can be used in ``vcl_init`` to emit an error message if the VCL
+load fails due to the return.
 
-**VMOD changes**
-================
+If you have set ``.proxy_header=1`` (to use the PROXYv1 protocol) for
+a backend addressed as a Unix domain socket (with a ``.path`` setting
+for the socket file), and have also defined a probe for the backend,
+then then the address family ``UNKNOWN`` is sent in the proxy header
+for the probe request. If you have set ``.proxy_header=2`` (for
+PROXYv2) for a UDS backend with a probe, then ``PROXY LOCAL`` is sent
+for the probe request.
 
-**XXX**
+VMODs
+=====
+
+Added the :ref:`func_fnmatch` function to :ref:`vmod_std(3)`, which
+you can use for shell-style wildcard matching (if you prefer that to
+regular expressions).
+
+:ref:`vmod_unix(3)` is now supported for SunOS and descendants. This
+entails changing the privilege set of the child process while the VMOD
+is loaded, see the documentation.
 
 **anything else**
 =================
@@ -62,26 +128,112 @@ Other changes
 
 * ``varnishd(1)``:
 
-  * **XXX**
+  * Some VCL compile-time error messages have been improved, for
+    example when a symbol is not found or arguments to VMOD calls are
+    missing.
 
   * **XXX**
 
 * ``varnishlog(1)``:
 
-  * **XXX**
+  * When a backend is unhealthy, ``Backend_health`` now reports some
+    diagnostic information in addition to the HTTP response and timing
+    information.
+
+  * The backend name logged for ``Backend_health`` is just the backend
+    name without the VCL prefix (as appears otherwise for backend
+    naming).
 
   * **XXX**
 
-* **other binaries in the distribution**
+* ``varnishadm(1)`` and ``varnish-cli(7)``
 
-  * **XXX**
+  * For a number of CLI commands, you can now use the ``-j`` argument
+    to get a JSON response, which may help in automation. These include:
+
+    * ``ping -j``
+
+    * **XXX...**
+
+    A JSON response in the CLI always includes a timestamp (epoch time in
+    seconds with millisecond precision).
+
+* ``varnishstat(1)`` and ``varnish-counters(7)``:
+
+  * We have added a number of counters to the ``VBE.*`` group to help
+    better diagnose error conditions with backends:
+
+    * ``VBE.*.unhealthy``: the number of fetches that were not
+      attempted because the backend was unhealthy
+
+    * ``.busy`` number of fetches that were not attempted because the
+      ``.max_connections`` limit was reached
+
+    * ``.fail``: number of failed attempts to open a connection to the
+      backend. Detailed reasons for the failures are given in the
+      ``.fail_*`` counters (shown at DIAG level), and in the log entry
+      ``Debug``. ``.fail`` is the sum of the values in the ``.fail_*``
+      counters.
+
+    * ``.fail_eaccess``, ``.fail_eaddrnotavail``,
+      ``.fail_econnrefused``, ``.fail_enetunreach`` and
+      ``.fail_etimedout``: these are the number of attempted
+      connections to the backend that failed with the given value of
+      ``errno(3)``.
+
+    * ``.fail_other``: number of connections to the backend that
+      failed for reasons other than those given by the other
+      ``.fail_*`` counters.
+
+    * ``.helddown``: the number of connections not attempted because
+      the backend was in the period set by one of the parameters
+      :ref:`ref_param_backend_local_error_holddown` or
+      :ref:`ref_param_backend_remote_error_holddown`
+
+  * In curses mode, the information in the header lines (uptimes and
+    cache hit rates) is always reported, even if you have defined a
+    filter that leaves them out of the stats table.
+
+  * Ban statistics are now reported more accurately (they had been
+    subject to inconsistencies due to race conditions).
+
+* ``varnishtest(1)`` and ``vtc(7)``:
+
+  * ``varnishtest`` and the ``vtc`` test script language now supports
+    testing for haproxy as well as Varnish. The ``haproxy`` directive
+    in a test can be used to define, configure, start and stop a
+    haproxy instance, and you can also script messages to send on the
+    haproxy CLI connection, and define expectations for the
+    responses. See the ``haproxy`` section in :ref:`vtc(7)` for
+    details.
+
+  * Related to haproxy support, you can now define a ``syslog``
+    instance in test scripts. This defines a syslog server, and allows
+    you to test expectations for syslog output from a haproxy
+    instance.
+
+  * Added the ``-keepalive`` argument for client and server scripts to
+    be used with the ``-repeat`` directive, which causes all test
+    iterations to run on the same connection, rather than open a new
+    connection each time. This makes the test run faster and use fewer
+    ephemeral ports.
 
   * **XXX**
 
 * Changes for developers:
 
-  * **XXX**
+  * The Varnish API soname version (for libvarnishapi.so) has been
+    bumped to 2.0.0.
 
-  * **XXX**
+  * We have improved support for the ``STRANDS`` data type, which you
+    may find easier to use than the varargs-based ``STRING_LIST``. See
+    ``vrt.h`` for details.  :ref:`vmod_blob(3)` has been refactored to
+    use ``STRANDS``, so you can look there for an example.
+
+  * We have fixed a bug that had limited the precision available for
+    the ``INT`` data type, so you now get the full 64 bits.
+
+  * Python 3 is now preferred in builds, and will likely be required
+    in future versions.
 
 *eof*

From geoff at uplex.de  Sun Sep  9 22:21:06 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Sun,  9 Sep 2018 22:21:06 +0000 (UTC)
Subject: [master] ca07c002f Add more release docs for 2018-09 to 'Upgrading'.
Message-ID: <20180909222106.ED00FAEB44@lists.varnish-cache.org>


commit ca07c002f4aa8c7e7436c230b0544724eab8f239
Author: Geoff Simmons <geoff at uplex.de>
Date:   Mon Sep 10 00:19:59 2018 +0200

    Add more release docs for 2018-09 to 'Upgrading'.

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index f91e6a188..7d3c06168 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -87,14 +87,36 @@ The documentation in :ref:`users-guide-handling_misbehaving_servers`
 has been expanded to discuss these matters in greater depth, look
 there for more details.
 
-``var2``
---------
-
-**XXX**
+``beresp.filters`` and support for backend response processing with VMODs
+-------------------------------------------------------------------------
+
+The ``beresp.filters`` variable is readable and writable in
+``vcl_backend_response``. This is a space-separated list of modules
+that we call VFPs, for "Varnish fetch processors", that may be applied
+to a backend response body as it is being fetched. In default Varnish,
+the list may include values such as ``gzip``, ``gunzip``, ``esi`` and
+``stream``, depending on how you have set the ``beresp.do_*``
+variables.
+
+This addition makes it possible for VMODs to define VFPs to filter or
+manipulate backend response bodies, which can be added by changing the
+list in ``beresp.filters``. VFPs are applied in the order given in
+``beresp.filters``, and you may have to ensure that a VFP is
+positioned correctly in the list, for example if it can only apply to
+uncompressed response bodies.
+
+This is a new capability, and at the time of release we only know of
+test VFPs implemented in VMODs. Over time we hope that an "ecology" of
+VFP code will develop that will enrich the features available to
+Varnish deployments.
 
 Other changes
 ~~~~~~~~~~~~~
 
+The ``Host`` header is mandatory for HTTP/1.1, as proscribed by the
+HTTP standard. If it is missing, then ``builtin.vcl`` causes a
+synthetic 400 "Bad request" response to be returned.
+
 You can now provide a string argument to ``return(fail("Foo!"))``,
 which can be used in ``vcl_init`` to emit an error message if the VCL
 load fails due to the return.
@@ -153,6 +175,10 @@ Other changes
 
     * ``ping -j``
 
+    * ``backend.list -j``
+
+    * ``help -j``
+
     * **XXX...**
 
     A JSON response in the CLI always includes a timestamp (epoch time in
@@ -190,6 +216,26 @@ Other changes
       :ref:`ref_param_backend_local_error_holddown` or
       :ref:`ref_param_backend_remote_error_holddown`
 
+  * Similarly, we have added a series of counters for better diagnostics
+    of session accept failures (failure to accept a connection from a
+    client). As before, the ``sess_fail`` counter gives the total number
+    of accept failures, and it is now augmented with the ``sess_fail_*``
+    counters. ``sess_fail`` is the sum of the values in ``sess_fail_*``.
+
+    * ``sess_fail_econnaborted``, ``sess_fail_eintr``,
+      ``sess_fail_emfile``, ``sess_fail_ebadf`` and
+      ``sess_fail_enomem``: the number of accept failures with the
+      indicated value of ``errno(3)``. The :ref:`varnish-counters(7)`
+      man page, and the "long descriptions" shown by ``varnishstat``,
+      give possible reasons why each of these may happen, and what
+      might be done to counter the problem.
+
+    * ``sess_fail_other``: number of accept failures for reasons
+      other than those given by the other ``sess_fail_*`` counters.
+      More details may appear in the ``Debug`` entry of the log
+      (:ref:`varnish-counters(7)` shows a ``varnishlog`` invocation
+      that may help).
+
   * In curses mode, the information in the header lines (uptimes and
     cache hit rates) is always reported, even if you have defined a
     filter that leaves them out of the stats table.
@@ -220,11 +266,34 @@ Other changes
 
   * **XXX**
 
+* For all of the utilities that access the Varnish log --
+  ``varnishlog(1)``, ``varnishncsa(1)``, ``varnishtop(1)`` and
+  ``varnishhist(1)`` -- it is now possible to set multiple ``-I`` and
+  ``-X`` command-line arguments.  So you can use multiple include and
+  exclude filters that apply regular expressions to selected log
+  messages.
+
 * Changes for developers:
 
+  * As mentioned above, VMODs can now implement VFPs that can be added
+    to backend response processing by changing ``beresp.filters``.
+    The interface for VFPs is defined in ``cache_filters.h``, and the
+    debug VMOD included in the distribution shows an example of a
+    VFP for rot13.
+
   * The Varnish API soname version (for libvarnishapi.so) has been
     bumped to 2.0.0.
 
+  * When ``PRIV_TASK`` and ``PRIV_TOP`` parameters are defined for a
+    VMOD method or function, space for the ``struct vrt_priv`` object
+    is allocated on the appropriate workspace before invocation -- the
+    task workspace (client or backend) for ``PRIV_TASK``, and the
+    client workspace for ``PRIV_TOP``. So it is no longer necessary
+    for the VMOD code to do the allocation. The address of the
+    allocated object is passed into the invocation. If the address is
+    NULL, then allocation failed due to workspace exhaustion (so your
+    VMOD should check for that).
+
   * We have improved support for the ``STRANDS`` data type, which you
     may find easier to use than the varargs-based ``STRING_LIST``. See
     ``vrt.h`` for details.  :ref:`vmod_blob(3)` has been refactored to

From geoff at uplex.de  Mon Sep 10 06:27:16 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Mon, 10 Sep 2018 06:27:16 +0000 (UTC)
Subject: [master] 03ba4e4fb Add more release notes for 2018-09 to 'Upgrading'.
Message-ID: <20180910062716.76F0561EBE@lists.varnish-cache.org>


commit 03ba4e4fbef6334b4db64618889955a36986523c
Author: Geoff Simmons <geoff at uplex.de>
Date:   Mon Sep 10 08:26:20 2018 +0200

    Add more release notes for 2018-09 to 'Upgrading'.

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 7d3c06168..2b09166d9 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -156,7 +156,15 @@ Other changes
 
   * **XXX**
 
-* ``varnishlog(1)``:
+* ``varnishlog(1)`` and ``vsl(7)``:
+
+  * The contents of ``FetchError`` log entries have been improved to
+    give better human-readable diagnostics for certain classes of
+    backend fetch failures.
+
+  * ``Debug`` log entries may also give more diagnostic information
+    about backend fetch failures and session accept failures (failure
+    to accept a client connection).
 
   * When a backend is unhealthy, ``Backend_health`` now reports some
     diagnostic information in addition to the HTTP response and timing
@@ -166,7 +174,9 @@ Other changes
     name without the VCL prefix (as appears otherwise for backend
     naming).
 
-  * **XXX**
+  * Added the log entry tag ``Filters``, which gives a list of the
+    filters applied to a response body (see ``beresp.filters``
+    discussed above).
 
 * ``varnishadm(1)`` and ``varnish-cli(7)``
 
@@ -184,6 +194,11 @@ Other changes
     A JSON response in the CLI always includes a timestamp (epoch time in
     seconds with millisecond precision).
 
+  * The ``backend.list`` command now lists both directors and
+    backends, with their health status. The command now has a ``-v``
+    option for verbose output, in which detailed health states for
+    each backend/director are displayed.
+
 * ``varnishstat(1)`` and ``varnish-counters(7)``:
 
   * We have added a number of counters to the ``VBE.*`` group to help
@@ -264,8 +279,16 @@ Other changes
     connection each time. This makes the test run faster and use fewer
     ephemeral ports.
 
+  * Added the ``-need-bytes`` argument for the ``process`` command,
+    see :ref:`vtc(7)`.
+
   * **XXX**
 
+* ``varnishhist(1)``:
+
+  * The ``-P min:max`` command-line parameters are now optional,
+    see :ref:`varnishhist(1)`.
+
 * For all of the utilities that access the Varnish log --
   ``varnishlog(1)``, ``varnishncsa(1)``, ``varnishtop(1)`` and
   ``varnishhist(1)`` -- it is now possible to set multiple ``-I`` and
@@ -302,6 +325,12 @@ Other changes
   * We have fixed a bug that had limited the precision available for
     the ``INT`` data type, so you now get the full 64 bits.
 
+  * Portions of what had previously been declared in
+    ``cache_director.h`` have been moved into ``vrt.h``, constituting
+    the public API for directors. The remainder in
+    ``cache_director.h`` is not public, and should not be used by a
+    VMOD intended for VRT ABI compatibility.
+
   * Python 3 is now preferred in builds, and will likely be required
     in future versions.
 

From phk at FreeBSD.org  Mon Sep 10 07:05:08 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 10 Sep 2018 07:05:08 +0000 (UTC)
Subject: [master] 9e8eecbb6 Optimize updating the index when we delete
 segments.
Message-ID: <20180910070508.A01B962A5C@lists.varnish-cache.org>


commit 9e8eecbb66887c938128d713f9e2c6a6b0a22de4
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 10 06:58:04 2018 +0000

    Optimize updating the index when we delete segments.
    
    Helps: #2764

diff --git a/bin/varnishd/common/common_vsmw.c b/bin/varnishd/common/common_vsmw.c
index 6d4545fc2..339394b58 100644
--- a/bin/varnishd/common/common_vsmw.c
+++ b/bin/varnishd/common/common_vsmw.c
@@ -127,22 +127,17 @@ vsmw_idx_head(const struct vsmw *vsmw, int fd)
 }
 
 static void
-vsmw_write_index(const struct vsmw *vsmw, int fd, const struct vsmwseg *seg)
+vsmw_fmt_index(const struct vsmw *vsmw, const struct vsmwseg *seg)
 {
-	ssize_t s;
 
 	CHECK_OBJ_NOTNULL(vsmw, VSMW_MAGIC);
 	CHECK_OBJ_NOTNULL(seg, VSMWSEG_MAGIC);
-	VSB_clear(vsmw->vsb);
 	VSB_printf(vsmw->vsb, "%s %zu %zu %s %s\n",
 	    seg->cluster->fn,
 	    seg->off,
 	    seg->len,
 	    seg->class,
 	    seg->id);
-	AZ(VSB_finish(vsmw->vsb));
-	s = write(fd, VSB_data(vsmw->vsb), VSB_len(vsmw->vsb));
-	assert(s == VSB_len(vsmw->vsb));
 }
 
 /*--------------------------------------------------------------------*/
@@ -174,11 +169,16 @@ static void
 vsmw_addseg(struct vsmw *vsmw, struct vsmwseg *seg)
 {
 	int fd;
+	ssize_t s;
 
 	VTAILQ_INSERT_TAIL(&vsmw->segs, seg, list);
 	fd = openat(vsmw->vdirfd, vsmw->idx, O_APPEND | O_WRONLY);
 	assert(fd >= 0);
-	vsmw_write_index(vsmw, fd, seg);
+	VSB_clear(vsmw->vsb);
+	vsmw_fmt_index(vsmw, seg);
+	AZ(VSB_finish(vsmw->vsb));
+	s = write(fd, VSB_data(vsmw->vsb), VSB_len(vsmw->vsb));
+	assert(s == VSB_len(vsmw->vsb));
 	AZ(close(fd));
 }
 
@@ -188,6 +188,7 @@ static void
 vsmw_delseg(struct vsmw *vsmw, struct vsmwseg *seg, int fixidx)
 {
 	char *t = NULL;
+	ssize_t s;
 	int fd;
 
 	CHECK_OBJ_NOTNULL(vsmw, VSMW_MAGIC);
@@ -206,8 +207,12 @@ vsmw_delseg(struct vsmw *vsmw, struct vsmwseg *seg, int fixidx)
 		    t, O_WRONLY|O_CREAT|O_EXCL, vsmw->mode);
 		assert(fd >= 0);
 		vsmw_idx_head(vsmw, fd);
+		VSB_clear(vsmw->vsb);
 		VTAILQ_FOREACH(seg, &vsmw->segs, list)
-			vsmw_write_index(vsmw, fd, seg);
+			vsmw_fmt_index(vsmw, seg);
+		AZ(VSB_finish(vsmw->vsb));
+		s = write(fd, VSB_data(vsmw->vsb), VSB_len(vsmw->vsb));
+		assert(s == VSB_len(vsmw->vsb));
 		AZ(close(fd));
 		AZ(renameat(vsmw->vdirfd, t, vsmw->vdirfd, vsmw->idx));
 		REPLACE(t, NULL);

From geoff at uplex.de  Mon Sep 10 07:09:07 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Mon, 10 Sep 2018 07:09:07 +0000 (UTC)
Subject: [master] 6afbe9062 Release doc fixes.
Message-ID: <20180910070907.1553662D33@lists.varnish-cache.org>


commit 6afbe90629f1eb4afd3cb8cab6c78289c8eaa239
Author: Geoff Simmons <geoff at uplex.de>
Date:   Mon Sep 10 09:07:45 2018 +0200

    Release doc fixes.

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 2b09166d9..af02d291e 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -163,8 +163,9 @@ Other changes
     backend fetch failures.
 
   * ``Debug`` log entries may also give more diagnostic information
-    about backend fetch failures and session accept failures (failure
-    to accept a client connection).
+    about session accept failures (failure to accept a client
+    connection). These must be viewed in raw grouping, since accept
+    failures are not part of any request/response transaction.
 
   * When a backend is unhealthy, ``Backend_health`` now reports some
     diagnostic information in addition to the HTTP response and timing
@@ -213,8 +214,8 @@ Other changes
     * ``.fail``: number of failed attempts to open a connection to the
       backend. Detailed reasons for the failures are given in the
       ``.fail_*`` counters (shown at DIAG level), and in the log entry
-      ``Debug``. ``.fail`` is the sum of the values in the ``.fail_*``
-      counters.
+      ``FetchError``. ``.fail`` is the sum of the values in the
+      ``.fail_*`` counters.
 
     * ``.fail_eaccess``, ``.fail_eaddrnotavail``,
       ``.fail_econnrefused``, ``.fail_enetunreach`` and

From geoff at uplex.de  Mon Sep 10 07:17:08 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Mon, 10 Sep 2018 07:17:08 +0000 (UTC)
Subject: [master] 5778fc7e5 Move an item about the 2018-08 release from
 'Changes' to 'Upgrading'.
Message-ID: <20180910071708.129F0630D5@lists.varnish-cache.org>


commit 5778fc7e5f89e0da05c57e0d5ebd87e53caacaf8
Author: Geoff Simmons <geoff at uplex.de>
Date:   Mon Sep 10 09:13:38 2018 +0200

    Move an item about the 2018-08 release from 'Changes' to 'Upgrading'.
    
    We have settled on using 'Upgrading' to list changes like this, while
    'Changes' is a free-form prose overview ("about this release").

diff --git a/doc/sphinx/whats-new/changes-6.1.rst b/doc/sphinx/whats-new/changes-6.1.rst
index 91d4b8491..ee3644694 100644
--- a/doc/sphinx/whats-new/changes-6.1.rst
+++ b/doc/sphinx/whats-new/changes-6.1.rst
@@ -10,8 +10,4 @@ Changes in Varnish 6.1
 
 **XXX**
 
-Varnish now won't rewrite the content-length header when responding to any HEAD
-request, making it possible to cache HEAD requests independently from the GET
-ones (peviously a HEAD request had to be a pass to avoid this rewriting).
-
 *eof*
diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index af02d291e..bbf1420a1 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -117,6 +117,11 @@ The ``Host`` header is mandatory for HTTP/1.1, as proscribed by the
 HTTP standard. If it is missing, then ``builtin.vcl`` causes a
 synthetic 400 "Bad request" response to be returned.
 
+Varnish now won't rewrite the content-length header when responding to
+any HEAD request, making it possible to cache responses to HEAD
+requests independently from the GET responses (previously a HEAD
+request had to be a pass to avoid this rewriting).
+
 You can now provide a string argument to ``return(fail("Foo!"))``,
 which can be used in ``vcl_init`` to emit an error message if the VCL
 load fails due to the return.

From geoff at uplex.de  Mon Sep 10 07:43:13 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Mon, 10 Sep 2018 07:43:13 +0000 (UTC)
Subject: [master] 7d7ff24a0 Elaborate a bit on the rationale for -p max_vcl in
 the release docs.
Message-ID: <20180910074313.9DDE0639A8@lists.varnish-cache.org>


commit 7d7ff24a05d336b285b8234a0099d9c7e0ac7b6b
Author: Geoff Simmons <geoff at uplex.de>
Date:   Mon Sep 10 09:42:00 2018 +0200

    Elaborate a bit on the rationale for -p max_vcl in the release docs.

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index bbf1420a1..8b67d2688 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -22,11 +22,18 @@ varnishd parameters
 We have added the :ref:`ref_param_max_vcl` parameter to set a
 threshold for the number of loaded VCL programs, since it is a common
 error to let previous VCL instances accumulate without discarding
-them. The default threshold is 100, and VCL labels are not counted
-against the total. The :ref:`ref_param_max_vcl_handling` parameter
-controls what happens when you reach the limit. By default you just
-get a warning from the VCL compiler, but you can set it to refuse to
-load more VCLs, or to ignore the threshold.
+them. The remnants of undiscarded VCLs take the form of files in the
+working directory of the management process. Over time, too many of
+these may take up significant storage space, and administrative
+operations such as ``vcl.list`` may become noticeably slow, or even
+time out, when Varnish has to iterate over many files.
+
+The default threshold in :ref:`ref_param_max_vcl` is 100, and VCL
+labels are not counted against the total. The
+:ref:`ref_param_max_vcl_handling` parameter controls what happens when
+you reach the limit. By default you just get a warning from the VCL
+compiler, but you can set it to refuse to load more VCLs, or to ignore
+the threshold.
 
 Added the :ref:`ref_param_backend_local_error_holddown` and
 :ref:`ref_param_backend_remote_error_holddown` parameters. These define

From geoff at uplex.de  Mon Sep 10 08:26:10 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Mon, 10 Sep 2018 08:26:10 +0000 (UTC)
Subject: [master] 90868639a Some editorial polish in the 2018-09 release notes.
Message-ID: <20180910082611.07152646EB@lists.varnish-cache.org>


commit 90868639a1ecc5e13541007c3874a040ae485055
Author: Geoff Simmons <geoff at uplex.de>
Date:   Mon Sep 10 10:25:22 2018 +0200

    Some editorial polish in the 2018-09 release notes.

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index f1d8bdae6..997746f12 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -78,7 +78,9 @@ to relieve load on the troubled backend::
 		set req.ttl = 60s;
 		set req.grace = 10s;
 	}
-	# ...
+
+	# If the backend is unhealthy, then permit cached responses
+	# that are older than 70s.
   }
 
 The evaluation of the ``beresp.keep`` timer has changed a
@@ -120,9 +122,10 @@ Varnish deployments.
 Other changes to VCL
 ~~~~~~~~~~~~~~~~~~~~
 
-* The ``Host`` header is mandatory for HTTP/1.1, as proscribed by the
-  HTTP standard. If it is missing, then ``builtin.vcl`` causes a
-  synthetic 400 "Bad request" response to be returned.
+* The ``Host`` header in client requests is mandatory for HTTP/1.1, as
+  proscribed by the HTTP standard. If it is missing, then
+  ``builtin.vcl`` causes a synthetic 400 "Bad request" response to be
+  returned.
 
 * You can now provide a string argument to ``return(fail("Foo!"))``,
   which can be used in ``vcl_init`` to emit an error message if the
@@ -176,7 +179,8 @@ Other changes
   * ``Debug`` log entries may also give more diagnostic information
     about session accept failures (failure to accept a client
     connection). These must be viewed in raw grouping, since accept
-    failures are not part of any request/response transaction.
+    failures are not part of any request/response transaction. The
+    ``Debug`` message begins with the phrase ``"Accept failed"``.
 
   * When a backend is unhealthy, ``Backend_health`` now reports some
     diagnostic information in addition to the HTTP response and timing
@@ -219,7 +223,7 @@ Other changes
     * ``VBE.*.unhealthy``: the number of fetches that were not
       attempted because the backend was unhealthy
 
-    * ``.busy`` number of fetches that were not attempted because the
+    * ``.busy``: number of fetches that were not attempted because the
       ``.max_connections`` limit was reached
 
     * ``.fail``: number of failed attempts to open a connection to the
@@ -325,9 +329,10 @@ Other changes
     task workspace (client or backend) for ``PRIV_TASK``, and the
     client workspace for ``PRIV_TOP``. So it is no longer necessary
     for the VMOD code to do the allocation. The address of the
-    allocated object is passed into the invocation. If the address is
-    NULL, then allocation failed due to workspace exhaustion (so your
-    VMOD should check for that).
+    allocated object is passed as the parameter to your implementation
+    of the method or function. If the address is NULL, then allocation
+    failed, probably due to workspace exhaustion (so your VMOD should
+    check for that).
 
   * We have improved support for the ``STRANDS`` data type, which you
     may find easier to use than the varargs-based ``STRING_LIST``. See

From geoff at uplex.de  Mon Sep 10 08:26:10 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Mon, 10 Sep 2018 08:26:10 +0000 (UTC)
Subject: [master] b1b826de1 Move some "misc" items in the release docs from
 VCL to varnishd changes.
Message-ID: <20180910082610.B9FBB646B2@lists.varnish-cache.org>


commit b1b826de1248eb666dc99eba12006f9feaa56057
Author: Geoff Simmons <geoff at uplex.de>
Date:   Mon Sep 10 09:57:12 2018 +0200

    Move some "misc" items in the release docs from VCL to varnishd changes.

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 8b67d2688..f1d8bdae6 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -117,29 +117,16 @@ test VFPs implemented in VMODs. Over time we hope that an "ecology" of
 VFP code will develop that will enrich the features available to
 Varnish deployments.
 
-Other changes
-~~~~~~~~~~~~~
-
-The ``Host`` header is mandatory for HTTP/1.1, as proscribed by the
-HTTP standard. If it is missing, then ``builtin.vcl`` causes a
-synthetic 400 "Bad request" response to be returned.
+Other changes to VCL
+~~~~~~~~~~~~~~~~~~~~
 
-Varnish now won't rewrite the content-length header when responding to
-any HEAD request, making it possible to cache responses to HEAD
-requests independently from the GET responses (previously a HEAD
-request had to be a pass to avoid this rewriting).
+* The ``Host`` header is mandatory for HTTP/1.1, as proscribed by the
+  HTTP standard. If it is missing, then ``builtin.vcl`` causes a
+  synthetic 400 "Bad request" response to be returned.
 
-You can now provide a string argument to ``return(fail("Foo!"))``,
-which can be used in ``vcl_init`` to emit an error message if the VCL
-load fails due to the return.
-
-If you have set ``.proxy_header=1`` (to use the PROXYv1 protocol) for
-a backend addressed as a Unix domain socket (with a ``.path`` setting
-for the socket file), and have also defined a probe for the backend,
-then then the address family ``UNKNOWN`` is sent in the proxy header
-for the probe request. If you have set ``.proxy_header=2`` (for
-PROXYv2) for a UDS backend with a probe, then ``PROXY LOCAL`` is sent
-for the probe request.
+* You can now provide a string argument to ``return(fail("Foo!"))``,
+  which can be used in ``vcl_init`` to emit an error message if the
+  VCL load fails due to the return.
 
 VMODs
 =====
@@ -166,7 +153,19 @@ Other changes
     example when a symbol is not found or arguments to VMOD calls are
     missing.
 
-  * **XXX**
+  * Varnish now won't rewrite the ``Content-Length`` header when
+    responding to any HEAD request, making it possible to cache
+    responses to HEAD requests independently from the GET responses
+    (previously a HEAD request had to be a pass to avoid this
+    rewriting).
+
+  * If you have set ``.proxy_header=1`` (to use the PROXYv1 protocol)
+    for a backend addressed as a Unix domain socket (with a ``.path``
+    setting for the socket file), and have also defined a probe for
+    the backend, then then the address family ``UNKNOWN`` is sent in
+    the proxy header for the probe request. If you have set
+    ``.proxy_header=2`` (for PROXYv2) for a UDS backend with a probe,
+    then ``PROXY LOCAL`` is sent for the probe request.
 
 * ``varnishlog(1)`` and ``vsl(7)``:
 

From phk at FreeBSD.org  Mon Sep 10 11:40:15 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 10 Sep 2018 11:40:15 +0000 (UTC)
Subject: [master] 7a0f2cf7e Back down a little bit on UTF-8 in .vcc files.
Message-ID: <20180910114016.15D2794366@lists.varnish-cache.org>


commit 7a0f2cf7e208035090b5f8ddb98a91e3681d11f6
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 10 11:38:52 2018 +0000

    Back down a little bit on UTF-8 in .vcc files.
    
    Not ready to tackle the Python2/Python3 monster a week before release.

diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py
index 7a4e362fc..25b8cc621 100755
--- a/lib/libvcc/vmodtool.py
+++ b/lib/libvcc/vmodtool.py
@@ -858,8 +858,9 @@ class vcc(object):
 
     def parse(self):
         global inputline
-        a = "\n" + open(self.inputfile, "r").read()
-        self.file_id = hashlib.sha256(bytearray(a)).hexdigest()
+        b = open(self.inputfile, "rb").read()
+        a = "\n" + b.decode("utf-8")
+        self.file_id = hashlib.sha256(b).hexdigest()
         s = a.split("\n$")
         self.copyright = s.pop(0).strip()
         while s:
diff --git a/lib/libvmod_debug/vmod.vcc b/lib/libvmod_debug/vmod.vcc
index 200b69de2..fecf9b150 100644
--- a/lib/libvmod_debug/vmod.vcc
+++ b/lib/libvmod_debug/vmod.vcc
@@ -25,7 +25,6 @@
 # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
 # SUCH DAMAGE.
 #
-# Test vmodtool.py's UTF-8 handling: ?blefl?sk med koldsk?l eller ?l
 
 $Module debug 3 Development, test and debug
 $ABI strict

From nils.goroll at uplex.de  Mon Sep 10 17:37:11 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon, 10 Sep 2018 17:37:11 +0000 (UTC)
Subject: [master] 5e2804f26 release docs: req.grace/req.ttl wording
Message-ID: <20180910173711.D81959CED9@lists.varnish-cache.org>


commit 5e2804f26402fa901bc13825bd5fcdc98a235796
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Sep 10 19:36:08 2018 +0200

    release docs: req.grace/req.ttl wording

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 997746f12..df71404f1 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -56,15 +56,15 @@ VCL variables
 ``req.ttl``, ``req.grace`` and keep
 -----------------------------------
 
-``req.ttl`` had been previously listed as deprecated, but it is now
-fully supported, since there are use cases that cannot be solved
-without it.
+``req.grace`` had been previously removed, but was now reintroduced,
+since there are use cases that cannot be solved without it. Similarly,
+``req.ttl`` used to be deprecated and is now fully supported again.
 
 ``req.ttl`` and ``req.grace`` set upper bounds on the TTL and grace
 times that are permitted for the current request -- if these variables
 are set and the TTL/grace of a cache object is longer than their
 settings, then a new response is fetched from the backend, despite the
-presence of the response in the cache.
+presence of an otherwise fresh cache object.
 
 A common application is to set shorter TTLs when the backend is known
 to be healthy, so that responses are fresher when all is well. But if

From nils.goroll at uplex.de  Mon Sep 10 18:11:07 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon, 10 Sep 2018 18:11:07 +0000 (UTC)
Subject: [master] e7ab96a9a add HTC errors to the documentation
Message-ID: <20180910181107.1702D9D999@lists.varnish-cache.org>


commit e7ab96a9a4d3bc8e6a5f3b492ecee0d0ce0d3b7e
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Sep 10 20:10:25 2018 +0200

    add HTC errors to the documentation

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 428a764ae..743821b7a 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -179,6 +179,9 @@ Other changes
     give better human-readable diagnostics for certain classes of
     backend fetch failures.
 
+    In particular, http connection (HTC) errors are now reported
+    symbolically in addition to the previous numerical value.
+
   * ``Debug`` log entries may also give more diagnostic information
     about session accept failures (failure to accept a client
     connection). These must be viewed in raw grouping, since accept
diff --git a/include/tbl/htc.h b/include/tbl/htc.h
index a46b75e28..a93c0e520 100644
--- a/include/tbl/htc.h
+++ b/include/tbl/htc.h
@@ -35,11 +35,11 @@
 HTC_STATUS(JUNK,	-5,	"junk",		"Received unexpected data")
 HTC_STATUS(CLOSE,	-4,	"close",	"Connection closed") // unused?
 HTC_STATUS(TIMEOUT,	-3,	"timeout",	"Timed out")
-HTC_STATUS(OVERFLOW,	-2,	"overflow",	"Buffer too small")
-HTC_STATUS(EOF,	-1,	"eof",		"EOF received")
+HTC_STATUS(OVERFLOW,	-2,	"overflow",	"Buffer/workspace too small")
+HTC_STATUS(EOF,	-1,	"eof",		"Unexpected end of input")
 HTC_STATUS(EMPTY,	 0,	"empty",	"Empty response")
 HTC_STATUS(MORE,	 1,	"more",	"More data required")
-HTC_STATUS(COMPLETE,	 2,	"complete",	"Data complete")
-HTC_STATUS(IDLE,	 3,	"idle",	"Return to waiter")
+HTC_STATUS(COMPLETE,	 2,	"complete",	"Data complete (no error)")
+HTC_STATUS(IDLE,	 3,	"idle",	"Connection was closed while idle")
 #undef HTC_STATUS
 /*lint -restore */
diff --git a/include/tbl/vsl_tags.h b/include/tbl/vsl_tags.h
index e56b7c85f..916fc4d79 100644
--- a/include/tbl/vsl_tags.h
+++ b/include/tbl/vsl_tags.h
@@ -165,9 +165,32 @@ SLTM(Length, 0, "Size of object body",
 	"Logs the size of a fetch object body.\n\n"
 )
 
+/* XXX generate HTC info from tbl include */
+#if 0
+#include <stdio.h>
+int main(void) {
+#define HTC_STATUS(e, n, s, l) \
+	printf("\t\"\\t* %s (%d): %s\\n\"\n", s, n, l);
+#include "include/tbl/htc.h"
+	return (0);
+}
+#endif
+
 SLTM(FetchError, 0, "Error while fetching object",
 	"Logs the error message of a failed fetch operation.\n\n"
-)
+	"Error messages should be self-explanatory, yet the http connection"
+	"(HTC) class of errors is reported with these symbols:\n\n"
+	"\t* junk (-5): Received unexpected data\n"
+	"\t* close (-4): Connection closed\n"
+	"\t* timeout (-3): Timed out\n"
+	"\t* overflow (-2): Buffer/workspace too small\n"
+	"\t* eof (-1): Unexpected end of input\n"
+	"\t* empty (0): Empty response\n"
+	"\t* more (1): More data required\n"
+	"\t* complete (2): Data complete (no error)\n"
+	"\t* idle (3): Connection was closed while idle\n"
+	"\nNotice that some HTC errors are never emitted."
+	)
 
 #define SLTH(tag, ind, req, resp, sdesc, ldesc) \
 	SLTM(Req##tag, (req ? 0 : SLT_F_UNUSED), "Client request " sdesc, ldesc)

From nils.goroll at uplex.de  Mon Sep 10 18:11:06 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon, 10 Sep 2018 18:11:06 +0000 (UTC)
Subject: [master] 32dbbfbdb release docs: better motivation for std.fnmatch()
Message-ID: <20180910181107.05C859D997@lists.varnish-cache.org>


commit 32dbbfbdb5447380de9075214a8b795944c11e79
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Sep 10 19:43:10 2018 +0200

    release docs: better motivation for std.fnmatch()

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index df71404f1..428a764ae 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -135,8 +135,11 @@ VMODs
 =====
 
 Added the :ref:`func_fnmatch` function to :ref:`vmod_std(3)`, which
-you can use for shell-style wildcard matching (if you prefer that to
-regular expressions).
+you can use for shell-style wildcard matching. :ref:`func_fnmatch` can
+be more efficient than regular expressions, in particular in cases
+where the pattern is dynamically defined.
+
+.. should we mention vmod_re for dynamic regexen?
 
 :ref:`vmod_unix(3)` is now supported for SunOS and descendants. This
 entails changing the privilege set of the child process while the VMOD

From dridi at varni.sh  Tue Sep 11 06:49:36 2018
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 11 Sep 2018 08:49:36 +0200
Subject: [master] ca07c002f Add more release docs for 2018-09 to
 'Upgrading'.
In-Reply-To: <20180909222106.ED00FAEB44@lists.varnish-cache.org>
References: <20180909222106.ED00FAEB44@lists.varnish-cache.org>
Message-ID: <CABoVN9D_7D-uoYth1NMmwKPAqiDPjV_oKhVYm13rrdubOYEOwQ@mail.gmail.com>

> +* For all of the utilities that access the Varnish log --
> +  ``varnishlog(1)``, ``varnishncsa(1)``, ``varnishtop(1)`` and
> +  ``varnishhist(1)`` -- it is now possible to set multiple ``-I`` and
> +  ``-X`` command-line arguments.  So you can use multiple include and
> +  exclude filters that apply regular expressions to selected log
> +  messages.
> +

This sounds both familiar and wrong. Wasn't it already the case, but
not documented as such?

Found it: 9938f01fe8a2b148013423c310c221273b8297b4

It was only a docfix!

> +  * When ``PRIV_TASK`` and ``PRIV_TOP`` parameters are defined for a
> +    VMOD method or function, space for the ``struct vrt_priv`` object
> +    is allocated on the appropriate workspace before invocation -- the
> +    task workspace (client or backend) for ``PRIV_TASK``, and the
> +    client workspace for ``PRIV_TOP``. So it is no longer necessary
> +    for the VMOD code to do the allocation. The address of the
> +    allocated object is passed into the invocation. If the address is
> +    NULL, then allocation failed due to workspace exhaustion (so your
> +    VMOD should check for that).
> +

I think this is done transparently and the API didn't change. Maybe needs
double check from someone who knows better? I don't.

Dridi

From nils.goroll at uplex.de  Tue Sep 11 06:55:48 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Tue, 11 Sep 2018 08:55:48 +0200
Subject: [master] ca07c002f Add more release docs for 2018-09 to
 'Upgrading'.
In-Reply-To: <CABoVN9D_7D-uoYth1NMmwKPAqiDPjV_oKhVYm13rrdubOYEOwQ@mail.gmail.com>
References: <20180909222106.ED00FAEB44@lists.varnish-cache.org>
 <CABoVN9D_7D-uoYth1NMmwKPAqiDPjV_oKhVYm13rrdubOYEOwQ@mail.gmail.com>
Message-ID: <51532fa9-ea3b-9a5d-f48d-17d9cfd6e67e@uplex.de>



On 11/09/18 08:49, Dridi Boukelmoune wrote:
> > When ``PRIV_TASK`` and ``PRIV_TOP`` parameters
> I think this is done transparently and the API didn't change. Maybe needs
> double check from someone who knows better? I don't.

it's still on my list


-- 

** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstra?e 32
22301 Hamburg

tel +49 40 28805731
mob +49 170 2723133
fax +49 40 42949753

xmpp://slink at jabber.int.uplex.de/

http://uplex.de/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-commit/attachments/20180911/17a91931/attachment.bin>

From dridi.boukelmoune at gmail.com  Tue Sep 11 07:05:13 2018
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Tue, 11 Sep 2018 07:05:13 +0000 (UTC)
Subject: [master] 3f433f96c New feature isn't that new
Message-ID: <20180911070514.04C48AE9D7@lists.varnish-cache.org>


commit 3f433f96c7151a1d592eb604c662d3978f1b936d
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Tue Sep 11 09:03:26 2018 +0200

    New feature isn't that new

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 743821b7a..3c635643f 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -313,10 +313,10 @@ Other changes
 
 * For all of the utilities that access the Varnish log --
   ``varnishlog(1)``, ``varnishncsa(1)``, ``varnishtop(1)`` and
-  ``varnishhist(1)`` -- it is now possible to set multiple ``-I`` and
-  ``-X`` command-line arguments.  So you can use multiple include and
-  exclude filters that apply regular expressions to selected log
-  messages.
+  ``varnishhist(1)`` -- it was already possible to set multiple ``-I``
+  and ``-X`` command-line arguments.  It is now properly documented
+  that you can use multiple include and exclude filters that apply
+  regular expressions to selected log records.
 
 * Changes for developers:
 

From nils.goroll at uplex.de  Tue Sep 11 07:21:08 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Tue, 11 Sep 2018 07:21:08 +0000 (UTC)
Subject: [master] 780cc21b2 release docs: point to the log for .fail_other
Message-ID: <20180911072108.C0BA7AEF64@lists.varnish-cache.org>


commit 780cc21b2005551501a89b49fb1dac69070b5873
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Tue Sep 11 09:16:35 2018 +0200

    release docs: point to the log for .fail_other

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 3c635643f..3606951fb 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -246,7 +246,9 @@ Other changes
 
     * ``.fail_other``: number of connections to the backend that
       failed for reasons other than those given by the other
-      ``.fail_*`` counters.
+      ``.fail_*`` counters. For such cases, details on the failure
+      can be extracted from the varnish log as described above for
+      ``FetchError``.
 
     * ``.helddown``: the number of connections not attempted because
       the backend was in the period set by one of the parameters

From nils.goroll at uplex.de  Tue Sep 11 07:21:08 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Tue, 11 Sep 2018 07:21:08 +0000 (UTC)
Subject: [master] 2423a5784 release docs: fix PRIV_TASK/PRIV_TOP
Message-ID: <20180911072108.CB977AEF6C@lists.varnish-cache.org>


commit 2423a578478c69cc78e52c8900a5c68797e9f998
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Tue Sep 11 09:19:48 2018 +0200

    release docs: fix PRIV_TASK/PRIV_TOP

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 3606951fb..fb050ca69 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -331,16 +331,29 @@ Other changes
   * The Varnish API soname version (for libvarnishapi.so) has been
     bumped to 2.0.0.
 
-  * When ``PRIV_TASK`` and ``PRIV_TOP`` parameters are defined for a
-    VMOD method or function, space for the ``struct vrt_priv`` object
-    is allocated on the appropriate workspace before invocation -- the
-    task workspace (client or backend) for ``PRIV_TASK``, and the
-    client workspace for ``PRIV_TOP``. So it is no longer necessary
-    for the VMOD code to do the allocation. The address of the
-    allocated object is passed as the parameter to your implementation
-    of the method or function. If the address is NULL, then allocation
-    failed, probably due to workspace exhaustion (so your VMOD should
-    check for that).
+  * Space required by varnish for maintaining the ``PRIV_TASK`` and
+    ``PRIV_TOP`` parameters is now taken from the appropriate
+    workspace rather than from the heap as before. For a failing
+    allocation, a VCL failure is triggered.
+
+    The net effect of this change is that in cases of a workspace
+    shortage, the almost unavoidable failure will happen earlier. The
+    amount of workspace required is slightly increased and scales with
+    the number of vmods per ``PRIV_TASK`` and ``PRIV_TOP`` parameter.
+
+    The VCL compiler (VCC) guarantees that if a vmod function is
+    called with a ``PRIV_*`` argument, that argument value is set.
+
+    There is no change with respect to the API the ``PRIV_*`` vmod
+    function arguments provide.
+
+  * ``VRT_priv_task()``, the function implementing the allocation of
+    the ``PRIV_TASK`` and ``PRIV_TOP`` parameters as described above,
+    is now more likely to return ``NULL`` for allocation failures for
+    the same reason.
+
+    Notice that explicit use of this function from within VMODs is
+    considered experimental as this interface may change.
 
   * We have improved support for the ``STRANDS`` data type, which you
     may find easier to use than the varargs-based ``STRING_LIST``. See

From geoff at uplex.de  Tue Sep 11 07:52:09 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Tue, 11 Sep 2018 07:52:09 +0000 (UTC)
Subject: [master] e9e935506 Another shot at motivating std.fnmatch() in the
 6.1 release notes.
Message-ID: <20180911075209.D5DE46BA5@lists.varnish-cache.org>


commit e9e9355068f925afd965d4bab9aba9fcdefef77b
Author: Geoff Simmons <geoff at uplex.de>
Date:   Tue Sep 11 09:50:08 2018 +0200

    Another shot at motivating std.fnmatch() in the 6.1 release notes.
    
    IMO the release notes are not an appropriate place to point to a
    third-party VMOD.

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index fb050ca69..0daf56df7 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -135,11 +135,13 @@ VMODs
 =====
 
 Added the :ref:`func_fnmatch` function to :ref:`vmod_std(3)`, which
-you can use for shell-style wildcard matching. :ref:`func_fnmatch` can
-be more efficient than regular expressions, in particular in cases
-where the pattern is dynamically defined.
-
-.. should we mention vmod_re for dynamic regexen?
+you can use for shell-style wildcard matching. Wildcard patterns may
+be a good fit for matching URLs, to match against a pattern like
+``/foo/*/bar/*``. The patterns can be built at runtime, if you need to
+do that, since they don't need the pre-compile step at VCL load time
+that is required for regular expressions. And if you are simply more
+comfortable with the wildcard syntax than with regular expressions,
+you now have the option.
 
 :ref:`vmod_unix(3)` is now supported for SunOS and descendants. This
 entails changing the privilege set of the child process while the VMOD

From nils.goroll at uplex.de  Wed Sep 12 06:58:12 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 12 Sep 2018 06:58:12 +0000 (UTC)
Subject: [master] e8344b806 upgrading WIP
Message-ID: <20180912065813.0170CA67CB@lists.varnish-cache.org>


commit e8344b8061e6151d0bc1a990c83459ed0f7b3d36
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Sep 12 08:54:19 2018 +0200

    upgrading WIP

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 0daf56df7..a8c3ae7b4 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -131,6 +131,9 @@ Other changes to VCL
   which can be used in ``vcl_init`` to emit an error message if the
   VCL load fails due to the return.
 
+* Additional ``import`` statements of an already imported vmod are now
+  ignored.
+
 VMODs
 =====
 
@@ -371,6 +374,13 @@ Other changes
     ``cache_director.h`` is not public, and should not be used by a
     VMOD intended for VRT ABI compatibility.
 
+  * The director API in ``vrt.h`` differs from the previous
+    interface:
+
+    *WIP slink*
+
+    * The director API which has changed:
+
   * Python 3 is now preferred in builds, and will likely be required
     in future versions.
 

From nils.goroll at uplex.de  Wed Sep 12 06:58:12 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 12 Sep 2018 06:58:12 +0000 (UTC)
Subject: [master] 4a5385858 vmod optional arguments documentation (not new for
 6.1)
Message-ID: <20180912065812.E75E5A67C8@lists.varnish-cache.org>


commit 4a5385858b5d0b904de82c55e53c4e7cf6626da0
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Sep 12 08:53:04 2018 +0200

    vmod optional arguments documentation (not new for 6.1)

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index 3b63f563f..b9c0fcfc8 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -140,6 +140,44 @@ different to user specified values.
 `Note` that default values have to be given in the native C-type
 syntax, see below. As a special case, ``NULL`` has to be given as ``0``.
 
+Optional arguments
+------------------
+
+The vmod.vcc declaration also allows for optional arguments in square
+brackets like so::
+
+	$Function VOID opt(PRIV_TASK, INT four = 4, [ STRING opt])
+
+With any optional argument present, the C function prototype looks
+completely different:
+
+	* Only the ``VRT_CTX`` and object pointer arguments (only for
+	  methods) remain positional
+
+	* All other arguments get passed in a struct as the last
+	  argument of the C function.
+
+The argument struct is simple, vmod authors should check the
+`vmodtool`-generated ``vcc_if.c`` file for the function and struct
+declarations:
+
+	* for each optional argument, a ``valid_``\ `argument` member
+	  is used to signal the presence of the respective optional
+	  argument.
+
+	  ``valid_`` argstruct members should only be used as truth
+	  values, irrespective of their actual data type.
+
+	* named arguments are passed in argument struct members by the
+	  same name and with the same data type.
+
+	* unnamed (positional) arguments are passed as ``arg``\ `n`
+	  with `n` starting at 1 and incrementing with the argument's
+	  position.
+
+	  Note that in particular also ``PRIV_*`` arguments (which are
+	  unnamed by definition) are passed as ``arg``\ `n`
+
 .. _ref-vmod-vcl-c-types:
 
 VCL and C data types

From nils.goroll at uplex.de  Wed Sep 12 06:58:13 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 12 Sep 2018 06:58:13 +0000 (UTC)
Subject: [master] 3f067ba24 doc vrt.h changes
Message-ID: <20180912065813.18CE0A67D0@lists.varnish-cache.org>


commit 3f067ba242f8e60a24981254e4f27157083f9ed2
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Sep 12 08:55:27 2018 +0200

    doc vrt.h changes

diff --git a/include/vrt.h b/include/vrt.h
index 88dd5214c..9d11f7565 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -57,9 +57,17 @@
  *	VRT_StrandsWS() added
  *	VRT_CollectStrands() added
  *	VRT_STRANDS_string() removed from vrt.h (never implemented)
- *	VRT_Healthy() changed prototype
  *	VRT_Vmod_Init signature changed
  *	VRT_Vmod_Fini changed to VRT_Vmod_Unload
+ *	// directors
+ *	VRT_backend_healthy() removed
+ *	VRT_Healthy() changed prototype
+ *	struct vdi_methods and callback prototypes added
+ *	struct director added;
+ *	VRT_AddDirector() added
+ *	VRT_SetHealth() added
+ *	VRT_DisableDirector() added
+ *	VRT_DelDirector() added
  * 7.0 (2018-03-15)
  *	lots of stuff moved from cache.h to cache_varnishd.h
  *	   (ie: from "$Abi vrt" to "$Abi strict")

From geoff at uplex.de  Wed Sep 12 08:12:13 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Wed, 12 Sep 2018 08:12:13 +0000 (UTC)
Subject: [master] af7ed69c8 Mention the obj.hits fix in vcl_hit in the 2018-09
 release docs.
Message-ID: <20180912081213.957A8A7F65@lists.varnish-cache.org>


commit af7ed69c83c43f260fa065a28f65170a9f04c2e3
Author: Geoff Simmons <geoff at uplex.de>
Date:   Wed Sep 12 10:10:31 2018 +0200

    Mention the obj.hits fix in vcl_hit in the 2018-09 release docs.

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index a8c3ae7b4..47729f7d8 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -119,6 +119,12 @@ test VFPs implemented in VMODs. Over time we hope that an "ecology" of
 VFP code will develop that will enrich the features available to
 Varnish deployments.
 
+``obj.hits``
+------------
+
+Has been fixed to return the correct value in ``vcl_hit`` (it had been
+0 in ``vcl_hit``).
+
 Other changes to VCL
 ~~~~~~~~~~~~~~~~~~~~
 

From geoff at uplex.de  Wed Sep 12 08:12:13 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Wed, 12 Sep 2018 08:12:13 +0000 (UTC)
Subject: [master] 53d20a98a Update changes.rst for the 2018-09 release.
Message-ID: <20180912081213.8BDB2A7F63@lists.varnish-cache.org>


commit 53d20a98a968cdff71a24ab6de8cd64e766632ca
Author: Geoff Simmons <geoff at uplex.de>
Date:   Wed Sep 12 10:10:07 2018 +0200

    Update changes.rst for the 2018-09 release.

diff --git a/doc/changes.rst b/doc/changes.rst
index 9c398cc53..f9a1eb4c1 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -27,15 +27,93 @@ individual releases. These documents are updated as part of the
 release process.
 
 
-=============================
-Varnish Cache Trunk (ongoing)
-=============================
+================================
+Varnish Cache 6.1.0 (2018-09-17)
+================================
 
-varnishadm
+* Added -p max_vcl and -p max_vcl_handling for warnings/errors when
+  there are too many undiscarded VCL instances. (2713_)
+
+* ``Content-Length`` header is not rewritten in response to a HEAD
+  request, allows responses to HEAD requests to be cached
+  independently from GET responses.
+
+.. _2713: https://github.com/varnishcache/varnish-cache/issues/2713
+
+VCL
+---
+
+* ``return(fail("mumble"))`` can have a string argument that is
+  emitted by VCC as an error message if the VCL load fails due to the
+  return. (2694_)
+
+* Improved VCC error messages (2696_)
+
+* Fixed ``obj.hits`` in ``vcl_hit`` (had been always 0) (2746_)
+
+.. _2746: https://github.com/varnishcache/varnish-cache/issues/2746
+.. _2696: https://github.com/varnishcache/varnish-cache/issues/2696
+.. _2694: https://github.com/varnishcache/varnish-cache/issues/2694
+
+bundled tools
+-------------
+
+* ``varnishhist``: Improved test coverage
+* ``varnishtest``: Added haproxy CLI send/expect facility
+
+C APIs (for vmod and utility authors)
+-------------------------------------
+
+* libvarnishapi so version bumped to 2.0.0 (2718_)
+
+* For VMOD methods/functions with PRIV_TASK or PRIV_TOP arguments, the
+  struct vrt_priv is allocated on the appropriate workspace. In the
+  out-of-workspace condition, VCL failure is invoked, and the VMOD
+  method/function is not called. (2708_)
+
+* Improved support for the VCL STRANDS type, VMOD blob refactored to
+  use STRANDS (2745_)
+
+.. _2718: https://github.com/varnishcache/varnish-cache/pull/2718
+.. _2745: https://github.com/varnishcache/varnish-cache/issues/2745
+.. _2708: https://github.com/varnishcache/varnish-cache/issues/2708
+
+Fixed bugs
 ----------
 
-* The output format of the ``backend.list`` CLI command has been
-  changed.
+* A series of bug fixes related to excessive object accumulation and
+  Transient storage use in the hit-for-miss case (2760_, 2754_, 2654_,
+  2763_)
+* A series of fixes related to Python and the vmodtool (2761_, 2759_,
+  2742_)
+* UB in varnishhist (2773_)
+* Allow to not have randomness in file_id (2436_)
+* b64.vtc unstable (2753_)
+* VCL_Poll ctx scope (2749_)
+
+.. _2436: https://github.com/varnishcache/varnish-cache/issues/2436
+.. _2654: https://github.com/varnishcache/varnish-cache/issues/2654
+.. _2742: https://github.com/varnishcache/varnish-cache/issues/2742
+.. _2749: https://github.com/varnishcache/varnish-cache/issues/2749
+.. _2753: https://github.com/varnishcache/varnish-cache/issues/2753
+.. _2754: https://github.com/varnishcache/varnish-cache/issues/2754
+.. _2759: https://github.com/varnishcache/varnish-cache/pull/2759
+.. _2760: https://github.com/varnishcache/varnish-cache/pull/2760
+.. _2761: https://github.com/varnishcache/varnish-cache/issues/2761
+.. _2763: https://github.com/varnishcache/varnish-cache/issues/2763
+.. _2773: https://github.com/varnishcache/varnish-cache/issues/2773
+
+================================
+Varnish Cache 6.0.1 (2018-08-29)
+================================
+
+* Added std.fnmatch() (2737_)
+* The variable req.grace is back. (2705_)
+* Importing the same VMOD multiple times is now allowed, if the file_id
+  is identical.
+
+.. _2705: https://github.com/varnishcache/varnish-cache/pull/2705
+.. _2737: https://github.com/varnishcache/varnish-cache/pull/2737
 
 varnishstat
 -----------
@@ -49,7 +127,7 @@ varnishstat
   * ``sess_fail_enomem``
   * ``sess_fail_other``
 
-  now break down the detailled reason for session accept failures, the
+  now break down the detailed reason for session accept failures, the
   sum of which continues to be counted in ``sess_fail``.
 
 VCL and bundled VMODs
@@ -57,9 +135,6 @@ VCL and bundled VMODs
 
 * VMOD unix now supports the ``getpeerucred(3)`` case.
 
-* Fetch Processors (VFPs) can now be configured from VCL via
-  ``beresp.filters``
-
 bundled tools
 -------------
 
@@ -67,20 +142,80 @@ bundled tools
   for custom profile definitions to also contain a prefix to match the
   tag against.
 
-logging
--------
+* ``varnishtest``: syslog instances now have to start with a capital S.
 
-* The backend name logged under the ``Backend_health`` tag has been
-  changed back from `vcl name`.`backend name` to just `backend name`
-
-C APIs (for vmod and utility authors)
--------------------------------------
+Fixed bugs which may influence VCL behavior
+--------------------------------------------
 
-* XXX VFP
+* When an object is out of grace but in keep, the client context goes
+  straight to vcl_miss instead of vcl_hit. The documentation has been
+  updated accordingly. (2705_)
 
-* XXX backend add / del
+Fixed bugs
+----------
 
-* XXX director api, cache_backend.h off-limits, moved to VRT
+* Several H2 bugs (2285_, 2572_, 2623_, 2624_, 2679_, 2690_, 2693_)
+* Make large integers work in VCL. (2603_)
+* Print usage on unknown or missing arguments (2608_)
+* Assert error in VPX_Send_Proxy() with proxy backends in pipe mode
+  (2613_)
+* Holddown times for certain backend connection errors (2622_)
+* Enforce Host requirement for HTTP/1.1 requests (2631_)
+* Introduction of '-' CLI prefix allowed empty commands to sneak
+  through. (2647_)
+* VUT apps can be stopped cleanly via vtc process -stop (2649_, 2650_)
+* VUT apps fail gracefully when removing a PID file fails
+* varnishd startup log should mention version (2661_)
+* In curses mode, always filter in the counters necessary for the
+  header lines. (2678_)
+* Assert error in ban_lurker_getfirst() (2681_)
+* Missing command entries in varnishadm help menu (2682_)
+* Handle string literal concatenation correctly (2685_)
+* varnishtop -1 does not work as documented (2686_)
+* Handle sigbus like sigsegv (2693_)
+* Panic on return (retry) of a conditional fetch (2700_)
+* Wrong turn at cache/cache_backend_probe.c:255: Unknown family
+  (2702_, 2726_)
+* VCL failure causes TASK_PRIV reference on reset workspace (2706_)
+* Accurate ban statistics except for a few remaining corner cases
+  (2716_)
+* Assert error in vca_make_session() (2719_)
+* Assert error in vca_tcp_opt_set() (2722_)
+* VCL compiling error on parenthesis (2727_)
+* Assert error in HTC_RxPipeline() (2731_)
+
+.. _2285: https://github.com/varnishcache/varnish-cache/issues/2285
+.. _2572: https://github.com/varnishcache/varnish-cache/issues/2572
+.. _2603: https://github.com/varnishcache/varnish-cache/issues/2603
+.. _2608: https://github.com/varnishcache/varnish-cache/issues/2608
+.. _2613: https://github.com/varnishcache/varnish-cache/issues/2613
+.. _2622: https://github.com/varnishcache/varnish-cache/issues/2622
+.. _2623: https://github.com/varnishcache/varnish-cache/issues/2623
+.. _2624: https://github.com/varnishcache/varnish-cache/issues/2624
+.. _2631: https://github.com/varnishcache/varnish-cache/issues/2631
+.. _2647: https://github.com/varnishcache/varnish-cache/issues/2647
+.. _2649: https://github.com/varnishcache/varnish-cache/issues/2649
+.. _2650: https://github.com/varnishcache/varnish-cache/pull/2650
+.. _2651: https://github.com/varnishcache/varnish-cache/pull/2651
+.. _2661: https://github.com/varnishcache/varnish-cache/issues/2661
+.. _2678: https://github.com/varnishcache/varnish-cache/issues/2678
+.. _2679: https://github.com/varnishcache/varnish-cache/issues/2679
+.. _2681: https://github.com/varnishcache/varnish-cache/issues/2681
+.. _2682: https://github.com/varnishcache/varnish-cache/issues/2682
+.. _2685: https://github.com/varnishcache/varnish-cache/issues/2685
+.. _2686: https://github.com/varnishcache/varnish-cache/issues/2686
+.. _2690: https://github.com/varnishcache/varnish-cache/issues/2690
+.. _2693: https://github.com/varnishcache/varnish-cache/issues/2693
+.. _2695: https://github.com/varnishcache/varnish-cache/issues/2695
+.. _2700: https://github.com/varnishcache/varnish-cache/issues/2700
+.. _2702: https://github.com/varnishcache/varnish-cache/issues/2702
+.. _2706: https://github.com/varnishcache/varnish-cache/issues/2706
+.. _2716: https://github.com/varnishcache/varnish-cache/issues/2716
+.. _2719: https://github.com/varnishcache/varnish-cache/issues/2719
+.. _2722: https://github.com/varnishcache/varnish-cache/issues/2722
+.. _2726: https://github.com/varnishcache/varnish-cache/pull/2726
+.. _2727: https://github.com/varnishcache/varnish-cache/issues/2727
+.. _2731: https://github.com/varnishcache/varnish-cache/issues/2731
 
 ================================
 Varnish Cache 6.0.0 (2018-03-15)

From nils.goroll at uplex.de  Wed Sep 12 08:24:13 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 12 Sep 2018 08:24:13 +0000 (UTC)
Subject: [master] 57336bb12 have a go at some vmod objects dev doc (not new
 for 6.1)
Message-ID: <20180912082413.36F64A853A@lists.varnish-cache.org>


commit 57336bb12a702c23f723b11a63e09e1eb80994d4
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Sep 12 10:22:51 2018 +0200

    have a go at some vmod objects dev doc (not new for 6.1)

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index b9c0fcfc8..657585f82 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -178,6 +178,73 @@ declarations:
 	  Note that in particular also ``PRIV_*`` arguments (which are
 	  unnamed by definition) are passed as ``arg``\ `n`
 
+.. _ref-vmod-vcl-c-objects:
+
+Objects and methods
+-------------------
+
+Varnish also supports a simple object model for vmods. Objects and
+methods are declared in the vcc file as::
+
+	$Object class(`constructor arguments`)
+	$Method .method(`method arguments`)
+
+
+For declared object classes of a vmod, object instances can then be
+created in ``vcl_init { }`` using the ``new`` statement::
+
+	sub vcl_init {
+		new foo = vmod.class(...);
+	}
+
+and have their methods called anywhere (including in ``vcl_init {}``
+after the instantiation)::
+
+	sub somewhere {
+		foo.method(...);
+	}
+
+Object instances are represented as pointers to vmod-implemented C
+structs. Varnish only provides space to store the address of object
+instances and ensures that the right object address gets passed to C
+functions implementing methods.
+
+	* Objects' scope and lifetime are the vcl
+
+	* Objects can only be created in ``vcl_init {}`` and have
+	  their destructors called by varnish after ``vcl_fini {}``
+	  has completed.
+
+vmod authors are advised to understand the prototypes in the
+`vmodtool`\ -generated ``vcc_if.c`` file:
+
+	* For ``$Object`` declarations, a constructor and destructor
+	  function must be implemented
+
+	* The constructor is named by the suffix ``__init``, always is
+	  of ``VOID`` return type and has the following arguments
+	  before the vcc-declared parameters:
+
+	  * ``VRT_CTX`` as usual
+	  * a pointer-pointer to return the address of the created
+	    oject
+	  * a string containing the vcl name of the object instance
+
+	* The destructor is named by the suffix ``__fini``, always is
+	  of ``VOID`` return type and has a single argument, the
+	  pointer-pointer to the address of the object. The destructor
+	  is expected clear the address of the object stored in that
+	  pointer-pointer.
+
+	* Methods gain the pointer to the object as an argument after
+	   the ``VRT_CTX``.
+
+As varnish is in no way involved in managing object instances other
+than passing their addresses, vmods need to implement all aspects of
+managing instances, in particular their memory management. As the
+lifetime of object instances is the vcl, they will usually be
+allocated from the heap.
+
 .. _ref-vmod-vcl-c-types:
 
 VCL and C data types
@@ -439,6 +506,20 @@ malloc would look like this::
 
 The per-call vmod_privs are freed before the per-vcl vmod_priv.
 
+Note on use with objects:
+
+``PRIV_TASK`` and ``PRIV_TOP`` arguments are not per object instance,
+but still per vmod as for ordinary vmod functions. Thus, vmods
+requiring per-task / per top-request state for object instances need
+to implement other means to associate storage with object instances.
+
+Using ``VRT_priv_task()`` to maintin per object instance state is a
+convenient yet inofficial interface which was not originally intended
+for this purpose and will likely be replaced with a more suitable
+interface.
+
+
+
 .. _ref-vmod-event-functions:
 
 Event functions
@@ -511,6 +592,7 @@ In the case where properly releasing resources may take some time, you can
 opt for an asynchronous worker, either by spawning a thread and tracking it, or
 by using Varnish's worker pools.
 
+
 When to lock, and when not to lock
 ==================================
 

From nils.goroll at uplex.de  Wed Sep 12 08:41:09 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 12 Sep 2018 08:41:09 +0000 (UTC)
Subject: [master] e2bbdaeea polish
Message-ID: <20180912084109.CC337A8AD8@lists.varnish-cache.org>


commit e2bbdaeeade1796d4263310e6180420bd00b5c88
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Sep 12 10:29:07 2018 +0200

    polish

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index 657585f82..eb35c1bf9 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -186,8 +186,8 @@ Objects and methods
 Varnish also supports a simple object model for vmods. Objects and
 methods are declared in the vcc file as::
 
-	$Object class(`constructor arguments`)
-	$Method .method(`method arguments`)
+	$Object class(...)
+	$Method .method(...)
 
 
 For declared object classes of a vmod, object instances can then be

From nils.goroll at uplex.de  Wed Sep 12 08:41:09 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 12 Sep 2018 08:41:09 +0000 (UTC)
Subject: [master] cc7fdab79 updating vmods should just work (not new in any
 way)
Message-ID: <20180912084109.DCFB8A8ADA@lists.varnish-cache.org>


commit cc7fdab7931899f6ec1064afaa549e0af14ae20d
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Sep 12 10:29:18 2018 +0200

    updating vmods should just work (not new in any way)
    
    We copy vmods using unique filenames for ages
    
    Ref: 3cf587668219166188e0dce304edb0bf8ab346ab
    
    Original text from 2014: 920541a8fa79b2ce9471571df0f973b1beeb25b8

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index eb35c1bf9..e91b87db1 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -610,33 +610,3 @@ unless they access VMOD specific global state, shared with other VCLs.
 
 Traffic in other VCLs which also import this VMOD, will be happening
 while housekeeping is going on.
-
-Updating VMODs
-==============
-
-A compiled VMOD is a shared library file which Varnish dlopen(3)'s
-using flags RTLD_NOW | RTLD_LOCAL.
-
-As a general rule, once a file is opened with dlopen(3) you should
-never modify it, but it is safe to rename it and put a new file
-under the name it had, which is how most tools installs and updates
-shared libraries.
-
-However, when you call dlopen(3) with the same filename multiple
-times it will give you the same single copy of the shared library
-file, without checking if it was updated in the meantime.
-
-This is obviously an oversight in the design of the dlopen(3) library
-function, but back in the late 1980s nobody could imagine why a
-program would ever want to have multiple different versions of the
-same shared library mapped at the same time.
-
-Varnish does that, and therefore you must restart the worker process
-before Varnish will discover an updated VMOD.
-
-If you want to test a new version of a VMOD, while being able to
-instantly switch back to the old version, you will have to install
-each version with a distinct filename or in a distinct subdirectory
-and use ``import foo from "...";`` to reference it in your VCL.
-
-We're not happy about this, but have found no sensible workarounds.

From nils.goroll at uplex.de  Wed Sep 12 09:10:09 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 12 Sep 2018 09:10:09 +0000 (UTC)
Subject: [master] c17b66493 document vmod $ABI
Message-ID: <20180912091010.0DFBEA9581@lists.varnish-cache.org>


commit c17b6649378f5af495c29ad9ca84c69681e9b21a
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Sep 12 10:50:06 2018 +0200

    document vmod $ABI
    
    Text mostly taken from ecdebdc792942165ba71b58ff6816decb2d89bef

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index df6f5c7d9..209c68188 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -51,6 +51,7 @@ data structures that do all the hard work.
 The std VMODs vmod.vcc file looks somewhat like this::
 
 	$Module std 3
+	$ABI strict
 	$Event event_function
 	$Function STRING toupper(STRING_LIST)
 	$Function STRING tolower(STRING_LIST)
@@ -59,7 +60,17 @@ The std VMODs vmod.vcc file looks somewhat like this::
 The first line gives the name of the module and the manual section where
 the documentation will reside.
 
-The second line specifies an optional "Event" function, which will be
+The ``$ABI`` line is optional (possible values ``strict`` (default)
+and ``vrt``) and allows to specify that a vmod is integrating with the
+blessed ``vrt`` interface provided by ``varnishd`` or go deeper in the
+stack. As a general rule of thumb you are considered "on your own" if
+your VMOD uses more than the VRT (Varnish RunTime), in which case it
+needs to be built for the exact Varnish version.
+
+``$ABI vrt`` means that a module complies to the VRT and only needs to
+be rebuilt when breaking changes are introduced to the VRT API.
+
+The third line specifies an optional "Event" function, which will be
 called whenever a VCL program which imports this VMOD is loaded or
 transitions to any of the warm, active, cold or discarded states.
 More on this below.

From nils.goroll at uplex.de  Wed Sep 12 09:10:09 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 12 Sep 2018 09:10:09 +0000 (UTC)
Subject: [master] 6d0e24109 polish
Message-ID: <20180912091010.0402CA956B@lists.varnish-cache.org>


commit 6d0e2410905db9e4e0d30e0e91101713094a425f
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Sep 12 10:41:59 2018 +0200

    polish

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index e91b87db1..df6f5c7d9 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -46,7 +46,7 @@ The vmod.vcc file
 The interface between your VMOD and the VCL compiler ("VCC") and the
 VCL runtime ("VRT") is defined in the vmod.vcc file which a python
 script called "vmodtool.py" turns into thaumaturgically challenged C
-data structures that does all the hard work.
+data structures that do all the hard work.
 
 The std VMODs vmod.vcc file looks somewhat like this::
 

From nils.goroll at uplex.de  Wed Sep 12 09:13:07 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 12 Sep 2018 09:13:07 +0000 (UTC)
Subject: [master] b10c39179 document STRANDS
Message-ID: <20180912091307.9D8AFA98C2@lists.varnish-cache.org>


commit b10c3917901aab79b2ca8fca0818e23e1df5be9a
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Sep 12 11:12:07 2018 +0200

    document STRANDS

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index 209c68188..96fe6f011 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -422,6 +422,16 @@ STRING_LIST
 	and make sure your workspace_client and workspace_backend params
 	are big enough.
 
+STRANDS
+	C-Type: ``const struct strands *``
+
+	Strands are like STRING_LIST, but without the drawbacks of
+	variable arguments: The list of strings gets passed in a
+	struct with the following members:
+
+	* ``int n``: the number of strings
+	* ``const char p[]``: the array of strings with `n` elements
+
 TIME
 	C-type: ``double``
 

From nils.goroll at uplex.de  Wed Sep 12 09:33:09 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 12 Sep 2018 09:33:09 +0000 (UTC)
Subject: [master] ebdf056a6 more on STRANDS
Message-ID: <20180912093309.5178AA9F94@lists.varnish-cache.org>


commit ebdf056a62d785a5f0d838bd6476444b4697bbac
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Sep 12 11:32:04 2018 +0200

    more on STRANDS

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index 96fe6f011..288653bab 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -390,6 +390,10 @@ STEVEDORE
 STRING_LIST
 	C-type: ``const char *, ...``
 
+	`Notice: New vmod developments for 6.1 and higher should
+	consider STRANDS as a better alternative to STRING_LIST, which
+	will eventually be replaced entirely.`
+
 	A multi-component text-string.  We try very hard to avoid
 	doing text-processing in Varnish, and this is one way we
 	to avoid that, by not editing separate pieces of a string
@@ -415,12 +419,8 @@ STRING_LIST
 	NULL, as described under STRING, that is why we do not use NULL
 	as the terminator.
 
-	Right now we only support STRING_LIST being the last argument to
-	a function, we may relax that at a latter time.
-
-	If you don't want to bother with STRING_LIST, just use STRING
-	and make sure your workspace_client and workspace_backend params
-	are big enough.
+	STRING_LIST must be the last argument to a function and the
+	function must not contain optional arguments.
 
 STRANDS
 	C-Type: ``const struct strands *``

From dridi at varni.sh  Wed Sep 12 09:40:20 2018
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 12 Sep 2018 11:40:20 +0200
Subject: [master] e8344b806 upgrading WIP
In-Reply-To: <20180912065813.0170CA67CB@lists.varnish-cache.org>
References: <20180912065813.0170CA67CB@lists.varnish-cache.org>
Message-ID: <CABoVN9AfOpbj891jMY95vdgzxKSRU9Uvus8yKfC+XLOY4Gn3AQ@mail.gmail.com>

On Wed, Sep 12, 2018 at 8:58 AM, Nils Goroll <nils.goroll at uplex.de> wrote:
>
> commit e8344b8061e6151d0bc1a990c83459ed0f7b3d36
> Author: Nils Goroll <nils.goroll at uplex.de>
> Date:   Wed Sep 12 08:54:19 2018 +0200
>
>     upgrading WIP
>
> diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
> index 0daf56df7..a8c3ae7b4 100644
> --- a/doc/sphinx/whats-new/upgrading-6.1.rst
> +++ b/doc/sphinx/whats-new/upgrading-6.1.rst
> @@ -131,6 +131,9 @@ Other changes to VCL
>    which can be used in ``vcl_init`` to emit an error message if the
>    VCL load fails due to the return.
>
> +* Additional ``import`` statements of an already imported vmod are now
> +  ignored.
> +
>  VMODs
>  =====
>
> @@ -371,6 +374,13 @@ Other changes
>      ``cache_director.h`` is not public, and should not be used by a
>      VMOD intended for VRT ABI compatibility.
>
> +  * The director API in ``vrt.h`` differs from the previous
> +    interface:
> +
> +    *WIP slink*

Instead of documenting it here, you could update the docs [1] and link to them?

[1] https://varnish-cache.org/docs/trunk/reference/directors.html

I hope you haven't made too much progress on that yet.

> +    * The director API which has changed:
> +
>    * Python 3 is now preferred in builds, and will likely be required
>      in future versions.
>
> _______________________________________________
> varnish-commit mailing list
> varnish-commit at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-commit

From dridi.boukelmoune at gmail.com  Wed Sep 12 09:51:06 2018
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Wed, 12 Sep 2018 09:51:06 +0000 (UTC)
Subject: [master] 6754d4993 Typo
Message-ID: <20180912095106.ECCC3AC6E8@lists.varnish-cache.org>


commit 6754d4993c32e473e1ae20241dc6d4441597bade
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed Sep 12 11:50:35 2018 +0200

    Typo

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index 288653bab..98733b390 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -430,7 +430,7 @@ STRANDS
 	struct with the following members:
 
 	* ``int n``: the number of strings
-	* ``const char p[]``: the array of strings with `n` elements
+	* ``const char **p``: the array of strings with `n` elements
 
 TIME
 	C-type: ``double``
@@ -535,7 +535,7 @@ requiring per-task / per top-request state for object instances need
 to implement other means to associate storage with object instances.
 
 Using ``VRT_priv_task()`` to maintin per object instance state is a
-convenient yet inofficial interface which was not originally intended
+convenient yet unofficial interface which was not originally intended
 for this purpose and will likely be replaced with a more suitable
 interface.
 

From nils.goroll at uplex.de  Wed Sep 12 09:56:34 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 12 Sep 2018 11:56:34 +0200
Subject: [master] 6754d4993 Typo
In-Reply-To: <20180912095106.ECCC3AC6E8@lists.varnish-cache.org>
References: <20180912095106.ECCC3AC6E8@lists.varnish-cache.org>
Message-ID: <cdbf563e-1e3c-449e-f902-f9a8e95751e7@uplex.de>

On 12/09/18 11:51, Dridi Boukelmoune wrote:
> -	* ``const char p[]``: the array of strings with `n` elements
> +	* ``const char **p``: the array of strings with `n` elements

sorry, I intended to use const char *p[] for clarity. WFU?

-- 

** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstra?e 32
22301 Hamburg

tel +49 40 28805731
mob +49 170 2723133
fax +49 40 42949753

xmpp://slink at jabber.int.uplex.de/

http://uplex.de/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-commit/attachments/20180912/831a4289/attachment.bin>

From dridi.boukelmoune at gmail.com  Wed Sep 12 09:57:06 2018
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Wed, 12 Sep 2018 09:57:06 +0000 (UTC)
Subject: [master] 668e051db Typo
Message-ID: <20180912095706.C44D7ACBF0@lists.varnish-cache.org>


commit 668e051db084c4510a04725ae4b9946dd04bebd9
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Wed Sep 12 11:53:59 2018 +0200

    Typo

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index 98733b390..8978ca4f2 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -534,7 +534,7 @@ but still per vmod as for ordinary vmod functions. Thus, vmods
 requiring per-task / per top-request state for object instances need
 to implement other means to associate storage with object instances.
 
-Using ``VRT_priv_task()`` to maintin per object instance state is a
+Using ``VRT_priv_task()`` to maintain per object instance state is a
 convenient yet unofficial interface which was not originally intended
 for this purpose and will likely be replaced with a more suitable
 interface.

From dridi at varni.sh  Wed Sep 12 09:58:35 2018
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 12 Sep 2018 11:58:35 +0200
Subject: [master] 6754d4993 Typo
In-Reply-To: <cdbf563e-1e3c-449e-f902-f9a8e95751e7@uplex.de>
References: <20180912095106.ECCC3AC6E8@lists.varnish-cache.org>
 <cdbf563e-1e3c-449e-f902-f9a8e95751e7@uplex.de>
Message-ID: <CABoVN9BwqjJ3pjCFaBqHF3QxxR9rZARUxJxOtw+sozEaCV3oXw@mail.gmail.com>

On Wed, Sep 12, 2018 at 11:56 AM, Nils Goroll <nils.goroll at uplex.de> wrote:
> On 12/09/18 11:51, Dridi Boukelmoune wrote:
>> -     * ``const char p[]``: the array of strings with `n` elements
>> +     * ``const char **p``: the array of strings with `n` elements
>
> sorry, I intended to use const char *p[] for clarity. WFU?

I'd rather use the type we actually have in the actual struct strands.
If you think that would help VMOD developers I don't mind.

From dridi at varni.sh  Wed Sep 12 10:05:14 2018
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 12 Sep 2018 12:05:14 +0200
Subject: [master] 668e051db Typo
In-Reply-To: <20180912095706.C44D7ACBF0@lists.varnish-cache.org>
References: <20180912095706.C44D7ACBF0@lists.varnish-cache.org>
Message-ID: <CABoVN9C94Y7F2PW4gEpiwEkY7nTmnnBEJrKeXOgzd+VifUxOAg@mail.gmail.com>

> -Using ``VRT_priv_task()`` to maintin per object instance state is a
> +Using ``VRT_priv_task()`` to maintain per object instance state is a
>  convenient yet unofficial interface which was not originally intended
>  for this purpose and will likely be replaced with a more suitable
>  interface.

So... This has been around since 2014, almost 4 years. I'm not
comfortable telling this is not blessed by $ABI vrt.

Either it is, or we can introduce a new "VRTC" prefix for VRT Candidate
functions, which is fine since this release already has a major VRT bump.

And 6.1 could introduce the concept of candidate APIs, especially since
slink already documented the $ABI rule of thumb, we now have a place
to explain the difference between VRTC and VRT if we go that route (and
we can add it to the release notes for 6.1 too).

I like the idea of giving ourselves an escape hatch when we aren't
sure what a VRT interface will look like, or when we want to expose it
while parts are still expected to move before it stabilizes. Less
major bumps ahead possibly.

Thoughts?

Dridi

From nils.goroll at uplex.de  Wed Sep 12 10:11:59 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 12 Sep 2018 12:11:59 +0200
Subject: [master] 668e051db Typo
In-Reply-To: <CABoVN9C94Y7F2PW4gEpiwEkY7nTmnnBEJrKeXOgzd+VifUxOAg@mail.gmail.com>
References: <20180912095706.C44D7ACBF0@lists.varnish-cache.org>
 <CABoVN9C94Y7F2PW4gEpiwEkY7nTmnnBEJrKeXOgzd+VifUxOAg@mail.gmail.com>
Message-ID: <0df030a8-028e-7400-7a58-10a7c65ff8aa@uplex.de>

On 12/09/18 12:05, Dridi Boukelmoune wrote:
> So... This has been around since 2014, almost 4 years. I'm not
> comfortable telling this is not blessed by $ABI vrt.

it is a part of VRT for use from VCC to generate the PRIV_TASK arguments to vmod
functions.

It was not intended by phk for how we (geoff, myself) use it all over the place.

A couple weeks back, I even wrote a prototype to replace it, and I guess we'll
come back to that after 6.1 soon-ish.

Nils

-- 

** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstra?e 32
22301 Hamburg

tel +49 40 28805731
mob +49 170 2723133
fax +49 40 42949753

xmpp://slink at jabber.int.uplex.de/

http://uplex.de/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-commit/attachments/20180912/d23d9ba6/attachment.bin>

From dridi at varni.sh  Wed Sep 12 10:17:35 2018
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 12 Sep 2018 12:17:35 +0200
Subject: [master] 668e051db Typo
In-Reply-To: <0df030a8-028e-7400-7a58-10a7c65ff8aa@uplex.de>
References: <20180912095706.C44D7ACBF0@lists.varnish-cache.org>
 <CABoVN9C94Y7F2PW4gEpiwEkY7nTmnnBEJrKeXOgzd+VifUxOAg@mail.gmail.com>
 <0df030a8-028e-7400-7a58-10a7c65ff8aa@uplex.de>
Message-ID: <CABoVN9BH3vfJsEhFFy2W-hL+1APMpCw+VY1dU52u2mF-FTn3VQ@mail.gmail.com>

On Wed, Sep 12, 2018 at 12:11 PM, Nils Goroll <nils.goroll at uplex.de> wrote:
> On 12/09/18 12:05, Dridi Boukelmoune wrote:
>> So... This has been around since 2014, almost 4 years. I'm not
>> comfortable telling this is not blessed by $ABI vrt.
>
> it is a part of VRT for use from VCC to generate the PRIV_TASK arguments to vmod
> functions.
>
> It was not intended by phk for how we (geoff, myself) use it all over the place.
>
> A couple weeks back, I even wrote a prototype to replace it, and I guess we'll
> come back to that after 6.1 soon-ish.

Then according to our guidelines [1] it shoud be demoted to vrt_priv_task().

Dridi

[1] https://varnish-cache.org/docs/trunk/phk/apispaces.html#general-namespace-rules

From nils.goroll at uplex.de  Wed Sep 12 10:21:53 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 12 Sep 2018 12:21:53 +0200
Subject: [master] 668e051db Typo
In-Reply-To: <CABoVN9BH3vfJsEhFFy2W-hL+1APMpCw+VY1dU52u2mF-FTn3VQ@mail.gmail.com>
References: <20180912095706.C44D7ACBF0@lists.varnish-cache.org>
 <CABoVN9C94Y7F2PW4gEpiwEkY7nTmnnBEJrKeXOgzd+VifUxOAg@mail.gmail.com>
 <0df030a8-028e-7400-7a58-10a7c65ff8aa@uplex.de>
 <CABoVN9BH3vfJsEhFFy2W-hL+1APMpCw+VY1dU52u2mF-FTn3VQ@mail.gmail.com>
Message-ID: <4d33a076-91d0-ac0a-579d-bf61d12025b6@uplex.de>

i'd want to leave this with phk. From my POV, I'd be happy to remove that
section from the docs or rename vrt_priv_task, but I'm not sure if the latter is
really worth it.

It's not that we'd be stringent with all our interfaces all the time and this
particular case does not constitute much of a problem, IMHO.

VRT_priv_task works just fine, the only point is that phk did not envision and
does not support the use case with arbitrary pointers.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-commit/attachments/20180912/52b572b8/attachment.bin>

From dridi at varni.sh  Wed Sep 12 10:44:56 2018
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 12 Sep 2018 12:44:56 +0200
Subject: [master] 53d20a98a Update changes.rst for the 2018-09 release.
In-Reply-To: <20180912081213.8BDB2A7F63@lists.varnish-cache.org>
References: <20180912081213.8BDB2A7F63@lists.varnish-cache.org>
Message-ID: <CABoVN9BgYj22wioLR-BkY-jiufNc7B3jOy8YFqH8acgbzwQU_Q@mail.gmail.com>

> +Fixed bugs
> +----------
>
> -* XXX director api, cache_backend.h off-limits, moved to VRT

Should we also update API spaces [1] to reflect that?

Dridi

[1] https://varnish-cache.org/docs/trunk/phk/apispaces.html#the-vmod-package-api-abi

From nils.goroll at uplex.de  Wed Sep 12 11:20:54 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 12 Sep 2018 13:20:54 +0200
Subject: [master] e8344b806 upgrading WIP
In-Reply-To: <CABoVN9AfOpbj891jMY95vdgzxKSRU9Uvus8yKfC+XLOY4Gn3AQ@mail.gmail.com>
References: <20180912065813.0170CA67CB@lists.varnish-cache.org>
 <CABoVN9AfOpbj891jMY95vdgzxKSRU9Uvus8yKfC+XLOY4Gn3AQ@mail.gmail.com>
Message-ID: <ab40c718-90de-8c27-ede5-60509c5f39b5@uplex.de>

On 12/09/18 11:40, Dridi Boukelmoune wrote:
> Instead of documenting it here, you could update the docs [1] and link to them?
> 
> [1] https://varnish-cache.org/docs/trunk/reference/directors.html
> 
> I hope you haven't made too much progress on that yet.

that was the plan, and then i noticed that the basis for all of it - vmod
objects, was not documented, then i noticed, that optional arguments weren't,
then I worked that stack and now I'm back at directors ;)

-- 

** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstra?e 32
22301 Hamburg

tel +49 40 28805731
mob +49 170 2723133
fax +49 40 42949753

xmpp://slink at jabber.int.uplex.de/

http://uplex.de/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-commit/attachments/20180912/dd5abbee/attachment-0001.bin>

From nils.goroll at uplex.de  Wed Sep 12 17:44:09 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 12 Sep 2018 17:44:09 +0000 (UTC)
Subject: [master] ab34ea9f2 update and (hopefully) improve the directors dev
 guide
Message-ID: <20180912174409.EA6D178BB@lists.varnish-cache.org>


commit ab34ea9f24235952596fce211f9a9b05a9ddd77d
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Sep 12 19:41:25 2018 +0200

    update and (hopefully) improve the directors dev guide
    
    Much more could be said, but I hope this helps understand the
    overall concepts a bit better.
    
    I've also taken the freedom to reword "cluster" in the director
    concept to "load balancer" because at least in my experience, the
    former is used more for some form of tighter binding between nodes,
    while the latter is a common term for what directors usually do.

diff --git a/doc/sphinx/reference/directors.rst b/doc/sphinx/reference/directors.rst
index 10e41d6ae..b359133ae 100644
--- a/doc/sphinx/reference/directors.rst
+++ b/doc/sphinx/reference/directors.rst
@@ -21,45 +21,78 @@ Backends vs Directors
 =====================
 
 The intuitive classification for backend and director is an endpoint for the
-former and a cluster for the latter, but the actual implementation is a bit
+former and a loadbalancer for the latter, but the actual implementation is a bit
 more subtle. VMODs can accept backend arguments and return backends in VCL (see
-:ref:`ref-vmod-vcl-c-types`), but the underlying C type is ``struct director``.
+:ref:`ref-vmod-vcl-c-types`), but the underlying C type is ``struct director``
+aka the ``VCL_BACKEND`` typedef.
 Under the hood director is a generic concept, and a backend is a kind of
 director.
 
 The line between the two is somewhat blurry at this point, let's look at some
 code instead::
 
+    // VRT interface from vrt.h
+
+    struct vdi_methods {
+        unsigned                        magic;
+    #define VDI_METHODS_MAGIC           0x4ec0c4bb
+        const char                      *type;
+        vdi_http1pipe_f                 *http1pipe;
+        vdi_healthy_f                   *healthy;
+        vdi_resolve_f                   *resolve;
+        vdi_gethdrs_f                   *gethdrs;
+        vdi_getip_f                     *getip;
+        vdi_finish_f                    *finish;
+        vdi_event_f                     *event;
+        vdi_destroy_f                   *destroy;
+        vdi_panic_f                     *panic;
+        vdi_list_f                      *list;
+    };
+
     struct director {
-            unsigned                magic;
-    #define DIRECTOR_MAGIC          0x3336351d
-            const char              *name;
-            char                    *vcl_name;
-            vdi_http1pipe_f         *http1pipe;
-            vdi_healthy_f           *healthy;
-            vdi_resolve_f           *resolve;
-            vdi_gethdrs_f           *gethdrs;
-            vdi_getbody_f           *getbody;
-            vdi_getip_f             *getip;
-            vdi_finish_f            *finish;
-            vdi_panic_f             *panic;
-            void                    *priv;
-            const void              *priv2;
+        unsigned                        magic;
+    #define DIRECTOR_MAGIC              0x3336351d
+        unsigned                        sick;
+        void                            *priv;
+        char                            *vcl_name;
+        struct vcldir                   *vdir;
     };
 
 A director can be summed up as:
 
-- a name (used for panics)
-- a VCL name
-- a set of operations
-- the associated state
+- being of a specific ``type`` with a set of operations which is
+  identical for all instances of that particular type
+- some instance specific attributes such as a ``vcl_name``, health
+  state and ``type``\ -specific private data
+
+The difference between a *load balancing* director and a *backend*
+director is mainly the functions they will implement.
+
+The fundamental steps towards a director implementation are:
+
+- implement the required functions
+- fill a ``struct vdi_methods`` with the name of your director type
+  and your function pointers
+- in your constructor or other initialization routine, allocate and
+  initialize your director-specific configuration state (aka private
+  data) and call ``VRT_AddDirector()`` with your ``struct
+  vdi_methods``, the pointer to your state and a printf format for the
+  name of your director instance
+- implement methods or functions returning ``VCL_BACKEND``
+- in your destructor or other finalizer, call ``VRT_DelDirector()``
 
-The difference between a *cluster* director and a *backend* director is mainly
-The functions they will implement.
+For forwards compatibility, it is strongly recommended for the last
+step not to destroy the actual director private state, but rather
+implement and declare in ``struct vdi_methods`` a ``destroy``
+callback.
 
+While vmods can implement functions returning directors,
+:ref:`ref-vmod-vcl-c-objects` are usually a more natural
+representation with vmod object instances being or referring to the
+director private data.
 
-Cluster Directors
-=================
+Load Balancing Directors
+========================
 
 As in :ref:`vmod_directors(3)`, you can write directors that will group
 backends sharing the same role, and pick them according to a strategy. If you
@@ -75,9 +108,10 @@ request, just like the backends you declare in VCL.
 Dynamic Backends
 ================
 
-If you want to speak HTTP/1 over TCP, but for some reason VCL does not fit the
-bill, you can instead reuse the whole backend facility. It allows you for
-instance to add and remove backends on-demand without the need to reload your
+If you want to speak HTTP/1 over TCP or UDS, but for some reason VCL
+does not fit the bill, you can instead reuse the whole backend
+facility. It allows you for instance to add and remove backends
+on-demand without the need to reload your
 VCL. You can then leverage your provisioning system.
 
 Consider the following snippet::
@@ -86,25 +120,24 @@ Consider the following snippet::
         .host = "localhost";
     }
 
-The VCL compiler turns this declaration into a ``struct vrt_backend``. When the
-VCL is loaded, Varnish calls ``VRT_new_backend`` in order to create the
-director. Varnish doesn't expose its data structure for actual backends, only
-the director abstraction and dynamic backends are built just like static
-backends, one *struct* at a time. You can get rid of the ``struct vrt_backend``
-as soon as you have the ``struct director``.
-
-A (dynamic) backend can't exceed its VCL's lifespan, because native backends
-are *owned* by VCLs. Though a dynamic backend can't outlive its VCL, it can be
-deleted any time with ``VRT_delete_backend``. The VCL will delete the remaining
-backends once discarded, you don't need to take care of it.
-
-.. XXX Consider using an object (see :ref:`ref-vmod-objects`) to manipulate dynamic
-
-Consider using an object to manipulate dynamic
-backends. They are tied to the VCL life cycle and make a handy data structure
-to keep track of backends and objects have a VCL name you can reuse for the
-director. It is also true for *cluster* directors that may reference native
-backends.
+The VCL compiler turns this declaration into a ``struct
+vrt_backend``. When the VCL is loaded, Varnish calls
+``VRT_new_backend`` (or rather ``VRT_new_backend_clustered`` for VSM
+efficiency) in order to create the director. Varnish doesn't expose
+its data structure for actual backends, only the director abstraction
+and dynamic backends are built just like static backends, one *struct*
+at a time. You can get rid of the ``struct vrt_backend`` as soon as
+you have the ``struct director``.
+
+A (dynamic) backend can't exceed its VCL's lifespan, because native
+backends are *owned* by VCLs. Though a dynamic backend can't outlive
+its VCL, it can be deleted any time with ``VRT_delete_backend``. The
+VCL will delete the remaining backends once discarded, you don't need
+to take care of it.
+
+.. XXX this does not quite work yet because the deleted backend could
+   be referenced, but at least that's where we want to get to. See
+   also https://github.com/varnishcache/varnish-cache/pull/2725
 
 Finally, Varnish will take care of event propagation for *all* native backends,
 but dynamic backends can only be created when the VCL is warm. If your backends
@@ -116,7 +149,7 @@ is cooling. You are also encouraged to comply with the
 :ref:`ref_vcl_temperature` in general.
 
 
-.. _ref-writing-a-director-cluster:
+.. _ref-writing-a-director-loadbalancer:
 
 Health Probes
 =============
@@ -125,9 +158,10 @@ It is possible in a VCL program to query the health of a director (see
 :ref:`func_healthy`). A director can report its health if it implements the
 ``healthy`` function, it is otherwise always considered healthy.
 
-Unless you are making a dynamic backend, you need to take care of the health
-probes yourselves. For *cluster* directors, being healthy typically means
-having at least one healthy underlying backend or director.
+Unless you are making a dynamic backend, you need to take care of the
+health probes yourselves. For *load balancing* directors, being
+healthy typically means having at least one healthy underlying backend
+or director.
 
 For dynamic backends, it is just a matter of assigning the ``probe`` field in
 the ``struct vrt_backend``. Once the director is created, the probe definition
@@ -138,19 +172,18 @@ probe and disable the feature on a cold VCL (see
 Instead of initializing your own probe definition, you can get a ``VCL_PROBE``
 directly built from VCL (see :ref:`ref-vmod-vcl-c-types`).
 
-What's the difference ?
-
 
 Custom Backends
 ===============
 
 If you want to implement a custom backend, have a look at how Varnish
-implements native backends. It is the canonical implementation, and though it
-provides other services like connection pooling or statistics, it is
-essentially a director which state is a ``struct backend``. Varnish native
-backends currently speak HTTP/1 over TCP, and as such, you need to make your
-own custom backend if you want Varnish to do otherwise such as connect over
-UDP or UNIX-domain sockets or speak a different protocol.
+implements native backends. It is the canonical implementation, and
+though it provides other services like connection pooling or
+statistics, it is essentially a director which state is a ``struct
+backend``. Varnish native backends currently speak HTTP/1 over TCP or
+UDS, and as such, you need to make your own custom backend if you want
+Varnish to do otherwise such as connect over UDP or speak a different
+protocol.
 
 If you want to leverage probes declarations in VCL, which have the advantage of
 being reusable since they are only specifications, you can. However, you need

From nils.goroll at uplex.de  Wed Sep 12 17:57:07 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 12 Sep 2018 17:57:07 +0000 (UTC)
Subject: [master] 677a71eba release docs: director api
Message-ID: <20180912175707.15E2A7D4C@lists.varnish-cache.org>


commit 677a71eba859a03fa87fbbbc43d87a683d06788a
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Sep 12 19:55:11 2018 +0200

    release docs: director api

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 47729f7d8..f1ca841b8 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -381,11 +381,18 @@ Other changes
     VMOD intended for VRT ABI compatibility.
 
   * The director API in ``vrt.h`` differs from the previous
-    interface:
-
-    *WIP slink*
-
-    * The director API which has changed:
+    interface. :ref:`ref-writing-a-director` has been updated
+    accordingly. In short, the most important changes include:
+
+    * ``struct director_methods`` is replaced by ``struct vdi_methods``
+      * signatures of various callbacks have changed
+    * ``VRT_AddDirector()`` and ``VRT_DelDirector()`` are to be used
+      for initialization and destruction.
+    * ``vdi_methods`` callbacks are not to be called from vmods any more
+    * ``VRT_Healthy()`` replaces calls to the ``healthy`` function
+    * The admin health is not to be manipulated by vmods any more
+    * director private state destruction is recommended to be
+      implemented via a ``destroy`` callback.
 
   * Python 3 is now preferred in builds, and will likely be required
     in future versions.

From geoff at uplex.de  Thu Sep 13 07:55:12 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Thu, 13 Sep 2018 07:55:12 +0000 (UTC)
Subject: [master] a02420caa Remove 'stream' from the release notes example
 about beresp.filters.
Message-ID: <20180913075513.2AF9B97DA0@lists.varnish-cache.org>


commit a02420caa37a9b6da49183c213035a5aae34cdcf
Author: Geoff Simmons <geoff at uplex.de>
Date:   Thu Sep 13 09:53:57 2018 +0200

    Remove 'stream' from the release notes example about beresp.filters.

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index f1ca841b8..8adba7689 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -103,9 +103,8 @@ The ``beresp.filters`` variable is readable and writable in
 ``vcl_backend_response``. This is a space-separated list of modules
 that we call VFPs, for "Varnish fetch processors", that may be applied
 to a backend response body as it is being fetched. In default Varnish,
-the list may include values such as ``gzip``, ``gunzip``, ``esi`` and
-``stream``, depending on how you have set the ``beresp.do_*``
-variables.
+the list may include values such as ``gzip``, ``gunzip``, and ``esi``,
+depending on how you have set the ``beresp.do_*`` variables.
 
 This addition makes it possible for VMODs to define VFPs to filter or
 manipulate backend response bodies, which can be added by changing the

From dridi at varni.sh  Thu Sep 13 08:24:06 2018
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Thu, 13 Sep 2018 10:24:06 +0200
Subject: [master] ab34ea9f2 update and (hopefully) improve the directors
 dev guide
In-Reply-To: <20180912174409.EA6D178BB@lists.varnish-cache.org>
References: <20180912174409.EA6D178BB@lists.varnish-cache.org>
Message-ID: <CABoVN9DHRSmKNVC0SPc8PPOSE5AzY6SU2=tU8gbZkmONpDZwGQ@mail.gmail.com>

On Wed, Sep 12, 2018 at 7:44 PM, Nils Goroll <nils.goroll at uplex.de> wrote:
>
> commit ab34ea9f24235952596fce211f9a9b05a9ddd77d
> Author: Nils Goroll <nils.goroll at uplex.de>
> Date:   Wed Sep 12 19:41:25 2018 +0200
>
>     update and (hopefully) improve the directors dev guide
>
>     Much more could be said, but I hope this helps understand the
>     overall concepts a bit better.
>
>     I've also taken the freedom to reword "cluster" in the director
>     concept to "load balancer" because at least in my experience, the
>     former is used more for some form of tighter binding between nodes,
>     while the latter is a common term for what directors usually do.

I chose not to use "load balancer" exactly for the opposite reason.
Usually the first reason to have clusters of backends is high
availability, and once we have a cluster in place then yes load
balancing may not be a bad idea. And as far as vmod_directors is
concerned, fallback does the opposite of balancing workloads.

Dridi

From geoff at uplex.de  Thu Sep 13 08:41:07 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Thu, 13 Sep 2018 08:41:07 +0000 (UTC)
Subject: [master] 6fdc89c1f Finish the release notes about JSON support in the
 CLI.
Message-ID: <20180913084108.156709B088@lists.varnish-cache.org>


commit 6fdc89c1fdd12bb83fadd8412faa1d7703761e36
Author: Geoff Simmons <geoff at uplex.de>
Date:   Thu Sep 13 10:40:20 2018 +0200

    Finish the release notes about JSON support in the CLI.

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 8adba7689..2a33f67e0 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -221,10 +221,9 @@ Other changes
 
     * ``help -j``
 
-    * **XXX...**
-
-    A JSON response in the CLI always includes a timestamp (epoch time in
-    seconds with millisecond precision).
+    A JSON response in the CLI always includes a timestamp (epoch time
+    in seconds with millisecond precision), indicating the time at
+    which the reponse was generated.
 
   * The ``backend.list`` command now lists both directors and
     backends, with their health status. The command now has a ``-v``

From hermunn at varnish-software.com  Thu Sep 13 13:03:12 2018
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Thu, 13 Sep 2018 13:03:12 +0000 (UTC)
Subject: [4.1] 721448ef0 Update the documentation on grace and keep
Message-ID: <20180913130312.3961BA2FFA@lists.varnish-cache.org>


commit 721448ef095eaaffbecb0981de6e79fd32360f21
Author: P?l Hermunn Johansen <hermunn at varnish-software.com>
Date:   Thu Jun 7 14:56:24 2018 +0200

    Update the documentation on grace and keep
    
    This is a back port of 7494d6adc9bad382cf267b352c21f5c1069bb14c,
    in which the main point is to clearly recomend using req.grace
    for the most common use case - using different grace time when
    the backend is healthy. To simplify things, the vcl-grace.rst
    file is just copied from master. It should be accurate also for
    the 4.1 branch.

diff --git a/doc/sphinx/users-guide/vcl-grace.rst b/doc/sphinx/users-guide/vcl-grace.rst
index dad7fd0cb..dddfac5f7 100644
--- a/doc/sphinx/users-guide/vcl-grace.rst
+++ b/doc/sphinx/users-guide/vcl-grace.rst
@@ -1,11 +1,15 @@
 .. _users-guide-handling_misbehaving_servers:
 
-Misbehaving servers
+Grace mode and keep
 -------------------
 
-A key feature of Varnish is its ability to shield you from misbehaving
-web- and application servers.
+Sometimes you want Varnish to serve content that is somewhat stale
+instead of waiting for a fresh object from the backend. For example,
+if you run a news site, serving a main page that is a few seconds old
+is not a problem if this gives your site faster load times.
 
+In Varnish this is achieved by using `grace mode`. A related idea
+is `keep`, which is also explained here.
 
 Grace mode
 ~~~~~~~~~~
@@ -19,52 +23,143 @@ If you are serving thousands of hits per second the queue of waiting
 requests can get huge. There are two potential problems - one is a
 thundering herd problem - suddenly releasing a thousand threads to
 serve content might send the load sky high. Secondly - nobody likes to
-wait. To deal with this we can instruct Varnish to keep
-the objects in cache beyond their TTL and to serve the waiting
-requests somewhat stale content.
+wait.
+
+Setting an object's `grace` to a positive value tells Varnish that it
+should serve the object to clients for some time after the TTL has
+expired, while Varnish fetches a new version of the object. The default
+value is controlled by the runtime parameter ``default_grace``.
+
+Keep
+~~~~
 
-So, in order to serve stale content we must first have some content to
-serve. So to make Varnish keep all objects for 2 minutes beyond their
-TTL use the following VCL::
+Setting an object's `keep` tells Varnish that it should keep an object
+in the cache for some additional time. The reasons to set `keep` is to
+use the object to construct a conditional GET backend request (with
+If-Modified-Since: and/or ?f-None-Match: headers), allowing the
+backend to reply with a 304 Not Modified response, which may be more
+efficient on the backend and saves re-transmitting the unchanged body.
+
+The values are additive, so if grace is 10 seconds and keep is 1 minute,
+then objects will survive in cache for 70 seconds after the TTL has
+expired.
+
+Setting grace and keep
+~~~~~~~~~~~~~~~~~~~~~~
+
+We can use VCL to make Varnish keep all objects for 10 minutes beyond
+their TTL with a grace period of 2 minutes::
 
   sub vcl_backend_response {
-    set beresp.grace = 2m;
+       set beresp.grace = 2m;
+       set beresp.keep = 8m;
   }
 
-Now Varnish will be allowed to serve objects that are up to two
-minutes out of date. When it does it will also schedule a refresh of
-the object. This will happen asynchronously and the moment the new
-object is in it will replace the one we've already got.
+The effect of grace and keep
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For most users setting the default grace and/or a suitable grace for
+each object is enough. The default VCL will do the right thing and
+behave as described above. However, if you want to customize how
+varnish behaves, then you should know some of the details on how this
+works.
+
+When ``sub vcl_recv`` ends with ``return (lookup)`` (which is the
+default behavior), Varnish will look for a matching object in its
+cache. Then, if it only found an object whose TTL has run out, Varnish
+will consider the following:
+
+* Is there already an ongoing backend request for the object?
+* Is the object within the `grace period`?
+
+Then, Varnish reacts using the following rules:
+
+* If the `grace period` has run out and there is no ongoing backend
+  request, then ``sub vcl_miss`` is called immediately, and the object
+  will be used as a 304 candidate.
+* If the `grace period` has run out and there is an ongoing backend
+  request, then the request will wait until the backend request
+  finishes.
+* If there is no backend request for the object, one is scheduled.
+* Assuming the object will be delivered, ``sub vcl_hit`` is called
+  immediately.
+
+Note that the backend fetch happens asynchronously, and the moment the
+new object is in it will replace the one we've already got.
+
+If you do not define your own ``sub vcl_hit``, then the default one is
+used. It looks like this::
+
+  sub vcl_hit {
+       if (obj.ttl >= 0s) {
+            // A pure unadulterated hit, deliver it
+            return (deliver);
+       }
+       if (obj.ttl + obj.grace > 0s) {
+            // Object is in grace, deliver it
+            // Automatically triggers a background fetch
+            return (deliver);
+       }
+       // fetch & deliver once we get the result
+       return (miss);
+  }
 
-You can influence how this logic works by adding code in vcl_hit. The
-default looks like this::
+The effect of the built-in VCL is in fact equivalent to the following::
 
   sub vcl_hit {
-     if (obj.ttl >= 0s) {
-         // A pure unadultered hit, deliver it
-         return (deliver);
-     }
-     if (obj.ttl + obj.grace > 0s) {
-         // Object is in grace, deliver it
-         // Automatically triggers a background fetch
-         return (deliver);
-     }
-     // fetch & deliver once we get the result
-     return (fetch);
+       return (deliver);
   }
 
-The grace logic is pretty obvious here. If you have enabled
-:ref:`users-guide-advanced_backend_servers-health` you can check if
-the backend is sick and only serve graced object then. Replace the
-second if-clause with something like this::
+This is because ``obj.ttl + obj.grace > 0s`` always will evaluate to
+true. However, the the VCL is as it is to show users how to
+differentiate between a pure hit and a `grace` hit. With the next
+major version of Varnish, the default VCL is planned to change to the
+latter, shorter version.
+
+Misbehaving servers
+~~~~~~~~~~~~~~~~~~~
+
+A key feature of Varnish is its ability to shield you from misbehaving
+web- and application servers.
+
+If you have enabled :ref:`users-guide-advanced_backend_servers-health`
+you can check if the backend is sick and modify the behavior when it
+comes to grace. This can done in the following way::
+
+  sub vcl_backend_response {
+       set beresp.grace = 24h;
+       // no keep - the grace should be enough for 304 candidates
+  }
+
+  sub vcl_recv {
+       if (std.healthy(req.backend_hint)) {
+            // change the behavior for healthy backends: Cap grace to 10s
+            set req.grace = 10s;
+       }
+  }
+
+In the example above, the special variable ``req.grace`` is set.  The
+effect is that, when the backend is healthy, objects with grace above
+10 seconds will have an `effective` grace of 10 seconds.  When the
+backend is sick, the default VCL kicks in, and the long grace is used.
+
+Additionally, you might want to stop cache insertion when a backend fetch
+returns an ``5xx`` error::
+
+  sub vcl_backend_response {
+       if (beresp.status >= 500 && bereq.is_bgfetch) {
+            return (abandon);
+       }
+  }
 
-   if (!std.healthy(req.backend_hint) && (obj.ttl + obj.grace > 0s)) {
-         return (deliver);
-   } else {
-         return (fetch);
-   }
+Summary
+~~~~~~~
 
-So, to sum up, grace mode solves two problems:
- * it serves stale content to avoid request pile-up.
- * it serves stale content if you allow it.
+Grace mode allows Varnish to deliver slightly stale content to clients
+while getting a fresh version from the backend. The result is faster
+load times at lower cost.
 
+It is possible to limit the grace during lookup by setting
+``req.grace`` and then change the behavior when it comes to
+grace. Often this is done to change the `effective` grace depending on
+the health of the backend.

From hermunn at varnish-software.com  Thu Sep 13 13:03:11 2018
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Thu, 13 Sep 2018 13:03:11 +0000 (UTC)
Subject: [4.1] d2d09318c Reintroduce the req.grace variable,
 change keep behavior
Message-ID: <20180913130312.2B039A2FF8@lists.varnish-cache.org>


commit d2d09318cfacbc61c7bc07cf502fddbae6ec3fbf
Author: P?l Hermunn Johansen <hermunn at varnish-software.com>
Date:   Tue Jun 12 16:33:07 2018 +0200

    Reintroduce the req.grace variable, change keep behavior
    
    This is a back port of ff38535acd6d9c3b87d7fbbcc9c6917f7106d216
    
    The req.grace variable can be set in vcl_recv to cap the grace
    of objects in the cache, in the same way as in 3.0.x
    
    The "keep" behavior changes with this patch. We now always go
    to vcl_miss when the expired object is out of grace, or we go
    to the waiting list. The result is that it is no longer
    possible to deliver a "keep" object in vcl_hit.
    
    Note that when we get to vcl_miss, we will still have the 304
    candidate, but without the detour by vcl_hit.
    
    This commit changes VCL, but only slightly, so we aim to back
    port this to earlier versions of Varnish Cache.
    
    Refs: #1799 and #2519
    
    Conflicts:
            bin/varnishd/cache/cache_hash.c
            bin/varnishd/cache/cache_req_fsm.c
            bin/varnishd/cache/cache_varnishd.h
            bin/varnishd/cache/cache_vrt_var.c
            doc/sphinx/reference/vcl_var.rst

diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 1089f0a41..f27b2f6c9 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -564,6 +564,7 @@ struct req {
 	uint8_t			digest[DIGEST_LEN];
 
 	double			d_ttl;
+	double			d_grace;
 
 	ssize_t			req_bodybytes;	/* Parsed req bodybytes */
 
@@ -697,6 +698,7 @@ void EXP_Clr(struct exp *e);
 
 double EXP_Ttl(const struct req *, const struct exp*);
 double EXP_When(const struct exp *exp);
+double EXP_Ttl_grace(const struct req *, const struct exp*);
 void EXP_Insert(struct worker *wrk, struct objcore *oc);
 void EXP_Inject(struct worker *wrk, struct objcore *oc, struct lru *lru);
 void EXP_Rearm(struct objcore *, double now, double ttl, double grace,
diff --git a/bin/varnishd/cache/cache_expire.c b/bin/varnishd/cache/cache_expire.c
index 28c79ab8a..4b0b95913 100644
--- a/bin/varnishd/cache/cache_expire.c
+++ b/bin/varnishd/cache/cache_expire.c
@@ -132,6 +132,24 @@ EXP_When(const struct exp *e)
 	return (when);
 }
 
+/*--------------------------------------------------------------------
+ * Calculate an object's effective ttl+grace time, taking req.grace into
+ * account if it is available.
+ */
+
+double
+EXP_Ttl_grace(const struct req *req, const struct exp *e)
+{
+	double g;
+
+	AN(e);
+
+	g = e->grace;
+	if (req != NULL && req->d_grace >= 0. && req->d_grace < g)
+		g = req->d_grace;
+	return (EXP_Ttl(req, e) + g);
+}
+
 /*--------------------------------------------------------------------
  * Post an objcore to the exp_thread's inbox.
  */
diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index 30e2a06f7..d1f5f1c56 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -455,36 +455,50 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp,
 		}
 	}
 
+	if (exp_oc != NULL && EXP_Ttl_grace(req, &exp_oc->exp) < req->t_req) {
+		/* the newest object is out of grace */
+		if (!busy_found) {
+			/*
+			 * here we get to insert the busy object. This
+			 * translates to a MISS with a 304 candidate.
+			 */
+			assert(oh->refcnt > 1);
+			assert(exp_oc->objhead == oh);
+			exp_oc->refcnt++;
+			/* Insert objcore in objecthead and release mutex */
+			*bocp = hsh_insert_busyobj(wrk, oh);
+			/* NB: no deref of objhead, new object inherits reference */
+			Lck_Unlock(&oh->mtx);
+			*ocp = exp_oc;
+			return (HSH_MISS);
+		} else {
+			/* we have no use for this very expired object */
+			exp_oc = NULL;
+		}
+	}
 	if (exp_oc != NULL) {
+		/*
+		 * here the object is within grace, so we expect it to
+		 * be delivered
+		 */
 		assert(oh->refcnt > 1);
 		assert(exp_oc->objhead == oh);
+		exp_oc->refcnt++;
 
 		if (!busy_found) {
 			*bocp = hsh_insert_busyobj(wrk, oh);
 			retval = HSH_EXPBUSY;
 		} else {
 			AZ(req->hash_ignore_busy);
-			/*
-			 * here we have a busy object, but if the stale object
-			 * is not under grace we go to the waiting list
-			 * instead of returning the stale object
-			 */
-			if (EXP_Ttl(req, &exp_oc->exp) + exp_oc->exp.grace < req->t_req)
-				retval = HSH_BUSY;
-			else
-				retval = HSH_EXP;
-		}
-		if (retval != HSH_BUSY) {
-			exp_oc->refcnt++;
-
-			if (exp_oc->hits < LONG_MAX)
-				exp_oc->hits++;
-			Lck_Unlock(&oh->mtx);
-			if (retval == HSH_EXP)
-				assert(HSH_DerefObjHead(wrk, &oh));
-			*ocp = exp_oc;
-			return (retval);
+			retval = HSH_EXP;
 		}
+		if (exp_oc->hits < LONG_MAX)
+			exp_oc->hits++;
+		Lck_Unlock(&oh->mtx);
+		if (retval == HSH_EXP)
+			assert(HSH_DerefObjHead(wrk, &oh));
+		*ocp = exp_oc;
+		return (retval);
 	}
 
 	if (!busy_found) {
diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index 25c548fb3..a8dcfe0ad 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -400,11 +400,11 @@ cnt_lookup(struct worker *wrk, struct req *req)
 
 	AZ(req->objcore);
 	if (lr == HSH_MISS) {
-		/* Found nothing */
-		AZ(oc);
+		/* Found nothing or an object that was out of grace */
 		AN(boc);
 		AN(boc->flags & OC_F_BUSY);
 		req->objcore = boc;
+		req->stale_oc = oc;
 		req->req_step = R_STP_MISS;
 		return (REQ_FSM_MORE);
 	}
@@ -702,6 +702,7 @@ cnt_recv(struct worker *wrk, struct req *req)
 
 	req->vdp_retval = 0;
 	req->d_ttl = -1;
+	req->d_grace = -1;
 	req->disable_esi = 0;
 	req->hash_always_miss = 0;
 	req->hash_ignore_busy = 0;
diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c
index bd67551ef..a294ab831 100644
--- a/bin/varnishd/cache/cache_vrt_var.c
+++ b/bin/varnishd/cache/cache_vrt_var.c
@@ -361,7 +361,7 @@ VRT_l_beresp_storage_hint(VRT_CTX, const char *str, ...)
 
 /*--------------------------------------------------------------------*/
 
-#define REQ_VAR_L(nm, elem, type,extra)					\
+#define REQ_VAR_L(nm, elem, type, extra)				\
 									\
 void									\
 VRT_l_req_##nm(VRT_CTX, type arg)			\
@@ -386,6 +386,8 @@ REQ_VAR_L(backend_hint, director_hint, const struct director *,)
 REQ_VAR_R(backend_hint, director_hint, const struct director *)
 REQ_VAR_L(ttl, d_ttl, double, if (!(arg>0.0)) arg = 0;)
 REQ_VAR_R(ttl, d_ttl, double)
+REQ_VAR_L(grace, d_grace, double, if (!(arg>0.0)) arg = 0;)
+REQ_VAR_R(grace, d_grace, double)
 
 /*--------------------------------------------------------------------*/
 
diff --git a/bin/varnishtest/tests/b00098.vtc b/bin/varnishtest/tests/b00062.vtc
similarity index 100%
rename from bin/varnishtest/tests/b00098.vtc
rename to bin/varnishtest/tests/b00062.vtc
diff --git a/bin/varnishtest/tests/b00064.vtc b/bin/varnishtest/tests/b00064.vtc
new file mode 100644
index 000000000..8fee088d2
--- /dev/null
+++ b/bin/varnishtest/tests/b00064.vtc
@@ -0,0 +1,126 @@
+varnishtest "Test that req.grace will hold a client when a miss is anticipated"
+
+barrier b1 cond 2
+
+server s1 {
+	rxreq
+	expect req.url == "/"
+	txresp -body "0"
+
+	rxreq
+	expect req.url == "/"
+	txresp -body "1"
+
+	# second time we get a request, we use some time to serve it
+	rxreq
+	expect req.url == "/"
+	barrier b1 sync
+	delay .1
+	txresp -body "2"
+
+	# Last request, to a different URL to catch it if varnish asks for "/" too many times
+	rxreq
+	expect req.url == "/2"
+	txresp -body "x"
+} -start
+
+varnish v1 -vcl+backend {
+	import ${vmod_std};
+
+	sub vcl_recv {
+		# When we know in vcl_recv that we will have no grace, it is now
+		# possible to signal this to the lookup function:
+		if (req.http.X-no-grace) {
+			set req.grace = 0s;
+		}
+	}
+	sub vcl_hit {
+		set req.http.X-grace = obj.grace;
+	}
+	sub vcl_backend_response {
+		set beresp.ttl = 0.1s;
+		set beresp.grace = 1m;
+	}
+	sub vcl_deliver {
+		if (req.http.X-grace) {
+			set resp.http.X-grace = req.http.X-grace;
+			set resp.http.X-req-grace = req.grace;
+		}
+	}
+} -start
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.status == 200
+	expect resp.body == "0"
+	expect resp.http.X-grace == <undef>
+	# let the object's ttl expire
+	delay .2
+	# get a genuinely fresh object by disabling grace
+	# we will not get to vcl_hit to see the grace
+	txreq -hdr "X-no-grace: true"
+	rxresp
+	expect resp.status == 200
+	expect resp.body == "1"
+	expect resp.http.X-grace == <undef>
+	expect resp.http.X-req-grace == <undef>
+} -run
+
+# let the latest object's ttl expire.
+delay .2
+
+varnish v1 -expect n_object == 1
+
+# c2 asks for the object under grace
+client c2 {
+	txreq
+	rxresp
+	# we did not disable grace in the request, so we should get the graced object here
+	expect resp.status == 200
+	expect resp.body == "1"
+	expect resp.http.X-grace == "60.000"
+	expect resp.http.X-req-grace < 0.
+} -start
+
+delay .1
+
+# c3 asks for graced object, but now we disable grace.
+client c3 {
+	txreq -hdr "X-no-grace: true"
+	rxresp
+	expect resp.status == 200
+	# Here we have disable grace and should get the object from the background fetch,
+	# which will take us into vcl_hit
+	expect resp.body == "2"
+	expect resp.http.X-grace == "60.000"
+	expect resp.http.X-req-grace == "0.000"
+} -start
+
+delay .1
+
+# c4 does not disable grace, and should get the grace object even
+# though c3 is waiting on the background thread to deliver a new
+# version.
+
+client c4 {
+	txreq
+	rxresp
+	barrier b1 sync
+	expect resp.status == 200
+	# We should get what c1 got in the very beginning
+	expect resp.body == "1"
+	expect resp.http.X-grace == "60.000"
+	expect resp.http.X-req-grace < 0.
+} -start
+
+client c2 -wait
+client c3 -wait
+client c4 -wait
+
+client c5 {
+	txreq -url "/2"
+	rxresp
+	expect resp.status == 200
+	expect resp.body == "x"
+} -run
diff --git a/bin/varnishtest/tests/r02705.vtc b/bin/varnishtest/tests/r02705.vtc
new file mode 100644
index 000000000..54bec95c2
--- /dev/null
+++ b/bin/varnishtest/tests/r02705.vtc
@@ -0,0 +1,51 @@
+varnishtest "No vcl_hit when grace has run out, with working IMS"
+
+server s1 {
+	rxreq
+	txresp -hdr {Etag: "Foo"} -body "1"
+
+	rxreq
+	txresp -status 304
+
+	rxreq
+	expect req.url == "/2"
+	txresp -body "3"
+} -start
+
+varnish v1 -vcl+backend {
+	sub vcl_backend_response {
+		set beresp.ttl = 1ms;
+		set beresp.grace = 0s;
+		set beresp.keep = 1m;
+		if (bereq.http.Hit) {
+			set beresp.http.Hit = bereq.http.Hit;
+		}
+	}
+	sub vcl_hit {
+		set req.http.Hit = "HIT";
+	}
+} -start
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.http.Hit == <undef>
+	expect resp.http.Etag == {"Foo"}
+	expect resp.status == 200
+	expect resp.body == "1"
+	delay .2
+	txreq
+	rxresp
+	expect resp.http.Hit == <undef>
+	expect resp.http.Etag == {"Foo"}
+	expect resp.status == 200
+	expect resp.body == "1"
+} -run
+
+client c2 {
+	txreq -url "/2"
+	rxresp
+	expect resp.http.Hit == <undef>
+	expect resp.body == "3"
+} -run
+
diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py
index a370b3274..a02c40229 100755
--- a/lib/libvcc/generate.py
+++ b/lib/libvcc/generate.py
@@ -268,6 +268,12 @@ sp_variables = [
 		( 'client',), """
 		"""
 	),
+	('req.grace',
+		'DURATION',
+		( 'client',),
+		( 'client',), """
+		"""
+	),
 	('req.xid',
 		'STRING',
 		( 'client',),

From hermunn at varnish-software.com  Thu Sep 13 13:03:12 2018
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Thu, 13 Sep 2018 13:03:12 +0000 (UTC)
Subject: [4.1] a8dc87802 Test case that shows return(abandon) to avoid cache
 insertion
Message-ID: <20180913130312.44B7EA2FFC@lists.varnish-cache.org>


commit a8dc87802fa83ec64eb762bce2a1107af09b7d6b
Author: P?l Hermunn Johansen <hermunn at varnish-software.com>
Date:   Fri Jun 15 10:54:48 2018 +0200

    Test case that shows return(abandon) to avoid cache insertion

diff --git a/bin/varnishtest/tests/b00063.vtc b/bin/varnishtest/tests/b00063.vtc
new file mode 100644
index 000000000..1adfe2680
--- /dev/null
+++ b/bin/varnishtest/tests/b00063.vtc
@@ -0,0 +1,90 @@
+varnishtest "Abandon background fetch when backend serves 5xx"
+
+barrier b1 cond 2
+barrier b2 cond 3
+
+server s1 {
+	# This is what we want to get to all client requests below
+	rxreq
+	expect req.url == "/1"
+	txresp -body "1"
+
+	# 503s will be abandoned when we have a bgfetch
+	rxreq
+	expect req.url == "/1"
+	txresp -status 503 -body "2"
+
+	# varnish will disconnect on a 503
+	accept
+	rxreq
+	expect req.url == "/1"
+	# wait until varnish has delivered 200 before replying
+	# with the 404
+	barrier b2 sync
+	delay .1
+	# this response will not be abandoned
+	txresp -status 404 -reason "Not Found" -body "3"
+
+	# some other resource at the end
+	rxreq
+	expect req.url == "/2"
+	txresp -body "4"
+} -start
+
+varnish v1 -vcl+backend {
+	sub vcl_backend_response {
+		if (beresp.status >= 500 && bereq.is_bgfetch) {
+			return (abandon);
+		}
+		if (beresp.status >= 400) {
+			set beresp.ttl = 1m;
+		} else {
+			set beresp.ttl = 1ms;
+		}
+		set beresp.grace = 1m;
+	}
+} -start
+
+client c1 {
+	txreq -url "/1"
+	rxresp
+	expect resp.status == 200
+	expect resp.body == "1"
+	delay .2
+	txreq -url "/1"
+	rxresp
+	expect resp.status == 200
+	expect resp.body == "1"
+	delay .2
+	barrier b1 sync
+	txreq -url "/1"
+	rxresp
+	expect resp.status == 200
+	expect resp.body == "1"
+	barrier b2 sync
+} -start
+
+client c2 {
+	barrier b1 sync
+	txreq -url "/1"
+	rxresp
+	expect resp.status == 200
+	expect resp.body == "1"
+	barrier b2 sync
+} -start
+
+client c1 -wait
+client c2 -wait
+
+client c3 {
+	delay .1
+	# We should now get a HIT on the 404:
+	txreq -url "/1"
+	rxresp
+	expect resp.status == 404
+	expect resp.body == "3"
+	# do a different resource to make sure we got the right number of reqs to /1
+	txreq -url "/2"
+	rxresp
+	expect resp.body == "4"
+} -run

From hermunn at varnish-software.com  Thu Sep 13 13:03:12 2018
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Thu, 13 Sep 2018 13:03:12 +0000 (UTC)
Subject: [4.1] 62163f4b6 Simplify cnt_lookup
Message-ID: <20180913130312.5C7A7A2FFF@lists.varnish-cache.org>


commit 62163f4b696683cc06c001532bd0641d2cfffbce
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Tue Jun 19 15:43:25 2018 +0200

    Simplify cnt_lookup
    
    This is a back port of 4a370dc4f91706b19c2e3d3103c4f13957349bbc
    
    Conflicts:
            bin/varnishd/cache/cache_req_fsm.c

diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index a8dcfe0ad..1aa432dfd 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -400,12 +400,17 @@ cnt_lookup(struct worker *wrk, struct req *req)
 
 	AZ(req->objcore);
 	if (lr == HSH_MISS) {
-		/* Found nothing or an object that was out of grace */
-		AN(boc);
-		AN(boc->flags & OC_F_BUSY);
-		req->objcore = boc;
-		req->stale_oc = oc;
-		req->req_step = R_STP_MISS;
+		if (boc != NULL) {
+			/* out-of-grace or ordinary miss */
+			AN(boc->flags & OC_F_BUSY);
+			req->objcore = boc;
+			req->stale_oc = oc;
+			req->req_step = R_STP_MISS;
+		} else {
+			/* hitpass */
+			AZ(oc);
+			req->req_step = R_STP_PASS;
+		}
 		return (REQ_FSM_MORE);
 	}
 

From dridi.boukelmoune at gmail.com  Fri Sep 14 12:42:13 2018
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Fri, 14 Sep 2018 12:42:13 +0000 (UTC)
Subject: [master] 7dc80a1c6 Simple addition on req.grace
Message-ID: <20180914124214.055BC91811@lists.varnish-cache.org>


commit 7dc80a1c6654b7edd9f4e5fade007c8531b9dcbe
Author: P?l Hermunn Johansen <hermunn at varnish-software.com>
Date:   Fri Sep 14 14:33:29 2018 +0200

    Simple addition on req.grace

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 2a33f67e0..73b8dfb72 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -88,9 +88,14 @@ bit. ``keep`` sets a lifetime in the cache in addition to TTL for
 objects that can be validated by a 304 "Not Modified" response from
 the backend to a conditional request (with ``If-None-Match`` or
 ``If-Modified-Since``). If an expired object is also out of grace
-time, it is no longer possible to deliver a "keep" object from
-``vcl_hit``. It is possible to validate a 304 candidate from
-``vcl_miss``.
+time, then ``vcl_hit`` will no longer be called, so it is impossible
+to deliver the "keep" object in this case.
+
+Note that the headers ``If-None-Match`` and ``If-Modified-Since``,
+together with the 304 behavior, are handled automatically by Varnish.
+If you, for some reason, need to explicitly disable this for a backend
+request, then you need do this by removing the headers in
+``vcl_backend_fetch``.
 
 The documentation in :ref:`users-guide-handling_misbehaving_servers`
 has been expanded to discuss these matters in greater depth, look

From geoff at uplex.de  Fri Sep 14 15:00:17 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Fri, 14 Sep 2018 15:00:17 +0000 (UTC)
Subject: [master] 2a785831a Formatting fix in the release notes
Message-ID: <20180914150018.140E8943D6@lists.varnish-cache.org>


commit 2a785831abee9cb727e0765bf90ead0f54943b31
Author: Geoff Simmons <geoff at uplex.de>
Date:   Fri Sep 14 16:59:27 2018 +0200

    Formatting fix in the release notes

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 73b8dfb72..65174d7e8 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -388,7 +388,7 @@ Other changes
     accordingly. In short, the most important changes include:
 
     * ``struct director_methods`` is replaced by ``struct vdi_methods``
-      * signatures of various callbacks have changed
+    * signatures of various callbacks have changed
     * ``VRT_AddDirector()`` and ``VRT_DelDirector()`` are to be used
       for initialization and destruction.
     * ``vdi_methods`` callbacks are not to be called from vmods any more

From geoff at uplex.de  Sun Sep 16 16:46:12 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Sun, 16 Sep 2018 16:46:12 +0000 (UTC)
Subject: [master] 7478e3b0e Remove XXX's and other placeholders from
 'Upgrading to 6.1'.
Message-ID: <20180916164612.AB68EA9527@lists.varnish-cache.org>


commit 7478e3b0e406badc3f470c9f9abf7e5d43d91156
Author: Geoff Simmons <geoff at uplex.de>
Date:   Sun Sep 16 18:44:56 2018 +0200

    Remove XXX's and other placeholders from 'Upgrading to 6.1'.

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 65174d7e8..131fd73a8 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -1,21 +1,9 @@
 .. _whatsnew_upgrading_6.1:
 
-**NOTE: The present document is work in progress for the September
-2018 release.** The version number 6.1.0 is provisional and may
-change. See :ref:`whatsnew_upgrading_6.0` for notes about the
-currently most recent Varnish release.
-
 %%%%%%%%%%%%%%%%%%%%%%%%
 Upgrading to Varnish 6.1
 %%%%%%%%%%%%%%%%%%%%%%%%
 
-.. _upd_6_1_headline:
-
-**Headline Changes**
-====================
-
-**XXX**
-
 varnishd parameters
 ===================
 
@@ -45,11 +33,6 @@ for details.
 Changes to VCL
 ==============
 
-**Headline VCL changes**
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-**XXX**
-
 VCL variables
 ~~~~~~~~~~~~~
 
@@ -160,11 +143,6 @@ you now have the option.
 entails changing the privilege set of the child process while the VMOD
 is loaded, see the documentation.
 
-**anything else**
-=================
-
-**XXX**
-
 Other changes
 =============
 
@@ -320,8 +298,6 @@ Other changes
   * Added the ``-need-bytes`` argument for the ``process`` command,
     see :ref:`vtc(7)`.
 
-  * **XXX**
-
 * ``varnishhist(1)``:
 
   * The ``-P min:max`` command-line parameters are now optional,

From geoff at uplex.de  Sun Sep 16 16:48:06 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Sun, 16 Sep 2018 16:48:06 +0000 (UTC)
Subject: [master] 355d63161 Remove another placeholder from the 6.1 release
 docs.
Message-ID: <20180916164806.BADBEA96ED@lists.varnish-cache.org>


commit 355d63161295605528fa38fd9748369a53027b07
Author: Geoff Simmons <geoff at uplex.de>
Date:   Sun Sep 16 18:47:12 2018 +0200

    Remove another placeholder from the 6.1 release docs.

diff --git a/doc/sphinx/whats-new/index.rst b/doc/sphinx/whats-new/index.rst
index 54438108b..f7b76016b 100644
--- a/doc/sphinx/whats-new/index.rst
+++ b/doc/sphinx/whats-new/index.rst
@@ -11,10 +11,6 @@ entail.
 Varnish 6.1
 -----------
 
-**NOTE: This documentation is work in progress for the September 2018
-release.** The version number 6.1.0 is provisional and may change. See
-the links listed below for notes about released versions of Varnish.
-
 .. toctree::
    :maxdepth: 2
 

From geoff at uplex.de  Sun Sep 16 17:20:08 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Sun, 16 Sep 2018 17:20:08 +0000 (UTC)
Subject: [master] 1bfed1784 Add a brief intro to 'Upgrading to 6.1'.
Message-ID: <20180916172008.A2642AC0A9@lists.varnish-cache.org>


commit 1bfed17848892445faff8338fff1eb59a3eb5ad4
Author: Geoff Simmons <geoff at uplex.de>
Date:   Sun Sep 16 19:19:01 2018 +0200

    Add a brief intro to 'Upgrading to 6.1'.

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 131fd73a8..94723e600 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -4,6 +4,12 @@
 Upgrading to Varnish 6.1
 %%%%%%%%%%%%%%%%%%%%%%%%
 
+A configuration for Varnish 6.0.x will run for version 6.1 without
+changes.  There has been a subtle change in the interpretation of the
+VCL variable ``beresp.keep`` under specific circumstances, as
+discussed below. Other than that, the changes in 6.1 are new features,
+described in the following.
+
 varnishd parameters
 ===================
 

From phk at FreeBSD.org  Mon Sep 17 06:42:11 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 17 Sep 2018 06:42:11 +0000 (UTC)
Subject: [master] a3a820fb9 Don't document Debug VSL's,
 they are not a supported and stable facility.
Message-ID: <20180917064211.87E446382F@lists.varnish-cache.org>


commit a3a820fb9b6311655545957ace4d2345e2c50e6a
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 17 06:41:15 2018 +0000

    Don't document Debug VSL's, they are not a supported and stable
    facility.

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 94723e600..c7e8686c9 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -181,12 +181,6 @@ Other changes
     In particular, http connection (HTC) errors are now reported
     symbolically in addition to the previous numerical value.
 
-  * ``Debug`` log entries may also give more diagnostic information
-    about session accept failures (failure to accept a client
-    connection). These must be viewed in raw grouping, since accept
-    failures are not part of any request/response transaction. The
-    ``Debug`` message begins with the phrase ``"Accept failed"``.
-
   * When a backend is unhealthy, ``Backend_health`` now reports some
     diagnostic information in addition to the HTTP response and timing
     information.

From phk at FreeBSD.org  Mon Sep 17 06:58:07 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 17 Sep 2018 06:58:07 +0000 (UTC)
Subject: [master] 9e90dcba0 Note reproducible builds
Message-ID: <20180917065807.58CC163ECF@lists.varnish-cache.org>


commit 9e90dcba0a28da1d3c67d4e7096a0ba1cad70c74
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 17 06:56:52 2018 +0000

    Note reproducible builds

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index c7e8686c9..8592ce30d 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -376,4 +376,7 @@ Other changes
   * Python 3 is now preferred in builds, and will likely be required
     in future versions.
 
+  * We belive builds are now reproducible, and intend to keep them
+    that way.
+
 *eof*

From phk at FreeBSD.org  Mon Sep 17 07:26:08 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 17 Sep 2018 07:26:08 +0000 (UTC)
Subject: [master] c6a10f462 Nothing much to put here this time.
Message-ID: <20180917072608.2E0A96477C@lists.varnish-cache.org>


commit c6a10f46237bac3b879c29f59923f551dec3dd47
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 17 07:25:20 2018 +0000

    Nothing much to put here this time.

diff --git a/doc/sphinx/whats-new/changes-6.1.rst b/doc/sphinx/whats-new/changes-6.1.rst
index ee3644694..37e2a3e83 100644
--- a/doc/sphinx/whats-new/changes-6.1.rst
+++ b/doc/sphinx/whats-new/changes-6.1.rst
@@ -1,13 +1,23 @@
 .. _whatsnew_changes_6.1:
 
-**NOTE: The present document is work in progress for the September
-2018 release.** The version number 6.1.0 is provisional and may
-change. See :ref:`whatsnew_changes_6.0` for notes about the currently
-most recent Varnish release.
-
 Changes in Varnish 6.1
 ======================
 
-**XXX**
+This is release is a maintenance release, so while there are many actual
+changes, and of course many bugfixes, they should not have little to no
+impact on running Varnish installations.
+
+Nothing to see here, really
+---------------------------
+
+Since new users often forget to `vcl.discard` their old VCLs, we have
+added a warning when you have more than 100 VCLs loaded.  There are
+parameters to set the threshold and decide what happens when it is
+exceeded (ignore/warn/error).
+
+We have made `req.http.Host` mandatory and handle requests without it
+on the fast DoS avoidance path.
+
+For all the details and new stuff, see :ref:`whatsnew_changes_6.1`
 
 *eof*

From phk at FreeBSD.org  Mon Sep 17 07:28:06 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 17 Sep 2018 07:28:06 +0000 (UTC)
Subject: [master] 537f890a9 But don't make it self-referential
Message-ID: <20180917072806.D036C64956@lists.varnish-cache.org>


commit 537f890a9a8c4e07ef191556e70e745bb9a1c61a
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 17 07:27:25 2018 +0000

    But don't make it self-referential

diff --git a/doc/sphinx/whats-new/changes-6.1.rst b/doc/sphinx/whats-new/changes-6.1.rst
index 37e2a3e83..3ceffc093 100644
--- a/doc/sphinx/whats-new/changes-6.1.rst
+++ b/doc/sphinx/whats-new/changes-6.1.rst
@@ -18,6 +18,6 @@ exceeded (ignore/warn/error).
 We have made `req.http.Host` mandatory and handle requests without it
 on the fast DoS avoidance path.
 
-For all the details and new stuff, see :ref:`whatsnew_changes_6.1`
+For all the details and new stuff, see :ref:`whatsnew_upgrading_6.1`
 
 *eof*

From dridi.boukelmoune at gmail.com  Mon Sep 17 07:43:11 2018
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 17 Sep 2018 07:43:11 +0000 (UTC)
Subject: [master] 37c42ca83 s/belive/believe/
Message-ID: <20180917074312.24FAD64EF6@lists.varnish-cache.org>


commit 37c42ca83b7a89b41252d70fb41c050440ead228
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Mon Sep 17 09:41:40 2018 +0200

    s/belive/believe/

diff --git a/doc/sphinx/phk/firstdesign.rst b/doc/sphinx/phk/firstdesign.rst
index 32f27b2a2..fee1943f8 100644
--- a/doc/sphinx/phk/firstdesign.rst
+++ b/doc/sphinx/phk/firstdesign.rst
@@ -176,7 +176,7 @@ where we kept thinking in the old frame of reference (ie: Squid)::
 	to ask are:
 
 		How long time after the expiry can we serve a cached copy
-		of this document while we have reason to belive the backend
+		of this document while we have reason to believe the backend
 		can supply us with an update ?
 
 		How long time after the expiry can we serve a cached copy
diff --git a/doc/sphinx/whats-new/changes-6.0.rst b/doc/sphinx/whats-new/changes-6.0.rst
index 55001dde4..41fd5052e 100644
--- a/doc/sphinx/whats-new/changes-6.0.rst
+++ b/doc/sphinx/whats-new/changes-6.0.rst
@@ -103,7 +103,7 @@ variant symbols, presently symbols which are different in
 ``vcl 4.0`` and ``vcl 4.1``.
 
 The "prototype" information in the VMOD shared library has been
-changed to JSON, (look in your vcc_if.c file if you don't belive
+changed to JSON, (look in your vcc_if.c file if you don't believe
 me), and this can express more detailed information, presently
 the optional arguments.
 
diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 8592ce30d..f78ed855d 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -376,7 +376,7 @@ Other changes
   * Python 3 is now preferred in builds, and will likely be required
     in future versions.
 
-  * We belive builds are now reproducible, and intend to keep them
+  * We believe builds are now reproducible, and intend to keep them
     that way.
 
 *eof*

From phk at FreeBSD.org  Mon Sep 17 07:44:07 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 17 Sep 2018 07:44:07 +0000 (UTC)
Subject: [master] 2cb5e5326 Fix production of VTC documentation.
Message-ID: <20180917074407.08DB46505A@lists.varnish-cache.org>


commit 2cb5e5326ed8d56abfade2454f5a90237c919d29
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 17 07:42:57 2018 +0000

    Fix production of VTC documentation.
    
    Fixes: #2777

diff --git a/bin/varnishtest/vtc_misc.c b/bin/varnishtest/vtc_misc.c
index 8ffe69643..196b35c65 100644
--- a/bin/varnishtest/vtc_misc.c
+++ b/bin/varnishtest/vtc_misc.c
@@ -217,6 +217,8 @@ cmd_shell(CMD_ARGS)
 }
 
 /* SECTION: err_shell err_shell
+ *
+ * NOTICE: err_shell is deprecated, use `shell -err -expect` instead.
  *
  * This is very similar to the the ``shell`` command, except it takes a first
  * string as argument before the command::
diff --git a/doc/sphinx/Makefile.am b/doc/sphinx/Makefile.am
index 5eb2a1361..54f930caf 100644
--- a/doc/sphinx/Makefile.am
+++ b/doc/sphinx/Makefile.am
@@ -188,6 +188,7 @@ VTCSYN_SRC = $(top_srcdir)/bin/varnishtest/vtc.c \
 	     $(top_srcdir)/bin/varnishtest/vtc_http.c \
 	     $(top_srcdir)/bin/varnishtest/vtc_http2.c \
 	     $(top_srcdir)/bin/varnishtest/vtc_logexp.c \
+	     $(top_srcdir)/bin/varnishtest/vtc_misc.c \
 	     $(top_srcdir)/bin/varnishtest/vtc_process.c \
 	     $(top_srcdir)/bin/varnishtest/vtc_syslog.c \
 	     $(top_srcdir)/bin/varnishtest/vtc_varnish.c

From nils.goroll at uplex.de  Mon Sep 17 08:21:08 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon, 17 Sep 2018 08:21:08 +0000 (UTC)
Subject: [master] 0c823a89e an attempt to clarify req.grace/req.ttl
Message-ID: <20180917082108.301B765BA9@lists.varnish-cache.org>


commit 0c823a89e5303ce8b2aeb264cacc6b040401ef6d
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Sep 17 10:19:58 2018 +0200

    an attempt to clarify req.grace/req.ttl

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index f78ed855d..3a0f8452a 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -49,11 +49,13 @@ VCL variables
 since there are use cases that cannot be solved without it. Similarly,
 ``req.ttl`` used to be deprecated and is now fully supported again.
 
-``req.ttl`` and ``req.grace`` set upper bounds on the TTL and grace
-times that are permitted for the current request -- if these variables
-are set and the TTL/grace of a cache object is longer than their
-settings, then a new response is fetched from the backend, despite the
-presence of an otherwise fresh cache object.
+``req.ttl`` and ``req.grace`` limit the ttl and grace times that are
+permitted for the current request. If ``req.ttl`` is set, it requires
+cache objects to have that minimum remaining ttl to be considered
+fresh. Likewise, ``req.grace`` sets an upper bound on the time an
+object has spent in grace to be considered eligible for grace mode
+(which is to deliver this object and fetch a fresh copy in the
+background).
 
 A common application is to set shorter TTLs when the backend is known
 to be healthy, so that responses are fresher when all is well. But if

From geoff at uplex.de  Mon Sep 17 08:50:14 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Mon, 17 Sep 2018 08:50:14 +0000 (UTC)
Subject: [master] 78ab678f3 Another rewording for req.ttl in the 6.1 release
 notes.
Message-ID: <20180917085015.3D2526E5B5@lists.varnish-cache.org>


commit 78ab678f3372c964a664a3a519af3e567b4022cd
Author: Geoff Simmons <geoff at uplex.de>
Date:   Mon Sep 17 10:48:39 2018 +0200

    Another rewording for req.ttl in the 6.1 release notes.

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 3a0f8452a..0f175b10f 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -50,12 +50,12 @@ since there are use cases that cannot be solved without it. Similarly,
 ``req.ttl`` used to be deprecated and is now fully supported again.
 
 ``req.ttl`` and ``req.grace`` limit the ttl and grace times that are
-permitted for the current request. If ``req.ttl`` is set, it requires
-cache objects to have that minimum remaining ttl to be considered
-fresh. Likewise, ``req.grace`` sets an upper bound on the time an
-object has spent in grace to be considered eligible for grace mode
-(which is to deliver this object and fetch a fresh copy in the
-background).
+permitted for the current request. If ``req.ttl`` is set, then cache
+objects are considered fresh (and may be cache hits) only if their
+remaining ttl is less than or equal to ``req.ttl``. Likewise,
+``req.grace`` sets an upper bound on the time an object has spent in
+grace to be considered eligible for grace mode (which is to deliver
+this object and fetch a fresh copy in the background).
 
 A common application is to set shorter TTLs when the backend is known
 to be healthy, so that responses are fresher when all is well. But if

From dridi.boukelmoune at gmail.com  Mon Sep 17 09:05:15 2018
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Mon, 17 Sep 2018 09:05:15 +0000 (UTC)
Subject: [master] 978bba40e Clarify what Debug means
Message-ID: <20180917090515.A95236EB82@lists.varnish-cache.org>


commit 978bba40ef426e2a65a4621c9c393e215cf0cbda
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Mon Sep 17 11:03:17 2018 +0200

    Clarify what Debug means

diff --git a/include/tbl/vsl_tags.h b/include/tbl/vsl_tags.h
index 916fc4d79..6c74b4429 100644
--- a/include/tbl/vsl_tags.h
+++ b/include/tbl/vsl_tags.h
@@ -51,6 +51,8 @@ SLTM(Debug, SLT_F_UNSAFE, "Debug messages",
 	"Debug messages can normally be ignored, but are sometimes"
 	" helpful during trouble-shooting.  Most debug messages must"
 	" be explicitly enabled with parameters.\n\n"
+	"Debug messages may be added, changed or removed without"
+	" prior notice and shouldn't be considered stable.\n\n"
 )
 
 SLTM(Error, 0, "Error messages",

From geoff at uplex.de  Mon Sep 17 09:25:08 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Mon, 17 Sep 2018 09:25:08 +0000 (UTC)
Subject: [master] 5275550c0 Grammar fix
Message-ID: <20180917092508.A42D1912CD@lists.varnish-cache.org>


commit 5275550c007e8f288e704189ed711da71b70ea7f
Author: Geoff Simmons <geoff at uplex.de>
Date:   Mon Sep 17 11:24:02 2018 +0200

    Grammar fix

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 0f175b10f..e70a0ddde 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -278,7 +278,7 @@ Other changes
 
 * ``varnishtest(1)`` and ``vtc(7)``:
 
-  * ``varnishtest`` and the ``vtc`` test script language now supports
+  * ``varnishtest`` and the ``vtc`` test script language now support
     testing for haproxy as well as Varnish. The ``haproxy`` directive
     in a test can be used to define, configure, start and stop a
     haproxy instance, and you can also script messages to send on the

From nils.goroll at uplex.de  Mon Sep 17 09:30:16 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon, 17 Sep 2018 09:30:16 +0000 (UTC)
Subject: [master] 515b59da9 mention UDS for SessError/SessOpen
Message-ID: <20180917093016.48446915C9@lists.varnish-cache.org>


commit 515b59da94bb2b8382c777374de03e33adc944fe
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Sep 17 11:27:16 2018 +0200

    mention UDS for SessError/SessOpen

diff --git a/include/tbl/vsl_tags.h b/include/tbl/vsl_tags.h
index d682cf194..0997d72f8 100644
--- a/include/tbl/vsl_tags.h
+++ b/include/tbl/vsl_tags.h
@@ -73,8 +73,8 @@ SLTM(SessError, 0, "Client connection accept failed",
 	"\t|  |  |  |  |  +- Detailed error message\n"
 	"\t|  |  |  |  +---- Error Number (errno) from accept(2)\n"
 	"\t|  |  |  +------- File descriptor number\n"
-	"\t|  |  +---------- Local TCP port\n"
-	"\t|  +------------- Local IPv4/6 address\n"
+	"\t|  |  +---------- Local TCP port / 0 for UDS\n"
+	"\t|  +------------- Local IPv4/6 address / 0.0.0.0 for UDS\n"
 	"\t+---------------- Socket name (from -a argument)\n"
 	"\n"
 )
@@ -86,11 +86,11 @@ SLTM(SessOpen, 0, "Client connection opened",
 	"\t%s %d %s %s %s %d\n"
 	"\t|  |  |  |  |  |\n"
 	"\t|  |  |  |  |  +- File descriptor number\n"
-	"\t|  |  |  |  +---- Local TCP port\n"
-	"\t|  |  |  +------- Local IPv4/6 address\n"
+	"\t|  |  |  |  +---- Local TCP port / 0 for UDS\n"
+	"\t|  |  |  +------- Local IPv4/6 address / 0.0.0.0 for UDS\n"
 	"\t|  |  +---------- Socket name (from -a argument)\n"
-	"\t|  +------------- Remote TCP port\n"
-	"\t+---------------- Remote IPv4/6 address\n"
+	"\t|  +------------- Remote TCP port / 0 for UDS\n"
+	"\t+---------------- Remote IPv4/6 address / 0.0.0.0 for UDS\n"
 	"\n"
 )
 

From nils.goroll at uplex.de  Mon Sep 17 09:30:16 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon, 17 Sep 2018 09:30:16 +0000 (UTC)
Subject: [master] be6949921 log accept failures under the SessError tag
Message-ID: <20180917093016.3A05F915C8@lists.varnish-cache.org>


commit be69499219db0704a136046369884531ee20bc01
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Sep 17 11:14:01 2018 +0200

    log accept failures under the SessError tag
    
    I consider it too late for a vtc, but this has been tested manually
    with ulimit -n 40. Example errors (tcp, uds)
    
             0 SessError      - a0 127.0.0.1 8080 3 24 Too many open files
             0 SessError      - a1 0.0.0.0 0 5 24 Too many open files

diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c
index 908613f47..2b920720a 100644
--- a/bin/varnishd/cache/cache_acceptor.c
+++ b/bin/varnishd/cache/cache_acceptor.c
@@ -451,6 +451,8 @@ vca_accept_task(struct worker *wrk, void *arg)
 	struct poolsock *ps;
 	struct listen_sock *ls;
 	int i;
+	char laddr[VTCP_ADDRBUFSIZE];
+	char lport[VTCP_PORTBUFSIZE];
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CAST_OBJ_NOTNULL(ps, arg, POOLSOCK_MAGIC);
@@ -510,9 +512,21 @@ vca_accept_task(struct worker *wrk, void *arg)
 				vca_pace_bad();
 				break;
 			}
+
+			i = errno;
 			wrk->stats->sess_fail++;
-			VSL(SLT_Debug, ls->sock, "Accept failed: %s",
-			    strerror(errno));
+
+			if (wa.acceptlsock->uds) {
+				strcpy(laddr, "0.0.0.0");
+				strcpy(lport, "0");
+			} else {
+				VTCP_myname(ls->sock, laddr, VTCP_ADDRBUFSIZE,
+				    lport, VTCP_ADDRBUFSIZE);
+			}
+
+			VSL(SLT_SessError, 0, "%s %s %s %d %d %s",
+			    wa.acceptlsock->name, laddr, lport,
+			    ls->sock, i, strerror(i));
 			(void)Pool_TrySumstat(wrk);
 			continue;
 		}
diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index e70a0ddde..f6e53da1e 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -183,6 +183,12 @@ Other changes
     In particular, http connection (HTC) errors are now reported
     symbolically in addition to the previous numerical value.
 
+  * Log entries under the new ``SessError`` tag now give more
+    diagnostic information about session accept failures (failure to
+    accept a client connection). These must be viewed in raw grouping,
+    since accept failures are not part of any request/response
+    transaction.
+
   * When a backend is unhealthy, ``Backend_health`` now reports some
     diagnostic information in addition to the HTTP response and timing
     information.
diff --git a/include/tbl/vsl_tags.h b/include/tbl/vsl_tags.h
index 6c74b4429..d682cf194 100644
--- a/include/tbl/vsl_tags.h
+++ b/include/tbl/vsl_tags.h
@@ -65,6 +65,20 @@ SLTM(CLI, 0, "CLI communication",
 
 /*---------------------------------------------------------------------*/
 
+SLTM(SessError, 0, "Client connection accept failed",
+	"Accepting a client connection has failed.\n\n"
+	"The format is::\n\n"
+	"\t%s %s %s %d %d %s\n"
+	"\t|  |  |  |  |  |\n"
+	"\t|  |  |  |  |  +- Detailed error message\n"
+	"\t|  |  |  |  +---- Error Number (errno) from accept(2)\n"
+	"\t|  |  |  +------- File descriptor number\n"
+	"\t|  |  +---------- Local TCP port\n"
+	"\t|  +------------- Local IPv4/6 address\n"
+	"\t+---------------- Socket name (from -a argument)\n"
+	"\n"
+)
+
 SLTM(SessOpen, 0, "Client connection opened",
 	"The first record for a client connection, with the socket-endpoints"
 	" of the connection.\n\n"

From geoff at uplex.de  Mon Sep 17 09:48:13 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Mon, 17 Sep 2018 09:48:13 +0000 (UTC)
Subject: [master] 9f6dd411a 6.1 release notes reference SessError to diagnose
 sess_fail_other.
Message-ID: <20180917094813.BE7A291EF0@lists.varnish-cache.org>


commit 9f6dd411ae558868948a38e366d2df319c053f53
Author: Geoff Simmons <geoff at uplex.de>
Date:   Mon Sep 17 11:46:33 2018 +0200

    6.1 release notes reference SessError to diagnose sess_fail_other.

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index f6e53da1e..27d5e4acb 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -271,7 +271,7 @@ Other changes
 
     * ``sess_fail_other``: number of accept failures for reasons
       other than those given by the other ``sess_fail_*`` counters.
-      More details may appear in the ``Debug`` entry of the log
+      More details may appear in the ``SessError`` entry of the log
       (:ref:`varnish-counters(7)` shows a ``varnishlog`` invocation
       that may help).
 

From geoff at uplex.de  Mon Sep 17 09:48:13 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Mon, 17 Sep 2018 09:48:13 +0000 (UTC)
Subject: [master] fc11b2419 VSC sess_fail_other docs reference the SessError
 log entry.
Message-ID: <20180917094813.B4C0B91EEF@lists.varnish-cache.org>


commit fc11b24198579b209f6ceeeecfdb38ba869850cc
Author: Geoff Simmons <geoff at uplex.de>
Date:   Mon Sep 17 11:45:31 2018 +0200

    VSC sess_fail_other docs reference the SessError log entry.

diff --git a/bin/varnishd/VSC_main.vsc b/bin/varnishd/VSC_main.vsc
index 56ad787b2..0bd491ca6 100644
--- a/bin/varnishd/VSC_main.vsc
+++ b/bin/varnishd/VSC_main.vsc
@@ -72,7 +72,7 @@
 	:oneliner:	Session accept failures: other
 
 	Detailed reason for sess_fail: neither of the above, see
-	Debug log (varnishlog -g raw -I Debug:^Accept).
+	SessError log (varnishlog -g raw -i SessError).
 
 .. varnish_vsc:: client_req_400
 	:oneliner:	Client requests received, subject to 400 errors

From nils.goroll at uplex.de  Mon Sep 17 10:01:08 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Mon, 17 Sep 2018 10:01:08 +0000 (UTC)
Subject: [master] 050a61569 docfix VBE.*.fail reference to the log
Message-ID: <20180917100108.321F392664@lists.varnish-cache.org>


commit 050a615699efdccfee29758e659e30f96376de1a
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Mon Sep 17 12:00:35 2018 +0200

    docfix VBE.*.fail reference to the log

diff --git a/bin/varnishd/VSC_vbe.vsc b/bin/varnishd/VSC_vbe.vsc
index cc0559831..f513e41ea 100644
--- a/bin/varnishd/VSC_vbe.vsc
+++ b/bin/varnishd/VSC_vbe.vsc
@@ -100,7 +100,7 @@
 	:oneliner:	Connections failed
 
 	Counter of failed opens. Detailed reasons are given in the
-	fail_* counters (DIAG level) and in Debug VSL.
+	fail_* counters (DIAG level) and in the log under the FetchError tag.
 
 	This counter is the sum of all detailed fail_* counters.
 

From geoff at uplex.de  Mon Sep 17 11:16:08 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Mon, 17 Sep 2018 11:16:08 +0000 (UTC)
Subject: [master] f223c6d5e Document the VRT version bump in the 6.1 release
 notes.
Message-ID: <20180917111608.E5AAC93E41@lists.varnish-cache.org>


commit f223c6d5e0f2731c8f33eb5b856952acb2090c40
Author: Geoff Simmons <geoff at uplex.de>
Date:   Mon Sep 17 13:15:13 2018 +0200

    Document the VRT version bump in the 6.1 release notes.

diff --git a/doc/sphinx/whats-new/upgrading-6.1.rst b/doc/sphinx/whats-new/upgrading-6.1.rst
index 27d5e4acb..a082b3cc5 100644
--- a/doc/sphinx/whats-new/upgrading-6.1.rst
+++ b/doc/sphinx/whats-new/upgrading-6.1.rst
@@ -329,6 +329,9 @@ Other changes
   * The Varnish API soname version (for libvarnishapi.so) has been
     bumped to 2.0.0.
 
+  * The VRT version has been bumped to 8.0. See ``vrt.h`` for details
+    on the changes since 7.0.
+
   * Space required by varnish for maintaining the ``PRIV_TASK`` and
     ``PRIV_TOP`` parameters is now taken from the appropriate
     workspace rather than from the heap as before. For a failing

From geoff at uplex.de  Mon Sep 17 11:16:08 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Mon, 17 Sep 2018 11:16:08 +0000 (UTC)
Subject: [master] 9ecac3938 Bump the VRT version to 8.0.
Message-ID: <20180917111608.DC0EF93E3F@lists.varnish-cache.org>


commit 9ecac3938bc09ae36e797f5b9d5fdb3ece34e652
Author: Geoff Simmons <geoff at uplex.de>
Date:   Mon Sep 17 13:14:37 2018 +0200

    Bump the VRT version to 8.0.

diff --git a/include/vrt.h b/include/vrt.h
index 9d11f7565..f7508446e 100644
--- a/include/vrt.h
+++ b/include/vrt.h
@@ -52,7 +52,7 @@
  * binary/load-time compatible, increment MAJOR version
  *
  *
- * TRUNK (2018-09-15)
+ * 8.0 (2018-09-15)
  *	VRT_Strands() added
  *	VRT_StrandsWS() added
  *	VRT_CollectStrands() added
@@ -114,7 +114,7 @@
  *	vrt_acl type added
  */
 
-#define VRT_MAJOR_VERSION	7U
+#define VRT_MAJOR_VERSION	8U
 
 #define VRT_MINOR_VERSION	0U
 

From hermunn at varnish-software.com  Mon Sep 17 12:02:13 2018
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Mon, 17 Sep 2018 12:02:13 +0000 (UTC)
Subject: [master] 4684c38ec Prepare for 6.1.0
Message-ID: <20180917120214.013C694F94@lists.varnish-cache.org>


commit 4684c38ecfc194b4f3b5b81594832dbb197a3bb9
Author: P?l Hermunn Johansen <hermunn at varnish-software.com>
Date:   Mon Sep 17 13:26:30 2018 +0200

    Prepare for 6.1.0

diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c
index dcf6e9df0..c1eece20d 100644
--- a/bin/varnishd/cache/cache_req_fsm.c
+++ b/bin/varnishd/cache/cache_req_fsm.c
@@ -150,7 +150,7 @@ cnt_deliver(struct worker *wrk, struct req *req)
 	http_PrintfHeader(req->resp, "Age: %.0f",
 	    floor(fmax(0., req->t_prev - req->objcore->t_origin)));
 
-	http_SetHeader(req->resp, "Via: 1.1 varnish (Varnish/6.0)");
+	http_SetHeader(req->resp, "Via: 1.1 varnish (Varnish/6.1)");
 
 	if (cache_param->http_gzip_support &&
 	    ObjCheckFlag(req->wrk, req->objcore, OF_GZIPED) &&
diff --git a/configure.ac b/configure.ac
index 7cbbe53be..cb2d76888 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2,7 +2,7 @@ AC_PREREQ(2.59)
 AC_COPYRIGHT([Copyright (c) 2006 Verdens Gang AS
 Copyright (c) 2006-2018 Varnish Software])
 AC_REVISION([$Id$])
-AC_INIT([Varnish], [trunk], [varnish-dev at varnish-cache.org])
+AC_INIT([Varnish], [6.1.0], [varnish-dev at varnish-cache.org])
 AC_CONFIG_SRCDIR(include/miniobj.h)
 AC_CONFIG_HEADERS([config.h])
 AC_CONFIG_MACRO_DIR([m4])

From hermunn at varnish-software.com  Tue Sep 18 09:52:13 2018
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Tue, 18 Sep 2018 09:52:13 +0000 (UTC)
Subject: [master] decf4be9f Back to trunk
Message-ID: <20180918095213.24B7461B9F@lists.varnish-cache.org>


commit decf4be9fe23a742a5dafc083c8b10e5e29b5d76
Author: P?l Hermunn Johansen <hermunn at varnish-software.com>
Date:   Tue Sep 18 11:48:26 2018 +0200

    Back to trunk
    
    Varnish Cache 6.1.0 has been released, and we are moving
    back to trunk. A 6.1 branch was created from the preivious
    "Prepare for.." commit.

diff --git a/configure.ac b/configure.ac
index cb2d76888..7cbbe53be 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2,7 +2,7 @@ AC_PREREQ(2.59)
 AC_COPYRIGHT([Copyright (c) 2006 Verdens Gang AS
 Copyright (c) 2006-2018 Varnish Software])
 AC_REVISION([$Id$])
-AC_INIT([Varnish], [6.1.0], [varnish-dev at varnish-cache.org])
+AC_INIT([Varnish], [trunk], [varnish-dev at varnish-cache.org])
 AC_CONFIG_SRCDIR(include/miniobj.h)
 AC_CONFIG_HEADERS([config.h])
 AC_CONFIG_MACRO_DIR([m4])
diff --git a/doc/changes.rst b/doc/changes.rst
index f9a1eb4c1..109bc066f 100644
--- a/doc/changes.rst
+++ b/doc/changes.rst
@@ -26,6 +26,9 @@ http://varnish-cache.org/docs/trunk/whats-new/index.html and via
 individual releases. These documents are updated as part of the
 release process.
 
+=============================
+Varnish Cache trunk (ongoing)
+=============================
 
 ================================
 Varnish Cache 6.1.0 (2018-09-17)

From hermunn at varnish-software.com  Tue Sep 18 14:55:12 2018
From: hermunn at varnish-software.com (Pål Hermunn Johansen)
Date: Tue, 18 Sep 2018 14:55:12 +0000 (UTC)
Subject: [4.1] 1d767f469 On startup, tell what varnish version this is
Message-ID: <20180918145512.949CC941C7@lists.varnish-cache.org>


commit 1d767f469957c7146a59ac8e836437384cb5a2e5
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Thu Apr 26 10:25:26 2018 +0100

    On startup, tell what varnish version this is
    
    Fixes #2661.
    
    Conflicts:
            bin/varnishd/mgt/mgt_main.c

diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index 368c6f4a9..b2ca48528 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -820,6 +820,7 @@ main(int argc, char * const *argv)
 
 	assert(pfh == NULL || !VPF_Write(pfh));
 
+	MGT_complain(C_DEBUG, "Version: %s", VCS_version);
 	MGT_complain(C_DEBUG, "Platform: %s\n", VSB_data(vident) + 1);
 
 	mgt_pid = getpid();	/* daemon() changed this */

From phk at FreeBSD.org  Mon Sep 24 11:24:17 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 24 Sep 2018 11:24:17 +0000 (UTC)
Subject: [master] 3d59f60e8 I börked varnishhist by calling the lround function the wrong place.
Message-ID: <20180924112417.469306E049@lists.varnish-cache.org>


commit 3d59f60e84869fa4adea88b9e433a478922860b1
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 24 11:23:07 2018 +0000

    I b?rked varnishhist by calling the lround function the wrong place.
    
    Fixes: #2780

diff --git a/bin/varnishhist/varnishhist.c b/bin/varnishhist/varnishhist.c
index 48bfce265..c4f99a04a 100644
--- a/bin/varnishhist/varnishhist.c
+++ b/bin/varnishhist/varnishhist.c
@@ -299,7 +299,7 @@ accumulate(struct VSL_data *vsl, struct VSL_transaction * const pt[],
 			continue;
 
 		/* select bucket */
-		i = HIST_RES * lround(log(value) / log_ten);
+		i = lround(HIST_RES * log(value) / log_ten);
 		if (i < hist_low * HIST_RES)
 			i = hist_low * HIST_RES;
 		if (i >= hist_high * HIST_RES)

From phk at FreeBSD.org  Mon Sep 24 11:26:15 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 24 Sep 2018 11:26:15 +0000 (UTC)
Subject: [master] b2075031a Don't implement VRT_* functions in libvarnish
Message-ID: <20180924112615.D87B56E1A7@lists.varnish-cache.org>


commit b2075031a85dbe34952eee03245261b63ea97922
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 24 06:43:17 2018 +0000

    Don't implement VRT_* functions in libvarnish

diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index af59b6a0b..9d47728c2 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -872,3 +872,9 @@ VRT_blob(VRT_CTX, const char *err, const void *src, size_t len)
 	p->priv = d;
 	return (p);
 }
+
+int
+VRT_VSA_GetPtr(const struct suckaddr *sua, const unsigned char ** dst)
+{
+	return (VSA_GetPtr(sua, dst));
+}
diff --git a/include/vsa.h b/include/vsa.h
index 70e4e502d..e5e342a26 100644
--- a/include/vsa.h
+++ b/include/vsa.h
@@ -55,4 +55,11 @@ struct suckaddr *VSA_Malloc(const void *s, unsigned  sal);
  */
 struct suckaddr *VSA_Build(void *d, const void *s, unsigned sal);
 
+/*
+ * This VRT interface is for the VCC generated ACL code, which needs
+ * to know the address family and a pointer to the actual address.
+ */
+  
+int VSA_GetPtr(const struct suckaddr *sua, const unsigned char ** dst);
+
 #endif
diff --git a/lib/libvarnish/vsa.c b/lib/libvarnish/vsa.c
index 4e0762f95..40d84fa83 100644
--- a/lib/libvarnish/vsa.c
+++ b/lib/libvarnish/vsa.c
@@ -43,7 +43,6 @@
 #include "vdef.h"
 #include "vas.h"
 #include "vsa.h"
-#include "vrt.h"
 #include "miniobj.h"
 
 /*
@@ -204,7 +203,7 @@ VSA_Init()
  */
 
 int
-VRT_VSA_GetPtr(const struct suckaddr *sua, const unsigned char ** dst)
+VSA_GetPtr(const struct suckaddr *sua, const unsigned char ** dst)
 {
 
 	AN(dst);

From phk at FreeBSD.org  Mon Sep 24 11:26:15 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 24 Sep 2018 11:26:15 +0000 (UTC)
Subject: [master] bd9a741e1 Fix trouble with 2GB+ VSB's on systems where int
 is 32 bit but size_t is 64 bit.
Message-ID: <20180924112615.CD17D6E1A5@lists.varnish-cache.org>


commit bd9a741e1d7e9ddd7aa6eceb0a78bc110016bbc7
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 24 06:38:58 2018 +0000

    Fix trouble with 2GB+ VSB's on systems where int is 32 bit but
    size_t is 64 bit.

diff --git a/lib/libvarnish/vsb.c b/lib/libvarnish/vsb.c
index 0bc0925ab..b44323854 100644
--- a/lib/libvarnish/vsb.c
+++ b/lib/libvarnish/vsb.c
@@ -51,8 +51,8 @@ __FBSDID("$FreeBSD: head/sys/kern/subr_vsb.c 222004 2011-05-17 06:36:32Z phk $")
  */
 #define	VSB_ISDYNAMIC(s)	((s)->s_flags & VSB_DYNAMIC)
 #define	VSB_ISDYNSTRUCT(s)	((s)->s_flags & VSB_DYNSTRUCT)
-#define	VSB_HASROOM(s)		((s)->s_len < (s)->s_size - 1)
-#define	VSB_FREESPACE(s)	((s)->s_size - ((s)->s_len + 1))
+#define	VSB_HASROOM(s)		((s)->s_len < (s)->s_size - 1L)
+#define	VSB_FREESPACE(s)	((s)->s_size - ((s)->s_len + 1L))
 #define	VSB_CANEXTEND(s)	((s)->s_flags & VSB_AUTOEXTEND)
 
 /*
@@ -114,10 +114,10 @@ CTASSERT(powerof2(VSB_MAXEXTENDSIZE));
 CTASSERT(powerof2(VSB_MAXEXTENDINCR));
 #endif
 
-static int
-VSB_extendsize(int size)
+static ssize_t
+VSB_extendsize(ssize_t size)
 {
-	int newsize;
+	ssize_t newsize;
 
 	if (size < (int)VSB_MAXEXTENDSIZE) {
 		newsize = VSB_MINEXTENDSIZE;
@@ -133,11 +133,11 @@ VSB_extendsize(int size)
 /*
  * Extend an vsb.
  */
-static int
-VSB_extend(struct vsb *s, int addlen)
+static ssize_t
+VSB_extend(struct vsb *s, ssize_t addlen)
 {
 	char *newbuf;
-	int newsize;
+	ssize_t newsize;
 
 	if (!VSB_CANEXTEND(s))
 		return (-1);

From phk at FreeBSD.org  Mon Sep 24 11:26:16 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 24 Sep 2018 11:26:16 +0000 (UTC)
Subject: [master] ab648c844 Eliminated varnishd from these tests
Message-ID: <20180924112616.0D6AB6E1AA@lists.varnish-cache.org>


commit ab648c844abef2bdd5e345519c00f30f88540651
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 24 06:47:36 2018 +0000

    Eliminated varnishd from these tests

diff --git a/bin/varnishtest/tests/a00015.vtc b/bin/varnishtest/tests/a00015.vtc
index 843637f0f..f363d66d9 100644
--- a/bin/varnishtest/tests/a00015.vtc
+++ b/bin/varnishtest/tests/a00015.vtc
@@ -12,9 +12,7 @@ server s1 {
 	txresp -hdr "Content-Type: text/plain" -body response
 } -start
 
-varnish v1 -vcl+backend {} -start -cliok "param.set debug +syncvsl"
-
-client c1 {
+client c1 -connect ${s1_sock} {
 	txreq -req POST -hdr "Content-Type: text/plain" -body request
 
 	# First, HTTP checks
diff --git a/bin/varnishtest/tests/a00018.vtc b/bin/varnishtest/tests/a00018.vtc
index 036aa4596..5500d7c6e 100644
--- a/bin/varnishtest/tests/a00018.vtc
+++ b/bin/varnishtest/tests/a00018.vtc
@@ -8,28 +8,9 @@ server s1 {
 	txresp -hdr "Bar: ${bar}"
 } -start
 
-varnish v1 -vcl {
-	backend default {
-		.host = "${s1_addr}";
-		.port = "${s1_port}";
-	}
-
-	sub vcl_deliver {
-		if (resp.http.Bar == "${bar}") {
-			set resp.http.Baz = "${baz}";
-		}
-	}
-} -start
-
-client c1 {
+client c1 -connect ${s1_sock} {
 	txreq -hdr "Foo: ${foo}"
 	rxresp
 	expect resp.status == 200
 	expect resp.http.Bar == "${bar}"
-	expect resp.http.Baz == "${baz}"
 } -run
-
-shell {
-	touch ${tmpdir}/tst
-	rm ${tmpdir}/tst
-}

From phk at FreeBSD.org  Mon Sep 24 11:26:16 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 24 Sep 2018 11:26:16 +0000 (UTC)
Subject: [master] bbb4514a8 Mark our extension to ZLib and #ifdef the code in
 varnishtest which reports stuff from it for debugging.
Message-ID: <20180924112616.388856E1AD@lists.varnish-cache.org>


commit bbb4514a882ed28e6a7a75c3a45948420c453675
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 24 06:48:49 2018 +0000

    Mark our extension to ZLib and #ifdef the code in varnishtest which
    reports stuff from it for debugging.

diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c
index d89a91a5a..952563b9f 100644
--- a/bin/varnishtest/vtc_http.c
+++ b/bin/varnishtest/vtc_http.c
@@ -771,6 +771,7 @@ cmd_http_gunzip(CMD_ARGS)
 	vtc_log(hp->vl, 3, "new bodylen %u", hp->bodyl);
 	vtc_dump(hp->vl, 4, "body", hp->body, hp->bodyl);
 	bprintf(hp->bodylen, "%u", hp->bodyl);
+#ifdef VGZ_EXTENSIONS
 	vtc_log(hp->vl, 4, "startbit = %ju %ju/%ju",
 	    (uintmax_t)vz.start_bit,
 	    (uintmax_t)vz.start_bit >> 3, (uintmax_t)vz.start_bit & 7);
@@ -780,6 +781,7 @@ cmd_http_gunzip(CMD_ARGS)
 	vtc_log(hp->vl, 4, "stopbit = %ju %ju/%ju",
 	    (uintmax_t)vz.stop_bit,
 	    (uintmax_t)vz.stop_bit >> 3, (uintmax_t)vz.stop_bit & 7);
+#endif
 	if (i != Z_STREAM_END)
 		vtc_log(hp->vl, hp->fatal,
 		    "Gunzip error = %d (%s) in:%jd out:%jd",
@@ -813,12 +815,13 @@ gzip_body(const struct http *hp, const char *txt, char **body, int *bodylen)
 	assert(Z_OK == deflateInit2(&vz,
 	    hp->gziplevel, Z_DEFLATED, 31, 9, Z_DEFAULT_STRATEGY));
 	assert(Z_STREAM_END == deflate(&vz, Z_FINISH));
+	*bodylen = vz.total_out;
+#ifdef VGZ_EXTENSIONS
 	i = vz.stop_bit & 7;
 	if (hp->gzipresidual >= 0 && hp->gzipresidual != i)
 		vtc_log(hp->vl, hp->fatal,
 		    "Wrong gzip residual got %d wanted %d",
 		    i, hp->gzipresidual);
-	*bodylen = vz.total_out;
 	vtc_log(hp->vl, 4, "startbit = %ju %ju/%ju",
 	    (uintmax_t)vz.start_bit,
 	    (uintmax_t)vz.start_bit >> 3, (uintmax_t)vz.start_bit & 7);
@@ -829,6 +832,7 @@ gzip_body(const struct http *hp, const char *txt, char **body, int *bodylen)
 	    (uintmax_t)vz.stop_bit,
 	    (uintmax_t)vz.stop_bit >> 3, (uintmax_t)vz.stop_bit & 7);
 	assert(Z_OK == deflateEnd(&vz));
+#endif
 }
 
 /**********************************************************************
diff --git a/lib/libvgz/vgz.h b/lib/libvgz/vgz.h
index 86831eeed..1bb81009d 100644
--- a/lib/libvgz/vgz.h
+++ b/lib/libvgz/vgz.h
@@ -104,6 +104,7 @@ typedef struct z_stream_s {
     uLong   adler;      /* Adler-32 or CRC-32 value of the uncompressed data */
     uLong   reserved;   /* reserved for future use */
 
+#define VGZ_EXTENSIONS 1
     uLong   start_bit;	/* Bit pos of first deflate block */
     uLong   stop_bit;	/* Bit pos after last deflate block */
     uLong   last_bit;	/* Bit pos of 'last' bit */

From phk at FreeBSD.org  Mon Sep 24 11:26:16 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 24 Sep 2018 11:26:16 +0000 (UTC)
Subject: [master] a852a43f9 Get varnishd out of and varnishtest-tests into
 a*vtc
Message-ID: <20180924112616.85E2C6E1B3@lists.varnish-cache.org>


commit a852a43f973146e6d084963b8bd36479171bb53c
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 24 07:09:30 2018 +0000

    Get varnishd out of and varnishtest-tests into a*vtc

diff --git a/bin/varnishtest/tests/a00000.vtc b/bin/varnishtest/tests/a00000.vtc
index 9316a5cb8..22e9f341b 100644
--- a/bin/varnishtest/tests/a00000.vtc
+++ b/bin/varnishtest/tests/a00000.vtc
@@ -41,96 +41,130 @@ shell -exit 77 -expect {TEST _.vtc skipped} {
 	exec varnishtest -v _.vtc || true
 }
 
-process p1 "ps -lw | grep '[p][s]' ; tty ; echo @" -start
-process p1 -expect-text 0 0 {@} -screen_dump -wait
-
-process p2 "stty -a ; echo '*'" -start
-process p2 -expect-text 0 0 {*} -screen_dump -wait
-
-process p4 -hexdump {stty raw -echo; stty -a ; echo "*" ; cat} -start
-process p4 -expect-text 0 0 "*" -screen_dump
-
-process p4 -write "\x1b[H\x1b[2Jzzzzzzz"
-process p4 -write "\x0c1\x1b[79C2\x08>\x1b[25;1H3\x1b[25;80H"
-process p4 -write "\x1b[H\x1b[2J1\x1b[79C2\x08>\x1b[25;1H3\x1b[25;80H"
-process p4 -write "4\x08>\x1b[A\x1b[Cv\x1b[22A^\x1b[79D^\x1b[;2H<\n\n\n\n"
-process p4 -write "\n\n\n\n\n\n\n\n\x1b[B\x1b[11B\x08<\x1b[24;Hv\x1b[12;1H"
-process p4 -write "111111112222222333333\x0d\x0a111111112"
-process p4 -write "222222333333\x0d\x0a111111112222222333333 UTF8: "
-process p4 -writehex {c2 a2 20}
-process p4 -writehex {e2 82 ac 20}
-process p4 -writehex {f0 90 80 80 20}
-process p4 -writehex {f0 9f 90 b0 20}
-process p4 -writehex {f0 a0 80 80 20}
-process p4 -writehex {f0 b0 80 80 20}
-process p4 -write "\x1b[22;24;25;27;30;47;49;97;107m"
-process p4 -write "\x1b[22;24;25;27;30m"
-process p4 -write "\x1b[47;49;97;107m"
-process p4 -write "\x0d\x0a111111112222222333333\x0d\x0a\x1b[12"
-process p4 -write ";12H\x1b[K\x1b[13;12H\x1b[0K\x1b[14;12H\x1b[1K\x1b"
-process p4 -write "[15;12H\x1b[2K\x1b[3;1Hline3 <\x0d\x0a"
-
-process p4 -need-bytes 310 -expect-text 3 1 "line3 <"
-process p4 -expect-cursor 4 1
-process p4 -expect-cursor 4 0
-process p4 -expect-cursor 0 1
-process p4 -screen-dump
+# Simple process tests
+
+process p1 "cat" -start
+process p1 -writeln "foo"
+process p1 -expect-text 2 1 foo
+process p1 -stop
+process p1 -wait
+shell "grep -q foo ${p1_out}"
+shell "test -f ${p1_err} -a ! -s ${p1_err}"
+
+process p2 -log "cat" -start
+process p2 -writeln "bar"
+process p2 -expect-text 2 1 bar
+process p2 -write "\x04"
+process p2 -wait
+shell "grep -q bar ${p2_out}"
+shell "test -f ${p2_err} -a ! -s ${p2_err}"
+
+process p3 -dump "cat" -start
+process p3 -writeln "baz"
+process p3 -expect-text 2 1 baz
+process p3 -kill KILL
+process p3 -wait
+shell "grep -q baz ${p3_out}"
+shell "test -f ${p3_err} -a ! -s ${p3_err}"
+
+process p4 -hexdump "cat" -start
+process p4 -writeln "b\001z"
+process p4 -expect-text 2 1 "b"
+process p4 -kill TERM
+process p4 -wait -screen_dump
+
+# Curses process tests
+
+process p5 "ps -lw | grep '[p][s]' ; tty ; echo @" -start
+process p5 -expect-text 0 0 {@} -screen_dump -wait
+
+process p6 "stty -a ; echo '*'" -start
+process p6 -expect-text 0 0 {*} -screen_dump -wait
+
+process p7 -hexdump {stty raw -echo; stty -a ; echo "*" ; cat} -start
+process p7 -expect-text 0 0 "*" -screen_dump
+
+process p7 -write "\x1b[H\x1b[2Jzzzzzzz"
+process p7 -write "\x0c1\x1b[79C2\x08>\x1b[25;1H3\x1b[25;80H"
+process p7 -write "\x1b[H\x1b[2J1\x1b[79C2\x08>\x1b[25;1H3\x1b[25;80H"
+process p7 -write "4\x08>\x1b[A\x1b[Cv\x1b[22A^\x1b[79D^\x1b[;2H<\n\n\n\n"
+process p7 -write "\n\n\n\n\n\n\n\n\x1b[B\x1b[11B\x08<\x1b[24;Hv\x1b[12;1H"
+process p7 -write "111111112222222333333\x0d\x0a111111112"
+process p7 -write "222222333333\x0d\x0a111111112222222333333 UTF8: "
+process p7 -writehex {c2 a2 20}
+process p7 -writehex {e2 82 ac 20}
+process p7 -writehex {f0 90 80 80 20}
+process p7 -writehex {f0 9f 90 b0 20}
+process p7 -writehex {f0 a0 80 80 20}
+process p7 -writehex {f0 b0 80 80 20}
+process p7 -write "\x1b[22;24;25;27;30;47;49;97;107m"
+process p7 -write "\x1b[22;24;25;27;30m"
+process p7 -write "\x1b[47;49;97;107m"
+process p7 -write "\x0d\x0a111111112222222333333\x0d\x0a\x1b[12"
+process p7 -write ";12H\x1b[K\x1b[13;12H\x1b[0K\x1b[14;12H\x1b[1K\x1b"
+process p7 -write "[15;12H\x1b[2K\x1b[3;1Hline3 <\x0d\x0a"
+
+process p7 -need-bytes 310 -expect-text 3 1 "line3 <"
+process p7 -expect-cursor 4 1
+process p7 -expect-cursor 4 0
+process p7 -expect-cursor 0 1
+process p7 -screen-dump
 
 # Also exercise CONS25 mode
-process p4 -write "\x1b[=1T"
-process p4 -write "\x1b[=2T"
-process p4 -write "\x1b[8z"
-process p4 -write "\x1b[0x"
-process p4 -write "\x1b[=1A"
-process p4 -write "\x1b[=1;2B"
-process p4 -write "\x1b[=1;2;3C"
-process p4 -write "\x1b[=1;2;3;4C"
-process p4 -write "\x1b[=1F"
-process p4 -write "\x1b[=1G"
-process p4 -write "\x1b[=1S"
-process p4 -writehex {0c 08 40 0d 0a 08}
-
-process p4 -expect-text 1 1 "@"
-process p4 -expect-cursor 1 80
-process p4 -writehex "0c 41 0e 42 0f"
-process p4 -expect-text 1 1 "A"
-process p4 -expect-text 0 0 "B"
-process p4 -write "\x1b[=0T"
-
-process p4 -writehex "0c 0a 0d 43 0a 08 08 0e 44 0f"
-
-process p4 -expect-text 3 1 "C"
-process p4 -expect-text 4 1 "D"
-process p4 -write "\x1b[2T"
-process p4 -expect-text 5 1 "C"
-process p4 -expect-text 6 1 "D"
-process p4 -write "\x1b[3S"
-process p4 -expect-text 3 1 "D"
-
-process p4 -write "\x1b[4;200H%"
-process p4 -expect-text 4 80 "%"
-
-process p4 -write "\x1b[7;7H\x09X\x09Y\x09Z\x1b[2ZW\x1b[2Ew\x1b[F*"
-
-process p4 -expect-text 7 17 "W"
-process p4 -expect-text 9 1 "w"
-process p4 -expect-text 8 1 "*"
-
-process p4 -write "\x1b[10;4HABCDEFGHIJKLMN\x1b(A#$%\x1b)A"
-process p4 -write "\x1b[8G\x1b[2X>"
-process p4 -expect-text 10 8 ">"
-process p4 -screen-dump
+process p7 -write "\x1b[=1T"
+process p7 -write "\x1b[=2T"
+process p7 -write "\x1b[8z"
+process p7 -write "\x1b[0x"
+process p7 -write "\x1b[=1A"
+process p7 -write "\x1b[=1;2B"
+process p7 -write "\x1b[=1;2;3C"
+process p7 -write "\x1b[=1;2;3;4C"
+process p7 -write "\x1b[=1F"
+process p7 -write "\x1b[=1G"
+process p7 -write "\x1b[=1S"
+process p7 -writehex {0c 08 40 0d 0a 08}
+
+process p7 -expect-text 1 1 "@"
+process p7 -expect-cursor 1 80
+process p7 -writehex "0c 41 0e 42 0f"
+process p7 -expect-text 1 1 "A"
+process p7 -expect-text 0 0 "B"
+process p7 -write "\x1b[=0T"
+
+process p7 -writehex "0c 0a 0d 43 0a 08 08 0e 44 0f"
+
+process p7 -expect-text 3 1 "C"
+process p7 -expect-text 4 1 "D"
+process p7 -write "\x1b[2T"
+process p7 -expect-text 5 1 "C"
+process p7 -expect-text 6 1 "D"
+process p7 -write "\x1b[3S"
+process p7 -expect-text 3 1 "D"
+
+process p7 -write "\x1b[4;200H%"
+process p7 -expect-text 4 80 "%"
+
+process p7 -write "\x1b[7;7H\x09X\x09Y\x09Z\x1b[2ZW\x1b[2Ew\x1b[F*"
+
+process p7 -expect-text 7 17 "W"
+process p7 -expect-text 9 1 "w"
+process p7 -expect-text 8 1 "*"
+
+process p7 -write "\x1b[10;4HABCDEFGHIJKLMN\x1b(A#$%\x1b)A"
+process p7 -write "\x1b[8G\x1b[2X>"
+process p7 -expect-text 10 8 ">"
+process p7 -screen-dump
 
 # Test responses
-process p4 -write "\x1b[3;1HA\x1b[5n"
-process p4 -write "\x1b[4;1HB\x1b[6n"
-process p4 -write "\x1b[5;1HC\x1b[15n"
-process p4 -write "\x1b[6;1HD\x1b[25n"
-process p4 -write "\x1b[7;1HE\x1b[26n"
-process p4 -write "\x1b[8;1HF\x1b[?26n"
-process p4 -write "\x1b[9;1HG\x1bPfutfutfut\x01"
-process p4 -write "\x1b[10;1HH\x1b]futfutfut\x01"
-process p4 -write "\x1b[11;1HI\x1b[>c"
-process p4 -write "\x1b[24;1HW"
-process p4 -expect-text 24 1 "W"
-process p4 -screen-dump
+process p7 -write "\x1b[3;1HA\x1b[5n"
+process p7 -write "\x1b[4;1HB\x1b[6n"
+process p7 -write "\x1b[5;1HC\x1b[15n"
+process p7 -write "\x1b[6;1HD\x1b[25n"
+process p7 -write "\x1b[7;1HE\x1b[26n"
+process p7 -write "\x1b[8;1HF\x1b[?26n"
+process p7 -write "\x1b[9;1HG\x1bPfutfutfut\x01"
+process p7 -write "\x1b[10;1HH\x1b]futfutfut\x01"
+process p7 -write "\x1b[11;1HI\x1b[>c"
+process p7 -write "\x1b[24;1HW"
+process p7 -expect-text 24 1 "W"
+process p7 -screen-dump
diff --git a/bin/varnishtest/tests/a00009.vtc b/bin/varnishtest/tests/a00009.vtc
deleted file mode 100644
index 4b82d4c7d..000000000
--- a/bin/varnishtest/tests/a00009.vtc
+++ /dev/null
@@ -1,55 +0,0 @@
-varnishtest "Code coverage of mgt_main, (VCL compiler and RSTdump etc)"
-
-shell "varnishd -b 127.0.0.1:80 -C 2> ${tmpdir}/_.c"
-shell -err -expect {VCL version declaration missing} {
-	echo 'bad vcl' > ${tmpdir}/bad.vcl
-	varnishd -f ${tmpdir}/bad.vcl -n ${tmpdir}
-}
-
-shell -err -expect {-x must be the first argument} "varnishd -d -x foo "
-shell -err -expect {-V must be the first argument} "varnishd -d -V foo "
-
-shell -err -expect {Too many arguments for -x} "varnishd -x foo bar"
-shell -err -expect {Invalid -x argument} "varnishd -x foo "
-
-# This one is tricky, the getopt message on stderr is not standardized.
-shell -err -expect {option} "varnishd -A "
-
-shell -err -expect {Usage: varnishd [options]} "varnishd -? "
-shell -err -expect {Too many arguments} "varnishd foo "
-
-shell "varnishd -x parameter > ${tmpdir}/_.param"
-shell "varnishd -x vsl > ${tmpdir}/_.vsl"
-shell "varnishd -x cli > ${tmpdir}/_.cli"
-shell "varnishd -x builtin > ${tmpdir}/_.builtin"
-
-shell -err -expect {-C needs either -b <backend> or -f <vcl_file>} {
-	varnishd -C
-}
-shell -err -expect {Cannot open -S file} {
-	varnishd -S ${tmpdir}/nonexistent -n ${tmpdir}/v0 -f ''
-}
-shell -err -expect {Unknown jail method "xyz"} "varnishd -jxyz -f '' "
-
-shell -err -expect {Invalid backslash sequence} {
-	varnishd -l 'xyz\kk,xyz\foo' -f ''
-}
-shell -err -expect {Invalid backslash sequence} {
-	varnishd -l 'ab\8cd' -f ''
-}
-shell -err -expect {Too many arguments for -V} "varnishd -V -V"
-shell -expect {Copyright (c) 2006} "varnishd -V"
-
-shell -err -expect {Only one of -d or -F can be specified} "varnishd -d -F "
-shell -err -expect {Only one of -b or -f can be specified} "varnishd -b a -f b "
-shell -err -expect {-d makes no sense with -C} "varnishd -C -b 127.0.0.1:80 -d "
-shell -err -expect {-F makes no sense with -C} "varnishd -C -b 127.0.0.1:80 -F "
-shell -err -expect {Neither -b nor -f given} { varnishd -n ${tmpdir}/v0  }
-
-# This check is kept last because it may be skipped
-
-feature persistent_storage
-
-shell -err -expect {-spersistent has been deprecated} {
-	varnishd -spersistent -f ''
-}
diff --git a/bin/varnishtest/tests/a00016.vtc b/bin/varnishtest/tests/a00016.vtc
deleted file mode 100644
index 4055b7a39..000000000
--- a/bin/varnishtest/tests/a00016.vtc
+++ /dev/null
@@ -1,25 +0,0 @@
-varnishtest "Test -I and -l arguments"
-
-shell -err -expect {Only one -I allowed} {
-	touch foo bar
-	varnishd -f '' -I foo -I bar -n ${tmpdir}/v0 -a :0
-}
-
-shell -err -expect {Error: -I file CLI command failed (104)} {
-	echo "vcl.list" > foo
-	echo "-foobar" >> foo
-	echo "vcl.load" >> foo
-	varnishd -f '' -I foo -n ${tmpdir}/v0 -a :0 -l 2m
-}
-
-shell -err -expect {Error: -l ...: Missing '"'} {
-	varnishd -f '' -I foo -n ${tmpdir}/v0 -a :0 -l '2m,"'
-}
-
-shell -err -expect {Error: Too many sub arguments} {
-	varnishd -f '' -I foo -n ${tmpdir}/v0 -a :0 -l 2m,2m,2m
-}
-
-shell -err -expect {Warning: Ignoring deprecated second subargument} {
-	varnishd -f '' -I foo -n ${tmpdir}/v0 -a :0 -l 2m,2m
-}
diff --git a/bin/varnishtest/tests/a00017.vtc b/bin/varnishtest/tests/a00017.vtc
deleted file mode 100644
index 21b184cfa..000000000
--- a/bin/varnishtest/tests/a00017.vtc
+++ /dev/null
@@ -1,80 +0,0 @@
-varnishtest "Code coverage of mgt_main, (VCL compiler and RSTdump etc)"
-
-# Test -F mode with no VCL loaded
-
-process p1 "exec varnishd -n ${tmpdir}/v1 -F -f '' -a :0 -l2m,1m" -log -start
-
-delay 2
-
-shell {
-	(
-	echo 'vcl 4.0;'
-	echo 'backend default {'
-	echo '  .host="${bad_backend}";'
-	echo '}'
-	) > ${tmpdir}/vcl
-}
-
-shell -expect {VCL compiled.} {
-	varnishadm -n ${tmpdir}/v1 vcl.load vcl1 ${tmpdir}/vcl
-}
-
-shell -expect {active      auto/warm          - vcl1} {
-	varnishadm -n ${tmpdir}/v1 vcl.list
-}
-
-shell {varnishadm -n ${tmpdir}/v1 start}
-
-shell {varnishadm -n ${tmpdir}/v1 debug.listen_address}
-
-shell -exit 1 -expect "Command failed with error code 500" {
-	varnishadm -n ${tmpdir}/v1 quit
-}
-
-shell -exit 1 -expect "Command failed with error code 102" {
-	varnishadm -n ${tmpdir}/v1 debug.panic.master -j
-}
-
-shell -exit 1 -expect "Command failed with error code 101" {
-	varnishadm -n ${tmpdir}/v1 123
-}
-
-shell "varnishadm -n ${tmpdir}/v1 param.set cli_limit 128"
-shell -expect "[response was truncated]" "varnishadm -n ${tmpdir}/v1 help"
-
-process p1 -expect-exit 64 -stop -wait
-
-# Test multiple -f options
-
-shell {
-	cat >${tmpdir}/ok1 <<-EOF
-	vcl 4.0;
-	backend ok1 {
-	  .host="${bad_backend}";
-	}
-	EOF
-
-	cat >${tmpdir}/ok2 <<-EOF
-	vcl 4.0;
-	backend ok2 {
-	  .host="${bad_backend}";
-	}
-	EOF
-}
-
-varnish v2 -arg "-f ${tmpdir}/ok1" -arg "-f ${tmpdir}/ok2" -start
-varnish v2 -cliexpect {available *auto/warm *0 boot0} "vcl.list"
-varnish v2 -cliexpect {active *auto/warm *0 boot} "vcl.list"
-varnish v2 -stop -wait
-
-# Test multiple -f options with a bad VCL
-
-shell -err -expect {Cannot read -f file} {
-	exec varnishd -n ${tmpdir}/v0 -F -a :0 -l2m,3m -f ${tmpdir}/ok1 \
-	    -f ${tmpdir}/ok2 -f ${tmpdir}/bad
-}
-
-shell -err -expect {Cannot read -f file} {
-	exec varnishd -n ${tmpdir}/v0 -F -a :0 -l2m,3m -f ${tmpdir}/ok1 \
-	    -f ${tmpdir}/bad -f ${tmpdir}/ok2
-}
diff --git a/bin/varnishtest/tests/a00019.vtc b/bin/varnishtest/tests/a00019.vtc
deleted file mode 100644
index a6d1317d5..000000000
--- a/bin/varnishtest/tests/a00019.vtc
+++ /dev/null
@@ -1,33 +0,0 @@
-varnishtest "vtc v_* macros when the listen address is UDS"
-
-varnish v1 -arg "-a ${tmpdir}/v1.sock -b '${bad_backend}'" -start
-
-varnish v1 -syntax 4.0 -errvcl {Compiled VCL version (4.0) not supported.} {
-	backend default { .host = "${bad_ip}"; }
-}
-
-varnish v1 -syntax 4.0 -errvcl \
-	{Unix socket backends only supported in VCL4.1 and higher.} \
-	{backend default { .path = "${tmpdir}/v1.sock"; }}
-
-varnish v2 -vcl {
-	backend default { .host = "${bad_ip}"; }
-
-	sub vcl_recv {
-		return(synth(200));
-	}
-
-	sub vcl_synth {
-		set resp.http.addr = "${v1_addr}";
-		set resp.http.port = "${v1_port}";
-		set resp.http.sock = "${v1_sock}";
-	}
-} -start
-
-client c1 -connect ${v2_sock} {
-	txreq
-	rxresp
-	expect resp.http.addr == "${tmpdir}/v1.sock"
-	expect resp.http.port == "-"
-	expect resp.http.sock == "${tmpdir}/v1.sock -"
-} -run
diff --git a/bin/varnishtest/tests/a00020.vtc b/bin/varnishtest/tests/a00020.vtc
deleted file mode 100644
index aba786e2e..000000000
--- a/bin/varnishtest/tests/a00020.vtc
+++ /dev/null
@@ -1,34 +0,0 @@
-varnishtest "vtc remote.ip, remote.port and remote.path"
-
-server s1 {
-	rxreq
-	expect remote.ip == "${localhost}"
-	expect remote.port > 0
-	expect remote.path == <undef>
-	txresp
-} -start
-
-varnish v1 -vcl+backend {} -start
-
-client c1 {
-	txreq
-	rxresp
-	expect remote.ip == "${v1_addr}"
-	expect remote.port == "${v1_port}"
-	expect remote.path == <undef>
-} -run
-
-varnish v1 -stop
-
-server s1 -wait
-server s1 -start
-
-varnish v2 -arg "-a ${tmpdir}/v2.sock" -vcl+backend {} -start
-
-client c1 -connect "${tmpdir}/v2.sock" {
-	txreq
-	rxresp
-	expect remote.ip == "0.0.0.0"
-	expect remote.port == 0
-	expect remote.path == "${tmpdir}/v2.sock"
-} -run
diff --git a/bin/varnishtest/tests/b00053.vtc b/bin/varnishtest/tests/b00053.vtc
index 0acb9daec..adcec5974 100644
--- a/bin/varnishtest/tests/b00053.vtc
+++ b/bin/varnishtest/tests/b00053.vtc
@@ -32,3 +32,37 @@ varnish v1 -expect cache_miss == 1
 varnish v1 -expect s_sess == 1
 varnish v1 -expect s_resp_bodybytes == 7
 varnish v1 -expect s_resp_hdrbytes == 178
+
+# varnishtest "vtc v_* macros when the listen address is UDS" (a00019)
+
+varnish v2 -arg "-a ${tmpdir}/v1.sock -b '${bad_backend}'" -start
+
+varnish v2 -syntax 4.0 -errvcl {Compiled VCL version (4.0) not supported.} {
+	backend default { .host = "${bad_ip}"; }
+}
+
+varnish v2 -syntax 4.0 -errvcl \
+	{Unix socket backends only supported in VCL4.1 and higher.} \
+	{backend default { .path = "${tmpdir}/v1.sock"; }}
+
+varnish v3 -vcl {
+	backend default { .host = "${bad_ip}"; }
+
+	sub vcl_recv {
+		return(synth(200));
+	}
+
+	sub vcl_synth {
+		set resp.http.addr = "${v1_addr}";
+		set resp.http.port = "${v1_port}";
+		set resp.http.sock = "${v1_sock}";
+	}
+} -start
+
+client c1 -connect ${v3_sock} {
+	txreq
+	rxresp
+	expect resp.http.addr == "${tmpdir}/v1.sock"
+	expect resp.http.port == "-"
+	expect resp.http.sock == "${tmpdir}/v1.sock -"
+} -run
diff --git a/bin/varnishtest/tests/u00000.vtc b/bin/varnishtest/tests/u00000.vtc
index 7e426ec96..ae91b169f 100644
--- a/bin/varnishtest/tests/u00000.vtc
+++ b/bin/varnishtest/tests/u00000.vtc
@@ -1,32 +1,162 @@
-varnishtest "Simple process tests"
-
-process p1 "cat" -start
-process p1 -writeln "foo"
-process p1 -expect-text 2 1 foo
-process p1 -stop
-process p1 -wait
-shell "grep -q foo ${p1_out}"
-shell "test -f ${p1_err} -a ! -s ${p1_err}"
-
-process p2 -log "cat" -start
-process p2 -writeln "bar"
-process p2 -expect-text 2 1 bar
-process p2 -write "\x04"
-process p2 -wait
-shell "grep -q bar ${p2_out}"
-shell "test -f ${p2_err} -a ! -s ${p2_err}"
-
-process p3 -dump "cat" -start
-process p3 -writeln "baz"
-process p3 -expect-text 2 1 baz
-process p3 -kill KILL
-process p3 -wait
-shell "grep -q baz ${p3_out}"
-shell "test -f ${p3_err} -a ! -s ${p3_err}"
-
-process p4 -hexdump "cat" -start
-process p4 -writeln "b\001z"
-process p4 -expect-text 2 1 "b"
-process p4 -kill TERM
-process p4 -wait -screen_dump
+varnishtest "Code coverage of mgt_main, (VCL compiler and RSTdump etc)"
 
+shell "varnishd -b 127.0.0.1:80 -C 2> ${tmpdir}/_.c"
+shell -err -expect {VCL version declaration missing} {
+	echo 'bad vcl' > ${tmpdir}/bad.vcl
+	varnishd -f ${tmpdir}/bad.vcl -n ${tmpdir}
+}
+
+shell -err -expect {-x must be the first argument} "varnishd -d -x foo "
+shell -err -expect {-V must be the first argument} "varnishd -d -V foo "
+
+shell -err -expect {Too many arguments for -x} "varnishd -x foo bar"
+shell -err -expect {Invalid -x argument} "varnishd -x foo "
+
+# This one is tricky, the getopt message on stderr is not standardized.
+shell -err -expect {option} "varnishd -A "
+
+shell -err -expect {Usage: varnishd [options]} "varnishd -? "
+shell -err -expect {Too many arguments} "varnishd foo "
+
+shell "varnishd -x parameter > ${tmpdir}/_.param"
+shell "varnishd -x vsl > ${tmpdir}/_.vsl"
+shell "varnishd -x cli > ${tmpdir}/_.cli"
+shell "varnishd -x builtin > ${tmpdir}/_.builtin"
+
+shell -err -expect {-C needs either -b <backend> or -f <vcl_file>} {
+	varnishd -C
+}
+shell -err -expect {Cannot open -S file} {
+	varnishd -S ${tmpdir}/nonexistent -n ${tmpdir}/v0 -f ''
+}
+shell -err -expect {Unknown jail method "xyz"} "varnishd -jxyz -f '' "
+
+shell -err -expect {Invalid backslash sequence} {
+	varnishd -l 'xyz\kk,xyz\foo' -f ''
+}
+shell -err -expect {Invalid backslash sequence} {
+	varnishd -l 'ab\8cd' -f ''
+}
+shell -err -expect {Too many arguments for -V} "varnishd -V -V"
+shell -expect {Copyright (c) 2006} "varnishd -V"
+
+shell -err -expect {Only one of -d or -F can be specified} "varnishd -d -F "
+shell -err -expect {Only one of -b or -f can be specified} "varnishd -b a -f b "
+shell -err -expect {-d makes no sense with -C} "varnishd -C -b 127.0.0.1:80 -d "
+shell -err -expect {-F makes no sense with -C} "varnishd -C -b 127.0.0.1:80 -F "
+shell -err -expect {Neither -b nor -f given} { varnishd -n ${tmpdir}/v0  }
+
+# This check is kept last because it may be skipped
+
+feature persistent_storage
+
+shell -err -expect {-spersistent has been deprecated} {
+	varnishd -spersistent -f ''
+}
+
+# Test -I and -l arguments (former a00016)
+
+shell -err -expect {Only one -I allowed} {
+	touch foo bar
+	varnishd -f '' -I foo -I bar -n ${tmpdir}/v0 -a :0
+}
+
+shell -err -expect {Error: -I file CLI command failed (104)} {
+	echo "vcl.list" > foo
+	echo "-foobar" >> foo
+	echo "vcl.load" >> foo
+	varnishd -f '' -I foo -n ${tmpdir}/v0 -a :0 -l 2m
+}
+
+shell -err -expect {Error: -l ...: Missing '"'} {
+	varnishd -f '' -I foo -n ${tmpdir}/v0 -a :0 -l '2m,"'
+}
+
+shell -err -expect {Error: Too many sub arguments} {
+	varnishd -f '' -I foo -n ${tmpdir}/v0 -a :0 -l 2m,2m,2m
+}
+
+shell -err -expect {Warning: Ignoring deprecated second subargument} {
+	varnishd -f '' -I foo -n ${tmpdir}/v0 -a :0 -l 2m,2m
+}
+
+# Code coverage of mgt_main, (VCL compiler and RSTdump etc) (former a00017)
+
+# Test -F mode with no VCL loaded
+
+process p1 "exec varnishd -n ${tmpdir}/v1 -F -f '' -a :0 -l2m,1m" -log -start
+
+delay 2
+
+shell {
+	(
+	echo 'vcl 4.0;'
+	echo 'backend default {'
+	echo '  .host="${bad_backend}";'
+	echo '}'
+	) > ${tmpdir}/vcl
+}
+
+shell -expect {VCL compiled.} {
+	varnishadm -n ${tmpdir}/v1 vcl.load vcl1 ${tmpdir}/vcl
+}
+
+shell -expect {active      auto/warm          - vcl1} {
+	varnishadm -n ${tmpdir}/v1 vcl.list
+}
+
+shell {varnishadm -n ${tmpdir}/v1 start}
+
+shell {varnishadm -n ${tmpdir}/v1 debug.listen_address}
+
+shell -exit 1 -expect "Command failed with error code 500" {
+	varnishadm -n ${tmpdir}/v1 quit
+}
+
+shell -exit 1 -expect "Command failed with error code 102" {
+	varnishadm -n ${tmpdir}/v1 debug.panic.master -j
+}
+
+shell -exit 1 -expect "Command failed with error code 101" {
+	varnishadm -n ${tmpdir}/v1 123
+}
+
+shell "varnishadm -n ${tmpdir}/v1 param.set cli_limit 128"
+shell -expect "[response was truncated]" "varnishadm -n ${tmpdir}/v1 help"
+
+process p1 -expect-exit 64 -stop -wait
+
+# Test multiple -f options
+
+shell {
+	cat >${tmpdir}/ok1 <<-EOF
+	vcl 4.0;
+	backend ok1 {
+	  .host="${bad_backend}";
+	}
+	EOF
+
+	cat >${tmpdir}/ok2 <<-EOF
+	vcl 4.0;
+	backend ok2 {
+	  .host="${bad_backend}";
+	}
+	EOF
+}
+
+varnish v2 -arg "-f ${tmpdir}/ok1" -arg "-f ${tmpdir}/ok2" -start
+varnish v2 -cliexpect {available *auto/warm *0 boot0} "vcl.list"
+varnish v2 -cliexpect {active *auto/warm *0 boot} "vcl.list"
+varnish v2 -stop -wait
+
+# Test multiple -f options with a bad VCL
+
+shell -err -expect {Cannot read -f file} {
+	exec varnishd -n ${tmpdir}/v0 -F -a :0 -l2m,3m -f ${tmpdir}/ok1 \
+	    -f ${tmpdir}/ok2 -f ${tmpdir}/bad
+}
+
+shell -err -expect {Cannot read -f file} {
+	exec varnishd -n ${tmpdir}/v0 -F -a :0 -l2m,3m -f ${tmpdir}/ok1 \
+	    -f ${tmpdir}/bad -f ${tmpdir}/ok2
+}
diff --git a/bin/varnishtest/tests/v00054.vtc b/bin/varnishtest/tests/v00054.vtc
index a230fb867..c92e8cce6 100644
--- a/bin/varnishtest/tests/v00054.vtc
+++ b/bin/varnishtest/tests/v00054.vtc
@@ -1,6 +1,40 @@
 varnishtest "client.identity is 0.0.0.0 if unset & client addr is UDS"
+# varnishtest "vtc remote.ip, remote.port and remote.path" (a00020)
 
-varnish v1 -arg "-a ${tmpdir}/v1.sock" -vcl {
+server s1 {
+	rxreq
+	expect remote.ip == "${localhost}"
+	expect remote.port > 0
+	expect remote.path == <undef>
+	txresp
+} -start
+
+varnish v1 -vcl+backend {} -start
+
+client c1 {
+	txreq
+	rxresp
+	expect remote.ip == "${v1_addr}"
+	expect remote.port == "${v1_port}"
+	expect remote.path == <undef>
+} -run
+
+varnish v1 -stop
+
+server s1 -wait
+server s1 -start
+
+varnish v2 -arg "-a ${tmpdir}/v2.sock" -vcl+backend {} -start
+
+client c1 -connect "${tmpdir}/v2.sock" {
+	txreq
+	rxresp
+	expect remote.ip == "0.0.0.0"
+	expect remote.port == 0
+	expect remote.path == "${tmpdir}/v2.sock"
+} -run
+
+varnish v3 -arg "-a ${tmpdir}/v3.sock" -vcl {
 	backend b { .host = "${bad_ip}"; }
 
 	sub vcl_recv {
@@ -16,7 +50,7 @@ varnish v1 -arg "-a ${tmpdir}/v1.sock" -vcl {
 	}
 } -start
 
-client c1 -connect "${tmpdir}/v1.sock" {
+client c2 -connect "${tmpdir}/v3.sock" {
 	txreq -url "/nobody"
 	rxresp
 	expect resp.status == 200

From phk at FreeBSD.org  Mon Sep 24 11:26:16 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 24 Sep 2018 11:26:16 +0000 (UTC)
Subject: [master] 64dc214d4 Make "vtest" an alias for "varnishtest" in the
 first line.
Message-ID: <20180924112616.B391B6E1BA@lists.varnish-cache.org>


commit 64dc214d4a10c4f841c5afe84382f783d534ceb8
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 24 07:16:10 2018 +0000

    Make "vtest" an alias for "varnishtest" in the first line.

diff --git a/bin/varnishtest/cmds.h b/bin/varnishtest/cmds.h
index e856e6084..d3244c258 100644
--- a/bin/varnishtest/cmds.h
+++ b/bin/varnishtest/cmds.h
@@ -43,6 +43,7 @@ CMD(shell)
 CMD(syslog)
 CMD(varnish)
 CMD(varnishtest)
+CMD(vtest)
 #undef CMD
 
 /*lint -restore */
diff --git a/bin/varnishtest/tests/a00000.vtc b/bin/varnishtest/tests/a00000.vtc
index 22e9f341b..8fb9ded32 100644
--- a/bin/varnishtest/tests/a00000.vtc
+++ b/bin/varnishtest/tests/a00000.vtc
@@ -10,7 +10,7 @@ shell {
 	echo 'shell "exit 9"' >> _.vtc
 }
 
-shell -exit 2 -expect {doesn't start with 'varnishtest'} {
+shell -exit 2 -expect {doesn't start with 'vtest' or 'varnishtest'} {
 	varnishtest -v _.vtc
 }
 
diff --git a/bin/varnishtest/tests/a00001.vtc b/bin/varnishtest/tests/a00001.vtc
index b6628d807..ea5b458f3 100644
--- a/bin/varnishtest/tests/a00001.vtc
+++ b/bin/varnishtest/tests/a00001.vtc
@@ -1,4 +1,4 @@
-varnishtest "Test Teken terminal emulator"
+vtest "Test Teken terminal emulator"
 
 feature cmd "vttest --version 2>&1 | grep -q Usage"
 
diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c
index d6168a2ae..cb0c2f07f 100644
--- a/bin/varnishtest/vtc_main.c
+++ b/bin/varnishtest/vtc_main.c
@@ -547,9 +547,11 @@ read_file(const char *fn, int ntest)
 		return (2);
 	}
 
-	if (strncmp(q, "varnishtest", 11) || !isspace(q[11])) {
+	if ((strncmp(q, "varnishtest", 11) || !isspace(q[11])) &&
+	    (strncmp(q, "vtest", 5) || !isspace(q[5]))) {
 		fprintf(stderr,
-		    "File \"%s\" doesn't start with 'varnishtest'\n", fn);
+		    "File \"%s\" doesn't start with"
+		    " 'vtest' or 'varnishtest'\n", fn);
 		free(p);
 		vtc_skip++;
 		return(2);
diff --git a/bin/varnishtest/vtc_misc.c b/bin/varnishtest/vtc_misc.c
index 196b35c65..0c47fe88a 100644
--- a/bin/varnishtest/vtc_misc.c
+++ b/bin/varnishtest/vtc_misc.c
@@ -49,17 +49,39 @@
 #include "vre.h"
 #include "vtim.h"
 
-/* SECTION: varnishtest varnishtest
+/* SECTION: vtest vtest
  *
  * This should be the first command in your vtc as it will identify the test
  * case with a short yet descriptive sentence. It takes exactly one argument, a
  * string, eg::
  *
- *         varnishtest "Check that varnishtest is actually a valid command"
+ *         vtest "Check that vtest is actually a valid command"
  *
  * It will also print that string in the log.
  */
 
+void v_matchproto_(cmd_f)
+cmd_vtest(CMD_ARGS)
+{
+
+	(void)priv;
+	(void)cmd;
+	(void)vl;
+
+	if (av == NULL)
+		return;
+	AZ(strcmp(av[0], "vtest"));
+
+	vtc_log(vl, 1, "TEST %s", av[1]);
+	AZ(av[2]);
+}
+
+/* SECTION: varnishtest varnishtest
+ *
+ * Alternate name for 'vtest', see above.
+ *
+ */
+
 void v_matchproto_(cmd_f)
 cmd_varnishtest(CMD_ARGS)
 {

From phk at FreeBSD.org  Mon Sep 24 11:26:16 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 24 Sep 2018 11:26:16 +0000 (UTC)
Subject: [master] 3ed8d7dba Whitespace OCD
Message-ID: <20180924112616.DB9886E1CD@lists.varnish-cache.org>


commit 3ed8d7dba5a8bfb1584147606ffc280babeee4ce
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 24 07:17:52 2018 +0000

    Whitespace OCD

diff --git a/bin/varnishtest/vtc_log.c b/bin/varnishtest/vtc_log.c
index d7ceefe89..94bee3d3f 100644
--- a/bin/varnishtest/vtc_log.c
+++ b/bin/varnishtest/vtc_log.c
@@ -155,7 +155,7 @@ vtc_log_emit(const struct vtclog *vl)
 		return;
 	AZ(pthread_mutex_lock(&vtclog_mtx));
 	assert(vtclog_left > l);
-	memcpy(vtclog_buf,VSB_data(vl->vsb), l);
+	memcpy(vtclog_buf, VSB_data(vl->vsb), l);
 	vtclog_buf += l;
 	*vtclog_buf = '\0';
 	vtclog_left -= l;
diff --git a/include/vsa.h b/include/vsa.h
index e5e342a26..67c3e6266 100644
--- a/include/vsa.h
+++ b/include/vsa.h
@@ -59,7 +59,7 @@ struct suckaddr *VSA_Build(void *d, const void *s, unsigned sal);
  * This VRT interface is for the VCC generated ACL code, which needs
  * to know the address family and a pointer to the actual address.
  */
-  
+
 int VSA_GetPtr(const struct suckaddr *sua, const unsigned char ** dst);
 
 #endif

From phk at FreeBSD.org  Mon Sep 24 11:26:17 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 24 Sep 2018 11:26:17 +0000 (UTC)
Subject: [master] 83b418220 A train-trip worth of python3 migration and other
 polishing
Message-ID: <20180924112617.274A76E1DB@lists.varnish-cache.org>


commit 83b4182202ea5d3beeb77c37b0d752c9d1828803
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 24 07:26:11 2018 +0000

    A train-trip worth of python3 migration and other polishing

diff --git a/bin/varnishtest/huffman_gen.py b/bin/varnishtest/huffman_gen.py
index 8da37b2ee..b89bbb43e 100755
--- a/bin/varnishtest/huffman_gen.py
+++ b/bin/varnishtest/huffman_gen.py
@@ -7,50 +7,50 @@ import sys
 regex = re.compile("^HPH\((.{4}), (.{10}), +(.{1,3})\)")
 
 if len(sys.argv) != 2:
-	print("{} takes one and only one argument".format(sys.argv[0]))
-	sys.exit(2)
+    print("{} takes one and only one argument".format(sys.argv[0]))
+    sys.exit(2)
 
 class sym:
-	def __init__(self, bigval, bigvall, chr = 0, esc = None):
-		self.vall = bigvall % 8 if bigvall % 8 else 8
-		self.val = bigval & ((1 << self.vall) - 1)
-		self.pfx = (bigval >> self.vall)# & 0xff
-		self.chr = chr
-		self.esc = esc
+    def __init__(self, bigval, bigvall, chr=0, esc=None):
+        self.vall = bigvall % 8 if bigvall % 8 else 8
+        self.val = bigval & ((1 << self.vall) - 1)
+        self.pfx = (bigval >> self.vall)# & 0xff
+        self.chr = chr
+        self.esc = esc
 
 tbls = {}
 msl = {} # max sym length
 
 f = open(sys.argv[1])
 for l in f:
-	grp = 1
-	match = regex.match(l)
-	if not match:
-		continue
-
-	chr = int(match.group(grp), 16)
-	grp += 1
-
-	val = int(match.group(grp), 16)
-	grp += 1
-
-	vall = int(match.group(grp))
-
-	s = sym(val, vall, chr)
-	if s.pfx not in tbls:
-		tbls[s.pfx] = {}
-
-	if (s.val in tbls[s.pfx]):
-		assert(tbls[s.pfx][s.val].e)
-	tbls[s.pfx][s.val] = s
-
-	# add the escape entry in the "previous" table
-	if s.pfx:
-		pp = s.pfx >> 8
-		pv = s.pfx & 0xff
-		if pp not in tbls:
-			tbls[pp] = {}
-		tbls[pp][pv] = sym(pv, 8, 0, "&tbl_{:x}".format(s.pfx))
+    grp = 1
+    match = regex.match(l)
+    if not match:
+        continue
+
+    char = int(match.group(grp), 16)
+    grp += 1
+
+    val = int(match.group(grp), 16)
+    grp += 1
+
+    vall = int(match.group(grp))
+
+    s = sym(val, vall, char)
+    if s.pfx not in tbls:
+        tbls[s.pfx] = {}
+
+    if s.val in tbls[s.pfx]:
+        assert tbls[s.pfx][s.val].e
+    tbls[s.pfx][s.val] = s
+
+    # add the escape entry in the "previous" table
+    if s.pfx:
+        pp = s.pfx >> 8
+        pv = s.pfx & 0xff
+        if pp not in tbls:
+            tbls[pp] = {}
+        tbls[pp][pv] = sym(pv, 8, 0, "&tbl_{:x}".format(s.pfx))
 f.close()
 
 # add the EOS case
@@ -64,33 +64,33 @@ print('''/* NB:  This file is machine generated, DO NOT EDIT!
 struct stbl;
 
 struct ssym {
-	uint8_t csm;		/* bits consumed */
-	uint8_t	chr;		/* character */
-	struct stbl *nxt;	/* next table */
+    uint8_t csm;        /* bits consumed */
+    uint8_t chr;        /* character */
+    struct stbl *nxt;       /* next table */
 };
 
 struct stbl {
-	int msk;
-	struct ssym *syms;
+    int msk;
+    struct ssym *syms;
 };
 ''')
 
 for pfx in sorted(tbls.keys(), reverse=True):
-	msl = max([ x.vall for x in tbls[pfx].values() ])
-	for s in tbls[pfx].values():
-		s.val = s.val << (msl - s.vall)
-
-	tbl = sorted(tbls[pfx].values(), key= lambda x: x.val)
-	print("\nstatic struct ssym sym_{:x}_array[] = {{".format(pfx))
-	for s in tbl:
-		for j in range(2 ** (msl - s.vall)):
-			print("	{} {{{}, {:3d}, {}}},".format(
-				"\t     " if j else "/* idx {:3d} */".format(s.val + j),
-				s.vall, s.chr % 256,
-				s.esc if s.esc else "NULL"))
-	print('''}};
+    msl = max([x.vall for x in tbls[pfx].values()])
+    for s in tbls[pfx].values():
+        s.val = s.val << (msl - s.vall)
+
+    tbl = sorted(tbls[pfx].values(), key=lambda x: x.val)
+    print("\nstatic struct ssym sym_{:x}_array[] = {{".format(pfx))
+    for s in tbl:
+        for j in range(2 ** (msl - s.vall)):
+            print("{} {{{}, {:3d}, {}}},".format(
+                "\t     " if j else "/* idx {:3d} */".format(s.val + j),
+                s.vall, s.chr % 256,
+                s.esc if s.esc else "NULL"))
+    print('''}};
 
 static struct stbl tbl_{:x} = {{
-	{},
-	sym_{:x}_array
+    {},
+    sym_{:x}_array
 }};'''.format(pfx, msl, pfx))
diff --git a/bin/varnishtest/witness.py b/bin/varnishtest/witness.py
index f57fdf221..a4f0fffa6 100644
--- a/bin/varnishtest/witness.py
+++ b/bin/varnishtest/witness.py
@@ -3,9 +3,9 @@
 # This script is in the public domain
 #
 # Run instructions:
-#	varnishtest -W -iv -j <pick_a_number> > _.w
-#	python witness.py
-#	dot -Tpng /tmp/_.dot > /tmp/_.png
+#       varnishtest -W -iv -j <pick_a_number> > _.w
+#       python witness.py
+#       dot -Tpng /tmp/_.dot > /tmp/_.png
 
 from __future__ import print_function
 
@@ -16,37 +16,37 @@ fi = open("_.w")
 fo = open("/tmp/_.dot", "w")
 
 fo.write('''digraph {
-	#rotate="90"
-	#page="8.2,11.7"
-	size="8.2,11.7"
-	rankdir="LR"
-	node [fontname="Inconsolata", fontsize="10"]
-	edge [fontname="Inconsolata", fontsize="10"]
+    #rotate="90"
+    #page="8.2,11.7"
+    size="8.2,11.7"
+    rankdir="LR"
+    node [fontname="Inconsolata", fontsize="10"]
+    edge [fontname="Inconsolata", fontsize="10"]
 ''')
 
 for i in fi:
-	l = "ROOT"
-	j = i.split()
-	if len(j) < 8:
-		continue
-	if j[1][0] != 'v':
-		continue
-	if j[3] != "vsl|":
-		continue
-	if j[5] != "Witness":
-		continue
-	t = j[7:]
-	tt = str(t)
-	if tt in d:
-		continue
-	d[tt] = True
-	for e in t:
-		f = e.split(",")
-		x = '"%s" -> "%s" [label="%s(%s)"]\n' % (l, f[0], f[1], f[2])
-		if not x in a:
-			a[x] = True
-			fo.write(x)
-		l = f[0]
+    l = "ROOT"
+    j = i.split()
+    if len(j) < 8:
+        continue
+    if j[1][0] != 'v':
+        continue
+    if j[3] != "vsl|":
+        continue
+    if j[5] != "Witness":
+        continue
+    t = j[7:]
+    tt = str(t)
+    if tt in d:
+        continue
+    d[tt] = True
+    for e in t:
+        f = e.split(",")
+        x = '"%s" -> "%s" [label="%s(%s)"]\n' % (l, f[0], f[1], f[2])
+        if not x in a:
+            a[x] = True
+            fo.write(x)
+        l = f[0]
 
 fo.write("}\n")
 
diff --git a/doc/sphinx/vtc-syntax.py b/doc/sphinx/vtc-syntax.py
index 298f4bb01..ba7bd6bd7 100644
--- a/doc/sphinx/vtc-syntax.py
+++ b/doc/sphinx/vtc-syntax.py
@@ -35,50 +35,49 @@ import re
 
 
 def parse_file(fn, cl, tl, sl):
-	p = False
-	section = ""
-	resec = re.compile("[ /]\* SECTION: ")
+    p = False
+    section = ""
+    resec = re.compile("[ /]\* SECTION: ")
 
-	f = open(fn, "r")
+    f = open(fn, "r")
 
-	for l in f:
-		if "*/" in l:
-			p = 0
-		if resec.match(l):
-			a = l.split()
-			section = a[2]
-			sl.append(section)
-			cl[section] = []
-			if len(a) > 3:
-				tl[section] = re.sub(
-					r"^[\t ]*\/?\* SECTION: [^ ]+ +",
-					"", l)
-			else:
-				tl[section] = ""
-			p = 1
-		elif p:
-			cl[section].append(re.sub(r"^ \* ?", "", l))
-	f.close()
+    for l in f:
+        if "*/" in l:
+            p = 0
+        if resec.match(l):
+            a = l.split()
+            section = a[2]
+            sl.append(section)
+            cl[section] = []
+            if len(a) > 3:
+                tl[section] = re.sub(
+                    r"^[\t ]*\/?\* SECTION: [^ ]+ +",
+                    "", l)
+            else:
+                tl[section] = ""
+            p = 1
+        elif p:
+            cl[section].append(re.sub(r"^ \* ?", "", l))
+    f.close()
 
 if __name__ == "__main__":
-	cl = {}
-	tl = {}
-	sl = []
-	for fn in sys.argv[1:]:
-		parse_file(fn, cl, tl, sl)
-	sl.sort()
-	for section in sl:
-		print(tl[section], end="")
-		a = section
-		c = section.count(".")
-		if c == 0:
-			r = "-"
-		elif c == 1:
-			r = "~"
-		elif c == 2:
-			r = "."
-		else:
-			r = "*"
-		print(re.sub(r".", r, tl[section]), end="")
-		print("".join(cl[section]))
-
+    cl = {}
+    tl = {}
+    sl = []
+    for fn in sys.argv[1:]:
+        parse_file(fn, cl, tl, sl)
+    sl.sort()
+    for section in sl:
+        print(tl[section], end="")
+        a = section
+        c = section.count(".")
+        if c == 0:
+            r = "-"
+        elif c == 1:
+            r = "~"
+        elif c == 2:
+            r = "."
+        else:
+            r = "*"
+        print(re.sub(r".", r, tl[section]), end="")
+        print("".join(cl[section]))
diff --git a/include/Makefile.am b/include/Makefile.am
index 27f40ec86..98050fd78 100644
--- a/include/Makefile.am
+++ b/include/Makefile.am
@@ -108,7 +108,7 @@ vcl.h: \
 	    $(top_srcdir)/include/vrt.h \
 	    $(top_srcdir)/doc/sphinx/reference/vcl_var.rst
 	mkdir -p $(top_builddir)/include/tbl
-	@PYTHON@ $(top_srcdir)/lib/libvcc/generate.py \
+	${PYTHON} $(top_srcdir)/lib/libvcc/generate.py \
 	    $(top_srcdir) $(top_builddir)
 
 GEN_H = \
@@ -128,7 +128,7 @@ vcs_version.h:
 	@if test -e $(top_srcdir)/.git || \
 	  ! test -f $(srcdir)/vmod_abi.h || \
 	  ! test -f $(srcdir)/vcs_version.h ; then \
-	    @PYTHON@ $(srcdir)/generate.py \
+	    ${PYTHON} $(srcdir)/generate.py \
 		$(top_srcdir) $(top_builddir) ; \
 	fi
 
diff --git a/include/generate.py b/include/generate.py
index 0d9d9b32b..e7d547593 100755
--- a/include/generate.py
+++ b/include/generate.py
@@ -32,22 +32,21 @@
 from __future__ import print_function
 
 import subprocess
-import collections
 import os
 import sys
 
 srcroot = "../.."
 buildroot = "../.."
 if len(sys.argv) == 3:
-	srcroot = sys.argv[1]
-	buildroot = sys.argv[2]
+    srcroot = sys.argv[1]
+    buildroot = sys.argv[2]
 elif len(sys.argv) != 1:
-	print("Two arguments or none")
-	exit(2)
+    print("Two arguments or none")
+    exit(2)
 
 #######################################################################
 def file_header(fo):
-	fo.write("""/*
+    fo.write("""/*
  * NB:  This file is machine generated, DO NOT EDIT!
  *
  * Edit and run include/generate.py instead.
@@ -66,25 +65,25 @@ v = subprocess.check_output([
 vcsfn = os.path.join(srcroot, "include", "vcs_version.h")
 
 try:
-	i = open(vcsfn).readline()
+    i = open(vcsfn).readline()
 except IOError:
-	i = ""
+    i = ""
 
 ident = "/* " + v + " */\n"
 
 if i != ident:
-	fo = open(vcsfn, "w")
-	fo.write(ident)
-	file_header(fo)
-	fo.write('#define VCS_Version "%s"\n' % v)
-	fo.close()
+    fo = open(vcsfn, "w")
+    fo.write(ident)
+    file_header(fo)
+    fo.write('#define VCS_Version "%s"\n' % v)
+    fo.close()
 
-	for i in open(os.path.join(buildroot, "Makefile")):
-		if i[:14] == "PACKAGE_STRING":
-			break
-	i = i.split("=")[1].strip()
+    for i in open(os.path.join(buildroot, "Makefile")):
+        if i[:14] == "PACKAGE_STRING":
+            break
+    i = i.split("=")[1].strip()
 
-	fo = open(os.path.join(srcroot, "include", "vmod_abi.h"), "w")
-	file_header(fo)
-	fo.write('#define VMOD_ABI_Version "%s %s"\n' % (i, v))
-	fo.close()
+    fo = open(os.path.join(srcroot, "include", "vmod_abi.h"), "w")
+    file_header(fo)
+    fo.write('#define VMOD_ABI_Version "%s %s"\n' % (i, v))
+    fo.close()
diff --git a/include/tbl/h2_settings.h b/include/tbl/h2_settings.h
index adfb416d4..c382e862d 100644
--- a/include/tbl/h2_settings.h
+++ b/include/tbl/h2_settings.h
@@ -47,6 +47,7 @@ H2_SETTING(					// rfc7540,l,2097,2103
 	0xffffffff,
 	0
 )
+
 #ifndef H2_SETTINGS_PARAM_ONLY
 H2_SETTING(					// rfc7540,l,2105,2114
 	ENABLE_PUSH,
@@ -58,6 +59,7 @@ H2_SETTING(					// rfc7540,l,2105,2114
 	H2CE_PROTOCOL_ERROR
 )
 #endif
+
 H2_SETTING(					// rfc7540,l,2116,2121
 	MAX_CONCURRENT_STREAMS,
 	max_concurrent_streams,
@@ -67,6 +69,7 @@ H2_SETTING(					// rfc7540,l,2116,2121
 	0xffffffff,
 	0
 )
+
 H2_SETTING(					// rfc7540,l,2139,2148
 	INITIAL_WINDOW_SIZE,
 	initial_window_size,
@@ -76,6 +79,7 @@ H2_SETTING(					// rfc7540,l,2139,2148
 	0x7fffffff,
 	H2CE_FLOW_CONTROL_ERROR
 )
+
 H2_SETTING(					// rfc7540,l,2150,2157
 	MAX_FRAME_SIZE,
 	max_frame_size,
@@ -85,6 +89,7 @@ H2_SETTING(					// rfc7540,l,2150,2157
 	0x00ffffff,
 	H2CE_PROTOCOL_ERROR
 )
+
 H2_SETTING(					// rfc7540,l,2159,2167
 	MAX_HEADER_LIST_SIZE,
 	max_header_list_size,
diff --git a/include/tbl/htc.h b/include/tbl/htc.h
index a93c0e520..4e9695491 100644
--- a/include/tbl/htc.h
+++ b/include/tbl/htc.h
@@ -36,10 +36,11 @@ HTC_STATUS(JUNK,	-5,	"junk",		"Received unexpected data")
 HTC_STATUS(CLOSE,	-4,	"close",	"Connection closed") // unused?
 HTC_STATUS(TIMEOUT,	-3,	"timeout",	"Timed out")
 HTC_STATUS(OVERFLOW,	-2,	"overflow",	"Buffer/workspace too small")
-HTC_STATUS(EOF,	-1,	"eof",		"Unexpected end of input")
+HTC_STATUS(EOF,		-1,	"eof",		"Unexpected end of input")
 HTC_STATUS(EMPTY,	 0,	"empty",	"Empty response")
-HTC_STATUS(MORE,	 1,	"more",	"More data required")
+HTC_STATUS(MORE,	 1,	"more",		"More data required")
 HTC_STATUS(COMPLETE,	 2,	"complete",	"Data complete (no error)")
-HTC_STATUS(IDLE,	 3,	"idle",	"Connection was closed while idle")
+HTC_STATUS(IDLE,	 3,	"idle",		"Connection was closed while idle")
 #undef HTC_STATUS
+
 /*lint -restore */
diff --git a/include/tbl/style.py b/include/tbl/style.py
index a29459c0a..a88c475a3 100644
--- a/include/tbl/style.py
+++ b/include/tbl/style.py
@@ -7,66 +7,68 @@ from __future__ import print_function
 import glob
 
 def check_file(fn):
-	s = 0
-	ll = []
-	for l in open(fn):
-		ll.append(l)
+    print("Check", fn)
+    ll = []
+    for l in open(fn):
+        ll.append(l)
 
-	assert ll.pop(0)[:2] == "/*"
+    assert ll.pop(0)[:2] == "/*"
 
-	while ll.pop(0) != " */\n":
-		continue
+    while ll.pop(0) != " */\n":
+        continue
 
-	assert len(ll) > 5
+    assert len(ll) > 5
 
-	assert ll.pop(0) == "\n"
-	assert ll.pop(0) == "/*lint -save -e525 -e539 */\n"
-	assert ll.pop(0) == "\n"
+    assert ll.pop(0) == "\n"
+    i = ll.pop(0)
+    assert i == "/*lint -save -e525 -e539 */\n" or \
+           i == "/*lint -save -e525 -e539 -e835 */\n"
+    assert ll.pop(0) == "\n"
 
-	assert ll.pop(-1) == "/*lint -restore */\n"
-	assert ll.pop(-1) == "\n"
+    assert ll.pop(-1) == "/*lint -restore */\n"
+    assert ll.pop(-1) == "\n"
 
-	for i in range(0, len(ll) -1):
-		assert ll[i] != "\n" or ll[i+1] != "\n"
-		assert ll[i] != ")\n" or ll[i+1] == "\n" or ll[i+1][0] == "#"
+    for i in range(0, len(ll) -1):
+        assert ll[i] != "\n" or ll[i+1] != "\n"
+        assert ll[i] != ")\n" or ll[i+1] == "\n" or ll[i+1][0] == "#"
 
-	m = {}
-	while len(ll) > 0:
-		i = ll.pop(0)
-		if i == "\n":
-			continue
-		l = i.lstrip()
-		if l[0] >= 'A' and l[0] <= 'Z':
-			j = l.split('(')
-			m[j[0]] = "Called"
-			l = l.split('//')[0]
-			l = l.split('/*')[0]
-			l = l.rstrip()
-			if l[-1] != ')':
-				while ll.pop(0) != ')\n':
-					continue
-		elif l[0] == "#":
-			j = l[1:].lstrip().split()
-			# print("#", j[0])
-			if j[0] == "define":
-				m[j[1].split("(")[0].strip()] = "Defined"
-			if j[0] == "undef":
-				m[j[1]] = "Undef"
-			while l[-2:] == "\\\n":
-				l = ll.pop(0)
-		else:
-			pass
-			# print(l)
-	rv = 0
-	for i in m:
-		if m[i] != "Undef":
-			print("ERROR", fn, i, m[i])
-			rv += 1
-	return rv
+    m = {}
+    while ll:
+        i = ll.pop(0)
+        if i == "\n":
+            continue
+        l = i.lstrip()
+        if l[0] >= 'A' and l[0] <= 'Z':
+            j = l.split('(')
+            m[j[0]] = "Called"
+            l = l.split('//')[0]
+            l = l.split('/*')[0]
+            l = l.rstrip()
+            if l[-1] != ')':
+                while ll.pop(0).strip() != ')':
+                    continue
+        elif l[0] == "#":
+            j = l[1:].lstrip().split()
+            # print("#", j[0])
+            if j[0] == "define":
+                m[j[1].split("(")[0].strip()] = "Defined"
+            if j[0] == "undef":
+                m[j[1]] = "Undef"
+            while l[-2:] == "\\\n":
+                l = ll.pop(0)
+        else:
+            pass
+            # print(l)
+    rv = 0
+    for i in m:
+        if m[i] != "Undef":
+            print("ERROR", fn, i, m[i])
+            rv += 1
+    return rv
 
 rv = 0
 for fn in glob.glob("*.h"):
-	rv += check_file(fn)
+    rv += check_file(fn)
 if rv != 0:
-	print(rv, "Errors")
+    print(rv, "Errors")
 exit(rv)
diff --git a/include/tbl/vsl_tags.h b/include/tbl/vsl_tags.h
index 0997d72f8..294da9431 100644
--- a/include/tbl/vsl_tags.h
+++ b/include/tbl/vsl_tags.h
@@ -181,16 +181,17 @@ SLTM(Length, 0, "Size of object body",
 	"Logs the size of a fetch object body.\n\n"
 )
 
-/* XXX generate HTC info from tbl include */
-#if 0
-#include <stdio.h>
-int main(void) {
-#define HTC_STATUS(e, n, s, l) \
-	printf("\t\"\\t* %s (%d): %s\\n\"\n", s, n, l);
-#include "include/tbl/htc.h"
-	return (0);
-}
-#endif
+/*
+ * XXX generate HTC info below from tbl include
+ *
+ * #include <stdio.h>
+ * int main(void) {
+ * #define HTC_STATUS(e, n, s, l) \
+ *	printf("\t\"\\t* %s (%d): %s\\n\"\n", s, n, l);
+ * #include "include/tbl/htc.h"
+ *	return (0);
+ * }
+ */
 
 SLTM(FetchError, 0, "Error while fetching object",
 	"Logs the error message of a failed fetch operation.\n\n"
diff --git a/include/tbl/waiters.h b/include/tbl/waiters.h
index f1f1d4900..f2fcf6446 100644
--- a/include/tbl/waiters.h
+++ b/include/tbl/waiters.h
@@ -44,3 +44,5 @@
 
 WAITER(poll)
 #undef WAITER
+
+/*lint -restore */
diff --git a/lib/libvarnishapi/generate.py b/lib/libvarnishapi/generate.py
index a7b505c74..d1357c985 100755
--- a/lib/libvarnishapi/generate.py
+++ b/lib/libvarnishapi/generate.py
@@ -37,122 +37,121 @@ import copy
 srcroot = "../.."
 buildroot = "../.."
 if len(sys.argv) == 3:
-	srcroot = sys.argv[1]
-	buildroot = sys.argv[2]
+    srcroot = sys.argv[1]
+    buildroot = sys.argv[2]
 
 #######################################################################
 # These are our tokens
 
 tokens = {
-	# Numerical comparisons
-	"T_EQ":         "==",
-	"T_NEQ":        "!=",
-	"T_LEQ":        "<=",
-	"T_GEQ":        ">=",
-
-	# String comparisons
-	"T_SEQ":        "eq",
-	"T_SNEQ":       "ne",
-
-	# Regular expression matching
-	"T_NOMATCH":    "!~",
-
-	# Boolean operators
-	"T_AND":        "and",
-	"T_OR":         "or",
-	"T_NOT":        "not",
-
-	# Miscellaneous
-	None:           "<>~[]{}():,",
-
-	# These have handwritten recognizers
-	"VAL":          None,
-	"EOI":          None,
-
-	# Special
-	"T_TRUE":       None,
-	"VXID":         "vxid",
+    # Numerical comparisons
+    "T_EQ":     "==",
+    "T_NEQ":    "!=",
+    "T_LEQ":    "<=",
+    "T_GEQ":    ">=",
+
+    # String comparisons
+    "T_SEQ":    "eq",
+    "T_SNEQ":       "ne",
+
+    # Regular expression matching
+    "T_NOMATCH":    "!~",
+
+    # Boolean operators
+    "T_AND":    "and",
+    "T_OR":     "or",
+    "T_NOT":    "not",
+
+    # Miscellaneous
+    None:       "<>~[]{}():,",
+
+    # These have handwritten recognizers
+    "VAL":      None,
+    "EOI":      None,
+
+    # Special
+    "T_TRUE":       None,
+    "VXID":     "vxid",
 }
 
 #######################################################################
 # Emit a function to recognize tokens in a string
 
 def emit_vxp_fixed_token(fo, tokens):
-	recog = list()
-	emit = dict()
-	for i in tokens:
-		j = tokens[i]
-		if (j != None):
-			recog.append(j)
-			emit[j] = i
-
-	recog.sort()
-	rrecog = copy.copy(recog)
-	rrecog.sort(key = lambda x: -len(x))
-
-	fo.write("""
+    recog = list()
+    emit = dict()
+    for i in tokens:
+        j = tokens[i]
+        if j is not None:
+            recog.append(j)
+            emit[j] = i
+
+    recog.sort()
+    rrecog = copy.copy(recog)
+    rrecog.sort(key=lambda x: -len(x))
+
+    fo.write("""
 unsigned
 vxp_fixed_token(const char *p, const char **q)
 {
 
 \tswitch (p[0]) {
 """)
-	last_initial = None
-	for i in recog:
-		if (i[0] == last_initial):
-			continue
-		last_initial = i[0]
-		fo.write("\tcase '%s':\n" % last_initial)
-		for j in rrecog:
-			if (j[0] != last_initial):
-				continue
-
-			fo.write("\t\tif (")
-			k = 1
-			l = len(j)
-			while (k < l):
-				fo.write("p[%d] == '%s'" % (k, j[k]))
-				fo.write(" &&\n\t\t    ")
-				k += 1
-			fo.write("(isword(p[%d]) ? !isword(p[%d]) : 1)) {\n" %
-				(l - 1, l))
-			fo.write("\t\t\t*q = p + %d;\n" % l)
-			fo.write("\t\t\treturn (%s);\n" % emit[j])
-			fo.write("\t\t}\n")
-		fo.write("\t\treturn (0);\n")
-
-	fo.write("\tdefault:\n\t\treturn (0);\n\t}\n}\n")
+    last_initial = None
+    for i in recog:
+        if i[0] == last_initial:
+            continue
+        last_initial = i[0]
+        fo.write("\tcase '%s':\n" % last_initial)
+        for j in rrecog:
+            if j[0] != last_initial:
+                continue
+
+            fo.write("\t\tif (")
+            k = 1
+            l = len(j)
+            while k < l:
+                fo.write("p[%d] == '%s'" % (k, j[k]))
+                fo.write(" &&\n\t\t    ")
+                k += 1
+            fo.write("(isword(p[%d]) ? !isword(p[%d]) : 1)) {\n" % (l - 1, l))
+            fo.write("\t\t\t*q = p + %d;\n" % l)
+            fo.write("\t\t\treturn (%s);\n" % emit[j])
+            fo.write("\t\t}\n")
+        fo.write("\t\treturn (0);\n")
+
+    fo.write("\tdefault:\n\t\treturn (0);\n\t}\n}\n")
 
 #######################################################################
 # Emit the vxp_tnames (token->string) conversion array
 
 def emit_vxp_tnames(fo, tokens):
-	fo.write("\nconst char * const vxp_tnames[256] = {\n")
-	l = list(tokens.keys())
-	l.sort()
-	for i in l:
-		j = tokens[i]
-		if j == None:
-			j = i
-		if i[0] == "'":
-			j = i
-		fo.write("\t[%s] = \"%s\",\n" % (i, j))
-	fo.write("};\n")
+    fo.write("\nconst char * const vxp_tnames[256] = {\n")
+    l = list(tokens.keys())
+    l.sort()
+    for i in l:
+        j = tokens[i]
+        if j is None:
+            j = i
+        if i[0] == "'":
+            j = i
+        fo.write("\t[%s] = \"%s\",\n" % (i, j))
+    fo.write("};\n")
 
 #######################################################################
 
 def polish_tokens(tokens):
-	# Expand single char tokens
-	st = tokens[None]
-	del tokens[None]
+    # Expand single char tokens
+    st = tokens[None]
+    del tokens[None]
 
-	for i in st:
-		tokens["'" + i + "'"] = i
+    for i in st:
+        tokens["'" + i + "'"] = i
 
 #######################################################################
 
 def file_header(fo):
-	fo.write("""/*
+    fo.write("""/*
  * NB:  This file is machine generated, DO NOT EDIT!
  *
  * Edit and run lib/libvarnishapi/generate.py instead
@@ -171,11 +170,11 @@ j = 128
 l = list(tokens.keys())
 l.sort()
 for i in l:
-	if i[0] == "'":
-		continue
-	fo.write("#define\t%s %d\n" % (i, j))
-	j += 1
-	assert j < 256
+    if i[0] == "'":
+        continue
+    fo.write("#define\t%s %d\n" % (i, j))
+    j += 1
+    assert j < 256
 
 fo.close()
 
diff --git a/lib/libvcc/generate.py b/lib/libvcc/generate.py
index 4328c7072..5dba1a37a 100755
--- a/lib/libvcc/generate.py
+++ b/lib/libvcc/generate.py
@@ -32,10 +32,6 @@
 
 from __future__ import print_function
 
-import subprocess
-import collections
-import os
-
 #######################################################################
 # These are our tokens
 
@@ -51,112 +47,112 @@ from os.path import join
 srcroot = "../.."
 buildroot = "../.."
 if len(sys.argv) == 3:
-	srcroot = sys.argv[1]
-	buildroot = sys.argv[2]
+    srcroot = sys.argv[1]
+    buildroot = sys.argv[2]
 elif len(sys.argv) != 1:
-	print("Two arguments or none")
-	exit(2)
+    print("Two arguments or none")
+    exit(2)
 
 tokens = {
-	"T_INC":	"++",
-	"T_DEC":	"--",
-	"T_CAND":	"&&",
-	"T_COR":	"||",
-	"T_LEQ":	"<=",
-	"T_EQ":		"==",
-	"T_NEQ":	"!=",
-	"T_GEQ":	">=",
-	"T_SHR":	">>",
-	"T_SHL":	"<<",
-	"T_INCR":	"+=",
-	"T_DECR":	"-=",
-	"T_MUL":	"*=",
-	"T_DIV":	"/=",
-	"T_NOMATCH":	"!~",
-
-	# Single char tokens, for convenience on one line
-	None:		"{}()*+-/%><=;!&.|~,",
-
-	# These have handwritten recognizers
-	"ID":		None,
-	"CNUM":		None,
-	"FNUM":		None,
-	"CSTR":		None,
-	"EOI":		None,
-	"CSRC":		None,
+    "T_INC":    "++",
+    "T_DEC":    "--",
+    "T_CAND":       "&&",
+    "T_COR":    "||",
+    "T_LEQ":    "<=",
+    "T_EQ":     "==",
+    "T_NEQ":    "!=",
+    "T_GEQ":    ">=",
+    "T_SHR":    ">>",
+    "T_SHL":    "<<",
+    "T_INCR":       "+=",
+    "T_DECR":       "-=",
+    "T_MUL":    "*=",
+    "T_DIV":    "/=",
+    "T_NOMATCH":    "!~",
+
+    # Single char tokens, for convenience on one line
+    None:       "{}()*+-/%><=;!&.|~,",
+
+    # These have handwritten recognizers
+    "ID":       None,
+    "CNUM":     None,
+    "FNUM":     None,
+    "CSTR":     None,
+    "EOI":      None,
+    "CSRC":     None,
 }
 
 #######################################################################
 # Our methods and actions
 
 returns = (
-	###############################################################
-	# Client side
-
-	('recv',
-		"C",
-		('fail', 'synth', 'restart', 'pass', 'pipe', 'hash', 'purge', 'vcl')
-	),
-	('pipe',
-		"C",
-		('fail', 'synth', 'pipe',)
-	),
-	('pass',
-		"C",
-		('fail', 'synth', 'restart', 'fetch',)
-	),
-	('hash',
-		"C",
-		('fail', 'lookup',)
-	),
-	('purge',
-		"C",
-		('fail', 'synth', 'restart',)
-	),
-	('miss',
-		"C",
-		('fail', 'synth', 'restart', 'pass', 'fetch',)
-	),
-	('hit',
-		"C",
-		('fail', 'synth', 'restart', 'pass', 'miss', 'deliver',)
-	),
-	('deliver',
-		"C",
-		('fail', 'synth', 'restart', 'deliver',)
-	),
-	('synth',
-		"C",
-		('fail', 'restart', 'deliver',)
-	),
-
-	###############################################################
-	# Backend-fetch
-
-	('backend_fetch',
-		"B",
-		('fail', 'fetch', 'abandon')
-	),
-	('backend_response',
-		"B",
-		('fail', 'deliver', 'retry', 'abandon', 'pass')
-	),
-	('backend_error',
-		"B",
-		('fail', 'deliver', 'retry', 'abandon')
-	),
-
-	###############################################################
-	# Housekeeping
-
-	('init',
-		"H",
-		('ok', 'fail')
-	),
-	('fini',
-		"H",
-		('ok',)
-	),
+    ###############################################################
+    # Client side
+
+    ('recv',
+     "C",
+     ('fail', 'synth', 'restart', 'pass', 'pipe', 'hash', 'purge', 'vcl')
+    ),
+    ('pipe',
+     "C",
+     ('fail', 'synth', 'pipe',)
+    ),
+    ('pass',
+     "C",
+     ('fail', 'synth', 'restart', 'fetch',)
+    ),
+    ('hash',
+     "C",
+     ('fail', 'lookup',)
+    ),
+    ('purge',
+     "C",
+     ('fail', 'synth', 'restart',)
+    ),
+    ('miss',
+     "C",
+     ('fail', 'synth', 'restart', 'pass', 'fetch',)
+    ),
+    ('hit',
+     "C",
+     ('fail', 'synth', 'restart', 'pass', 'miss', 'deliver',)
+    ),
+    ('deliver',
+     "C",
+     ('fail', 'synth', 'restart', 'deliver',)
+    ),
+    ('synth',
+     "C",
+     ('fail', 'restart', 'deliver',)
+    ),
+
+    ###############################################################
+    # Backend-fetch
+
+    ('backend_fetch',
+     "B",
+     ('fail', 'fetch', 'abandon')
+    ),
+    ('backend_response',
+     "B",
+     ('fail', 'deliver', 'retry', 'abandon', 'pass')
+    ),
+    ('backend_error',
+     "B",
+     ('fail', 'deliver', 'retry', 'abandon')
+    ),
+
+    ###############################################################
+    # Housekeeping
+
+    ('init',
+     "H",
+     ('ok', 'fail')
+    ),
+    ('fini',
+     "H",
+     ('ok',)
+    ),
 )
 
 #######################################################################
@@ -170,152 +166,151 @@ returns = (
 varprotos = {}
 
 def varproto(s):
-	if not s in varprotos:
-		fh.write(s + ";\n")
-		varprotos[s] = True
+    if not s in varprotos:
+        fh.write(s + ";\n")
+        varprotos[s] = True
 
 class vardef(object):
-	def __init__(self, nam, typ, rd, wr, wu, vlo, vhi):
-		self.nam = nam
-		self.typ = typ
-		self.rd = rd
-		self.wr = wr
-		self.uns = wu
-		self.vlo = vlo
-		self.vhi = vhi
-
-		self.emit()
-
-	def emit(self):
-		fh.write("\n")
-		fo.write("\n")
-		cnam = self.nam.replace(".", "_")
-		ctyp = vcltypes[self.typ]
-
-		# fo.write("\t{ \"%s\", %s,\n" % (nm, self.typ))
-		fo.write("\tsym = VCC_MkSym(tl, \"%s\", " % self.nam)
-		if (self.typ == "HEADER"):
-			fo.write(" SYM_NONE, %d, %d);\n" % (self.vlo, self.vhi))
-			fo.write("\tAN(sym);\n");
-			fo.write("\tsym->wildcard = vcc_Var_Wildcard;\n")
-		else:
-			fo.write(" SYM_VAR, %d, %d);\n" % (self.vlo, self.vhi))
-		fo.write("\tAN(sym);\n")
-		fo.write("\tsym->type = %s;\n" % self.typ)
-		fo.write("\tsym->eval = vcc_Eval_Var;\n")
-
-		if self.typ == "HEADER":
-			fo.write('\tsym->rname = "HDR_')
-			fo.write(self.nam.split(".")[0].upper())
-			fo.write('";\n')
-		elif len(self.rd):
-			fo.write('\tsym->rname = "VRT_r_%s(ctx)";\n' % cnam)
-			varproto("VCL_" + self.typ + " VRT_r_%s(VRT_CTX)" % cnam)
-		fo.write("\tsym->r_methods =\n")
-		restrict(fo, self.rd)
-		fo.write(";\n")
-
-		if self.typ == "HEADER":
-			fo.write('\tsym->lname = "HDR_')
-			fo.write(self.nam.split(".")[0].upper())
-			fo.write('";\n')
-		elif len(self.wr):
-			fo.write('\tsym->lname = "VRT_l_%s(ctx, ";\n' % cnam)
-			s = "void VRT_l_%s(VRT_CTX, " % cnam
-			if self.typ != "STRING" and self.typ != "BODY":
-				s += "VCL_" + self.typ + ")"
-			else:
-				s += ctyp.c + ", ...)"
-			varproto(s);
-		fo.write("\tsym->w_methods =\n")
-		restrict(fo, self.wr)
-		fo.write(";\n")
-
-		if len(self.uns):
-			varproto("void VRT_u_%s(VRT_CTX)" % cnam)
-			fo.write('\tsym->uname = "VRT_u_%s(ctx)";\n' % cnam)
-		fo.write('\tsym->u_methods =\n')
-		restrict(fo, self.uns)
-		fo.write(";\n")
+    def __init__(self, nam, typ, rd, wr, wu, vlo, vhi):
+        self.nam = nam
+        self.typ = typ
+        self.rd = rd
+        self.wr = wr
+        self.uns = wu
+        self.vlo = vlo
+        self.vhi = vhi
+
+        self.emit()
+
+    def emit(self):
+        fh.write("\n")
+        fo.write("\n")
+        cnam = self.nam.replace(".", "_")
+        ctyp = vcltypes[self.typ]
+
+        # fo.write("\t{ \"%s\", %s,\n" % (nm, self.typ))
+        fo.write("\tsym = VCC_MkSym(tl, \"%s\", " % self.nam)
+        if self.typ == "HEADER":
+            fo.write(" SYM_NONE, %d, %d);\n" % (self.vlo, self.vhi))
+            fo.write("\tAN(sym);\n")
+            fo.write("\tsym->wildcard = vcc_Var_Wildcard;\n")
+        else:
+            fo.write(" SYM_VAR, %d, %d);\n" % (self.vlo, self.vhi))
+        fo.write("\tAN(sym);\n")
+        fo.write("\tsym->type = %s;\n" % self.typ)
+        fo.write("\tsym->eval = vcc_Eval_Var;\n")
+
+        if self.typ == "HEADER":
+            fo.write('\tsym->rname = "HDR_')
+            fo.write(self.nam.split(".")[0].upper())
+            fo.write('";\n')
+        elif self.rd:
+            fo.write('\tsym->rname = "VRT_r_%s(ctx)";\n' % cnam)
+            varproto("VCL_" + self.typ + " VRT_r_%s(VRT_CTX)" % cnam)
+        fo.write("\tsym->r_methods =\n")
+        restrict(fo, self.rd)
+        fo.write(";\n")
+
+        if self.typ == "HEADER":
+            fo.write('\tsym->lname = "HDR_')
+            fo.write(self.nam.split(".")[0].upper())
+            fo.write('";\n')
+        elif self.wr:
+            fo.write('\tsym->lname = "VRT_l_%s(ctx, ";\n' % cnam)
+            s = "void VRT_l_%s(VRT_CTX, " % cnam
+            if self.typ != "STRING" and self.typ != "BODY":
+                s += "VCL_" + self.typ + ")"
+            else:
+                s += ctyp.c + ", ...)"
+            varproto(s)
+        fo.write("\tsym->w_methods =\n")
+        restrict(fo, self.wr)
+        fo.write(";\n")
+
+        if self.uns:
+            varproto("void VRT_u_%s(VRT_CTX)" % cnam)
+            fo.write('\tsym->uname = "VRT_u_%s(ctx)";\n' % cnam)
+        fo.write('\tsym->u_methods =\n')
+        restrict(fo, self.uns)
+        fo.write(";\n")
 
 def parse_vcl(x):
-	vlo,vhi = (0,99)
-	x = x.split()
-	if x[0] == "VCL" and x[1] == "<=":
-		vhi = int(float(x[2]) * 10)
-	elif x[0] == "VCL" and x[1] == ">=":
-		vlo = int(float(x[2]) * 10)
-	else:
-		print("Unknown variable version spec")
-		print("XXX", x, vlo, vhi)
-		exit(2)
-	return vlo,vhi
+    vlo, vhi = (0, 99)
+    x = x.split()
+    if x[0] == "VCL" and x[1] == "<=":
+        vhi = int(float(x[2]) * 10)
+    elif x[0] == "VCL" and x[1] == ">=":
+        vlo = int(float(x[2]) * 10)
+    else:
+        print("Unknown variable version spec")
+        print("XXX", x, vlo, vhi)
+        exit(2)
+    return vlo, vhi
 
 def parse_var(ln):
-	l1 = ln.pop(0).split("``")
-	assert len(l1) in (1,3)
-	vn = l1[0].strip()
-	if vn[-1] == '*':
-		vn = vn[:-1]
-	if len(l1) == 3:
-		vlo,vhi = parse_vcl(l1[1])
-	else:
-		vlo,vhi = 0,99
-	vr = []
-	vw = []
-	vu = []
-	while True:
-		l = ln.pop(0)
-		if l == "":
-			continue
-		j = l.split()
-		if j[0] == "Type:":
-			assert len(j) == 2
-			vt = j[1]
-			continue
-		if j[0] == "Readable" and j[1] == "from:":
-			for i in j[2:]:
-				vr.append(i.strip(",."))
-			continue
-		if j[0] == "Writable" and j[1] == "from:":
-			for i in j[2:]:
-				vw.append(i.strip(",."))
-			continue
-		if j[0] == "Unsetable" and j[1] == "from:":
-			for i in j[2:]:
-				vu.append(i.strip(",."))
-			continue
-		break
-	if vn[:8] != "storage.":
-		vardef(vn, vt, vr, vw, vu, vlo, vhi)
+    l1 = ln.pop(0).split("``")
+    assert len(l1) in (1, 3)
+    vn = l1[0].strip()
+    if vn[-1] == '*':
+        vn = vn[:-1]
+    if len(l1) == 3:
+        vlo, vhi = parse_vcl(l1[1])
+    else:
+        vlo, vhi = 0, 99
+    vr = []
+    vw = []
+    vu = []
+    while True:
+        l = ln.pop(0)
+        if l == "":
+            continue
+        j = l.split()
+        if j[0] == "Type:":
+            assert len(j) == 2
+            vt = j[1]
+            continue
+        if j[0] == "Readable" and j[1] == "from:":
+            for i in j[2:]:
+                vr.append(i.strip(",."))
+            continue
+        if j[0] == "Writable" and j[1] == "from:":
+            for i in j[2:]:
+                vw.append(i.strip(",."))
+            continue
+        if j[0] == "Unsetable" and j[1] == "from:":
+            for i in j[2:]:
+                vu.append(i.strip(",."))
+            continue
+        break
+    if vn[:8] != "storage.":
+        vardef(vn, vt, vr, vw, vu, vlo, vhi)
 
 def parse_var_doc(fn):
-	s = 0
-	l = []
-	for i in open(fn):
-		l.append(i.rstrip())
-	for n in range(0, len(l)):
-		j = l[n].split()
-		if len(j) != 2 or j[0] != "Type:" or not l[n][0].isspace():
-			continue
-		m = n
-		while m < len(l) and ( l[m] == "" or l[m][0].isspace() ):
-			m += 1
-		parse_var(l[n-2:m-1])
+    l = []
+    for i in open(fn):
+        l.append(i.rstrip())
+    for n in range(0, len(l)):
+        j = l[n].split()
+        if len(j) != 2 or j[0] != "Type:" or not l[n][0].isspace():
+            continue
+        m = n
+        while m < len(l) and (l[m] == "" or l[m][0].isspace()):
+            m += 1
+        parse_var(l[n-2:m-1])
 
 stv_variables = (
-	('free_space',	'BYTES',	"0.", 'storage.<name>.free_space', """
-	Free space available in the named stevedore. Only available for
-	the malloc stevedore.
-	"""),
-	('used_space',	'BYTES',	"0.", 'storage.<name>.used_space', """
-	Used space in the named stevedore. Only available for the malloc
-	stevedore.
-	"""),
-	('happy',	'BOOL',		"0", 'storage.<name>.happy', """
-	Health status for the named stevedore. Not available in any of the
-	current stevedores.
-	"""),
+    ('free_space', 'BYTES', "0.", 'storage.<name>.free_space', """
+    Free space available in the named stevedore. Only available for
+    the malloc stevedore.
+    """),
+    ('used_space', 'BYTES', "0.", 'storage.<name>.used_space', """
+    Used space in the named stevedore. Only available for the malloc
+    stevedore.
+    """),
+    ('happy', 'BOOL', "0", 'storage.<name>.happy', """
+    Health status for the named stevedore. Not available in any of the
+    current stevedores.
+    """),
 )
 
 #######################################################################
@@ -324,11 +319,11 @@ stv_variables = (
 vcltypes = {}
 
 class vcltype(object):
-	def __init__(self, name, ctype, internal=False):
-		self.name = name
-		self.c = ctype
-		self.internal = internal
-		vcltypes[name] = self
+    def __init__(self, name, ctype, internal=False):
+        self.name = name
+        self.c = ctype
+        self.internal = internal
+        vcltypes[name] = self
 
 
 vcltype("STRINGS", "void", True)
@@ -338,19 +333,19 @@ vcltype("SUB", "void*", True)
 fi = open(join(srcroot, "include/vrt.h"))
 
 for i in fi:
-	j = i.split()
-	if len(j) < 3:
-		continue
-	if j[0] != "typedef":
-		continue
-	if j[-1][-1] != ";":
-		continue
-	if j[-1][-2] == ")":
-		continue
-	if j[-1][:4] != "VCL_":
-		continue
-	d = " ".join(j[1:-1])
-	vcltype(j[-1][4:-1], d)
+    j = i.split()
+    if len(j) < 3:
+        continue
+    if j[0] != "typedef":
+        continue
+    if j[-1][-1] != ";":
+        continue
+    if j[-1][-2] == ")":
+        continue
+    if j[-1][:4] != "VCL_":
+        continue
+    d = " ".join(j[1:-1])
+    vcltype(j[-1][4:-1], d)
 fi.close()
 
 #######################################################################
@@ -360,20 +355,20 @@ fi.close()
 
 #######################################################################
 def emit_vcl_fixed_token(fo, tokens):
-	"Emit a function to recognize tokens in a string"
-	recog = list()
-	emit = dict()
-	for i in tokens:
-		j = tokens[i]
-		if j is not None:
-			recog.append(j)
-			emit[j] = i
-
-	recog.sort()
-	rrecog = copy.copy(recog)
-	rrecog.sort(key=lambda x: -len(x))
-
-	fo.write("""
+    "Emit a function to recognize tokens in a string"
+    recog = list()
+    emit = dict()
+    for i in tokens:
+        j = tokens[i]
+        if j is not None:
+            recog.append(j)
+            emit[j] = i
+
+    recog.sort()
+    rrecog = copy.copy(recog)
+    rrecog.sort(key=lambda x: -len(x))
+
+    fo.write("""
 #define M1()\tdo {*q = p + 1; return (p[0]); } while (0)
 #define M2(c,t)\tdo {if (p[1] == (c)) { *q = p + 2; return (t); }} while (0)
 
@@ -383,131 +378,130 @@ vcl_fixed_token(const char *p, const char **q)
 
 \tswitch (p[0]) {
 """)
-	last_initial = None
-	for i in recog:
-		if (i[0] == last_initial):
-			continue
-		last_initial = i[0]
-		fo.write("\tcase '%s':\n" % last_initial)
-		need_ret = True
-		for j in rrecog:
-			if (j[0] != last_initial):
-				continue
-			if len(j) == 2:
-				fo.write("\t\tM2('%s', %s);\n" %
-				    (j[1], emit[j]))
-			elif len(j) == 1:
-				fo.write("\t\tM1();\n")
-				need_ret = False
-			else:
-				fo.write("\t\tif (")
-				k = 1
-				l = len(j)
-				while (k < l):
-					fo.write("p[%d] == '%s'" % (k, j[k]))
-					fo.write(" &&")
-					if (k % 3) == 0:
-						fo.write("\n\t\t    ")
-					else:
-						fo.write(" ")
-					k += 1
-				fo.write("!isvar(p[%d])) {\n" % l)
-				fo.write("\t\t\t*q = p + %d;\n" % l)
-				fo.write("\t\t\treturn (%s);\n" % emit[j])
-				fo.write("\t\t}\n")
-		if need_ret:
-			fo.write("\t\treturn (0);\n")
-	fo.write("\tdefault:\n\t\treturn (0);\n\t}\n}\n")
+    last_initial = None
+    for i in recog:
+        if i[0] == last_initial:
+            continue
+        last_initial = i[0]
+        fo.write("\tcase '%s':\n" % last_initial)
+        need_ret = True
+        for j in rrecog:
+            if j[0] != last_initial:
+                continue
+            if len(j) == 2:
+                fo.write("\t\tM2('%s', %s);\n" % (j[1], emit[j]))
+            elif len(j) == 1:
+                fo.write("\t\tM1();\n")
+                need_ret = False
+            else:
+                fo.write("\t\tif (")
+                k = 1
+                l = len(j)
+                while k < l:
+                    fo.write("p[%d] == '%s'" % (k, j[k]))
+                    fo.write(" &&")
+                    if (k % 3) == 0:
+                        fo.write("\n\t\t    ")
+                    else:
+                        fo.write(" ")
+                    k += 1
+                fo.write("!isvar(p[%d])) {\n" % l)
+                fo.write("\t\t\t*q = p + %d;\n" % l)
+                fo.write("\t\t\treturn (%s);\n" % emit[j])
+                fo.write("\t\t}\n")
+        if need_ret:
+            fo.write("\t\treturn (0);\n")
+    fo.write("\tdefault:\n\t\treturn (0);\n\t}\n}\n")
 
 
 #######################################################################
 def emit_vcl_tnames(fo, tokens):
-	"Emit the vcl_tnames (token->string) conversion array"
-	fo.write("\nconst char * const vcl_tnames[256] = {\n")
-	l = list(tokens.keys())
-	l.sort()
-	for i in l:
-		j = tokens[i]
-		if j is None:
-			j = i
-		if i[0] == "'":
-			j = i
-		fo.write("\t[%s] = \"%s\",\n" % (i, j))
-	fo.write("};\n")
+    "Emit the vcl_tnames (token->string) conversion array"
+    fo.write("\nconst char * const vcl_tnames[256] = {\n")
+    l = list(tokens.keys())
+    l.sort()
+    for i in l:
+        j = tokens[i]
+        if j is None:
+            j = i
+        if i[0] == "'":
+            j = i
+        fo.write("\t[%s] = \"%s\",\n" % (i, j))
+    fo.write("};\n")
 
 
 #######################################################################
 def emit_file(fo, fd, bn):
-	"Read a C-source file and spit out code that outputs it with VSB_cat()"
-	fn = join(fd, bn)
-
-	fi = open(fn)
-	fc = fi.read()
-	fi.close()
-
-	w = 66		# Width of lines, after white space prefix
-	maxlen = 10240  # Max length of string literal
-
-	x = 0
-	l = 0
-	fo.write("\n\t/* %s */\n\n" % fn)
-	fo.write('\tVSB_cat(sb, "/* ---===### %s ###===--- */\\n\\n");\n' % bn)
-	for c in fc:
-		if l == 0:
-			fo.write("\tVSB_cat(sb, \"")
-			l += 12
-			x += 12
-		if x == 0:
-			fo.write("\t    \"")
-		d = c
-		if c == '\n':
-			d = "\\n"
-		elif c == '\t':
-			d = "\\t"
-		elif c == '"':
-			d = "\\\""
-		elif c == '\\':
-			d = "\\\\"
-
-		if c == '\n' and x > w - 20:
-			fo.write(d + "\"\n")
-			x = 0
-			continue
-		if c.isspace() and x > w - 10:
-			fo.write(d + "\"\n")
-			x = 0
-			continue
-
-		fo.write(d)
-		x += len(d)
-		l += len(d)
-		if l > maxlen:
-			fo.write("\");\n")
-			l = 0
-			x = 0
-		if x > w - 3:
-			fo.write("\"\n")
-			x = 0
-	if x != 0:
-		fo.write("\"\n")
-	if l != 0:
-		fo.write("\t);\n")
-	fo.write('\tVSB_cat(sb, "\\n");\n')
+    "Read a C-source file and spit out code that outputs it with VSB_cat()"
+    fn = join(fd, bn)
+
+    fi = open(fn)
+    fc = fi.read()
+    fi.close()
+
+    w = 66      # Width of lines, after white space prefix
+    maxlen = 10240  # Max length of string literal
+
+    x = 0
+    l = 0
+    fo.write("\n\t/* %s */\n\n" % fn)
+    fo.write('\tVSB_cat(sb, "/* ---===### %s ###===--- */\\n\\n");\n' % bn)
+    for c in fc:
+        if l == 0:
+            fo.write("\tVSB_cat(sb, \"")
+            l += 12
+            x += 12
+        if x == 0:
+            fo.write("\t    \"")
+        d = c
+        if c == '\n':
+            d = "\\n"
+        elif c == '\t':
+            d = "\\t"
+        elif c == '"':
+            d = "\\\""
+        elif c == '\\':
+            d = "\\\\"
+
+        if c == '\n' and x > w - 20:
+            fo.write(d + "\"\n")
+            x = 0
+            continue
+        if c.isspace() and x > w - 10:
+            fo.write(d + "\"\n")
+            x = 0
+            continue
+
+        fo.write(d)
+        x += len(d)
+        l += len(d)
+        if l > maxlen:
+            fo.write("\");\n")
+            l = 0
+            x = 0
+        if x > w - 3:
+            fo.write("\"\n")
+            x = 0
+    if x != 0:
+        fo.write("\"\n")
+    if l != 0:
+        fo.write("\t);\n")
+    fo.write('\tVSB_cat(sb, "\\n");\n')
 
 #######################################################################
 
 
 def polish_tokens(tokens):
-	"Expand single char tokens"
-	st = tokens[None]
-	del tokens[None]
-	for i in st:
-		tokens["'" + i + "'"] = i
+    "Expand single char tokens"
+    st = tokens[None]
+    del tokens[None]
+    for i in st:
+        tokens["'" + i + "'"] = i
 
 
 #######################################################################
 def file_header(fo):
-	fo.write("""/*
+    fo.write("""/*
  * NB:  This file is machine generated, DO NOT EDIT!
  *
  * Edit and run lib/libvcc/generate.py instead.
@@ -516,10 +510,10 @@ def file_header(fo):
 """)
 
 def lint_start(fo):
-	fo.write('/*lint -save -e525 -e539 */\n\n')
+    fo.write('/*lint -save -e525 -e539 */\n\n')
 
 def lint_end(fo):
-	fo.write('\n/*lint -restore */\n')
+    fo.write('\n/*lint -restore */\n')
 
 #######################################################################
 
@@ -531,11 +525,11 @@ file_header(fo)
 
 j = 128
 for i in sorted(tokens.keys()):
-	if i[0] == "'":
-		continue
-	fo.write("#define\t%s %d\n" % (i, j))
-	j += 1
-	assert j < 256
+    if i[0] == "'":
+        continue
+    fo.write("#define\t%s %d\n" % (i, j))
+    j += 1
+    assert j < 256
 
 fo.close()
 
@@ -546,14 +540,14 @@ vcls = list()
 vcls_client = list()
 vcls_backend = list()
 for i in returns:
-	vcls.append(i[0])
-	for j in i[1]:
-		if j == "B":
-			vcls_backend.append(i[0])
-		elif j == "C":
-			vcls_client.append(i[0])
-	for j in i[2]:
-		rets[j] = True
+    vcls.append(i[0])
+    for j in i[1]:
+        if j == "B":
+            vcls_backend.append(i[0])
+        elif j == "C":
+            vcls_client.append(i[0])
+    for j in i[2]:
+        rets[j] = True
 
 #######################################################################
 
@@ -566,25 +560,25 @@ lint_start(fo)
 fo.write("#ifdef VCL_RET_MAC\n")
 ll = sorted(returns)
 for i in sorted(rets.keys()):
-	fo.write("VCL_RET_MAC(%s, %s" % (i.lower(), i.upper()))
-	s = ",\n\t"
-	for j in ll:
-		if i in j[2]:
-			fo.write("%sVCL_MET_%s" % (s, j[0].upper()))
-			s = " |\n\t"
-	fo.write("\n)\n\n")
+    fo.write("VCL_RET_MAC(%s, %s" % (i.lower(), i.upper()))
+    s = ",\n\t"
+    for j in ll:
+        if i in j[2]:
+            fo.write("%sVCL_MET_%s" % (s, j[0].upper()))
+            s = " |\n\t"
+    fo.write("\n)\n\n")
 fo.write("#undef VCL_RET_MAC\n")
 fo.write("#endif\n")
 
 fo.write("\n#ifdef VCL_MET_MAC\n")
 for i in ll:
-	fo.write("VCL_MET_MAC(%s, %s, %s," %
-	    (i[0].lower(), i[0].upper(), i[1]))
-	p = " (\n\t"
-	for j in sorted(i[2]):
-		fo.write("%s(1U << VCL_RET_%s)" % (p, j.upper()))
-		p = " |\n\t"
-	fo.write(")\n)\n\n")
+    fo.write("VCL_MET_MAC(%s, %s, %s," %
+             (i[0].lower(), i[0].upper(), i[1]))
+    p = " (\n\t"
+    for j in sorted(i[2]):
+        fo.write("%s(1U << VCL_RET_%s)" % (p, j.upper()))
+        p = " |\n\t"
+    fo.write(")\n)\n\n")
 fo.write("#undef VCL_MET_MAC\n")
 fo.write("#endif\n")
 lint_end(fo)
@@ -609,40 +603,40 @@ fo.write("""
 
 
 def tbl40(a, b):
-	while len(a.expandtabs()) < 40:
-		a += "\t"
-	return a + b
+    while len(a.expandtabs()) < 40:
+        a += "\t"
+    return a + b
 
 fo.write("\n/* VCL Methods */\n")
 task = {}
 n = 1
 for i in returns:
-	fo.write(tbl40("#define VCL_MET_%s" % i[0].upper(), "(1U << %d)\n" % n))
-	if not i[1] in task:
-		task[i[1]] = []
-	task[i[1]].append("VCL_MET_" + i[0].upper())
-	n += 1
+    fo.write(tbl40("#define VCL_MET_%s" % i[0].upper(), "(1U << %d)\n" % n))
+    if not i[1] in task:
+        task[i[1]] = []
+    task[i[1]].append("VCL_MET_" + i[0].upper())
+    n += 1
 
 fo.write("\n" + tbl40("#define VCL_MET_MAX", "%d\n" % n))
 fo.write("\n" + tbl40("#define VCL_MET_MASK", "0x%x\n" % ((1 << n) - 1)))
 
 fo.write("\n")
 for i in sorted(task.keys()):
-	fo.write(tbl40("#define VCL_MET_TASK_%s" % i.upper(),
-		       "( " + (" | \\\n\t\t\t\t\t  ").join(task[i]) + " )\n"))
+    fo.write(tbl40("#define VCL_MET_TASK_%s" % i.upper(),
+                   "( " + (" | \\\n\t\t\t\t\t  ").join(task[i]) + " )\n"))
 
 
 fo.write("\n/* VCL Returns */\n")
 n = 1
 for i in sorted(rets.keys()):
-	fo.write(tbl40("#define VCL_RET_%s" % i.upper(), "%d\n" % n))
-	n += 1
+    fo.write(tbl40("#define VCL_RET_%s" % i.upper(), "%d\n" % n))
+    n += 1
 
 fo.write("\n" + tbl40("#define VCL_RET_MAX", "%d\n" % n))
 
 fo.write("\n/* VCL Types */\n")
 for vcltype in sorted(vcltypes.keys()):
-	fo.write("extern const struct vrt_type VCL_TYPE_%s[1];\n" % vcltype)
+    fo.write("extern const struct vrt_type VCL_TYPE_%s[1];\n" % vcltype)
 
 
 fo.write("""
@@ -653,26 +647,26 @@ typedef void vcl_fini_f(VRT_CTX);
 typedef void vcl_func_f(VRT_CTX);
 
 struct VCL_conf {
-	unsigned			magic;
-#define VCL_CONF_MAGIC			0x7406c509	/* from /dev/random */
+    unsigned            magic;
+#define VCL_CONF_MAGIC          0x7406c509      /* from /dev/random */
 
-	unsigned			syntax;
-	VCL_BACKEND			*default_director;
-	const struct vrt_backend_probe	*default_probe;
-	unsigned			nref;
-	const struct vrt_ref		*ref;
+    unsigned            syntax;
+    VCL_BACKEND             *default_director;
+    const struct vrt_backend_probe  *default_probe;
+    unsigned            nref;
+    const struct vrt_ref        *ref;
 
-	unsigned			nsrc;
-	const char			**srcname;
-	const char			**srcbody;
+    unsigned            nsrc;
+    const char              **srcname;
+    const char              **srcbody;
 
-	unsigned			nvmod;
+    unsigned            nvmod;
 
-	vcl_event_f			*event_vcl;
+    vcl_event_f             *event_vcl;
 """)
 
 for i in returns:
-	fo.write("\tvcl_func_f\t\t\t*" + i[0] + "_func;\n")
+    fo.write("\tvcl_func_f\t\t\t*" + i[0] + "_func;\n")
 
 fo.write("\n};\n")
 fo.close()
@@ -681,44 +675,44 @@ fo.close()
 
 
 def restrict(fo, spec):
-	d = dict()
-	for j in spec:
-		if j[:4] == "vcl_":
-			j = j[4:]
-		if j == 'all':
-			for i in vcls:
-				d[i] = True
-		elif j == 'backend':
-			for i in vcls_backend:
-				d[i] = True
-		elif j == 'client':
-			for i in vcls_client:
-				d[i] = True
-		elif j == 'both':
-			for i in vcls_client:
-				d[i] = True
-			for i in vcls_backend:
-				d[i] = True
-		else:
-			if not j in vcls:
-				print("JJ", j)
-			assert j in vcls
-			d[j] = True
-	p = ""
-	l = list(d.keys())
-	l.sort()
-	w = 0
-	fo.write("\t\t")
-	for j in l:
-		x = p + "VCL_MET_" + j.upper()
-		if w + len(x) > 60:
-			fo.write("\n\t\t")
-			w = 0
-		fo.write(x)
-		w += len(x)
-		p = " | "
-	if len(d) == 0:
-		fo.write("0")
+    d = dict()
+    for j in spec:
+        if j[:4] == "vcl_":
+            j = j[4:]
+        if j == 'all':
+            for i in vcls:
+                d[i] = True
+        elif j == 'backend':
+            for i in vcls_backend:
+                d[i] = True
+        elif j == 'client':
+            for i in vcls_client:
+                d[i] = True
+        elif j == 'both':
+            for i in vcls_client:
+                d[i] = True
+            for i in vcls_backend:
+                d[i] = True
+        else:
+            if not j in vcls:
+                print("JJ", j)
+            assert j in vcls
+            d[j] = True
+    p = ""
+    l = list(d.keys())
+    l.sort()
+    w = 0
+    fo.write("\t\t")
+    for j in l:
+        x = p + "VCL_MET_" + j.upper()
+        if w + len(x) > 60:
+            fo.write("\n\t\t")
+            w = 0
+        fo.write(x)
+        w += len(x)
+        p = " | "
+    if not d:
+        fo.write("0")
 
 #######################################################################
 
@@ -736,7 +730,7 @@ fo.write("""
 void
 vcc_Var_Init(struct vcc *tl)
 {
-	struct symbol *sym;
+    struct symbol *sym;
 """)
 
 
@@ -744,22 +738,22 @@ parse_var_doc(join(srcroot, "doc/sphinx/reference/vcl_var.rst"))
 fo.write("}\n")
 
 for i in stv_variables:
-	fh.write(vcltypes[i[1]].c + " VRT_Stv_" + i[0] + "(const char *);\n")
+    fh.write(vcltypes[i[1]].c + " VRT_Stv_" + i[0] + "(const char *);\n")
 
 fo.write("\n/* VCL type identifiers */\n")
 
 for vn in sorted(vcltypes.keys()):
-	v = vcltypes[vn]
-	if v.internal:
-		continue
-	fo.write("const struct vrt_type VCL_TYPE_%s[1] = { {\n" % v.name)
-	fo.write("\t.magic = VRT_TYPE_MAGIC,\n")
-	fo.write('\t.lname = "%s",\n' % v.name.lower())
-	fo.write('\t.uname = "%s",\n' % v.name)
-	fo.write('\t.ctype = "%s",\n' % v.c)
-	if v.c != "void":
-		fo.write('\t.szof = sizeof(VCL_%s),\n' % v.name)
-	fo.write("}};\n")
+    v = vcltypes[vn]
+    if v.internal:
+        continue
+    fo.write("const struct vrt_type VCL_TYPE_%s[1] = { {\n" % v.name)
+    fo.write("\t.magic = VRT_TYPE_MAGIC,\n")
+    fo.write('\t.lname = "%s",\n' % v.name.lower())
+    fo.write('\t.uname = "%s",\n' % v.name)
+    fo.write('\t.ctype = "%s",\n' % v.c)
+    if v.c != "void":
+        fo.write('\t.szof = sizeof(VCL_%s),\n' % v.name)
+    fo.write("}};\n")
 
 fo.close()
 fh.close()
@@ -800,7 +794,7 @@ file_header(ft)
 lint_start(ft)
 
 for vcltype in sorted(vcltypes.keys()):
-	ft.write("VCC_TYPE(" + vcltype + ")\n")
+    ft.write("VCC_TYPE(" + vcltype + ")\n")
 ft.write("#undef VCC_TYPE\n")
 lint_end(ft)
 ft.close()
@@ -813,10 +807,10 @@ file_header(fo)
 lint_start(fo)
 
 for i in stv_variables:
-	ct = vcltypes[i[1]]
-	fo.write("VRTSTVVAR(" + i[0] + ",\t" + i[1] + ",\t")
-	fo.write(ct.c + ",\t" + i[2] + ")")
-	fo.write("\n")
+    ct = vcltypes[i[1]]
+    fo.write("VRTSTVVAR(" + i[0] + ",\t" + i[1] + ",\t")
+    fo.write(ct.c + ",\t" + i[2] + ")")
+    fo.write("\n")
 
 fo.write("#undef VRTSTVVAR\n")
 lint_end(fo)
diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py
index 25b8cc621..dd157449f 100755
--- a/lib/libvcc/vmodtool.py
+++ b/lib/libvcc/vmodtool.py
@@ -41,7 +41,6 @@ import sys
 import re
 import optparse
 import unittest
-import random
 import copy
 import json
 import hashlib
@@ -254,7 +253,7 @@ class CType(object):
     def json(self, jl):
         jl.append([self.vt])
         while jl[-1][-1] is None:
-                jl[-1].pop(-1)
+            jl[-1].pop(-1)
 
 #######################################################################
 
@@ -290,7 +289,7 @@ class arg(CType):
     def json(self, jl):
         jl.append([self.vt, self.nm, self.defval, self.spec])
         if self.opt:
-                jl[-1].append(True)
+            jl[-1].append(True)
         while jl[-1][-1] is None:
             jl[-1].pop(-1)
 
@@ -366,7 +365,7 @@ class ProtoType(object):
             err("%s(): Illegal C-name\n" % self.cname(), warn=False)
 
         if len(wl) == 2 and wl[0] == '(' and wl[1] == ')':
-                return
+            return
 
         if wl[0] != "(":
             err("Syntax error: Expected '(', got '%s'" % wl[0], warn=False)
@@ -386,17 +385,17 @@ class ProtoType(object):
             if not wl:
                 break
             if wl[0] == '[':
-                    wl.pop(0)
-                    t = arg(wl, names, st.vcc.enums, ']')
-                    if t.nm is None:
-                        err("Optional arguments must have names", warn=False)
-                    t.opt = True
-                    x = wl.pop(0)
-                    if x != ']':
-                        err("Expected ']' found '%s'" % x, warn=False)
-                    self.argstruct = True
+                wl.pop(0)
+                t = arg(wl, names, st.vcc.enums, ']')
+                if t.nm is None:
+                    err("Optional arguments must have names", warn=False)
+                t.opt = True
+                x = wl.pop(0)
+                if x != ']':
+                    err("Expected ']' found '%s'" % x, warn=False)
+                self.argstruct = True
             else:
-                    t = arg(wl, names, st.vcc.enums, ',')
+                t = arg(wl, names, st.vcc.enums, ',')
             if t.nm is None:
                 t.nm2 = "arg%d" % n
             else:
@@ -466,10 +465,10 @@ class ProtoType(object):
         s = self.retval.ct + " " + name + '('
         ll = args
         if self.argstruct:
-                ll.append(self.argstructname() + "*")
+            ll.append(self.argstructname() + "*")
         else:
-                for i in self.args:
-                    ll.append(i.ct)
+            for i in self.args:
+                ll.append(i.ct)
         s += ", ".join(ll)
         return s + ');'
 
@@ -483,31 +482,31 @@ class ProtoType(object):
     def argstructure(self):
         s = "\n" + self.argstructname() + " {\n"
         for i in self.args:
-                if i.opt:
-                        assert i.nm is not None
-                        s += "\tchar\t\t\tvalid_%s;\n" % i.nm
+            if i.opt:
+                assert i.nm is not None
+                s += "\tchar\t\t\tvalid_%s;\n" % i.nm
         for i in self.args:
-                s += "\t" + i.ct
-                if len(i.ct) < 8:
-                        s += "\t"
-                if len(i.ct) < 16:
-                        s += "\t"
-                s += "\t" + i.nm2 + ";\n"
+            s += "\t" + i.ct
+            if len(i.ct) < 8:
+                s += "\t"
+            if len(i.ct) < 16:
+                s += "\t"
+            s += "\t" + i.nm2 + ";\n"
         s += "};\n"
         return s
 
     def cstuff(self, args, where):
         s = ""
         if where == 'h':
-                if self.argstruct:
-                        s += self.argstructure()
-                s += lwrap(self.proto(args, self.cname(True)))
+            if self.argstruct:
+                s += self.argstructure()
+            s += lwrap(self.proto(args, self.cname(True)))
         elif where == 'c':
-                s += lwrap(self.typedef(args))
+            s += lwrap(self.typedef(args))
         elif where == 'o':
-                if self.argstruct:
-                        s += self.argstructure()
-                s += lwrap(self.typedef(args))
+            if self.argstruct:
+                s += self.argstructure()
+            s += lwrap(self.typedef(args))
         else:
             assert False
         return s
@@ -517,9 +516,9 @@ class ProtoType(object):
         self.retval.json(ll)
         ll.append('Vmod_%s_Func.%s' % (self.st.vcc.modname, cfunc))
         if self.argstruct:
-                ll.append(self.argstructname())
+            ll.append(self.argstructname())
         else:
-                ll.append("")
+            ll.append("")
         for i in self.args:
             i.json(ll)
         jl.append(ll)
diff --git a/lib/libvcc/vsctool.py b/lib/libvcc/vsctool.py
index a665dff20..6c96388e5 100644
--- a/lib/libvcc/vsctool.py
+++ b/lib/libvcc/vsctool.py
@@ -27,9 +27,13 @@
 # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
 # SUCH DAMAGE.
 
-"""
-This program compiles a .vsc file to C language constructs.
-"""
+'''
+This program compiles a `.vsc` file to `.c`, `.h` and `.rst` formats.
+
+Note: A `.vsc` file is *almost* a `.rst` file, or at last *almost*
+the same general syntax as a `.rst` file, but for now we process
+it with this program to get a *real* `.rst` file.
+'''
 
 from __future__ import print_function
 
@@ -37,385 +41,386 @@ import getopt
 import json
 import sys
 import collections
-import struct
 
-TYPES = [ "counter", "gauge", "bitmap" ]
-CTYPES = [ "uint64_t" ]
-LEVELS = [ "info", "diag", "debug" ]
-FORMATS = [ "integer", "bytes", "bitmap", "duration" ]
+# Parameters of 'varnish_vsc_begin', first element is default
+TYPES = ["counter", "gauge", "bitmap"]
+CTYPES = ["uint64_t"]
+LEVELS = ["info", "diag", "debug"]
+FORMATS = ["integer", "bytes", "bitmap", "duration"]
 
 PARAMS = {
-	"type":		["counter", TYPES],
-	"ctype":	["uint64_t", CTYPES],
-	"level":	["info", LEVELS],
-	"oneliner":	True,
-	"format":	[ "integer", FORMATS],
+    "type": TYPES,
+    "ctype": CTYPES,
+    "level": LEVELS,
+    "oneliner": None,
+    "format": FORMATS,
 }
 
-# http://python3porting.com/problems.html#bytes-strings-and-unicode
-if sys.version_info < (3,):
-	def b(x):
-		return x
-else:
-	import codecs
-	def b(x):
-		return codecs.latin_1_encode(x)[0]
-
 def genhdr(fo, name):
-	fo.write('/*\n')
-	fo.write(' * NB:  This file is machine generated, DO NOT EDIT!\n')
-	fo.write(' *\n')
-	fo.write(' * Edit ' + name +
-	    '.vsc and run lib/libvcc/vsctool.py instead.\n')
-	fo.write(' */\n')
-	fo.write('\n')
 
-#######################################################################
+    '''Emit .[ch] file boiler-plate warning'''
 
-class vscset(object):
-	def __init__(self, name, m):
-		self.name = name
-		self.struct = "struct VSC_" + name
-		self.mbrs = []
-		self.head = m
-		self.completed = False
-		self.off = 0
-
-	def addmbr(self, m):
-		assert not self.completed
-		self.mbrs.append(m)
-		m.param["index"] = self.off
-		self.off += 8
-
-	def complete(self):
-		self.completed = True
-
-	def emit_json(self, fo):
-		dd = collections.OrderedDict()
-		dd["version"] = "1"
-		dd["name"] = self.name
-		dd["oneliner"] = self.head.param["oneliner"].strip()
-		dd["order"] = int(self.head.param["order"])
-		dd["docs"] = "\n".join(self.head.getdoc())
-		dd["elements"] = len(self.mbrs)
-		el = collections.OrderedDict()
-		dd["elem"] = el
-		for i in self.mbrs:
-			ed = collections.OrderedDict()
-			el[i.arg] = ed
-			for j in PARAMS:
-				if j in i.param:
-					ed[j] = i.param[j]
-			ed["index"] = i.param["index"]
-			ed["name"] = i.arg
-			ed["docs"] = "\n".join(i.getdoc())
-		s=json.dumps(dd, separators=(",",":")) + "\0"
-		fo.write("\nstatic const unsigned char")
-		fo.write(" vsc_%s_json[%d] = {\n" % (self.name, len(s)))
-		bz = bytearray(s, encoding="ascii")
-		t = "\t"
-		for i in bz:
-			t += "%d," % i
-			if len(t) >= 69:
-				fo.write(t + "\n")
-				t = "\t"
-		if len(t) > 1:
-			fo.write(t[:-1])
-		fo.write("\n};\n")
-		s = json.dumps(dd, indent=2, separators=(',', ': '))
-		fo.write("\n")
-		for i in s.split("\n"):
-			j = "// " + i
-			if len(j) > 72:
-				fo.write(j[:72] + "[...]\n")
-			else:
-				fo.write(j + "\n")
-		fo.write("\n")
-
-
-	def emit_h(self):
-		fon="VSC_" + self.name + ".h"
-		fo = open(fon, "w")
-		genhdr(fo, self.name)
-		fo.write(self.struct + " {\n")
-		for i in self.mbrs:
-			fo.write("\tuint64_t\t%s;\n" % i.arg)
-		fo.write("};\n")
-		fo.write("\n")
-
-		fo.write("#define VSC_" + self.name +
-		    "_size PRNDUP(sizeof(" + self.struct + "))\n\n")
-
-		fo.write(self.struct + " *VSC_" + self.name + "_New")
-		fo.write("(struct vsmw_cluster *,\n")
-		fo.write("    struct vsc_seg **, const char *fmt, ...);\n")
-
-		fo.write("void VSC_" + self.name + "_Destroy")
-		fo.write("(struct vsc_seg **);\n")
-
-		if 'sumfunction' in self.head.param:
-			fo.write("void VSC_" + self.name + "_Summ")
-			fo.write("(" + self.struct + " *, ")
-			fo.write("const " + self.struct + " *);\n")
-
-	def emit_c(self):
-		fon="VSC_" + self.name + ".c"
-		fo = open(fon, "w")
-		genhdr(fo, self.name)
-		fo.write('#include "config.h"\n')
-		fo.write('#include <stdio.h>\n')
-		fo.write('#include <stdarg.h>\n')
-		fo.write('#include "vdef.h"\n')
-		fo.write('#include "vas.h"\n')
-		fo.write('#include "vrt.h"\n')
-		fo.write('#include "VSC_%s.h"\n' % self.name)
-
-		fo.write("\n")
-		fo.write('static const char vsc_%s_name[] = "%s";\n' %
-		    (self.name, self.name.upper()))
-
-		fo.write("\n")
-		fo.write("#define PARANOIA(a,n)\t\t\t\t\\\n")
-		fo.write("    _Static_assert(\t\t\t\t\\\n")
-		fo.write("\toffsetof(" + self.struct + ", a) == n,\t\\\n")
-		fo.write("\t\"VSC element '\" #a \"' at wrong offset\")\n\n")
-
-		for i in self.mbrs:
-			fo.write("PARANOIA(" + i.arg)
-			fo.write(", %d);\n" % (i.param["index"]))
-
-		fo.write("#undef PARANOIA\n")
-
-		self.emit_json(fo)
-
-		fo.write("\n")
-		fo.write(self.struct + "*\n")
-		fo.write("VSC_" + self.name + "_New")
-		fo.write("(struct vsmw_cluster *vc,\n")
-		fo.write("    struct vsc_seg **sg, const char *fmt, ...)\n")
-		fo.write("{\n")
-		fo.write("\tva_list ap;\n")
-		fo.write("\t" + self.struct + " *retval;\n")
-		fo.write("\n")
-		fo.write("\tva_start(ap, fmt);\n")
-		fo.write("\tretval = VRT_VSC_Alloc")
-		fo.write("(vc, sg, vsc_" + self.name + "_name, ")
-		fo.write("VSC_" + self.name + "_size,\n")
-		fo.write("\t    vsc_" + self.name + "_json, ")
-		fo.write("sizeof vsc_" + self.name + "_json, fmt, ap);\n")
-		fo.write("\tva_end(ap);\n")
-		fo.write("\treturn(retval);\n")
-		fo.write("}\n")
-
-		fo.write("\n")
-		fo.write("void\n")
-		fo.write("VSC_" + self.name + "_Destroy")
-		fo.write("(struct vsc_seg **sg)\n")
-		fo.write("{\n")
-		fo.write("\tstruct vsc_seg *p;\n")
-		fo.write("\n")
-		fo.write("\tAN(sg);\n")
-		fo.write("\tp = *sg;\n")
-		fo.write("\t*sg = NULL;\n")
-		fo.write('\tVRT_VSC_Destroy(vsc_%s_name, p);\n' % self.name)
-		fo.write("}\n")
-
-		if 'sumfunction' in self.head.param:
-			fo.write("\n")
-			fo.write("void\n")
-			fo.write("VSC_" + self.name + "_Summ")
-			fo.write("(" + self.struct + " *dst, ")
-			fo.write("const " + self.struct + " *src)\n")
-			fo.write("{\n")
-			fo.write("\n")
-			fo.write("\tAN(dst);\n")
-			fo.write("\tAN(src);\n")
-			for i in self.mbrs:
-				s1 = "\tdst->" + i.arg + " +="
-				s2 = "src->" + i.arg + ";"
-				if len((s1 + " " + s2).expandtabs()) < 79:
-					fo.write(s1 + " " + s2 + "\n")
-				else:
-					fo.write(s1 + "\n\t    " + s2 + "\n")
-			fo.write("}\n")
+    fo.write('/*\n')
+    fo.write(' * NB:  This file is machine generated, DO NOT EDIT!\n')
+    fo.write(' *\n')
+    fo.write(' * Edit ' + name +
+             '.vsc and run lib/libvcc/vsctool.py instead.\n')
+    fo.write(' */\n')
+    fo.write('\n')
 
 #######################################################################
 
-class directive(object):
-	def __init__(self, s):
-		ll = s.split("\n")
-		i = ll.pop(0).split("::", 2)
-		self.cmd = i[0]
-		self.arg = i[1].strip()
-		assert len(self.arg.split()) == 1
-
-		self.param = {}
-		while len(ll):
-			j = ll[0].split(":",2)
-			if len(j) != 3 or not j[0].isspace():
-				break
-			self.param[j[1]] = j[2].strip()
-			ll.pop(0)
-		self.ldoc = ll
-
-	def getdoc(self):
-		while len(self.ldoc) and self.ldoc[0].strip() == "":
-			self.ldoc.pop(0)
-		while len(self.ldoc) and self.ldoc[-1].strip() == "":
-			self.ldoc.pop(-1)
-		return self.ldoc
-
-	def moredoc(self, s):
-		self.getdoc()
-		self.ldoc += s.split("\n")
-
-	def emit_rst(self, fo):
-		fo.write("\n.. " + self.cmd + ":: " + self.arg + "\n")
-		self.emit_rst_doc(fo)
-
-	def emit_rst_doc(self, fo):
-		fo.write("\n".join(self.ldoc))
-
-	def emit_h(self, fo):
-		return
-
-class rst_vsc_begin(directive):
-	def __init__(self, s):
-		super(rst_vsc_begin, self).__init__(s)
-
-	def vscset(self, ss):
-		ss.append(vscset(self.arg, self))
-
-	def emit_rst(self, fo):
-		fo.write("\n..\n\t" + self.cmd + ":: " + self.arg + "\n")
-
-		s = self.arg.upper() + " ? " + self.param["oneliner"]
-		fo.write("\n")
-		fo.write(s + "\n")
-		fo.write("=" * len(s) + "\n")
-
-		self.emit_rst_doc(fo)
-
-class rst_vsc(directive):
-	def __init__(self, s):
-		super(rst_vsc, self).__init__(s)
-
-		for i,v in PARAMS.items():
-			if v is not True:
-				self.do_default(i, v[0], v[1])
-
-		for p in self.param:
-			if p in PARAMS:
-				continue
-			sys.stderr.write("Unknown parameter ")
-			sys.stderr.write("'" + p + "'")
-			sys.stderr.write(" on field '" + self.arg + "'\n")
-			exit(2)
-
-	def do_default(self, p, v, s):
-		if p not in self.param:
-			self.param[p] = v
-		if self.param[p] not in s:
-			sys.stderr.write("Wrong " + p + " '" + self.param[p])
-			sys.stderr.write("' on field '" + self.arg + "'\n")
-			exit(2)
-
-
-	def emit_rst(self, fo):
-		fo.write("\n``%s`` ? " % self.arg)
-		fo.write("`%s` - " % self.param["type"])
-		fo.write("%s\n\n" % self.param["level"])
-
-		fo.write("\t" + self.param["oneliner"] + "\n")
-		self.emit_rst_doc(fo)
-
-	def vscset(self, ss):
-		ss[-1].addmbr(self)
-
-
-class rst_vsc_end(directive):
-	def __init__(self, s):
-		super(rst_vsc_end, self).__init__(s)
-
-	def vscset(self, ss):
-		ss[-1].complete()
-
-	def emit_rst(self, fo):
-		fo.write("\n..\n\t" + self.cmd + ":: " + self.arg + "\n")
-		self.emit_rst_doc(fo)
-
-class other(object):
-	def __init__(self, s):
-		self.s = s
-
-	def emit_rst(self, fo):
-		fo.write(self.s)
-
-	def emit_h(self, fo):
-		return
-
-	def vscset(self, ss):
-		return
+class CounterSet(object):
+
+    '''
+        A set of counters
+
+        In the `.vsc` file a CounterSet is everything between a
+
+            .. varnish_vsc_begin::
+
+        and the subsequent
+
+            .. varnish_vsc_end::
+    '''
+
+    def __init__(self, name, m):
+        self.name = name
+        self.struct = "struct VSC_" + name
+        self.mbrs = []
+        self.head = m
+        self.completed = False
+        self.off = 0
+
+    def addmbr(self, m):
+        '''Add a counter'''
+        assert not self.completed
+        self.mbrs.append(m)
+        retval = self.off
+        self.off += 8
+        return retval
+
+    def complete(self, arg):
+        '''Mark set completed'''
+        assert arg == self.name
+        self.completed = True
+
+    def emit_json(self, fo):
+        '''Emit JSON as compact C byte-array and as readable C-comments'''
+        assert self.completed
+        dd = collections.OrderedDict()
+        dd["version"] = "1"
+        dd["name"] = self.name
+        dd["oneliner"] = self.head.param["oneliner"].strip()
+        dd["order"] = int(self.head.param["order"])
+        dd["docs"] = "\n".join(self.head.getdoc())
+        dd["elements"] = len(self.mbrs)
+        el = collections.OrderedDict()
+        dd["elem"] = el
+        for i in self.mbrs:
+            ed = collections.OrderedDict()
+            el[i.arg] = ed
+            for j in PARAMS:
+                if j in i.param:
+                    ed[j] = i.param[j]
+            ed["index"] = i.param["index"]
+            ed["name"] = i.arg
+            ed["docs"] = "\n".join(i.getdoc())
+        s = json.dumps(dd, separators=(",", ":")) + "\0"
+        fo.write("\nstatic const unsigned char")
+        fo.write(" vsc_%s_json[%d] = {\n" % (self.name, len(s)))
+        bz = bytearray(s, encoding="ascii")
+        t = "\t"
+        for i in bz:
+            t += "%d," % i
+            if len(t) >= 69:
+                fo.write(t + "\n")
+                t = "\t"
+        if len(t) > 1:
+            fo.write(t[:-1])
+        fo.write("\n};\n")
+        s = json.dumps(dd, indent=2, separators=(',', ': '))
+        fo.write("\n")
+        for i in s.split("\n"):
+            j = "// " + i
+            if len(j) > 72:
+                fo.write(j[:72] + "[...]\n")
+            else:
+                fo.write(j + "\n")
+        fo.write("\n")
+
+    def emit_h(self):
+        '''Emit .h file'''
+        assert self.completed
+        fon = "VSC_" + self.name + ".h"
+        fo = open(fon, "w")
+        genhdr(fo, self.name)
+        fo.write(self.struct + " {\n")
+        for i in self.mbrs:
+            fo.write("\tuint64_t\t%s;\n" % i.arg)
+        fo.write("};\n")
+        fo.write("\n")
+
+        fo.write("#define VSC_" + self.name +
+                 "_size PRNDUP(sizeof(" + self.struct + "))\n\n")
+
+        fo.write(self.struct + " *VSC_" + self.name + "_New")
+        fo.write("(struct vsmw_cluster *,\n")
+        fo.write("    struct vsc_seg **, const char *fmt, ...);\n")
+
+        fo.write("void VSC_" + self.name + "_Destroy")
+        fo.write("(struct vsc_seg **);\n")
+
+        if 'sumfunction' in self.head.param:
+            fo.write("void VSC_" + self.name + "_Summ")
+            fo.write("(" + self.struct + " *, ")
+            fo.write("const " + self.struct + " *);\n")
+
+    def emit_c_paranoia(self, fo):
+        '''Emit asserts to make sure compiler gets same byte index'''
+        fo.write("#define PARANOIA(a,n)\t\t\t\t\\\n")
+        fo.write("    _Static_assert(\t\t\t\t\\\n")
+        fo.write("\toffsetof(" + self.struct + ", a) == n,\t\\\n")
+        fo.write("\t\"VSC element '\" #a \"' at wrong offset\")\n\n")
+
+        for i in self.mbrs:
+            fo.write("PARANOIA(" + i.arg)
+            fo.write(", %d);\n" % (i.param["index"]))
+
+        fo.write("#undef PARANOIA\n")
+
+    def emit_c_sumfunc(self, fo):
+        '''Emit a function summ up countersets'''
+        fo.write("\n")
+        fo.write("void\n")
+        fo.write("VSC_" + self.name + "_Summ")
+        fo.write("(" + self.struct + " *dst, ")
+        fo.write("const " + self.struct + " *src)\n")
+        fo.write("{\n")
+        fo.write("\n")
+        fo.write("\tAN(dst);\n")
+        fo.write("\tAN(src);\n")
+        for i in self.mbrs:
+            s1 = "\tdst->" + i.arg + " +="
+            s2 = "src->" + i.arg + ";"
+            if len((s1 + " " + s2).expandtabs()) < 79:
+                fo.write(s1 + " " + s2 + "\n")
+            else:
+                fo.write(s1 + "\n\t    " + s2 + "\n")
+        fo.write("}\n")
+
+    def emit_c_newfunc(self, fo):
+        '''Emit New function'''
+        fo.write("\n")
+        fo.write(self.struct + "*\n")
+        fo.write("VSC_" + self.name + "_New")
+        fo.write("(struct vsmw_cluster *vc,\n")
+        fo.write("    struct vsc_seg **sg, const char *fmt, ...)\n")
+        fo.write("{\n")
+        fo.write("\tva_list ap;\n")
+        fo.write("\t" + self.struct + " *retval;\n")
+        fo.write("\n")
+        fo.write("\tva_start(ap, fmt);\n")
+        fo.write("\tretval = VRT_VSC_Alloc")
+        fo.write("(vc, sg, vsc_" + self.name + "_name, ")
+        fo.write("VSC_" + self.name + "_size,\n")
+        fo.write("\t    vsc_" + self.name + "_json, ")
+        fo.write("sizeof vsc_" + self.name + "_json, fmt, ap);\n")
+        fo.write("\tva_end(ap);\n")
+        fo.write("\treturn(retval);\n")
+        fo.write("}\n")
+
+    def emit_c_destroyfunc(self, fo):
+        '''Emit Destroy function'''
+        fo.write("\n")
+        fo.write("void\n")
+        fo.write("VSC_" + self.name + "_Destroy")
+        fo.write("(struct vsc_seg **sg)\n")
+        fo.write("{\n")
+        fo.write("\tstruct vsc_seg *p;\n")
+        fo.write("\n")
+        fo.write("\tAN(sg);\n")
+        fo.write("\tp = *sg;\n")
+        fo.write("\t*sg = NULL;\n")
+        fo.write('\tVRT_VSC_Destroy(vsc_%s_name, p);\n' % self.name)
+        fo.write("}\n")
+
+    def emit_c(self):
+        '''Emit .c file'''
+        assert self.completed
+        fon = "VSC_" + self.name + ".c"
+        fo = open(fon, "w")
+        genhdr(fo, self.name)
+        fo.write('#include "config.h"\n')
+        fo.write('#include <stdio.h>\n')
+        fo.write('#include <stdarg.h>\n')
+        fo.write('#include "vdef.h"\n')
+        fo.write('#include "vas.h"\n')
+        fo.write('#include "vrt.h"\n')
+        fo.write('#include "VSC_%s.h"\n' % self.name)
+
+        fo.write("\n")
+        fo.write('static const char vsc_%s_name[] = "%s";\n' %
+                 (self.name, self.name.upper()))
+
+        fo.write("\n")
+
+        self.emit_c_paranoia(fo)
+        self.emit_json(fo)
+        self.emit_c_newfunc(fo)
+        self.emit_c_destroyfunc(fo)
+        if 'sumfunction' in self.head.param:
+            self.emit_c_sumfunc(fo)
 
 #######################################################################
 
-class vsc_file(object):
-	def __init__(self, fin):
-		self.c = []
-		scs = open(fin).read().split("\n.. ")
-		self.c.append(other(scs[0]))
-		ld = None
-		for i in scs[1:]:
-			j = i.split(None, 1)
-			f = {
-				"varnish_vsc_begin::":	rst_vsc_begin,
-				"varnish_vsc::":	rst_vsc,
-				"varnish_vsc_end::":	rst_vsc_end,
-			}.get(j[0])
-			if f is None:
-				s = "\n.. " + i
-				o = other(s)
-				if ld is not None:
-					ld.moredoc(s)
-			else:
-				o = f(i)
-				ld = o
-			self.c.append(o)
-
-		self.vscset = []
-		for i in self.c:
-			i.vscset(self.vscset)
-
-	def emit_h(self):
-		for i in self.vscset:
-			i.emit_h()
-
-	def emit_c(self):
-		for i in self.vscset:
-			i.emit_c()
-
-	def emit_rst(self):
-		fo = sys.stdout
-		for i in self.c:
-			i.emit_rst(fo)
+class OurDirective(object):
+
+    '''
+        One of our `.. blablabla::` directives in the source file
+    '''
+
+    def __init__(self, s):
+        ll = s.split("\n")
+        i = ll.pop(0).split("::", 2)
+        self.cmd = i[0]
+        self.arg = i[1].strip()
+        assert len(self.arg.split()) == 1
+
+        self.param = {}
+        while ll:
+            j = ll[0].split(":", 2)
+            if len(j) != 3 or not j[0].isspace():
+                break
+            self.param[j[1]] = j[2].strip()
+            ll.pop(0)
+        self.ldoc = ll
+
+    def getdoc(self):
+        '''
+        Get docs for JSON
+
+        Note that these docs end with the first '\n.. ' sequence
+        in the .vsc file, so that we can put a longer and more
+        complex description into the .RST docs than the "long"
+        explanation varnishstat(1) and similar programs provide.
+        '''
+        while self.ldoc and self.ldoc[0].strip() == "":
+            self.ldoc.pop(0)
+        while self.ldoc and self.ldoc[-1].strip() == "":
+            self.ldoc.pop(-1)
+        return self.ldoc
+
+    def emit_rst(self, fo):
+        '''Emit the documentation as .rst'''
+        assert False
+
+class RstVscDirectiveBegin(OurDirective):
+
+    '''
+        `varnish_vsc_begin` directive
+    '''
+
+    def __init__(self, s, vsc_set, fo):
+        super(RstVscDirectiveBegin, self).__init__(s)
+        vsc_set.append(CounterSet(self.arg, self))
+        if fo:
+            fo.write("\n..\n\t" + self.cmd + ":: " + self.arg + "\n")
+
+            s = self.arg.upper() + " ? " + self.param["oneliner"]
+            fo.write("\n")
+            fo.write(s + "\n")
+            fo.write("=" * len(s) + "\n")
+            fo.write("\n".join(self.ldoc))
+
+class RstVscDirective(OurDirective):
+
+    '''
+        `varnish_vsc` directive - one counter
+    '''
+
+    def __init__(self, s, vsc_set, fo):
+        assert not vsc_set or vsc_set[-1].complete
+        super(RstVscDirective, self).__init__(s)
+
+        for i, v in PARAMS.items():
+            if v is not None:
+                if i not in self.param:
+                    self.param[i] = v[0]
+                if self.param[i] not in v:
+                    sys.stderr.write("Wrong " + i + " '" + self.param[i])
+                    sys.stderr.write("' on field '" + self.arg + "'\n")
+                    exit(2)
+
+        for p in self.param:
+            if p in PARAMS:
+                continue
+            sys.stderr.write("Unknown parameter ")
+            sys.stderr.write("'" + p + "'")
+            sys.stderr.write(" on field '" + self.arg + "'\n")
+            exit(2)
+        self.param["index"] = vsc_set[-1].addmbr(self)
+        if fo:
+            fo.write("\n``%s`` ? " % self.arg)
+            fo.write("`%s` - " % self.param["type"])
+            fo.write("%s\n\n" % self.param["level"])
+
+            fo.write("\t" + self.param["oneliner"] + "\n")
+            fo.write("\n".join(self.ldoc))
+
+class RstVscDirectiveEnd(OurDirective):
+
+    '''
+        `varnish_vsc_end` directive
+    '''
+
+    def __init__(self, s, vsc_set, fo):
+        super(RstVscDirectiveEnd, self).__init__(s)
+        vsc_set[-1].complete(self.arg)
+        if fo:
+            fo.write("\n..\n\t" + self.cmd + ":: " + self.arg + "\n")
+            fo.write("\n".join(self.ldoc))
 
 #######################################################################
 
-if __name__ == "__main__":
-
-	optlist, args = getopt.getopt(sys.argv[1:], "chr")
+def mainfunc(argv):
+
+    '''Process a .vsc file'''
+
+    optlist, args = getopt.getopt(argv[1:], "chr")
+
+    if len(args) != 1:
+        sys.stderr.write("Need exactly one filename argument\n")
+        exit(2)
+
+    rstfile = None
+    for f, v in optlist:
+        if f == '-r':
+            rstfile = sys.stdout
+
+    vscset = []
+    scs = open(args[0]).read().split("\n.. ")
+    if rstfile:
+        rstfile.write(scs[0])
+    for i in scs[1:]:
+        j = i.split(None, 1)
+        f = {
+            "varnish_vsc_begin::":  RstVscDirectiveBegin,
+            "varnish_vsc::":        RstVscDirective,
+            "varnish_vsc_end::":    RstVscDirectiveEnd,
+        }.get(j[0])
+        if f is not None:
+            f(i, vscset, rstfile)
+        elif rstfile:
+            rstfile.write("\n.. " + i)
+
+    for i in vscset:
+        for f, v in optlist:
+            if f == '-h':
+                i.emit_h()
+            if f == '-c':
+                i.emit_c()
 
-	fo = sys.stdout
-
-	if len(args) != 1:
-		sys.stderr.write("Need exactly one filename argument\n")
-		exit(2)
+if __name__ == "__main__":
 
-	vf = vsc_file(args[0])
-	for f,v in optlist:
-		if f == '-r':
-			vf.emit_rst()
-		if f == '-h':
-			vf.emit_h()
-		if f == '-c':
-			vf.emit_c()
+    mainfunc(sys.argv)
diff --git a/tools/gcov_digest.py b/tools/gcov_digest.py
index 138f7e8cf..188fb499d 100644
--- a/tools/gcov_digest.py
+++ b/tools/gcov_digest.py
@@ -32,19 +32,19 @@ found in a subdirectory tree.
 
 Options:
 
-	-g gcov-program
-		default:gcov6
+    -g gcov-program
+        default:gcov6
 
-	-o output-filename
-		default: stdout
+    -o output-filename
+        default: stdout
 
-	-x exclude-subdir
-		default ".git" and ".deps"
+    -x exclude-subdir
+        default ".git" and ".deps"
 
  Arguments:
 
-	directories to process.
-	default: .
+    directories to process.
+    default: .
 
 """
 
@@ -61,141 +61,141 @@ lengths = {}
 exclude = [".git", ".deps",]
 
 def process_gcov(fn, sn):
-	""" Sum .gcov file into counts, then delete it """
-	dd = counts.get(sn)
-	if dd is None:
-		dd = {}
-	for ln in open(fn):
-		d = ln.split(":")
-		cnt = d[0].strip()
-		ll = d[1]
-		if cnt == "-":
-			continue
-		if cnt == "#####":
-			cnt = 0
-		else:
-			cnt = int(cnt)
-		lno = int(d[1])
-		if lno not in dd:
-			dd[lno] = 0
-		dd[lno] += cnt
-	counts[sn] = dd
-	pl = lengths.get(sn)
-	ll = ll.strip()
-	if d[2] == "/*EOF*/\n":
-		ll = pl
-	elif pl != ll and not pl is None:
-		print("CONFLICT", fn, ll, pl)
-		ll = "-1"
-	lengths[sn] = ll
-	os.remove(fn)
+    """ Sum .gcov file into counts, then delete it """
+    dd = counts.get(sn)
+    if dd is None:
+        dd = {}
+    for ln in open(fn):
+        d = ln.split(":")
+        cnt = d[0].strip()
+        ll = d[1]
+        if cnt == "-":
+            continue
+        if cnt == "#####":
+            cnt = 0
+        else:
+            cnt = int(cnt)
+        lno = int(d[1])
+        if lno not in dd:
+            dd[lno] = 0
+        dd[lno] += cnt
+    counts[sn] = dd
+    pl = lengths.get(sn)
+    ll = ll.strip()
+    if d[2] == "/*EOF*/\n":
+        ll = pl
+    elif pl != ll and not pl is None:
+        print("CONFLICT", fn, ll, pl)
+        ll = "-1"
+    lengths[sn] = ll
+    os.remove(fn)
 
 def run_gcov(prog, subdir):
-	""" Walk tree, run gcov, process output """
-	for root, dirs, files in os.walk(subdir):
-		for i in exclude:
-			if i in dirs:
-				dirs.remove(i)
-		if " ".join(files).find(".gcda") == -1:
-			continue
-		for fn in files:
-			if fn[-2:] != ".o":
-				continue
-
-			# if we find the .o file in a .../.libs the sources
-			# must be found relative to the parent directory
-
-			if root[-6:] == "/.libs":
-				x = subprocess.check_output(
-				    ["cd " + root + "/.. && " +
-				     "exec " + prog + " -r .libs/" + fn],
-				    stderr=subprocess.STDOUT, shell=True,
-				    universal_newlines=True)
-				pf = ".."
-			else:
-				x = subprocess.check_output(
-				    ["cd " + root + " && " +
-				     "exec " + prog + " -r " + fn],
-				    stderr=subprocess.STDOUT, shell=True,
-				    universal_newlines=True)
-				pf = ""
-
-			for ln in x.split("\n"):
-				ln = ln.split()
-				if len(ln) == 0:
-					continue
-				if ln[0] == "Creating":
-					gn = ln[1].strip("'")
-					assert gn[-5:] == ".gcov"
-					sn = gn[:-5]
-					process_gcov(
-					    os.path.join(root, pf, gn), sn)
+    """ Walk tree, run gcov, process output """
+    for root, dirs, files in os.walk(subdir):
+        for i in exclude:
+            if i in dirs:
+                dirs.remove(i)
+        if " ".join(files).find(".gcda") == -1:
+            continue
+        for fn in files:
+            if fn[-2:] != ".o":
+                continue
+
+            # if we find the .o file in a .../.libs the sources
+            # must be found relative to the parent directory
+
+            if root[-6:] == "/.libs":
+                x = subprocess.check_output(
+                    ["cd " + root + "/.. && " +
+                     "exec " + prog + " -r .libs/" + fn],
+                    stderr=subprocess.STDOUT, shell=True,
+                    universal_newlines=True)
+                pf = ".."
+            else:
+                x = subprocess.check_output(
+                    ["cd " + root + " && " +
+                     "exec " + prog + " -r " + fn],
+                    stderr=subprocess.STDOUT, shell=True,
+                    universal_newlines=True)
+                pf = ""
+
+            for ln in x.split("\n"):
+                ln = ln.split()
+                if not ln:
+                    continue
+                if ln[0] == "Creating":
+                    gn = ln[1].strip("'")
+                    assert gn[-5:] == ".gcov"
+                    sn = gn[:-5]
+                    process_gcov(
+                        os.path.join(root, pf, gn), sn)
 
 def produce_output(fdo):
-	"""
-	Produce compact output file
-
-	Format:
-		linefm [lineto] count
-
-		"+" in linefm means "previous line + 1"
-		"." in count means "same as previous count"
-	"""
-
-	for sn, cnt in counts.items():
-		fdo.write("/" + sn + " " + str(lengths[sn]) + "\n")
-		lnos = list(cnt.items())
-		lnos.sort()
-		pln = -1
-		pcn = -1
-		while len(lnos) > 0:
-			ln, cn = lnos.pop(0)
-			lnl = ln
-			while len(lnos) > 0:
-				lnn, cnn = lnos[0]
-				if lnl + 1 != lnn or cnn != cn:
-					break
-				lnos.pop(0)
-				lnl = lnn
-			if ln == pln + 1:
-				s = "+ "
-			else:
-				s = "%d " % ln
-
-			if ln != lnl:
-				s += "%d " % lnl
-				pln = lnl
-			else:
-				pln = ln
-
-			if cn == pcn:
-				s += "."
-			else:
-				s += "%d" % cn
-				pcn = cn
-			fdo.write(s + "\n")
+    """
+    Produce compact output file
+
+    Format:
+        linefm [lineto] count
+
+        "+" in linefm means "previous line + 1"
+        "." in count means "same as previous count"
+    """
+
+    for sn, cnt in counts.items():
+        fdo.write("/" + sn + " " + str(lengths[sn]) + "\n")
+        lnos = list(cnt.items())
+        lnos.sort()
+        pln = -1
+        pcn = -1
+        while lnos:
+            ln, cn = lnos.pop(0)
+            lnl = ln
+            while lnos:
+                lnn, cnn = lnos[0]
+                if lnl + 1 != lnn or cnn != cn:
+                    break
+                lnos.pop(0)
+                lnl = lnn
+            if ln == pln + 1:
+                s = "+ "
+            else:
+                s = "%d " % ln
+
+            if ln != lnl:
+                s += "%d " % lnl
+                pln = lnl
+            else:
+                pln = ln
+
+            if cn == pcn:
+                s += "."
+            else:
+                s += "%d" % cn
+                pcn = cn
+            fdo.write(s + "\n")
 
 if __name__ == "__main__":
 
-	optlist, args = getopt.getopt(sys.argv[1:], "g:o:x:")
-
-	fo = sys.stdout
-	gcovprog = "gcov6"
-
-	for f, v in optlist:
-		if f == '-o' and v == '-':
-			fo = sys.stdout
-		elif f == '-o':
-			fo = open(v, "w")
-		elif f == '-g':
-			gcovprog = v
-		elif f == '-x':
-			exclude.append(v)
-		else:
-			assert False
-	if len(args) == 0:
-		args = ["."]
-	for dn in args:
-		run_gcov(gcovprog, dn)
-
-	produce_output(fo)
+    optlist, args = getopt.getopt(sys.argv[1:], "g:o:x:")
+
+    fo = sys.stdout
+    gcovprog = "gcov6"
+
+    for f, v in optlist:
+        if f == '-o' and v == '-':
+            fo = sys.stdout
+        elif f == '-o':
+            fo = open(v, "w")
+        elif f == '-g':
+            gcovprog = v
+        elif f == '-x':
+            exclude.append(v)
+        else:
+            assert False
+    if not args:
+        args = ["."]
+    for dn in args:
+        run_gcov(gcovprog, dn)
+
+    produce_output(fo)
diff --git a/tools/include_wash.py b/tools/include_wash.py
index 1284e30e1..4df6eeaf3 100644
--- a/tools/include_wash.py
+++ b/tools/include_wash.py
@@ -5,41 +5,39 @@ from __future__ import print_function
 import os
 
 def check(fn):
-	l = []
-	for i in open(fn):
-		i = i.strip()
-		if len(i) == 0:
-			continue
-		if i[0] != "#":
-			continue
-		if i.find("include") == -1:
-			continue
-		if i.find('"') == -1:
-			l.append(i.split('<')[1].split('>')[0])
-		else:
-			l.append(i.split('"')[1])
-	if "vrt.h" in l:
-		vrt = l.index("vrt.h")
-		if not "vdef.h" in l:
-			print(fn, "vdef.h not included with vrt.h")
-		vdef = l.index("vdef.h")
-		if vdef > vrt:
-			print(fn, "vdef.h included after vrt.h")
-		for i in ("stddef.h", "stdint.h", "cache/cache.h", "cache.h"):
-			if i in l:
-				print(fn, i + " included with vrt.h")
+    l = []
+    for i in open(fn):
+        i = i.strip()
+        if not i:
+            continue
+        if i[0] != "#":
+            continue
+        if i.find("include") == -1:
+            continue
+        if i.find('"') == -1:
+            l.append(i.split('<')[1].split('>')[0])
+        else:
+            l.append(i.split('"')[1])
+    if "vrt.h" in l:
+        vrt = l.index("vrt.h")
+        if "vdef.h" not in l:
+            print(fn, "vdef.h not included with vrt.h")
+        vdef = l.index("vdef.h")
+        if vdef > vrt:
+            print(fn, "vdef.h included after vrt.h")
+        for i in ("stddef.h", "stdint.h", "cache/cache.h", "cache.h"):
+            if i in l:
+                print(fn, i + " included with vrt.h")
 
-	for i in ("cache/cache.h", "cache.h"):
-		if i in l:
-			for i in (
-			    "stddef.h", "stdint.h", "vrt.h",
-			    "math.h", "pthread.h", "stdarg.h", "sys/types.h",
-			    "vdef.h", "miniobj.h", "vas.h", "vqueue.h",
-			):
-				if i in l:
-					print(fn, i + " included with cache.h")
+    for i in ("cache/cache.h", "cache.h"):
+        if i in l:
+            for i in ("stddef.h", "stdint.h", "vrt.h",
+                      "math.h", "pthread.h", "stdarg.h", "sys/types.h",
+                      "vdef.h", "miniobj.h", "vas.h", "vqueue.h"):
+                if i in l:
+                    print(fn, i + " included with cache.h")
 
 for (dir, dns, fns) in os.walk("."):
-	for f in fns:
-		if f[-2:] == ".c":
-			check(dir + "/" + f)
+    for f in fns:
+        if f[-2:] == ".c":
+            check(dir + "/" + f)

From phk at FreeBSD.org  Mon Sep 24 11:26:17 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 24 Sep 2018 11:26:17 +0000 (UTC)
Subject: [master] f2abaf869 Update Teken from FreeBSD source tree
Message-ID: <20180924112617.60BD76E1DF@lists.varnish-cache.org>


commit f2abaf8698a34a44d8a80d80756b0f843560e31d
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 24 07:31:46 2018 +0000

    Update Teken from FreeBSD source tree

diff --git a/bin/varnishtest/gensequences b/bin/varnishtest/gensequences
index 8c2299a61..e82e56ec7 100644
--- a/bin/varnishtest/gensequences
+++ b/bin/varnishtest/gensequences
@@ -25,7 +25,7 @@
 # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
 # SUCH DAMAGE.
 #
-# $FreeBSD: head/sys/teken/gensequences 223574 2011-06-26 18:25:10Z ed $
+# $FreeBSD: head/sys/teken/gensequences 333925 2018-05-20 14:21:20Z dumbbell $
 
 function die(msg) {
 	print msg;
@@ -35,6 +35,15 @@ function die(msg) {
 function cchar(str) {
 	if (str == "^[")
 		return "\\x1B";
+	if (str == "SP")
+		return " ";
+
+	return str;
+}
+
+function csequence(str) {
+	if (str == "SP")
+		return " ";
 
 	return str;
 }
@@ -57,11 +66,11 @@ while (getline > 0) {
 	prefix = "";
 	l_prefix_name[""] = "teken_state_init";
 	for (i = 1; i < nsequences; i++) {
-		n = prefix sequence[i];
+		n = prefix csequence(sequence[i]);
 		l_prefix_parent[n] = prefix;
 		l_prefix_suffix[n] = sequence[i];
 		if (!l_prefix_name[n])
-			l_prefix_name[n] = "teken_state_" "" ++npr;
+			l_prefix_name[n] = "teken_state_" ++npr;
 		prefix = n;
 	}
 
diff --git a/bin/varnishtest/sequences b/bin/varnishtest/sequences
index af92df04b..d8f30306b 100644
--- a/bin/varnishtest/sequences
+++ b/bin/varnishtest/sequences
@@ -23,93 +23,94 @@
 # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
 # SUCH DAMAGE.
 #
-# $FreeBSD: head/sys/teken/sequences 322662 2017-08-18 15:40:40Z bde $
+# $FreeBSD: head/sys/teken/sequences 334316 2018-05-29 08:41:44Z dumbbell $
 
 # File format is as follows:
 #	Abbr		Abbreviation of sequence name
 #	Name		Sequence name (will be converted to C function name)
 #	Sequence	Bytes that form the sequence
-#	Arguments	Standard value of arguments passed to this sequence
+#	Args		Standard value of arguments passed to this sequence
 #			- `n' non-zero number (0 gets converted to 1)
 #			- `r' regular numeric argument
 #			- `v' means a variable number of arguments
 
-# Abbr	Name					Sequence	Arguments
-CBT	Cursor Backward Tabulation		^[ [ Z		n
-CHT	Cursor Forward Tabulation		^[ [ I		n
-CNL	Cursor Next Line			^[ [ E		n
-CPL	Cursor Previous Line			^[ [ F		n
-CPR	Cursor Position Report			^[ [ n		r
-CUB	Cursor Backward				^[ [ D		n
-CUD	Cursor Down				^[ [ B		n
-CUD	Cursor Down				^[ [ e		n
-CUF	Cursor Forward				^[ [ C		n
-CUF	Cursor Forward				^[ [ a		n
-CUP	Cursor Position				^[ [ H		n n
-CUP	Cursor Position				^[ [ f		n n
-CUU	Cursor Up				^[ [ A		n
-DA1	Primary Device Attributes		^[ [ c		r
-DA2	Secondary Device Attributes		^[ [ > c	r
-DC	Delete character			^[ [ P		n
-DCS	Device Control String			^[ P
-DECALN	Alignment test				^[ # 8
-DECDHL	Double Height Double Width Line Top	^[ # 3
-DECDHL	Double Height Double Width Line Bottom	^[ # 4
-DECDWL	Single Height Double Width Line		^[ # 6
-DECKPAM	Keypad application mode			^[ =
-DECKPNM	Keypad numeric mode			^[ >
-DECRC	Restore cursor				^[ 8
-DECRC	Restore cursor				^[ [ u
-DECRM	Reset DEC mode				^[ [ ? l	r
-DECSC	Save cursor				^[ 7
-DECSC	Save cursor				^[ [ s
-DECSM	Set DEC mode				^[ [ ? h	r
-DECSTBM	Set top and bottom margins		^[ [ r		r r
-DECSWL	Single Height Single Width Line		^[ # 5
-DL	Delete line				^[ [ M		n
-DSR	Device Status Report			^[ [ ? n	r
-ECH	Erase character				^[ [ X		n
-ED	Erase display				^[ [ J		r
-EL	Erase line				^[ [ K		r
-G0SCS0	G0 SCS Special Graphics			^[ ( 0
-G0SCS1	G0 SCS US ASCII				^[ ( 1
-G0SCS2	G0 SCS Special Graphics			^[ ( 2
-G0SCSA	G0 SCS UK National			^[ ( A
-G0SCSB	G0 SCS US ASCII				^[ ( B
-G1SCS0	G1 SCS Special Graphics			^[ ) 0
-G1SCS1	G1 SCS US ASCII				^[ ) 1
-G1SCS2	G1 SCS Special Graphics			^[ ) 2
-G1SCSA	G1 SCS UK National			^[ ) A
-G1SCSB	G1 SCS US ASCII				^[ ) B
-HPA	Horizontal Position Absolute		^[ [ G		n
-HPA	Horizontal Position Absolute		^[ [ `		n
-HTS	Horizontal Tab Set			^[ H
-ICH	Insert character			^[ [ @		n
-IL	Insert line				^[ [ L		n
-IND	Index					^[ D
-NEL	Next line				^[ E
-OSC	Operating System Command		^[ ]
-RI	Reverse index				^[ M
-RIS	Reset to Initial State			^[ c
-RM	Reset Mode				^[ [ l		r
-SD	Pan Up					^[ [ T		n
-SGR	Set Graphic Rendition			^[ [ m		v
-SM	Set Mode				^[ [ h		r
-ST	String Terminator			^[ \\
-SU	Pan Down				^[ [ S		n
-TBC	Tab Clear				^[ [ g		r
-VPA	Vertical Position Absolute		^[ [ d		n
+# Abbr		Name					Sequence	Args
+CBT		Cursor Backward Tabulation		^[ [ Z		n
+CHT		Cursor Forward Tabulation		^[ [ I		n
+CNL		Cursor Next Line			^[ [ E		n
+CPL		Cursor Previous Line			^[ [ F		n
+CPR		Cursor Position Report			^[ [ n		r
+CUB		Cursor Backward				^[ [ D		n
+CUD		Cursor Down				^[ [ B		n
+CUD		Cursor Down				^[ [ e		n
+CUF		Cursor Forward				^[ [ C		n
+CUF		Cursor Forward				^[ [ a		n
+CUP		Cursor Position				^[ [ H		n n
+CUP		Cursor Position				^[ [ f		n n
+CUU		Cursor Up				^[ [ A		n
+DA1		Primary Device Attributes		^[ [ c		r
+DA2		Secondary Device Attributes		^[ [ > c	r
+DC		Delete character			^[ [ P		n
+DCS		Device Control String			^[ P
+DECALN		Alignment test				^[ # 8
+DECDHL		Double Height Double Width Line Top	^[ # 3
+DECDHL		Double Height Double Width Line Bottom	^[ # 4
+DECDWL		Single Height Double Width Line		^[ # 6
+DECKPAM		Keypad application mode			^[ =
+DECKPNM		Keypad numeric mode			^[ >
+DECRC		Restore cursor				^[ 8
+DECRC		Restore cursor				^[ [ u
+DECRM		Reset DEC mode				^[ [ ? l	r
+DECSC		Save cursor				^[ 7
+DECSC		Save cursor				^[ [ s
+DECSCUSR	Set Cursor Style			^[ [ SP q	r
+DECSM		Set DEC mode				^[ [ ? h	r
+DECSTBM		Set top and bottom margins		^[ [ r		r r
+DECSWL		Single Height Single Width Line		^[ # 5
+DL		Delete line				^[ [ M		n
+DSR		Device Status Report			^[ [ ? n	r
+ECH		Erase character				^[ [ X		n
+ED		Erase display				^[ [ J		r
+EL		Erase line				^[ [ K		r
+G0SCS0		G0 SCS Special Graphics			^[ ( 0
+G0SCS1		G0 SCS US ASCII				^[ ( 1
+G0SCS2		G0 SCS Special Graphics			^[ ( 2
+G0SCSA		G0 SCS UK National			^[ ( A
+G0SCSB		G0 SCS US ASCII				^[ ( B
+G1SCS0		G1 SCS Special Graphics			^[ ) 0
+G1SCS1		G1 SCS US ASCII				^[ ) 1
+G1SCS2		G1 SCS Special Graphics			^[ ) 2
+G1SCSA		G1 SCS UK National			^[ ) A
+G1SCSB		G1 SCS US ASCII				^[ ) B
+HPA		Horizontal Position Absolute		^[ [ G		n
+HPA		Horizontal Position Absolute		^[ [ `		n
+HTS		Horizontal Tab Set			^[ H
+ICH		Insert character			^[ [ @		n
+IL		Insert line				^[ [ L		n
+IND		Index					^[ D
+NEL		Next line				^[ E
+OSC		Operating System Command		^[ ]
+RI		Reverse index				^[ M
+RIS		Reset to Initial State			^[ c
+RM		Reset Mode				^[ [ l		r
+SD		Pan Up					^[ [ T		n
+SGR		Set Graphic Rendition			^[ [ m		v
+SM		Set Mode				^[ [ h		r
+ST		String Terminator			^[ \\
+SU		Pan Down				^[ [ S		n
+TBC		Tab Clear				^[ [ g		r
+VPA		Vertical Position Absolute		^[ [ d		n
 
 # Cons25 compatibility sequences
-C25BLPD	Cons25 set bell pitch duration		^[ [ = B	r r
-C25BORD	Cons25 set border			^[ [ = A	r
-C25DBG	Cons25 set default background		^[ [ = G	r
-C25DFG	Cons25 set default foreground		^[ [ = F	r
-C25GCS	Cons25 set global cursor shape		^[ [ = C	v
-C25LCT	Cons25 set local cursor type		^[ [ = S	r
-C25MODE	Cons25 set terminal mode		^[ [ = T	r
-C25SGR	Cons25 set graphic rendition		^[ [ x		r r
-C25VTSW	Cons25 switch virtual terminal		^[ [ z		r
+C25BLPD		Cons25 set bell pitch duration		^[ [ = B	r r
+C25BORD		Cons25 set border			^[ [ = A	r
+C25DBG		Cons25 set default background		^[ [ = G	r
+C25DFG		Cons25 set default foreground		^[ [ = F	r
+C25GCS		Cons25 set global cursor shape		^[ [ = C	v
+C25LCT		Cons25 set local cursor type		^[ [ = S	r
+C25MODE		Cons25 set terminal mode		^[ [ = T	r
+C25SGR		Cons25 set graphic rendition		^[ [ x		r r
+C25VTSW		Cons25 switch virtual terminal		^[ [ z		r
 
 # VT52 compatibility
-#DECID	VT52 DECID				^[ Z
+#DECID		VT52 DECID				^[ Z
diff --git a/bin/varnishtest/teken.c b/bin/varnishtest/teken.c
index ad06c57c6..21eb71752 100644
--- a/bin/varnishtest/teken.c
+++ b/bin/varnishtest/teken.c
@@ -25,7 +25,7 @@
  * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
  * SUCH DAMAGE.
  *
- * $FreeBSD: head/sys/teken/teken.c 326272 2017-11-27 15:23:17Z pfg $
+ * $FreeBSD: head/sys/teken/teken.c 333683 2018-05-16 18:12:49Z cem $
  */
 
 #include "config.h"
@@ -35,14 +35,13 @@
 #include <stdint.h>
 #include <stdio.h>
 #include <string.h>
+#define	teken_assert(x)		assert(x)
 
 #include "vdef.h"
 #include "vas.h"
 
-#define	teken_assert(x)		assert(x)
-
 /* debug messages */
-#define	teken_printf(...)
+#define	teken_printf(x,...)
 
 /* Private flags for t_stateflags. */
 #define	TS_FIRSTDIGIT	0x0001	/* First numeric digit in escape sequence. */
@@ -128,20 +127,36 @@ teken_funcs_copy(const teken_t *t, const teken_rect_t *r, const teken_pos_t *p)
 	t->t_funcs->tf_copy(t->t_softc, r, p);
 }
 
+static inline void
+teken_funcs_pre_input(const teken_t *t)
+{
+
+	if (t->t_funcs->tf_pre_input != NULL)
+		t->t_funcs->tf_pre_input(t->t_softc);
+}
+
+static inline void
+teken_funcs_post_input(const teken_t *t)
+{
+
+	if (t->t_funcs->tf_post_input != NULL)
+		t->t_funcs->tf_post_input(t->t_softc);
+}
+
 static inline void
 teken_funcs_param(const teken_t *t, int cmd, unsigned int value)
 {
 
-	if (t->t_funcs->tf_param != NULL)
-		t->t_funcs->tf_param(t->t_softc, cmd, value);
+	teken_assert(t->t_funcs->tf_param != NULL);
+	t->t_funcs->tf_param(t->t_softc, cmd, value);
 }
 
 static inline void
 teken_funcs_respond(const teken_t *t, const void *buf, size_t len)
 {
 
-	if (t->t_funcs->tf_respond != NULL)
-		t->t_funcs->tf_respond(t->t_softc, buf, len);
+	teken_assert(t->t_funcs->tf_respond != NULL);
+	t->t_funcs->tf_respond(t->t_softc, buf, len);
 }
 
 #include "teken_subr.h"
@@ -288,8 +303,10 @@ teken_input(teken_t *t, const void *buf, size_t len)
 {
 	const char *c = buf;
 
+	teken_funcs_pre_input(t);
 	while (len-- > 0)
 		teken_input_byte(t, *c++);
+	teken_funcs_post_input(t);
 }
 
 const teken_pos_t *
diff --git a/bin/varnishtest/teken.h b/bin/varnishtest/teken.h
index 986c3bd36..eb59817d7 100644
--- a/bin/varnishtest/teken.h
+++ b/bin/varnishtest/teken.h
@@ -25,7 +25,7 @@
  * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
  * SUCH DAMAGE.
  *
- * $FreeBSD: head/sys/teken/teken.h 326272 2017-11-27 15:23:17Z pfg $
+ * $FreeBSD: head/sys/teken/teken.h 333669 2018-05-16 09:01:02Z dumbbell $
  */
 
 #ifndef _TEKEN_H_
@@ -93,6 +93,8 @@ typedef void tf_putchar_t(void *, const teken_pos_t *, teken_char_t,
 typedef void tf_fill_t(void *, const teken_rect_t *, teken_char_t,
     const teken_attr_t *);
 typedef void tf_copy_t(void *, const teken_rect_t *, const teken_pos_t *);
+typedef void tf_pre_input_t(void *);
+typedef void tf_post_input_t(void *);
 typedef void tf_param_t(void *, int, unsigned int);
 #define	TP_SHOWCURSOR	0
 #define	TP_KEYPADAPP	1
@@ -114,6 +116,8 @@ typedef struct {
 	tf_putchar_t	*tf_putchar;
 	tf_fill_t	*tf_fill;
 	tf_copy_t	*tf_copy;
+	tf_pre_input_t	*tf_pre_input;
+	tf_post_input_t	*tf_post_input;
 	tf_param_t	*tf_param;
 	tf_respond_t	*tf_respond;
 } teken_funcs_t;
diff --git a/bin/varnishtest/teken_scs.h b/bin/varnishtest/teken_scs.h
index fd99de15d..719f2a98e 100644
--- a/bin/varnishtest/teken_scs.h
+++ b/bin/varnishtest/teken_scs.h
@@ -25,7 +25,7 @@
  * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
  * SUCH DAMAGE.
  *
- * $FreeBSD: head/sys/teken/teken_scs.h 326272 2017-11-27 15:23:17Z pfg $
+ * $FreeBSD: head/sys/teken/teken_scs.h 332297 2018-04-08 19:23:50Z phk $
  */
 
 static inline teken_char_t
diff --git a/bin/varnishtest/teken_subr.h b/bin/varnishtest/teken_subr.h
index b67ef5cc0..22d06bb19 100644
--- a/bin/varnishtest/teken_subr.h
+++ b/bin/varnishtest/teken_subr.h
@@ -25,7 +25,7 @@
  * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
  * SUCH DAMAGE.
  *
- * $FreeBSD: head/sys/teken/teken_subr.h 326272 2017-11-27 15:23:17Z pfg $
+ * $FreeBSD: head/sys/teken/teken_subr.h 333995 2018-05-21 20:35:16Z dumbbell $
  */
 
 static void teken_subr_cursor_up(teken_t *, unsigned int);
@@ -371,6 +371,27 @@ teken_subr_cursor_up(teken_t *t, unsigned int nrows)
 	teken_funcs_cursor(t);
 }
 
+static void
+teken_subr_set_cursor_style(teken_t *t, unsigned int style)
+{
+
+	/* TODO */
+	(void)t;
+	(void)style;
+
+	/*
+	 * CSI Ps SP q
+	 *   Set cursor style (DECSCUSR), VT520.
+	 *     Ps = 0  -> blinking block.
+	 *     Ps = 1  -> blinking block (default).
+	 *     Ps = 2  -> steady block.
+	 *     Ps = 3  -> blinking underline.
+	 *     Ps = 4  -> steady underline.
+	 *     Ps = 5  -> blinking bar (xterm).
+	 *     Ps = 6  -> steady bar (xterm).
+	 */
+}
+
 static void
 teken_subr_delete_character(const teken_t *t, unsigned int ncols)
 {
diff --git a/bin/varnishtest/teken_subr_compat.h b/bin/varnishtest/teken_subr_compat.h
index 9c84f1331..7db4d858d 100644
--- a/bin/varnishtest/teken_subr_compat.h
+++ b/bin/varnishtest/teken_subr_compat.h
@@ -25,7 +25,7 @@
  * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
  * SUCH DAMAGE.
  *
- * $FreeBSD: head/sys/teken/teken_subr_compat.h 326272 2017-11-27 15:23:17Z pfg $
+ * $FreeBSD: head/sys/teken/teken_subr_compat.h 332297 2018-04-08 19:23:50Z phk $
  */
 
 static void
diff --git a/bin/varnishtest/teken_wcwidth.h b/bin/varnishtest/teken_wcwidth.h
index 73401698f..70d92060f 100644
--- a/bin/varnishtest/teken_wcwidth.h
+++ b/bin/varnishtest/teken_wcwidth.h
@@ -7,7 +7,7 @@
  *
  * Latest version: http://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c
  *
- * $FreeBSD: head/sys/teken/teken_wcwidth.h 186681 2009-01-01 13:26:53Z ed $
+ * $FreeBSD: head/sys/teken/teken_wcwidth.h 332297 2018-04-08 19:23:50Z phk $
  */
 
 struct interval {

From phk at FreeBSD.org  Mon Sep 24 11:26:17 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 24 Sep 2018 11:26:17 +0000 (UTC)
Subject: [master] d3ee2909d Introduce support for 'groups' inside a VSC
Message-ID: <20180924112617.85FA76E1E3@lists.varnish-cache.org>


commit d3ee2909ddf9589fdb5d472ccd2a1ce0f4690434
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 24 07:33:39 2018 +0000

    Introduce support for 'groups' inside a VSC

diff --git a/lib/libvcc/vsctool.py b/lib/libvcc/vsctool.py
index 6c96388e5..d4e210deb 100644
--- a/lib/libvcc/vsctool.py
+++ b/lib/libvcc/vsctool.py
@@ -53,6 +53,7 @@ PARAMS = {
     "ctype": CTYPES,
     "level": LEVELS,
     "oneliner": None,
+    "group": None,
     "format": FORMATS,
 }
 
@@ -88,22 +89,30 @@ class CounterSet(object):
         self.name = name
         self.struct = "struct VSC_" + name
         self.mbrs = []
+        self.groups = {}
         self.head = m
         self.completed = False
         self.off = 0
 
-    def addmbr(self, m):
+    def addmbr(self, m, g):
         '''Add a counter'''
         assert not self.completed
         self.mbrs.append(m)
         retval = self.off
         self.off += 8
+        if g is not None:
+            if g not in self.groups:
+                self.groups[g] = []
+            self.groups[g].append(m)
         return retval
 
     def complete(self, arg):
         '''Mark set completed'''
         assert arg == self.name
         self.completed = True
+        self.gnames = list(self.groups.keys())
+        self.gnames.sort()
+
 
     def emit_json(self, fo):
         '''Emit JSON as compact C byte-array and as readable C-comments'''
@@ -152,15 +161,24 @@ class CounterSet(object):
     def emit_h(self):
         '''Emit .h file'''
         assert self.completed
+
         fon = "VSC_" + self.name + ".h"
         fo = open(fon, "w")
         genhdr(fo, self.name)
+
         fo.write(self.struct + " {\n")
         for i in self.mbrs:
             fo.write("\tuint64_t\t%s;\n" % i.arg)
         fo.write("};\n")
         fo.write("\n")
 
+        for i in self.gnames:
+            fo.write(self.struct + "_" + i + " {\n")
+            for j in self.groups[i]:
+                fo.write("\tuint64_t\t%s;\n" % j.arg)
+            fo.write("};\n")
+            fo.write("\n")
+
         fo.write("#define VSC_" + self.name +
                  "_size PRNDUP(sizeof(" + self.struct + "))\n\n")
 
@@ -175,6 +193,10 @@ class CounterSet(object):
             fo.write("void VSC_" + self.name + "_Summ")
             fo.write("(" + self.struct + " *, ")
             fo.write("const " + self.struct + " *);\n")
+            for i in self.gnames:
+                fo.write("void VSC_" + self.name + "_Summ_" + i)
+                fo.write("(" + self.struct + " *, ")
+                fo.write("const " + self.struct + "_" + i + " *);\n")
 
     def emit_c_paranoia(self, fo):
         '''Emit asserts to make sure compiler gets same byte index'''
@@ -189,18 +211,27 @@ class CounterSet(object):
 
         fo.write("#undef PARANOIA\n")
 
-    def emit_c_sumfunc(self, fo):
+    def emit_c_sumfunc(self, fo, g=None):
         '''Emit a function summ up countersets'''
         fo.write("\n")
         fo.write("void\n")
         fo.write("VSC_" + self.name + "_Summ")
+        if g is not None:
+            fo.write("_" + g)
         fo.write("(" + self.struct + " *dst, ")
-        fo.write("const " + self.struct + " *src)\n")
+        fo.write("const " + self.struct)
+        if g is not None:
+            fo.write("_" + g)
+        fo.write(" *src)\n")
         fo.write("{\n")
         fo.write("\n")
         fo.write("\tAN(dst);\n")
         fo.write("\tAN(src);\n")
-        for i in self.mbrs:
+        if g:
+            l = self.groups[g]
+        else:
+            l = self.mbrs
+        for i in l:
             s1 = "\tdst->" + i.arg + " +="
             s2 = "src->" + i.arg + ";"
             if len((s1 + " " + s2).expandtabs()) < 79:
@@ -271,6 +302,8 @@ class CounterSet(object):
         self.emit_c_destroyfunc(fo)
         if 'sumfunction' in self.head.param:
             self.emit_c_sumfunc(fo)
+            for i in self.gnames:
+                self.emit_c_sumfunc(fo, i)
 
 #######################################################################
 
@@ -359,7 +392,7 @@ class RstVscDirective(OurDirective):
             sys.stderr.write("'" + p + "'")
             sys.stderr.write(" on field '" + self.arg + "'\n")
             exit(2)
-        self.param["index"] = vsc_set[-1].addmbr(self)
+        self.param["index"] = vsc_set[-1].addmbr(self, self.param.get("group"))
         if fo:
             fo.write("\n``%s`` ? " % self.arg)
             fo.write("`%s` - " % self.param["type"])

From phk at FreeBSD.org  Mon Sep 24 11:26:17 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 24 Sep 2018 11:26:17 +0000 (UTC)
Subject: [master] 254f6b367 Move all the ban counters into a "ban-mtx" group,
 update them directly in VSC_C_main and protect them with the ban-mtx.
Message-ID: <20180924112617.B89116E1FC@lists.varnish-cache.org>


commit 254f6b367fc762b1aa067647232ecc9d94df2cab
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 24 07:41:49 2018 +0000

    Move all the ban counters into a "ban-mtx" group, update them directly
    in VSC_C_main and protect them with the ban-mtx.

diff --git a/bin/varnishd/VSC_main.vsc b/bin/varnishd/VSC_main.vsc
index 0bd491ca6..ffa04f22e 100644
--- a/bin/varnishd/VSC_main.vsc
+++ b/bin/varnishd/VSC_main.vsc
@@ -595,6 +595,7 @@
 
 .. varnish_vsc:: bans
 	:type:	gauge
+	:group: ban_mtx
 	:oneliner:	Count of bans
 
 	Number of all bans in system, including bans superseded by newer
@@ -603,6 +604,7 @@
 .. varnish_vsc:: bans_completed
 	:type:	gauge
 	:level:	diag
+	:group: ban_mtx
 	:oneliner:	Number of bans marked 'completed'
 
 	Number of bans which are no longer active, either because they got
@@ -611,6 +613,7 @@
 .. varnish_vsc:: bans_obj
 	:type:	gauge
 	:level:	diag
+	:group: ban_mtx
 	:oneliner:	Number of bans using obj.*
 
 	Number of bans which use obj.* variables.  These bans can possibly
@@ -619,6 +622,7 @@
 .. varnish_vsc:: bans_req
 	:type:	gauge
 	:level:	diag
+	:group: ban_mtx
 	:oneliner:	Number of bans using req.*
 
 	Number of bans which use req.* variables.  These bans can not be
@@ -626,18 +630,21 @@
 
 .. varnish_vsc:: bans_added
 	:level:	diag
+	:group: ban_mtx
 	:oneliner:	Bans added
 
 	Counter of bans added to ban list.
 
 .. varnish_vsc:: bans_deleted
 	:level:	diag
+	:group: ban_mtx
 	:oneliner:	Bans deleted
 
 	Counter of bans deleted from ban list.
 
 .. varnish_vsc:: bans_tested
 	:level:	diag
+	:group: ban_mtx
 	:oneliner:	Bans tested against objects (lookup)
 
 	Count of how many bans and objects have been tested against each
@@ -645,12 +652,14 @@
 
 .. varnish_vsc:: bans_obj_killed
 	:level:	diag
+	:group: ban_mtx
 	:oneliner:	Objects killed by bans (lookup)
 
 	Number of objects killed by bans during object lookup.
 
 .. varnish_vsc:: bans_lurker_tested
 	:level:	diag
+	:group: ban_mtx
 	:oneliner:	Bans tested against objects (lurker)
 
 	Count of how many bans and objects have been tested against each
@@ -658,6 +667,7 @@
 
 .. varnish_vsc:: bans_tests_tested
 	:level:	diag
+	:group: ban_mtx
 	:oneliner:	Ban tests tested against objects (lookup)
 
 	Count of how many tests and objects have been tested against each
@@ -666,6 +676,7 @@
 
 .. varnish_vsc:: bans_lurker_tests_tested
 	:level:	diag
+	:group: ban_mtx
 	:oneliner:	Ban tests tested against objects (lurker)
 
 	Count of how many tests and objects have been tested against each
@@ -674,12 +685,14 @@
 
 .. varnish_vsc:: bans_lurker_obj_killed
 	:level:	diag
+	:group: ban_mtx
 	:oneliner:	Objects killed by bans (lurker)
 
 	Number of objects killed by the ban-lurker.
 
 .. varnish_vsc:: bans_lurker_obj_killed_cutoff
 	:level:	diag
+	:group: ban_mtx
 	:oneliner:	Objects killed by bans for cutoff (lurker)
 
 	Number of objects killed by the ban-lurker to keep the number of
@@ -687,12 +700,14 @@
 
 .. varnish_vsc:: bans_dups
 	:level:	diag
+	:group: ban_mtx
 	:oneliner:	Bans superseded by other bans
 
 	Count of bans replaced by later identical bans.
 
 .. varnish_vsc:: bans_lurker_contention
 	:level:	diag
+	:group: ban_mtx
 	:oneliner:	Lurker gave way for lookup
 
 	Number of times the ban-lurker had to wait for lookups.
@@ -701,6 +716,7 @@
 	:type:	gauge
 	:format:	bytes
 	:level:	diag
+	:group: ban_mtx
 	:oneliner:	Bytes used by the persisted ban lists
 
 	Number of bytes used by the persisted ban lists.
@@ -709,6 +725,7 @@
 	:type:	gauge
 	:format:	bytes
 	:level:	diag
+	:group: ban_mtx
 	:oneliner:	Extra bytes in persisted ban lists due to fragmentation
 
 	Number of extra bytes accumulated through dropped and completed
diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index e8dc2f3de..093a08b00 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -579,7 +579,7 @@ struct sess {
 struct ban_proto *BAN_Build(void);
 const char *BAN_AddTest(struct ban_proto *,
     const char *, const char *, const char *);
-const char *BAN_Commit(struct ban_proto *b, struct VSC_main *stats);
+const char *BAN_Commit(struct ban_proto *b);
 void BAN_Abandon(struct ban_proto *b);
 
 /* cache_cli.c [CLI] */
diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index b3203a042..5e8712503 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -152,7 +152,7 @@ ban_equal(const uint8_t *bs1, const uint8_t *bs2)
 }
 
 void
-ban_mark_completed(struct ban *b, struct VSC_main *stats)
+ban_mark_completed(struct ban *b)
 {
 	unsigned ln;
 
@@ -166,7 +166,7 @@ ban_mark_completed(struct ban *b, struct VSC_main *stats)
 		b->spec[BANS_FLAGS] |= BANS_FLAG_COMPLETED;
 		VWMB();
 		vbe32enc(b->spec + BANS_LENGTH, BANS_HEAD_LEN);
-		stats->bans_completed++;
+		VSC_C_main->bans_completed++;
 		bans_persisted_fragmentation += ln - ban_len(b->spec);
 		/*
 		 * XXX absolute update of gauges - may be inaccurate for
@@ -367,7 +367,6 @@ ban_reload(const uint8_t *ban, unsigned len)
 	struct ban *b, *b2;
 	int duplicate = 0;
 	double t0, t1, t2 = 9e99;
-	struct VSC_main *stats = VSC_C_main;	// XXX accurate?
 	ASSERT_CLI();
 	Lck_AssertHeld(&ban_mtx);
 
@@ -386,8 +385,8 @@ ban_reload(const uint8_t *ban, unsigned len)
 			duplicate = 1;
 	}
 
-	stats->bans++;
-	stats->bans_added++;
+	VSC_C_main->bans++;
+	VSC_C_main->bans_added++;
 
 	b2 = ban_alloc();
 	AN(b2);
@@ -395,13 +394,13 @@ ban_reload(const uint8_t *ban, unsigned len)
 	AN(b2->spec);
 	memcpy(b2->spec, ban, len);
 	if (ban[BANS_FLAGS] & BANS_FLAG_REQ) {
-		stats->bans_req++;
+		VSC_C_main->bans_req++;
 		b2->flags |= BANS_FLAG_REQ;
 	}
 	if (duplicate)
-		stats->bans_dups++;
+		VSC_C_main->bans_dups++;
 	if (duplicate || (ban[BANS_FLAGS] & BANS_FLAG_COMPLETED))
-		ban_mark_completed(b2, stats);
+		ban_mark_completed(b2);
 	if (b == NULL)
 		VTAILQ_INSERT_TAIL(&ban_head, b2, list);
 	else
@@ -418,8 +417,8 @@ ban_reload(const uint8_t *ban, unsigned len)
 		if (b->flags & BANS_FLAG_COMPLETED)
 			continue;
 		if (ban_equal(b->spec, ban)) {
-			ban_mark_completed(b, stats);
-			stats->bans_dups++;
+			ban_mark_completed(b);
+			VSC_C_main->bans_dups++;
 		}
 	}
 }
@@ -548,7 +547,6 @@ BAN_CheckObject(struct worker *wrk, struct objcore *oc, struct req *req)
 	struct vsl_log *vsl;
 	struct ban *b0, *bn;
 	unsigned tests;
-	struct VSC_main *stats;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(oc, OBJCORE_MAGIC);
@@ -557,7 +555,6 @@ BAN_CheckObject(struct worker *wrk, struct objcore *oc, struct req *req)
 	assert(oc->refcnt > 0);
 
 	vsl = req->vsl;
-	stats = wrk->stats;
 
 	CHECK_OBJ_NOTNULL(oc->ban, BAN_MAGIC);
 
@@ -600,8 +597,8 @@ BAN_CheckObject(struct worker *wrk, struct objcore *oc, struct req *req)
 
 	Lck_Lock(&ban_mtx);
 	bn->refcount--;
-	stats->bans_tested++;
-	stats->bans_tests_tested += tests;
+	VSC_C_main->bans_tested++;
+	VSC_C_main->bans_tests_tested += tests;
 
 	if (b == bn) {
 		/* not banned */
@@ -612,6 +609,8 @@ BAN_CheckObject(struct worker *wrk, struct objcore *oc, struct req *req)
 		oc->ban = b0;
 		b = NULL;
 	}
+	if (b != NULL)
+		VSC_C_main->bans_obj_killed++;
 
 	if (VTAILQ_LAST(&ban_head, banhead_s)->refcount == 0)
 		ban_kick_lurker();
@@ -624,7 +623,6 @@ BAN_CheckObject(struct worker *wrk, struct objcore *oc, struct req *req)
 		return (0);
 	} else {
 		VSLb(vsl, SLT_ExpBan, "%u banned lookup", ObjGetXID(wrk, oc));
-		stats->bans_obj_killed++;
 		return (1);
 	}
 }
@@ -672,7 +670,7 @@ ccf_ban(struct cli *cli, const char * const *av, void *priv)
 
 	if (err == NULL) {
 		// XXX racy - grab wstat lock?
-		err = BAN_Commit(bp, VSC_C_main);
+		err = BAN_Commit(bp);
 	}
 
 	if (err != NULL) {
@@ -821,9 +819,9 @@ BAN_Init(void)
 	bp = BAN_Build();
 	AN(bp);
 	AZ(pthread_cond_init(&ban_lurker_cond, NULL));
-	AZ(BAN_Commit(bp, VSC_C_main));
+	AZ(BAN_Commit(bp));
 	Lck_Lock(&ban_mtx);
-	ban_mark_completed(VTAILQ_FIRST(&ban_head), VSC_C_main);
+	ban_mark_completed(VTAILQ_FIRST(&ban_head));
 	Lck_Unlock(&ban_mtx);
 }
 
diff --git a/bin/varnishd/cache/cache_ban.h b/bin/varnishd/cache/cache_ban.h
index d534f6e99..62fc767bb 100644
--- a/bin/varnishd/cache/cache_ban.h
+++ b/bin/varnishd/cache/cache_ban.h
@@ -109,7 +109,7 @@ extern pthread_cond_t	ban_lurker_cond;
 extern uint64_t bans_persisted_bytes;
 extern uint64_t bans_persisted_fragmentation;
 
-void ban_mark_completed(struct ban *b, struct VSC_main *stats);
+void ban_mark_completed(struct ban *);
 unsigned ban_len(const uint8_t *banspec);
 void ban_info_new(const uint8_t *ban, unsigned len);
 void ban_info_drop(const uint8_t *ban, unsigned len);
diff --git a/bin/varnishd/cache/cache_ban_build.c b/bin/varnishd/cache/cache_ban_build.c
index 95f82bcc8..1ff460057 100644
--- a/bin/varnishd/cache/cache_ban_build.c
+++ b/bin/varnishd/cache/cache_ban_build.c
@@ -246,7 +246,7 @@ BAN_AddTest(struct ban_proto *bp,
  */
 
 const char *
-BAN_Commit(struct ban_proto *bp, struct VSC_main *stats)
+BAN_Commit(struct ban_proto *bp)
 {
 	struct ban  *b, *bi;
 	ssize_t ln;
@@ -294,8 +294,8 @@ BAN_Commit(struct ban_proto *bp, struct VSC_main *stats)
 	VTAILQ_INSERT_HEAD(&ban_head, b, list);
 	ban_start = b;
 
-	stats->bans++;
-	stats->bans_added++;
+	VSC_C_main->bans++;
+	VSC_C_main->bans_added++;
 	bans_persisted_bytes += ln;
 	/*
 	 * XXX absolute update of gauges - may be inaccurate for Pool_Sumstat
@@ -304,9 +304,9 @@ BAN_Commit(struct ban_proto *bp, struct VSC_main *stats)
 	VSC_C_main->bans_persisted_bytes = bans_persisted_bytes;
 
 	if (b->flags & BANS_FLAG_OBJ)
-		stats->bans_obj++;
+		VSC_C_main->bans_obj++;
 	if (b->flags & BANS_FLAG_REQ)
-		stats->bans_req++;
+		VSC_C_main->bans_req++;
 
 	if (bi != NULL)
 		ban_info_new(b->spec, ln);	/* Notify stevedores */
@@ -317,8 +317,8 @@ BAN_Commit(struct ban_proto *bp, struct VSC_main *stats)
 		    bi = VTAILQ_NEXT(bi, list)) {
 			if (!(bi->flags & BANS_FLAG_COMPLETED) &&
 			    ban_equal(b->spec, bi->spec)) {
-				ban_mark_completed(bi, stats);
-				stats->bans_dups++;
+				ban_mark_completed(bi);
+				VSC_C_main->bans_dups++;
 			}
 		}
 	}
diff --git a/bin/varnishd/cache/cache_ban_lurker.c b/bin/varnishd/cache/cache_ban_lurker.c
index 343009dd5..0680941ae 100644
--- a/bin/varnishd/cache/cache_ban_lurker.c
+++ b/bin/varnishd/cache/cache_ban_lurker.c
@@ -62,7 +62,7 @@ ban_kick_lurker(void)
  */
 
 static int
-ban_cleantail(const struct ban *victim, struct VSC_main *stats)
+ban_cleantail(const struct ban *victim)
 {
 	struct ban *b, *bt;
 	struct banhead_s freelist = VTAILQ_HEAD_INITIALIZER(freelist);
@@ -78,13 +78,13 @@ ban_cleantail(const struct ban *victim, struct VSC_main *stats)
 		if (b != VTAILQ_FIRST(&ban_head) && b->refcount == 0) {
 			assert(VTAILQ_EMPTY(&b->objcore));
 			if (b->flags & BANS_FLAG_COMPLETED)
-				stats->bans_completed--;
+				VSC_C_main->bans_completed--;
 			if (b->flags & BANS_FLAG_OBJ)
-				stats->bans_obj--;
+				VSC_C_main->bans_obj--;
 			if (b->flags & BANS_FLAG_REQ)
-				stats->bans_req--;
-			stats->bans--;
-			stats->bans_deleted++;
+				VSC_C_main->bans_req--;
+			VSC_C_main->bans--;
+			VSC_C_main->bans_deleted++;
 			VTAILQ_REMOVE(&ban_head, b, list);
 			VTAILQ_INSERT_TAIL(&freelist, b, list);
 			bans_persisted_fragmentation +=
@@ -125,7 +125,7 @@ ban_cleantail(const struct ban *victim, struct VSC_main *stats)
  */
 
 static struct objcore *
-ban_lurker_getfirst(struct vsl_log *vsl, struct ban *bt, struct VSC_main *stats)
+ban_lurker_getfirst(struct vsl_log *vsl, struct ban *bt)
 {
 	struct objhead *oh;
 	struct objcore *oc, *noc;
@@ -154,8 +154,8 @@ ban_lurker_getfirst(struct vsl_log *vsl, struct ban *bt, struct VSC_main *stats)
 			assert(move_oc == 0);
 
 			/* hold off to give lookup a chance and reiterate */
+			VSC_C_main->bans_lurker_contention++;
 			Lck_Unlock(&ban_mtx);
-			stats->bans_lurker_contention++;
 			VSL_Flush(vsl, 0);
 			VTIM_sleep(cache_param->ban_lurker_holdoff);
 			Lck_Lock(&ban_mtx);
@@ -209,10 +209,9 @@ ban_lurker_test_ban(struct worker *wrk, struct vsl_log *vsl, struct ban *bt,
 	struct objcore *oc;
 	unsigned tests;
 	int i;
-	struct VSC_main *stats;
+	uint64_t tested = 0, tested_tests = 0, lok = 0, lokc = 0;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	stats = wrk->stats;
 
 	/*
 	 * First see if there is anything to do, and if so, insert markers
@@ -232,9 +231,16 @@ ban_lurker_test_ban(struct worker *wrk, struct vsl_log *vsl, struct ban *bt,
 			VTIM_sleep(cache_param->ban_lurker_sleep);
 			ban_batch = 0;
 		}
-		oc = ban_lurker_getfirst(vsl, bt, stats);
-		if (oc == NULL)
+		oc = ban_lurker_getfirst(vsl, bt);
+		if (oc == NULL) {
+			Lck_Lock(&ban_mtx);
+			VSC_C_main->bans_lurker_tested += tested;
+			VSC_C_main->bans_lurker_tests_tested += tested_tests;
+			VSC_C_main->bans_lurker_obj_killed += lok;
+			VSC_C_main->bans_lurker_obj_killed_cutoff += lokc;
+			Lck_Unlock(&ban_mtx);
 			return;
+		}
 		i = 0;
 		VTAILQ_FOREACH_REVERSE_SAFE(bl, obans, banhead_s, l_list, bln) {
 			if (oc->ban != bt) {
@@ -256,21 +262,20 @@ ban_lurker_test_ban(struct worker *wrk, struct vsl_log *vsl, struct ban *bt,
 				tests = 0;
 				i = ban_evaluate(wrk, bl->spec, oc, NULL,
 				    &tests);
-				stats->bans_lurker_tested++;
-				stats->bans_lurker_tests_tested += tests;
+				tested++;
+				tested_tests += tests;
 			}
 			if (i) {
 				if (kill) {
 					VSLb(vsl, SLT_ExpBan,
 					    "%u killed for lurker cutoff",
 					    ObjGetXID(wrk, oc));
-					stats->
-					    bans_lurker_obj_killed_cutoff++;
+					lokc++;
 				} else {
 					VSLb(vsl, SLT_ExpBan,
 					    "%u banned by lurker",
 					    ObjGetXID(wrk, oc));
-					stats->bans_lurker_obj_killed++;
+					lok++;
 				}
 				HSH_Kill(oc);
 				break;
@@ -278,6 +283,11 @@ ban_lurker_test_ban(struct worker *wrk, struct vsl_log *vsl, struct ban *bt,
 		}
 		if (i == 0 && oc->ban == bt) {
 			Lck_Lock(&ban_mtx);
+			VSC_C_main->bans_lurker_tested += tested;
+			VSC_C_main->bans_lurker_tests_tested += tested_tests;
+			VSC_C_main->bans_lurker_obj_killed += lok;
+			VSC_C_main->bans_lurker_obj_killed_cutoff += lokc;
+			tested = tested_tests = lok = lokc = 0;
 			if (oc->ban == bt) {
 				bt->refcount--;
 				VTAILQ_REMOVE(&bt->objcore, oc, ban_list);
@@ -311,14 +321,12 @@ ban_lurker_work(struct worker *wrk, struct vsl_log *vsl)
 	struct banhead_s obans;
 	double d, dt, n;
 	unsigned count = 0, cutoff = UINT_MAX;
-	struct VSC_main *stats;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	stats = wrk->stats;
 
 	dt = 49.62;		// Random, non-magic
 	if (cache_param->ban_lurker_sleep == 0) {
-		(void)ban_cleantail(NULL, stats);
+		(void)ban_cleantail(NULL);
 		return (dt);
 	}
 	if (cache_param->ban_cutoff > 0)
@@ -358,7 +366,7 @@ ban_lurker_work(struct worker *wrk, struct vsl_log *vsl)
 	 * containted the first oban, all obans were on the tail and we're
 	 * done.
 	 */
-	if (ban_cleantail(VTAILQ_FIRST(&obans), stats))
+	if (ban_cleantail(VTAILQ_FIRST(&obans)))
 		return (dt);
 
 	if (VTAILQ_FIRST(&obans) == NULL)
@@ -386,7 +394,7 @@ ban_lurker_work(struct worker *wrk, struct vsl_log *vsl)
 
 	Lck_Lock(&ban_mtx);
 	VTAILQ_FOREACH(b, &obans, l_list) {
-		ban_mark_completed(b, stats);
+		ban_mark_completed(b);
 		if (b == bd)
 			break;
 	}
diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c
index 9d47728c2..58186b2eb 100644
--- a/bin/varnishd/cache/cache_vrt.c
+++ b/bin/varnishd/cache/cache_vrt.c
@@ -669,26 +669,6 @@ VRT_synth_page(VRT_CTX, const char *str, ...)
 
 /*--------------------------------------------------------------------*/
 
-static struct VSC_main *
-vrt_stats(VRT_CTX)
-{
-	struct worker *wrk;
-
-	if (ctx->req) {
-		CHECK_OBJ_NOTNULL(ctx->req, REQ_MAGIC);
-		wrk = ctx->req->wrk;
-	} else if (ctx->bo) {
-		CHECK_OBJ_NOTNULL(ctx->bo, BUSYOBJ_MAGIC);
-		wrk = ctx->bo->wrk;
-	} else {
-		// XXX
-		return (VSC_C_main);
-	}
-
-	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
-	return (wrk->stats);
-}
-
 VCL_VOID
 VRT_ban_string(VRT_CTX, VCL_STRING str)
 {
@@ -744,7 +724,7 @@ VRT_ban_string(VRT_CTX, VCL_STRING str)
 			break;
 		}
 		if (av[++i] == NULL) {
-			err = BAN_Commit(bp, vrt_stats(ctx));
+			err = BAN_Commit(bp);
 			if (err == NULL)
 				bp = NULL;
 			else

From phk at FreeBSD.org  Mon Sep 24 11:26:17 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 24 Sep 2018 11:26:17 +0000 (UTC)
Subject: [master] 0b9657931 Move the wrk+pool summed stats into group 'wrk'.
Message-ID: <20180924112617.E295E6E212@lists.varnish-cache.org>


commit 0b9657931b71546d7ab573f456c3a88e94627e34
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 24 07:46:48 2018 +0000

    Move the wrk+pool summed stats into group 'wrk'.
    
    This eliminates read-add-zero-write on counters outside this group.

diff --git a/bin/varnishd/VSC_main.vsc b/bin/varnishd/VSC_main.vsc
index ffa04f22e..80a486a24 100644
--- a/bin/varnishd/VSC_main.vsc
+++ b/bin/varnishd/VSC_main.vsc
@@ -5,10 +5,11 @@
 .. varnish_vsc_begin::	main
 	:oneliner:	Main counters
 	:order:		10
-	:sumfunction:
+	:sumfunction:	wrk_wrk wrk
 
 .. varnish_vsc:: summs
 	:level:	debug
+	:group: wrk
 	:oneliner:	stat summ operations
 
 	Number of times per-thread statistics were summed into the
@@ -21,16 +22,19 @@
 	How long the child process has been running.
 
 .. varnish_vsc:: sess_conn
+	:group: wrk
 	:oneliner:	Sessions accepted
 
 	Count of sessions successfully accepted
 
 .. varnish_vsc:: sess_drop
+	:group: wrk
 	:oneliner:	Sessions dropped
 
 	Count of sessions silently dropped due to lack of worker thread.
 
 .. varnish_vsc:: sess_fail
+	:group: wrk
 	:oneliner:	Session accept failures
 
 	Count of failures to accept TCP connection.
@@ -39,64 +43,75 @@
 	give more detailed information.
 
 .. varnish_vsc:: sess_fail_econnaborted
+	:group: wrk
 	:oneliner:	Session accept failures: connection aborted
 
 	Detailed reason for sess_fail: Connection aborted by the
 	client, usually harmless.
 
 .. varnish_vsc:: sess_fail_eintr
+	:group: wrk
 	:oneliner:	Session accept failures: interrupted system call
 
 	Detailed reason for sess_fail: The accept() call was
 	interrupted, usually harmless
 
 .. varnish_vsc:: sess_fail_emfile
+	:group: wrk
 	:oneliner:	Session accept failures: too many open files
 
 	Detailed reason for sess_fail: No file descriptor was
 	available. Consider raising RLIMIT_NOFILE (see ulimit -n).
 
 .. varnish_vsc:: sess_fail_ebadf
+	:group: wrk
 	:oneliner:	Session accept failures: bad file descriptor
 
 	Detailed reason for sess_fail: The listen socket file
 	descriptor was invalid. Should never happen.
 
 .. varnish_vsc:: sess_fail_enomem
+	:group: wrk
 	:oneliner:	Session accept failures: not enough memory
 
 	Detailed reason for sess_fail: Most likely insufficient
 	socket buffer memory. Should never happen
 
 .. varnish_vsc:: sess_fail_other
+	:group: wrk
 	:oneliner:	Session accept failures: other
 
 	Detailed reason for sess_fail: neither of the above, see
 	SessError log (varnishlog -g raw -i SessError).
 
 .. varnish_vsc:: client_req_400
+	:group: wrk
 	:oneliner:	Client requests received, subject to 400 errors
 
 	400 means we couldn't make sense of the request, it was malformed
 	in some drastic way.
 
 .. varnish_vsc:: client_req_417
+	:group: wrk
 	:oneliner:	Client requests received, subject to 417 errors
 
 	417 means that something went wrong with an Expect: header.
 
 .. varnish_vsc:: client_req
+	:group: wrk
 	:oneliner:	Good client requests received
 
 	The count of parseable client requests seen.
 
 .. varnish_vsc:: cache_hit
+	:group: wrk
 	:oneliner:	Cache hits
 
 	Count of cache hits.  A cache hit indicates that an object has been
 	delivered to a client without fetching it from a backend server.
 
 .. varnish_vsc:: cache_hit_grace
+	:group: wrk
 	:oneliner:	Cache grace hits
 
 	Count of cache hits with grace. A cache hit with grace is a cache
@@ -104,6 +119,7 @@
 	included in the cache_hit counter.
 
 .. varnish_vsc:: cache_hitpass
+	:group: wrk
 	:oneliner:	Cache hits for pass.
 
 	Count of hits for pass. A cache hit for pass indicates that Varnish
@@ -112,6 +128,7 @@
 	decision is being used.
 
 .. varnish_vsc:: cache_hitmiss
+	:group: wrk
 	:oneliner:	Cache hits for miss.
 
 	Count of hits for miss. A cache hit for miss indicates that Varnish
@@ -120,6 +137,7 @@
 	cached decision is being used.
 
 .. varnish_vsc:: cache_miss
+	:group: wrk
 	:oneliner:	Cache misses
 
 	Count of misses. A cache miss indicates the object was fetched from
@@ -161,56 +179,67 @@
 
 
 .. varnish_vsc:: fetch_head
+	:group: wrk
 	:oneliner:	Fetch no body (HEAD)
 
 	beresp with no body because the request is HEAD.
 
 .. varnish_vsc:: fetch_length
+	:group: wrk
 	:oneliner:	Fetch with Length
 
 	beresp.body with Content-Length.
 
 .. varnish_vsc:: fetch_chunked
+	:group: wrk
 	:oneliner:	Fetch chunked
 
 	beresp.body with Chunked.
 
 .. varnish_vsc:: fetch_eof
+	:group: wrk
 	:oneliner:	Fetch EOF
 
 	beresp.body with EOF.
 
 .. varnish_vsc:: fetch_bad
+	:group: wrk
 	:oneliner:	Fetch bad T-E
 
 	beresp.body length/fetch could not be determined.
 
 .. varnish_vsc:: fetch_none
+	:group: wrk
 	:oneliner:	Fetch no body
 
 	beresp.body empty
 
 .. varnish_vsc:: fetch_1xx
+	:group: wrk
 	:oneliner:	Fetch no body (1xx)
 
 	beresp with no body because of 1XX response.
 
 .. varnish_vsc:: fetch_204
+	:group: wrk
 	:oneliner:	Fetch no body (204)
 
 	beresp with no body because of 204 response.
 
 .. varnish_vsc:: fetch_304
+	:group: wrk
 	:oneliner:	Fetch no body (304)
 
 	beresp with no body because of 304 response.
 
 .. varnish_vsc:: fetch_failed
+	:group: wrk
 	:oneliner:	Fetch failed (all causes)
 
 	beresp fetch failed.
 
 .. varnish_vsc:: fetch_no_thread
+	:group: wrk
 	:oneliner:	Fetch failed (no thread)
 
 	beresp fetch failed, no thread available.
@@ -259,12 +288,14 @@
 	per second. See also parameter thread_queue_limit.
 
 .. varnish_vsc:: busy_sleep
+	:group: wrk
 	:oneliner:	Number of requests sent to sleep on busy objhdr
 
 	Number of requests sent to sleep without a worker thread because
 	they found a busy object.
 
 .. varnish_vsc:: busy_wakeup
+	:group: wrk
 	:oneliner:	Number of requests woken after sleep on busy objhdr
 
 	Number of requests taken off the busy object sleep list and rescheduled.
@@ -295,6 +326,7 @@
 
 .. varnish_vsc:: n_object
 	:type:	gauge
+	:group: wrk
 	:oneliner:	object structs made
 
 	Approximate number of HTTP objects (headers + body, if present) in
@@ -303,12 +335,14 @@
 .. varnish_vsc:: n_vampireobject
 	:type:	gauge
 	:level:	diag
+	:group: wrk
 	:oneliner:	unresurrected objects
 
 	Number of unresurrected objects
 
 .. varnish_vsc:: n_objectcore
 	:type:	gauge
+	:group: wrk
 	:oneliner:	objectcore structs made
 
 	Approximate number of object metadata elements in the cache. Each
@@ -317,6 +351,7 @@
 
 .. varnish_vsc:: n_objecthead
 	:type:	gauge
+	:group: wrk
 	:oneliner:	objecthead structs made
 
 	Approximate number of different hash entries in the cache.
@@ -358,72 +393,83 @@
 
 
 .. varnish_vsc:: s_sess
+	:group: wrk
 	:oneliner:	Total sessions seen
 
 
 .. varnish_vsc:: s_pipe
+	:group: wrk
 	:oneliner:	Total pipe sessions seen
 
 
 .. varnish_vsc:: s_pass
+	:group: wrk
 	:oneliner:	Total pass-ed requests seen
 
 
 .. varnish_vsc:: s_fetch
+	:group: wrk
 	:oneliner:	Total backend fetches initiated
 
 
 .. varnish_vsc:: s_synth
+	:group: wrk
 	:oneliner:	Total synthetic responses made
 
 
 .. varnish_vsc:: s_req_hdrbytes
-	:oneliner:	Request header bytes
 	:format:	bytes
+	:group:		wrk
+	:oneliner:	Request header bytes
 
 	Total request header bytes received
 
 .. varnish_vsc:: s_req_bodybytes
-	:oneliner:	Request body bytes
 	:format:	bytes
+	:group:		wrk
+	:oneliner:	Request body bytes
 
 	Total request body bytes received
 
 .. varnish_vsc:: s_resp_hdrbytes
-	:oneliner:	Response header bytes
 	:format:	bytes
+	:group:		wrk
+	:oneliner:	Response header bytes
 
 	Total response header bytes transmitted
 
 .. varnish_vsc:: s_resp_bodybytes
-	:oneliner:	Response body bytes
 	:format:	bytes
+	:group:		wrk
+	:oneliner:	Response body bytes
 
 	Total response body bytes transmitted
-	:format:	bytes
 
 .. varnish_vsc:: s_pipe_hdrbytes
-	:oneliner:	Pipe request header bytes
 	:format:	bytes
+	:group:		wrk
+	:oneliner:	Pipe request header bytes
 
 	Total request bytes received for piped sessions
 
 .. varnish_vsc:: s_pipe_in
-	:oneliner:	Piped bytes from client
 	:format:	bytes
+	:group:		wrk
+	:oneliner:	Piped bytes from client
 
 	Total number of bytes forwarded from clients in pipe sessions
 
 .. varnish_vsc:: s_pipe_out
-	:oneliner:	Piped bytes to client
 	:format:	bytes
+	:group:		wrk
+	:oneliner:	Piped bytes to client
 
 	Total number of bytes forwarded to clients in pipe sessions
 
 .. varnish_vsc:: sess_closed
+	:group: wrk
 	:oneliner:	Session Closed
 
-
 .. varnish_vsc:: sess_closed_err
 	:oneliner:	Session Closed with error
 
@@ -431,11 +477,12 @@
 	for detailed breakdown
 
 .. varnish_vsc:: sess_readahead
+	:group: wrk
 	:oneliner:	Session Read Ahead
 
-
 .. varnish_vsc:: sess_herd
 	:level:	diag
+	:group: wrk
 	:oneliner:	Session herd
 
 	Number of times the timeout_linger triggered
@@ -589,6 +636,7 @@
 
 
 .. varnish_vsc:: vcl_fail
+	:group: wrk
 	:oneliner:	VCL failures
 
 	Count of failures which prevented VCL from completing.
@@ -754,6 +802,7 @@
 	Number of objects received by expiry thread for handling.
 
 .. varnish_vsc:: hcb_nolock
+	:group: wrk
 	:level:	debug
 	:oneliner:	HCB Lookups without lock
 
diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h
index 093a08b00..e62bc9e8d 100644
--- a/bin/varnishd/cache/cache.h
+++ b/bin/varnishd/cache/cache.h
@@ -86,6 +86,7 @@ enum {
 
 struct VSC_lck;
 struct VSC_main;
+struct VSC_main_wrk;
 struct backend;
 struct ban;
 struct ban_proto;
@@ -246,7 +247,7 @@ struct worker {
 	struct objhead		*nobjhead;
 	struct objcore		*nobjcore;
 	void			*nhashpriv;
-	struct VSC_main		*stats;
+	struct VSC_main_wrk	*stats;
 	struct vsl_log		*vsl;		// borrowed from req/bo
 
 	struct pool_task	task;
diff --git a/bin/varnishd/cache/cache_pool.c b/bin/varnishd/cache/cache_pool.c
index f2bcc23f7..35be8ad00 100644
--- a/bin/varnishd/cache/cache_pool.c
+++ b/bin/varnishd/cache/cache_pool.c
@@ -50,20 +50,12 @@ static VTAILQ_HEAD(,pool)	pools = VTAILQ_HEAD_INITIALIZER(pools);
  * Summing of stats into global stats counters
  */
 
-static void
-pool_sumstat(const struct VSC_main *src)
-{
-
-	Lck_AssertHeld(&wstat_mtx);
-	VSC_main_Summ(VSC_C_main, src);
-}
-
 void
 Pool_Sumstat(const struct worker *wrk)
 {
 
 	Lck_Lock(&wstat_mtx);
-	pool_sumstat(wrk->stats);
+	VSC_main_Summ_wrk(VSC_C_main, wrk->stats);
 	Lck_Unlock(&wstat_mtx);
 	memset(wrk->stats, 0, sizeof *wrk->stats);
 }
@@ -73,7 +65,7 @@ Pool_TrySumstat(const struct worker *wrk)
 {
 	if (Lck_Trylock(&wstat_mtx))
 		return (0);
-	pool_sumstat(wrk->stats);
+	VSC_main_Summ_wrk(VSC_C_main, wrk->stats);
 	Lck_Unlock(&wstat_mtx);
 	memset(wrk->stats, 0, sizeof *wrk->stats);
 	return (1);
@@ -121,14 +113,14 @@ Pool_PurgeStat(unsigned nobj)
 void v_matchproto_(task_func_t)
 pool_stat_summ(struct worker *wrk, void *priv)
 {
-	struct VSC_main *src;
+	struct VSC_main_wrk *src;
 
 	CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 	CHECK_OBJ_NOTNULL(wrk->pool, POOL_MAGIC);
 	AN(priv);
 	src = priv;
 	Lck_Lock(&wstat_mtx);
-	pool_sumstat(src);
+	VSC_main_Summ_wrk(VSC_C_main, src);
 	Lck_Unlock(&wstat_mtx);
 	memset(src, 0, sizeof *src);
 	AZ(wrk->pool->b_stat);
diff --git a/bin/varnishd/cache/cache_pool.h b/bin/varnishd/cache/cache_pool.h
index c954aee8c..e7a6a618b 100644
--- a/bin/varnishd/cache/cache_pool.h
+++ b/bin/varnishd/cache/cache_pool.h
@@ -53,8 +53,8 @@ struct pool {
 	uintmax_t			sdropped;
 	uintmax_t			rdropped;
 	uintmax_t			nqueued;
-	struct VSC_main			*a_stat;
-	struct VSC_main			*b_stat;
+	struct VSC_main_wrk		*a_stat;
+	struct VSC_main_wrk		*b_stat;
 
 	struct mempool			*mpl_req;
 	struct mempool			*mpl_sess;
diff --git a/bin/varnishd/cache/cache_req.c b/bin/varnishd/cache/cache_req.c
index 036056740..991487201 100644
--- a/bin/varnishd/cache/cache_req.c
+++ b/bin/varnishd/cache/cache_req.c
@@ -44,7 +44,7 @@
 #include "vtim.h"
 
 void
-Req_AcctLogCharge(struct VSC_main *ds, struct req *req)
+Req_AcctLogCharge(struct VSC_main_wrk *ds, struct req *req)
 {
 	struct acct_req *a;
 
diff --git a/bin/varnishd/cache/cache_varnishd.h b/bin/varnishd/cache/cache_varnishd.h
index cb1d295a5..0654b9f6b 100644
--- a/bin/varnishd/cache/cache_varnishd.h
+++ b/bin/varnishd/cache/cache_varnishd.h
@@ -318,7 +318,7 @@ void Req_Release(struct req *);
 void Req_Rollback(struct req *req);
 void Req_Cleanup(struct sess *sp, struct worker *wrk, struct req *req);
 void Req_Fail(struct req *req, enum sess_close reason);
-void Req_AcctLogCharge(struct VSC_main *, struct req *);
+void Req_AcctLogCharge(struct VSC_main_wrk *, struct req *);
 
 /* cache_req_body.c */
 int VRB_Ignore(struct req *);
diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c
index 7cd02533d..be98b4e5e 100644
--- a/bin/varnishd/cache/cache_wrk.c
+++ b/bin/varnishd/cache/cache_wrk.c
@@ -61,7 +61,7 @@ wrk_bgthread(void *arg)
 {
 	struct bgthread *bt;
 	struct worker wrk;
-	struct VSC_main ds;
+	struct VSC_main_wrk ds;
 
 	CAST_OBJ_NOTNULL(bt, arg, BGTHREAD_MAGIC);
 	THR_SetName(bt->name);
@@ -98,7 +98,7 @@ WRK_Thread(struct pool *qp, size_t stacksize, unsigned thread_workspace)
 {
 	// child_signal_handler stack overflow check uses struct worker addr
 	struct worker *w, ww;
-	struct VSC_main ds;
+	struct VSC_main_wrk ds;
 	unsigned char ws[thread_workspace];
 
 	AN(qp);
@@ -133,11 +133,11 @@ WRK_Thread(struct pool *qp, size_t stacksize, unsigned thread_workspace)
  */
 
 static void
-pool_addstat(struct VSC_main *dst, struct VSC_main *src)
+pool_addstat(struct VSC_main_wrk *dst, struct VSC_main_wrk *src)
 {
 
 	dst->summs++;
-	VSC_main_Summ(dst, src);
+	VSC_main_Summ_wrk_wrk(dst, src);
 	memset(src, 0, sizeof *src);
 }
 
diff --git a/lib/libvcc/vsctool.py b/lib/libvcc/vsctool.py
index d4e210deb..eb795e076 100644
--- a/lib/libvcc/vsctool.py
+++ b/lib/libvcc/vsctool.py
@@ -93,6 +93,7 @@ class CounterSet(object):
         self.head = m
         self.completed = False
         self.off = 0
+        self.gnames = None
 
     def addmbr(self, m, g):
         '''Add a counter'''
@@ -168,7 +169,13 @@ class CounterSet(object):
 
         fo.write(self.struct + " {\n")
         for i in self.mbrs:
-            fo.write("\tuint64_t\t%s;\n" % i.arg)
+            s = "\tuint64_t\t%s;" % i.arg
+            g = i.param.get("group")
+            if g is not None:
+                while len(s.expandtabs()) < 64:
+                    s += "\t"
+                s += "/* %s */" % g
+            fo.write(s + "\n")
         fo.write("};\n")
         fo.write("\n")
 
@@ -189,14 +196,19 @@ class CounterSet(object):
         fo.write("void VSC_" + self.name + "_Destroy")
         fo.write("(struct vsc_seg **);\n")
 
-        if 'sumfunction' in self.head.param:
-            fo.write("void VSC_" + self.name + "_Summ")
-            fo.write("(" + self.struct + " *, ")
-            fo.write("const " + self.struct + " *);\n")
-            for i in self.gnames:
-                fo.write("void VSC_" + self.name + "_Summ_" + i)
-                fo.write("(" + self.struct + " *, ")
-                fo.write("const " + self.struct + "_" + i + " *);\n")
+        sf = self.head.param.get('sumfunction')
+        if sf is not None:
+            for i in sf.split():
+                j = i.split("_")
+                assert len(j) <= 2
+                if len(j) == 1:
+                    fo.write("void VSC_" + self.name + "_Summ_" + i)
+                    fo.write("(" + self.struct + " *, ")
+                    fo.write("const " + self.struct + "_" + i + " *);\n")
+                else:
+                    fo.write("void VSC_" + self.name + "_Summ_" + i)
+                    fo.write("(" + self.struct + "_" + j[0] + " *, ")
+                    fo.write("const " + self.struct + "_" + j[1] + " *);\n")
 
     def emit_c_paranoia(self, fo):
         '''Emit asserts to make sure compiler gets same byte index'''
@@ -211,27 +223,23 @@ class CounterSet(object):
 
         fo.write("#undef PARANOIA\n")
 
-    def emit_c_sumfunc(self, fo, g=None):
+    def emit_c_sumfunc(self, fo, tgt):
         '''Emit a function summ up countersets'''
         fo.write("\n")
         fo.write("void\n")
         fo.write("VSC_" + self.name + "_Summ")
-        if g is not None:
-            fo.write("_" + g)
-        fo.write("(" + self.struct + " *dst, ")
-        fo.write("const " + self.struct)
-        if g is not None:
-            fo.write("_" + g)
-        fo.write(" *src)\n")
+        fo.write("_" + tgt[0])
+        if len(tgt) > 1:
+            fo.write("_" + tgt[1])
+            fo.write("(" + self.struct + "_" + tgt[1])
+        else:
+            fo.write("(" + self.struct)
+        fo.write(" *dst, const " + self.struct + "_" + tgt[0] + " *src)\n")
         fo.write("{\n")
         fo.write("\n")
         fo.write("\tAN(dst);\n")
         fo.write("\tAN(src);\n")
-        if g:
-            l = self.groups[g]
-        else:
-            l = self.mbrs
-        for i in l:
+        for i in self.groups[tgt[0]]:
             s1 = "\tdst->" + i.arg + " +="
             s2 = "src->" + i.arg + ";"
             if len((s1 + " " + s2).expandtabs()) < 79:
@@ -300,10 +308,10 @@ class CounterSet(object):
         self.emit_json(fo)
         self.emit_c_newfunc(fo)
         self.emit_c_destroyfunc(fo)
-        if 'sumfunction' in self.head.param:
-            self.emit_c_sumfunc(fo)
-            for i in self.gnames:
-                self.emit_c_sumfunc(fo, i)
+        sf = self.head.param.get('sumfunction')
+        if sf is not None:
+            for i in sf.split():
+                self.emit_c_sumfunc(fo, i.split("_"))
 
 #######################################################################
 

From phk at FreeBSD.org  Mon Sep 24 11:26:18 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 24 Sep 2018 11:26:18 +0000 (UTC)
Subject: [master] 9bdc5f75d Implement and test ECMA-48 "REP" sequence.
Message-ID: <20180924112618.246906E21A@lists.varnish-cache.org>


commit 9bdc5f75d661a1659c4df60799612a7524a6caa7
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 24 08:31:37 2018 +0000

    Implement and test ECMA-48 "REP" sequence.
    
    Fixes: #2668

diff --git a/bin/varnishtest/gensequences b/bin/varnishtest/gensequences
index e82e56ec7..4337186b8 100644
--- a/bin/varnishtest/gensequences
+++ b/bin/varnishtest/gensequences
@@ -158,6 +158,7 @@ for (p in l_prefix_name) {
 
 	if (l_prefix_name[p] != "teken_state_init") {
 		print "";
+		print "\tt->t_last = 0;";
 		print "\tteken_state_switch(t, teken_state_init);";
 	}
 	print "}";
diff --git a/bin/varnishtest/sequences b/bin/varnishtest/sequences
index d8f30306b..50f589626 100644
--- a/bin/varnishtest/sequences
+++ b/bin/varnishtest/sequences
@@ -114,3 +114,6 @@ C25VTSW		Cons25 switch virtual terminal		^[ [ z		r
 
 # VT52 compatibility
 #DECID		VT52 DECID				^[ Z
+
+# ECMA-48
+REP		Repeat last graphic char		^[ [ b		n
diff --git a/bin/varnishtest/teken.h b/bin/varnishtest/teken.h
index eb59817d7..c08a2bbb8 100644
--- a/bin/varnishtest/teken.h
+++ b/bin/varnishtest/teken.h
@@ -157,6 +157,7 @@ struct __teken {
 
 	unsigned int	 t_utf8_left;
 	teken_char_t	 t_utf8_partial;
+	teken_char_t	 t_last;
 
 	unsigned int	 t_curscs;
 	teken_scs_t	*t_saved_curscs;
diff --git a/bin/varnishtest/teken_subr.h b/bin/varnishtest/teken_subr.h
index 22d06bb19..644e502f6 100644
--- a/bin/varnishtest/teken_subr.h
+++ b/bin/varnishtest/teken_subr.h
@@ -798,10 +798,11 @@ teken_subr_primary_device_attributes(const teken_t *t, unsigned int request)
 }
 
 static void
-teken_subr_do_putchar(const teken_t *t, const teken_pos_t *tp, teken_char_t c,
+teken_subr_do_putchar(teken_t *t, const teken_pos_t *tp, teken_char_t c,
     int width)
 {
 
+	t->t_last = c;
 	if (t->t_stateflags & TS_INSERT &&
 	    tp->tp_col < t->t_winsize.tp_col - width) {
 		teken_rect_t ctr;
@@ -1334,3 +1335,12 @@ teken_subr_vertical_position_absolute(teken_t *t, unsigned int row)
 	t->t_stateflags &= ~TS_WRAPPED;
 	teken_funcs_cursor(t);
 }
+
+static void
+teken_subr_repeat_last_graphic_char(teken_t *t, unsigned int rpts)
+{
+
+	for (; t->t_last != 0 && rpts > 0; rpts--)
+		teken_subr_regular_character(t, t->t_last);
+}
+
diff --git a/bin/varnishtest/tests/a00001.vtc b/bin/varnishtest/tests/a00001.vtc
index ea5b458f3..3647e01e3 100644
--- a/bin/varnishtest/tests/a00001.vtc
+++ b/bin/varnishtest/tests/a00001.vtc
@@ -204,6 +204,27 @@ process p4 -writehex 0d
 process p4 -expect-text 21 11 "Enter choice number (0 - 12):"
 process p4 -screen_dump
 
+# 11. Test non-VT100 (e.g., VT220, XTERM) terminals
+process p4 -writehex "31 31 0d"
+process p4 -expect-text 0 0 "Menu 11: Non-VT100 Tests"
+
+process p4 -writehex "37 0d"
+process p4 -expect-text 0 0 "Menu 11.7: Miscellaneous ISO-6429 (ECMA-48) Tests"
+
+process p4 -writehex "32 0d"
+process p4 -expect-text 0 0 "Push <RETURN>"
+process p4 -screen_dump
+process p4 -expect-text 20 1 "Test Repeat (REP)"
+process p4 -expect-text 1 1 " ++ "
+process p4 -expect-text 2 2 " ++ "
+process p4 -expect-text 17 17 " ++ "
+process p4 -expect-text 18 18 "*++*"
+process p4 -writehex "0d"
+process p4 -expect-text 0 0 "Menu 11.7: Miscellaneous ISO-6429 (ECMA-48) Tests"
+process p4 -writehex "30 0d"
+process p4 -expect-text 0 0 "Menu 11: Non-VT100 Tests"
+process p4 -writehex "30 0d"
+
 # 0. Exit
 process p4 -writehex "30 0d"
 process p4 -expect-text 12 30 "That's all, folks!"

From phk at FreeBSD.org  Mon Sep 24 11:43:07 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Mon, 24 Sep 2018 11:43:07 +0000 (UTC)
Subject: [master] 156bb87cf Be Sun-c compiler compatible
Message-ID: <20180924114307.ACA999274C@lists.varnish-cache.org>


commit 156bb87cfcd8248528d251ade29aa00d10330b46
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Sep 24 11:41:42 2018 +0000

    Be Sun-c compiler compatible

diff --git a/bin/varnishtest/teken.c b/bin/varnishtest/teken.c
index 21eb71752..48df7e066 100644
--- a/bin/varnishtest/teken.c
+++ b/bin/varnishtest/teken.c
@@ -41,7 +41,7 @@
 #include "vas.h"
 
 /* debug messages */
-#define	teken_printf(x,...)
+#define	teken_printf(...)
 
 /* Private flags for t_stateflags. */
 #define	TS_FIRSTDIGIT	0x0001	/* First numeric digit in escape sequence. */

From daghf at varnish-software.com  Mon Sep 24 13:00:15 2018
From: daghf at varnish-software.com (Dag Haavi Finstad)
Date: Mon, 24 Sep 2018 13:00:15 +0000 (UTC)
Subject: [master] aa2dd82ab Fix libvgz build error on gcc7
Message-ID: <20180924130015.BE25995CC1@lists.varnish-cache.org>


commit aa2dd82abe9441396080f464cf4ad54f9b7ca523
Author: Dag Haavi Finstad <daghf at varnish-software.com>
Date:   Mon Sep 24 13:27:36 2018 +0200

    Fix libvgz build error on gcc7
    
    This adds -Wno-implicit-fallthrough for libvgz to satisfy gcc7.
    
    The -Wno-unknown-warning-option is there to satisfy older gcc/clang that
    don't recognize -Wno-implicit-fallthrough.

diff --git a/configure.ac b/configure.ac
index 7cbbe53be..1f9a2f8b2 100644
--- a/configure.ac
+++ b/configure.ac
@@ -261,6 +261,11 @@ if test "$ac_cv_have_viz" = no; then
 fi
 CFLAGS="${save_CFLAGS}"
 
+if test "x$GCC" = "xyes"; then
+	libvgz_extra_cflags="${libvgz_extra_cflags} -Wno-unknown-warning-option -Wno-implicit-fallthrough"
+	AC_SUBST(libvgz_extra_cflags)
+fi
+
 SAN_CFLAGS=
 SAN_LDFLAGS=
 UBSAN_CFLAGS=

From fgsch at lodoss.net  Mon Sep 24 22:28:10 2018
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 24 Sep 2018 22:28:10 +0000 (UTC)
Subject: [master] bf730ab1a Correct output
Message-ID: <20180924222810.BB19DB178E@lists.varnish-cache.org>


commit bf730ab1a788b9414ee5f0236718de7c92f2c2b4
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Mon Sep 24 23:27:04 2018 +0100

    Correct output

diff --git a/bin/varnishtest/vtc_misc.c b/bin/varnishtest/vtc_misc.c
index 0c47fe88a..b015f3f8f 100644
--- a/bin/varnishtest/vtc_misc.c
+++ b/bin/varnishtest/vtc_misc.c
@@ -129,9 +129,10 @@ cmd_varnishtest(CMD_ARGS)
  * \-match REGEXP
  *	Expect regexp to match the stdout+err output.
  */
-/* SECTION: client-server.spec.shell shell
+/* SECTION: client-server.spec.shell
  *
- * Same as for the top-level shell.
+ * shell
+ *	Same as for the top-level shell.
  */
 
 static void

From fgsch at lodoss.net  Mon Sep 24 22:28:10 2018
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 24 Sep 2018 22:28:10 +0000 (UTC)
Subject: [master] 5f74f5ff5 Spelling
Message-ID: <20180924222810.B3D1DB178C@lists.varnish-cache.org>


commit 5f74f5ff565cad3b8671470979eb8e39f50baeff
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Mon Sep 24 23:25:46 2018 +0100

    Spelling

diff --git a/bin/varnishtest/vtc_http2.c b/bin/varnishtest/vtc_http2.c
index 764710f35..404b68da0 100644
--- a/bin/varnishtest/vtc_http2.c
+++ b/bin/varnishtest/vtc_http2.c
@@ -1213,7 +1213,7 @@ cmd_var_resolve(const struct stream *s, const char *spec, char *buf)
 /* SECTION: stream.spec.frame_sendhex sendhex
  *
  * Push bytes directly on the wire. sendhex takes exactly one argument: a string
- * describing the bytes, in hex notation, will possible whitespaces between
+ * describing the bytes, in hex notation, with possible whitespaces between
  * them. Here's an example::
  *
  *	sendhex "00 00 08 00 0900	8d"
diff --git a/lib/libvmod_std/vmod_std.c b/lib/libvmod_std/vmod_std.c
index 03a39453e..d35d89f6c 100644
--- a/lib/libvmod_std/vmod_std.c
+++ b/lib/libvmod_std/vmod_std.c
@@ -298,7 +298,7 @@ vmod_syntax(VRT_CTX, VCL_REAL r)
 	assert(ctx->syntax == 40 || ctx->syntax == 41);
 	/*
 	 * We need to be careful because non-integer numbers have imprecise
-	 * IEE754 represenation (4.1 is 0x1.0666666666666p+2 = 4.09999...)
+	 * IEE754 representation (4.1 is 0x1.0666666666666p+2 = 4.09999...)
 	 * By scaling up and rounding, this is taken care of.
 	 */
 	return (round(r * 10) <= ctx->syntax);

From fgsch at lodoss.net  Mon Sep 24 22:28:10 2018
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 24 Sep 2018 22:28:10 +0000 (UTC)
Subject: [master] 91d1cd7d5 No need to mark the overflow twice
Message-ID: <20180924222810.D9895B1791@lists.varnish-cache.org>


commit 91d1cd7d55c9d2c1409c58458e932d560f0c4c98
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Mon Sep 24 23:27:29 2018 +0100

    No need to mark the overflow twice

diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c
index 8e41d37b1..2e61ca7cf 100644
--- a/bin/varnishd/cache/cache_http.c
+++ b/bin/varnishd/cache/cache_http.c
@@ -379,7 +379,6 @@ http_CollectHdrSep(struct http *hp, const char *hdr, const char *sep)
 			if (b + x >= e) {
 				http_fail(hp);
 				VSLb(hp->vsl, SLT_LostHeader, "%s", hdr + 1);
-				WS_MarkOverflow(hp->ws);
 				WS_Release(hp->ws, 0);
 				return;
 			}

From fgsch at lodoss.net  Mon Sep 24 22:48:08 2018
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 24 Sep 2018 22:48:08 +0000 (UTC)
Subject: [master] 562b5bc78 More spelling and consistency
Message-ID: <20180924224808.85C0D4167@lists.varnish-cache.org>


commit 562b5bc78ddf3a8f0bb28713098369d46ff45a49
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Mon Sep 24 23:32:41 2018 +0100

    More spelling and consistency

diff --git a/bin/varnishtest/vtc_misc.c b/bin/varnishtest/vtc_misc.c
index b015f3f8f..650b659e7 100644
--- a/bin/varnishtest/vtc_misc.c
+++ b/bin/varnishtest/vtc_misc.c
@@ -101,7 +101,7 @@ cmd_varnishtest(CMD_ARGS)
 /* SECTION: shell shell
  *
  * Pass the string given as argument to a shell. If you have multiple
- * commands to run, you can use curly barces to describe a multi-lines
+ * commands to run, you can use curly brackets to describe a multi-lines
  * script, eg::
  *
  *         shell {
diff --git a/doc/sphinx/users-guide/vcl-syntax.rst b/doc/sphinx/users-guide/vcl-syntax.rst
index f4bc219da..4ab5d7f74 100644
--- a/doc/sphinx/users-guide/vcl-syntax.rst
+++ b/doc/sphinx/users-guide/vcl-syntax.rst
@@ -3,7 +3,7 @@ VCL Syntax
 
 VCL has inherited a lot from C and it reads much like simple C or Perl.
 
-Blocks are delimited by curly braces, statements end with semicolons,
+Blocks are delimited by curly brackets, statements end with semicolons,
 and comments may be written as in C, C++ or Perl according to your own
 preferences.
 

From fgsch at lodoss.net  Mon Sep 24 22:48:08 2018
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Mon, 24 Sep 2018 22:48:08 +0000 (UTC)
Subject: [master] 4e8926cc5 Always report the ws id in lowercase
Message-ID: <20180924224808.A17164169@lists.varnish-cache.org>


commit 4e8926cc58e0376e7ee4115f9091dacd595b98c0
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Mon Sep 24 23:44:13 2018 +0100

    Always report the ws id in lowercase

diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c
index 2e61ca7cf..30988ab56 100644
--- a/bin/varnishd/cache/cache_http.c
+++ b/bin/varnishd/cache/cache_http.c
@@ -102,6 +102,7 @@ http_fail(const struct http *hp)
 {
 
 	VSC_C_main->losthdr++;
+	hp->ws->id[0] |= 0x20;		// cheesy tolower()
 	VSLb(hp->vsl, SLT_Error, "out of workspace (%s)", hp->ws->id);
 	WS_MarkOverflow(hp->ws);
 }
diff --git a/bin/varnishtest/tests/r01739.vtc b/bin/varnishtest/tests/r01739.vtc
index 7b8d6565d..843ea6933 100644
--- a/bin/varnishtest/tests/r01739.vtc
+++ b/bin/varnishtest/tests/r01739.vtc
@@ -17,7 +17,7 @@ varnish v1 -vcl+backend {
 
 logexpect l1 -v v1 -g raw {
 	expect * 1002 FetchError {^Workspace overflow}
-	expect * =    Error      {^out of workspace [(]Bo[)]}
+	expect * =    Error      {^out of workspace [(]bo[)]}
 } -start
 
 client c1 {
diff --git a/bin/varnishtest/tests/r01990.vtc b/bin/varnishtest/tests/r01990.vtc
index 5ee40bda5..d86f61c55 100644
--- a/bin/varnishtest/tests/r01990.vtc
+++ b/bin/varnishtest/tests/r01990.vtc
@@ -26,9 +26,9 @@ logexpect l1 -v v1 -g raw {
 	expect * 1002 FetchError   {^out of workspace}
 	expect * =    BerespStatus {^503}
 	expect * =    BerespReason {^Backend fetch failed}
-	expect * =    Error        {^out of workspace [(]Bo[)]}
+	expect * =    Error        {^out of workspace [(]bo[)]}
 	expect * =    LostHeader   {^Date:}
-	expect * =    Error        {^out of workspace [(]Bo[)]}
+	expect * =    Error        {^out of workspace [(]bo[)]}
 	expect * =    LostHeader   {^299}
 } -start
 

From geoff at uplex.de  Tue Sep 25 08:12:12 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Tue, 25 Sep 2018 08:12:12 +0000 (UTC)
Subject: [master] 97de384b2 Document the 'changed' parameter for param.show.
Message-ID: <20180925081213.00DC565214@lists.varnish-cache.org>


commit 97de384b2f8209d1766f57828d8ad1631f3fe51e
Author: Geoff Simmons <geoff at uplex.de>
Date:   Tue Sep 25 10:10:39 2018 +0200

    Document the 'changed' parameter for param.show.

diff --git a/include/tbl/cli_cmds.h b/include/tbl/cli_cmds.h
index d162bada5..7b3d1de19 100644
--- a/include/tbl/cli_cmds.h
+++ b/include/tbl/cli_cmds.h
@@ -134,9 +134,14 @@ CLI_CMD(VCL_LABEL,
 
 CLI_CMD(PARAM_SHOW,
 	"param.show",
-	"param.show [-l] [<param>]",
+	"param.show [-l] [<param>|changed]",
 	"Show parameters and their values.",
-	"",
+
+	"The long form with ``-l`` shows additional information, including"
+	" documentation and minimum, maximum and default values, if defined"
+	" for the parameter. If a parameter is specified with ``<param>``,"
+	" show only that parameter. If ``changed`` is specified, show only"
+	" those parameters whose values differ from their defaults.",
 	0, 2
 )
 

From geoff at uplex.de  Tue Sep 25 14:35:14 2018
From: geoff at uplex.de (Geoff Simmons)
Date: Tue, 25 Sep 2018 14:35:14 +0000 (UTC)
Subject: [master] b2f5cf3c1 Start skeleton release notes for the next version.
Message-ID: <20180925143514.2354AA12DB@lists.varnish-cache.org>


commit b2f5cf3c1d75f5ab72df40de87c0ddbab6639a35
Author: Geoff Simmons <geoff at uplex.de>
Date:   Tue Sep 25 16:31:17 2018 +0200

    Start skeleton release notes for the next version.
    
    Restructured so that:
    
    * 'Upgrading' is limited to work that has to be done to upgrade from
      a current deployment to the new version.
    
    * 'Changes' is a comprehensive, user-level description of changes and
      new features.

diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst
new file mode 100644
index 000000000..fbda9d6b9
--- /dev/null
+++ b/doc/sphinx/whats-new/changes-trunk.rst
@@ -0,0 +1,73 @@
+**Note: This is a working document for a future release, with running
+updates for changes in the development branch. For changes in the
+released versions of Varnish, see:** :ref:`whats-new-index`
+
+.. _whatsnew_changes_CURRENT:
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+Changes in Varnish **$NEXT_RELEASE**
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+For information about updating your current Varnish deployment to the
+new version, see :ref:`whatsnew_upgrading_CURRENT`.
+
+A more detailed and technical account of changes in Varnish, with
+links to issues that have been fixed and pull requests that have been
+merged, may be found in the `change log`_.
+
+.. _change log: https://github.com/varnishcache/varnish-cache/blob/master/doc/changes.rst
+
+varnishd
+========
+
+Parameters
+~~~~~~~~~~
+
+**XXX changes in -p parameters**
+
+Other changes in varnishd
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Changes to VCL
+==============
+
+VCL variables
+~~~~~~~~~~~~~
+
+**XXX new, deprecated or removed variables, or changed semantics**
+
+Other changes to VCL
+~~~~~~~~~~~~~~~~~~~~
+
+VMODs
+=====
+
+**XXX changes in the bundled VMODs**
+
+varnishlog
+==========
+
+**XXX changes concerning varnishlog(1) and/or vsl(7)**
+
+varnishadm
+==========
+
+**XXX changes concerning varnishadm(1) and/or varnish-cli(7)**
+
+varnishstat
+===========
+
+**XXX changes concerning varnishstat(1) and/or varnish-counters(7)**
+
+varnishtest
+===========
+
+**XXX changes concerning varnishtest(1) and/or vtc(7)**
+
+Changes for developers and VMOD authors
+=======================================
+
+**XXX changes concerning VRT, the public APIs, source code organization,
+builds etc.**
+
+*eof*
diff --git a/doc/sphinx/whats-new/index.rst b/doc/sphinx/whats-new/index.rst
index f7b76016b..f98c16e40 100644
--- a/doc/sphinx/whats-new/index.rst
+++ b/doc/sphinx/whats-new/index.rst
@@ -8,6 +8,19 @@ This section describes the changes and improvements between different
 versions of Varnish, and what upgrading between the different versions
 entail.
 
+Varnish **$NEXT_RELEASE**
+-------------------------
+
+**Note: These are working documents for a future release, with running
+updates for changes in the development branch. For changes in the
+released versions of Varnish, see the chapters listed below.**
+
+.. toctree::
+   :maxdepth: 2
+
+   changes-trunk
+   upgrading-trunk
+
 Varnish 6.1
 -----------
 
diff --git a/doc/sphinx/whats-new/upgrading-trunk.rst b/doc/sphinx/whats-new/upgrading-trunk.rst
new file mode 100644
index 000000000..6143fde99
--- /dev/null
+++ b/doc/sphinx/whats-new/upgrading-trunk.rst
@@ -0,0 +1,33 @@
+**Note: This is a working document for a future release, with running
+updates for changes in the development branch. For changes in the
+released versions of Varnish, see:** :ref:`whats-new-index`
+
+.. _whatsnew_upgrading_CURRENT:
+
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+Upgrading to Varnish **$NEXT_RELEASE**
+%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
+
+**XXX: how to upgrade from previous deployments to this
+version. Limited to work that has to be done for an upgrade, new
+features are listed in "Changes". Explicitly mention what does *not*
+have to be changed, especially in VCL. May include, but is not limited
+to:**
+
+* Elements of VCL that have been removed or are deprecated, or whose
+  semantics have changed.
+
+* -p parameters that have been removed or are deprecated, or whose
+  semantics have changed.
+
+* Changes in the CLI.
+
+* Changes in the output or interpretation of stats or the log, including
+  changes affecting varnishncsa/-hist/-top.
+
+* Changes that may be necessary in VTCs or in the use of varnishtest.
+
+* Changes in public APIs that may require changes in VMODs or VAPI/VUT
+  clients.
+
+*eof*

From phk at FreeBSD.org  Tue Sep 25 19:44:11 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 25 Sep 2018 19:44:11 +0000 (UTC)
Subject: [master] aba1633b1 Make varnishtest (more) agnostic about its own
 name.
Message-ID: <20180925194411.19B77A97C9@lists.varnish-cache.org>


commit aba1633b128d84bdc053ea1fefbd909e88155ea4
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Sep 25 09:37:01 2018 +0000

    Make varnishtest (more) agnostic about its own name.

diff --git a/bin/varnishtest/tests/a00000.vtc b/bin/varnishtest/tests/a00000.vtc
index 8fb9ded32..3f012b105 100644
--- a/bin/varnishtest/tests/a00000.vtc
+++ b/bin/varnishtest/tests/a00000.vtc
@@ -10,7 +10,7 @@ shell {
 	echo 'shell "exit 9"' >> _.vtc
 }
 
-shell -exit 2 -expect {doesn't start with 'vtest' or 'varnishtest'} {
+shell -exit 2 -expect {doesn't start with 'vtest' or 'varnishtes} {
 	varnishtest -v _.vtc
 }
 
diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c
index cb0c2f07f..a06f6f08e 100644
--- a/bin/varnishtest/vtc_main.c
+++ b/bin/varnishtest/vtc_main.c
@@ -52,6 +52,8 @@
 #include "vtim.h"
 #include "vct.h"
 
+static const char *argv0;
+
 struct vtc_tst {
 	unsigned		magic;
 #define TST_MAGIC		0x618d8b88
@@ -124,7 +126,7 @@ parse_D_opt(char *arg)
 static void
 usage(void)
 {
-	fprintf(stderr, "usage: varnishtest [options] file ...\n");
+	fprintf(stderr, "usage: %s [options] file ...\n", argv0);
 #define FMT "    %-28s # %s\n"
 	fprintf(stderr, FMT, "-b size",
 	    "Set internal buffer size (default: 1M)");
@@ -577,6 +579,12 @@ main(int argc, char * const *argv)
 	uintmax_t bufsiz;
 	const char *p;
 
+	argv0 = strrchr(argv[0], '/');
+	if (argv0 == NULL)
+		argv0 = argv[0];
+	else
+		argv0++;
+
 	if (getenv("TMPDIR") != NULL)
 		tmppath = strdup(getenv("TMPDIR"));
 	else

From phk at FreeBSD.org  Tue Sep 25 19:44:11 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 25 Sep 2018 19:44:11 +0000 (UTC)
Subject: [master] 252ce718e More 'vtest' vs. 'varnishtest' agnosticism.
Message-ID: <20180925194411.3907EA97CC@lists.varnish-cache.org>


commit 252ce718eda3093ce268a48385b459b5a66e80c1
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Sep 25 10:05:07 2018 +0000

    More 'vtest' vs. 'varnishtest' agnosticism.

diff --git a/bin/varnishtest/huffman_gen.py b/bin/varnishtest/huffman_gen.py
index b89bbb43e..ab6b30e87 100755
--- a/bin/varnishtest/huffman_gen.py
+++ b/bin/varnishtest/huffman_gen.py
@@ -58,7 +58,7 @@ s = sym(63, 6, 0)
 tbls[0xffffff][63] = s
 
 print('''/* NB:  This file is machine generated, DO NOT EDIT!
- * edit bin/varnishtest/huffman_input instead
+ * edit 'huffman_input' instead
  */
 
 struct stbl;
diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c
index 952563b9f..6c4fcc8da 100644
--- a/bin/varnishtest/vtc_http.c
+++ b/bin/varnishtest/vtc_http.c
@@ -76,9 +76,9 @@ extern const struct cmds http_cmds[];
  *
  * Be careful though, servers will by default listen to the 127.0.0.1 IP and
  * will pick a random port, and publish 3 macros: sNAME_addr, sNAME_port and
- * sNAME_sock, but only once they are started. For varnishtest to
- * create the vcl with the correct values, the server must be started when you
- * use -vcl+backend.
+ * sNAME_sock, but only once they are started.
+ * For 'varnish -vcl+backend' to create the vcl with the correct values, the
+ * server must be started first.
  *
  * SECTION: client-server.args Arguments
  *
@@ -1452,7 +1452,7 @@ cmd_http_chunked(CMD_ARGS)
 /* SECTION: client-server.spec.chunkedlen
  *
  * chunkedlen NUMBER
- *         Do as ``chunked`` except that varnishtest will generate the string
+ *         Do as ``chunked`` except that the string will be generated
  *         for you, with a length of NUMBER characters.
  */
 
diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c
index a06f6f08e..4e5cf6b2b 100644
--- a/bin/varnishtest/vtc_main.c
+++ b/bin/varnishtest/vtc_main.c
@@ -597,7 +597,9 @@ main(int argc, char * const *argv)
 
 	params_vsb = VSB_new_auto();
 	AN(params_vsb);
-	p = getenv("VARNISHTEST_DURATION");
+	p = getenv("VTEST_DURATION");
+	if (p == NULL)
+		p = getenv("VARNISHTEST_DURATION");
 	if (p != NULL)
 		vtc_maxdur = atoi(p);
 
diff --git a/bin/varnishtest/vtc_misc.c b/bin/varnishtest/vtc_misc.c
index 650b659e7..f04003da7 100644
--- a/bin/varnishtest/vtc_misc.c
+++ b/bin/varnishtest/vtc_misc.c
@@ -112,8 +112,8 @@ cmd_varnishtest(CMD_ARGS)
  *
  * By default a zero exit code is expected, otherwise the vtc will fail.
  *
- * Notice that the commandstring is prefixed with "exec 2>&1;" to join
- * stderr and stdout back to the varnishtest process.
+ * Notice that the commandstring is prefixed with "exec 2>&1;" to combine
+ * stderr and stdout back to the test process.
  *
  * Optional arguments:
  *
@@ -352,9 +352,9 @@ cmd_delay(CMD_ARGS)
  * dns
  *        DNS lookups are working
  * topbuild
- *        varnishtest has been started with '-i'
+ *        The test has been started with '-i'
  * root
- *        varnishtest has been invoked by the root user
+ *        The test has been invoked by the root user
  * user_varnish
  *        The varnish user is present
  * user_vcache
diff --git a/bin/varnishtest/vtc_process.c b/bin/varnishtest/vtc_process.c
index be943cfaf..10d545b38 100644
--- a/bin/varnishtest/vtc_process.c
+++ b/bin/varnishtest/vtc_process.c
@@ -856,7 +856,7 @@ process_close(struct process *p)
  *	Shorthand for -start -wait.
  *
  *	In most cases, if you just want to start a process and wait for it
- *	to finish, you can use the varnishtest ``shell`` command instead.
+ *	to finish, you can use the ``shell`` command instead.
  *	The following commands are equivalent::
  *
  *	    shell "do --something"

From phk at FreeBSD.org  Tue Sep 25 20:05:07 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 25 Sep 2018 20:05:07 +0000 (UTC)
Subject: [master] acb336442 Add missing copyright information
Message-ID: <20180925200508.12426AC069@lists.varnish-cache.org>


commit acb336442c7d22c62e2c9a669fe602d325356b69
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Sep 25 19:54:50 2018 +0000

    Add missing copyright information

diff --git a/bin/varnishtest/hpack.h b/bin/varnishtest/hpack.h
index 94b734715..a443ee4af 100644
--- a/bin/varnishtest/hpack.h
+++ b/bin/varnishtest/hpack.h
@@ -1,3 +1,31 @@
+/*-
+ * Copyright (c) 2008-2016 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Guillaume Quintard <guillaume.quintard at gmail.com>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
 #include <stdint.h>
 
 enum hpk_result{
diff --git a/bin/varnishtest/huffman_input b/bin/varnishtest/huffman_input
index bfde068cd..117f87f06 100644
--- a/bin/varnishtest/huffman_input
+++ b/bin/varnishtest/huffman_input
@@ -1,3 +1,4 @@
+# For Copyright information see RFC7541 [BSD3]
        (  0)  |11111111|11000                             1ff8  [13]
        (  1)  |11111111|11111111|1011000                7fffd8  [23]
        (  2)  |11111111|11111111|11111110|0010         fffffe2  [28]
diff --git a/bin/varnishtest/vtc_h2_enctbl.h b/bin/varnishtest/vtc_h2_enctbl.h
index 5e69f3303..186a293d9 100644
--- a/bin/varnishtest/vtc_h2_enctbl.h
+++ b/bin/varnishtest/vtc_h2_enctbl.h
@@ -1,3 +1,7 @@
+/*-
+ * For Copyright information see RFC7541 [BSD3]
+ */
+
 HPACK(0,    0x1ff8,      13)
 HPACK(1,    0x7fffd8,    23)
 HPACK(2,    0xfffffe2,   28)
diff --git a/bin/varnishtest/vtc_h2_stattbl.h b/bin/varnishtest/vtc_h2_stattbl.h
index 1e5486245..e845cf85b 100644
--- a/bin/varnishtest/vtc_h2_stattbl.h
+++ b/bin/varnishtest/vtc_h2_stattbl.h
@@ -1,3 +1,6 @@
+/*-
+ * For Copyright information see RFC7541 [BSD3]
+ */
 STAT_HDRS(1,   ":authority",                   "")
 STAT_HDRS(2,   ":method",                      "GET")
 STAT_HDRS(3,   ":method",                      "POST")
diff --git a/bin/varnishtest/vtc_http.h b/bin/varnishtest/vtc_http.h
index a6c5a95f4..5cd079890 100644
--- a/bin/varnishtest/vtc_http.h
+++ b/bin/varnishtest/vtc_http.h
@@ -1,3 +1,31 @@
+/*-
+ * Copyright (c) 2008-2015 Varnish Software AS
+ * All rights reserved.
+ *
+ * Author: Poul-Henning Kamp <phk at phk.freebsd.dk>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
 #define MAX_HDR		50
 
 struct http {

From phk at FreeBSD.org  Tue Sep 25 20:14:07 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Tue, 25 Sep 2018 20:14:07 +0000 (UTC)
Subject: [master] 7161dcf9d Work around autocrappery screwing up argv[0]
Message-ID: <20180925201407.B0AEFAC3F2@lists.varnish-cache.org>


commit 7161dcf9d9f5380ffd0483ee9a163270ac04f1df
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Tue Sep 25 20:12:44 2018 +0000

    Work around autocrappery screwing up argv[0]

diff --git a/bin/varnishtest/tests/a00000.vtc b/bin/varnishtest/tests/a00000.vtc
index 3f012b105..2d3604f24 100644
--- a/bin/varnishtest/tests/a00000.vtc
+++ b/bin/varnishtest/tests/a00000.vtc
@@ -1,6 +1,6 @@
 varnishtest "Test varnishtest itself"
 
-shell -exit 1 -expect {usage: varnishtest} {varnishtest -h}
+shell -exit 1 -expect {varnishtest [options]} {varnishtest -h}
 
 shell -exit 1 -match {-D.*Define macro} {varnishtest -h}
 

From nils.goroll at uplex.de  Wed Sep 26 06:03:40 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 26 Sep 2018 08:03:40 +0200
Subject: [master] 83b418220 A train-trip worth of python3 migration and
 other polishing
In-Reply-To: <20180924112617.274A76E1DB@lists.varnish-cache.org>
References: <20180924112617.274A76E1DB@lists.varnish-cache.org>
Message-ID: <1f6cba19-6d92-c771-eae3-afae099f5e80@uplex.de>

phk,

now that it seems you've done the ground work, putting back the wisdom about
apple pork into the debug vmod vcc file, it appears to me that VCC UTF-8 now
works with python3, but not 2.7

Is removing python2 support the way forward or can we get support for unicode
across both releases? I am too little a snake handler to feel confident about
hacking this myself.

Nils

-- 

** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstra?e 32
22301 Hamburg

tel +49 40 28805731
mob +49 170 2723133
fax +49 40 42949753

xmpp://slink at jabber.int.uplex.de/

http://uplex.de/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-commit/attachments/20180926/91250e76/attachment.bin>

From nils.goroll at uplex.de  Wed Sep 26 05:42:59 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 26 Sep 2018 07:42:59 +0200
Subject: [master] 4e8926cc5 Always report the ws id in lowercase
In-Reply-To: <20180924224808.A17164169@lists.varnish-cache.org>
References: <20180924224808.A17164169@lists.varnish-cache.org>
Message-ID: <8cb51acb-fa89-20df-7067-2195de4045c4@uplex.de>

Hi Federico,

can you explain why this is a good idea? Isn't the overflow maker relevant in
panics?

Nils

On 25/09/2018 00:48, Federico G. Schwindt wrote:
> 
> commit 4e8926cc58e0376e7ee4115f9091dacd595b98c0
> Author: Federico G. Schwindt <fgsch at lodoss.net>
> Date:   Mon Sep 24 23:44:13 2018 +0100
> 
>     Always report the ws id in lowercase
> 
> diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c
> index 2e61ca7cf..30988ab56 100644
> --- a/bin/varnishd/cache/cache_http.c
> +++ b/bin/varnishd/cache/cache_http.c
> @@ -102,6 +102,7 @@ http_fail(const struct http *hp)
>  {
>  
>  	VSC_C_main->losthdr++;
> +	hp->ws->id[0] |= 0x20;		// cheesy tolower()
>  	VSLb(hp->vsl, SLT_Error, "out of workspace (%s)", hp->ws->id);
>  	WS_MarkOverflow(hp->ws);
>  }
> diff --git a/bin/varnishtest/tests/r01739.vtc b/bin/varnishtest/tests/r01739.vtc
> index 7b8d6565d..843ea6933 100644
> --- a/bin/varnishtest/tests/r01739.vtc
> +++ b/bin/varnishtest/tests/r01739.vtc
> @@ -17,7 +17,7 @@ varnish v1 -vcl+backend {
>  
>  logexpect l1 -v v1 -g raw {
>  	expect * 1002 FetchError {^Workspace overflow}
> -	expect * =    Error      {^out of workspace [(]Bo[)]}
> +	expect * =    Error      {^out of workspace [(]bo[)]}
>  } -start
>  
>  client c1 {
> diff --git a/bin/varnishtest/tests/r01990.vtc b/bin/varnishtest/tests/r01990.vtc
> index 5ee40bda5..d86f61c55 100644
> --- a/bin/varnishtest/tests/r01990.vtc
> +++ b/bin/varnishtest/tests/r01990.vtc
> @@ -26,9 +26,9 @@ logexpect l1 -v v1 -g raw {
>  	expect * 1002 FetchError   {^out of workspace}
>  	expect * =    BerespStatus {^503}
>  	expect * =    BerespReason {^Backend fetch failed}
> -	expect * =    Error        {^out of workspace [(]Bo[)]}
> +	expect * =    Error        {^out of workspace [(]bo[)]}
>  	expect * =    LostHeader   {^Date:}
> -	expect * =    Error        {^out of workspace [(]Bo[)]}
> +	expect * =    Error        {^out of workspace [(]bo[)]}
>  	expect * =    LostHeader   {^299}
>  } -start
>  
> _______________________________________________
> varnish-commit mailing list
> varnish-commit at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-commit
> 


-- 

** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstra?e 32
22301 Hamburg

tel +49 40 28805731
mob +49 170 2723133
fax +49 40 42949753

xmpp://slink at jabber.int.uplex.de/

http://uplex.de/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-commit/attachments/20180926/fbd22229/attachment.bin>

From nils.goroll at uplex.de  Wed Sep 26 06:39:12 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 26 Sep 2018 06:39:12 +0000 (UTC)
Subject: [master] 663e2b8c1 remove markers about an issue now solved
Message-ID: <20180926063912.F22A0631A0@lists.varnish-cache.org>


commit 663e2b8c1de67b73cec3b09ba4f0f92f5e14e922
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Sep 26 08:12:38 2018 +0200

    remove markers about an issue now solved

diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c
index 5e8712503..6ec3577d2 100644
--- a/bin/varnishd/cache/cache_ban.c
+++ b/bin/varnishd/cache/cache_ban.c
@@ -168,10 +168,6 @@ ban_mark_completed(struct ban *b)
 		vbe32enc(b->spec + BANS_LENGTH, BANS_HEAD_LEN);
 		VSC_C_main->bans_completed++;
 		bans_persisted_fragmentation += ln - ban_len(b->spec);
-		/*
-		 * XXX absolute update of gauges - may be inaccurate for
-		 * Pool_Sumstat race
-		 */
 		VSC_C_main->bans_persisted_fragmentation =
 		    bans_persisted_fragmentation;
 	}
@@ -319,10 +315,6 @@ ban_export(void)
 	assert(VSB_len(vsb) == ln);
 	STV_BanExport((const uint8_t *)VSB_data(vsb), VSB_len(vsb));
 	VSB_destroy(&vsb);
-	/*
-	 * XXX absolute update of gauges - may be inaccurate for Pool_Sumstat
-	 * race
-	 */
 	VSC_C_main->bans_persisted_bytes =
 	    bans_persisted_bytes = ln;
 	VSC_C_main->bans_persisted_fragmentation =
@@ -406,10 +398,6 @@ ban_reload(const uint8_t *ban, unsigned len)
 	else
 		VTAILQ_INSERT_BEFORE(b, b2, list);
 	bans_persisted_bytes += len;
-	/*
-	 * XXX absolute update of gauges - may be inaccurate for Pool_Sumstat
-	 * race
-	 */
 	VSC_C_main->bans_persisted_bytes = bans_persisted_bytes;
 
 	/* Hunt down older duplicates */
diff --git a/bin/varnishd/cache/cache_ban_lurker.c b/bin/varnishd/cache/cache_ban_lurker.c
index 0680941ae..defe55d91 100644
--- a/bin/varnishd/cache/cache_ban_lurker.c
+++ b/bin/varnishd/cache/cache_ban_lurker.c
@@ -89,10 +89,6 @@ ban_cleantail(const struct ban *victim)
 			VTAILQ_INSERT_TAIL(&freelist, b, list);
 			bans_persisted_fragmentation +=
 			    ban_len(b->spec);
-			/*
-			 * XXX absolute update of gauges - may be inaccurate for
-			 * Pool_Sumstat race
-			 */
 			VSC_C_main->bans_persisted_fragmentation =
 			    bans_persisted_fragmentation;
 			ban_info_drop(b->spec, ban_len(b->spec));

From nils.goroll at uplex.de  Wed Sep 26 06:39:12 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 26 Sep 2018 06:39:12 +0000 (UTC)
Subject: [master] bc13919d5 avoid holding the ban_mtx unless we need to
Message-ID: <20180926063913.05B36631A1@lists.varnish-cache.org>


commit bc13919d58aac92203bd4bcb431878adde96f7e6
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Sep 26 08:21:37 2018 +0200

    avoid holding the ban_mtx unless we need to

diff --git a/bin/varnishd/cache/cache_ban_lurker.c b/bin/varnishd/cache/cache_ban_lurker.c
index defe55d91..1729e04a9 100644
--- a/bin/varnishd/cache/cache_ban_lurker.c
+++ b/bin/varnishd/cache/cache_ban_lurker.c
@@ -229,6 +229,11 @@ ban_lurker_test_ban(struct worker *wrk, struct vsl_log *vsl, struct ban *bt,
 		}
 		oc = ban_lurker_getfirst(vsl, bt);
 		if (oc == NULL) {
+			if (tested == 0 && lokc == 0) {
+				AZ(tested_tests);
+				AZ(lok);
+				return;
+			}
 			Lck_Lock(&ban_mtx);
 			VSC_C_main->bans_lurker_tested += tested;
 			VSC_C_main->bans_lurker_tests_tested += tested_tests;

From nils.goroll at uplex.de  Wed Sep 26 07:26:10 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 26 Sep 2018 07:26:10 +0000 (UTC)
Subject: [4.1] 44bf9f3cd clarify that we handle references correctly
Message-ID: <20180926072610.65E8864277@lists.varnish-cache.org>


commit 44bf9f3cde555c6548c6f2efeb4ce5433995cb70
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Wed Sep 26 09:14:20 2018 +0200

    clarify that we handle references correctly
    
    This is no semantic change, but rather than indirectly checking
    via the retval, we might also check the reason for keeping a
    reference (or rather, not).

diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c
index d1f5f1c56..4d4e67afe 100644
--- a/bin/varnishd/cache/cache_hash.c
+++ b/bin/varnishd/cache/cache_hash.c
@@ -495,7 +495,7 @@ HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp,
 		if (exp_oc->hits < LONG_MAX)
 			exp_oc->hits++;
 		Lck_Unlock(&oh->mtx);
-		if (retval == HSH_EXP)
+		if (*bocp == NULL)
 			assert(HSH_DerefObjHead(wrk, &oh));
 		*ocp = exp_oc;
 		return (retval);

From nils.goroll at uplex.de  Wed Sep 26 07:26:10 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Wed, 26 Sep 2018 07:26:10 +0000 (UTC)
Subject: [4.1] 00ad6d205 stabilize b63.vtc
Message-ID: <20180926072610.5A76B64276@lists.varnish-cache.org>


commit 00ad6d2055f5cbd8d462a7fca006481ec5b94b55
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Mon Jul 2 15:34:05 2018 +0000

    stabilize b63.vtc

diff --git a/bin/varnishtest/tests/b00063.vtc b/bin/varnishtest/tests/b00063.vtc
index 1adfe2680..da5a20448 100644
--- a/bin/varnishtest/tests/b00063.vtc
+++ b/bin/varnishtest/tests/b00063.vtc
@@ -45,6 +45,10 @@ varnish v1 -vcl+backend {
 	}
 } -start
 
+logexpect l1 -v v1 -g raw {
+	expect * *	ExpKill		EXP_Rearm
+} -start
+
 client c1 {
 	txreq -url "/1"
 	rxresp
@@ -76,6 +80,9 @@ client c2 {
 client c1 -wait
 client c2 -wait
 
+# Make sure the expiry has happened
+logexpect l1 -wait
+
 client c3 {
 	delay .1
 	# We should now get a HIT on the 404:

From phk at FreeBSD.org  Wed Sep 26 11:14:11 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 26 Sep 2018 11:14:11 +0000 (UTC)
Subject: [master] 109f441c2 Move 'delay',
 'shell' and 'barrier' to a new category of "global" commands.
Message-ID: <20180926111411.7F2D794BAF@lists.varnish-cache.org>


commit 109f441c20992ad2acb96002aaccd8ec1718a5e5
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Sep 26 11:10:14 2018 +0000

    Move 'delay', 'shell' and 'barrier' to a new category of "global" commands.

diff --git a/bin/varnishtest/cmds.h b/bin/varnishtest/cmds.h
index d3244c258..fd140efe2 100644
--- a/bin/varnishtest/cmds.h
+++ b/bin/varnishtest/cmds.h
@@ -29,21 +29,29 @@
 
 /*lint -save -e525 -e539 */
 
-CMD(barrier)
-CMD(client)
-CMD(delay)
-CMD(err_shell)
-CMD(feature)
-CMD(haproxy)
-CMD(logexpect)
-CMD(process)
-CMD(server)
-CMD(setenv)
-CMD(shell)
-CMD(syslog)
-CMD(varnish)
-CMD(varnishtest)
-CMD(vtest)
-#undef CMD
+#ifndef CMD_GLOBAL
+  #define CMD_GLOBAL(x)
+#endif
+CMD_GLOBAL(barrier)
+CMD_GLOBAL(delay)
+CMD_GLOBAL(shell)
+#undef CMD_GLOBAL
+
+#ifndef CMD_TOP
+  #define CMD_TOP(x)
+#endif
+CMD_TOP(client)
+CMD_TOP(err_shell)
+CMD_TOP(feature)
+CMD_TOP(haproxy)
+CMD_TOP(logexpect)
+CMD_TOP(process)
+CMD_TOP(server)
+CMD_TOP(setenv)
+CMD_TOP(syslog)
+CMD_TOP(varnish)
+CMD_TOP(varnishtest)
+CMD_TOP(vtest)
+#undef CMD_TOP
 
 /*lint -restore */
diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c
index 276566514..2b98596a8 100644
--- a/bin/varnishtest/vtc.c
+++ b/bin/varnishtest/vtc.c
@@ -290,6 +290,13 @@ macro_expand(struct vtclog *vl, const char *text)
  * Static checkers like Coverity may bitch about this, but we don't care.
  */
 
+static const struct cmds global_cmds[] = {
+#define CMD_GLOBAL(n) { #n, cmd_##n },
+#include "cmds.h"
+	{ NULL, NULL }
+};
+
+
 void
 parse_string(const char *spec, const struct cmds *cmd, void *priv,
     struct vtclog *vl)
@@ -410,6 +417,12 @@ parse_string(const char *spec, const struct cmds *cmd, void *priv,
 			if (!strcmp(token_s[0], cp->name))
 				break;
 
+		if (cp->name == NULL) {
+			for (cp = global_cmds; cp->name != NULL; cp++)
+				if (!strcmp(token_s[0], cp->name))
+					break;
+		}
+
 		if (cp->name == NULL)
 			vtc_fatal(vl, "Unknown command: \"%s\"", token_s[0]);
 
@@ -435,7 +448,7 @@ reset_cmds(const struct cmds *cmd)
  */
 
 static const struct cmds cmds[] = {
-#define CMD(n) { #n, cmd_##n },
+#define CMD_TOP(n) { #n, cmd_##n },
 #include "cmds.h"
 	{ NULL, NULL }
 };
diff --git a/bin/varnishtest/vtc.h b/bin/varnishtest/vtc.h
index 6ebff08a5..182f258e9 100644
--- a/bin/varnishtest/vtc.h
+++ b/bin/varnishtest/vtc.h
@@ -69,7 +69,8 @@ void parse_string(const char *spec, const struct cmds *cmd, void *priv,
     struct vtclog *vl);
 int fail_out(void);
 
-#define CMD(n) cmd_f cmd_##n;
+#define CMD_GLOBAL(n) cmd_f cmd_##n;
+#define CMD_TOP(n) cmd_f cmd_##n;
 #include "cmds.h"
 
 extern volatile sig_atomic_t vtc_error; /* Error, bail out */
diff --git a/bin/varnishtest/vtc_barrier.c b/bin/varnishtest/vtc_barrier.c
index 3f0f69065..7f2f0c1ce 100644
--- a/bin/varnishtest/vtc_barrier.c
+++ b/bin/varnishtest/vtc_barrier.c
@@ -348,8 +348,7 @@ barrier_sync(struct barrier *b, struct vtclog *vl)
 
 /* SECTION: barrier barrier
  *
- * NOTE: this can be used from the top-level as well as from client and server
- * specifications.
+ * NOTE: This command is available everywhere commands are given.
  *
  * Barriers allows you to synchronize different threads to make sure events
  * occur in the right order. It's even possible to use them in VCL.
diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c
index 6c4fcc8da..03aa7f7e5 100644
--- a/bin/varnishtest/vtc_http.c
+++ b/bin/varnishtest/vtc_http.c
@@ -1659,17 +1659,6 @@ cmd_http_fatal(CMD_ARGS)
 
 #define cmd_http_non_fatal cmd_http_fatal
 
-/* SECTION: client-server.spec.delay
- *
- * delay
- *	Same as for the top-level delay.
- *
- * SECTION: client-server.spec.barrier
- *
- * barrier
- *	Same as for the top-level barrier
- */
-
 static const char PREFACE[24] = {
 	0x50, 0x52, 0x49, 0x20, 0x2a, 0x20, 0x48, 0x54,
 	0x54, 0x50, 0x2f, 0x32, 0x2e, 0x30, 0x0d, 0x0a,
@@ -1815,7 +1804,6 @@ cmd_http_write_body(CMD_ARGS)
  */
 
 const struct cmds http_cmds[] = {
-#define CMD(n) { #n, cmd_##n },
 #define CMD_HTTP(n) { #n, cmd_http_##n },
 	/* session */
 	CMD_HTTP(accept)
@@ -1864,13 +1852,7 @@ const struct cmds http_cmds[] = {
 	CMD_HTTP(expect)
 	CMD_HTTP(expect_close)
 	CMD_HTTP(expect_pattern)
-
-	/* general purpose */
-	CMD(barrier)
-	CMD(delay)
-	CMD(shell)
 #undef CMD_HTTP
-#undef CMD
 	{ NULL, NULL }
 };
 
diff --git a/bin/varnishtest/vtc_http2.c b/bin/varnishtest/vtc_http2.c
index 404b68da0..ed72ea37a 100644
--- a/bin/varnishtest/vtc_http2.c
+++ b/bin/varnishtest/vtc_http2.c
@@ -2445,7 +2445,6 @@ cmd_write_body(CMD_ARGS)
  * client or a server.
  */
 static const struct cmds stream_cmds[] = {
-#define CMD(n) { #n, cmd_##n },
 #define CMD_STREAM(n) { #n, cmd_##n },
 	/* spec */
 	CMD_STREAM(expect)
@@ -2475,14 +2474,8 @@ static const struct cmds stream_cmds[] = {
 	CMD_STREAM(txsettings)
 	CMD_STREAM(txwinup)
 	CMD_STREAM(write_body)
-
-	/* general purpose */
-	CMD(barrier)
-	CMD(delay)
-	CMD(shell)
 	{ NULL, NULL }
 #undef CMD_STREAM
-#undef CMD
 };
 
 static void *
diff --git a/bin/varnishtest/vtc_misc.c b/bin/varnishtest/vtc_misc.c
index f04003da7..2fcf5e0e2 100644
--- a/bin/varnishtest/vtc_misc.c
+++ b/bin/varnishtest/vtc_misc.c
@@ -99,6 +99,8 @@ cmd_varnishtest(CMD_ARGS)
 }
 
 /* SECTION: shell shell
+ *
+ * NOTE: This command is available everywhere commands are given.
  *
  * Pass the string given as argument to a shell. If you have multiple
  * commands to run, you can use curly brackets to describe a multi-lines
@@ -311,13 +313,12 @@ cmd_setenv(CMD_ARGS)
 }
 
 /* SECTION: delay delay
+ *
+ * NOTE: This command is available everywhere commands are given.
  *
  * Sleep for the number of seconds specified in the argument. The number
  * can include a fractional part, e.g. 1.5.
- */
-/* SECTION: stream.spec.delay delay
  *
- * Same as for the top-level delay.
  */
 void
 cmd_delay(CMD_ARGS)

From fgsch at lodoss.net  Wed Sep 26 13:06:09 2018
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Wed, 26 Sep 2018 13:06:09 +0000 (UTC)
Subject: [master] fe4fd1646 Python 3 takes priority over python 2
Message-ID: <20180926130609.D66C196EE1@lists.varnish-cache.org>


commit fe4fd1646a8565c27f7169104834dfb2099ca943
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Wed Sep 26 07:03:28 2018 -0600

    Python 3 takes priority over python 2
    
    Take 2. Let's see if this time sticks.

diff --git a/varnish.m4 b/varnish.m4
index 8a92d5f10..71df96d9c 100644
--- a/varnish.m4
+++ b/varnish.m4
@@ -129,7 +129,9 @@ AC_DEFUN([_VARNISH_CHECK_DEVEL], [
 # _VARNISH_CHECK_PYTHON
 # ---------------------
 AC_DEFUN([_VARNISH_CHECK_PYTHON], [
-
+	m4_define_default([_AM_PYTHON_INTERPRETER_LIST],
+[python3.9 python3.8 python3.7 python3.6 python3.5 python3.4 python2.7 dnl
+python python2 python3])
 	AM_PATH_PYTHON([2.7], [], [
 		AC_MSG_ERROR([Python >= 2.7 is required.])
 	])

From fgsch at lodoss.net  Wed Sep 26 13:21:08 2018
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Wed, 26 Sep 2018 13:21:08 +0000 (UTC)
Subject: [master] dc4186746 Cleanup travis' osx job and bump image
Message-ID: <20180926132109.0339D974C1@lists.varnish-cache.org>


commit dc41867460688d66634d46931072eaf1b9443d28
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Wed Sep 26 07:19:26 2018 -0600

    Cleanup travis' osx job and bump image

diff --git a/.travis.yml b/.travis.yml
index 4441126b0..1b0751c8f 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -16,7 +16,7 @@ matrix:
     env: CLANG=6.0 SAN_FLAGS="--enable-asan --enable-ubsan"
     sudo: required
   - os: osx
-    osx_image: xcode9.3
+    osx_image: xcode10
     compiler: clang
   allow_failures:
   - os: osx
@@ -36,7 +36,6 @@ before_install:
   - |
     if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
       brew update
-      brew upgrade python
       brew install docutils sphinx-doc nghttp2
       export PATH="/usr/local/opt/sphinx-doc/bin:$PATH"
     elif [[ -n "$CLANG" ]]; then
@@ -59,9 +58,6 @@ before_install:
   - ./configure ${CONFIGURE_ARGS}
 script:
   - |
-    if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
-      export PYTHONPATH=`brew --prefix`/lib/python2.7/site-packages
-    fi
     if [[ "$TRAVIS_PULL_REQUEST" != "false" ]]; then
       make -j3 distcheck
     else

From fgsch at lodoss.net  Wed Sep 26 15:12:11 2018
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Wed, 26 Sep 2018 15:12:11 +0000 (UTC)
Subject: [master] ae50ca8b3 Switch to clang 7 in our travis job
Message-ID: <20180926151211.44BACA1784@lists.varnish-cache.org>


commit ae50ca8b362d1a0e3f75273fbe1d644545bb6c41
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Wed Sep 26 09:10:54 2018 -0600

    Switch to clang 7 in our travis job

diff --git a/.travis.yml b/.travis.yml
index 1b0751c8f..da2bcbb8b 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -13,14 +13,14 @@ matrix:
   - os: linux
     dist: trusty
     compiler: clang
-    env: CLANG=6.0 SAN_FLAGS="--enable-asan --enable-ubsan"
+    env: CLANG=7 SAN_FLAGS="--enable-asan --enable-ubsan"
     sudo: required
   - os: osx
     osx_image: xcode10
     compiler: clang
   allow_failures:
   - os: osx
-  - env: CLANG=6.0 SAN_FLAGS="--enable-asan --enable-ubsan"
+  - env: CLANG=7 SAN_FLAGS="--enable-asan --enable-ubsan"
 addons:
   apt:
     packages:

From fgsch at users.noreply.github.com  Wed Sep 26 16:50:07 2018
From: fgsch at users.noreply.github.com (Federico G. Schwindt)
Date: Wed, 26 Sep 2018 16:50:07 +0000 (UTC)
Subject: [master] e476f9f6c Fix varnish_vsc metrics type
Message-ID: <20180926165007.43F14A5318@lists.varnish-cache.org>


commit e476f9f6c67bfcf56b6c09bd8aea57df4df5ed36
Author: Emmanuel Hocdet <manu at gandi.net>
Date:   Wed Sep 26 12:03:07 2018 +0200

    Fix varnish_vsc metrics type

diff --git a/bin/varnishd/VSC_main.vsc b/bin/varnishd/VSC_main.vsc
index 80a486a24..8ca97887e 100644
--- a/bin/varnishd/VSC_main.vsc
+++ b/bin/varnishd/VSC_main.vsc
@@ -363,20 +363,17 @@
 	Number of backends known to us.
 
 .. varnish_vsc:: n_expired
-	:type:	gauge
 	:oneliner:	Number of expired objects
 
 	Number of objects that expired from cache because of old age.
 
 .. varnish_vsc:: n_lru_nuked
-	:type:	gauge
 	:oneliner:	Number of LRU nuked objects
 
 	How many objects have been forcefully evicted from storage to make
 	room for a new object.
 
 .. varnish_vsc:: n_lru_moved
-	:type:	gauge
 	:level:	diag
 	:oneliner:	Number of LRU moved objects
 
@@ -780,12 +777,10 @@
 	bans in the persistent ban lists.
 
 .. varnish_vsc:: n_purges
-	:type:	gauge
 	:oneliner:	Number of purge operations executed
 
 
 .. varnish_vsc:: n_obj_purged
-	:type:	gauge
 	:oneliner:	Number of purged objects
 
 

From phk at FreeBSD.org  Wed Sep 26 19:43:05 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Wed, 26 Sep 2018 19:43:05 +0000 (UTC)
Subject: [master] 6b933111a Dup(2) stderr before calling VCLS on -I argument.
Message-ID: <20180926194305.A7DF3A8526@lists.varnish-cache.org>


commit 6b933111a4e0267ed06307a6d3d3e0660e6dafca
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Wed Sep 26 19:40:05 2018 +0000

    Dup(2) stderr before calling VCLS on -I argument.
    
    Fixes: #2782

diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c
index 7a427d5bf..d03ff2106 100644
--- a/bin/varnishd/mgt/mgt_child.c
+++ b/bin/varnishd/mgt/mgt_child.c
@@ -278,6 +278,8 @@ mgt_launch_child(struct cli *cli)
 	/* Open pipe for mgt->child CLI */
 	AZ(pipe(cp));
 	heritage.cli_in = cp[0];
+	assert(cp[0] > STDERR_FILENO);	// See #2782
+	assert(cp[1] > STDERR_FILENO);
 	MCH_Fd_Inherit(heritage.cli_in, "cli_in");
 	child_cli_out = cp[1];
 
diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c
index aada3ce2c..237fb0f7f 100644
--- a/bin/varnishd/mgt/mgt_main.c
+++ b/bin/varnishd/mgt/mgt_main.c
@@ -861,7 +861,8 @@ main(int argc, char * const *argv)
 
 	if (I_fd >= 0) {
 		fprintf(stderr, "BEGIN of -I file processing\n");
-		mgt_cli_setup(I_fd, 2, 1, "-I file", mgt_I_close, stderr);
+		/* We must dup stderr, because VCLS closes the output fd */
+		mgt_cli_setup(I_fd, dup(2), 1, "-I file", mgt_I_close, stderr);
 		while (I_fd >= 0) {
 			o = VEV_Once(mgt_evb);
 			if (o != 1)

From phk at FreeBSD.org  Thu Sep 27 06:08:13 2018
From: phk at FreeBSD.org (Poul-Henning Kamp)
Date: Thu, 27 Sep 2018 06:08:13 +0000 (UTC)
Subject: [master] 9b13d33d7 Convert this test to UTF-8
Message-ID: <20180927060814.123AC6F32@lists.varnish-cache.org>


commit 9b13d33d7cc7fdd6b44c4f7db02e97747984bc8e
Author: Poul-Henning Kamp <phk at FreeBSD.org>
Date:   Thu Sep 27 05:58:35 2018 +0000

    Convert this test to UTF-8

diff --git a/bin/varnishtest/tests/r00545.vtc b/bin/varnishtest/tests/r00545.vtc
index 0888af11a..e293c926b 100644
--- a/bin/varnishtest/tests/r00545.vtc
+++ b/bin/varnishtest/tests/r00545.vtc
@@ -7,12 +7,12 @@ server s1 {
 
 varnish v1 -vcl+backend {
 	sub vcl_deliver {
-		set resp.http.foo = "???";
+		set resp.http.foo = "??????";
 	}
 } -start
 
 client c1 {
 	txreq
 	rxresp
-	expect resp.http.foo == "???"
+	expect resp.http.foo == "??????"
 } -run

From nils.goroll at uplex.de  Thu Sep 27 09:12:09 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Thu, 27 Sep 2018 09:12:09 +0000 (UTC)
Subject: [master] a2acba9af explain a relevant detail of the worker thread
 signaling
Message-ID: <20180927091209.EF56C61454@lists.varnish-cache.org>


commit a2acba9afe5d672fa80707a421150905e04ebcb3
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Thu Sep 27 10:27:44 2018 +0200

    explain a relevant detail of the worker thread signaling
    
    Over time, I have repeatedly stared at this code again and again
    wondering if (and why) our cv signaling is correct, just to end up
    with the same insight each time (but first overlooking #2719)
    
    Being fully aware that we do not want to plaster our code with
    outdated comments, I hope this explanation is warranted to save myself
    (and others, hopefully) from wasting precious life time on reiterating
    over the same question.

diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c
index be98b4e5e..5b88a8338 100644
--- a/bin/varnishd/cache/cache_wrk.c
+++ b/bin/varnishd/cache/cache_wrk.c
@@ -27,6 +27,27 @@
  * SUCH DAMAGE.
  *
  * Worker thread stuff unrelated to the worker thread pools.
+ *
+ * --
+ * signaling_note:
+ *
+ * note on worker wakeup signaling through the wrk condition variable (cv)
+ *
+ * In the general case, a cv needs to be signaled while holding the
+ * corresponding mutex, otherwise the signal may be posted before the waiting
+ * thread could register itself on the cv and, consequently, the signal may be
+ * missed.
+ *
+ * In our case, any worker thread which we wake up comes from the idle queue,
+ * where it put itself under the mutex, releasing that mutex implicitly via
+ * Lck_CondWait() (which calls some variant of pthread_cond_wait). So we avoid
+ * additional mutex contention knowing that any worker thread on the idle queue
+ * is blocking on the cv.
+ *
+ * Except -- when it isn't, because it woke up for releasing its VCL
+ * Reference. To account for this case, we check if the task function has been
+ * set in the meantime, which in turn requires all of the task preparation to be
+ * done holding the pool mutex. (see also #2719)
  */
 
 #include "config.h"
@@ -220,6 +241,7 @@ Pool_Task_Arg(struct worker *wrk, enum task_prio prio, task_func_t *func,
 	wrk2->task.func = func;
 	wrk2->task.priv = wrk2->aws->f;
 	Lck_Unlock(&pp->mtx);
+	// see signaling_note at the top for explanation
 	if (retval)
 		AZ(pthread_cond_signal(&wrk2->cond));
 	return (retval);
@@ -252,6 +274,7 @@ Pool_Task(struct pool *pp, struct pool_task *task, enum task_prio prio)
 		wrk->task.func = task->func;
 		wrk->task.priv = task->priv;
 		Lck_Unlock(&pp->mtx);
+		// see signaling_note at the top for explanation
 		AZ(pthread_cond_signal(&wrk->cond));
 		return (0);
 	}
@@ -346,6 +369,7 @@ Pool_Work_Thread(struct pool *pp, struct worker *wrk)
 			VTAILQ_INSERT_HEAD(&pp->idle_queue, &wrk->task, list);
 			pp->nidle++;
 			do {
+				// see signaling_note at the top for explanation
 				i = Lck_CondWait(&wrk->cond, &pp->mtx,
 				    wrk->vcl == NULL ?  0 : wrk->lastused+60.);
 				if (i == ETIMEDOUT)

From nils.goroll at uplex.de  Thu Sep 27 09:21:06 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Thu, 27 Sep 2018 09:21:06 +0000 (UTC)
Subject: [master] d8cc41bc0 shrink the critical section under the pool mutex
 by a bit
Message-ID: <20180927092106.47B3F617F9@lists.varnish-cache.org>


commit d8cc41bc0bee5a8fd1402963cc693e6890acedf0
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Thu Sep 27 11:17:27 2018 +0200

    shrink the critical section under the pool mutex by a bit
    
    Checking and preparing our worker struct does not need to happen
    under the lock.

diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c
index 5b88a8338..589ce68cb 100644
--- a/bin/varnishd/cache/cache_wrk.c
+++ b/bin/varnishd/cache/cache_wrk.c
@@ -326,13 +326,12 @@ Pool_Work_Thread(struct pool *pp, struct worker *wrk)
 	CHECK_OBJ_NOTNULL(pp, POOL_MAGIC);
 	wrk->pool = pp;
 	while (1) {
-		Lck_Lock(&pp->mtx);
-
 		CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC);
 
 		WS_Reset(wrk->aws, 0);
 		AZ(wrk->vsl);
 
+		Lck_Lock(&pp->mtx);
 		if (pp->nidle < pool_reserve())
 			prio_lim = TASK_QUEUE_RESERVE + 1;
 		else

From nils.goroll at uplex.de  Thu Sep 27 18:24:06 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Thu, 27 Sep 2018 18:24:06 +0000 (UTC)
Subject: [master] 527f1bd0b no need to sum stats which we don't use
Message-ID: <20180927182406.8EA12A0B6A@lists.varnish-cache.org>


commit 527f1bd0b74e758c6f8b8ae189b293151e61cd15
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Thu Sep 27 20:21:32 2018 +0200

    no need to sum stats which we don't use

diff --git a/bin/varnishd/cache/cache_ban_lurker.c b/bin/varnishd/cache/cache_ban_lurker.c
index 1729e04a9..a56119c62 100644
--- a/bin/varnishd/cache/cache_ban_lurker.c
+++ b/bin/varnishd/cache/cache_ban_lurker.c
@@ -422,7 +422,6 @@ ban_lurker(struct worker *wrk, void *priv)
 		d += VTIM_real();
 		Lck_Lock(&ban_mtx);
 		if (gen == ban_generation) {
-			Pool_Sumstat(wrk);
 			(void)Lck_CondWait(&ban_lurker_cond, &ban_mtx, d);
 			ban_batch = 0;
 		}

From nils.goroll at uplex.de  Thu Sep 27 18:34:06 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Thu, 27 Sep 2018 18:34:06 +0000 (UTC)
Subject: [master] eedfa1e4d Revert "no need to sum stats which we don't use"
Message-ID: <20180927183406.CAFC5A0FCA@lists.varnish-cache.org>


commit eedfa1e4d7395b67cc17ebfece2ebde6333a515a
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Thu Sep 27 20:32:25 2018 +0200

    Revert "no need to sum stats which we don't use"
    
    my apologies: as long as we pass around a struct wrk, some other
    function could use the wrk->stats - in other words, the fact that
    the ban stats are decoupled from the wrk stats does not make the
    latter any less relevant.
    
    This reverts commit 527f1bd0b74e758c6f8b8ae189b293151e61cd15.

diff --git a/bin/varnishd/cache/cache_ban_lurker.c b/bin/varnishd/cache/cache_ban_lurker.c
index a56119c62..1729e04a9 100644
--- a/bin/varnishd/cache/cache_ban_lurker.c
+++ b/bin/varnishd/cache/cache_ban_lurker.c
@@ -422,6 +422,7 @@ ban_lurker(struct worker *wrk, void *priv)
 		d += VTIM_real();
 		Lck_Lock(&ban_mtx);
 		if (gen == ban_generation) {
+			Pool_Sumstat(wrk);
 			(void)Lck_CondWait(&ban_lurker_cond, &ban_mtx, d);
 			ban_batch = 0;
 		}

From dridi.boukelmoune at gmail.com  Fri Sep 28 13:39:15 2018
From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune)
Date: Fri, 28 Sep 2018 13:39:15 +0000 (UTC)
Subject: [master] bd7d12b86 VSC_Arg succeeds with non-zero
Message-ID: <20180928133915.F145C651B2@lists.varnish-cache.org>


commit bd7d12b86934da8a1befd202f4793bd850bd2fb4
Author: Dridi Boukelmoune <dridi.boukelmoune at gmail.com>
Date:   Fri Sep 28 15:37:40 2018 +0200

    VSC_Arg succeeds with non-zero
    
    Fixes #2787

diff --git a/bin/varnishstat/varnishstat.c b/bin/varnishstat/varnishstat.c
index 74a8f5adf..be9e64c79 100644
--- a/bin/varnishstat/varnishstat.c
+++ b/bin/varnishstat/varnishstat.c
@@ -319,10 +319,10 @@ main(int argc, char * const *argv)
 
 	if (curses) {
 		if (has_f) {
-			AZ(VSC_Arg(vsc, 'f', "MGT.uptime"));
-			AZ(VSC_Arg(vsc, 'f', "MAIN.uptime"));
-			AZ(VSC_Arg(vsc, 'f', "MAIN.cache_hit"));
-			AZ(VSC_Arg(vsc, 'f', "MAIN.cache_miss"));
+			AN(VSC_Arg(vsc, 'f', "MGT.uptime"));
+			AN(VSC_Arg(vsc, 'f', "MAIN.uptime"));
+			AN(VSC_Arg(vsc, 'f', "MAIN.cache_hit"));
+			AN(VSC_Arg(vsc, 'f', "MAIN.cache_miss"));
 		}
 		do_curses(vd, vsc, 1.0);
 	}

From nils.goroll at uplex.de  Fri Sep 28 15:23:04 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Fri, 28 Sep 2018 17:23:04 +0200
Subject: [master] bd7d12b86 VSC_Arg succeeds with non-zero
In-Reply-To: <20180928133915.F145C651B2@lists.varnish-cache.org>
References: <20180928133915.F145C651B2@lists.varnish-cache.org>
Message-ID: <5d19926e-bef0-cb35-67fe-b319834bdf18@uplex.de>

wasn't the purpose to ensure that we always have the uptime fields filtered?

-- 

** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstra?e 32
22301 Hamburg

tel +49 40 28805731
mob +49 170 2723133
fax +49 40 42949753

xmpp://slink at jabber.int.uplex.de/

http://uplex.de/

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-commit/attachments/20180928/b2523beb/attachment-0001.bin>

From dridi at varni.sh  Fri Sep 28 15:33:39 2018
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 28 Sep 2018 17:33:39 +0200
Subject: [master] bd7d12b86 VSC_Arg succeeds with non-zero
In-Reply-To: <5d19926e-bef0-cb35-67fe-b319834bdf18@uplex.de>
References: <20180928133915.F145C651B2@lists.varnish-cache.org>
 <5d19926e-bef0-cb35-67fe-b319834bdf18@uplex.de>
Message-ID: <CABoVN9BkoNJSnY8t80OOBtjwpZkDBeJbKfbPMHzK-FzgLn-Esg@mail.gmail.com>

On Fri, Sep 28, 2018 at 5:23 PM Nils Goroll <nils.goroll at uplex.de> wrote:
>
> wasn't the purpose to ensure that we always have the uptime fields filtered?

Yes, but there is no code coverage of this "if (has_f)" branch so we
didn't realize that the assertion was the other way around until you
opened an issue.

Someone(tm) should write a test case for 2678, but I have no time for
that right now :(

Dridi

From nils.goroll at uplex.de  Fri Sep 28 15:35:01 2018
From: nils.goroll at uplex.de (Nils Goroll)
Date: Fri, 28 Sep 2018 17:35:01 +0200
Subject: [master] bd7d12b86 VSC_Arg succeeds with non-zero
In-Reply-To: <5d19926e-bef0-cb35-67fe-b319834bdf18@uplex.de>
References: <20180928133915.F145C651B2@lists.varnish-cache.org>
 <5d19926e-bef0-cb35-67fe-b319834bdf18@uplex.de>
Message-ID: <88aed0e7-a6f8-ecb6-b58c-751e22eb107c@uplex.de>

On 28/09/2018 17:23, Nils Goroll wrote:
> wasn't the purpose to ensure that we always have the uptime fields filtered?

nevermind, the patch works

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-commit/attachments/20180928/cefe775c/attachment.bin>

From fgsch at lodoss.net  Sat Sep 29 13:01:11 2018
From: fgsch at lodoss.net (Federico G. Schwindt)
Date: Sat, 29 Sep 2018 13:01:11 +0000 (UTC)
Subject: [master] 5e263a0a1 Plug minor leak when we fail loading a vcl
Message-ID: <20180929130111.912C4B0FEA@lists.varnish-cache.org>


commit 5e263a0a17d88f1de25b34fc35a4d07656665ebf
Author: Federico G. Schwindt <fgsch at lodoss.net>
Date:   Sat Sep 29 06:57:50 2018 -0600

    Plug minor leak when we fail loading a vcl

diff --git a/bin/varnishd/cache/cache_vcl.c b/bin/varnishd/cache/cache_vcl.c
index dd3267ea2..6827b7379 100644
--- a/bin/varnishd/cache/cache_vcl.c
+++ b/bin/varnishd/cache/cache_vcl.c
@@ -347,6 +347,7 @@ vcl_KillBackends(struct vcl *vcl)
 		REPLACE(vdir->cli_name, NULL);
 		AN(vdir->methods->destroy);
 		vdir->methods->destroy(vdir->dir);
+		FREE_OBJ(vdir->dir);
 		FREE_OBJ(vdir);
 	}
 }