From nils.goroll at uplex.de Mon Feb 3 13:15:07 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 3 Feb 2020 13:15:07 +0000 (UTC) Subject: [master] f5dcaeabb Update RedHat / CentOS install instructions for >= 8 Message-ID: <20200203131507.DE0B3AD92E@lists.varnish-cache.org> commit f5dcaeabba2525c5609b66001f4743ba2fbbf7f8 Author: Nils Goroll Date: Mon Feb 3 14:11:00 2020 +0100 Update RedHat / CentOS install instructions for >= 8 Basically taken from #3204 by @gquintard - sphinx install for release 8 - included libunwind though it is optional - left out libconfig libcurl and openssh-clients because varnish-cache does not need them diff --git a/doc/sphinx/installation/install_source.rst b/doc/sphinx/installation/install_source.rst index b9a95cc14..39d30a4cc 100644 --- a/doc/sphinx/installation/install_source.rst +++ b/doc/sphinx/installation/install_source.rst @@ -74,16 +74,34 @@ Build dependencies on Red Hat / CentOS .. gawk '/^BuildRequires/ {print "* `" $2 "`"}' ../../../redhat/varnish.spec | sort | uniq | egrep -v '(systemd)' -To build Varnish on a Red Hat or CentOS system, this command should -install required packages (replace ``sudo yum install`` if needed):: +in the following shell commands, replace ``sudo yum install`` if needed. - sudo yum install \ +Install sphinx + +* On Red Hat / CentOS 8, sphinx is not included in the default + repositories, so execute these steps to include it from the + PowerTools repository:: + + sudo dnf install -y 'dnf-command(config-manager)' + sudo yum config-manager --set-enabled PowerTools + sudo yum install -y diffutils python3-sphinx + +* On Red Hat / CentOS <= 7, install sphinx:: + + sudo yum install -y python-sphinx + +The following step should conclude installation of the required +packages:: + + yum install -y \ make \ autoconf \ automake \ jemalloc-devel \ + libconfig-devel \ libedit-devel \ libtool \ + libunwind-devel \ ncurses-devel \ pcre-devel \ pkgconfig \ From nils.goroll at uplex.de Mon Feb 3 13:19:05 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 3 Feb 2020 13:19:05 +0000 (UTC) Subject: [master] 0e192d6fa glitches from previous commit Message-ID: <20200203131905.D8F0FADC86@lists.varnish-cache.org> commit 0e192d6fa1691ebe9a4a3dcb600579cd43a0c1ba Author: Nils Goroll Date: Mon Feb 3 14:17:59 2020 +0100 glitches from previous commit Sorry, I had missed a fixup before push diff --git a/doc/sphinx/installation/install_source.rst b/doc/sphinx/installation/install_source.rst index 39d30a4cc..488fe235c 100644 --- a/doc/sphinx/installation/install_source.rst +++ b/doc/sphinx/installation/install_source.rst @@ -98,15 +98,13 @@ packages:: autoconf \ automake \ jemalloc-devel \ - libconfig-devel \ libedit-devel \ libtool \ libunwind-devel \ ncurses-devel \ pcre-devel \ pkgconfig \ - python3-docutils \ - python3-sphinx + python3-docutils Optionally, to rebuild the svg files:: From dridi at varni.sh Mon Feb 3 13:24:36 2020 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 3 Feb 2020 13:24:36 +0000 Subject: [master] f5dcaeabb Update RedHat / CentOS install instructions for >= 8 In-Reply-To: <20200203131507.DE0B3AD92E@lists.varnish-cache.org> References: <20200203131507.DE0B3AD92E@lists.varnish-cache.org> Message-ID: On Mon, Feb 3, 2020 at 1:15 PM Nils Goroll wrote: > > > commit f5dcaeabba2525c5609b66001f4743ba2fbbf7f8 > Author: Nils Goroll > Date: Mon Feb 3 14:11:00 2020 +0100 > > Update RedHat / CentOS install instructions for >= 8 > > Basically taken from #3204 by @gquintard > > - sphinx install for release 8 > - included libunwind though it is optional > - left out libconfig libcurl and openssh-clients because > varnish-cache does not need them > > diff --git a/doc/sphinx/installation/install_source.rst b/doc/sphinx/installation/install_source.rst > index b9a95cc14..39d30a4cc 100644 > --- a/doc/sphinx/installation/install_source.rst > +++ b/doc/sphinx/installation/install_source.rst > @@ -74,16 +74,34 @@ Build dependencies on Red Hat / CentOS > > .. gawk '/^BuildRequires/ {print "* `" $2 "`"}' ../../../redhat/varnish.spec | sort | uniq | egrep -v '(systemd)' > > -To build Varnish on a Red Hat or CentOS system, this command should > -install required packages (replace ``sudo yum install`` if needed):: > +in the following shell commands, replace ``sudo yum install`` if needed. > > - sudo yum install \ > +Install sphinx > + > +* On Red Hat / CentOS 8, sphinx is not included in the default > + repositories, so execute these steps to include it from the > + PowerTools repository:: > + > + sudo dnf install -y 'dnf-command(config-manager)' > + sudo yum config-manager --set-enabled PowerTools > + sudo yum install -y diffutils python3-sphinx > + > +* On Red Hat / CentOS <= 7, install sphinx:: > + > + sudo yum install -y python-sphinx I believe that PowerTools was also needed to install -devel packages, Guillaume will correct me if I'm wrong. The reason why this step is done in CircleCI configuration is because we have multiple `yum install` commands so it would not be convenient to add --enablerepo to all of them. > +The following step should conclude installation of the required > +packages:: > + > + yum install -y \ Without the first step from above, a single command should be needed: yum -y --enablerepo PowerTools install [pkgs] > make \ > autoconf \ > automake \ > jemalloc-devel \ > + libconfig-devel \ > libedit-devel \ > libtool \ > + libunwind-devel \ We only need libunwind if we enable it at configure time, I would leave it out of the default list of dependencies. My 2 cents, Dridi > ncurses-devel \ > pcre-devel \ > pkgconfig \ > _______________________________________________ > varnish-commit mailing list > varnish-commit at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-commit From nils.goroll at uplex.de Mon Feb 3 13:28:13 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 3 Feb 2020 14:28:13 +0100 Subject: [master] f5dcaeabb Update RedHat / CentOS install instructions for >= 8 In-Reply-To: References: <20200203131507.DE0B3AD92E@lists.varnish-cache.org> Message-ID: On 03/02/2020 14:24, Dridi Boukelmoune wrote: > We only need libunwind if we enable it at configure time, I would > leave it out of the default list of dependencies. please feel free to correct the .rst -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg tel +49 40 28805731 mob +49 170 2723133 fax +49 40 42949753 xmpp://slink at jabber.int.uplex.de/ http://uplex.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From nils.goroll at uplex.de Mon Feb 3 15:14:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 3 Feb 2020 15:14:06 +0000 (UTC) Subject: [master] 6c4ae7ed4 make parameter canary test 64bit Message-ID: <20200203151406.C90A7B1714@lists.varnish-cache.org> commit 6c4ae7ed4554471d1a68b9ae340463984499382e Author: Nils Goroll Date: Mon Feb 3 16:12:06 2020 +0100 make parameter canary test 64bit as decided by bugwash. Closes #3203 diff --git a/bin/varnishtest/tests/v00004.vtc b/bin/varnishtest/tests/v00004.vtc index 5f7a7aec8..0b52fa11b 100644 --- a/bin/varnishtest/tests/v00004.vtc +++ b/bin/varnishtest/tests/v00004.vtc @@ -1,5 +1,7 @@ varnishtest "test if our default paramers make sense ..." +feature 64bit + # ... for our definition of a standard use case: # - 2019 header madness # - 5 ESI levels down @@ -154,10 +156,6 @@ varnish v1 -vcl+backend { sub recv9 { std.log("STK recv9 " + debug.stk() + " WS " + vtc.workspace_free(client)); - if (vtc.workspace_free(client) < 48 * 1024) { - std.log("XXX #3203 skipping"); - return; - } set req.http.regex = regsub(req.http.cookie, "(.*)", "\1\1\1\1\1\1\1\1"); std.log("WS " + vtc.workspace_free(client)); # hey geoff, this is deliberate From guillaume at varnish-software.com Mon Feb 3 21:14:10 2020 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 3 Feb 2020 21:14:10 +0000 (UTC) Subject: [master] fd19ab82f [cci] remove unneeded deps Message-ID: <20200203211410.69EEC61435@lists.varnish-cache.org> commit fd19ab82f5b98723281793074de150841b824f70 Author: Guillaume Quintard Date: Mon Feb 3 10:52:10 2020 -0800 [cci] remove unneeded deps diff --git a/.circleci/config.yml b/.circleci/config.yml index de6444d6a..9b5dee3e3 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -20,17 +20,13 @@ jobs: automake \ jemalloc-devel \ git \ - libconfig-devel \ - libcurl-devel \ libedit-devel \ libtool \ libunwind-devel \ make \ - openssh-clients \ pcre-devel \ python3 \ - python-sphinx \ - sudo + python-sphinx - checkout - run: name: Create the dist tarball @@ -107,13 +103,10 @@ jobs: yum install -y \ automake \ jemalloc-devel \ - libconfig-devel \ - libcurl-devel \ libedit-devel \ libtool \ libunwind-devel \ make \ - openssh-clients \ pcre-devel \ python3 \ sudo @@ -127,7 +120,6 @@ jobs: build-essential \ ca-certificates \ graphviz \ - libconfig-dev \ libedit-dev \ libjemalloc-dev \ libncurses-dev \ @@ -145,7 +137,6 @@ jobs: build-base \ ca-certificates \ gzip \ - libconfig-dev \ libedit-dev \ libtool \ libunwind-dev \ From guillaume at varnish-software.com Mon Feb 3 21:14:10 2020 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 3 Feb 2020 21:14:10 +0000 (UTC) Subject: [master] 5e378f43c [cci] use artifacts to expose packages Message-ID: <20200203211410.7858061437@lists.varnish-cache.org> commit 5e378f43c1b067d794d7b806ff768a01597ab1b6 Author: Guillaume Quintard Date: Mon Feb 3 11:17:56 2020 -0800 [cci] use artifacts to expose packages diff --git a/.circleci/config.yml b/.circleci/config.yml index 9b5dee3e3..74a8c0cbe 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -72,13 +72,11 @@ jobs: - alpine.tar.gz distcheck: parameters: - release: - description: the release name (stretch|buster|xenial|bionic) - default: "" - type: string dist: description: the Linux distribution (debian|ubuntu) - default: "" + type: string + release: + description: the release name (stretch|buster|xenial|bionic) type: string extra_conf: description: platform-specific configure arguments @@ -177,6 +175,15 @@ jobs: --preserve-env=ASAN_OPTIONS,LSAN_OPTIONS,TSAN_OPTIONS,UBSAN_OPTIONS \ make distcheck VERBOSE=1 -j 12 -k build_apks: + parameters: + dist: + description: the Linux distribution (alpine) + default: alpine + type: string + release: + description: the release version + default: latest + type: string description: Build alpine apks docker: - image: alpine @@ -224,19 +231,25 @@ jobs: command: | mkdir apks cp /home/builder/packages/x86_64/*.apk apks + - run: + name: Import the packages into the workspace + command: | + mkdir -p packages/<< parameters.dist >>/<< parameters.release >> + mv /home/builder/packages/x86_64/*.apk packages/<< parameters.dist >>/<< parameters.release >> - persist_to_workspace: root: . paths: - - apks/*.apk + - packages/<< parameters.dist >>/<< parameters.release >>/varnish* + - store_artifacts: + path: packages/<< parameters.dist >>/<< parameters.release >> + destination: ./ build_debs: parameters: - release: - description: the release name (stretch|buster|xenial|bionic) - default: "" - type: string dist: description: the Linux distribution (debian|ubuntu) - default: "" + type: string + release: + description: the release name (stretch|buster|xenial|bionic) type: string description: Build << parameters.release >> debs docker: @@ -278,19 +291,24 @@ jobs: - run: name: Import the packages into the workspace command: | - mkdir debs - mv ../*.deb ../*.dsc debs/ + mkdir -p packages/<< parameters.dist >>/<< parameters.release >> + mv ../*.deb ../*.dsc packages/<< parameters.dist >>/<< parameters.release >> - persist_to_workspace: root: . paths: - - debs/varnish*.deb - - debs/varnish*.dsc + - packages/<< parameters.dist >>/<< parameters.release >>/varnish* + - store_artifacts: + path: packages/<< parameters.dist >>/<< parameters.release >> + destination: ./ build_rpms: parameters: + dist: + description: the Linux distribution (centos|amazonlinux) + type: string release: - description: the Centos version (7|8) + description: the distribution version (7|8|2) type: string - description: Build Centos << parameters.release >> rpms + description: Build << parameters.dist >>:<< parameters.release >> rpms docker: - image: centos:<< parameters.release >> environment: @@ -358,12 +376,19 @@ jobs: yum-builddep -y "$DIST_DIR"/redhat/varnish.spec rpmbuild -bs "$DIST_DIR"/redhat/varnish.spec rpmbuild --rebuild "$RESULT_DIR"/varnish-*.src.rpm + - run: + name: Prepare the packages for storage + command: | + mkdir -p packages/<< parameters.dist >>/<< parameters.release >> + mv rpms/*/*.rpm packages/<< parameters.dist >>/<< parameters.release >> - persist_to_workspace: root: . paths: - - rpms/*.rpm - - rpms/*/*.rpm - push_packages: + - packages/<< parameters.dist >>/<< parameters.release >>/*.rpm + - store_artifacts: + path: packages/<< parameters.dist >>/<< parameters.release >> + destination: ./ + collect_packages: docker: - image: centos:7 steps: @@ -372,7 +397,7 @@ jobs: - run: name: Tar the packages command: | - tar cvzf packages.tar.gz rpms/*.rpm debs/*.deb debs/*.dsc apks/*.apk + tar cvzf packages.tar.gz packages - store_artifacts: destination: packages.tar.gz path: packages.tar.gz @@ -409,16 +434,18 @@ workflows: <<: *pkg_req - build_rpms: name: build_centos_7 + dist: centos release: "7" <<: *pkg_req - build_rpms: name: build_centos_8 + dist: centos release: "8" <<: *pkg_req - build_apks: name: build_alpine <<: *pkg_req - - push_packages: + - collect_packages: requires: - build_debian_stretch - build_debian_buster From guillaume at varnish-software.com Mon Feb 3 21:14:10 2020 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 3 Feb 2020 21:14:10 +0000 (UTC) Subject: [master] 32f160a46 [cci] unneeded env var Message-ID: <20200203211410.950BF6143A@lists.varnish-cache.org> commit 32f160a46be44da6cdffa69a1e8b064aa3c9cfc6 Author: Guillaume Quintard Date: Mon Feb 3 11:53:37 2020 -0800 [cci] unneeded env var diff --git a/.circleci/config.yml b/.circleci/config.yml index 74a8c0cbe..44fee78dc 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -313,7 +313,6 @@ jobs: - image: centos:<< parameters.release >> environment: DIST_DIR: build - DIST: el7 steps: - run: name: Install packaging tools From martin at varnish-software.com Tue Feb 4 10:01:07 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:01:07 +0000 (UTC) Subject: [master] f8b9667f2 Handle out of session workspace in http1_new_session() Message-ID: <20200204100107.14CB6A2A72@lists.varnish-cache.org> commit f8b9667f23c9fd90c5e2260409f82f5f92f4b007 Author: Martin Blix Grydeland Date: Thu Dec 12 13:12:48 2019 +0100 Handle out of session workspace in http1_new_session() If proxy protocol is in use, it is possible to fill the session workspace exactly before entering http1_new_session(), which will cause it to assert when calling SES_Reserve_proto_priv(). with this patch we will close the session gracefully. diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c index 4668c5f8c..81ab2e3f6 100644 --- a/bin/varnishd/http1/cache_http1_fsm.c +++ b/bin/varnishd/http1/cache_http1_fsm.c @@ -114,7 +114,15 @@ http1_new_session(struct worker *wrk, void *arg) CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); HTC_RxInit(req->htc, req->ws); - XXXAN(SES_Reserve_proto_priv(sp, &u)); + if (!SES_Reserve_proto_priv(sp, &u)) { + /* Out of session workspace. Free the req, close the sess, + * and do not set a new task func, which will exit the + * worker thread. */ + VSL(SLT_Error, req->sp->vxid, "insufficient workspace"); + Req_Release(req); + SES_Delete(sp, SC_RX_JUNK, NAN); + return; + } http1_setstate(sp, H1NEWREQ); wrk->task.func = http1_req; wrk->task.priv = req; diff --git a/bin/varnishtest/tests/f00005.vtc b/bin/varnishtest/tests/f00005.vtc new file mode 100644 index 000000000..866d941dd --- /dev/null +++ b/bin/varnishtest/tests/f00005.vtc @@ -0,0 +1,64 @@ +varnishtest "proxy ws panic" + +server s1 { + rxreq + txresp +} -start + +varnish v1 -proto "PROXY" -vcl+backend {}-start + +# Too large proxy payload using TLV +client c1 { + sendhex { +0d 0a 0d 0a 00 0d 0a 51 +55 49 54 0a 21 21 00 93 +aa bb cc dd ee ff 11 22 +33 44 55 66 77 88 99 aa +bb cc dd ee ff 11 22 33 +44 55 66 77 88 99 aa bb +88 da 0d 73 02 00 3c 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 20 00 2d 01 01 +00 00 00 21 00 07 54 4c +53 76 31 2e 32 23 00 1b +45 43 44 48 45 2d 52 53 +41 2d 41 45 53 32 35 36 +2d 47 43 4d 2d 53 48 41 +33 38 34 + } + expect_close +} -run + +# Reduced size proxy payload to verify Varnish is still running +client c1 { + sendhex { +0d 0a 0d 0a 00 0d 0a 51 +55 49 54 0a 21 21 00 8b +aa bb cc dd ee ff 11 22 +33 44 55 66 77 88 99 aa +bb cc dd ee ff 11 22 33 +44 55 66 77 88 99 aa bb +88 da 0d 73 02 00 34 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 20 00 2d 01 01 +00 00 00 21 00 07 54 4c +53 76 31 2e 32 23 00 1b +45 43 44 48 45 2d 52 53 +41 2d 41 45 53 32 35 36 +2d 47 43 4d 2d 53 48 41 +33 38 34 + } + txreq + rxresp +} -run From martin at varnish-software.com Tue Feb 4 10:01:07 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:01:07 +0000 (UTC) Subject: [master] 613fa4334 Remove extra call to SES_Reserve_proto_priv Message-ID: <20200204100107.288E0A2A75@lists.varnish-cache.org> commit 613fa4334b50889b82304e6d75f76e2686101b1b Author: Martin Blix Grydeland Date: Thu Dec 12 13:16:40 2019 +0100 Remove extra call to SES_Reserve_proto_priv In h2_init_sess, an extra call was always made to SES_Reseve_proto_priv(), even though it was already reserved. This wasted a pointer worth of session workspace. This patch removes the extra call. diff --git a/bin/varnishd/http2/cache_http2_session.c b/bin/varnishd/http2/cache_http2_session.c index 618f26b33..4cf946546 100644 --- a/bin/varnishd/http2/cache_http2_session.c +++ b/bin/varnishd/http2/cache_http2_session.c @@ -132,7 +132,6 @@ h2_init_sess(const struct worker *wrk, struct sess *sp, AZ(VHT_Init(h2->dectbl, h2->local_settings.header_table_size)); - XXXAN(SES_Reserve_proto_priv(sp, &up)); *up = (uintptr_t)h2; } AN(up); From martin at varnish-software.com Tue Feb 4 10:01:07 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:01:07 +0000 (UTC) Subject: [master] 05027d746 Remove call to SES_Reserve_proto_priv in h2_init_sess Message-ID: <20200204100107.3F474A2A79@lists.varnish-cache.org> commit 05027d746a11fa3010e2d5f59f96dc97ca3b640b Author: Martin Blix Grydeland Date: Thu Dec 12 13:51:17 2019 +0100 Remove call to SES_Reserve_proto_priv in h2_init_sess h2_init_sess can only be reached through H1 with either previous knowledge or opportunistic upgrade. Because of this the proto_priv session attribute will always be set before entry. This patch simplifies and removes dead code containing a call to SES_Reserve_proto_priv. Note: Better diff with the --ignore-all-space option diff --git a/bin/varnishd/http2/cache_http2_session.c b/bin/varnishd/http2/cache_http2_session.c index 4cf946546..fec88569b 100644 --- a/bin/varnishd/http2/cache_http2_session.c +++ b/bin/varnishd/http2/cache_http2_session.c @@ -101,41 +101,37 @@ h2_init_sess(const struct worker *wrk, struct sess *sp, uintptr_t *up; struct h2_sess *h2; - if (SES_Get_proto_priv(sp, &up)) { - /* Already reserved if we came via H1 */ - XXXAN(SES_Reserve_proto_priv(sp, &up)); - *up = 0; - } - if (*up == 0) { - if (srq == NULL) - srq = Req_New(wrk, sp); - AN(srq); - h2 = h2s; - AN(h2); - INIT_OBJ(h2, H2_SESS_MAGIC); - h2->srq = srq; - h2->htc = srq->htc; - h2->ws = srq->ws; - h2->vsl = srq->vsl; - VSL_Flush(h2->vsl, 0); - h2->vsl->wid = sp->vxid; - h2->htc->rfd = &sp->fd; - h2->sess = sp; - h2->rxthr = pthread_self(); - AZ(pthread_cond_init(h2->winupd_cond, NULL)); - VTAILQ_INIT(&h2->streams); - VTAILQ_INIT(&h2->txqueue); - h2_local_settings(&h2->local_settings); - h2->remote_settings = H2_proto_settings; - h2->decode = decode; - - AZ(VHT_Init(h2->dectbl, - h2->local_settings.header_table_size)); - - *up = (uintptr_t)h2; - } - AN(up); - CAST_OBJ_NOTNULL(h2, (void*)(*up), H2_SESS_MAGIC); + /* proto_priv session attribute will always have been set up by H1 + * before reaching here. */ + AZ(SES_Get_proto_priv(sp, &up)); + assert(*up == 0); + + if (srq == NULL) + srq = Req_New(wrk, sp); + AN(srq); + h2 = h2s; + AN(h2); + INIT_OBJ(h2, H2_SESS_MAGIC); + h2->srq = srq; + h2->htc = srq->htc; + h2->ws = srq->ws; + h2->vsl = srq->vsl; + VSL_Flush(h2->vsl, 0); + h2->vsl->wid = sp->vxid; + h2->htc->rfd = &sp->fd; + h2->sess = sp; + h2->rxthr = pthread_self(); + AZ(pthread_cond_init(h2->winupd_cond, NULL)); + VTAILQ_INIT(&h2->streams); + VTAILQ_INIT(&h2->txqueue); + h2_local_settings(&h2->local_settings); + h2->remote_settings = H2_proto_settings; + h2->decode = decode; + + AZ(VHT_Init(h2->dectbl, h2->local_settings.header_table_size)); + + *up = (uintptr_t)h2; + return (h2); } From martin at varnish-software.com Tue Feb 4 10:01:07 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:01:07 +0000 (UTC) Subject: [master] 618e9f92b Handle badly formatted proxy TLVs Message-ID: <20200204100107.58557A2A7E@lists.varnish-cache.org> commit 618e9f92b13b25d5c5549a0cfd9284c063d1c1d1 Author: Martin Blix Grydeland Date: Thu Dec 12 14:53:48 2019 +0100 Handle badly formatted proxy TLVs Proxy TLVs claiming to have PP2_TYPE_SSL sub-TLVs without complete payload would cause a Varnish assert. This patch fixes the parsing of the TLVs. diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c index 7efe6cc2f..3cc0f48a0 100644 --- a/bin/varnishd/proxy/cache_proxy_proto.c +++ b/bin/varnishd/proxy/cache_proxy_proto.c @@ -276,8 +276,9 @@ vpx_tlv_itern(struct vpx_tlv_iter *vpi) return (1); } -#define VPX_TLV_FOREACH(ptr, len, itv) \ - for(vpx_tlv_iter0(itv, ptr, len); vpx_tlv_itern(itv);) +#define VPX_TLV_FOREACH(ptr, len, itv) \ + for (vpx_tlv_iter0(itv, ptr, len); \ + (vpi->e == NULL) && vpx_tlv_itern(itv);) int VPX_tlv(const struct req *req, int typ, void **dst, int *len) @@ -453,6 +454,10 @@ vpx_proto2(const struct worker *wrk, struct req *req) VPX_TLV_FOREACH(d, l, vpi) { if (vpi->t == PP2_TYPE_SSL) { + if (vpi->l < 5) { + vpi->e = "Length Error"; + break; + } VPX_TLV_FOREACH((char*)vpi->p + 5, vpi->l - 5, vpi2) { } vpi->e = vpi2->e; diff --git a/bin/varnishtest/tests/f00005.vtc b/bin/varnishtest/tests/f00005.vtc index 866d941dd..b1b097fce 100644 --- a/bin/varnishtest/tests/f00005.vtc +++ b/bin/varnishtest/tests/f00005.vtc @@ -35,6 +35,18 @@ bb cc dd ee ff 11 22 33 expect_close } -run +# Badly formatted TLV proxy payload +client c1 { + sendhex { +0d 0a 0d 0a 00 0d 0a 51 +55 49 54 0a +21 11 00 13 00 ff 20 ff 10 ff 03 21 20 30 00 20 +20 00 00 19 00 02 29 20 00 00 00 41 20 9e 15 15 +d6 00 00 08 00 00 00 00 00 07 7a 20 b1 3f 43 20 + } + expect_close +} -run + # Reduced size proxy payload to verify Varnish is still running client c1 { sendhex { From martin at varnish-software.com Tue Feb 4 10:01:07 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:01:07 +0000 (UTC) Subject: [master] 2d8fc1a78 Take sizeof pool_task into account when reserving WS in SES_Wait Message-ID: <20200204100107.74919A2A82@lists.varnish-cache.org> commit 2d8fc1a784a1e26d78c30174923a2b14ee2ebf62 Author: Martin Blix Grydeland Date: Mon Dec 16 17:04:43 2019 +0100 Take sizeof pool_task into account when reserving WS in SES_Wait The assert on WS_ReserveSize() in ses_handle() can not trip because sizeof (struct pool_task) is less than sizeof (struct waited). But to safe guard against future problems if that were to change, this patch makes sure that the session workspace can hold the largest of them before entering the waiter, erroring out if not. diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index 1ba60742f..973304b88 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -419,6 +419,7 @@ ses_handle(struct waited *wp, enum wait_event ev, vtim_real now) wp->magic = 0; wp = NULL; + /* The WS was reserved in SES_Wait() */ WS_Release(sp->ws, 0); switch (ev) { @@ -431,6 +432,7 @@ ses_handle(struct waited *wp, enum wait_event ev, vtim_real now) case WAITER_ACTION: pp = sp->pool; CHECK_OBJ_NOTNULL(pp, POOL_MAGIC); + /* SES_Wait() guarantees the next will not assert. */ assert(sizeof *tp <= WS_ReserveSize(sp->ws, sizeof *tp)); tp = (void*)sp->ws->f; tp->func = xp->unwait; @@ -454,6 +456,7 @@ SES_Wait(struct sess *sp, const struct transport *xp) { struct pool *pp; struct waited *wp; + unsigned u; CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); CHECK_OBJ_NOTNULL(xp, TRANSPORT_MAGIC); @@ -467,10 +470,15 @@ SES_Wait(struct sess *sp, const struct transport *xp) VTCP_nonblocking(sp->fd); /* - * put struct waited on the workspace + * Put struct waited on the workspace. Make sure that the + * workspace can hold enough space for the largest of struct + * waited and pool_task, as pool_task will be needed when coming + * off the waiter again. */ - if (WS_ReserveSize(sp->ws, sizeof(struct waited)) - < sizeof(struct waited)) { + u = sizeof (struct waited); + if (sizeof (struct pool_task) > u) + u = sizeof (struct pool_task); + if (!WS_ReserveSize(sp->ws, u)) { SES_Delete(sp, SC_OVERLOAD, NAN); return; } From martin at varnish-software.com Tue Feb 4 10:02:07 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:02:07 +0000 (UTC) Subject: [6.0] 2081eeb6e Add WS_ReserveAll() to replace WS_Reserve(ws, 0) Message-ID: <20200204100207.9ECF4A317D@lists.varnish-cache.org> commit 2081eeb6e4ae64f95e2c6b9abdac7ca8a99bc5d2 Author: Nils Goroll Date: Mon Apr 8 15:19:30 2019 +0200 Add WS_ReserveAll() to replace WS_Reserve(ws, 0) ... to un-confuse the interface Notes on changes from WS_Reserve(): * Removed the first WS_Assert because all we change is ws->r and we got a specific assert on it. * it follows from PAOK(ws->e) && PAOK(ws->f) in WS_Assert() that PAOK(ws->r) && PAOK(b), so we remove the PRNDDN() Ref: varnishcache/varnish-cache#2967 (cherry picked from commit d001cdd2045aa484a968d09859cddf59dee372cb) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 6dbefa84e..4455277b4 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -762,6 +762,7 @@ void WRK_BgThread(pthread_t *thr, const char *name, bgthread_t *func, void WS_Init(struct ws *ws, const char *id, void *space, unsigned len); unsigned WS_Reserve(struct ws *ws, unsigned bytes); +unsigned WS_ReserveAll(struct ws *); unsigned WS_ReserveLumps(struct ws *ws, size_t sz); void WS_MarkOverflow(struct ws *ws); void WS_Release(struct ws *ws, unsigned bytes); diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c index 0bd31f2ca..9f1558aa0 100644 --- a/bin/varnishd/cache/cache_hash.c +++ b/bin/varnishd/cache/cache_hash.c @@ -626,7 +626,7 @@ double grace, double keep) CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC); - ospc = WS_Reserve(wrk->aws, 0); + ospc = WS_ReserveAll(wrk->aws); assert(ospc >= sizeof *ocp); /* * Because of "soft" purges, there might be oc's in the list that has diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c index 070ead2e6..78d96293b 100644 --- a/bin/varnishd/cache/cache_http.c +++ b/bin/varnishd/cache/cache_http.c @@ -374,7 +374,7 @@ http_CollectHdrSep(struct http *hp, const char *hdr, const char *sep) } if (b == NULL) { /* Found second header, start our collection */ - ml = WS_Reserve(hp->ws, 0); + ml = WS_ReserveAll(hp->ws); b = hp->ws->f; e = b + ml; x = Tlen(hp->hd[f]); @@ -1208,7 +1208,7 @@ http_PrintfHeader(struct http *to, const char *fmt, ...) unsigned l, n; CHECK_OBJ_NOTNULL(to, HTTP_MAGIC); - l = WS_Reserve(to->ws, 0); + l = WS_ReserveAll(to->ws); va_start(ap, fmt); n = vsnprintf(to->ws->f, l, fmt, ap); va_end(ap); diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index a8a75b7e7..e18accc57 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -204,7 +204,7 @@ HTC_RxInit(struct http_conn *htc, struct ws *ws) CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC); htc->ws = ws; - (void)WS_Reserve(htc->ws, 0); + (void)WS_ReserveAll(htc->ws); htc->rxbuf_b = ws->f; htc->rxbuf_e = ws->f; if (htc->pipeline_b != NULL) { diff --git a/bin/varnishd/cache/cache_vary.c b/bin/varnishd/cache/cache_vary.c index 46d6930fb..4af92f69c 100644 --- a/bin/varnishd/cache/cache_vary.c +++ b/bin/varnishd/cache/cache_vary.c @@ -230,7 +230,7 @@ VRY_Prep(struct req *req) AZ(req->vary_b); AZ(req->vary_l); AZ(req->vary_e); - (void)WS_Reserve(req->ws, 0); + (void)WS_ReserveAll(req->ws); } else { AN(req->ws->r); } diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c index 4b49e1d3a..343481346 100644 --- a/bin/varnishd/cache/cache_vrt.c +++ b/bin/varnishd/cache/cache_vrt.c @@ -271,7 +271,7 @@ VRT_String(struct ws *ws, const char *h, const char *p, va_list ap) unsigned u, x; va_list aq; - u = WS_Reserve(ws, 0); + u = WS_ReserveAll(ws); e = b = ws->f; e += u; @@ -395,7 +395,7 @@ VRT_StrandsWS(struct ws *ws, const char *h, VCL_STRANDS s) int i; AN(s); - u = WS_Reserve(ws, 0); + u = WS_ReserveAll(ws); for (i = 0; i < s->n; i++) if (s->p[i] != NULL && *s->p[i] != '\0') { @@ -578,7 +578,7 @@ VRT_IP_string(VRT_CTX, VCL_IP ip) CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); if (ip == NULL) return (NULL); - len = WS_Reserve(ctx->ws, 0); + len = WS_ReserveAll(ctx->ws); if (len == 0) { WS_Release(ctx->ws, 0); return (NULL); diff --git a/bin/varnishd/cache/cache_vrt_re.c b/bin/varnishd/cache/cache_vrt_re.c index 20a0cf6fc..449f07126 100644 --- a/bin/varnishd/cache/cache_vrt_re.c +++ b/bin/varnishd/cache/cache_vrt_re.c @@ -127,7 +127,7 @@ VRT_regsub(VRT_CTX, int all, const char *str, void *re, return(str); } - u = WS_Reserve(ctx->ws, 0); + u = WS_ReserveAll(ctx->ws); res_e = res_b = b0 = ctx->ws->f; res_e += u; diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c index 3ec5ec017..e0a9b1d48 100644 --- a/bin/varnishd/cache/cache_ws.c +++ b/bin/varnishd/cache/cache_ws.c @@ -202,7 +202,7 @@ WS_Printf(struct ws *ws, const char *fmt, ...) va_list ap; char *p; - u = WS_Reserve(ws, 0); + u = WS_ReserveAll(ws); p = ws->f; va_start(ap, fmt); v = vsnprintf(p, u, fmt, ap); @@ -227,6 +227,30 @@ WS_Snapshot(struct ws *ws) return (ws->f == ws->s ? 0 : (uintptr_t)ws->f); } +/* + * WS_Release() must be called in all cases + */ +unsigned +WS_ReserveAll(struct ws *ws) +{ + unsigned b; + + assert(ws->r == NULL); + + ws->r = ws->e; + b = pdiff(ws->f, ws->r); + + WS_Assert(ws); + DSL(DBG_WORKSPACE, 0, "WS_ReserveAll(%p) = %u", ws, b); + + return (b); +} + +/* + * bytes == 0 argument is deprecated - use WS_ReserveAll + * + * XXX rename to WS_ReserveSize and macro-wrap WS_Reserve to emit #warn ? + */ unsigned WS_Reserve(struct ws *ws, unsigned bytes) { @@ -253,11 +277,7 @@ WS_Reserve(struct ws *ws, unsigned bytes) unsigned WS_ReserveLumps(struct ws *ws, size_t sz) { - unsigned u; - - u = WS_Reserve(ws, 0); - u /= sz; - return (u); + return (WS_ReserveAll(ws) / sz); } void diff --git a/bin/varnishd/http2/cache_http2_deliver.c b/bin/varnishd/http2/cache_http2_deliver.c index 46f7654d9..94018edd7 100644 --- a/bin/varnishd/http2/cache_http2_deliver.c +++ b/bin/varnishd/http2/cache_http2_deliver.c @@ -229,7 +229,7 @@ h2_build_headers(struct vsb *resp, struct req *req) uint8_t buf[6]; ssize_t sz, sz1; - l = WS_Reserve(req->ws, 0); + l = WS_ReserveAll(req->ws); AN(VSB_new(resp, req->ws->f, l, VSB_FIXEDLEN)); if (l < 10) { WS_Release(req->ws, 0); diff --git a/bin/varnishd/http2/cache_http2_hpack.c b/bin/varnishd/http2/cache_http2_hpack.c index f2ef11cf7..d43262916 100644 --- a/bin/varnishd/http2/cache_http2_hpack.c +++ b/bin/varnishd/http2/cache_http2_hpack.c @@ -175,7 +175,7 @@ h2h_decode_init(const struct h2_sess *h2) d = h2->decode; INIT_OBJ(d, H2H_DECODE_MAGIC); VHD_Init(d->vhd); - d->out_l = WS_Reserve(h2->new_req->http->ws, 0); + d->out_l = WS_ReserveAll(h2->new_req->http->ws); /* * Can't do any work without any buffer * space. Require non-zero size. diff --git a/lib/libvmod_blob/vmod_blob.c b/lib/libvmod_blob/vmod_blob.c index dfb03cafa..0deb37f08 100644 --- a/lib/libvmod_blob/vmod_blob.c +++ b/lib/libvmod_blob/vmod_blob.c @@ -348,7 +348,7 @@ vmod_decode(VRT_CTX, VCL_ENUM decs, VCL_INT length, VCL_STRANDS strings) } buf = WS_Front(ctx->ws); - space = WS_Reserve(ctx->ws, 0); + space = WS_ReserveAll(ctx->ws); if (length <= 0) length = -1; @@ -389,7 +389,7 @@ encode(VRT_CTX, enum encoding enc, enum case_e kase, VCL_BLOB b) CHECK_OBJ_NOTNULL(ctx->ws, WS_MAGIC); snap = WS_Snapshot(ctx->ws); buf = WS_Front(ctx->ws); - space = WS_Reserve(ctx->ws, 0); + space = WS_ReserveAll(ctx->ws); len = func[enc].encode(enc, kase, buf, space, b->priv, b->len); diff --git a/lib/libvmod_std/vmod_std.c b/lib/libvmod_std/vmod_std.c index ef801f11b..f36f4bc1c 100644 --- a/lib/libvmod_std/vmod_std.c +++ b/lib/libvmod_std/vmod_std.c @@ -74,7 +74,7 @@ vmod_updown(VRT_CTX, int up, const char *s, va_list ap) const char *p; CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - u = WS_Reserve(ctx->ws, 0); + u = WS_ReserveAll(ctx->ws); e = b = ctx->ws->f; e += u; p = s; diff --git a/lib/libvmod_vtc/vmod_vtc.c b/lib/libvmod_vtc/vmod_vtc.c index 197fe81f0..18374e8a5 100644 --- a/lib/libvmod_vtc/vmod_vtc.c +++ b/lib/libvmod_vtc/vmod_vtc.c @@ -156,7 +156,7 @@ vmod_workspace_alloc(VRT_CTX, VCL_ENUM which, VCL_INT size) WS_Assert(ws); if (size < 0) { - size += WS_Reserve(ws, 0); + size += WS_ReserveAll(ws); WS_Release(ws, 0); } if (size <= 0) { @@ -183,7 +183,7 @@ vmod_workspace_free(VRT_CTX, VCL_ENUM which) return(-1); WS_Assert(ws); - u = WS_Reserve(ws, 0); + u = WS_ReserveAll(ws); WS_Release(ws, 0); return (u); } From martin at varnish-software.com Tue Feb 4 10:02:07 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:02:07 +0000 (UTC) Subject: [6.0] fa3c8df3a Add a v_* attribute for deprecated functions Message-ID: <20200204100207.B861DA3186@lists.varnish-cache.org> commit fa3c8df3afe8d69572265c27dd9e3a198e3506a6 Author: Nils Goroll Date: Mon Apr 8 16:11:30 2019 +0200 Add a v_* attribute for deprecated functions This works with gcc 6.3.0 and clang 3.8.1-24 The test for __GNUC__ is deliberately simple and might not catch all compilers which would potentially support deprecation marks. While more specifics could be added, the aim is to raise awareness with developers and we consider it quite unlikely that anyone does not compile with one of the main stream compilers at all. (cherry picked from commit 1594037cb896970c2cbfdab101d0059ac00201f9) diff --git a/include/vdef.h b/include/vdef.h index 3a5e40329..f76407f80 100644 --- a/include/vdef.h +++ b/include/vdef.h @@ -83,6 +83,12 @@ #define v_noreturn_ __attribute__((__noreturn__)) +#ifdef __GNUC__ +# define v_deprecated_ __attribute__((deprecated)) +#else +# define v_deprecated_ +#endif + /********************************************************************* * Pointer alignment magic */ From martin at varnish-software.com Tue Feb 4 10:02:07 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:02:07 +0000 (UTC) Subject: [6.0] 534e045e0 Deprecate WS_Reserve() and replace it with WS_ReserveSize() Message-ID: <20200204100207.D0573A318A@lists.varnish-cache.org> commit 534e045e040fc213f72aa38a85806f152e3ef5e0 Author: Nils Goroll Date: Mon Apr 8 16:26:07 2019 +0200 Deprecate WS_Reserve() and replace it with WS_ReserveSize() WS_ReserveSize() does not leave the workspace reserved when the reservation fails, so WS_Release() must be called for retval > 0 only. Besides the debug string, it is identical to WS_Reserve() except for assert(bytes > 0); Follow-up varnishcache/varnish-cache#2967 Note: The WS_Reserve() function has not been deprecated as that can potentially create problems for the build process of VMODs. (cherry picked from commit 4e33359772a3d751b2ef4b5c4b40259f1bcd6903) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 4455277b4..9e47c7215 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -761,7 +761,10 @@ void WRK_BgThread(pthread_t *thr, const char *name, bgthread_t *func, /* cache_ws.c */ void WS_Init(struct ws *ws, const char *id, void *space, unsigned len); + +/* WS_Reserve(): Use WS_ReserveSize() or WS_ReserveAll() */ unsigned WS_Reserve(struct ws *ws, unsigned bytes); +unsigned WS_ReserveSize(struct ws *, unsigned); unsigned WS_ReserveAll(struct ws *); unsigned WS_ReserveLumps(struct ws *ws, size_t sz); void WS_MarkOverflow(struct ws *ws); diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index e18accc57..8c9325515 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -113,7 +113,7 @@ ses_reserve_attr(struct sess *sp, enum sess_attr a, void **dst, int sz) assert(a < SA_LAST); assert(sz >= 0); AN(dst); - o = WS_Reserve(sp->ws, sz); + o = WS_ReserveSize(sp->ws, sz); assert(o >= sz); *dst = sp->ws->f; o = sp->ws->f - sp->ws->s; @@ -417,7 +417,7 @@ ses_handle(struct waited *wp, enum wait_event ev, vtim_real now) case WAITER_ACTION: pp = sp->pool; CHECK_OBJ_NOTNULL(pp, POOL_MAGIC); - assert(sizeof *tp <= WS_Reserve(sp->ws, sizeof *tp)); + assert(sizeof *tp <= WS_ReserveSize(sp->ws, sizeof *tp)); tp = (void*)sp->ws->f; tp->func = xp->unwait; tp->priv = sp; @@ -455,7 +455,7 @@ SES_Wait(struct sess *sp, const struct transport *xp) /* * put struct waited on the workspace */ - if (WS_Reserve(sp->ws, sizeof(struct waited)) + if (WS_ReserveSize(sp->ws, sizeof(struct waited)) < sizeof(struct waited)) { SES_Delete(sp, SC_OVERLOAD, NAN); return; diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c index 14bbfd4fd..fa3fa366a 100644 --- a/bin/varnishd/cache/cache_wrk.c +++ b/bin/varnishd/cache/cache_wrk.c @@ -239,7 +239,7 @@ Pool_Task_Arg(struct worker *wrk, enum task_prio prio, task_func_t *func, retval = 0; } AZ(wrk2->task.func); - assert(arg_len <= WS_Reserve(wrk2->aws, arg_len)); + assert(arg_len <= WS_ReserveSize(wrk2->aws, arg_len)); memcpy(wrk2->aws->f, arg, arg_len); wrk2->task.func = func; wrk2->task.priv = wrk2->aws->f; diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c index e0a9b1d48..7b8df6d23 100644 --- a/bin/varnishd/cache/cache_ws.c +++ b/bin/varnishd/cache/cache_ws.c @@ -247,9 +247,34 @@ WS_ReserveAll(struct ws *ws) } /* - * bytes == 0 argument is deprecated - use WS_ReserveAll - * - * XXX rename to WS_ReserveSize and macro-wrap WS_Reserve to emit #warn ? + * WS_Release() must be called for retval > 0 only + */ +unsigned +WS_ReserveSize(struct ws *ws, unsigned bytes) +{ + unsigned b2; + + WS_Assert(ws); + assert(ws->r == NULL); + assert(bytes > 0); + + b2 = PRNDDN(ws->e - ws->f); + if (bytes < b2) + b2 = PRNDUP(bytes); + + if (ws->f + b2 > ws->e) { + WS_MarkOverflow(ws); + return (0); + } + ws->r = ws->f + b2; + DSL(DBG_WORKSPACE, 0, "WS_ReserveSize(%p, %u/%u) = %u", + ws, b2, bytes, pdiff(ws->f, ws->r)); + WS_Assert(ws); + return (pdiff(ws->f, ws->r)); +} + +/* + * XXX remove for 2020-03-15 release */ unsigned WS_Reserve(struct ws *ws, unsigned bytes) diff --git a/lib/libvmod_proxy/vmod_proxy.c b/lib/libvmod_proxy/vmod_proxy.c index fa7b9617e..58f2db295 100644 --- a/lib/libvmod_proxy/vmod_proxy.c +++ b/lib/libvmod_proxy/vmod_proxy.c @@ -105,7 +105,7 @@ tlv_string(VRT_CTX, int tlv) if (VPX_tlv(ctx->req, tlv, (void **)&dst, &len)) return (NULL); - if (!WS_Reserve(ctx->ws, len+1)) + if (!WS_ReserveSize(ctx->ws, len+1)) return (NULL); d = ctx->ws->f; memcpy(d, dst, len); From martin at varnish-software.com Tue Feb 4 10:02:07 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:02:07 +0000 (UTC) Subject: [6.0] 95c9fc6f7 add a facility to test WS_ReserveSize() Message-ID: <20200204100207.EB11BA3190@lists.varnish-cache.org> commit 95c9fc6f709478e90457f67fa13277fe8d59574e Author: Nils Goroll Date: Tue Nov 26 13:21:37 2019 +0100 add a facility to test WS_ReserveSize() (cherry picked from commit ed3b095c0414af453493580a654f63a02d22e0e8) diff --git a/lib/libvmod_vtc/vmod.vcc b/lib/libvmod_vtc/vmod.vcc index 248f4b2d6..a846a5fcf 100644 --- a/lib/libvmod_vtc/vmod.vcc +++ b/lib/libvmod_vtc/vmod.vcc @@ -96,6 +96,14 @@ as much as needed to leave that many bytes free. The actual allocation size may be higher to comply with memory alignment requirements of the CPU architecture. A failed allocation fails the transaction. +$Function BYTES workspace_reserve(ENUM { client, backend, session, thread }, + INT size) + +Attempt to reserve *size* bytes and release the reservation right +away. Return the size of the reservation. + +See `vtc.workspace_alloc()` for semantics of the *size* argument. + $Function INT workspace_free(ENUM { client, backend, session, thread }) Find how much unallocated space there is left in a workspace. diff --git a/lib/libvmod_vtc/vmod_vtc.c b/lib/libvmod_vtc/vmod_vtc.c index 18374e8a5..ce717db5a 100644 --- a/lib/libvmod_vtc/vmod_vtc.c +++ b/lib/libvmod_vtc/vmod_vtc.c @@ -170,6 +170,34 @@ vmod_workspace_alloc(VRT_CTX, VCL_ENUM which, VCL_INT size) memset(p, '\0', size); } +VCL_BYTES v_matchproto_(td_vtc_workspace_reserve) +vmod_workspace_reserve(VRT_CTX, VCL_ENUM which, VCL_INT size) +{ + struct ws *ws; + unsigned r; + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + + ws = vtc_ws_find(ctx, which); + if (ws == NULL) + return (0); + WS_Assert(ws); + + if (size < 0) { + size += WS_ReserveAll(ws); + WS_Release(ws, 0); + } + if (size <= 0) { + VRT_fail(ctx, "Attempted negative WS reservation"); + return (0); + } + r = WS_ReserveSize(ws, size); + if (r == 0) + return (0); + WS_Release(ws, 0); + return (1); +} + VCL_INT v_matchproto_(td_vtc_workspace_free) vmod_workspace_free(VRT_CTX, VCL_ENUM which) { From martin at varnish-software.com Tue Feb 4 10:02:08 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:02:08 +0000 (UTC) Subject: [6.0] c3fa617d4 WS_ReserveSize() must not hold a reservation for zero return value Message-ID: <20200204100208.125AAA3194@lists.varnish-cache.org> commit c3fa617d4b2ae0b7b61bb9e36ffa100a8adb5631 Author: Nils Goroll Date: Tue Nov 26 13:27:09 2019 +0100 WS_ReserveSize() must not hold a reservation for zero return value This originates from a3d47c258fb7938f67a053f6d041257edb69afe9, but was overlooked in 4e33359772a3d751b2ef4b5c4b40259f1bcd6903: When there is insufficient space to fulfil the reservation request, we must not leave the workspace reserved. Fixes varnishcache/varnish-cache#3131 (cherry picked from commit 505b7bd9643006fa8e3977f920564ce12a2b24a2) diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c index 7b8df6d23..e98e2cc69 100644 --- a/bin/varnishd/cache/cache_ws.c +++ b/bin/varnishd/cache/cache_ws.c @@ -262,7 +262,7 @@ WS_ReserveSize(struct ws *ws, unsigned bytes) if (bytes < b2) b2 = PRNDUP(bytes); - if (ws->f + b2 > ws->e) { + if (ws->f + b2 >= ws->e) { WS_MarkOverflow(ws); return (0); } diff --git a/bin/varnishtest/tests/r03131.vtc b/bin/varnishtest/tests/r03131.vtc new file mode 100644 index 000000000..2722b748a --- /dev/null +++ b/bin/varnishtest/tests/r03131.vtc @@ -0,0 +1,24 @@ +varnishtest "Test workspace functions in vmod_vtc" + +varnish v1 -vcl { + import vtc; + import std; + + backend dummy {.host = "${bad_backend}"; } + + sub vcl_recv { + return (synth(200)); + } + + sub vcl_synth { + set resp.http.res1 = vtc.workspace_reserve(client, 1024 * 1024); + vtc.workspace_alloc(client, -1); + set resp.http.res2 = vtc.workspace_reserve(client, 8); + set resp.http.res3 = vtc.workspace_reserve(client, 8); + } +} -start + +client c1 { + txreq + rxresp +} -run From martin at varnish-software.com Tue Feb 4 10:02:08 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:02:08 +0000 (UTC) Subject: [6.0] 752d360ae fix copy-pasta vtc description Message-ID: <20200204100208.2AED1A31A4@lists.varnish-cache.org> commit 752d360ae4cf6793ed0691c5b9cc046bceb31b9d Author: Nils Goroll Date: Tue Nov 26 14:09:32 2019 +0100 fix copy-pasta vtc description (cherry picked from commit 815331b3aecca0eb7e4e29f6dd4d778124c2ac9d) diff --git a/bin/varnishtest/tests/r03131.vtc b/bin/varnishtest/tests/r03131.vtc index 2722b748a..43f0995b0 100644 --- a/bin/varnishtest/tests/r03131.vtc +++ b/bin/varnishtest/tests/r03131.vtc @@ -1,4 +1,4 @@ -varnishtest "Test workspace functions in vmod_vtc" +varnishtest "Test WS_ReserveSize() overflow behavior" varnish v1 -vcl { import vtc; From martin at varnish-software.com Tue Feb 4 10:02:08 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:02:08 +0000 (UTC) Subject: [6.0] efaa633ca Add Session Attribute workspace overflow handling Message-ID: <20200204100208.44591A31B4@lists.varnish-cache.org> commit efaa633ca71ad09bd7f68c5d7fb1886b15a92d1e Author: Nils Goroll Date: Tue Nov 26 14:56:24 2019 +0100 Add Session Attribute workspace overflow handling Notes: * for the acceptor, I think it makes sense to keep AN assertion (pun!) because varnish is not viable if the session workspace is too small to even hold the attributes initialized in the acceptor. If this was an issue, we should rather revisit the minimum values for the session workspace * for h1 and h2 session setup, I have used XXXAN() because I am not sure how we should best handle allocation failures. * The relevant bit, for now, is the proxy code which may allocate arbitrarily long TLV attributes, so this is the code for which we now actually handle errors and test that we do On the vtc: I added the test to o00005.vtc because there existed a previous overflow test from 267504b8143bdb8ef96240455a3aa788f96b579b, but that only tested for the one case of a WS overflow which was already handled. Fixes varnishcache/varnish-cache#3145 (cherry picked from commit 287dc4a6745c374e0b229bfa861d664989a3a9e8) diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c index f5f102075..0029a68d3 100644 --- a/bin/varnishd/cache/cache_acceptor.c +++ b/bin/varnishd/cache/cache_acceptor.c @@ -319,17 +319,17 @@ vca_mk_tcp(const struct wrk_accept *wa, struct sockaddr_storage ss; socklen_t sl; - SES_Reserve_remote_addr(sp, &sa); + AN(SES_Reserve_remote_addr(sp, &sa)); AN(VSA_Build(sa, &wa->acceptaddr, wa->acceptaddrlen)); sp->sattr[SA_CLIENT_ADDR] = sp->sattr[SA_REMOTE_ADDR]; VTCP_name(sa, raddr, VTCP_ADDRBUFSIZE, rport, VTCP_PORTBUFSIZE); - SES_Set_String_Attr(sp, SA_CLIENT_IP, raddr); - SES_Set_String_Attr(sp, SA_CLIENT_PORT, rport); + AN(SES_Set_String_Attr(sp, SA_CLIENT_IP, raddr)); + AN(SES_Set_String_Attr(sp, SA_CLIENT_PORT, rport)); sl = sizeof ss; AZ(getsockname(sp->fd, (void*)&ss, &sl)); - SES_Reserve_local_addr(sp, &sa); + AN(SES_Reserve_local_addr(sp, &sa)); AN(VSA_Build(sa, &ss, sl)); sp->sattr[SA_SERVER_ADDR] = sp->sattr[SA_LOCAL_ADDR]; VTCP_name(sa, laddr, VTCP_ADDRBUFSIZE, lport, VTCP_PORTBUFSIZE); @@ -342,13 +342,13 @@ vca_mk_uds(struct wrk_accept *wa, struct sess *sp, char *laddr, char *lport, struct suckaddr *sa; (void) wa; - SES_Reserve_remote_addr(sp, &sa); + AN(SES_Reserve_remote_addr(sp, &sa)); AZ(SES_Set_remote_addr(sp, bogo_ip)); sp->sattr[SA_CLIENT_ADDR] = sp->sattr[SA_REMOTE_ADDR]; sp->sattr[SA_LOCAL_ADDR] = sp->sattr[SA_REMOTE_ADDR]; sp->sattr[SA_SERVER_ADDR] = sp->sattr[SA_REMOTE_ADDR]; - SES_Set_String_Attr(sp, SA_CLIENT_IP, "0.0.0.0"); - SES_Set_String_Attr(sp, SA_CLIENT_PORT, "0"); + AN(SES_Set_String_Attr(sp, SA_CLIENT_IP, "0.0.0.0")); + AN(SES_Set_String_Attr(sp, SA_CLIENT_PORT, "0")); strcpy(laddr, "0.0.0.0"); strcpy(raddr, "0.0.0.0"); diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index 8c9325515..debb0c2cf 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -104,8 +104,8 @@ ses_set_attr(const struct sess *sp, enum sess_attr a, const void *src, int sz) return (0); } -static void -ses_reserve_attr(struct sess *sp, enum sess_attr a, void **dst, int sz) +static int +ses_res_attr(struct sess *sp, enum sess_attr a, void **dst, int sz) { ssize_t o; @@ -114,12 +114,14 @@ ses_reserve_attr(struct sess *sp, enum sess_attr a, void **dst, int sz) assert(sz >= 0); AN(dst); o = WS_ReserveSize(sp->ws, sz); - assert(o >= sz); + if (o < sz) + return (0); *dst = sp->ws->f; o = sp->ws->f - sp->ws->s; WS_Release(sp->ws, sz); assert(o >= 0 && o <= 0xffff); sp->sattr[a] = (uint16_t)o; + return (1); } #define SESS_ATTR(UP, low, typ, len) \ @@ -135,16 +137,16 @@ ses_reserve_attr(struct sess *sp, enum sess_attr a, void **dst, int sz) return (ses_get_attr(sp, SA_##UP, (void**)dst)); \ } \ \ - void \ + int \ SES_Reserve_##low(struct sess *sp, typ **dst) \ { \ - assert(len >= 0); \ - ses_reserve_attr(sp, SA_##UP, (void**)dst, len); \ + assert(len > 0); \ + return (ses_res_attr(sp, SA_##UP, (void**)dst, len)); \ } #include "tbl/sess_attr.h" -void +int SES_Set_String_Attr(struct sess *sp, enum sess_attr a, const char *src) { void *q; @@ -158,8 +160,10 @@ SES_Set_String_Attr(struct sess *sp, enum sess_attr a, const char *src) default: WRONG("wrong sess_attr"); } - ses_reserve_attr(sp, a, &q, strlen(src) + 1); + if (! ses_res_attr(sp, a, &q, strlen(src) + 1)) + return (0); strcpy(q, src); + return (1); } const char * diff --git a/bin/varnishd/cache/cache_varnishd.h b/bin/varnishd/cache/cache_varnishd.h index 5d036f6da..939a0fe17 100644 --- a/bin/varnishd/cache/cache_varnishd.h +++ b/bin/varnishd/cache/cache_varnishd.h @@ -351,9 +351,9 @@ enum htc_status_e HTC_RxStuff(struct http_conn *, htc_complete_f *, #define SESS_ATTR(UP, low, typ, len) \ int SES_Set_##low(const struct sess *sp, const typ *src); \ - void SES_Reserve_##low(struct sess *sp, typ **dst); + int SES_Reserve_##low(struct sess *sp, typ **dst); #include "tbl/sess_attr.h" -void SES_Set_String_Attr(struct sess *sp, enum sess_attr a, const char *src); +int SES_Set_String_Attr(struct sess *sp, enum sess_attr a, const char *src); enum htc_status_e { diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c index 1c216dc78..91b8e2695 100644 --- a/bin/varnishd/http1/cache_http1_fsm.c +++ b/bin/varnishd/http1/cache_http1_fsm.c @@ -113,7 +113,7 @@ http1_new_session(struct worker *wrk, void *arg) CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); HTC_RxInit(req->htc, req->ws); - SES_Reserve_proto_priv(sp, &u); + XXXAN(SES_Reserve_proto_priv(sp, &u)); http1_setstate(sp, H1NEWREQ); wrk->task.func = http1_req; wrk->task.priv = req; diff --git a/bin/varnishd/http2/cache_http2_session.c b/bin/varnishd/http2/cache_http2_session.c index 140007ffc..ab71cc2aa 100644 --- a/bin/varnishd/http2/cache_http2_session.c +++ b/bin/varnishd/http2/cache_http2_session.c @@ -102,7 +102,7 @@ h2_init_sess(const struct worker *wrk, struct sess *sp, if (SES_Get_proto_priv(sp, &up)) { /* Already reserved if we came via H1 */ - SES_Reserve_proto_priv(sp, &up); + XXXAN(SES_Reserve_proto_priv(sp, &up)); *up = 0; } if (*up == 0) { @@ -131,7 +131,7 @@ h2_init_sess(const struct worker *wrk, struct sess *sp, AZ(VHT_Init(h2->dectbl, h2->local_settings.header_table_size)); - SES_Reserve_proto_priv(sp, &up); + XXXAN(SES_Reserve_proto_priv(sp, &up)); *up = (uintptr_t)h2; } AN(up); diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c index 9232254d3..0c7be9bb3 100644 --- a/bin/varnishd/proxy/cache_proxy_proto.c +++ b/bin/varnishd/proxy/cache_proxy_proto.c @@ -50,6 +50,13 @@ struct vpx_tlv { char tlv[1]; }; +static inline int +vpx_ws_err(const struct req *req) +{ + VSL(SLT_Error, req->sp->vxid, "insufficient workspace"); + return (-1); +} + /********************************************************************** * PROXY 1 protocol */ @@ -109,17 +116,23 @@ vpx_proto1(const struct worker *wrk, const struct req *req) return (-1); } - SES_Reserve_client_addr(req->sp, &sa); + if (! SES_Reserve_client_addr(req->sp, &sa)) + return (vpx_ws_err(req)); + if (VSS_ResolveOne(sa, fld[1], fld[3], pfam, SOCK_STREAM, AI_NUMERICHOST | AI_NUMERICSERV) == NULL) { VSL(SLT_ProxyGarbage, req->sp->vxid, "PROXY1: Cannot resolve source address"); return (-1); } - SES_Set_String_Attr(req->sp, SA_CLIENT_IP, fld[1]); - SES_Set_String_Attr(req->sp, SA_CLIENT_PORT, fld[3]); + if (! SES_Set_String_Attr(req->sp, SA_CLIENT_IP, fld[1])) + return (vpx_ws_err(req)); + if (! SES_Set_String_Attr(req->sp, SA_CLIENT_PORT, fld[3])) + return (vpx_ws_err(req)); + + if (! SES_Reserve_server_addr(req->sp, &sa)) + return (vpx_ws_err(req)); - SES_Reserve_server_addr(req->sp, &sa); if (VSS_ResolveOne(sa, fld[2], fld[4], pfam, SOCK_STREAM, AI_NUMERICHOST | AI_NUMERICSERV) == NULL) { VSL(SLT_ProxyGarbage, req->sp->vxid, @@ -379,14 +392,16 @@ vpx_proto2(const struct worker *wrk, struct req *req) /* dst/server */ memcpy(&sin4.sin_addr, p + 20, 4); memcpy(&sin4.sin_port, p + 26, 2); - SES_Reserve_server_addr(req->sp, &sa); + if (! SES_Reserve_server_addr(req->sp, &sa)) + return (vpx_ws_err(req)); AN(VSA_Build(sa, &sin4, sizeof sin4)); VTCP_name(sa, ha, sizeof ha, pa, sizeof pa); /* src/client */ memcpy(&sin4.sin_addr, p + 16, 4); memcpy(&sin4.sin_port, p + 24, 2); - SES_Reserve_client_addr(req->sp, &sa); + if (! SES_Reserve_client_addr(req->sp, &sa)) + return (vpx_ws_err(req)); AN(VSA_Build(sa, &sin4, sizeof sin4)); break; case 0x21: @@ -405,14 +420,16 @@ vpx_proto2(const struct worker *wrk, struct req *req) /* dst/server */ memcpy(&sin6.sin6_addr, p + 32, 16); memcpy(&sin6.sin6_port, p + 50, 2); - SES_Reserve_server_addr(req->sp, &sa); + if (! SES_Reserve_server_addr(req->sp, &sa)) + return (vpx_ws_err(req)); AN(VSA_Build(sa, &sin6, sizeof sin6)); VTCP_name(sa, ha, sizeof ha, pa, sizeof pa); /* src/client */ memcpy(&sin6.sin6_addr, p + 16, 16); memcpy(&sin6.sin6_port, p + 48, 2); - SES_Reserve_client_addr(req->sp, &sa); + if (! SES_Reserve_client_addr(req->sp, &sa)) + return (vpx_ws_err(req)); AN(VSA_Build(sa, &sin6, sizeof sin6)); break; default: @@ -424,8 +441,10 @@ vpx_proto2(const struct worker *wrk, struct req *req) AN(sa); VTCP_name(sa, hb, sizeof hb, pb, sizeof pb); - SES_Set_String_Attr(req->sp, SA_CLIENT_IP, hb); - SES_Set_String_Attr(req->sp, SA_CLIENT_PORT, pb); + if (! SES_Set_String_Attr(req->sp, SA_CLIENT_IP, hb)) + return (vpx_ws_err(req)); + if (! SES_Set_String_Attr(req->sp, SA_CLIENT_PORT, pb)) + return (vpx_ws_err(req)); VSL(SLT_Proxy, req->sp->vxid, "2 %s %s %s %s", hb, pb, ha, pa); @@ -452,15 +471,13 @@ vpx_proto2(const struct worker *wrk, struct req *req) return (-1); } tlv = WS_Alloc(req->sp->ws, sizeof *tlv + tlv_len); - if (tlv == NULL) { - VSL(SLT_ProxyGarbage, req->sp->vxid, - "PROXY2: TLV overflows WS"); - return (-1); - } + if (tlv == NULL) + return (vpx_ws_err(req)); INIT_OBJ(tlv, VPX_TLV_MAGIC); tlv->len = tlv_len; memcpy(tlv->tlv, tlv_start, tlv_len); - SES_Reserve_proxy_tlv(req->sp, &up); + if (! SES_Reserve_proxy_tlv(req->sp, &up)) + return (vpx_ws_err(req)); *up = (uintptr_t)tlv; return (0); } diff --git a/bin/varnishtest/tests/o00005.vtc b/bin/varnishtest/tests/o00005.vtc index 3ae1aacfe..d84b92d4d 100644 --- a/bin/varnishtest/tests/o00005.vtc +++ b/bin/varnishtest/tests/o00005.vtc @@ -5,7 +5,8 @@ server s1 { txresp } -start -varnish v1 -proto "PROXY" -vcl+backend { +varnish v1 -arg "-p pool_sess=0,0,0" -proto "PROXY" -vcl+backend { + import vtc; import proxy; sub vcl_deliver { @@ -20,6 +21,7 @@ varnish v1 -proto "PROXY" -vcl+backend { set resp.http.key = proxy.cert_key(); set resp.http.sign = proxy.cert_sign(); set resp.http.cn = proxy.client_cert_cn(); + set resp.http.ws_free = vtc.workspace_free(session); } } -start @@ -243,4 +245,68 @@ client c1 { expect_close } -run +delay 1 + varnish v1 -expect ws_session_overflow == 1 + +# error handling elsewhere in the proxy code +# request is the same as initial + +varnish v1 -cliok "param.set workspace_session 450" +varnish v1 -cliok "param.set pool_sess 10,100,1" + +delay 1 + +# get rid of the surplus session mpl +client c10 -proxy1 "1.2.3.4:1111 5.6.7.8:5678" { + txreq + rxresp +} -start +client c11 -proxy1 "1.2.3.4:1111 5.6.7.8:5678" { + txreq + rxresp +} -start +client c12 -proxy1 "1.2.3.4:1111 5.6.7.8:5678" { + txreq + rxresp +} -start +client c13 -proxy1 "1.2.3.4:1111 5.6.7.8:5678" { + txreq + rxresp +} -start +client c14 -proxy1 "1.2.3.4:1111 5.6.7.8:5678" { + txreq + rxresp +} -start + +client c10 -wait +client c11 -wait +client c12 -wait +client c13 -wait +client c14 -wait + +client c2 { + # PROXY2 with CRC32C TLV + sendhex { + 0d 0a 0d 0a 00 0d 0a 51 55 49 54 0a + 21 11 00 65 + d9 46 b5 21 + 5f 8e a8 22 + ed 96 + 01 bb + 03 00 04 95 03 ee 75 + 01 00 02 68 32 + 02 00 0a 68 6f 63 64 65 74 2e 6e 65 74 + 20 00 3d + 01 00 00 00 00 + 21 00 07 54 4c 53 76 31 2e 33 + 25 00 05 45 43 32 35 36 + 24 00 0a 52 53 41 2d 53 48 41 32 35 36 + 23 00 16 41 45 41 44 2d 41 45 53 31 32 38 + 2d 47 43 4d 2d 53 48 41 32 35 36 + } + txreq + expect_close +} -run + +varnish v1 -expect ws_session_overflow == 2 From martin at varnish-software.com Tue Feb 4 10:02:08 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:02:08 +0000 (UTC) Subject: [6.0] 883480956 Simplify WS allocation in tlv_string Message-ID: <20200204100208.5BC7EA31B9@lists.varnish-cache.org> commit 883480956faa113987813f13a162e4648e9761e9 Author: Emmanuel Hocdet Date: Thu Nov 14 11:14:07 2019 +0100 Simplify WS allocation in tlv_string Patch by @ehocdet, commit message edited by @nigoroll: The root cause of #3131 was misdiagnosed to the extent that, while this change had prevented it, the root cause was a bug in WS_ReserveSize() fixed in 505b7bd9643006fa8e3977f920564ce12a2b24a2 The previous tlv_string() code was correct except for the fact that error handling should have checked for WS_ReserveSize(ctx->ws, len+1) <= len (also spotted by @ehocdet). Someone had mentioned at some point that we would not want to VRT_fail(), but I think this must have been related to the proxy transport code, not the proxy vmod. Ref varnishcache/varnish-cache#3131 (cherry picked from commit e74f9e871bc0eab014b7dd359d6ae83153c3ee37) diff --git a/lib/libvmod_proxy/vmod_proxy.c b/lib/libvmod_proxy/vmod_proxy.c index 58f2db295..222de9c53 100644 --- a/lib/libvmod_proxy/vmod_proxy.c +++ b/lib/libvmod_proxy/vmod_proxy.c @@ -105,12 +105,13 @@ tlv_string(VRT_CTX, int tlv) if (VPX_tlv(ctx->req, tlv, (void **)&dst, &len)) return (NULL); - if (!WS_ReserveSize(ctx->ws, len+1)) + d = WS_Alloc(ctx->ws, len+1); + if (d == NULL) { + VRT_fail(ctx, "proxy.TLV: out of workspace"); return (NULL); - d = ctx->ws->f; + } memcpy(d, dst, len); d[len] = '\0'; - WS_Release(ctx->ws, len+1); return (d); } From martin at varnish-software.com Tue Feb 4 10:02:08 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:02:08 +0000 (UTC) Subject: [6.0] 69483e434 Try to make the proxy code session workspace overflow test on 32bit Message-ID: <20200204100208.74AB0A31C4@lists.varnish-cache.org> commit 69483e434bb23e5176526aafe937d4220f511a1b Author: Nils Goroll Date: Thu Nov 28 11:19:08 2019 +0100 Try to make the proxy code session workspace overflow test on 32bit Ref varnishcache/varnish-cache#3145 / 287dc4a6745c374e0b229bfa861d664989a3a9e8 (cherry picked from commit e1a57eb7a7350ac4c8a0235dc991bef400b118c1) diff --git a/bin/varnishtest/tests/o00005.vtc b/bin/varnishtest/tests/o00005.vtc index d84b92d4d..2ac11d1a8 100644 --- a/bin/varnishtest/tests/o00005.vtc +++ b/bin/varnishtest/tests/o00005.vtc @@ -249,10 +249,16 @@ delay 1 varnish v1 -expect ws_session_overflow == 1 -# error handling elsewhere in the proxy code -# request is the same as initial - -varnish v1 -cliok "param.set workspace_session 450" +# workspace overflow handling elsewhere in the proxy code +# +# the workspace_session size is chosen to fail as closely as possible +# below the minimum required value for the vtc to work (= the +# workspace to overflow) both on 64 and 32 bit. +# +# This value is fragile, ideally we would want something like +# vtc.alloc(-x), yet there is no vcl code being run before we parse +# proxy headers +varnish v1 -cliok "param.set workspace_session 410" varnish v1 -cliok "param.set pool_sess 10,100,1" delay 1 From martin at varnish-software.com Tue Feb 4 10:02:08 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:02:08 +0000 (UTC) Subject: [6.0] 435c19a21 more tweaking for 32bit vtest machines Message-ID: <20200204100208.8E931A31D0@lists.varnish-cache.org> commit 435c19a21416c2bf6acdcd6f17b36c3a7576a35b Author: Nils Goroll Date: Thu Nov 28 12:21:17 2019 +0100 more tweaking for 32bit vtest machines Ref varnishcache/varnish-cache#3145 / 287dc4a6745c374e0b229bfa861d664989a3a9e8 (cherry picked from commit d6dec03101a67c38addf549435ae51df38f1d7db) diff --git a/bin/varnishtest/tests/o00005.vtc b/bin/varnishtest/tests/o00005.vtc index 2ac11d1a8..b09a8fcaf 100644 --- a/bin/varnishtest/tests/o00005.vtc +++ b/bin/varnishtest/tests/o00005.vtc @@ -258,7 +258,7 @@ varnish v1 -expect ws_session_overflow == 1 # This value is fragile, ideally we would want something like # vtc.alloc(-x), yet there is no vcl code being run before we parse # proxy headers -varnish v1 -cliok "param.set workspace_session 410" +varnish v1 -cliok "param.set workspace_session 402" varnish v1 -cliok "param.set pool_sess 10,100,1" delay 1 From martin at varnish-software.com Tue Feb 4 10:02:08 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:02:08 +0000 (UTC) Subject: [6.0] 6848eb7d3 Handle out of session workspace in http1_new_session() Message-ID: <20200204100208.A72AFA31D9@lists.varnish-cache.org> commit 6848eb7d3281c9ff94624f6560f66740e975811a Author: Martin Blix Grydeland Date: Thu Dec 12 13:12:48 2019 +0100 Handle out of session workspace in http1_new_session() If proxy protocol is in use, it is possible to fill the session workspace exactly before entering http1_new_session(), which will cause it to assert when calling SES_Reserve_proto_priv(). with this patch we will close the session gracefully. diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c index 91b8e2695..8746c2ef8 100644 --- a/bin/varnishd/http1/cache_http1_fsm.c +++ b/bin/varnishd/http1/cache_http1_fsm.c @@ -113,7 +113,15 @@ http1_new_session(struct worker *wrk, void *arg) CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); HTC_RxInit(req->htc, req->ws); - XXXAN(SES_Reserve_proto_priv(sp, &u)); + if (!SES_Reserve_proto_priv(sp, &u)) { + /* Out of session workspace. Free the req, close the sess, + * and do not set a new task func, which will exit the + * worker thread. */ + VSL(SLT_Error, req->sp->vxid, "insufficient workspace"); + Req_Release(req); + SES_Delete(sp, SC_RX_JUNK, NAN); + return; + } http1_setstate(sp, H1NEWREQ); wrk->task.func = http1_req; wrk->task.priv = req; diff --git a/bin/varnishtest/tests/f00005.vtc b/bin/varnishtest/tests/f00005.vtc new file mode 100644 index 000000000..866d941dd --- /dev/null +++ b/bin/varnishtest/tests/f00005.vtc @@ -0,0 +1,64 @@ +varnishtest "proxy ws panic" + +server s1 { + rxreq + txresp +} -start + +varnish v1 -proto "PROXY" -vcl+backend {}-start + +# Too large proxy payload using TLV +client c1 { + sendhex { +0d 0a 0d 0a 00 0d 0a 51 +55 49 54 0a 21 21 00 93 +aa bb cc dd ee ff 11 22 +33 44 55 66 77 88 99 aa +bb cc dd ee ff 11 22 33 +44 55 66 77 88 99 aa bb +88 da 0d 73 02 00 3c 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 20 00 2d 01 01 +00 00 00 21 00 07 54 4c +53 76 31 2e 32 23 00 1b +45 43 44 48 45 2d 52 53 +41 2d 41 45 53 32 35 36 +2d 47 43 4d 2d 53 48 41 +33 38 34 + } + expect_close +} -run + +# Reduced size proxy payload to verify Varnish is still running +client c1 { + sendhex { +0d 0a 0d 0a 00 0d 0a 51 +55 49 54 0a 21 21 00 8b +aa bb cc dd ee ff 11 22 +33 44 55 66 77 88 99 aa +bb cc dd ee ff 11 22 33 +44 55 66 77 88 99 aa bb +88 da 0d 73 02 00 34 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 20 00 2d 01 01 +00 00 00 21 00 07 54 4c +53 76 31 2e 32 23 00 1b +45 43 44 48 45 2d 52 53 +41 2d 41 45 53 32 35 36 +2d 47 43 4d 2d 53 48 41 +33 38 34 + } + txreq + rxresp +} -run From martin at varnish-software.com Tue Feb 4 10:02:08 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:02:08 +0000 (UTC) Subject: [6.0] 1e3e6e2c5 Remove extra call to SES_Reserve_proto_priv Message-ID: <20200204100208.BFD42A31ED@lists.varnish-cache.org> commit 1e3e6e2c523f5f96729c12bd9597c919bff686cf Author: Martin Blix Grydeland Date: Thu Dec 12 13:16:40 2019 +0100 Remove extra call to SES_Reserve_proto_priv In h2_init_sess, an extra call was always made to SES_Reseve_proto_priv(), even though it was already reserved. This wasted a pointer worth of session workspace. This patch removes the extra call. diff --git a/bin/varnishd/http2/cache_http2_session.c b/bin/varnishd/http2/cache_http2_session.c index ab71cc2aa..5818707f4 100644 --- a/bin/varnishd/http2/cache_http2_session.c +++ b/bin/varnishd/http2/cache_http2_session.c @@ -131,7 +131,6 @@ h2_init_sess(const struct worker *wrk, struct sess *sp, AZ(VHT_Init(h2->dectbl, h2->local_settings.header_table_size)); - XXXAN(SES_Reserve_proto_priv(sp, &up)); *up = (uintptr_t)h2; } AN(up); From martin at varnish-software.com Tue Feb 4 10:02:08 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:02:08 +0000 (UTC) Subject: [6.0] 8164a366a Remove call to SES_Reserve_proto_priv in h2_init_sess Message-ID: <20200204100208.D6E0BA31FC@lists.varnish-cache.org> commit 8164a366a085a85e9df18067529a55b141c1cfc2 Author: Martin Blix Grydeland Date: Thu Dec 12 13:51:17 2019 +0100 Remove call to SES_Reserve_proto_priv in h2_init_sess h2_init_sess can only be reached through H1 with either previous knowledge or opportunistic upgrade. Because of this the proto_priv session attribute will always be set before entry. This patch simplifies and removes dead code containing a call to SES_Reserve_proto_priv. Note: Better diff with the --ignore-all-space option diff --git a/bin/varnishd/http2/cache_http2_session.c b/bin/varnishd/http2/cache_http2_session.c index 5818707f4..9d3683845 100644 --- a/bin/varnishd/http2/cache_http2_session.c +++ b/bin/varnishd/http2/cache_http2_session.c @@ -100,41 +100,37 @@ h2_init_sess(const struct worker *wrk, struct sess *sp, uintptr_t *up; struct h2_sess *h2; - if (SES_Get_proto_priv(sp, &up)) { - /* Already reserved if we came via H1 */ - XXXAN(SES_Reserve_proto_priv(sp, &up)); - *up = 0; - } - if (*up == 0) { - if (srq == NULL) - srq = Req_New(wrk, sp); - AN(srq); - h2 = h2s; - AN(h2); - INIT_OBJ(h2, H2_SESS_MAGIC); - h2->srq = srq; - h2->htc = srq->htc; - h2->ws = srq->ws; - h2->vsl = srq->vsl; - VSL_Flush(h2->vsl, 0); - h2->vsl->wid = sp->vxid; - h2->htc->rfd = &sp->fd; - h2->sess = sp; - h2->rxthr = pthread_self(); - AZ(pthread_cond_init(h2->winupd_cond, NULL)); - VTAILQ_INIT(&h2->streams); - VTAILQ_INIT(&h2->txqueue); - h2_local_settings(&h2->local_settings); - h2->remote_settings = H2_proto_settings; - h2->decode = decode; - - AZ(VHT_Init(h2->dectbl, - h2->local_settings.header_table_size)); - - *up = (uintptr_t)h2; - } - AN(up); - CAST_OBJ_NOTNULL(h2, (void*)(*up), H2_SESS_MAGIC); + /* proto_priv session attribute will always have been set up by H1 + * before reaching here. */ + AZ(SES_Get_proto_priv(sp, &up)); + assert(*up == 0); + + if (srq == NULL) + srq = Req_New(wrk, sp); + AN(srq); + h2 = h2s; + AN(h2); + INIT_OBJ(h2, H2_SESS_MAGIC); + h2->srq = srq; + h2->htc = srq->htc; + h2->ws = srq->ws; + h2->vsl = srq->vsl; + VSL_Flush(h2->vsl, 0); + h2->vsl->wid = sp->vxid; + h2->htc->rfd = &sp->fd; + h2->sess = sp; + h2->rxthr = pthread_self(); + AZ(pthread_cond_init(h2->winupd_cond, NULL)); + VTAILQ_INIT(&h2->streams); + VTAILQ_INIT(&h2->txqueue); + h2_local_settings(&h2->local_settings); + h2->remote_settings = H2_proto_settings; + h2->decode = decode; + + AZ(VHT_Init(h2->dectbl, h2->local_settings.header_table_size)); + + *up = (uintptr_t)h2; + return (h2); } From martin at varnish-software.com Tue Feb 4 10:02:08 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:02:08 +0000 (UTC) Subject: [6.0] ce025dca6 Handle badly formatted proxy TLVs Message-ID: <20200204100208.F0B1DA3206@lists.varnish-cache.org> commit ce025dca6d1861a237de343c67c8abbee01d70a3 Author: Martin Blix Grydeland Date: Thu Dec 12 14:53:48 2019 +0100 Handle badly formatted proxy TLVs Proxy TLVs claiming to have PP2_TYPE_SSL sub-TLVs without complete payload would cause a Varnish assert. This patch fixes the parsing of the TLVs. diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c index 0c7be9bb3..64a3a2b08 100644 --- a/bin/varnishd/proxy/cache_proxy_proto.c +++ b/bin/varnishd/proxy/cache_proxy_proto.c @@ -276,8 +276,9 @@ vpx_tlv_itern(struct vpx_tlv_iter *vpi) return (1); } -#define VPX_TLV_FOREACH(ptr, len, itv) \ - for(vpx_tlv_iter0(itv, ptr, len); vpx_tlv_itern(itv);) +#define VPX_TLV_FOREACH(ptr, len, itv) \ + for (vpx_tlv_iter0(itv, ptr, len); \ + (vpi->e == NULL) && vpx_tlv_itern(itv);) int VPX_tlv(const struct req *req, int typ, void **dst, int *len) @@ -453,6 +454,10 @@ vpx_proto2(const struct worker *wrk, struct req *req) VPX_TLV_FOREACH(d, l, vpi) { if (vpi->t == PP2_TYPE_SSL) { + if (vpi->l < 5) { + vpi->e = "Length Error"; + break; + } VPX_TLV_FOREACH((char*)vpi->p + 5, vpi->l - 5, vpi2) { } vpi->e = vpi2->e; diff --git a/bin/varnishtest/tests/f00005.vtc b/bin/varnishtest/tests/f00005.vtc index 866d941dd..b1b097fce 100644 --- a/bin/varnishtest/tests/f00005.vtc +++ b/bin/varnishtest/tests/f00005.vtc @@ -35,6 +35,18 @@ bb cc dd ee ff 11 22 33 expect_close } -run +# Badly formatted TLV proxy payload +client c1 { + sendhex { +0d 0a 0d 0a 00 0d 0a 51 +55 49 54 0a +21 11 00 13 00 ff 20 ff 10 ff 03 21 20 30 00 20 +20 00 00 19 00 02 29 20 00 00 00 41 20 9e 15 15 +d6 00 00 08 00 00 00 00 00 07 7a 20 b1 3f 43 20 + } + expect_close +} -run + # Reduced size proxy payload to verify Varnish is still running client c1 { sendhex { From martin at varnish-software.com Tue Feb 4 10:02:09 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:02:09 +0000 (UTC) Subject: [6.0] c18b3baa1 Assert Message-ID: <20200204100209.15E4EA321F@lists.varnish-cache.org> commit c18b3baa187e798eb7225a64d6ba5a4e904aefc1 Author: Dridi Boukelmoune Date: Mon Dec 16 12:16:17 2019 +0100 Assert (cherry picked from commit 7da6220dfd86416200c43d84d40fce0cae1b5bae) diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c index e98e2cc69..de04e896d 100644 --- a/bin/varnishd/cache/cache_ws.c +++ b/bin/varnishd/cache/cache_ws.c @@ -235,6 +235,7 @@ WS_ReserveAll(struct ws *ws) { unsigned b; + WS_Assert(ws); assert(ws->r == NULL); ws->r = ws->e; From martin at varnish-software.com Tue Feb 4 10:02:09 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:02:09 +0000 (UTC) Subject: [6.0] ff4e69265 Fix WS_ReserveSize calls when bytes is equal to free workspace Message-ID: <20200204100209.2E5A8A3236@lists.varnish-cache.org> commit ff4e69265f2cbedf6c7a002e71ced8f255a6a0b3 Author: Martin Blix Grydeland Date: Mon Dec 16 16:37:27 2019 +0100 Fix WS_ReserveSize calls when bytes is equal to free workspace Currently, with the 505b7bd9643006fa8e3977f920564ce12a2b24a2 patch, when calling WS_ReserveSize with bytes equal to the amount of workspace that is currently available, it will return zero and mark overflow. This patch redoes the patch, and changes it to return zero and overflow only when the requested number of bytes is larger than what is available. diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c index de04e896d..5de7a6830 100644 --- a/bin/varnishd/cache/cache_ws.c +++ b/bin/varnishd/cache/cache_ws.c @@ -263,7 +263,7 @@ WS_ReserveSize(struct ws *ws, unsigned bytes) if (bytes < b2) b2 = PRNDUP(bytes); - if (ws->f + b2 >= ws->e) { + if (bytes > b2) { WS_MarkOverflow(ws); return (0); } From martin at varnish-software.com Tue Feb 4 10:02:09 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:02:09 +0000 (UTC) Subject: [6.0] e703aa1a2 Take sizeof pool_task into account when reserving WS in SES_Wait Message-ID: <20200204100209.4689DA323C@lists.varnish-cache.org> commit e703aa1a23c0c5c4fe3f22db73b64963e76b6b2d Author: Martin Blix Grydeland Date: Mon Dec 16 17:04:43 2019 +0100 Take sizeof pool_task into account when reserving WS in SES_Wait The assert on WS_ReserveSize() in ses_handle() can not trip because sizeof (struct pool_task) is less than sizeof (struct waited). But to safe guard against future problems if that were to change, this patch makes sure that the session workspace can hold the largest of them before entering the waiter, erroring out if not. diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index debb0c2cf..45bb80fa0 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -409,6 +409,7 @@ ses_handle(struct waited *wp, enum wait_event ev, vtim_real now) wp->magic = 0; wp = NULL; + /* The WS was reserved in SES_Wait() */ WS_Release(sp->ws, 0); switch (ev) { @@ -421,6 +422,7 @@ ses_handle(struct waited *wp, enum wait_event ev, vtim_real now) case WAITER_ACTION: pp = sp->pool; CHECK_OBJ_NOTNULL(pp, POOL_MAGIC); + /* SES_Wait() guarantees the next will not assert. */ assert(sizeof *tp <= WS_ReserveSize(sp->ws, sizeof *tp)); tp = (void*)sp->ws->f; tp->func = xp->unwait; @@ -444,6 +446,7 @@ SES_Wait(struct sess *sp, const struct transport *xp) { struct pool *pp; struct waited *wp; + unsigned u; CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); CHECK_OBJ_NOTNULL(xp, TRANSPORT_MAGIC); @@ -457,10 +460,15 @@ SES_Wait(struct sess *sp, const struct transport *xp) VTCP_nonblocking(sp->fd); /* - * put struct waited on the workspace + * Put struct waited on the workspace. Make sure that the + * workspace can hold enough space for the largest of struct + * waited and pool_task, as pool_task will be needed when coming + * off the waiter again. */ - if (WS_ReserveSize(sp->ws, sizeof(struct waited)) - < sizeof(struct waited)) { + u = sizeof (struct waited); + if (sizeof (struct pool_task) > u) + u = sizeof (struct pool_task); + if (!WS_ReserveSize(sp->ws, u)) { SES_Delete(sp, SC_OVERLOAD, NAN); return; } From martin at varnish-software.com Tue Feb 4 10:02:09 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:02:09 +0000 (UTC) Subject: [6.0] dfb6a1c9a Changelog for Varnish LTS 6.0.6 Message-ID: <20200204100209.5F330A3259@lists.varnish-cache.org> commit dfb6a1c9a219d4c91ab8760ecb46f384aeadec14 Author: Martin Blix Grydeland Date: Fri Jan 31 12:44:41 2020 +0100 Changelog for Varnish LTS 6.0.6 diff --git a/doc/changes.rst b/doc/changes.rst index b2eeba60e..da4adaeff 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -26,6 +26,41 @@ http://varnish-cache.org/docs/trunk/whats-new/index.html and via individual releases. These documents are updated as part of the release process. +================================ +Varnish Cache 6.0.6 (2020-02-04) +================================ + +* Fix an H2 locking bug during error handling. (3086_) + +* Replace `python` with `python3` in all build scripts. + +* Introduce `none` backends, which are empty backends that are always + sick. This is documented in the ``vcl(7)`` manual page. + +* Fix an assertion panic when labelling a VCL twice. (2834_) + +* Fix semantics of VCL `auto` state management. (2836_) + +* Fix a probe scheduling timeout sorting error. (3115_) + +* Allow switching to the error state from the backend fetch and backend + response states. This makes it possible to execute `return (error)` + from within the `vcl_backend_fetch()` and `vcl_backend_response()` VCL + functions. + +* Implement the `If-Range` header as specified in RFC 7233 section + 3.2. (RFC-7233-32_) + +* Fix a denial of service vulnerability when using the proxy protocol + version 2. (VSV00005_) + +.. _3086: https://github.com/varnishcache/varnish-cache/issues/3086 +.. _2834: https://github.com/varnishcache/varnish-cache/issues/2834 +.. _2836: https://github.com/varnishcache/varnish-cache/issues/2836 +.. _3115: https://github.com/varnishcache/varnish-cache/issues/3115 +.. _RFC-7233-32: https://tools.ietf.org/html/rfc7233#section-3.2 +.. _VSV00005: https://varnish-cache.org/security/VSV00005.html + ================================ Varnish Cache 6.0.5 (2019-10-21) ================================ From martin at varnish-software.com Tue Feb 4 10:02:09 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:02:09 +0000 (UTC) Subject: [6.0] 29a1a8243 Prepare for 6.0.6 Message-ID: <20200204100209.77EBCA326A@lists.varnish-cache.org> commit 29a1a8243dbef3d973aec28dc90403188c1dc8e7 Author: Martin Blix Grydeland Date: Fri Jan 31 12:49:09 2020 +0100 Prepare for 6.0.6 diff --git a/configure.ac b/configure.ac index 5348d9fd4..1768ee068 100644 --- a/configure.ac +++ b/configure.ac @@ -2,7 +2,7 @@ AC_PREREQ(2.59) AC_COPYRIGHT([Copyright (c) 2006 Verdens Gang AS Copyright (c) 2006-2019 Varnish Software]) AC_REVISION([$Id$]) -AC_INIT([Varnish], [6.0.5], [varnish-dev at varnish-cache.org]) +AC_INIT([Varnish], [6.0.6], [varnish-dev at varnish-cache.org]) AC_CONFIG_SRCDIR(include/miniobj.h) AC_CONFIG_HEADERS([config.h]) AC_CONFIG_MACRO_DIR([m4]) From martin at varnish-software.com Tue Feb 4 10:03:07 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:03:07 +0000 (UTC) Subject: [6.2] c6df6a5fa Add WS_ReserveAll() to replace WS_Reserve(ws, 0) Message-ID: <20200204100307.DA456A3C41@lists.varnish-cache.org> commit c6df6a5fa9e5575b079d602895eb429db417c5cd Author: Nils Goroll Date: Mon Apr 8 15:19:30 2019 +0200 Add WS_ReserveAll() to replace WS_Reserve(ws, 0) ... to un-confuse the interface Notes on changes from WS_Reserve(): * Removed the first WS_Assert because all we change is ws->r and we got a specific assert on it. * it follows from PAOK(ws->e) && PAOK(ws->f) in WS_Assert() that PAOK(ws->r) && PAOK(b), so we remove the PRNDDN() Ref: varnishcache/varnish-cache#2967 (cherry picked from commit d001cdd2045aa484a968d09859cddf59dee372cb) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 163c52948..cc2fc993d 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -765,6 +765,7 @@ void WRK_BgThread(pthread_t *thr, const char *name, bgthread_t *func, void WS_Init(struct ws *ws, const char *id, void *space, unsigned len); unsigned WS_Reserve(struct ws *ws, unsigned bytes); +unsigned WS_ReserveAll(struct ws *); unsigned WS_ReserveLumps(struct ws *ws, size_t sz); void WS_MarkOverflow(struct ws *ws); void WS_Release(struct ws *ws, unsigned bytes); diff --git a/bin/varnishd/cache/cache_hash.c b/bin/varnishd/cache/cache_hash.c index f43e214ff..1395f7101 100644 --- a/bin/varnishd/cache/cache_hash.c +++ b/bin/varnishd/cache/cache_hash.c @@ -650,7 +650,7 @@ HSH_Purge(struct worker *wrk, struct objhead *oh, vtim_real ttl_now, CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); CHECK_OBJ_NOTNULL(oh, OBJHEAD_MAGIC); - ospc = WS_Reserve(wrk->aws, 0); + ospc = WS_ReserveAll(wrk->aws); assert(ospc >= sizeof *ocp); /* * Because of "soft" purges, there might be oc's in the list that has diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c index 63d25799b..f77fcf30c 100644 --- a/bin/varnishd/cache/cache_http.c +++ b/bin/varnishd/cache/cache_http.c @@ -394,7 +394,7 @@ http_CollectHdrSep(struct http *hp, const char *hdr, const char *sep) } if (b == NULL) { /* Found second header, start our collection */ - ml = WS_Reserve(hp->ws, 0); + ml = WS_ReserveAll(hp->ws); b = hp->ws->f; e = b + ml; x = Tlen(hp->hd[f]); @@ -1228,7 +1228,7 @@ http_PrintfHeader(struct http *to, const char *fmt, ...) unsigned l, n; CHECK_OBJ_NOTNULL(to, HTTP_MAGIC); - l = WS_Reserve(to->ws, 0); + l = WS_ReserveAll(to->ws); va_start(ap, fmt); n = vsnprintf(to->ws->f, l, fmt, ap); va_end(ap); diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index 259110d7b..eac4773fc 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -203,7 +203,7 @@ HTC_RxInit(struct http_conn *htc, struct ws *ws) CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC); htc->ws = ws; - (void)WS_Reserve(htc->ws, 0); + (void)WS_ReserveAll(htc->ws); htc->rxbuf_b = ws->f; htc->rxbuf_e = ws->f; if (htc->pipeline_b != NULL) { diff --git a/bin/varnishd/cache/cache_vary.c b/bin/varnishd/cache/cache_vary.c index 46d6930fb..4af92f69c 100644 --- a/bin/varnishd/cache/cache_vary.c +++ b/bin/varnishd/cache/cache_vary.c @@ -230,7 +230,7 @@ VRY_Prep(struct req *req) AZ(req->vary_b); AZ(req->vary_l); AZ(req->vary_e); - (void)WS_Reserve(req->ws, 0); + (void)WS_ReserveAll(req->ws); } else { AN(req->ws->r); } diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c index 2edfb85b6..bf1676653 100644 --- a/bin/varnishd/cache/cache_vrt.c +++ b/bin/varnishd/cache/cache_vrt.c @@ -261,7 +261,7 @@ VRT_String(struct ws *ws, const char *h, const char *p, va_list ap) unsigned u, x; va_list aq; - u = WS_Reserve(ws, 0); + u = WS_ReserveAll(ws); e = b = ws->f; e += u; @@ -385,7 +385,7 @@ VRT_StrandsWS(struct ws *ws, const char *h, VCL_STRANDS s) int i; AN(s); - u = WS_Reserve(ws, 0); + u = WS_ReserveAll(ws); for (i = 0; i < s->n; i++) if (s->p[i] != NULL && *s->p[i] != '\0') { @@ -573,7 +573,7 @@ VRT_IP_string(VRT_CTX, VCL_IP ip) CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); if (ip == NULL) return (NULL); - len = WS_Reserve(ctx->ws, 0); + len = WS_ReserveAll(ctx->ws); if (len == 0) { WS_Release(ctx->ws, 0); return (NULL); diff --git a/bin/varnishd/cache/cache_vrt_filter.c b/bin/varnishd/cache/cache_vrt_filter.c index 271be6ccd..f192c3a7a 100644 --- a/bin/varnishd/cache/cache_vrt_filter.c +++ b/bin/varnishd/cache/cache_vrt_filter.c @@ -251,7 +251,7 @@ filter_on_ws(struct ws *ws, filter_list_t *func, void *arg) AN(func); AN(arg); - u = WS_Reserve(ws, 0); + u = WS_ReserveAll(ws); if (u == 0) { WS_Release(ws, 0); WS_MarkOverflow(ws); diff --git a/bin/varnishd/cache/cache_vrt_re.c b/bin/varnishd/cache/cache_vrt_re.c index 9cd0de3e7..ad1f44de8 100644 --- a/bin/varnishd/cache/cache_vrt_re.c +++ b/bin/varnishd/cache/cache_vrt_re.c @@ -127,7 +127,7 @@ VRT_regsub(VRT_CTX, int all, const char *str, void *re, return (str); } - u = WS_Reserve(ctx->ws, 0); + u = WS_ReserveAll(ctx->ws); res_e = res_b = b0 = ctx->ws->f; res_e += u; diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c index 3ec5ec017..e0a9b1d48 100644 --- a/bin/varnishd/cache/cache_ws.c +++ b/bin/varnishd/cache/cache_ws.c @@ -202,7 +202,7 @@ WS_Printf(struct ws *ws, const char *fmt, ...) va_list ap; char *p; - u = WS_Reserve(ws, 0); + u = WS_ReserveAll(ws); p = ws->f; va_start(ap, fmt); v = vsnprintf(p, u, fmt, ap); @@ -227,6 +227,30 @@ WS_Snapshot(struct ws *ws) return (ws->f == ws->s ? 0 : (uintptr_t)ws->f); } +/* + * WS_Release() must be called in all cases + */ +unsigned +WS_ReserveAll(struct ws *ws) +{ + unsigned b; + + assert(ws->r == NULL); + + ws->r = ws->e; + b = pdiff(ws->f, ws->r); + + WS_Assert(ws); + DSL(DBG_WORKSPACE, 0, "WS_ReserveAll(%p) = %u", ws, b); + + return (b); +} + +/* + * bytes == 0 argument is deprecated - use WS_ReserveAll + * + * XXX rename to WS_ReserveSize and macro-wrap WS_Reserve to emit #warn ? + */ unsigned WS_Reserve(struct ws *ws, unsigned bytes) { @@ -253,11 +277,7 @@ WS_Reserve(struct ws *ws, unsigned bytes) unsigned WS_ReserveLumps(struct ws *ws, size_t sz) { - unsigned u; - - u = WS_Reserve(ws, 0); - u /= sz; - return (u); + return (WS_ReserveAll(ws) / sz); } void diff --git a/bin/varnishd/http2/cache_http2_deliver.c b/bin/varnishd/http2/cache_http2_deliver.c index 7ee71ac99..26e4cc5da 100644 --- a/bin/varnishd/http2/cache_http2_deliver.c +++ b/bin/varnishd/http2/cache_http2_deliver.c @@ -225,9 +225,11 @@ h2_build_headers(struct vsb *resp, struct req *req) uint8_t buf[6]; ssize_t sz, sz1; - l = WS_Reserve(req->ws, 0); - if (l < 10) + l = WS_ReserveAll(req->ws); + if (l < 10) { + WS_Release(req->ws, 0); return (-1); + } AN(VSB_new(resp, req->ws->f, l, VSB_FIXEDLEN)); diff --git a/bin/varnishd/http2/cache_http2_hpack.c b/bin/varnishd/http2/cache_http2_hpack.c index 0524f3e32..d55c37614 100644 --- a/bin/varnishd/http2/cache_http2_hpack.c +++ b/bin/varnishd/http2/cache_http2_hpack.c @@ -175,7 +175,7 @@ h2h_decode_init(const struct h2_sess *h2) d = h2->decode; INIT_OBJ(d, H2H_DECODE_MAGIC); VHD_Init(d->vhd); - d->out_l = WS_Reserve(h2->new_req->http->ws, 0); + d->out_l = WS_ReserveAll(h2->new_req->http->ws); assert(d->out_l > 0); /* Can't do any work without any buffer space. Require non-zero size. */ d->out = h2->new_req->http->ws->f; diff --git a/lib/libvmod_blob/vmod_blob.c b/lib/libvmod_blob/vmod_blob.c index 7820e810c..5fe6be553 100644 --- a/lib/libvmod_blob/vmod_blob.c +++ b/lib/libvmod_blob/vmod_blob.c @@ -339,7 +339,7 @@ vmod_decode(VRT_CTX, VCL_ENUM decs, VCL_INT length, VCL_STRANDS strings) CHECK_OBJ_NOTNULL(ctx->ws, WS_MAGIC); buf = WS_Front(ctx->ws); - space = WS_Reserve(ctx->ws, 0); + space = WS_ReserveAll(ctx->ws); if (length <= 0) length = -1; @@ -378,7 +378,7 @@ encode(VRT_CTX, enum encoding enc, enum case_e kase, VCL_BLOB b) CHECK_OBJ_NOTNULL(ctx->ws, WS_MAGIC); snap = WS_Snapshot(ctx->ws); buf = WS_Front(ctx->ws); - space = WS_Reserve(ctx->ws, 0); + space = WS_ReserveAll(ctx->ws); len = func[enc].encode(enc, kase, buf, space, b->blob, b->len); diff --git a/lib/libvmod_std/vmod_std.c b/lib/libvmod_std/vmod_std.c index d35d89f6c..11e5f8667 100644 --- a/lib/libvmod_std/vmod_std.c +++ b/lib/libvmod_std/vmod_std.c @@ -72,7 +72,7 @@ vmod_updown(VRT_CTX, int up, const char *s, va_list ap) const char *p; CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - u = WS_Reserve(ctx->ws, 0); + u = WS_ReserveAll(ctx->ws); e = b = ctx->ws->f; e += u; p = s; diff --git a/lib/libvmod_vtc/vmod_vtc.c b/lib/libvmod_vtc/vmod_vtc.c index 72320ba94..ee9d23fa0 100644 --- a/lib/libvmod_vtc/vmod_vtc.c +++ b/lib/libvmod_vtc/vmod_vtc.c @@ -155,7 +155,7 @@ vmod_workspace_alloc(VRT_CTX, VCL_ENUM which, VCL_INT size) WS_Assert(ws); if (size < 0) { - size += WS_Reserve(ws, 0); + size += WS_ReserveAll(ws); WS_Release(ws, 0); } if (size <= 0) { @@ -182,7 +182,7 @@ vmod_workspace_free(VRT_CTX, VCL_ENUM which) return(-1); WS_Assert(ws); - u = WS_Reserve(ws, 0); + u = WS_ReserveAll(ws); WS_Release(ws, 0); return (u); } From martin at varnish-software.com Tue Feb 4 10:03:07 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:03:07 +0000 (UTC) Subject: [6.2] e518b5cbd Add a v_* attribute for deprecated functions Message-ID: <20200204100308.00320A3C49@lists.varnish-cache.org> commit e518b5cbdb8d932506973e88a0b6c71ee4637344 Author: Nils Goroll Date: Mon Apr 8 16:11:30 2019 +0200 Add a v_* attribute for deprecated functions This works with gcc 6.3.0 and clang 3.8.1-24 The test for __GNUC__ is deliberately simple and might not catch all compilers which would potentially support deprecation marks. While more specifics could be added, the aim is to raise awareness with developers and we consider it quite unlikely that anyone does not compile with one of the main stream compilers at all. (cherry picked from commit 1594037cb896970c2cbfdab101d0059ac00201f9) diff --git a/include/vdef.h b/include/vdef.h index acf94813f..2c78fa041 100644 --- a/include/vdef.h +++ b/include/vdef.h @@ -81,6 +81,12 @@ #define v_noreturn_ __attribute__((__noreturn__)) +#ifdef __GNUC__ +# define v_deprecated_ __attribute__((deprecated)) +#else +# define v_deprecated_ +#endif + /********************************************************************* * Pointer alignment magic */ From martin at varnish-software.com Tue Feb 4 10:03:08 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:03:08 +0000 (UTC) Subject: [6.2] 68de849b1 Deprecate WS_Reserve() and replace it with WS_ReserveSize() Message-ID: <20200204100308.1D778A3C59@lists.varnish-cache.org> commit 68de849b1e874403e9d1eb0145c10c6b5d35f541 Author: Nils Goroll Date: Mon Apr 8 16:26:07 2019 +0200 Deprecate WS_Reserve() and replace it with WS_ReserveSize() WS_ReserveSize() does not leave the workspace reserved when the reservation fails, so WS_Release() must be called for retval > 0 only. Besides the debug string, it is identical to WS_Reserve() except for assert(bytes > 0); Follow-up varnishcache/varnish-cache#2967 Note: The WS_Reserve() function has not been deprecated as that can potentially create problems for the build process of VMODs. (cherry picked from commit 4e33359772a3d751b2ef4b5c4b40259f1bcd6903) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index cc2fc993d..c67dca563 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -764,7 +764,10 @@ void WRK_BgThread(pthread_t *thr, const char *name, bgthread_t *func, /* cache_ws.c */ void WS_Init(struct ws *ws, const char *id, void *space, unsigned len); + +/* WS_Reserve(): Use WS_ReserveSize() or WS_ReserveAll() */ unsigned WS_Reserve(struct ws *ws, unsigned bytes); +unsigned WS_ReserveSize(struct ws *, unsigned); unsigned WS_ReserveAll(struct ws *); unsigned WS_ReserveLumps(struct ws *ws, size_t sz); void WS_MarkOverflow(struct ws *ws); diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index eac4773fc..8c9e2d7c9 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -112,7 +112,7 @@ ses_reserve_attr(struct sess *sp, enum sess_attr a, void **dst, int sz) assert(a < SA_LAST); assert(sz >= 0); AN(dst); - o = WS_Reserve(sp->ws, sz); + o = WS_ReserveSize(sp->ws, sz); assert(o >= sz); *dst = sp->ws->f; o = sp->ws->f - sp->ws->s; @@ -415,7 +415,7 @@ ses_handle(struct waited *wp, enum wait_event ev, vtim_real now) case WAITER_ACTION: pp = sp->pool; CHECK_OBJ_NOTNULL(pp, POOL_MAGIC); - assert(sizeof *tp <= WS_Reserve(sp->ws, sizeof *tp)); + assert(sizeof *tp <= WS_ReserveSize(sp->ws, sizeof *tp)); tp = (void*)sp->ws->f; tp->func = xp->unwait; tp->priv = sp; @@ -453,7 +453,7 @@ SES_Wait(struct sess *sp, const struct transport *xp) /* * put struct waited on the workspace */ - if (WS_Reserve(sp->ws, sizeof(struct waited)) + if (WS_ReserveSize(sp->ws, sizeof(struct waited)) < sizeof(struct waited)) { SES_Delete(sp, SC_OVERLOAD, NAN); return; diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c index a0fc4eacb..c6a37b862 100644 --- a/bin/varnishd/cache/cache_wrk.c +++ b/bin/varnishd/cache/cache_wrk.c @@ -238,7 +238,7 @@ Pool_Task_Arg(struct worker *wrk, enum task_prio prio, task_func_t *func, retval = 0; } AZ(wrk2->task.func); - assert(arg_len <= WS_Reserve(wrk2->aws, arg_len)); + assert(arg_len <= WS_ReserveSize(wrk2->aws, arg_len)); memcpy(wrk2->aws->f, arg, arg_len); wrk2->task.func = func; wrk2->task.priv = wrk2->aws->f; diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c index e0a9b1d48..7b8df6d23 100644 --- a/bin/varnishd/cache/cache_ws.c +++ b/bin/varnishd/cache/cache_ws.c @@ -247,9 +247,34 @@ WS_ReserveAll(struct ws *ws) } /* - * bytes == 0 argument is deprecated - use WS_ReserveAll - * - * XXX rename to WS_ReserveSize and macro-wrap WS_Reserve to emit #warn ? + * WS_Release() must be called for retval > 0 only + */ +unsigned +WS_ReserveSize(struct ws *ws, unsigned bytes) +{ + unsigned b2; + + WS_Assert(ws); + assert(ws->r == NULL); + assert(bytes > 0); + + b2 = PRNDDN(ws->e - ws->f); + if (bytes < b2) + b2 = PRNDUP(bytes); + + if (ws->f + b2 > ws->e) { + WS_MarkOverflow(ws); + return (0); + } + ws->r = ws->f + b2; + DSL(DBG_WORKSPACE, 0, "WS_ReserveSize(%p, %u/%u) = %u", + ws, b2, bytes, pdiff(ws->f, ws->r)); + WS_Assert(ws); + return (pdiff(ws->f, ws->r)); +} + +/* + * XXX remove for 2020-03-15 release */ unsigned WS_Reserve(struct ws *ws, unsigned bytes) diff --git a/lib/libvmod_proxy/vmod_proxy.c b/lib/libvmod_proxy/vmod_proxy.c index fa7b9617e..58f2db295 100644 --- a/lib/libvmod_proxy/vmod_proxy.c +++ b/lib/libvmod_proxy/vmod_proxy.c @@ -105,7 +105,7 @@ tlv_string(VRT_CTX, int tlv) if (VPX_tlv(ctx->req, tlv, (void **)&dst, &len)) return (NULL); - if (!WS_Reserve(ctx->ws, len+1)) + if (!WS_ReserveSize(ctx->ws, len+1)) return (NULL); d = ctx->ws->f; memcpy(d, dst, len); From martin at varnish-software.com Tue Feb 4 10:03:08 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:03:08 +0000 (UTC) Subject: [6.2] 4921b0084 add a facility to test WS_ReserveSize() Message-ID: <20200204100308.398C4A3C65@lists.varnish-cache.org> commit 4921b008487555e2a32d542309e81eefc7870b42 Author: Nils Goroll Date: Tue Nov 26 13:21:37 2019 +0100 add a facility to test WS_ReserveSize() (cherry picked from commit ed3b095c0414af453493580a654f63a02d22e0e8) diff --git a/lib/libvmod_vtc/vmod.vcc b/lib/libvmod_vtc/vmod.vcc index e1644fffc..907d8720b 100644 --- a/lib/libvmod_vtc/vmod.vcc +++ b/lib/libvmod_vtc/vmod.vcc @@ -96,6 +96,14 @@ as much as needed to leave that many bytes free. The actual allocation size may be higher to comply with memory alignment requirements of the CPU architecture. A failed allocation fails the transaction. +$Function BYTES workspace_reserve(ENUM { client, backend, session, thread }, + INT size) + +Attempt to reserve *size* bytes and release the reservation right +away. Return the size of the reservation. + +See `vtc.workspace_alloc()` for semantics of the *size* argument. + $Function INT workspace_free(ENUM { client, backend, session, thread }) Find how much unallocated space there is left in a workspace. diff --git a/lib/libvmod_vtc/vmod_vtc.c b/lib/libvmod_vtc/vmod_vtc.c index ee9d23fa0..a2342c110 100644 --- a/lib/libvmod_vtc/vmod_vtc.c +++ b/lib/libvmod_vtc/vmod_vtc.c @@ -169,6 +169,34 @@ vmod_workspace_alloc(VRT_CTX, VCL_ENUM which, VCL_INT size) memset(p, '\0', size); } +VCL_BYTES v_matchproto_(td_vtc_workspace_reserve) +vmod_workspace_reserve(VRT_CTX, VCL_ENUM which, VCL_INT size) +{ + struct ws *ws; + unsigned r; + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + + ws = vtc_ws_find(ctx, which); + if (ws == NULL) + return (0); + WS_Assert(ws); + + if (size < 0) { + size += WS_ReserveAll(ws); + WS_Release(ws, 0); + } + if (size <= 0) { + VRT_fail(ctx, "Attempted negative WS reservation"); + return (0); + } + r = WS_ReserveSize(ws, size); + if (r == 0) + return (0); + WS_Release(ws, 0); + return (1); +} + VCL_INT v_matchproto_(td_vtc_workspace_free) vmod_workspace_free(VRT_CTX, VCL_ENUM which) { From martin at varnish-software.com Tue Feb 4 10:03:08 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:03:08 +0000 (UTC) Subject: [6.2] ccc936474 WS_ReserveSize() must not hold a reservation for zero return value Message-ID: <20200204100308.53A81A3C74@lists.varnish-cache.org> commit ccc936474ce64d710394df3565efaf48a8169b8e Author: Nils Goroll Date: Tue Nov 26 13:27:09 2019 +0100 WS_ReserveSize() must not hold a reservation for zero return value This originates from a3d47c258fb7938f67a053f6d041257edb69afe9, but was overlooked in 4e33359772a3d751b2ef4b5c4b40259f1bcd6903: When there is insufficient space to fulfil the reservation request, we must not leave the workspace reserved. Fixes varnishcache/varnish-cache#3131 (cherry picked from commit 505b7bd9643006fa8e3977f920564ce12a2b24a2) diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c index 7b8df6d23..e98e2cc69 100644 --- a/bin/varnishd/cache/cache_ws.c +++ b/bin/varnishd/cache/cache_ws.c @@ -262,7 +262,7 @@ WS_ReserveSize(struct ws *ws, unsigned bytes) if (bytes < b2) b2 = PRNDUP(bytes); - if (ws->f + b2 > ws->e) { + if (ws->f + b2 >= ws->e) { WS_MarkOverflow(ws); return (0); } diff --git a/bin/varnishtest/tests/r03131.vtc b/bin/varnishtest/tests/r03131.vtc new file mode 100644 index 000000000..2722b748a --- /dev/null +++ b/bin/varnishtest/tests/r03131.vtc @@ -0,0 +1,24 @@ +varnishtest "Test workspace functions in vmod_vtc" + +varnish v1 -vcl { + import vtc; + import std; + + backend dummy {.host = "${bad_backend}"; } + + sub vcl_recv { + return (synth(200)); + } + + sub vcl_synth { + set resp.http.res1 = vtc.workspace_reserve(client, 1024 * 1024); + vtc.workspace_alloc(client, -1); + set resp.http.res2 = vtc.workspace_reserve(client, 8); + set resp.http.res3 = vtc.workspace_reserve(client, 8); + } +} -start + +client c1 { + txreq + rxresp +} -run From martin at varnish-software.com Tue Feb 4 10:03:08 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:03:08 +0000 (UTC) Subject: [6.2] 299e29838 fix copy-pasta vtc description Message-ID: <20200204100308.6F64FA3C84@lists.varnish-cache.org> commit 299e298382dbe609363ed5a0fad74d4cc17b446f Author: Nils Goroll Date: Tue Nov 26 14:09:32 2019 +0100 fix copy-pasta vtc description (cherry picked from commit 815331b3aecca0eb7e4e29f6dd4d778124c2ac9d) diff --git a/bin/varnishtest/tests/r03131.vtc b/bin/varnishtest/tests/r03131.vtc index 2722b748a..43f0995b0 100644 --- a/bin/varnishtest/tests/r03131.vtc +++ b/bin/varnishtest/tests/r03131.vtc @@ -1,4 +1,4 @@ -varnishtest "Test workspace functions in vmod_vtc" +varnishtest "Test WS_ReserveSize() overflow behavior" varnish v1 -vcl { import vtc; From martin at varnish-software.com Tue Feb 4 10:03:08 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:03:08 +0000 (UTC) Subject: [6.2] dd33a9251 Add Session Attribute workspace overflow handling Message-ID: <20200204100308.8D0D1A3C91@lists.varnish-cache.org> commit dd33a92513f9339c7e3cd977ff0aae56688c13fe Author: Nils Goroll Date: Tue Nov 26 14:56:24 2019 +0100 Add Session Attribute workspace overflow handling Notes: * for the acceptor, I think it makes sense to keep AN assertion (pun!) because varnish is not viable if the session workspace is too small to even hold the attributes initialized in the acceptor. If this was an issue, we should rather revisit the minimum values for the session workspace * for h1 and h2 session setup, I have used XXXAN() because I am not sure how we should best handle allocation failures. * The relevant bit, for now, is the proxy code which may allocate arbitrarily long TLV attributes, so this is the code for which we now actually handle errors and test that we do On the vtc: I added the test to o00005.vtc because there existed a previous overflow test from 267504b8143bdb8ef96240455a3aa788f96b579b, but that only tested for the one case of a WS overflow which was already handled. Fixes varnishcache/varnish-cache#3145 (cherry picked from commit 287dc4a6745c374e0b229bfa861d664989a3a9e8) diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c index 8300d1b5d..ee0b6063e 100644 --- a/bin/varnishd/cache/cache_acceptor.c +++ b/bin/varnishd/cache/cache_acceptor.c @@ -317,17 +317,17 @@ vca_mk_tcp(const struct wrk_accept *wa, struct sockaddr_storage ss; socklen_t sl; - SES_Reserve_remote_addr(sp, &sa); + AN(SES_Reserve_remote_addr(sp, &sa)); AN(VSA_Build(sa, &wa->acceptaddr, wa->acceptaddrlen)); sp->sattr[SA_CLIENT_ADDR] = sp->sattr[SA_REMOTE_ADDR]; VTCP_name(sa, raddr, VTCP_ADDRBUFSIZE, rport, VTCP_PORTBUFSIZE); - SES_Set_String_Attr(sp, SA_CLIENT_IP, raddr); - SES_Set_String_Attr(sp, SA_CLIENT_PORT, rport); + AN(SES_Set_String_Attr(sp, SA_CLIENT_IP, raddr)); + AN(SES_Set_String_Attr(sp, SA_CLIENT_PORT, rport)); sl = sizeof ss; AZ(getsockname(sp->fd, (void*)&ss, &sl)); - SES_Reserve_local_addr(sp, &sa); + AN(SES_Reserve_local_addr(sp, &sa)); AN(VSA_Build(sa, &ss, sl)); sp->sattr[SA_SERVER_ADDR] = sp->sattr[SA_LOCAL_ADDR]; VTCP_name(sa, laddr, VTCP_ADDRBUFSIZE, lport, VTCP_PORTBUFSIZE); @@ -340,13 +340,13 @@ vca_mk_uds(struct wrk_accept *wa, struct sess *sp, char *laddr, char *lport, struct suckaddr *sa; (void) wa; - SES_Reserve_remote_addr(sp, &sa); + AN(SES_Reserve_remote_addr(sp, &sa)); AZ(SES_Set_remote_addr(sp, bogo_ip)); sp->sattr[SA_CLIENT_ADDR] = sp->sattr[SA_REMOTE_ADDR]; sp->sattr[SA_LOCAL_ADDR] = sp->sattr[SA_REMOTE_ADDR]; sp->sattr[SA_SERVER_ADDR] = sp->sattr[SA_REMOTE_ADDR]; - SES_Set_String_Attr(sp, SA_CLIENT_IP, "0.0.0.0"); - SES_Set_String_Attr(sp, SA_CLIENT_PORT, "0"); + AN(SES_Set_String_Attr(sp, SA_CLIENT_IP, "0.0.0.0")); + AN(SES_Set_String_Attr(sp, SA_CLIENT_PORT, "0")); strcpy(laddr, "0.0.0.0"); strcpy(raddr, "0.0.0.0"); diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index 8c9e2d7c9..c9a0f5180 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -103,8 +103,8 @@ ses_set_attr(const struct sess *sp, enum sess_attr a, const void *src, int sz) return (0); } -static void -ses_reserve_attr(struct sess *sp, enum sess_attr a, void **dst, int sz) +static int +ses_res_attr(struct sess *sp, enum sess_attr a, void **dst, int sz) { ssize_t o; @@ -113,12 +113,14 @@ ses_reserve_attr(struct sess *sp, enum sess_attr a, void **dst, int sz) assert(sz >= 0); AN(dst); o = WS_ReserveSize(sp->ws, sz); - assert(o >= sz); + if (o < sz) + return (0); *dst = sp->ws->f; o = sp->ws->f - sp->ws->s; WS_Release(sp->ws, sz); assert(o >= 0 && o <= 0xffff); sp->sattr[a] = (uint16_t)o; + return (1); } #define SESS_ATTR(UP, low, typ, len) \ @@ -134,16 +136,16 @@ ses_reserve_attr(struct sess *sp, enum sess_attr a, void **dst, int sz) return (ses_get_attr(sp, SA_##UP, (void**)dst)); \ } \ \ - void \ + int \ SES_Reserve_##low(struct sess *sp, typ **dst) \ { \ - assert(len >= 0); \ - ses_reserve_attr(sp, SA_##UP, (void**)dst, len); \ + assert(len > 0); \ + return (ses_res_attr(sp, SA_##UP, (void**)dst, len)); \ } #include "tbl/sess_attr.h" -void +int SES_Set_String_Attr(struct sess *sp, enum sess_attr a, const char *src) { void *q; @@ -157,8 +159,10 @@ SES_Set_String_Attr(struct sess *sp, enum sess_attr a, const char *src) default: WRONG("wrong sess_attr"); } - ses_reserve_attr(sp, a, &q, strlen(src) + 1); + if (! ses_res_attr(sp, a, &q, strlen(src) + 1)) + return (0); strcpy(q, src); + return (1); } const char * diff --git a/bin/varnishd/cache/cache_varnishd.h b/bin/varnishd/cache/cache_varnishd.h index f5c4bf656..4d6ce1022 100644 --- a/bin/varnishd/cache/cache_varnishd.h +++ b/bin/varnishd/cache/cache_varnishd.h @@ -378,9 +378,9 @@ enum htc_status_e HTC_RxStuff(struct http_conn *, htc_complete_f *, #define SESS_ATTR(UP, low, typ, len) \ int SES_Set_##low(const struct sess *sp, const typ *src); \ - void SES_Reserve_##low(struct sess *sp, typ **dst); + int SES_Reserve_##low(struct sess *sp, typ **dst); #include "tbl/sess_attr.h" -void SES_Set_String_Attr(struct sess *sp, enum sess_attr a, const char *src); +int SES_Set_String_Attr(struct sess *sp, enum sess_attr a, const char *src); enum htc_status_e { diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c index 4f6e8efd1..2b937ec86 100644 --- a/bin/varnishd/http1/cache_http1_fsm.c +++ b/bin/varnishd/http1/cache_http1_fsm.c @@ -112,7 +112,7 @@ http1_new_session(struct worker *wrk, void *arg) CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); HTC_RxInit(req->htc, req->ws); - SES_Reserve_proto_priv(sp, &u); + XXXAN(SES_Reserve_proto_priv(sp, &u)); http1_setstate(sp, H1NEWREQ); wrk->task.func = http1_req; wrk->task.priv = req; diff --git a/bin/varnishd/http2/cache_http2_session.c b/bin/varnishd/http2/cache_http2_session.c index efd560db6..788879bbf 100644 --- a/bin/varnishd/http2/cache_http2_session.c +++ b/bin/varnishd/http2/cache_http2_session.c @@ -101,7 +101,7 @@ h2_init_sess(const struct worker *wrk, struct sess *sp, if (SES_Get_proto_priv(sp, &up)) { /* Already reserved if we came via H1 */ - SES_Reserve_proto_priv(sp, &up); + XXXAN(SES_Reserve_proto_priv(sp, &up)); *up = 0; } if (*up == 0) { @@ -130,7 +130,7 @@ h2_init_sess(const struct worker *wrk, struct sess *sp, AZ(VHT_Init(h2->dectbl, h2->local_settings.header_table_size)); - SES_Reserve_proto_priv(sp, &up); + XXXAN(SES_Reserve_proto_priv(sp, &up)); *up = (uintptr_t)h2; } AN(up); diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c index 86fc2f97d..f5fecad02 100644 --- a/bin/varnishd/proxy/cache_proxy_proto.c +++ b/bin/varnishd/proxy/cache_proxy_proto.c @@ -49,6 +49,13 @@ struct vpx_tlv { char tlv[1]; }; +static inline int +vpx_ws_err(const struct req *req) +{ + VSL(SLT_Error, req->sp->vxid, "insufficient workspace"); + return (-1); +} + /********************************************************************** * PROXY 1 protocol */ @@ -128,10 +135,19 @@ vpx_proto1(const struct worker *wrk, const struct req *req) freeaddrinfo(res); return (-1); } - SES_Reserve_client_addr(req->sp, &sa); + if (! SES_Reserve_client_addr(req->sp, &sa)) { + freeaddrinfo(res); + return (vpx_ws_err(req)); + } AN(VSA_Build(sa, res->ai_addr, res->ai_addrlen)); - SES_Set_String_Attr(req->sp, SA_CLIENT_IP, fld[1]); - SES_Set_String_Attr(req->sp, SA_CLIENT_PORT, fld[3]); + if (! SES_Set_String_Attr(req->sp, SA_CLIENT_IP, fld[1])) { + freeaddrinfo(res); + return (vpx_ws_err(req)); + } + if (! SES_Set_String_Attr(req->sp, SA_CLIENT_PORT, fld[3])) { + freeaddrinfo(res); + return (vpx_ws_err(req)); + } freeaddrinfo(res); i = getaddrinfo(fld[2], fld[4], &hints, &res); @@ -149,7 +165,10 @@ vpx_proto1(const struct worker *wrk, const struct req *req) freeaddrinfo(res); return (-1); } - SES_Reserve_server_addr(req->sp, &sa); + if (! SES_Reserve_server_addr(req->sp, &sa)) { + freeaddrinfo(res); + return (vpx_ws_err(req)); + } AN(VSA_Build(sa, res->ai_addr, res->ai_addrlen)); freeaddrinfo(res); @@ -403,14 +422,16 @@ vpx_proto2(const struct worker *wrk, struct req *req) /* dst/server */ memcpy(&sin4.sin_addr, p + 20, 4); memcpy(&sin4.sin_port, p + 26, 2); - SES_Reserve_server_addr(req->sp, &sa); + if (! SES_Reserve_server_addr(req->sp, &sa)) + return (vpx_ws_err(req)); AN(VSA_Build(sa, &sin4, sizeof sin4)); VTCP_name(sa, ha, sizeof ha, pa, sizeof pa); /* src/client */ memcpy(&sin4.sin_addr, p + 16, 4); memcpy(&sin4.sin_port, p + 24, 2); - SES_Reserve_client_addr(req->sp, &sa); + if (! SES_Reserve_client_addr(req->sp, &sa)) + return (vpx_ws_err(req)); AN(VSA_Build(sa, &sin4, sizeof sin4)); break; case 0x21: @@ -429,14 +450,16 @@ vpx_proto2(const struct worker *wrk, struct req *req) /* dst/server */ memcpy(&sin6.sin6_addr, p + 32, 16); memcpy(&sin6.sin6_port, p + 50, 2); - SES_Reserve_server_addr(req->sp, &sa); + if (! SES_Reserve_server_addr(req->sp, &sa)) + return (vpx_ws_err(req)); AN(VSA_Build(sa, &sin6, sizeof sin6)); VTCP_name(sa, ha, sizeof ha, pa, sizeof pa); /* src/client */ memcpy(&sin6.sin6_addr, p + 16, 16); memcpy(&sin6.sin6_port, p + 48, 2); - SES_Reserve_client_addr(req->sp, &sa); + if (! SES_Reserve_client_addr(req->sp, &sa)) + return (vpx_ws_err(req)); AN(VSA_Build(sa, &sin6, sizeof sin6)); break; default: @@ -448,8 +471,10 @@ vpx_proto2(const struct worker *wrk, struct req *req) AN(sa); VTCP_name(sa, hb, sizeof hb, pb, sizeof pb); - SES_Set_String_Attr(req->sp, SA_CLIENT_IP, hb); - SES_Set_String_Attr(req->sp, SA_CLIENT_PORT, pb); + if (! SES_Set_String_Attr(req->sp, SA_CLIENT_IP, hb)) + return (vpx_ws_err(req)); + if (! SES_Set_String_Attr(req->sp, SA_CLIENT_PORT, pb)) + return (vpx_ws_err(req)); VSL(SLT_Proxy, req->sp->vxid, "2 %s %s %s %s", hb, pb, ha, pa); @@ -476,15 +501,13 @@ vpx_proto2(const struct worker *wrk, struct req *req) return (-1); } tlv = WS_Alloc(req->sp->ws, sizeof *tlv + tlv_len); - if (tlv == NULL) { - VSL(SLT_ProxyGarbage, req->sp->vxid, - "PROXY2: TLV overflows WS"); - return (-1); - } + if (tlv == NULL) + return (vpx_ws_err(req)); INIT_OBJ(tlv, VPX_TLV_MAGIC); tlv->len = tlv_len; memcpy(tlv->tlv, tlv_start, tlv_len); - SES_Reserve_proxy_tlv(req->sp, &up); + if (! SES_Reserve_proxy_tlv(req->sp, &up)) + return (vpx_ws_err(req)); *up = (uintptr_t)tlv; return (0); } diff --git a/bin/varnishtest/tests/o00005.vtc b/bin/varnishtest/tests/o00005.vtc index 3ae1aacfe..d84b92d4d 100644 --- a/bin/varnishtest/tests/o00005.vtc +++ b/bin/varnishtest/tests/o00005.vtc @@ -5,7 +5,8 @@ server s1 { txresp } -start -varnish v1 -proto "PROXY" -vcl+backend { +varnish v1 -arg "-p pool_sess=0,0,0" -proto "PROXY" -vcl+backend { + import vtc; import proxy; sub vcl_deliver { @@ -20,6 +21,7 @@ varnish v1 -proto "PROXY" -vcl+backend { set resp.http.key = proxy.cert_key(); set resp.http.sign = proxy.cert_sign(); set resp.http.cn = proxy.client_cert_cn(); + set resp.http.ws_free = vtc.workspace_free(session); } } -start @@ -243,4 +245,68 @@ client c1 { expect_close } -run +delay 1 + varnish v1 -expect ws_session_overflow == 1 + +# error handling elsewhere in the proxy code +# request is the same as initial + +varnish v1 -cliok "param.set workspace_session 450" +varnish v1 -cliok "param.set pool_sess 10,100,1" + +delay 1 + +# get rid of the surplus session mpl +client c10 -proxy1 "1.2.3.4:1111 5.6.7.8:5678" { + txreq + rxresp +} -start +client c11 -proxy1 "1.2.3.4:1111 5.6.7.8:5678" { + txreq + rxresp +} -start +client c12 -proxy1 "1.2.3.4:1111 5.6.7.8:5678" { + txreq + rxresp +} -start +client c13 -proxy1 "1.2.3.4:1111 5.6.7.8:5678" { + txreq + rxresp +} -start +client c14 -proxy1 "1.2.3.4:1111 5.6.7.8:5678" { + txreq + rxresp +} -start + +client c10 -wait +client c11 -wait +client c12 -wait +client c13 -wait +client c14 -wait + +client c2 { + # PROXY2 with CRC32C TLV + sendhex { + 0d 0a 0d 0a 00 0d 0a 51 55 49 54 0a + 21 11 00 65 + d9 46 b5 21 + 5f 8e a8 22 + ed 96 + 01 bb + 03 00 04 95 03 ee 75 + 01 00 02 68 32 + 02 00 0a 68 6f 63 64 65 74 2e 6e 65 74 + 20 00 3d + 01 00 00 00 00 + 21 00 07 54 4c 53 76 31 2e 33 + 25 00 05 45 43 32 35 36 + 24 00 0a 52 53 41 2d 53 48 41 32 35 36 + 23 00 16 41 45 41 44 2d 41 45 53 31 32 38 + 2d 47 43 4d 2d 53 48 41 32 35 36 + } + txreq + expect_close +} -run + +varnish v1 -expect ws_session_overflow == 2 From martin at varnish-software.com Tue Feb 4 10:03:08 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:03:08 +0000 (UTC) Subject: [6.2] 33b8240b5 Simplify WS allocation in tlv_string Message-ID: <20200204100308.A4F0BA3C96@lists.varnish-cache.org> commit 33b8240b55156c303856868b822f054c6d86147c Author: Emmanuel Hocdet Date: Thu Nov 14 11:14:07 2019 +0100 Simplify WS allocation in tlv_string Patch by @ehocdet, commit message edited by @nigoroll: The root cause of #3131 was misdiagnosed to the extent that, while this change had prevented it, the root cause was a bug in WS_ReserveSize() fixed in 505b7bd9643006fa8e3977f920564ce12a2b24a2 The previous tlv_string() code was correct except for the fact that error handling should have checked for WS_ReserveSize(ctx->ws, len+1) <= len (also spotted by @ehocdet). Someone had mentioned at some point that we would not want to VRT_fail(), but I think this must have been related to the proxy transport code, not the proxy vmod. Ref varnishcache/varnish-cache#3131 (cherry picked from commit e74f9e871bc0eab014b7dd359d6ae83153c3ee37) diff --git a/lib/libvmod_proxy/vmod_proxy.c b/lib/libvmod_proxy/vmod_proxy.c index 58f2db295..222de9c53 100644 --- a/lib/libvmod_proxy/vmod_proxy.c +++ b/lib/libvmod_proxy/vmod_proxy.c @@ -105,12 +105,13 @@ tlv_string(VRT_CTX, int tlv) if (VPX_tlv(ctx->req, tlv, (void **)&dst, &len)) return (NULL); - if (!WS_ReserveSize(ctx->ws, len+1)) + d = WS_Alloc(ctx->ws, len+1); + if (d == NULL) { + VRT_fail(ctx, "proxy.TLV: out of workspace"); return (NULL); - d = ctx->ws->f; + } memcpy(d, dst, len); d[len] = '\0'; - WS_Release(ctx->ws, len+1); return (d); } From martin at varnish-software.com Tue Feb 4 10:03:08 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:03:08 +0000 (UTC) Subject: [6.2] f286ed20f Try to make the proxy code session workspace overflow test on 32bit Message-ID: <20200204100308.BB76AA3C9E@lists.varnish-cache.org> commit f286ed20f6f5e7329e2a2967e6492f9a2d97207c Author: Nils Goroll Date: Thu Nov 28 11:19:08 2019 +0100 Try to make the proxy code session workspace overflow test on 32bit Ref varnishcache/varnish-cache#3145 / 287dc4a6745c374e0b229bfa861d664989a3a9e8 (cherry picked from commit e1a57eb7a7350ac4c8a0235dc991bef400b118c1) diff --git a/bin/varnishtest/tests/o00005.vtc b/bin/varnishtest/tests/o00005.vtc index d84b92d4d..2ac11d1a8 100644 --- a/bin/varnishtest/tests/o00005.vtc +++ b/bin/varnishtest/tests/o00005.vtc @@ -249,10 +249,16 @@ delay 1 varnish v1 -expect ws_session_overflow == 1 -# error handling elsewhere in the proxy code -# request is the same as initial - -varnish v1 -cliok "param.set workspace_session 450" +# workspace overflow handling elsewhere in the proxy code +# +# the workspace_session size is chosen to fail as closely as possible +# below the minimum required value for the vtc to work (= the +# workspace to overflow) both on 64 and 32 bit. +# +# This value is fragile, ideally we would want something like +# vtc.alloc(-x), yet there is no vcl code being run before we parse +# proxy headers +varnish v1 -cliok "param.set workspace_session 410" varnish v1 -cliok "param.set pool_sess 10,100,1" delay 1 From martin at varnish-software.com Tue Feb 4 10:03:08 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:03:08 +0000 (UTC) Subject: [6.2] 9296c5471 more tweaking for 32bit vtest machines Message-ID: <20200204100308.D52E3A3CA7@lists.varnish-cache.org> commit 9296c547138bc6898d3ed8238019a0a5db4aa9b7 Author: Nils Goroll Date: Thu Nov 28 12:21:17 2019 +0100 more tweaking for 32bit vtest machines Ref varnishcache/varnish-cache#3145 / 287dc4a6745c374e0b229bfa861d664989a3a9e8 (cherry picked from commit d6dec03101a67c38addf549435ae51df38f1d7db) diff --git a/bin/varnishtest/tests/o00005.vtc b/bin/varnishtest/tests/o00005.vtc index 2ac11d1a8..b09a8fcaf 100644 --- a/bin/varnishtest/tests/o00005.vtc +++ b/bin/varnishtest/tests/o00005.vtc @@ -258,7 +258,7 @@ varnish v1 -expect ws_session_overflow == 1 # This value is fragile, ideally we would want something like # vtc.alloc(-x), yet there is no vcl code being run before we parse # proxy headers -varnish v1 -cliok "param.set workspace_session 410" +varnish v1 -cliok "param.set workspace_session 402" varnish v1 -cliok "param.set pool_sess 10,100,1" delay 1 From martin at varnish-software.com Tue Feb 4 10:03:08 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:03:08 +0000 (UTC) Subject: [6.2] e20d258a8 Handle out of session workspace in http1_new_session() Message-ID: <20200204100308.ED756A3CC3@lists.varnish-cache.org> commit e20d258a89774577d14273ae759d69b6d33717e9 Author: Martin Blix Grydeland Date: Thu Dec 12 13:12:48 2019 +0100 Handle out of session workspace in http1_new_session() If proxy protocol is in use, it is possible to fill the session workspace exactly before entering http1_new_session(), which will cause it to assert when calling SES_Reserve_proto_priv(). with this patch we will close the session gracefully. diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c index 2b937ec86..fe8de28e0 100644 --- a/bin/varnishd/http1/cache_http1_fsm.c +++ b/bin/varnishd/http1/cache_http1_fsm.c @@ -112,7 +112,15 @@ http1_new_session(struct worker *wrk, void *arg) CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); HTC_RxInit(req->htc, req->ws); - XXXAN(SES_Reserve_proto_priv(sp, &u)); + if (!SES_Reserve_proto_priv(sp, &u)) { + /* Out of session workspace. Free the req, close the sess, + * and do not set a new task func, which will exit the + * worker thread. */ + VSL(SLT_Error, req->sp->vxid, "insufficient workspace"); + Req_Release(req); + SES_Delete(sp, SC_RX_JUNK, NAN); + return; + } http1_setstate(sp, H1NEWREQ); wrk->task.func = http1_req; wrk->task.priv = req; diff --git a/bin/varnishtest/tests/f00005.vtc b/bin/varnishtest/tests/f00005.vtc new file mode 100644 index 000000000..866d941dd --- /dev/null +++ b/bin/varnishtest/tests/f00005.vtc @@ -0,0 +1,64 @@ +varnishtest "proxy ws panic" + +server s1 { + rxreq + txresp +} -start + +varnish v1 -proto "PROXY" -vcl+backend {}-start + +# Too large proxy payload using TLV +client c1 { + sendhex { +0d 0a 0d 0a 00 0d 0a 51 +55 49 54 0a 21 21 00 93 +aa bb cc dd ee ff 11 22 +33 44 55 66 77 88 99 aa +bb cc dd ee ff 11 22 33 +44 55 66 77 88 99 aa bb +88 da 0d 73 02 00 3c 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 20 00 2d 01 01 +00 00 00 21 00 07 54 4c +53 76 31 2e 32 23 00 1b +45 43 44 48 45 2d 52 53 +41 2d 41 45 53 32 35 36 +2d 47 43 4d 2d 53 48 41 +33 38 34 + } + expect_close +} -run + +# Reduced size proxy payload to verify Varnish is still running +client c1 { + sendhex { +0d 0a 0d 0a 00 0d 0a 51 +55 49 54 0a 21 21 00 8b +aa bb cc dd ee ff 11 22 +33 44 55 66 77 88 99 aa +bb cc dd ee ff 11 22 33 +44 55 66 77 88 99 aa bb +88 da 0d 73 02 00 34 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 61 61 61 61 61 +61 61 61 20 00 2d 01 01 +00 00 00 21 00 07 54 4c +53 76 31 2e 32 23 00 1b +45 43 44 48 45 2d 52 53 +41 2d 41 45 53 32 35 36 +2d 47 43 4d 2d 53 48 41 +33 38 34 + } + txreq + rxresp +} -run From martin at varnish-software.com Tue Feb 4 10:03:09 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:03:09 +0000 (UTC) Subject: [6.2] e5bdc9b96 Remove call to SES_Reserve_proto_priv in h2_init_sess Message-ID: <20200204100309.3CAB7A3CEA@lists.varnish-cache.org> commit e5bdc9b96327097d7f4e948149370e8fbad36954 Author: Martin Blix Grydeland Date: Thu Dec 12 13:51:17 2019 +0100 Remove call to SES_Reserve_proto_priv in h2_init_sess h2_init_sess can only be reached through H1 with either previous knowledge or opportunistic upgrade. Because of this the proto_priv session attribute will always be set before entry. This patch simplifies and removes dead code containing a call to SES_Reserve_proto_priv. Note: Better diff with the --ignore-all-space option diff --git a/bin/varnishd/http2/cache_http2_session.c b/bin/varnishd/http2/cache_http2_session.c index 977669589..ea32cad27 100644 --- a/bin/varnishd/http2/cache_http2_session.c +++ b/bin/varnishd/http2/cache_http2_session.c @@ -99,41 +99,37 @@ h2_init_sess(const struct worker *wrk, struct sess *sp, uintptr_t *up; struct h2_sess *h2; - if (SES_Get_proto_priv(sp, &up)) { - /* Already reserved if we came via H1 */ - XXXAN(SES_Reserve_proto_priv(sp, &up)); - *up = 0; - } - if (*up == 0) { - if (srq == NULL) - srq = Req_New(wrk, sp); - AN(srq); - h2 = h2s; - AN(h2); - INIT_OBJ(h2, H2_SESS_MAGIC); - h2->srq = srq; - h2->htc = srq->htc; - h2->ws = srq->ws; - h2->vsl = srq->vsl; - VSL_Flush(h2->vsl, 0); - h2->vsl->wid = sp->vxid; - h2->htc->rfd = &sp->fd; - h2->sess = sp; - h2->rxthr = pthread_self(); - AZ(pthread_cond_init(h2->winupd_cond, NULL)); - VTAILQ_INIT(&h2->streams); - VTAILQ_INIT(&h2->txqueue); - h2_local_settings(&h2->local_settings); - h2->remote_settings = H2_proto_settings; - h2->decode = decode; - - AZ(VHT_Init(h2->dectbl, - h2->local_settings.header_table_size)); - - *up = (uintptr_t)h2; - } - AN(up); - CAST_OBJ_NOTNULL(h2, (void*)(*up), H2_SESS_MAGIC); + /* proto_priv session attribute will always have been set up by H1 + * before reaching here. */ + AZ(SES_Get_proto_priv(sp, &up)); + assert(*up == 0); + + if (srq == NULL) + srq = Req_New(wrk, sp); + AN(srq); + h2 = h2s; + AN(h2); + INIT_OBJ(h2, H2_SESS_MAGIC); + h2->srq = srq; + h2->htc = srq->htc; + h2->ws = srq->ws; + h2->vsl = srq->vsl; + VSL_Flush(h2->vsl, 0); + h2->vsl->wid = sp->vxid; + h2->htc->rfd = &sp->fd; + h2->sess = sp; + h2->rxthr = pthread_self(); + AZ(pthread_cond_init(h2->winupd_cond, NULL)); + VTAILQ_INIT(&h2->streams); + VTAILQ_INIT(&h2->txqueue); + h2_local_settings(&h2->local_settings); + h2->remote_settings = H2_proto_settings; + h2->decode = decode; + + AZ(VHT_Init(h2->dectbl, h2->local_settings.header_table_size)); + + *up = (uintptr_t)h2; + return (h2); } From martin at varnish-software.com Tue Feb 4 10:03:09 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:03:09 +0000 (UTC) Subject: [6.2] 841c1e00b Remove extra call to SES_Reserve_proto_priv Message-ID: <20200204100309.186FAA3CDB@lists.varnish-cache.org> commit 841c1e00b978a5ecf53f6c72cbd749e1c1730b12 Author: Martin Blix Grydeland Date: Thu Dec 12 13:16:40 2019 +0100 Remove extra call to SES_Reserve_proto_priv In h2_init_sess, an extra call was always made to SES_Reseve_proto_priv(), even though it was already reserved. This wasted a pointer worth of session workspace. This patch removes the extra call. diff --git a/bin/varnishd/http2/cache_http2_session.c b/bin/varnishd/http2/cache_http2_session.c index 788879bbf..977669589 100644 --- a/bin/varnishd/http2/cache_http2_session.c +++ b/bin/varnishd/http2/cache_http2_session.c @@ -130,7 +130,6 @@ h2_init_sess(const struct worker *wrk, struct sess *sp, AZ(VHT_Init(h2->dectbl, h2->local_settings.header_table_size)); - XXXAN(SES_Reserve_proto_priv(sp, &up)); *up = (uintptr_t)h2; } AN(up); From martin at varnish-software.com Tue Feb 4 10:03:09 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:03:09 +0000 (UTC) Subject: [6.2] 0c38acb6d Handle badly formatted proxy TLVs Message-ID: <20200204100309.674ABA3D07@lists.varnish-cache.org> commit 0c38acb6da78081c9c666a9a2426e7ba945c5210 Author: Martin Blix Grydeland Date: Thu Dec 12 14:53:48 2019 +0100 Handle badly formatted proxy TLVs Proxy TLVs claiming to have PP2_TYPE_SSL sub-TLVs without complete payload would cause a Varnish assert. This patch fixes the parsing of the TLVs. diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c index f5fecad02..b84f6242c 100644 --- a/bin/varnishd/proxy/cache_proxy_proto.c +++ b/bin/varnishd/proxy/cache_proxy_proto.c @@ -306,8 +306,9 @@ vpx_tlv_itern(struct vpx_tlv_iter *vpi) return (1); } -#define VPX_TLV_FOREACH(ptr, len, itv) \ - for(vpx_tlv_iter0(itv, ptr, len); vpx_tlv_itern(itv);) +#define VPX_TLV_FOREACH(ptr, len, itv) \ + for (vpx_tlv_iter0(itv, ptr, len); \ + (vpi->e == NULL) && vpx_tlv_itern(itv);) int VPX_tlv(const struct req *req, int typ, void **dst, int *len) @@ -483,6 +484,10 @@ vpx_proto2(const struct worker *wrk, struct req *req) VPX_TLV_FOREACH(d, l, vpi) { if (vpi->t == PP2_TYPE_SSL) { + if (vpi->l < 5) { + vpi->e = "Length Error"; + break; + } VPX_TLV_FOREACH((char*)vpi->p + 5, vpi->l - 5, vpi2) { } vpi->e = vpi2->e; diff --git a/bin/varnishtest/tests/f00005.vtc b/bin/varnishtest/tests/f00005.vtc index 866d941dd..b1b097fce 100644 --- a/bin/varnishtest/tests/f00005.vtc +++ b/bin/varnishtest/tests/f00005.vtc @@ -35,6 +35,18 @@ bb cc dd ee ff 11 22 33 expect_close } -run +# Badly formatted TLV proxy payload +client c1 { + sendhex { +0d 0a 0d 0a 00 0d 0a 51 +55 49 54 0a +21 11 00 13 00 ff 20 ff 10 ff 03 21 20 30 00 20 +20 00 00 19 00 02 29 20 00 00 00 41 20 9e 15 15 +d6 00 00 08 00 00 00 00 00 07 7a 20 b1 3f 43 20 + } + expect_close +} -run + # Reduced size proxy payload to verify Varnish is still running client c1 { sendhex { From martin at varnish-software.com Tue Feb 4 10:03:09 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:03:09 +0000 (UTC) Subject: [6.2] 6d221929f Assert Message-ID: <20200204100309.8347FA3D10@lists.varnish-cache.org> commit 6d221929f761ea6e22e5384aee9981e513edae89 Author: Dridi Boukelmoune Date: Mon Dec 16 12:16:17 2019 +0100 Assert (cherry picked from commit 7da6220dfd86416200c43d84d40fce0cae1b5bae) diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c index e98e2cc69..de04e896d 100644 --- a/bin/varnishd/cache/cache_ws.c +++ b/bin/varnishd/cache/cache_ws.c @@ -235,6 +235,7 @@ WS_ReserveAll(struct ws *ws) { unsigned b; + WS_Assert(ws); assert(ws->r == NULL); ws->r = ws->e; From martin at varnish-software.com Tue Feb 4 10:03:09 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:03:09 +0000 (UTC) Subject: [6.2] bb3a1e362 Fix WS_ReserveSize calls when bytes is equal to free workspace Message-ID: <20200204100309.9F289A3D1E@lists.varnish-cache.org> commit bb3a1e362393ed78a9f55c1de32d58c6f2846420 Author: Martin Blix Grydeland Date: Mon Dec 16 16:37:27 2019 +0100 Fix WS_ReserveSize calls when bytes is equal to free workspace Currently, with the 505b7bd9643006fa8e3977f920564ce12a2b24a2 patch, when calling WS_ReserveSize with bytes equal to the amount of workspace that is currently available, it will return zero and mark overflow. This patch redoes the patch, and changes it to return zero and overflow only when the requested number of bytes is larger than what is available. diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c index de04e896d..5de7a6830 100644 --- a/bin/varnishd/cache/cache_ws.c +++ b/bin/varnishd/cache/cache_ws.c @@ -263,7 +263,7 @@ WS_ReserveSize(struct ws *ws, unsigned bytes) if (bytes < b2) b2 = PRNDUP(bytes); - if (ws->f + b2 >= ws->e) { + if (bytes > b2) { WS_MarkOverflow(ws); return (0); } From martin at varnish-software.com Tue Feb 4 10:03:09 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:03:09 +0000 (UTC) Subject: [6.2] 676e3d2fd Take sizeof pool_task into account when reserving WS in SES_Wait Message-ID: <20200204100309.C115BA3D2C@lists.varnish-cache.org> commit 676e3d2fdf2d44ab2017d0b54eb9cad3535b952c Author: Martin Blix Grydeland Date: Mon Dec 16 17:04:43 2019 +0100 Take sizeof pool_task into account when reserving WS in SES_Wait The assert on WS_ReserveSize() in ses_handle() can not trip because sizeof (struct pool_task) is less than sizeof (struct waited). But to safe guard against future problems if that were to change, this patch makes sure that the session workspace can hold the largest of them before entering the waiter, erroring out if not. diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index c9a0f5180..b6731b820 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -407,6 +407,7 @@ ses_handle(struct waited *wp, enum wait_event ev, vtim_real now) wp->magic = 0; wp = NULL; + /* The WS was reserved in SES_Wait() */ WS_Release(sp->ws, 0); switch (ev) { @@ -419,6 +420,7 @@ ses_handle(struct waited *wp, enum wait_event ev, vtim_real now) case WAITER_ACTION: pp = sp->pool; CHECK_OBJ_NOTNULL(pp, POOL_MAGIC); + /* SES_Wait() guarantees the next will not assert. */ assert(sizeof *tp <= WS_ReserveSize(sp->ws, sizeof *tp)); tp = (void*)sp->ws->f; tp->func = xp->unwait; @@ -442,6 +444,7 @@ SES_Wait(struct sess *sp, const struct transport *xp) { struct pool *pp; struct waited *wp; + unsigned u; CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); CHECK_OBJ_NOTNULL(xp, TRANSPORT_MAGIC); @@ -455,10 +458,15 @@ SES_Wait(struct sess *sp, const struct transport *xp) VTCP_nonblocking(sp->fd); /* - * put struct waited on the workspace + * Put struct waited on the workspace. Make sure that the + * workspace can hold enough space for the largest of struct + * waited and pool_task, as pool_task will be needed when coming + * off the waiter again. */ - if (WS_ReserveSize(sp->ws, sizeof(struct waited)) - < sizeof(struct waited)) { + u = sizeof (struct waited); + if (sizeof (struct pool_task) > u) + u = sizeof (struct pool_task); + if (!WS_ReserveSize(sp->ws, u)) { SES_Delete(sp, SC_OVERLOAD, NAN); return; } From martin at varnish-software.com Tue Feb 4 10:03:09 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:03:09 +0000 (UTC) Subject: [6.2] d67a60e56 Create changelog for 6.2.3 Message-ID: <20200204100309.E8110A3D3A@lists.varnish-cache.org> commit d67a60e56a29747f1e5d34e97c903d052376ec17 Author: Martin Blix Grydeland Date: Fri Jan 31 13:12:46 2020 +0100 Create changelog for 6.2.3 diff --git a/doc/changes.rst b/doc/changes.rst index d421bed9b..70183dd35 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -24,6 +24,18 @@ http://varnish-cache.org/docs/trunk/whats-new/index.html and via individual releases. These documents are updated as part of the release process. +================================ +Varnish Cache 6.2.3 (2020-02-04) +================================ + +Bugs fixed +---------- + +* Fix a denial of service vulnerability when using the proxy protocol + version 2. (VSV00005_) + +.. _VSV00005: https://varnish-cache.org/security/VSV00005.html + ================================ Varnish Cache 6.2.2 (2019-10-21) ================================ From martin at varnish-software.com Tue Feb 4 10:03:10 2020 From: martin at varnish-software.com (Martin Blix Grydeland) Date: Tue, 4 Feb 2020 10:03:10 +0000 (UTC) Subject: [6.2] 84d239c93 Prepare for 6.2.3 Message-ID: <20200204100310.0BCA3A3D46@lists.varnish-cache.org> commit 84d239c93e756ae255b6abb459c1052a36a409e9 Author: Martin Blix Grydeland Date: Fri Jan 31 13:13:16 2020 +0100 Prepare for 6.2.3 diff --git a/configure.ac b/configure.ac index 5586e5f04..f825cd470 100644 --- a/configure.ac +++ b/configure.ac @@ -2,7 +2,7 @@ AC_PREREQ(2.59) AC_COPYRIGHT([Copyright (c) 2006 Verdens Gang AS Copyright (c) 2006-2019 Varnish Software]) AC_REVISION([$Id$]) -AC_INIT([Varnish], [6.2.2], [varnish-dev at varnish-cache.org]) +AC_INIT([Varnish], [6.2.3], [varnish-dev at varnish-cache.org]) AC_CONFIG_SRCDIR(include/miniobj.h) AC_CONFIG_HEADERS([config.h]) AC_CONFIG_MACRO_DIR([m4]) From nils.goroll at uplex.de Tue Feb 4 11:43:05 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 4 Feb 2020 11:43:05 +0000 (UTC) Subject: [master] f75bce17e link to the business page Message-ID: <20200204114305.9AE22AF676@lists.varnish-cache.org> commit f75bce17edbda8c30b083390210463dd0a83fbc9 Author: Nils Goroll Date: Tue Feb 4 12:42:03 2020 +0100 link to the business page diff --git a/README.rst b/README.rst index f18de281c..89c0853a0 100644 --- a/README.rst +++ b/README.rst @@ -11,12 +11,14 @@ to . Please see CONTRIBUTING for how to contribute patches and report bugs. -Questions about commercial support and services related to Varnish -should be addressed to . +For questions about commercial support and services related to Varnish +see the `Varnish HTTP Cache Business page +`_ . .. |ccibadge| image:: https://circleci.com/gh/varnishcache/varnish-cache/tree/master.svg?style=svg :target: https://circleci.com/gh/varnishcache/varnish-cache/tree/master .. _vtest: https://varnish-cache.org/vtest/ + CircleCI tests: |ccibadge| More platforms are tested via vtest_ From dridi.boukelmoune at gmail.com Tue Feb 4 17:49:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 4 Feb 2020 17:49:06 +0000 (UTC) Subject: [master] f78dcb878 Various typos Message-ID: <20200204174906.C721397AA@lists.varnish-cache.org> commit f78dcb8782bf615bfd5112ac0a255ca4551f6843 Author: Dridi Boukelmoune Date: Tue Feb 4 18:11:36 2020 +0100 Various typos diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c index fad9b236b..df95f3322 100644 --- a/bin/varnishd/cache/cache_http.c +++ b/bin/varnishd/cache/cache_http.c @@ -48,7 +48,7 @@ const char H__Proto[] = "\007:proto:"; const char H__Reason[] = "\010:reason:"; /*-------------------------------------------------------------------- - * These two functions are in an incestous relationship with the + * These two functions are in an incestuous relationship with the * order of macros in include/tbl/vsl_tags_http.h * * The http->logtag is the SLT_*Method enum, and we add to that, to @@ -109,8 +109,9 @@ http_fail(const struct http *hp) WS_MarkOverflow(hp->ws); } -/*--------------------------------------------------------------------*/ -/* List of canonical HTTP response code names from RFC2616 */ +/*-------------------------------------------------------------------- + * List of canonical HTTP response code names from RFC2616 + */ static struct http_msg { unsigned nbr; @@ -345,7 +346,7 @@ http_CountHdr(const struct http *hp, const char *hdr) } /*-------------------------------------------------------------------- - * This function collapses multiple headerlines of the same name. + * This function collapses multiple header lines of the same name. * The lines are joined with a comma, according to [rfc2616, 4.2bot, p32] */ @@ -472,7 +473,7 @@ http_GetHdr(const struct http *hp, const char *hdr, const char **ptr) /*----------------------------------------------------------------------------- * Split source string at any of the separators, return pointer to first - * and last+1 char of substrings, with whitespace trimed at both ends. + * and last+1 char of substrings, with whitespace trimmed at both ends. * If sep being an empty string is shorthand for VCT::SP * If stop is NULL, src is NUL terminated. */ @@ -606,7 +607,7 @@ http_GetHdrToken(const struct http *hp, const char *hdr, } /*-------------------------------------------------------------------- - * Find a given headerfields Q value. + * Find a given header field's quality value (qvalue). */ double @@ -655,7 +656,7 @@ http_GetHdrQ(const struct http *hp, const char *hdr, const char *field) } /*-------------------------------------------------------------------- - * Find a given headerfields value. + * Find a given header field's value. */ int From dridi.boukelmoune at gmail.com Tue Feb 4 17:49:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 4 Feb 2020 17:49:06 +0000 (UTC) Subject: [master] 0aec0c9e3 Document more variables default values in vcl(7) Message-ID: <20200204174906.DA88597AD@lists.varnish-cache.org> commit 0aec0c9e33178f7709936f4243e7592f35c6f114 Author: Dridi Boukelmoune Date: Tue Feb 4 18:20:11 2020 +0100 Document more variables default values in vcl(7) And harmonize the reStructuredText format of default values entries. diff --git a/doc/sphinx/reference/vcl_var.rst b/doc/sphinx/reference/vcl_var.rst index dd9f34975..62c768fdd 100644 --- a/doc/sphinx/reference/vcl_var.rst +++ b/doc/sphinx/reference/vcl_var.rst @@ -367,7 +367,7 @@ req.hash_ignore_busy Writable from: client - Default: `false` + Default: ``false``. Ignore any busy object during cache lookup. @@ -383,7 +383,7 @@ req.hash_always_miss Writable from: client - Default: `false` + Default: ``false``. Force a cache miss for this request, even if perfectly good matching objects are in the cache. @@ -602,7 +602,7 @@ bereq.connect_timeout Default: ``.connect_timeout`` attribute from the :ref:`backend_definition`, which defaults to the - ``connect_timeout`` parameter, see :ref:`varnishd(1)` + ``connect_timeout`` parameter, see :ref:`varnishd(1)`. The time in seconds to wait for a backend connection to be established. @@ -618,7 +618,7 @@ bereq.first_byte_timeout Default: ``.first_byte_timeout`` attribute from the :ref:`backend_definition`, which defaults to the - ``first_byte_timeout`` parameter, see :ref:`varnishd(1)` + ``first_byte_timeout`` parameter, see :ref:`varnishd(1)`. The time in seconds to wait getting the first byte back from the backend. Not available in pipe mode. @@ -634,7 +634,7 @@ bereq.between_bytes_timeout Default: ``.between_bytes_timeout`` attribute from the :ref:`backend_definition`, which defaults to the - ``between_bytes_timeout`` parameter, see :ref:`varnishd(1)` + ``between_bytes_timeout`` parameter, see :ref:`varnishd(1)`. The time in seconds to wait between each received byte from the backend. Not available in pipe mode. @@ -740,7 +740,7 @@ beresp.do_esi Writable from: vcl_backend_response, vcl_backend_error - Default: false + Default: ``false``. Set it to true to parse the object for ESI directives. Will only be honored if req.esi is true. @@ -754,7 +754,7 @@ beresp.do_stream Writable from: vcl_backend_response, vcl_backend_error - Default: true + Default: ``true``. Deliver the object to the client while fetching the whole object into varnish. @@ -774,7 +774,7 @@ beresp.do_gzip Writable from: vcl_backend_response, vcl_backend_error - Default: false + Default: ``false``. Set to `true` to gzip the object while storing it. @@ -789,7 +789,7 @@ beresp.do_gunzip Writable from: vcl_backend_response, vcl_backend_error - Default: false + Default: ``false``. Set to `true` to gunzip the object while storing it in the cache. @@ -834,6 +834,10 @@ beresp.ttl Writable from: vcl_backend_response, vcl_backend_error + Default: Cache-Control ``s-maxage`` or ``max-age`` directives, + or a value computed from the Expires header's deadline, or the + ``default_ttl`` parameter. + The object's remaining time to live, in seconds. @@ -843,6 +847,8 @@ beresp.age Readable from: vcl_backend_response, vcl_backend_error + Default: Age header, or zero. + The age of the object. @@ -854,6 +860,9 @@ beresp.grace Writable from: vcl_backend_response, vcl_backend_error + Default: Cache-Control ``stale-while-revalidate`` directive, + or ``default_grace`` parameter. + Set to a period to enable grace. @@ -865,6 +874,8 @@ beresp.keep Writable from: vcl_backend_response, vcl_backend_error + Default: ``default_keep`` parameter. + Set to a period to enable conditional backend requests. The keep time is cache lifetime in addition to the ttl. From dridi.boukelmoune at gmail.com Tue Feb 4 17:49:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 4 Feb 2020 17:49:06 +0000 (UTC) Subject: [master] 0cf29671a Fix man/vcl.7 build dependencies Message-ID: <20200204174906.F2E4F97B0@lists.varnish-cache.org> commit 0cf29671a4f2be0efd4ebdbb605781e0f258f495 Author: Dridi Boukelmoune Date: Tue Feb 4 18:36:55 2020 +0100 Fix man/vcl.7 build dependencies Other man pages seem to have bogus dependencies. diff --git a/man/Makefile.am b/man/Makefile.am index 0226e7299..20b17dce9 100644 --- a/man/Makefile.am +++ b/man/Makefile.am @@ -35,8 +35,9 @@ varnish-cli.7: $(top_builddir)/doc/sphinx/reference/varnish-cli.rst varnish-counters.7: $(top_builddir)/doc/sphinx/reference/varnish-counters.rst $(BUILD_MAN) $(top_srcdir)/doc/sphinx/reference/varnish-counters.rst $@ -vcl.7: $(top_builddir)/doc/sphinx/reference/vcl.rst \ - $(top_builddir)/bin/varnishd/builtin.vcl +vcl.7: $(top_srcdir)/doc/sphinx/reference/vcl.rst \ + $(top_srcdir)/doc/sphinx/reference/vcl_var.rst \ + $(top_srcdir)/bin/varnishd/builtin.vcl $(BUILD_MAN) $(top_srcdir)/doc/sphinx/reference/vcl.rst $@ vsl.7: $(top_builddir)/doc/sphinx/reference/vsl.rst \ From nils.goroll at uplex.de Wed Feb 5 08:39:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 5 Feb 2020 08:39:06 +0000 (UTC) Subject: [master] 91822c85e assert umem cache sanity Message-ID: <20200205083906.9DB4FA46A3@lists.varnish-cache.org> commit 91822c85e1717033df16fd2302ecd75fcb8b7694 Author: Nils Goroll Date: Wed Feb 5 09:37:54 2020 +0100 assert umem cache sanity Ref #3210 diff --git a/bin/varnishd/storage/storage_umem.c b/bin/varnishd/storage/storage_umem.c index db45305f6..bc3458e9c 100644 --- a/bin/varnishd/storage/storage_umem.c +++ b/bin/varnishd/storage/storage_umem.c @@ -201,6 +201,11 @@ smu_alloc(const struct stevedore *st, size_t size) } smu->sz = size; smu->s.space = size; +#ifndef BUG3210 + assert(smu->sc == smu_sc); + assert(smu->s.priv == smu); + AZ(smu->s.len); +#endif return (&smu->s); } From nils.goroll at uplex.de Fri Feb 7 12:25:07 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 7 Feb 2020 12:25:07 +0000 (UTC) Subject: [master] fff0dc7af make test agnostic to the order of events in the VSL Message-ID: <20200207122507.F065DA2D13@lists.varnish-cache.org> commit fff0dc7af810be220cbee9849bd8f5d2bc4245bf Author: Nils Goroll Date: Fri Feb 7 13:22:00 2020 +0100 make test agnostic to the order of events in the VSL diff --git a/bin/varnishtest/tests/b00049.vtc b/bin/varnishtest/tests/b00049.vtc index af3cac740..4ad6d4988 100644 --- a/bin/varnishtest/tests/b00049.vtc +++ b/bin/varnishtest/tests/b00049.vtc @@ -9,6 +9,9 @@ varnish v1 -vcl+backend { } -start logexpect l1 -v v1 -g raw { expect * 1004 BogoHeader "Illegal char 0x20 in header name" +} -start + +logexpect l2 -v v1 -g raw { expect * 1006 BogoHeader "Illegal char 0x2f in header name" } -start @@ -46,3 +49,4 @@ client c1 { } -run logexpect l1 -wait +logexpect l2 -wait From nils.goroll at uplex.de Fri Feb 7 12:40:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 7 Feb 2020 12:40:06 +0000 (UTC) Subject: [master] 41eb79a69 stabilize timing-sensitive test Message-ID: <20200207124007.00CD9A34EF@lists.varnish-cache.org> commit 41eb79a69f131bb2ab03033130011904dc8e96db Author: Nils Goroll Date: Fri Feb 7 13:33:41 2020 +0100 stabilize timing-sensitive test There are two one-second delays on the server side, but varnishd might read the response a bit longer after the first delay such that the second interval is shortened from the varnish perspective. diff --git a/bin/varnishtest/tests/s00004.vtc b/bin/varnishtest/tests/s00004.vtc index 675a0c8d4..d86903f7c 100644 --- a/bin/varnishtest/tests/s00004.vtc +++ b/bin/varnishtest/tests/s00004.vtc @@ -37,7 +37,7 @@ logexpect l1 -v v1 -g request { expect * = Timestamp {Start: \S+ 0\.000000 0\.000000} expect * = Timestamp {Bereq: \S+ 0\.\d+ 0\.\d+} expect * = Timestamp {Beresp: \S+ 1\.\d+ [01]\.\d+} - expect * = Timestamp {BerespBody: \S+ 2\.\d+ 1\.\d+} + expect * = Timestamp {BerespBody: \S+ 2\.\d+ (1\.\d+|0\.9)} expect * = End expect 0 1003 Begin {req 1001 restart} expect * = Timestamp {Start: \S+ 2\.\d+ 0\.\d+} From nils.goroll at uplex.de Fri Feb 7 12:56:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 7 Feb 2020 12:56:06 +0000 (UTC) Subject: [master] 27215f470 use backend None instead of bad ip Message-ID: <20200207125606.48B4FA3C91@lists.varnish-cache.org> commit 27215f4702cfcecbd534c36841f1d9af5b55387e Author: Nils Goroll Date: Fri Feb 7 13:54:22 2020 +0100 use backend None instead of bad ip Does this stabilize a vtest false negative which I do not understand? diff --git a/bin/varnishtest/tests/r02946.vtc b/bin/varnishtest/tests/r02946.vtc index 4801178ad..ecda00483 100644 --- a/bin/varnishtest/tests/r02946.vtc +++ b/bin/varnishtest/tests/r02946.vtc @@ -1,7 +1,7 @@ varnishtest "#2946 - objcore leak for backend_synth" varnish v1 -vcl { - backend bad { .host = "${bad_backend}"; } + backend bad None; sub vcl_backend_error { if (bereq.http.abandon) { From phk at FreeBSD.org Mon Feb 10 09:15:08 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 10 Feb 2020 09:15:08 +0000 (UTC) Subject: [master] 4113c8037 Add functions to treat a WS as a VSB. Message-ID: <20200210091508.5472AB2275@lists.varnish-cache.org> commit 4113c8037aab183059c0e6bce32de7530823ba79 Author: Poul-Henning Kamp Date: Mon Feb 10 09:13:32 2020 +0000 Add functions to treat a WS as a VSB. Usage pattern: struct vsb vsb[1]; char *p; WS_VSB_new(vsb, ctx->ws); VSB_printf(vsb, "blablabla"); p = WS_VSB_finish(vsb); if (p == NULL) return (FAILURE); diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 360855e44..e83096935 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -795,6 +795,9 @@ void *WS_Printf(struct ws *ws, const char *fmt, ...) v_printflike_(2, 3); int WS_Inside(const struct ws *, const void *, const void *); void WS_Assert_Allocated(const struct ws *ws, const void *ptr, ssize_t len); +void WS_VSB_new(struct vsb *, struct ws *); +char *WS_VSB_finish(struct vsb *, struct ws *); + static inline char* WS_Front(const struct ws *ws) { diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c index f224b9ffd..7a4b07ae5 100644 --- a/bin/varnishd/cache/cache_vrt.c +++ b/bin/varnishd/cache/cache_vrt.c @@ -440,63 +440,41 @@ VRT_Strands(char *d, size_t dl, VCL_STRANDS s) VCL_STRING VRT_StrandsWS(struct ws *ws, const char *h, VCL_STRANDS s) { - char *b; - const char *q = NULL, *e; - VCL_STRING r; - unsigned u, x; + const char *q = NULL; + struct vsb vsb[1]; int i; + WS_Assert(ws); AN(s); - u = WS_ReserveAll(ws); - for (i = 0; i < s->n; i++) + for (i = 0; i < s->n; i++) { if (s->p[i] != NULL && *s->p[i] != '\0') { q = s->p[i]; break; } - - if (h != NULL && q == NULL && WS_Inside(ws, h, NULL)) { - WS_Release(ws, 0); - return (h); } - if (h == NULL) { - if (q == NULL) { - WS_Release(ws, 0); + if (q == NULL) { + if (h == NULL) return (""); - } - if (WS_Inside(ws, q, NULL)) { - for (i++; i < s->n; i++) - if (s->p[i] != NULL && *s->p[i] != '\0') - break; - if (i == s->n) { - WS_Release(ws, 0); - return (q); - } - } + if (WS_Inside(ws, h, NULL)) + return (h); + } else if (h == NULL && WS_Inside(ws, q, NULL)) { + for (i++; i < s->n; i++) + if (s->p[i] != NULL && *s->p[i] != '\0') + break; + if (i == s->n) + return (q); } - b = WS_Front(ws); - e = b + u; - - if (h != NULL) { - x = strlen(h); - if (b + x < e) - memcpy(b, h, x); - b += x; - if (b < e) - *b = ' '; - b++; - } - r = VRT_Strands(b, e > b ? e - b : 0, s); - if (r == NULL || r == e) { - WS_MarkOverflow(ws); - WS_Release(ws, 0); - return (NULL); + WS_VSB_new(vsb, ws); + if (h != NULL) + VSB_cat(vsb, h); + for (i = 0; i < s->n; i++) { + if (s->p[i] != NULL && *s->p[i] != '\0') + VSB_cat(vsb, s->p[i]); } - b = WS_Front(ws); - WS_Release(ws, r - b); - return (b); + return (WS_VSB_finish(vsb, ws)); } /*-------------------------------------------------------------------- diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c index 2ea57116a..aad4db73a 100644 --- a/bin/varnishd/cache/cache_ws.c +++ b/bin/varnishd/cache/cache_ws.c @@ -342,3 +342,53 @@ WS_Overflowed(const struct ws *ws) return (0); return (1); } + +/*--------------------------------------------------------------------- + * Build a VSB on a workspace. + * Usage pattern: + * + * struct vsb vsb[1]; + * char *p; + * + * WS_VSB_new(vsb, ctx->ws); + * VSB_printf(vsb, "blablabla"); + * p = WS_VSB_finish(vsb); + * if (p == NULL) + * return (FAILURE); + */ + +void +WS_VSB_new(struct vsb *vsb, struct ws *ws) +{ + unsigned u; + + WS_Assert(ws); + u = WS_ReserveAll(ws); + if (WS_Overflowed(ws) || u < 2) { + WS_MarkOverflow(ws); + /* Create a malloced-buffer VSB, and fail it up front */ + AN(VSB_new(vsb, NULL, 2, 0)); + VSB_cat(vsb, "XXX"); + } else { + AN(VSB_new(vsb, WS_Front(ws), u, 0)); + } +} + +char * +WS_VSB_finish(struct vsb *vsb, struct ws *ws) +{ + char *p; + + WS_Assert(ws); + if (!VSB_finish(vsb)) { + p = VSB_data(vsb); + if (p == WS_Front(ws)) { + WS_Release(ws, VSB_len(vsb) + 1); + VSB_delete(vsb); + return (p); + } + } + VSB_delete(vsb); + WS_Release(ws, 0); + return (NULL); +} From phk at FreeBSD.org Mon Feb 10 09:15:08 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 10 Feb 2020 09:15:08 +0000 (UTC) Subject: [master] 4454f75d2 White space OCD Message-ID: <20200210091508.7267CB2278@lists.varnish-cache.org> commit 4454f75d200c03c12841377cec03c3293d892780 Author: Poul-Henning Kamp Date: Mon Feb 10 09:14:24 2020 +0000 White space OCD diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c index 71c82d2a3..ba8797c7c 100644 --- a/bin/varnishadm/varnishadm.c +++ b/bin/varnishadm/varnishadm.c @@ -260,7 +260,7 @@ pass_answer(int fd) printf("%s\n", answer); free(answer); } - if (status == CLIS_TRUNCATED) + if (status == CLIS_TRUNCATED) printf("[response was truncated]\n"); (void)fflush(stdout); } diff --git a/bin/varnishtest/tests/m00003.vtc b/bin/varnishtest/tests/m00003.vtc index 3dd8a12d6..d6e8b571e 100644 --- a/bin/varnishtest/tests/m00003.vtc +++ b/bin/varnishtest/tests/m00003.vtc @@ -10,7 +10,7 @@ server s1 { shell { echo "vcl 4.1; import std; backend dummy None;" >${tmpdir}/test.vcl varnishd -pvmod_path=${topbuild}/lib/libvmod_std/.libs \ - -C -f ${tmpdir}/test.vcl 2>/dev/null + -C -f ${tmpdir}/test.vcl 2>/dev/null } varnish v1 -arg "-pvmod_path=${topbuild}/lib/libvmod_std/.libs/" -vcl+backend { From phk at FreeBSD.org Mon Feb 10 09:45:07 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 10 Feb 2020 09:45:07 +0000 (UTC) Subject: [master] c2a512359 Make it possible to get the length from WS_VSB_finish() (ie: H2 headers). Message-ID: <20200210094507.12349B2FA1@lists.varnish-cache.org> commit c2a5123598061154e89875148dbe96edfb428d83 Author: Poul-Henning Kamp Date: Mon Feb 10 09:33:05 2020 +0000 Make it possible to get the length from WS_VSB_finish() (ie: H2 headers). diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index e83096935..b1f8bdc40 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -796,7 +796,7 @@ int WS_Inside(const struct ws *, const void *, const void *); void WS_Assert_Allocated(const struct ws *ws, const void *ptr, ssize_t len); void WS_VSB_new(struct vsb *, struct ws *); -char *WS_VSB_finish(struct vsb *, struct ws *); +char *WS_VSB_finish(struct vsb *, struct ws *, size_t *); static inline char* WS_Front(const struct ws *ws) diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c index 7a4b07ae5..fe26401eb 100644 --- a/bin/varnishd/cache/cache_vrt.c +++ b/bin/varnishd/cache/cache_vrt.c @@ -474,7 +474,7 @@ VRT_StrandsWS(struct ws *ws, const char *h, VCL_STRANDS s) if (s->p[i] != NULL && *s->p[i] != '\0') VSB_cat(vsb, s->p[i]); } - return (WS_VSB_finish(vsb, ws)); + return (WS_VSB_finish(vsb, ws, NULL)); } /*-------------------------------------------------------------------- diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c index aad4db73a..4d2c382db 100644 --- a/bin/varnishd/cache/cache_ws.c +++ b/bin/varnishd/cache/cache_ws.c @@ -352,7 +352,7 @@ WS_Overflowed(const struct ws *ws) * * WS_VSB_new(vsb, ctx->ws); * VSB_printf(vsb, "blablabla"); - * p = WS_VSB_finish(vsb); + * p = WS_VSB_finish(vsb, NULL); * if (p == NULL) * return (FAILURE); */ @@ -375,7 +375,7 @@ WS_VSB_new(struct vsb *vsb, struct ws *ws) } char * -WS_VSB_finish(struct vsb *vsb, struct ws *ws) +WS_VSB_finish(struct vsb *vsb, struct ws *ws, size_t *szp) { char *p; @@ -384,11 +384,15 @@ WS_VSB_finish(struct vsb *vsb, struct ws *ws) p = VSB_data(vsb); if (p == WS_Front(ws)) { WS_Release(ws, VSB_len(vsb) + 1); + if (szp != NULL) + *szp = VSB_len(vsb); VSB_delete(vsb); return (p); } } VSB_delete(vsb); WS_Release(ws, 0); + if (szp) + *szp = 0; return (NULL); } From phk at FreeBSD.org Mon Feb 10 09:45:07 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 10 Feb 2020 09:45:07 +0000 (UTC) Subject: [master] 7c4dcd662 Simplify h2_deliver() with a WS_VSB Message-ID: <20200210094507.30CE0B2FA4@lists.varnish-cache.org> commit 7c4dcd662736336f4e3c9248a145c46c085f3838 Author: Poul-Henning Kamp Date: Mon Feb 10 09:43:27 2020 +0000 Simplify h2_deliver() with a WS_VSB Note XXX relative to #3202 diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c index 4d2c382db..4553c6abf 100644 --- a/bin/varnishd/cache/cache_ws.c +++ b/bin/varnishd/cache/cache_ws.c @@ -365,7 +365,6 @@ WS_VSB_new(struct vsb *vsb, struct ws *ws) WS_Assert(ws); u = WS_ReserveAll(ws); if (WS_Overflowed(ws) || u < 2) { - WS_MarkOverflow(ws); /* Create a malloced-buffer VSB, and fail it up front */ AN(VSB_new(vsb, NULL, 2, 0)); VSB_cat(vsb, "XXX"); @@ -390,6 +389,7 @@ WS_VSB_finish(struct vsb *vsb, struct ws *ws, size_t *szp) return (p); } } + WS_MarkOverflow(ws); VSB_delete(vsb); WS_Release(ws, 0); if (szp) diff --git a/bin/varnishd/http2/cache_http2_deliver.c b/bin/varnishd/http2/cache_http2_deliver.c index c279ec54f..a1cdbf9c1 100644 --- a/bin/varnishd/http2/cache_http2_deliver.c +++ b/bin/varnishd/http2/cache_http2_deliver.c @@ -221,7 +221,7 @@ static const uint8_t h2_500_resp[] = { 0x1f, 0x27, 0x07, 'V', 'a', 'r', 'n', 'i', 's', 'h', }; -static int +static void h2_build_headers(struct vsb *resp, struct req *req) { unsigned u, l; @@ -232,14 +232,6 @@ h2_build_headers(struct vsb *resp, struct req *req) uint8_t buf[6]; ssize_t sz, sz1; - l = WS_ReserveAll(req->ws); - if (l < 10) { - WS_Release(req->ws, 0); - return (-1); - } - - AN(VSB_new(resp, req->ws->f, l, VSB_FIXEDLEN)); - l = h2_status(buf, req->resp->status); VSB_bcat(resp, buf, l); @@ -281,17 +273,17 @@ h2_build_headers(struct vsb *resp, struct req *req) h2_enc_len(resp, 7, sz, 0); VSB_bcat(resp, r, sz); } - return (VSB_finish(resp)); } void v_matchproto_(vtr_deliver_f) h2_deliver(struct req *req, struct boc *boc, int sendbody) { - ssize_t sz; + size_t sz; const char *r; struct sess *sp; struct h2_req *r2; - struct vsb resp; + struct vsb resp[1]; + uintptr_t ss; CHECK_OBJ_NOTNULL(req, REQ_MAGIC); CHECK_OBJ_ORNULL(boc, BOC_MAGIC); @@ -302,9 +294,13 @@ h2_deliver(struct req *req, struct boc *boc, int sendbody) VSLb(req->vsl, SLT_RespProtocol, "HTTP/2.0"); - if (h2_build_headers(&resp, req)) { - // We ran out of workspace, return minimal 500 - WS_MarkOverflow(req->ws); + ss = WS_Snapshot(req->ws); + + WS_VSB_new(resp, req->ws); + h2_build_headers(resp, req); + r = WS_VSB_finish(resp, req->ws, &sz); + + if (r == NULL) { VSLb(req->vsl, SLT_Error, "workspace_client overflow"); VSLb(req->vsl, SLT_RespStatus, "500"); VSLb(req->vsl, SLT_RespReason, "Internal Server Error"); @@ -313,9 +309,6 @@ h2_deliver(struct req *req, struct boc *boc, int sendbody) r = (const char*)h2_500_resp; sz = sizeof h2_500_resp; sendbody = 0; - } else { - sz = VSB_len(&resp); - r = req->ws->f; } AZ(req->wrk->v1l); @@ -328,7 +321,8 @@ h2_deliver(struct req *req, struct boc *boc, int sendbody) sz, r, &req->acct.resp_hdrbytes); H2_Send_Rel(r2->h2sess, r2); - WS_Release(req->ws, 0); + if (!WS_Overflowed(req->ws)) // XXX: remove if when #3202 is fixed + WS_Reset(req->ws, ss); /* XXX someone into H2 please add appropriate error handling */ if (sendbody) { From phk at FreeBSD.org Mon Feb 10 11:24:06 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 10 Feb 2020 11:24:06 +0000 (UTC) Subject: [master] 6e25bf69b Two more uses for WS_VSB() Message-ID: <20200210112406.CDB77C0148@lists.varnish-cache.org> commit 6e25bf69b9d236d1479371beb678f486dcf5fa71 Author: Poul-Henning Kamp Date: Mon Feb 10 11:23:36 2020 +0000 Two more uses for WS_VSB() diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c index df95f3322..2297121e0 100644 --- a/bin/varnishd/cache/cache_http.c +++ b/bin/varnishd/cache/cache_http.c @@ -1228,28 +1228,32 @@ http_ForceHeader(struct http *to, const char *hdr, const char *val) void http_PrintfHeader(struct http *to, const char *fmt, ...) { - va_list ap; - unsigned l, n; + va_list ap, ap2; + struct vsb vsb[1]; + size_t sz; + char *p; CHECK_OBJ_NOTNULL(to, HTTP_MAGIC); - l = WS_ReserveAll(to->ws); + va_start(ap, fmt); - n = vsnprintf(to->ws->f, l, fmt, ap); - va_end(ap); - if (n + 1 >= l || to->nhd >= to->shd) { + va_copy(ap2, ap); + + WS_VSB_new(vsb, to->ws); + VSB_vprintf(vsb, fmt, ap); + p = WS_VSB_finish(vsb, to->ws, &sz); + + if (p == NULL || to->nhd >= to->shd) { http_fail(to); - va_start(ap, fmt); - VSLbv(to->vsl, SLT_LostHeader, fmt, ap); - va_end(ap); - WS_Release(to->ws, 0); - return; + VSLbv(to->vsl, SLT_LostHeader, fmt, ap2); + } else { + to->hd[to->nhd].b = p; + to->hd[to->nhd].e = p + sz; + to->hdf[to->nhd] = 0; + http_VSLH(to, to->nhd); + to->nhd++; } - to->hd[to->nhd].b = to->ws->f; - to->hd[to->nhd].e = to->ws->f + n; - to->hdf[to->nhd] = 0; - WS_Release(to->ws, n + 1); - http_VSLH(to, to->nhd); - to->nhd++; + va_end(ap); + va_end(ap2); } void diff --git a/bin/varnishd/cache/cache_vrt_filter.c b/bin/varnishd/cache/cache_vrt_filter.c index b6444d886..2bbdcf862 100644 --- a/bin/varnishd/cache/cache_vrt_filter.c +++ b/bin/varnishd/cache/cache_vrt_filter.c @@ -248,30 +248,17 @@ typedef void filter_list_t(void *, struct vsb *vsb); static const char * filter_on_ws(struct ws *ws, filter_list_t *func, void *arg) { - unsigned u; struct vsb vsb[1]; + const char *p; AN(func); AN(arg); - u = WS_ReserveAll(ws); - if (u == 0) { - WS_Release(ws, 0); - WS_MarkOverflow(ws); - return (NULL); - } - AN(VSB_new(vsb, ws->f, u, VSB_FIXEDLEN)); + WS_VSB_new(vsb, ws); func(arg, vsb); - if (VSB_finish(vsb)) { - WS_Release(ws, 0); - WS_MarkOverflow(ws); - return (NULL); - } - if (VSB_len(vsb)) { - WS_Release(ws, VSB_len(vsb) + 1); - return (VSB_data(vsb) + 1); - } - WS_Release(ws, 0); - return (""); + p = WS_VSB_finish(vsb, ws, NULL); + if (p == NULL) + p = ""; + return (p); } /*-------------------------------------------------------------------- diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c index 4553c6abf..687d15d2d 100644 --- a/bin/varnishd/cache/cache_ws.c +++ b/bin/varnishd/cache/cache_ws.c @@ -361,16 +361,14 @@ void WS_VSB_new(struct vsb *vsb, struct ws *ws) { unsigned u; + static char bogus[2]; // Smallest possible vsb WS_Assert(ws); u = WS_ReserveAll(ws); - if (WS_Overflowed(ws) || u < 2) { - /* Create a malloced-buffer VSB, and fail it up front */ - AN(VSB_new(vsb, NULL, 2, 0)); - VSB_cat(vsb, "XXX"); - } else { + if (WS_Overflowed(ws) || u < 2) + AN(VSB_new(vsb, bogus, sizeof bogus, 0)); + else AN(VSB_new(vsb, WS_Front(ws), u, 0)); - } } char * From phk at FreeBSD.org Mon Feb 10 12:29:07 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 10 Feb 2020 12:29:07 +0000 (UTC) Subject: [master] 0c3ec1692 Simplify std.tolower() and std.toupper() with WS_VSB Message-ID: <20200210122907.34A2E6738@lists.varnish-cache.org> commit 0c3ec169282427e629f211dc17c4afd1df2bb9da Author: Poul-Henning Kamp Date: Mon Feb 10 11:40:22 2020 +0000 Simplify std.tolower() and std.toupper() with WS_VSB diff --git a/lib/libvmod_std/vmod_std.c b/lib/libvmod_std/vmod_std.c index 51a9b7e16..1fe02fdcf 100644 --- a/lib/libvmod_std/vmod_std.c +++ b/lib/libvmod_std/vmod_std.c @@ -46,6 +46,7 @@ #include "vrnd.h" #include "vtcp.h" #include "vsa.h" +#include "vsb.h" #include "vtim.h" #include "vcl.h" @@ -66,56 +67,34 @@ vmod_set_ip_tos(VRT_CTX, VCL_INT tos) IPPROTO_IP, IP_TOS, &itos, sizeof(itos))); } -static const char * -vmod_updown(VRT_CTX, int up, VCL_STRANDS s) -{ - unsigned u; - char *b, *e; - const char *p; - int i; - - CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - u = WS_ReserveAll(ctx->ws); - e = b = ctx->ws->f; - e += u; - for (i = 0; i < s->n && b < e; i++) { - p = s->p[i]; - while (p != NULL && *p != '\0' && b < e) { - if (up) - *b++ = (char)toupper(*p++); - else - *b++ = (char)tolower(*p++); - } - } - if (b < e) - *b = '\0'; - b++; - if (b > e) { - WS_MarkOverflow(ctx->ws); - WS_Release(ctx->ws, 0); - return (NULL); - } else { - e = b; - b = ctx->ws->f; - WS_Release(ctx->ws, e - b); - return (b); - } -} - VCL_STRING v_matchproto_(td_std_toupper) vmod_toupper(VRT_CTX, VCL_STRANDS s) { + const char *p; + struct vsb vsb[1]; + int i; CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - return (vmod_updown(ctx, 1, s)); + WS_VSB_new(vsb, ctx->ws); + for (i = 0; i < s->n; i++) + for (p = s->p[i]; p != NULL && *p != '\0'; p++) + VSB_putc(vsb, (char)toupper(*p)); + return (WS_VSB_finish(vsb, ctx->ws, NULL)); } VCL_STRING v_matchproto_(td_std_tolower) vmod_tolower(VRT_CTX, VCL_STRANDS s) { + const char *p; + struct vsb vsb[1]; + int i; CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - return (vmod_updown(ctx, 0, s)); + WS_VSB_new(vsb, ctx->ws); + for (i = 0; i < s->n; i++) + for (p = s->p[i]; p != NULL && *p != '\0'; p++) + VSB_putc(vsb, (char)tolower(*p)); + return (WS_VSB_finish(vsb, ctx->ws, NULL)); } VCL_REAL v_matchproto_(td_std_random) From phk at FreeBSD.org Mon Feb 10 12:29:07 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 10 Feb 2020 12:29:07 +0000 (UTC) Subject: [master] c22c39a8a Get rid of the hardcoded lists of programs and vmods, instead read the actual directories and filter on prefix strings. Message-ID: <20200210122907.4CB64673C@lists.varnish-cache.org> commit c22c39a8a9ec69665ad8d77ced735951ecae2b47 Author: Poul-Henning Kamp Date: Mon Feb 10 12:27:42 2020 +0000 Get rid of the hardcoded lists of programs and vmods, instead read the actual directories and filter on prefix strings. diff --git a/bin/varnishtest/programs.h b/bin/varnishtest/programs.h deleted file mode 100644 index 05c4f136d..000000000 --- a/bin/varnishtest/programs.h +++ /dev/null @@ -1,41 +0,0 @@ -/*- - * Copyright (c) 2013 Varnish Software AS - * All rights reserved. - * - * Author: Poul-Henning Kamp - * - * SPDX-License-Identifier: BSD-2-Clause - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * 1. Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in the - * documentation and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND - * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE - * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL - * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS - * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) - * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT - * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY - * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF - * SUCH DAMAGE. - * - */ - -VTC_PROG(haproxy) -VTC_PROG(varnishadm) -VTC_PROG(varnishd) -VTC_PROG(varnishhist) -VTC_PROG(varnishlog) -VTC_PROG(varnishncsa) -VTC_PROG(varnishstat) -VTC_PROG(varnishtest) -VTC_PROG(varnishtop) -#undef VTC_PROG diff --git a/bin/varnishtest/vmods.h b/bin/varnishtest/vmods.h deleted file mode 100644 index 2af7644d9..000000000 --- a/bin/varnishtest/vmods.h +++ /dev/null @@ -1,39 +0,0 @@ -/*- - * Copyright (c) 2013 Varnish Software AS - * All rights reserved. - * - * Author: Poul-Henning Kamp - * - * SPDX-License-Identifier: BSD-2-Clause - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * 1. Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in the - * documentation and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND - * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE - * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL - * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS - * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) - * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT - * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY - * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF - * SUCH DAMAGE. - * - */ - -VTC_VMOD(std) -VTC_VMOD(debug) -VTC_VMOD(directors) -VTC_VMOD(purge) -VTC_VMOD(vtc) -VTC_VMOD(blob) -VTC_VMOD(unix) -VTC_VMOD(proxy) diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c index b0cc128b2..6b7ebf0a6 100644 --- a/bin/varnishtest/vtc_main.c +++ b/bin/varnishtest/vtc_main.c @@ -35,12 +35,14 @@ #include #include +#include #include #include #include #include #include #include +#include #include "vtc.h" @@ -458,14 +460,44 @@ start_test(void) * Find the abs path to top of source dir from Makefile, if that * fails, fall back on "../../" * - * Set path to all programs build directories + * Set PATH to all programs build directories + * Set vmod_path to all vmods build directories * */ +static void +build_path(const char *topbuilddir, const char *subdir, + const char *pfx, const char *sfx, struct vsb *vsb) +{ + char buf[PATH_MAX]; + DIR *dir; + struct dirent *de; + struct stat st; + const char *sep = ""; + + bprintf(buf, "%s/%s/", topbuilddir, subdir); + dir = opendir(buf); + XXXAN(dir); + while (1) { + de = readdir(dir); + if (de == NULL) + break; + if (strncmp(de->d_name, pfx, strlen(pfx))) + continue; + bprintf(buf, "%s/%s/%s", topbuilddir, subdir, de->d_name); + if (!stat(buf, &st) && S_ISDIR(st.st_mode)) { + VSB_cat(vsb, sep); + VSB_cat(vsb, buf); + VSB_cat(vsb, sfx); + sep = ":"; + } + } + AZ(closedir(dir)); +} + static void i_mode(void) { - const char *sep; struct vsb *vsb; char *p, *q; char *topbuild; @@ -511,39 +543,27 @@ i_mode(void) } AN(topbuild); extmacro_def("topbuild", "%s", topbuild); + /* * Build $PATH which can find all programs in the build tree */ + VSB_clear(vsb); VSB_cat(vsb, "PATH="); - sep = ""; -#define VTC_PROG(l) \ - do { \ - VSB_printf(vsb, "%s%s/bin/" #l, sep, topbuild); \ - sep = ":"; \ - } while (0); -#include "programs.h" - + build_path(topbuild, "bin", "varnish", "", vsb); VSB_printf(vsb, ":%s", getenv("PATH")); AZ(VSB_finish(vsb)); - AZ(putenv(strdup(VSB_data(vsb)))); /* * Build vmod_path which can find all VMODs in the build tree */ + VSB_clear(vsb); - sep = ""; -#define VTC_VMOD(l) \ - do { \ - VSB_printf(vsb, "%s%s/lib/libvmod_" #l "/.libs", \ - sep, topbuild); \ - sep = ":"; \ - } while (0); -#include "vmods.h" -#undef VTC_VMOD + build_path(topbuild, "lib", "libvmod_", "/.libs", vsb); AZ(VSB_finish(vsb)); vmod_path = strdup(VSB_data(vsb)); AN(vmod_path); + free(topbuild); VSB_destroy(&vsb); From phk at FreeBSD.org Mon Feb 10 12:50:06 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 10 Feb 2020 12:50:06 +0000 (UTC) Subject: [master] 069dbdfac Forgot to remove programs.h and vmods.h from autocrappery Message-ID: <20200210125006.459097390@lists.varnish-cache.org> commit 069dbdfac6393f575f34ed6c58bd304693dc19f8 Author: Poul-Henning Kamp Date: Mon Feb 10 12:49:03 2020 +0000 Forgot to remove programs.h and vmods.h from autocrappery diff --git a/bin/varnishtest/Makefile.am b/bin/varnishtest/Makefile.am index dc67a98c0..7324b7473 100644 --- a/bin/varnishtest/Makefile.am +++ b/bin/varnishtest/Makefile.am @@ -25,9 +25,7 @@ bin_PROGRAMS = varnishtest varnishtest_SOURCES = \ hpack.h \ - programs.h \ cmds.h \ - vmods.h \ vtc.h \ teken.c \ teken.h \ From nils.goroll at uplex.de Tue Feb 11 14:08:02 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 11 Feb 2020 14:08:02 +0000 (UTC) Subject: [master] b0af060fb simple fix for fedora/gcc-10.0.1: -Werror=format-overflow, by some reason hit on s390x Message-ID: <20200211140802.9EFE3616ED@lists.varnish-cache.org> commit b0af060fb688b8fc2ff3817ea99430432668b291 Author: Ingvar Hagelund Date: Tue Feb 11 12:56:54 2020 +0100 simple fix for fedora/gcc-10.0.1: -Werror=format-overflow, by some reason hit on s390x diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c index 1ec748cb6..09e49d258 100644 --- a/bin/varnishtest/vtc_varnish.c +++ b/bin/varnishtest/vtc_varnish.c @@ -121,7 +121,7 @@ varnish_ask_cli(const struct varnish *v, const char *cmd, char **repl) i = VCLI_ReadResult(v->cli_fd, &retval, &r, vtc_maxdur); if (i != 0 && !vtc_stop) vtc_fatal(v->vl, "CLI failed (%s) = %d %u %s", - cmd, i, retval, r); + cmd != NULL ? cmd : "NULL", i, retval, r); vtc_log(v->vl, 3, "CLI RX %u", retval); vtc_dump(v->vl, 4, "CLI RX", r, -1); if (repl != NULL) From dridi at varni.sh Wed Feb 12 14:06:17 2020 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 12 Feb 2020 14:06:17 +0000 Subject: [master] b0af060fb simple fix for fedora/gcc-10.0.1: -Werror=format-overflow, by some reason hit on s390x In-Reply-To: <20200211140802.9EFE3616ED@lists.varnish-cache.org> References: <20200211140802.9EFE3616ED@lists.varnish-cache.org> Message-ID: On Tue, Feb 11, 2020 at 2:08 PM Nils Goroll wrote: > > > commit b0af060fb688b8fc2ff3817ea99430432668b291 > Author: Ingvar Hagelund > Date: Tue Feb 11 12:56:54 2020 +0100 > > simple fix for fedora/gcc-10.0.1: -Werror=format-overflow, by some reason hit on s390x > > diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c > index 1ec748cb6..09e49d258 100644 > --- a/bin/varnishtest/vtc_varnish.c > +++ b/bin/varnishtest/vtc_varnish.c > @@ -121,7 +121,7 @@ varnish_ask_cli(const struct varnish *v, const char *cmd, char **repl) > i = VCLI_ReadResult(v->cli_fd, &retval, &r, vtc_maxdur); > if (i != 0 && !vtc_stop) > vtc_fatal(v->vl, "CLI failed (%s) = %d %u %s", > - cmd, i, retval, r); > + cmd != NULL ? cmd : "NULL", i, retval, r); We shouldn't blindly take this change, we should instead ask the GCC or Fedora folks (or their intersection) why this is happening. We are hiding a potential bug under the rug here. GCC 10 turned some flags on by default and this resulted in one legitimate report [1] in Hitch, I'm not convinced by this one. Dridi [1] https://github.com/varnish/hitch/issues/326 Dridi > vtc_log(v->vl, 3, "CLI RX %u", retval); > vtc_dump(v->vl, 4, "CLI RX", r, -1); > if (repl != NULL) > _______________________________________________ > varnish-commit mailing list > varnish-commit at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-commit From nils.goroll at uplex.de Wed Feb 12 14:47:10 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 12 Feb 2020 15:47:10 +0100 Subject: [master] b0af060fb simple fix for fedora/gcc-10.0.1: -Werror=format-overflow, by some reason hit on s390x In-Reply-To: References: <20200211140802.9EFE3616ED@lists.varnish-cache.org> Message-ID: <1c18d52a-408c-18ce-f442-326f250ab193@uplex.de> >> - cmd, i, retval, r); >> + cmd != NULL ? cmd : "NULL", i, retval, r); > > We shouldn't blindly take this change, we should instead ask the GCC > or Fedora folks (or their intersection) why this is happening. We are > hiding a potential bug under the rug here. The gcc error from https://github.com/varnishcache/varnish-cache/issues/3216 is: inlined from 'varnish_launch' at vtc_varnish.c:500:6: vtc_varnish.c:121:3: error: '%s' directive argument is null [-Werror=format-overflow=] So in my mind the warning is legit, but can only happen with inling. Can you explain which potential bug you are suspecting? Nils -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg tel +49 40 28805731 mob +49 170 2723133 fax +49 40 42949753 xmpp://slink at jabber.int.uplex.de/ http://uplex.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From dridi at varni.sh Wed Feb 12 15:55:41 2020 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 12 Feb 2020 15:55:41 +0000 Subject: [master] b0af060fb simple fix for fedora/gcc-10.0.1: -Werror=format-overflow, by some reason hit on s390x In-Reply-To: <1c18d52a-408c-18ce-f442-326f250ab193@uplex.de> References: <20200211140802.9EFE3616ED@lists.varnish-cache.org> <1c18d52a-408c-18ce-f442-326f250ab193@uplex.de> Message-ID: On Wed, Feb 12, 2020 at 2:47 PM Nils Goroll wrote: > > >> - cmd, i, retval, r); > >> + cmd != NULL ? cmd : "NULL", i, retval, r); > > > > We shouldn't blindly take this change, we should instead ask the GCC > > or Fedora folks (or their intersection) why this is happening. We are > > hiding a potential bug under the rug here. > > The gcc error from https://github.com/varnishcache/varnish-cache/issues/3216 is: > > inlined from 'varnish_launch' at vtc_varnish.c:500:6: > vtc_varnish.c:121:3: error: '%s' directive argument is null > [-Werror=format-overflow=] > > So in my mind the warning is legit, but can only happen with inling. > > Can you explain which potential bug you are suspecting? This looks like GCC tripping on its own feet: it wants to inline a variant of this function where cmd is known to be null because (I suspect) a large-enough portion of the function can be eliminated, enough to shrink below the inlining threshold on s390x. So far so good, then it encounters a function with a format __attribute__, but it can't tell whether it might overflow its destination (the answer is no, it can't) and yet it insists on complaining about a potential destination overflow because the argument to a %s format is known to be null. As long as the format __attribute__ can't tell the difference between a printf-like or an sn?printf-like function it shouldn't trigger spurious overflow checks. The printf documentation says that the argument to a %s format is a pointer to an array of char, but doesn't say explicitly that this pointer must not be null. This is at best off topic and otherwise irrelevant since on the target system glibc will print "(null)" for that argument. I can accept an undefined behavior claim better than Wformat-overflow but if we acknowledge that then we need to fix a lot of format arguments to ensure non-null %s directives. To me this should have been escalated to GCC or Fedora instead, that's Ingvar's job as a package maintainer for Fedora: something I would have pointed out if it hadn't been merged so fast. Dridi From nils.goroll at uplex.de Wed Feb 12 16:37:58 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 12 Feb 2020 17:37:58 +0100 Subject: [master] b0af060fb simple fix for fedora/gcc-10.0.1: -Werror=format-overflow, by some reason hit on s390x In-Reply-To: References: <20200211140802.9EFE3616ED@lists.varnish-cache.org> <1c18d52a-408c-18ce-f442-326f250ab193@uplex.de> Message-ID: <8f376fee-6c7b-3a87-b98e-fb934fd6239e@uplex.de> To summarize an IRC conversion with Dridi in case there are any other simple minded people like me who admire pedants ;) : - the publicly available *1) c99 draft states p.279 If no l (length) modifier is present, the argument shall be a pointer to the initial element of an array of character type I read this as excluding NULL - Dridis point is that the gcc warning should not be for -Wformat-overflow, but rather for -Wformat / -Wnonnull Nils 1) http://www.open-std.org/jtc1/sc22/WG14/www/docs/n1256.pdf -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg tel +49 40 28805731 mob +49 170 2723133 fax +49 40 42949753 xmpp://slink at jabber.int.uplex.de/ http://uplex.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From nils.goroll at uplex.de Thu Feb 13 09:15:09 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 13 Feb 2020 09:15:09 +0000 (UTC) Subject: [master] c67af5f5f disambiguate vcl_max Message-ID: <20200213091509.83D47A4855@lists.varnish-cache.org> commit c67af5f5f8bb4cf95df14d7d0eb62f73955bdc7c Author: Nils Goroll Date: Thu Feb 13 10:08:15 2020 +0100 disambiguate vcl_max we used vcl_max both for the max vcl version and the maximum number of vcls to load. diff --git a/bin/varnishd/cache/cache_vcl.c b/bin/varnishd/cache/cache_vcl.c index 8c0686597..2e1f6a69f 100644 --- a/bin/varnishd/cache/cache_vcl.c +++ b/bin/varnishd/cache/cache_vcl.c @@ -430,7 +430,8 @@ VCL_Open(const char *fn, struct vsb *msg) (void)dlclose(dlh); return (NULL); } - if (cnf->syntax < heritage.min_vcl || cnf->syntax > heritage.max_vcl) { + if (cnf->syntax < heritage.min_vcl_version || + cnf->syntax > heritage.max_vcl_version) { VSB_printf(msg, "Compiled VCL version (%.1f) not supported.\n", .1 * cnf->syntax); (void)dlclose(dlh); diff --git a/bin/varnishd/common/heritage.h b/bin/varnishd/common/heritage.h index e3550ada6..d33fb4d3f 100644 --- a/bin/varnishd/common/heritage.h +++ b/bin/varnishd/common/heritage.h @@ -87,8 +87,8 @@ struct heritage { struct vsmw *proc_vsmw; - unsigned min_vcl; - unsigned max_vcl; + unsigned min_vcl_version; + unsigned max_vcl_version; }; extern struct heritage heritage; diff --git a/bin/varnishd/mgt/mgt_acceptor.c b/bin/varnishd/mgt/mgt_acceptor.c index 1d513bc31..dd00d77bc 100644 --- a/bin/varnishd/mgt/mgt_acceptor.c +++ b/bin/varnishd/mgt/mgt_acceptor.c @@ -269,8 +269,8 @@ MAC_Arg(const char *spec) ARGV_ERR("Unix domain socket addresses must be" " absolute paths in -a (%s)\n", la->endpoint); - if (*la->endpoint == '/' && heritage.min_vcl < 41) - heritage.min_vcl = 41; + if (*la->endpoint == '/' && heritage.min_vcl_version < 41) + heritage.min_vcl_version = 41; for (int i = 2; av[i] != NULL; i++) { char *eq, *val; diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c index 4753ef69b..0b21c7c61 100644 --- a/bin/varnishd/mgt/mgt_main.c +++ b/bin/varnishd/mgt/mgt_main.c @@ -286,7 +286,7 @@ mgt_initialize(struct cli *cli) MCF_InitParams(cli); - VCC_VCL_Range(&heritage.min_vcl, &heritage.max_vcl); + VCC_VCL_Range(&heritage.min_vcl_version, &heritage.max_vcl_version); cli_check(cli); } From nils.goroll at uplex.de Thu Feb 13 09:29:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 13 Feb 2020 09:29:06 +0000 (UTC) Subject: [master] 9bba6622d centralize setting ctx->syntax Message-ID: <20200213092906.48353A4ED5@lists.varnish-cache.org> commit 9bba6622d9f4e056c72cc0c80fea57ea744d0d0b Author: Nils Goroll Date: Thu Feb 13 10:01:36 2020 +0100 centralize setting ctx->syntax ... and ensure it always gets set during cli operations also diff --git a/bin/varnishd/cache/cache_vcl.c b/bin/varnishd/cache/cache_vcl.c index 2e1f6a69f..c1c11e6a1 100644 --- a/bin/varnishd/cache/cache_vcl.c +++ b/bin/varnishd/cache/cache_vcl.c @@ -83,6 +83,7 @@ VCL_Bo2Ctx(struct vrt_ctx *ctx, struct busyobj *bo) CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); CHECK_OBJ_NOTNULL(bo->wrk, WORKER_MAGIC); ctx->vcl = bo->vcl; + ctx->syntax = ctx->vcl->conf->syntax; ctx->vsl = bo->vsl; ctx->http_bereq = bo->bereq; ctx->http_beresp = bo->beresp; @@ -102,6 +103,7 @@ VCL_Req2Ctx(struct vrt_ctx *ctx, struct req *req) CHECK_OBJ_NOTNULL(req, REQ_MAGIC); ctx->vcl = req->vcl; + ctx->syntax = ctx->vcl->conf->syntax; ctx->vsl = req->vsl; ctx->http_req = req->http; CHECK_OBJ_NOTNULL(req->top, REQTOP_MAGIC); @@ -586,6 +588,7 @@ vcl_cancel_load(struct vrt_ctx *ctx, struct cli *cli, ctx = VCL_Get_CliCtx(0); ctx->vcl = vcl; ctx->method = VCL_MET_FINI; + ctx->syntax = ctx->vcl->conf->syntax; AZ(vcl_send_event(ctx, VCL_EVENT_DISCARD)); ctx->method = 0; vcl_KillBackends(vcl); @@ -623,6 +626,7 @@ vcl_load(struct cli *cli, struct vrt_ctx *ctx, vcl->temp = VCL_TEMP_INIT; ctx->vcl = vcl; + ctx->syntax = ctx->vcl->conf->syntax; VSB_clear(ctx->msg); ctx->method = VCL_MET_INIT; @@ -801,6 +805,7 @@ vcl_cli_state(struct cli *cli, const char * const *av, void *priv) ctx = VCL_Get_CliCtx(1); ctx->vcl = vcl_find(av[2]); AN(ctx->vcl); + ctx->syntax = ctx->vcl->conf->syntax; if (vcl_set_state(ctx, av[3])) { AZ(VSB_finish(ctx->msg)); VCLI_SetResult(cli, CLIS_CANT); diff --git a/bin/varnishd/cache/cache_vrt_vcl.c b/bin/varnishd/cache/cache_vrt_vcl.c index 2d7b39a36..eda1fe3db 100644 --- a/bin/varnishd/cache/cache_vrt_vcl.c +++ b/bin/varnishd/cache/cache_vrt_vcl.c @@ -442,7 +442,6 @@ vcl_call_method(struct worker *wrk, struct req *req, struct busyobj *bo, VCL_Bo2Ctx(&ctx, bo); } assert(ctx.now != 0); - ctx.syntax = ctx.vcl->conf->syntax; ctx.specific = specific; ctx.method = method; aws = WS_Snapshot(wrk->aws); From dridi.boukelmoune at gmail.com Thu Feb 13 11:40:05 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 13 Feb 2020 11:40:05 +0000 (UTC) Subject: [master] cee2d1aa0 Polish Message-ID: <20200213114005.D6102A7D54@lists.varnish-cache.org> commit cee2d1aa0a7261ccef272cd223c60191b33f86fb Author: Dridi Boukelmoune Date: Thu Feb 13 12:33:32 2020 +0100 Polish diff --git a/bin/varnishstat/varnishstat_bindings.h b/bin/varnishstat/varnishstat_bindings.h index 7d8cbc2ba..be9ec8c8e 100644 --- a/bin/varnishstat/varnishstat_bindings.h +++ b/bin/varnishstat/varnishstat_bindings.h @@ -51,7 +51,7 @@ BINDING(PAGEUP, "\tNavigate the counter list one page up.") BINDING_KEY(KEY_NPAGE, "PAGEDOWN", " or ") BINDING_KEY(' ', "SPACE", " or ") -BINDING_KEY(BINDING_CTRL('F'), "CTRL-F",) +BINDING_KEY(BINDING_CTRL('f'), "CTRL-F",) BINDING(PAGEDOWN, "\tNavigate the counter list one page down.") BINDING_KEY(KEY_HOME, "HOME", " or ") diff --git a/bin/varnishstat/varnishstat_curses.c b/bin/varnishstat/varnishstat_curses.c index f61612fb6..9543448c5 100644 --- a/bin/varnishstat/varnishstat_curses.c +++ b/bin/varnishstat/varnishstat_curses.c @@ -881,7 +881,7 @@ handle_keypress(int ch) #define BINDING_KEY(chr, name, or) \ case chr: #define BINDING(name, desc) \ - kb = KB_ ## name; \ + kb = KB_ ## name; \ break; #define BINDING_SIG #include "varnishstat_bindings.h" From dridi.boukelmoune at gmail.com Thu Feb 13 11:40:05 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 13 Feb 2020 11:40:05 +0000 (UTC) Subject: [master] 1f1234827 Use AM_SUBST_NOTMAKE directly Message-ID: <20200213114005.E860CA7D57@lists.varnish-cache.org> commit 1f1234827d315c6678c97d448253f53489367ae4 Author: Dridi Boukelmoune Date: Thu Feb 13 12:34:55 2020 +0100 Use AM_SUBST_NOTMAKE directly I don't remember how I ended up with its private counterpart, conditionally even, but this has been part of automake since version 1.10 so there's no reason to beat around the bush. diff --git a/varnish.m4 b/varnish.m4 index 13a273f39..a7ef062ab 100644 --- a/varnish.m4 +++ b/varnish.m4 @@ -226,8 +226,7 @@ clean-vmod-$1: " AC_SUBST(m4_toupper(BUILD_VMOD_$1), [$VMOD_RULES]) - m4_ifdef([_AM_SUBST_NOTMAKE], - [_AM_SUBST_NOTMAKE(m4_toupper(BUILD_VMOD_$1))]) + AM_SUBST_NOTMAKE(m4_toupper(BUILD_VMOD_$1)) ]) # VARNISH_VMODS(NAMES) @@ -418,8 +417,7 @@ clean-vsc-$1: " AC_SUBST(m4_toupper(BUILD_VSC_$1), [$VSC_RULES]) - m4_ifdef([_AM_SUBST_NOTMAKE], - [_AM_SUBST_NOTMAKE(m4_toupper(BUILD_VSC_$1))]) + AM_SUBST_NOTMAKE(m4_toupper(BUILD_VSC_$1)) ]) # VARNISH_COUNTERS(NAMES) @@ -488,8 +486,7 @@ clean-vut-$1: " AC_SUBST(m4_toupper(GENERATE_$1_DOCS), [$VUT_RULES]) - m4_ifdef([_AM_SUBST_NOTMAKE], - [_AM_SUBST_NOTMAKE(m4_toupper(GENERATE_$1_DOCS))]) + AM_SUBST_NOTMAKE(m4_toupper(GENERATE_$1_DOCS)) ]) From nils.goroll at uplex.de Thu Feb 13 21:11:07 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 13 Feb 2020 21:11:07 +0000 (UTC) Subject: [master] 89198513a RX_CLOSE_IDLE accounted as sc_rx_close_idle for timeout_idle hit Message-ID: <20200213211107.70F3D617E@lists.varnish-cache.org> commit 89198513a4fba2ed2370918bc7c78c61c7febdd4 Author: Nils Goroll Date: Thu Feb 13 22:07:42 2020 +0100 RX_CLOSE_IDLE accounted as sc_rx_close_idle for timeout_idle hit RX_TIMEOUT was issued when the timeout_idle was hit waiting for another client request and accounted as an error in sess_closed_err. We now turn this case into an RX_CLOSE_IDLE which does not get accounted as an error. Closes #3208 diff --git a/bin/varnishd/VSC_main.vsc b/bin/varnishd/VSC_main.vsc index ca5704267..a7cacb7cc 100644 --- a/bin/varnishd/VSC_main.vsc +++ b/bin/varnishd/VSC_main.vsc @@ -539,6 +539,14 @@ Number of session closes with Error RX_TIMEOUT (Receive timeout) +.. varnish_vsc:: sc_rx_close_idle + :level: diag + :oneliner: Session Err RX_CLOSE_IDLE + + Number of session closes with Error RX_CLOSE_IDLE: + timeout_idle has been exceeded while waiting for a + client request. + .. varnish_vsc:: sc_tx_pipe :level: diag :oneliner: Session OK TX_PIPE diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index 973304b88..9549f12b8 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -424,7 +424,7 @@ ses_handle(struct waited *wp, enum wait_event ev, vtim_real now) switch (ev) { case WAITER_TIMEOUT: - SES_Delete(sp, SC_RX_TIMEOUT, now); + SES_Delete(sp, SC_RX_CLOSE_IDLE, now); break; case WAITER_REMCLOSE: SES_Delete(sp, SC_REM_CLOSE, now); diff --git a/include/tbl/sess_close.h b/include/tbl/sess_close.h index 61d8f3b99..97483147c 100644 --- a/include/tbl/sess_close.h +++ b/include/tbl/sess_close.h @@ -40,6 +40,7 @@ SESS_CLOSE(RX_BODY, rx_body, 1, "Failure receiving req.body") SESS_CLOSE(RX_JUNK, rx_junk, 1, "Received junk data") SESS_CLOSE(RX_OVERFLOW, rx_overflow, 1, "Received buffer overflow") SESS_CLOSE(RX_TIMEOUT, rx_timeout, 1, "Receive timeout") +SESS_CLOSE(RX_CLOSE_IDLE, rx_close_idle,0, "timeout_idle reached") SESS_CLOSE(TX_PIPE, tx_pipe, 0, "Piped transaction") SESS_CLOSE(TX_ERROR, tx_error, 1, "Error transaction") SESS_CLOSE(TX_EOF, tx_eof, 0, "EOF transmission") From dridi.boukelmoune at gmail.com Thu Feb 13 21:45:09 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 13 Feb 2020 21:45:09 +0000 (UTC) Subject: [master] 8abc20da6 Enable automake maintainer mode for development Message-ID: <20200213214509.DCA4B6FEB@lists.varnish-cache.org> commit 8abc20da6066dc14b410e94c48c464c2639cbb49 Author: Dridi Boukelmoune Date: Thu Feb 13 20:20:43 2020 +0100 Enable automake maintainer mode for development In maintainer mode, whenever a file managed by autoconf or automake is changed it is regenerated on the fly. For Makefile.am files this also applies to any fragment .am file referenced by the include directive. The configure script disables the maintainer mode by default, which is probably a sane behavior because downstream usage of dist archives don't need the various autotools to be present, unless a Makefile or similar is patched. This is explicitly enabled by the autogen.des script, in order to make this change only impact development, saving most of the time a configure step after amending something in the build. Refs #3184 diff --git a/autogen.des b/autogen.des index ecb24d9fa..6fe0dcbca 100755 --- a/autogen.des +++ b/autogen.des @@ -34,6 +34,7 @@ export CONFIG_SHELL # env MAKE=gmake \ $SRCDIR/configure \ $DST \ + --enable-maintainer-mode \ --enable-developer-warnings \ --enable-debugging-symbols \ --enable-dependency-tracking \ From dridi.boukelmoune at gmail.com Thu Feb 13 21:45:09 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 13 Feb 2020 21:45:09 +0000 (UTC) Subject: [master] 4eeec9744 Ensure everything is built before running tests Message-ID: <20200213214509.F34EA6FEF@lists.varnish-cache.org> commit 4eeec97447aed237c67070fbf94030edb9967061 Author: Dridi Boukelmoune Date: Thu Feb 13 20:26:09 2020 +0100 Ensure everything is built before running tests With a growing list of bundled VMODs we want to split the main test suite to colocate VMOD-specific tests with the rest of its sources. Because a VMOD test case may involve other binaries, and will always involve at least varnishtest and varnishd, we insist that make check kicks in only after a complete recursive build. This doesn't work in this scenario: cd lib/libvmod_ && make check The assumption is that at this point a full build was already performed and changes are only happening in that sub directory. Refs #3184 diff --git a/Makefile.am b/Makefile.am index c2cbd1c13..60c388610 100644 --- a/Makefile.am +++ b/Makefile.am @@ -50,6 +50,11 @@ vtest-clean: # solved it better, but we don't, so use this at least for now. LICENSE: all +# XXX: This is a similar hack to ensure we have a built varnishtest +# (and technically any other binary that may be involved in a VTC) +# before we try to run tests anywhere in the tree. +check-recursive: all + cscope: -rm -f cscope* find . -name '*.[hcS]' > cscope.files From dridi.boukelmoune at gmail.com Thu Feb 13 21:45:10 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 13 Feb 2020 21:45:10 +0000 (UTC) Subject: [master] 182e96ff7 Extract the VTC "make check" logic to vtc.am Message-ID: <20200213214510.264816FF2@lists.varnish-cache.org> commit 182e96ff73925dea016195832e31443cbf4992ed Author: Dridi Boukelmoune Date: Thu Feb 13 20:39:13 2020 +0100 Extract the VTC "make check" logic to vtc.am The effect is currently the same, but for VMODs we shouldn't need a complete configure step. Refs #3184 diff --git a/bin/varnishtest/Makefile.am b/bin/varnishtest/Makefile.am index 7324b7473..4ebff33a8 100644 --- a/bin/varnishtest/Makefile.am +++ b/bin/varnishtest/Makefile.am @@ -1,18 +1,11 @@ # -VTC_LOG_COMPILER = ./varnishtest -v -i -TEST_EXTENSIONS = .vtc TESTS = @VTC_TESTS@ -# Make sure we run check-local first -check-am: check-local -# See if list of checks have changed, recheck -check-local: - @mkdir -p tests ; \ - LC_ALL=C; \ - if [ "$$(cd $(srcdir) && echo tests/*.vtc)" != "@VTC_TESTS@" ]; then \ - cd $(top_builddir) && ./config.status --recheck ; \ - fi +vtc-refresh-tests: + @cd $(top_builddir) && ./config.status --recheck + +include $(top_srcdir)/vtc.am DISTCLEANFILES = _.ok diff --git a/vtc.am b/vtc.am new file mode 100644 index 000000000..26ad611c3 --- /dev/null +++ b/vtc.am @@ -0,0 +1,21 @@ +# This file helps set up a varnishtest execution without a traditional +# pre-defined list of tests. Test cases are expected to be located in a +# tests/ directory relative to the Makefile's source directory. +# +# When a difference is detected, a refresh is triggered by calling the +# vtc-refresh-tests target that must be defined in the including Makefile. +# +# The current assumption is that all the tests for the Makefile are VTCs +# and may be revisited if needed. + +VTC_LOG_COMPILER = $(top_builddir)/bin/varnishtest/varnishtest -v -i +TEST_EXTENSIONS = .vtc + +check-am: vtc-check-tests + +vtc-check-tests: + @mkdir -p tests \ + LC_ALL=C; \ + if [ "$$(cd $(srcdir) && echo tests/*.vtc)" != "$(TESTS)" ]; then \ + $(MAKE) $(AM_MAKEFLAGS) vtc-refresh-tests ; \ + fi From dridi.boukelmoune at gmail.com Thu Feb 13 21:45:10 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 13 Feb 2020 21:45:10 +0000 (UTC) Subject: [master] 6ed45b8be Sort vmodtool's imports Message-ID: <20200213214510.4F34E6FFB@lists.varnish-cache.org> commit 6ed45b8be8113a8d2362cea5e75581c5d57698b2 Author: Dridi Boukelmoune Date: Thu Feb 13 20:40:58 2020 +0100 Sort vmodtool's imports diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py index 7f73d9c41..158e8bd95 100755 --- a/lib/libvcc/vmodtool.py +++ b/lib/libvcc/vmodtool.py @@ -36,13 +36,13 @@ Read the vmod.vcc file (inputvcc) and produce: vmod_${name}.man.rst -- Extracted documentation (rst2man input) """ -import os -import sys -import re -import optparse import copy -import json import hashlib +import json +import optparse +import os +import re +import sys import time AMBOILERPLATE = ''' From dridi.boukelmoune at gmail.com Thu Feb 13 21:45:10 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 13 Feb 2020 21:45:10 +0000 (UTC) Subject: [master] 4ace3ef6b Generate test suite setup in vmodtool Message-ID: <20200213214510.A35BF7002@lists.varnish-cache.org> commit 4ace3ef6b1e670549f7f120c31d5f924622a9158 Author: Dridi Boukelmoune Date: Thu Feb 13 20:49:42 2020 +0100 Generate test suite setup in vmodtool That part of the automake boilerplate is generated conditionally, and since we check automake_boilerplate.am files in the list of tests is printed with one per line to be diff-friendly. A test was "randomly" chosen to be the first VMOD-specific test. Generated automake_boilerplate.am files no longer start nor end with empty lines. This is probably going to mandate some changes in vtest to collect VMOD-specific test suites. Refs #3184 diff --git a/.gitignore b/.gitignore index f2b991fc5..2fe1a1b6d 100644 --- a/.gitignore +++ b/.gitignore @@ -13,6 +13,7 @@ _* *~ *.sw[op] *.trs +*.log # Various auto-tools artifacts /aclocal.m4 @@ -22,8 +23,6 @@ _* /config.guess /config.h /config.h.in -/config.h.in~ -/config.log /config.status /config.sub /configure @@ -108,22 +107,16 @@ cscope.*out /nbproject/private/ # Test droppings -/bin/varnishd/*.log /bin/varnishd/vhp_decode_test /bin/varnishd/vhp_table_test -/bin/varnishtest/*.log -/bin/varnishtest/tests/*.log /bin/varnishtest/tests/*.log-t -/include/*.log /include/_vrt.c /include/_vrt_test /include/vrt_test /include/vbm_test -/lib/libvarnish/*.log /lib/libvarnish/binheap /lib/libvarnish/vjsn_test /lib/libvarnish/vnum_c_test -/lib/libvarnishapi/*.log /lib/libvarnishapi/vsl_glob_test /lib/libvarnishapi/vsl_glob_test_coverage diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py index 158e8bd95..3522545c8 100755 --- a/lib/libvcc/vmodtool.py +++ b/lib/libvcc/vmodtool.py @@ -37,6 +37,7 @@ Read the vmod.vcc file (inputvcc) and produce: """ import copy +import glob import hashlib import json import optparse @@ -45,7 +46,7 @@ import re import sys import time -AMBOILERPLATE = ''' +AMBOILERPLATE = '''\ # Generated by vmodtool.py --boilerplate. AM_LDFLAGS = $(AM_LT_LDFLAGS) @@ -84,7 +85,17 @@ EXTRA_DIST = vmod.vcc automake_boilerplate.am CLEANFILES = $(builddir)/PFX.c $(builddir)/PFX.h \\ \t$(builddir)/vmod_XXX.rst \\ \t$(builddir)/vmod_XXX.man.rst +''' + +AMBOILERPLATE_CHECK = ''' +TESTS = \\ +\tXXX + +vtc-refresh-tests: +\t at PYTHON@ $(vmodtool) $(vmodtoolargs) $(srcdir)/vmod.vcc +\t at cd $(top_builddir) && ./config.status --file=$(subdir)/Makefile +include $(top_srcdir)/vtc.am ''' PRIVS = { @@ -975,6 +986,9 @@ class vcc(object): fo = self.openfile("automake_boilerplate.am") fo.write(AMBOILERPLATE.replace("XXX", self.modname) .replace("PFX", self.pfx)) + tests = glob.glob("tests/*.vtc") + if len(tests) > 0: + fo.write(AMBOILERPLATE_CHECK.replace("XXX", " \\\n\t".join(tests))) fo.close() def mkdefs(self, fo): diff --git a/lib/libvmod_blob/automake_boilerplate.am b/lib/libvmod_blob/automake_boilerplate.am index e3873eb7c..39118cb4d 100644 --- a/lib/libvmod_blob/automake_boilerplate.am +++ b/lib/libvmod_blob/automake_boilerplate.am @@ -1,4 +1,3 @@ - # Generated by vmodtool.py --boilerplate. AM_LDFLAGS = $(AM_LT_LDFLAGS) @@ -37,4 +36,3 @@ EXTRA_DIST = vmod.vcc automake_boilerplate.am CLEANFILES = $(builddir)/vcc_if.c $(builddir)/vcc_if.h \ $(builddir)/vmod_blob.rst \ $(builddir)/vmod_blob.man.rst - diff --git a/lib/libvmod_debug/automake_boilerplate.am b/lib/libvmod_debug/automake_boilerplate.am index 44f1f6691..5c8c8afbb 100644 --- a/lib/libvmod_debug/automake_boilerplate.am +++ b/lib/libvmod_debug/automake_boilerplate.am @@ -1,4 +1,3 @@ - # Generated by vmodtool.py --boilerplate. AM_LDFLAGS = $(AM_LT_LDFLAGS) @@ -37,4 +36,3 @@ EXTRA_DIST = vmod.vcc automake_boilerplate.am CLEANFILES = $(builddir)/vcc_if.c $(builddir)/vcc_if.h \ $(builddir)/vmod_debug.rst \ $(builddir)/vmod_debug.man.rst - diff --git a/lib/libvmod_directors/automake_boilerplate.am b/lib/libvmod_directors/automake_boilerplate.am index f1b6cac7e..2d7a779fb 100644 --- a/lib/libvmod_directors/automake_boilerplate.am +++ b/lib/libvmod_directors/automake_boilerplate.am @@ -1,4 +1,3 @@ - # Generated by vmodtool.py --boilerplate. AM_LDFLAGS = $(AM_LT_LDFLAGS) @@ -37,4 +36,3 @@ EXTRA_DIST = vmod.vcc automake_boilerplate.am CLEANFILES = $(builddir)/vcc_if.c $(builddir)/vcc_if.h \ $(builddir)/vmod_directors.rst \ $(builddir)/vmod_directors.man.rst - diff --git a/lib/libvmod_proxy/automake_boilerplate.am b/lib/libvmod_proxy/automake_boilerplate.am index 683f8b80f..c81749811 100644 --- a/lib/libvmod_proxy/automake_boilerplate.am +++ b/lib/libvmod_proxy/automake_boilerplate.am @@ -1,4 +1,3 @@ - # Generated by vmodtool.py --boilerplate. AM_LDFLAGS = $(AM_LT_LDFLAGS) @@ -37,4 +36,3 @@ EXTRA_DIST = vmod.vcc automake_boilerplate.am CLEANFILES = $(builddir)/vcc_if.c $(builddir)/vcc_if.h \ $(builddir)/vmod_proxy.rst \ $(builddir)/vmod_proxy.man.rst - diff --git a/lib/libvmod_purge/automake_boilerplate.am b/lib/libvmod_purge/automake_boilerplate.am index ab6b314f3..961c5e967 100644 --- a/lib/libvmod_purge/automake_boilerplate.am +++ b/lib/libvmod_purge/automake_boilerplate.am @@ -1,4 +1,3 @@ - # Generated by vmodtool.py --boilerplate. AM_LDFLAGS = $(AM_LT_LDFLAGS) @@ -37,4 +36,3 @@ EXTRA_DIST = vmod.vcc automake_boilerplate.am CLEANFILES = $(builddir)/vcc_if.c $(builddir)/vcc_if.h \ $(builddir)/vmod_purge.rst \ $(builddir)/vmod_purge.man.rst - diff --git a/lib/libvmod_std/automake_boilerplate.am b/lib/libvmod_std/automake_boilerplate.am index 3538cfdd5..0153fcd64 100644 --- a/lib/libvmod_std/automake_boilerplate.am +++ b/lib/libvmod_std/automake_boilerplate.am @@ -1,4 +1,3 @@ - # Generated by vmodtool.py --boilerplate. AM_LDFLAGS = $(AM_LT_LDFLAGS) @@ -38,3 +37,11 @@ CLEANFILES = $(builddir)/vcc_if.c $(builddir)/vcc_if.h \ $(builddir)/vmod_std.rst \ $(builddir)/vmod_std.man.rst +TESTS = \ + tests/b00001.vtc + +vtc-refresh-tests: + @PYTHON@ $(vmodtool) $(vmodtoolargs) $(srcdir)/vmod.vcc + @cd $(top_builddir) && ./config.status --file=$(subdir)/Makefile + +include $(top_srcdir)/vtc.am diff --git a/bin/varnishtest/tests/m00002.vtc b/lib/libvmod_std/tests/b00001.vtc similarity index 100% rename from bin/varnishtest/tests/m00002.vtc rename to lib/libvmod_std/tests/b00001.vtc diff --git a/lib/libvmod_unix/automake_boilerplate.am b/lib/libvmod_unix/automake_boilerplate.am index 967bffec8..f2582e27f 100644 --- a/lib/libvmod_unix/automake_boilerplate.am +++ b/lib/libvmod_unix/automake_boilerplate.am @@ -1,4 +1,3 @@ - # Generated by vmodtool.py --boilerplate. AM_LDFLAGS = $(AM_LT_LDFLAGS) @@ -37,4 +36,3 @@ EXTRA_DIST = vmod.vcc automake_boilerplate.am CLEANFILES = $(builddir)/vcc_if.c $(builddir)/vcc_if.h \ $(builddir)/vmod_unix.rst \ $(builddir)/vmod_unix.man.rst - diff --git a/lib/libvmod_vtc/automake_boilerplate.am b/lib/libvmod_vtc/automake_boilerplate.am index d82f03a78..feb939901 100644 --- a/lib/libvmod_vtc/automake_boilerplate.am +++ b/lib/libvmod_vtc/automake_boilerplate.am @@ -1,4 +1,3 @@ - # Generated by vmodtool.py --boilerplate. AM_LDFLAGS = $(AM_LT_LDFLAGS) @@ -37,4 +36,3 @@ EXTRA_DIST = vmod.vcc automake_boilerplate.am CLEANFILES = $(builddir)/vcc_if.c $(builddir)/vcc_if.h \ $(builddir)/vmod_vtc.rst \ $(builddir)/vmod_vtc.man.rst - From dridi.boukelmoune at gmail.com Thu Feb 13 21:45:10 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 13 Feb 2020 21:45:10 +0000 (UTC) Subject: [master] 92d3f677e We don't need $(VTC_TESTS) in Makefiles Message-ID: <20200213214510.DF7337007@lists.varnish-cache.org> commit 92d3f677e1320205db79c092900b2bc04e3f1367 Author: Dridi Boukelmoune Date: Thu Feb 13 22:33:41 2020 +0100 We don't need $(VTC_TESTS) in Makefiles We only need the @VTC_TESTS@ substitution, and doing so saves around 13kB per Makefile. diff --git a/configure.ac b/configure.ac index 297e8223e..61c19dbcc 100644 --- a/configure.ac +++ b/configure.ac @@ -817,6 +817,7 @@ AC_DEFINE_UNQUOTED([VCC_CC],"$VCC_CC",[C compiler command line for VCL code]) # Stupid automake needs this VTC_TESTS="$(cd $srcdir/bin/varnishtest && echo tests/*.vtc)" AC_SUBST(VTC_TESTS) +AM_SUBST_NOTMAKE(VTC_TESTS) VSC_SRC="$(cd $srcdir/bin/varnishd && echo *.vsc)" VSC_GEN_C="$(echo $VSC_SRC | sed 's:\.vsc:.c:g')" From dridi.boukelmoune at gmail.com Thu Feb 13 22:25:14 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 13 Feb 2020 22:25:14 +0000 (UTC) Subject: [master] dec8e14bb Fix distcheck Message-ID: <20200213222514.E7D589869@lists.varnish-cache.org> commit dec8e14bb515cafc80687883fc3b27fc684dac25 Author: Dridi Boukelmoune Date: Thu Feb 13 23:24:45 2020 +0100 Fix distcheck diff --git a/Makefile.am b/Makefile.am index 60c388610..c70a16293 100644 --- a/Makefile.am +++ b/Makefile.am @@ -22,7 +22,8 @@ EXTRA_DIST = \ autogen.sh \ varnishapi.pc.in \ varnish.m4 \ - varnish-legacy.m4 + varnish-legacy.m4 \ + vtc.am AM_DISTCHECK_CONFIGURE_FLAGS = \ --enable-developer-warnings \ diff --git a/vtc.am b/vtc.am index 26ad611c3..840687a43 100644 --- a/vtc.am +++ b/vtc.am @@ -14,7 +14,7 @@ TEST_EXTENSIONS = .vtc check-am: vtc-check-tests vtc-check-tests: - @mkdir -p tests \ + @mkdir -p tests ; \ LC_ALL=C; \ if [ "$$(cd $(srcdir) && echo tests/*.vtc)" != "$(TESTS)" ]; then \ $(MAKE) $(AM_MAKEFLAGS) vtc-refresh-tests ; \ From dridi.boukelmoune at gmail.com Fri Feb 14 09:23:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 14 Feb 2020 09:23:07 +0000 (UTC) Subject: [master] ae01dfe73 Distribute VMODs tests Message-ID: <20200214092307.6F895105FB9@lists.varnish-cache.org> commit ae01dfe73c6f0c2f5291dcfcbf6b93e5091aeb42 Author: Dridi Boukelmoune Date: Fri Feb 14 10:22:32 2020 +0100 Distribute VMODs tests diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py index 3522545c8..7683e2614 100755 --- a/lib/libvcc/vmodtool.py +++ b/lib/libvcc/vmodtool.py @@ -91,6 +91,8 @@ AMBOILERPLATE_CHECK = ''' TESTS = \\ \tXXX +EXTRA_DIST += $(TESTS) + vtc-refresh-tests: \t at PYTHON@ $(vmodtool) $(vmodtoolargs) $(srcdir)/vmod.vcc \t at cd $(top_builddir) && ./config.status --file=$(subdir)/Makefile diff --git a/lib/libvmod_std/automake_boilerplate.am b/lib/libvmod_std/automake_boilerplate.am index 0153fcd64..f51194a0e 100644 --- a/lib/libvmod_std/automake_boilerplate.am +++ b/lib/libvmod_std/automake_boilerplate.am @@ -40,6 +40,8 @@ CLEANFILES = $(builddir)/vcc_if.c $(builddir)/vcc_if.h \ TESTS = \ tests/b00001.vtc +EXTRA_DIST += $(TESTS) + vtc-refresh-tests: @PYTHON@ $(vmodtool) $(vmodtoolargs) $(srcdir)/vmod.vcc @cd $(top_builddir) && ./config.status --file=$(subdir)/Makefile From dridi.boukelmoune at gmail.com Mon Feb 17 10:23:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 17 Feb 2020 10:23:07 +0000 (UTC) Subject: [master] 5e602798c Sort VMOD tests Message-ID: <20200217102307.2C3E2117ACE@lists.varnish-cache.org> commit 5e602798c36dadf63acaf93995101f6259ce5916 Author: Dridi Boukelmoune Date: Mon Feb 17 08:11:29 2020 +0100 Sort VMOD tests diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py index 7683e2614..2a8a9444c 100755 --- a/lib/libvcc/vmodtool.py +++ b/lib/libvcc/vmodtool.py @@ -990,6 +990,7 @@ class vcc(object): .replace("PFX", self.pfx)) tests = glob.glob("tests/*.vtc") if len(tests) > 0: + tests.sort() fo.write(AMBOILERPLATE_CHECK.replace("XXX", " \\\n\t".join(tests))) fo.close() From dridi.boukelmoune at gmail.com Mon Feb 17 10:23:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 17 Feb 2020 10:23:07 +0000 (UTC) Subject: [master] f074c2e26 mv lib/libvmod_std/tests/{,std_}b00001.vtc Message-ID: <20200217102307.49F8B117AD1@lists.varnish-cache.org> commit f074c2e26387d676ad2c80a72390600051f3c097 Author: Dridi Boukelmoune Date: Mon Feb 17 10:27:45 2020 +0100 mv lib/libvmod_std/tests/{,std_}b00001.vtc I forgot the namespace when I first split the test suite. diff --git a/lib/libvmod_std/automake_boilerplate.am b/lib/libvmod_std/automake_boilerplate.am index f51194a0e..9ff7f9425 100644 --- a/lib/libvmod_std/automake_boilerplate.am +++ b/lib/libvmod_std/automake_boilerplate.am @@ -38,7 +38,7 @@ CLEANFILES = $(builddir)/vcc_if.c $(builddir)/vcc_if.h \ $(builddir)/vmod_std.man.rst TESTS = \ - tests/b00001.vtc + tests/std_b00001.vtc EXTRA_DIST += $(TESTS) diff --git a/lib/libvmod_std/tests/b00001.vtc b/lib/libvmod_std/tests/std_b00001.vtc similarity index 100% rename from lib/libvmod_std/tests/b00001.vtc rename to lib/libvmod_std/tests/std_b00001.vtc From dridi.boukelmoune at gmail.com Mon Feb 17 10:27:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 17 Feb 2020 10:27:06 +0000 (UTC) Subject: [master] 6b64399f8 Whitespace OCD Message-ID: <20200217102706.DFAA4117EE3@lists.varnish-cache.org> commit 6b64399f8e7e243aa0182627b2d32efe800d0733 Author: Dridi Boukelmoune Date: Mon Feb 17 11:25:00 2020 +0100 Whitespace OCD diff --git a/lib/libvarnish/vct.c b/lib/libvarnish/vct.c index cbc6da20a..1c301d37e 100644 --- a/lib/libvarnish/vct.c +++ b/lib/libvarnish/vct.c @@ -79,22 +79,22 @@ const uint16_t vct_typtab[256] = { [0x1d] = VCT_CTL, [0x1e] = VCT_CTL, [0x1f] = VCT_CTL, - [0x20] = VCT_SP, - [0x21] = VCT_TCHAR, - [0x22] = VCT_SEPARATOR, - [0x23] = VCT_TCHAR, - [0x24] = VCT_TCHAR, - [0x25] = VCT_TCHAR, - [0x26] = VCT_TCHAR, - [0x27] = VCT_TCHAR, - [0x28] = VCT_SEPARATOR, - [0x29] = VCT_SEPARATOR, - [0x2a] = VCT_TCHAR, - [0x2b] = VCT_TCHAR, - [0x2c] = VCT_SEPARATOR, + [0x20] = VCT_SP, + [0x21] = VCT_TCHAR, + [0x22] = VCT_SEPARATOR, + [0x23] = VCT_TCHAR, + [0x24] = VCT_TCHAR, + [0x25] = VCT_TCHAR, + [0x26] = VCT_TCHAR, + [0x27] = VCT_TCHAR, + [0x28] = VCT_SEPARATOR, + [0x29] = VCT_SEPARATOR, + [0x2a] = VCT_TCHAR, + [0x2b] = VCT_TCHAR, + [0x2c] = VCT_SEPARATOR, [0x2d] = VCT_XMLNAME | VCT_TCHAR | VCT_ID, [0x2e] = VCT_XMLNAME | VCT_TCHAR, - [0x2f] = VCT_SEPARATOR, + [0x2f] = VCT_SEPARATOR, [0x30] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, [0x31] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, [0x32] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, @@ -105,13 +105,13 @@ const uint16_t vct_typtab[256] = { [0x37] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, [0x38] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, [0x39] = VCT_DIGIT | VCT_HEX | VCT_XMLNAME, - [0x3a] = VCT_SEPARATOR | VCT_XMLNAMESTART, - [0x3b] = VCT_SEPARATOR, - [0x3c] = VCT_SEPARATOR, - [0x3d] = VCT_SEPARATOR, - [0x3e] = VCT_SEPARATOR, - [0x3f] = VCT_SEPARATOR, - [0x40] = VCT_SEPARATOR, + [0x3a] = VCT_SEPARATOR | VCT_XMLNAMESTART, + [0x3b] = VCT_SEPARATOR, + [0x3c] = VCT_SEPARATOR, + [0x3d] = VCT_SEPARATOR, + [0x3e] = VCT_SEPARATOR, + [0x3f] = VCT_SEPARATOR, + [0x40] = VCT_SEPARATOR, [0x41] = VCT_UPALPHA | VCT_HEX | VCT_XMLNAMESTART, [0x42] = VCT_UPALPHA | VCT_HEX | VCT_XMLNAMESTART, [0x43] = VCT_UPALPHA | VCT_HEX | VCT_XMLNAMESTART, @@ -138,10 +138,10 @@ const uint16_t vct_typtab[256] = { [0x58] = VCT_UPALPHA | VCT_XMLNAMESTART, [0x59] = VCT_UPALPHA | VCT_XMLNAMESTART, [0x5a] = VCT_UPALPHA | VCT_XMLNAMESTART, - [0x5b] = VCT_SEPARATOR, - [0x5c] = VCT_SEPARATOR, - [0x5d] = VCT_SEPARATOR, - [0x5e] = VCT_TCHAR, + [0x5b] = VCT_SEPARATOR, + [0x5c] = VCT_SEPARATOR, + [0x5d] = VCT_SEPARATOR, + [0x5e] = VCT_TCHAR, [0x5f] = VCT_XMLNAMESTART | VCT_TCHAR | VCT_ID, [0x60] = VCT_TCHAR, [0x61] = VCT_LOALPHA | VCT_HEX | VCT_XMLNAMESTART, @@ -170,9 +170,9 @@ const uint16_t vct_typtab[256] = { [0x78] = VCT_LOALPHA | VCT_XMLNAMESTART, [0x79] = VCT_LOALPHA | VCT_XMLNAMESTART, [0x7a] = VCT_LOALPHA | VCT_XMLNAMESTART, - [0x7b] = VCT_SEPARATOR, + [0x7b] = VCT_SEPARATOR, [0x7c] = VCT_TCHAR, - [0x7d] = VCT_SEPARATOR, + [0x7d] = VCT_SEPARATOR, [0x7e] = VCT_TCHAR, [0x7f] = VCT_CTL, [0xb7] = VCT_XMLNAME, From nils.goroll at uplex.de Mon Feb 17 17:05:07 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 17 Feb 2020 17:05:07 +0000 (UTC) Subject: [master] 746384b20 demo how a priv_task can veto being rolled back Message-ID: <20200217170507.92F6E63C22@lists.varnish-cache.org> commit 746384b20cbc24ff8afd2df35e1510087404fbf4 Author: Nils Goroll Date: Mon Feb 17 18:02:19 2020 +0100 demo how a priv_task can veto being rolled back or rather: fail the task if being rolled back diff --git a/bin/varnishtest/tests/m00017.vtc b/bin/varnishtest/tests/m00017.vtc index c31322d50..0f1ddba47 100644 --- a/bin/varnishtest/tests/m00017.vtc +++ b/bin/varnishtest/tests/m00017.vtc @@ -204,6 +204,7 @@ server s1 -wait # Dont panic varnish v1 -vcl+backend { import std; + import debug; import vtc; sub vcl_recv { @@ -211,6 +212,12 @@ varnish v1 -vcl+backend { } sub vcl_deliver { + if (req.url ~ "/veto") { + debug.fail_rollback(); + } + if (req.url ~ "/ok") { + debug.ok_rollback(); + } std.rollback(req); set resp.http.test = req.http.test; vtc.workspace_alloc(client, -200); @@ -218,9 +225,9 @@ varnish v1 -vcl+backend { } # XXX server keeps params, so we need to reset previous -repeat 5 -server s1 -repeat 1 { +server s1 -repeat 3 { rxreq - expect req.url == "/6" + expect req.url ~ "^/6" txresp } -start @@ -229,6 +236,13 @@ client c4 { rxresp expect resp.status == 200 expect resp.http.test == "1" + txreq -url /6/veto/ok -hdr "test: 1" + rxresp + expect resp.status == 200 + expect resp.http.test == "1" + txreq -url /6/veto -hdr "test: 1" + rxresp + expect resp.status == 503 } -run server s1 -wait @@ -251,7 +265,7 @@ varnish v1 -vcl+backend { } } -server s1 { +server s1 -repeat 1 { rxreq expect req.url == "/7" txresp diff --git a/lib/libvmod_debug/vmod.vcc b/lib/libvmod_debug/vmod.vcc index 35ce24631..5fe13d252 100644 --- a/lib/libvmod_debug/vmod.vcc +++ b/lib/libvmod_debug/vmod.vcc @@ -292,3 +292,11 @@ Get the stringified client ip from the session attr $Function STRING client_port() Get the stringified client port from the session attr + +$Function VOID fail_rollback() + +fail any rollback before ok_rollback() is called + +$Function VOID ok_rollback() + +Allow rollbacks. Must be called before the end of the task. diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c index 9f46a2436..ddd3b1f1d 100644 --- a/lib/libvmod_debug/vmod_debug.c +++ b/lib/libvmod_debug/vmod_debug.c @@ -1085,3 +1085,52 @@ xyzzy_client_port(VRT_CTX) return (SES_Get_String_Attr(ctx->sp, SA_CLIENT_PORT)); } + +static void +fail_f(void *priv) +{ + VRT_CTX; + + CAST_OBJ_NOTNULL(ctx, priv, VRT_CTX_MAGIC); + VRT_fail(ctx, "thou shalt not rollet back"); +} + +VCL_VOID v_matchproto_(td_xyzzy_debug_fail_rollback) +xyzzy_fail_rollback(VRT_CTX) +{ + struct vmod_priv *p; + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + + p = VRT_priv_task(ctx, (void *)xyzzy_fail_rollback); + if (p == NULL) { + VRT_fail(ctx, "no priv task - out of ws?"); + return; + } + + if (p->priv != NULL) { + assert(p->priv == ctx); + assert(p->free == fail_f); + return; + } + + p->priv = TRUST_ME(ctx); + p->free = fail_f; +} + +VCL_VOID v_matchproto_(td_xyzzy_debug_ok_rollback) +xyzzy_ok_rollback(VRT_CTX) +{ + struct vmod_priv *p; + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + + p = VRT_priv_task(ctx, (void *)xyzzy_fail_rollback); + if (p == NULL) { + VRT_fail(ctx, "no priv task - out of ws?"); + return; + } + + p->priv = NULL; + p->free = NULL; +} From dridi.boukelmoune at gmail.com Mon Feb 17 17:59:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 17 Feb 2020 17:59:07 +0000 (UTC) Subject: [master] f64a54e37 Import vmod_cookie from varnish/varnish-modules Message-ID: <20200217175907.ABC1E650DA@lists.varnish-cache.org> commit f64a54e375fb8692d23bb8f6c8c191623fbbb443 Author: Dridi Boukelmoune Date: Mon Feb 17 18:50:40 2020 +0100 Import vmod_cookie from varnish/varnish-modules This is in essence the same VMOD so migrating from one to the other should result in no VCL change, except for the removal of DEPRECATED cookie.filter_except(). The test suite was refreshed to take advantage of recent features and mostly operates in vcl_synth as a result, not wasting cycles bringing needless backends up. The C code and VCC descriptor were also improved for better code style compliance and documentation consistency, without changing the VMOD's behavior. On the C side actual changes mostly consisted in adding missing assertions but there were notable changes: - usage of VRE_Free instead of free (with test case) - WS_VSB_* facility usage for the one use case There are other opportunities for improvement, like usage of more modern facilities like VRT_fail, or a named Lck instead of a plain mutex. Closes #3184 diff --git a/configure.ac b/configure.ac index 61c19dbcc..87fe4b11f 100644 --- a/configure.ac +++ b/configure.ac @@ -853,6 +853,7 @@ AC_CONFIG_FILES([ lib/libvarnishapi/Makefile lib/libvcc/Makefile lib/libvgz/Makefile + lib/libvmod_cookie/Makefile lib/libvmod_debug/Makefile lib/libvmod_std/Makefile lib/libvmod_directors/Makefile diff --git a/lib/Makefile.am b/lib/Makefile.am index 907ee4a5e..1cd630954 100644 --- a/lib/Makefile.am +++ b/lib/Makefile.am @@ -5,6 +5,7 @@ SUBDIRS = \ libvarnishapi \ libvcc \ libvgz \ + libvmod_cookie \ libvmod_debug \ libvmod_std \ libvmod_directors \ diff --git a/lib/libvmod_cookie/Makefile.am b/lib/libvmod_cookie/Makefile.am new file mode 100644 index 000000000..e9ed0cf2e --- /dev/null +++ b/lib/libvmod_cookie/Makefile.am @@ -0,0 +1,5 @@ +libvmod_cookie_la_SOURCES = \ + vmod_cookie.c + +# Use vmodtool.py generated automake boilerplate +include $(srcdir)/automake_boilerplate.am diff --git a/lib/libvmod_cookie/automake_boilerplate.am b/lib/libvmod_cookie/automake_boilerplate.am new file mode 100644 index 000000000..ded993389 --- /dev/null +++ b/lib/libvmod_cookie/automake_boilerplate.am @@ -0,0 +1,64 @@ +# Generated by vmodtool.py --boilerplate. + +AM_LDFLAGS = $(AM_LT_LDFLAGS) + +AM_CPPFLAGS = \ + -I$(top_srcdir)/include \ + -I$(top_srcdir)/bin/varnishd \ + -I$(top_builddir)/include + +vmoddir = $(pkglibdir)/vmods +vmodtool = $(top_srcdir)/lib/libvcc/vmodtool.py +vmodtoolargs ?= --strict --boilerplate + +vmod_LTLIBRARIES = libvmod_cookie.la + +libvmod_cookie_la_CFLAGS = \ + @SAN_CFLAGS@ + +libvmod_cookie_la_LDFLAGS = \ + -export-symbols-regex 'Vmod_cookie_Data' \ + $(AM_LDFLAGS) \ + $(VMOD_LDFLAGS) \ + @SAN_LDFLAGS@ + +nodist_libvmod_cookie_la_SOURCES = vcc_if.c vcc_if.h + +$(libvmod_cookie_la_OBJECTS): vcc_if.h + +vcc_if.h vmod_cookie.rst vmod_cookie.man.rst: vcc_if.c + +vcc_if.c: $(vmodtool) $(srcdir)/vmod.vcc + @PYTHON@ $(vmodtool) $(vmodtoolargs) $(srcdir)/vmod.vcc + +EXTRA_DIST = vmod.vcc automake_boilerplate.am + +CLEANFILES = $(builddir)/vcc_if.c $(builddir)/vcc_if.h \ + $(builddir)/vmod_cookie.rst \ + $(builddir)/vmod_cookie.man.rst + +TESTS = \ + tests/cookie_b00000.vtc \ + tests/cookie_b00001.vtc \ + tests/cookie_b00002.vtc \ + tests/cookie_b00003.vtc \ + tests/cookie_b00004.vtc \ + tests/cookie_b00005.vtc \ + tests/cookie_b00006.vtc \ + tests/cookie_b00007.vtc \ + tests/cookie_b00008.vtc \ + tests/cookie_b00009.vtc \ + tests/cookie_b00010.vtc \ + tests/cookie_b00011.vtc \ + tests/cookie_b00012.vtc \ + tests/cookie_b00013.vtc \ + tests/cookie_r00028.vtc \ + tests/cookie_v00000.vtc + +EXTRA_DIST += $(TESTS) + +vtc-refresh-tests: + @PYTHON@ $(vmodtool) $(vmodtoolargs) $(srcdir)/vmod.vcc + @cd $(top_builddir) && ./config.status --file=$(subdir)/Makefile + +include $(top_srcdir)/vtc.am diff --git a/lib/libvmod_cookie/tests/cookie_b00000.vtc b/lib/libvmod_cookie/tests/cookie_b00000.vtc new file mode 100644 index 000000000..a83633cfd --- /dev/null +++ b/lib/libvmod_cookie/tests/cookie_b00000.vtc @@ -0,0 +1,22 @@ +varnishtest "Test vmod_cookie" + +varnish v1 -vcl { + import cookie; + backend be none; + sub vcl_recv { return (synth(200)); } + sub vcl_synth { + cookie.set("cookie1", "cookie1BAD"); + cookie.set("cookie2", "cookie2value"); + cookie.set("cookie3", "cookie3value"); + cookie.set("cookie4", "cookie4value"); + cookie.set("cookie1", "cookie1value"); # overrides cookie1 + cookie.delete("cookie2"); + set resp.http.X-foo = cookie.get_string(); + } +} -start + +client c1 { + txreq -url "/" + rxresp + expect resp.http.X-foo == "cookie1=cookie1value; cookie3=cookie3value; cookie4=cookie4value;" +} -run diff --git a/lib/libvmod_cookie/tests/cookie_b00001.vtc b/lib/libvmod_cookie/tests/cookie_b00001.vtc new file mode 100644 index 000000000..9b45f740e --- /dev/null +++ b/lib/libvmod_cookie/tests/cookie_b00001.vtc @@ -0,0 +1,21 @@ +varnishtest "Test cookie.clean()" + +varnish v1 -vcl { + import cookie; + backend be none; + sub vcl_recv { return (synth(200)); } + sub vcl_synth { + cookie.clean(); + cookie.set("cookie1", "cookie1BAD"); + set resp.http.X-foo = cookie.get_string(); + cookie.clean(); + set resp.http.X-bar = cookie.get_string(); + } +} -start + +client c1 { + txreq + rxresp + expect resp.http.X-foo == "cookie1=cookie1BAD;" + expect resp.http.X-bar == "" +} -run diff --git a/lib/libvmod_cookie/tests/cookie_b00002.vtc b/lib/libvmod_cookie/tests/cookie_b00002.vtc new file mode 100644 index 000000000..d440f841d --- /dev/null +++ b/lib/libvmod_cookie/tests/cookie_b00002.vtc @@ -0,0 +1,37 @@ +varnishtest "NULL/empty value checks" + +varnish v1 -vcl { + import cookie; + backend be none; + sub vcl_recv { return (synth(200)); } + sub vcl_synth { + # nothing in here at this point. + set req.http.x-foo = cookie.get_string(); + + # XXX: We might want to revisit these assumptions since "=" + # (empty name and empty value) is as correct as "name=value" + # for a cookie. See rfc6265 section 5.2 for reference. + + # empty name + cookie.set("", "foo"); + + # empty value + cookie.set("cookie1", ""); + + # proper NULL + cookie.set(req.http.null, "foo"); + + # double delete + cookie.delete("cookie2"); + cookie.delete("cookie2"); + + cookie.delete(req.http.null); + set resp.http.x-foo = cookie.get_string(); + } +} -start + +client c1 { + txreq -url "/" + rxresp + expect resp.http.x-foo == "" +} -run diff --git a/lib/libvmod_cookie/tests/cookie_b00003.vtc b/lib/libvmod_cookie/tests/cookie_b00003.vtc new file mode 100644 index 000000000..af9653035 --- /dev/null +++ b/lib/libvmod_cookie/tests/cookie_b00003.vtc @@ -0,0 +1,31 @@ +varnishtest "Test cookie.keep()" + +varnish v1 -vcl { + import cookie; + backend be none; + sub vcl_recv { return (synth(200)); } + sub vcl_synth { + cookie.set("biscuit", "standard"); + cookie.set("bredela", "eggwhites"); + cookie.set("chocolatechip", "verychippy"); + cookie.set("empire", "jellytots"); + cookie.keep("bredela,empire,baz"); + set resp.http.X-foo = cookie.get_string(); + + # Test exotic admin-supplied filter strings. + cookie.parse("bredela=eggwhites; empire=jellytots;"); + cookie.keep(",,,,bredela, ,empire,baz,"); + set resp.http.X-bar = cookie.get_string(); + + cookie.keep(req.http.none); + set resp.http.X-baz = cookie.get_string(); + } +} -start + +client c1 { + txreq -url "/" + rxresp + expect resp.http.X-foo == "bredela=eggwhites; empire=jellytots;" + expect resp.http.X-bar == "bredela=eggwhites; empire=jellytots;" + expect resp.http.X-baz == "" +} -run diff --git a/lib/libvmod_cookie/tests/cookie_b00004.vtc b/lib/libvmod_cookie/tests/cookie_b00004.vtc new file mode 100644 index 000000000..d996fbee2 --- /dev/null +++ b/lib/libvmod_cookie/tests/cookie_b00004.vtc @@ -0,0 +1,32 @@ +varnishtest "Test cookie.filter()" + +varnish v1 -vcl { + import cookie; + backend be none; + sub vcl_recv { return (synth(200)); } + sub vcl_synth { + cookie.set("biscuit", "standard"); + cookie.set("bredela", "eggwhites"); + cookie.set("chocolatechip", "verychippy"); + cookie.set("empire", "jellytots"); + cookie.filter("bredela,empire,baz"); + set resp.http.X-foo = cookie.get_string(); + + # Test exotic admin-supplied filter strings. + cookie.parse("bredela=eggwhites; empire=jellytots;"); + cookie.filter(",,,,bredela, ,baz,"); + set resp.http.X-bar = cookie.get_string(); + + cookie.parse("foo=bar; bar=baz;"); + cookie.filter(req.http.none); + set resp.http.X-baz = cookie.get_string(); + } +} -start + +client c1 { + txreq -url "/" + rxresp + expect resp.http.X-foo == "biscuit=standard; chocolatechip=verychippy;" + expect resp.http.X-bar == "empire=jellytots;" + expect resp.http.X-baz == "foo=bar; bar=baz;" +} -run diff --git a/lib/libvmod_cookie/tests/cookie_b00005.vtc b/lib/libvmod_cookie/tests/cookie_b00005.vtc new file mode 100644 index 000000000..7d8c4b7cd --- /dev/null +++ b/lib/libvmod_cookie/tests/cookie_b00005.vtc @@ -0,0 +1,21 @@ +varnishtest "Test missing cookie" + +varnish v1 -vcl { + import cookie; + backend be none; + sub vcl_recv { return (synth(200)); } + sub vcl_synth { + cookie.set("cookie1", "cookie1value"); + cookie.set("cookie2", "cookie2value"); + set resp.http.X-foo = cookie.get("cookie2"); + # Make sure we handle this gracefully. + set resp.http.X-bar = "" + cookie.get("non-existing"); + } +} -start + +client c1 { + txreq -url "/" + rxresp + expect resp.http.X-foo == "cookie2value" + expect resp.http.X-bar == "" +} -run diff --git a/lib/libvmod_cookie/tests/cookie_b00006.vtc b/lib/libvmod_cookie/tests/cookie_b00006.vtc new file mode 100644 index 000000000..f171b5e20 --- /dev/null +++ b/lib/libvmod_cookie/tests/cookie_b00006.vtc @@ -0,0 +1,50 @@ +varnishtest "Test vmod_cookie concurrency" + +barrier b1 cond 2 + +server s1 { + rxreq + barrier b1 sync + expect req.url == "/s1" + txresp +} -start + +server s2 { + rxreq + expect req.url == "/s2" + txresp +} -start + +varnish v1 -vcl+backend { + import cookie; + + sub vcl_recv { + cookie.parse(req.http.cookie); + + if (req.url == "/s1") { + set req.backend_hint = s1; + } + else { + set req.backend_hint = s2; + } + } + + sub vcl_deliver { + set resp.http.x-val = cookie.get("a"); + } +} -start + +client c1 { + txreq -url "/s1" -hdr "Cookie: a=bar" + rxresp + expect resp.http.x-val == "bar" +} -start + +client c2 { + barrier b1 sync + txreq -url "/s2" -hdr "Cookie: a=foo" + rxresp + expect resp.http.x-val == "foo" +} -run + +client c1 -wait diff --git a/lib/libvmod_cookie/tests/cookie_b00007.vtc b/lib/libvmod_cookie/tests/cookie_b00007.vtc new file mode 100644 index 000000000..418e91120 --- /dev/null +++ b/lib/libvmod_cookie/tests/cookie_b00007.vtc @@ -0,0 +1,23 @@ +varnishtest "Test cookie.isset()" + +varnish v1 -vcl { + import cookie; + backend be none; + sub vcl_recv { return (synth(200)); } + sub vcl_synth { + cookie.set("cookie1", "cookie1value"); + set resp.http.does = cookie.isset("cookie1"); + set resp.http.does-not = cookie.isset("non-existent"); + set resp.http.null = cookie.isset(""); + set resp.http.null2 = cookie.isset(req.http.probably-null); + } +} -start + +client c1 { + txreq -url "/" + rxresp + expect resp.http.does == "true" + expect resp.http.does-not == "false" + expect resp.http.null == "false" + expect resp.http.null2 == "false" +} -run diff --git a/lib/libvmod_cookie/tests/cookie_b00008.vtc b/lib/libvmod_cookie/tests/cookie_b00008.vtc new file mode 100644 index 000000000..af1bfcae9 --- /dev/null +++ b/lib/libvmod_cookie/tests/cookie_b00008.vtc @@ -0,0 +1,25 @@ +varnishtest "Test large cookies" + +varnish v1 -cliok "param.set workspace_client 64k" -vcl { + import cookie; + backend be none; + sub vcl_recv { return (synth(200)); } + sub vcl_synth { + cookie.parse(req.http.cookie); + set resp.http.cookiestring = cookie.get_string(); + set resp.http.cookie1 = cookie.isset("cookie1"); + } +} -start + +client c1 { + # Insanely long cookie name. + txreq -url "/" -hdr "Cookie: phohx8aingie6Ide7peephie5paip6ang4thooh4ooquai8ohvah7eiqueeki8ooth7viequ0Tha5thewiSheih5jaimaiTahr1wi8WooQuoe7loothieThahweeneichoo8cufeelu3tie5cei1iShiemiezoofox6ahcaevaihocheungai2aeghaichaingee0EiGie3Ees5ujaem5uquahpieFeelei7Ohngei1afaTooph4aiquum1aewaidatheshuh1fohhoor0hoo6aeTeiy9xougahf3jeapooshuhoob5deiwareingahth7ahf2fafeer8Oobiewai3rei8ofejohjeiye4die8Na7ze6eixajauCairoth0lek8vioyuom6eih0egho2aingoo7coh1at3niochu6osahthi0ue1Luchae1eifeupiuwaa0raidiewaijese4oozee4eihie5shaBaoreacooNg8uW9eru9kigh3Feesi8iex2pu7ohfaiBiezael6ifaujiek4nutae1aalohchoteiPuaM2chiefaicaopheKohsh6Ho1wiephieseef1daj3Pahfie2ooch8shaing5baXeiLiep9lahfe9uDaxeehielais2eix3iekiew8aiter9Foo8noo2hae7ohdie1iB7hoop3podeengooSothoojui4AhXu5Nain8ohqu8if1ue5iTheimei5oghie9sheiv4Hejah1veixahcaixie8ahyieT8Phay4daeTei1aRiemae6oicheef2miiNuoxeil1kae2nea1roh9Rei1keiwaT2eoJaiNgie0den6aideif3uechaishaec4cai2eozieb9aeN9sai9ahnielohdaeGh2kaeleiteitai0ietoo7eiCha0baiW7dai0im1jul5OWijaLo2ohh3kooxu2oFah3loob6feiw7pie9eighu8ik4chae0Athou2fah5ieQuuic0Mu1j=cookievalue" + rxresp + expect resp.http.cookiestring == "phohx8aingie6Ide7peephie5paip6ang4thooh4ooquai8ohvah7eiqueeki8ooth7viequ0Tha5thewiSheih5jaimaiTahr1wi8WooQuoe7loothieThahweeneichoo8cufeelu3tie5cei1iShiemiezoofox6ahcaevaihocheungai2aeghaichaingee0EiGie3Ees5ujaem5uquahpieFeelei7Ohngei1afaTooph4aiquum1aewaidatheshuh1fohhoor0hoo6aeTeiy9xougahf3jeapooshuhoob5deiwareingahth7ahf2fafeer8Oobiewai3rei8ofejohjeiye4die8Na7ze6eixajauCairoth0lek8vioyuom6eih0egho2aingoo7coh1at3niochu6osahthi0ue1Luchae1eifeupiuwaa0raidiewaijese4oozee4eihie5shaBaoreacooNg8uW9eru9kigh3Feesi8iex2pu7ohfaiBiezael6ifaujiek4nutae1aalohchoteiPuaM2chiefaicaopheKohsh6Ho1wiephieseef1daj3Pahfie2ooch8shaing5baXeiLiep9lahfe9uDaxeehielais2eix3iekiew8aiter9Foo8noo2hae7ohdie1iB7hoop3podeengooSothoojui4AhXu5Nain8ohqu8if1ue5iTheimei5oghie9sheiv4Hejah1veixahcaixie8ahyieT8Phay4daeTei1aRiemae6oicheef2miiNuoxeil1kae2nea1roh9Rei1keiwaT2eoJaiNgie0den6aideif3uechaishaec4cai2eozieb9aeN9sai9ahnielohdaeGh2kaeleiteitai0ietoo7eiCha0baiW7dai0im1jul5OWijaLo2ohh3kooxu2oFah3loob6feiw7pie9eighu8ik4chae0Athou2fah5ieQuuic0Mu1j=cookievalue;" + + # Insane 6KB cookie value. + txreq -url "/" -hdr "Cookie: cookie1=foobarbazfoobarbazphohx8aingie6Ide7peephie5paip6ang4thooh4ooquai8ohvah7eiqueeki8ooth7viequ0Tha5thewiSheih5jaimaiTahr1wi8WooQuoe7loothieThahweeneichoo8cufeelu3tie5cei1iShiemiezoofox6ahcaevaihocheungai2aeghaichaingee0EiGie3Ees5ujaem5uquahpieFeelei7Ohngei1afaTooph4aiquum1aewaidatheshuh1fohhoor0hoo6aeTeiy9xougahf3jeapooshuhoob5deiwareingahth7ahf2fafeer8Oobiewai3rei8ofejohjeiye4die8Na7ze6eixajauCairoth0lek8vioyuom6eih0egho2aingoo7coh1at3niochu6osahthi0ue1Luchae1eifeupiuwaa0raidiewaijese4oozee4eihie5shaBaoreacooNg8uW9eru9kigh3Feesi8iex2pu7ohfaiBiezael6ifaujiek4nutae1aalohchoteiPuaM2chiefaicaopheKohsh6Ho1wiephieseef1daj3Pahfie2ooch8shaing5baXeiLiep9lahfe9uDaxeehielais2eix3iekiew8aiter9Foo8noo2hae7ohdie1iB7hoop3podeengooSothoojui4AhXu5Nain8ohqu8if1ue5iTheimei5oghie9sheiv4Hejah1veixahcaixie8ahyieT8Phay4daeTei1aRiemae6oicheef2miiNuoxeil1kae2nea1roh9Rei1keiwaT2eoJaiNgie0den6aideif3uechaishaec4cai2eozieb9aeN9sai9ahnielohdaeGh2kaeleiteitai0ietoo7eiCha0baiW7dai0im1jul5OWijaLo2ohh3kooxu2oFah3loob6feiw7pie9eighu8ik4chae0Athou2fah5ieQuuic0Mu1jphohx8aingie6Ide7peephie5paip6ang4thooh4ooquai8ohvah7eiqueeki8ooth7viequ0Tha5thewiSheih5jaimaiTahr1wi8WooQuoe7loothieThahweeneichoo8cufeelu3tie5cei1iShiemiezoofox6ahcaevaihocheungai2aeghaichaingee0EiGie3Ees5ujaem5uquahpieFeelei7Ohngei1afaTooph4aiquum1aewaidatheshuh1fohhoor0hoo6aeTeiy9xougahf3jeapooshuhoob5deiwareingahth7ahf2fafeer8Oobiewai3rei8ofejohjeiye4die8Na7ze6eixajauCairoth0lek8vioyuom6eih0egho2aingoo7coh1at3niochu6osahthi0ue1Luchae1eifeupiuwaa0raidiewaijese4oozee4eihie5shaBaoreacooNg8uW9eru9kigh3Feesi8iex2pu7ohfaiBiezael6ifaujiek4nutae1aalohchoteiPuaM2chiefaicaopheKohsh6Ho1wiephieseef1daj3Pahfie2ooch8shaing5baXeiLiep9lahfe9uDaxeehielais2eix3iekiew8aiter9Foo8noo2hae7ohdie1iB7hoop3podeengooSothoojui4AhXu5Nain8ohqu8if1ue5iTheimei5oghie9sheiv4Hejah1veixahcaixie8ahyieT8Phay4daeTei1aRiemae6oicheef2miiNuoxeil1kae2nea1roh9Rei1keiwaT2eoJaiNgie0den6aideif3uechaishaec4cai2eozieb9aeN9sai9ahnielohdaeGh2kaeleiteitai0ietoo7eiCha0baiW7dai0im1jul5OWijaLo2ohh3kooxu2oFah3loob6feiw7pie9eighu8ik4chae0Athou2fah5ieQuuic0Mu1jphohx8aingie6Ide7peephie5paip6ang4thooh4ooquai8ohvah7eiqueeki8ooth7viequ0Tha5thewiSheih5jaimaiTahr1wi8WooQuoe7loothieThahweeneichoo8cufeelu3tie5cei1iShiemiezoofox6ahcaevaihocheungai2aeghaichaingee0EiGie3Ees5ujaem5uquahpieFeelei7Ohngei1afaTooph4aiquum1aewaidatheshuh1fohhoor0hoo6aeTeiy9xougahf3jeapooshuhoob5deiwareingahth7ahf2fafeer8Oobiewai3rei8ofejohjeiye4die8Na7ze6eixajauCairoth0lek8vioyuom6eih0egho2aingoo7coh1at3niochu6osahthi0ue1Luchae1eifeupiuwaa0raidiewaijese4oozee4eihie5shaBaoreacooNg8uW9eru9kigh3Feesi8iex2pu7ohfaiBiezael6ifaujiek4nutae1aalohchoteiPuaM2chiefaicaopheKohsh6Ho1wiephieseef1daj3Pahfie2ooch8shaing5baXeiLiep9lahfe9uDaxeehielais2eix3iekiew8aiter9Foo8noo2hae7ohdie1iB7hoop3podeengooSothoojui4AhXu5Nain8ohqu8if1ue5iTheimei5oghie9sheiv4Hejah1veixahcaixie8ahyieT8Phay4daeTei1aRiemae6oicheef2miiNuoxeil1kae2nea1roh9Rei1keiwaT2eoJaiNgie0den6aideif3uechaishaec4cai2eozieb9aeN9sai9ahnielohdaeGh2kaeleiteitai0ietoo7eiCha0baiW7dai0im1jul5OWijaLo2ohh3kooxu2oFah3loob6feiw7pie9eighu8ik4chae0Athou2fah5ieQuuic0Mu1jphohx8aingie6Ide7peephie5paip6ang4thooh4ooquai8ohvah7eiqueeki8ooth7viequ0Tha5thewiSheih5jaimaiTahr1wi8WooQuoe7loothieThahweeneichoo8cufeelu3tie5cei1iShiemiezoofox6ahcaevaihocheungai2aeghaichaingee0EiGie3Ees5ujaem5uquahpieFeelei7Ohngei1afaTooph4aiquum1aewaidatheshuh1fohhoor0hoo6aeTeiy9xougahf3jeapooshuhoob5deiwareingahth7ahf2fafeer8Oobiewai3rei8ofejohjeiye4die8Na7ze6eixajauCairoth0lek8vioyuom6eih0egho2aingoo7coh1at3niochu6osahthi0ue1Luchae1eifeupiuwaa0raidiewaijese4oozee4eihie5shaBaoreacooNg8uW9eru9kigh3Feesi8iex2pu7ohfaiBiezael6ifaujiek4nutae1aalohchoteiPuaM2chiefaicaopheKohsh6Ho1wiephieseef1daj3Pahfie2ooch8shaing5baXeiLiep9lahfe9uDaxeehielais2eix3iekiew8aiter9Foo8noo2hae7ohdie1iB7hoop3podeengooSothoojui4AhXu5Nain8ohqu8if1ue5iTheimei5oghie9sheiv4Hejah1veixahcaixie8ahyieT8Phay4daeTei1aRiemae6oicheef2miiNuoxeil1kae2nea1roh9Rei1keiwaT2eoJaiNgie0den6aideif3uechaishaec4cai2eozieb9aeN9sai9ahnielohdaeGh2kaeleiteitai0ietoo7eiCha0baiW7dai0im1jul5OWijaLo2ohh3kooxu2oFah3loob6feiw7pie9eighu8ik4chae0Athou2fah5ieQuuic0Mu1jphohx8aingie6Ide7peephie5paip6ang4thooh4ooquai8ohvah7eiqueeki8ooth7viequ0Tha5thewiSheih5jaimaiTahr1wi8WooQuoe7loothieThahweeneichoo8cufeelu3tie5cei1iShiemiezoofox6ahcaevaihocheungai2aeghaichaingee0EiGie3Ees5ujaem5uquahpieFeelei7Ohngei1afaTooph4aiquum1aewaidatheshuh1fohhoor0hoo6aeTeiy9xougahf3jeapooshuhoob5deiwareingahth7ahf2fafeer8Oobiewai3rei8ofejohjeiye4die8Na7ze6eixajauCairoth0lek8vioyuom6eih0egho2aingoo7coh1at3niochu6osahthi0ue1Luchae1eifeupiuwaa0raidiewaijese4oozee4eihie5shaBaoreacooNg8uW9eru9kigh3Feesi8iex2pu7ohfaiBiezael6ifaujiek4nutae1aalohchoteiPuaM2chiefaicaopheKohsh6Ho1wiephieseef1daj3Pahfie2ooch8shaing5baXeiLiep9lahfe9uDaxeehielais2eix3iekiew8aiter9Foo8noo2hae7ohdie1iB7hoop3podeengooSothoojui4AhXu5Nain8ohqu8if1ue5iTheimei5oghie9sheiv4Hejah1veixahcaixie8ahyieT8Phay4daeTei1aRiemae6oicheef2miiNuoxeil1kae2nea1roh9Rei1keiwaT2eoJaiNgie0den6aideif3uechaishaec4cai2eozieb9aeN9sai9ahnielohdaeGh2kaeleiteitai0ietoo7eiCha0baiW7dai0im1jul5OWijaLo2ohh3kooxu2oFah3loob6feiw7pie9eighu8ik4chae0Athou2fah5ieQuuic0Mu1jphohx8aingie6Ide7peephie5paip6ang4thooh4ooquai8ohvah7eiqueeki8ooth7viequ0Tha5thewiSheih5jaimaiTahr1wi8WooQuoe7loothieThahweeneichoo8cufeelu3tie5cei1iShiemiezoofox6ahcaevaihocheungai2aeghaichaingee0EiGie3Ees5ujaem5uquahpieFeelei7Ohngei1afaTooph4aiquum1aewaidatheshuh1fohhoor0hoo6aeTeiy9xougahf3jeapooshuhoob5deiwareingahth7ahf2fafeer8Oobiewai3rei8ofejohjeiye4die8Na7ze6eixajauCairoth0lek8vioyuom6eih0egho2aingoo7coh1at3niochu6osahthi0ue1Luchae1eifeupiuwaa0raidiewaijese4oozee4eihie5shaBaoreacooNg8uW9eru9kigh3Feesi8iex2pu7ohfaiBiezael6ifaujiek4nutae1aalohchoteiPuaM2chiefaicaopheKohsh6Ho1wiephieseef1daj3Pahfie2ooch8shaing5baXeiLiep9lahfe9uDaxeehielais2eix3iekiew8aiter9Foo8noo2hae7ohdie1iB7hoop3podeengooSothoojui4AhXu5Nain8ohqu8if1ue5iTheimei5oghie9sheiv4Hejah1veixahcaixie8ahyieT8Phay4daeTei1aRiemae6oicheef2miiNuoxeil1kae2nea1roh9Rei1keiwaT2eoJaiNgie0den6aideif3uechaishaec4cai2eozieb9aeN9sai9ahnielohdaeGh2kaeleiteitai0ietoo7eiCha0baiW7dai0im1jul5OWijaLo2ohh3kooxu2oFah3loob6feiw7pie9eighu8ik4chae0Athou2fah5ieQuuic0Mu1j;" + rxresp + # We support long cookie values, should be fine. + expect resp.http.cookie1 == "true" +} -run diff --git a/lib/libvmod_cookie/tests/cookie_b00009.vtc b/lib/libvmod_cookie/tests/cookie_b00009.vtc new file mode 100644 index 000000000..de7a8d731 --- /dev/null +++ b/lib/libvmod_cookie/tests/cookie_b00009.vtc @@ -0,0 +1,51 @@ +varnishtest "Test cookie parser" + +varnish v1 -vcl { + import cookie; + backend be none; + sub vcl_recv { return (synth(200)); } + sub vcl_synth { + cookie.parse(req.http.cookie); + set resp.http.X-foo = cookie.get_string(); + } + +} -start + +client c1 { + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" +} -run + +client c2 { + txreq -hdr "Cookie: __utmc=253898641; __utma=253898641.654622101.1372224466.1372224466.1372224466.1; __utmb=253898641.44.10.1372224466; __utmz=253898641.1372224466.1.1.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=index%2Bof%2Bccnp%2Bpdf" + rxresp + expect resp.http.X-foo == "__utmc=253898641; __utma=253898641.654622101.1372224466.1372224466.1372224466.1; __utmb=253898641.44.10.1372224466; __utmz=253898641.1372224466.1.1.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=index%2Bof%2Bccnp%2Bpdf;" +} -run + +client c3 { + txreq -hdr "Cookie: " + rxresp + expect resp.http.X-foo == "" +} -run + +# An empty cookie is a non-existing cookie for us. +client c4 { + txreq -hdr "Cookie: emptycookie=" + rxresp + expect resp.http.X-foo == "" +} -run + +# A single cookie should also work. +client c5 { + txreq -hdr "Cookie: cookie1=foobarbaz" + rxresp + expect resp.http.X-foo == "cookie1=foobarbaz;" +} -run + +# Don't overflow the buffer with an edge case +client c6 { + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128a;=" -hdr "X-Not-Cookie: sessionid=a707505310ddf259bb290d3ca63fc561" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128a;" +} -run diff --git a/lib/libvmod_cookie/tests/cookie_b00010.vtc b/lib/libvmod_cookie/tests/cookie_b00010.vtc new file mode 100644 index 000000000..fc8cbdd35 --- /dev/null +++ b/lib/libvmod_cookie/tests/cookie_b00010.vtc @@ -0,0 +1,17 @@ +varnishtest "Test rfc1123 string formatting function" + +varnish v1 -vcl { + import cookie; + backend be none; + sub vcl_recv { return (synth(200)); } + sub vcl_synth { + set resp.http.x-date = cookie.format_rfc1123(now, 1d); + } +} -start + + +client c1 { + txreq -url "/" + rxresp + expect resp.http.date != +} -run diff --git a/lib/libvmod_cookie/tests/cookie_b00011.vtc b/lib/libvmod_cookie/tests/cookie_b00011.vtc new file mode 100644 index 000000000..f9d394b3f --- /dev/null +++ b/lib/libvmod_cookie/tests/cookie_b00011.vtc @@ -0,0 +1,24 @@ +varnishtest "Test cookie.get_re()" + +varnish v1 -vcl { + import cookie; + backend be none; + sub vcl_recv { return (synth(200)); } + sub vcl_synth { + cookie.set("biscuit", "standard"); + cookie.set("chocolatechip", "verychippy"); + cookie.set("empire", "jellytots"); + + set resp.http.X-first = cookie.get_re("DOES_NOT_EXIST"); + set resp.http.X-second = cookie.get_re("biscuit"); + set resp.http.X-third = cookie.get_re("DOES_NOT_EXIST_EITHER"); + } +} -start + +client c1 { + txreq -url "/" + rxresp + expect resp.http.X-first == "" + expect resp.http.X-second == "standard" + expect resp.http.X-third == "" +} -run diff --git a/lib/libvmod_cookie/tests/cookie_b00012.vtc b/lib/libvmod_cookie/tests/cookie_b00012.vtc new file mode 100644 index 000000000..e0584af98 --- /dev/null +++ b/lib/libvmod_cookie/tests/cookie_b00012.vtc @@ -0,0 +1,33 @@ +varnishtest "Test cookie.filter_re()" + +varnish v1 -vcl { + import cookie; + backend be none; + sub vcl_recv { return (synth(200)); } + sub vcl_synth { + cookie.set("biscuit", "standard"); + cookie.set("bredela", "eggwhites"); + cookie.set("empire", "jellytots"); + + set resp.http.X-foo = cookie.get_string(); + cookie.filter_re("^NOT-MATCHING-ANYTHING$"); + set resp.http.X-bar = cookie.get_string(); + + cookie.filter_re("^bredela"); + set resp.http.X-baz = cookie.get_string(); + + cookie.filter_re(".*"); + set resp.http.X-qux = cookie.get_string(); + } +} -start + +client c1 { + txreq -url "/" + rxresp + expect resp.http.X-foo == resp.http.X-bar + + expect resp.http.X-baz != resp.http.X-foo + expect resp.http.X-baz == "biscuit=standard; empire=jellytots;" + + expect resp.http.X-qux == "" +} -run diff --git a/lib/libvmod_cookie/tests/cookie_b00013.vtc b/lib/libvmod_cookie/tests/cookie_b00013.vtc new file mode 100644 index 000000000..31624563a --- /dev/null +++ b/lib/libvmod_cookie/tests/cookie_b00013.vtc @@ -0,0 +1,23 @@ +varnishtest "Test cookie.keep_re()" + +varnish v1 -vcl { + import cookie; + backend be none; + sub vcl_recv { return (synth(200)); } + sub vcl_synth { + cookie.parse("foo=bar; baz=qux;"); + cookie.keep_re("NOTHING_MATCHES_SO_NOTHING_KEPT$"); + set resp.http.X-empty = cookie.get_string(); + + cookie.parse("biscuit=standard; bredela=eggwhites; empire=jellytots;"); + cookie.keep_re("^b"); + set resp.http.X-bees = cookie.get_string(); + } +} -start + +client c1 { + txreq -url "/" + rxresp + expect resp.http.X-empty == "" + expect resp.http.X-bees == "biscuit=standard; bredela=eggwhites;" +} -run diff --git a/lib/libvmod_cookie/tests/cookie_r00028.vtc b/lib/libvmod_cookie/tests/cookie_r00028.vtc new file mode 100644 index 000000000..4e606887b --- /dev/null +++ b/lib/libvmod_cookie/tests/cookie_r00028.vtc @@ -0,0 +1,101 @@ +varnishtest "Test issue https://github.com/varnish/varnish-modules/issues/28" + +varnish v1 -vcl { + import cookie; + backend be none; + sub vcl_recv { return (synth(200)); } + sub vcl_synth { + cookie.parse(req.http.cookie); + set resp.http.X-foo = cookie.get_string(); + } +} -start + +client c1 { + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + + +} + +client c2 { + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + + txreq -hdr "Cookie: csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560" + rxresp + expect resp.http.X-foo == "csrftoken=0e0c3616e41a6bd561b72b7f5fc1128f; sessionid=a707505310ddf259bb290d3ca63fc560;" + +} + +client c1 -repeat 2 -run +client c2 -repeat 2 -run diff --git a/lib/libvmod_cookie/vmod.vcc b/lib/libvmod_cookie/vmod.vcc new file mode 100644 index 000000000..3cbdfe261 --- /dev/null +++ b/lib/libvmod_cookie/vmod.vcc @@ -0,0 +1,219 @@ +$Module cookie 3 "Varnish Cookie Module" + +DESCRIPTION +=========== + +Handle HTTP cookies easier in Varnish VCL. + +Parses a cookie header into an internal data store, where per-cookie +get/set/delete functions are available. A filter_except() method removes all +but a set comma-separated list of cookies. A filter() method removes a comma- +separated list of cookies. + +Regular expressions can be used for either selecting cookies, deleting matching +cookies and deleting non-matching cookie names. + +A convenience function for formatting the Set-Cookie Expires date field +is also included. + +The state loaded with cookie.parse() has a lifetime of the current request +or backend request context. To pass variables to the backend request, store +the contents as fake bereq headers. + +Filtering example:: + + import cookie; + + sub vcl_recv { + if (req.http.cookie) { + cookie.parse(req.http.cookie); + # Either delete the ones you want to get rid of: + cookie.delete("cookie2"); + # or delete all but a few: + cookie.keep("SESSIONID,PHPSESSID"); + + # Store it back into req so it will be passed to the backend. + set req.http.cookie = cookie.get_string(); + + # If empty, unset so the builtin VCL can consider it for caching. + if (req.http.cookie == "") { + unset req.http.cookie; + } + } + } + +$ABI strict +$Function VOID clean(PRIV_TASK) + +Clean up previously parsed cookies. It is not necessary to run clean() +in normal operations. + +Example:: + + sub vcl_recv { + cookie.clean(); + } + +$Function VOID delete(PRIV_TASK, STRING cookiename) + +Delete ``cookiename`` from internal vmod storage if it exists. + +Example:: + + sub vcl_recv { + cookie.parse("cookie1: value1; cookie2: value2;"); + cookie.delete("cookie2"); + # get_string() will now yield "cookie1: value1"; + } + +$Function VOID filter(PRIV_TASK, STRING filterstring) + +Delete all cookies from internal vmod storage that are in the +comma-separated argument cookienames. + +Example:: + + sub vcl_recv { + cookie.parse("cookie1: value1; cookie2: value2; cookie3: value3"); + cookie.filter("cookie1,cookie2"); + # get_string() will now yield + # "cookie3: value3"; + } + + +$Function VOID filter_re(PRIV_TASK, PRIV_CALL, STRING expression) + +Delete all cookies from internal vmod storage that matches the +regular expression ``expression``. + +Example:: + + sub vcl_recv { + cookie.parse("cookie1: value1; cookie2: value2; cookie3: value3"); + cookie.filter_re("^cookie[12]$"); + # get_string() will now yield + # "cookie3: value3"; + } + +$Function VOID keep(PRIV_TASK, STRING filterstring) + +Delete all cookies from internal vmod storage that is not in the +comma-separated argument cookienames. + +Example:: + + sub vcl_recv { + cookie.parse("cookie1: value1; cookie2: value2; cookie3: value3"); + cookie.keep("cookie1,cookie2"); + # get_string() will now yield + # "cookie1: value1; cookie2: value2;"; + } + +$Function VOID keep_re(PRIV_TASK, PRIV_CALL, STRING expression) + +Delete all cookies from internal vmod storage that does not match +expression ``expression``. + +Example:: + + sub vcl_recv { + cookie.parse("cookie1: value1; cookie2: value2; cookie3: value3"); + cookie.keep_re("^cookie1,cookie2"); + # get_string() will now yield + # "cookie1: value1; cookie2: value2;"; + } + + + + +$Function STRING format_rfc1123(TIME now, DURATION timedelta) + +Get a RFC1123 formatted date string suitable for inclusion in a +Set-Cookie response header. + +Care should be taken if the response has multiple Set-Cookie headers. +In that case the header vmod should be used. + +Example:: + + sub vcl_deliver { + # Set a userid cookie on the client that lives for 5 minutes. + set resp.http.Set-Cookie = "userid=" + req.http.userid + + "; Expires=" + cookie.format_rfc1123(now, 5m) + "; httpOnly"; + } + +$Function STRING get(PRIV_TASK, STRING cookiename) + +Get the value of ``cookiename``, as stored in internal vmod storage. If +``cookiename`` does not exist an empty string is returned. + +Example:: + + import std; + sub vcl_recv { + cookie.parse("cookie1: value1; cookie2: value2;"); + std.log("cookie1 value is: " + cookie.get("cookie1")); + } + +$Function STRING get_re(PRIV_TASK, PRIV_CALL, STRING expression) + +Get the value of the first cookie in internal vmod storage that matches +regular expression ``expression``. If nothing matches, an empty string +is returned. + +Example:: + + import std; + sub vcl_recv { + cookie.parse("cookie1: value1; cookie2: value2;"); + std.log("cookie1 value is: " + cookie.get_re("^cookie1$")); + } + +$Function STRING get_string(PRIV_TASK) + +Get a Cookie string value with all cookies in internal vmod storage. Does +not modify internal storage. + +Example:: + + sub vcl_recv { + cookie.parse(req.http.cookie); + cookie.filter_except("SESSIONID,PHPSESSID"); + set req.http.cookie = cookie.get_string(); + } + +$Function BOOL isset(PRIV_TASK, STRING cookiename) + +Check if ``cookiename`` is set in the internal vmod storage. + +Example:: + + import std; + sub vcl_recv { + cookie.parse("cookie1: value1; cookie2: value2;"); + if (cookie.isset("cookie2")) { + std.log("cookie2 is set."); + } + } + +$Function VOID parse(PRIV_TASK, STRING cookieheader) + +Parse the cookie string in ``cookieheader``. If state already exists, +``clean()`` will be run first. + +Example:: + + sub vcl_recv { + cookie.parse(req.http.Cookie); + } + +$Function VOID set(PRIV_TASK, STRING cookiename, STRING value) + +Set the internal vmod storage for ``cookiename`` to ``value``. + +Example:: + + sub vcl_recv { + cookie.set("cookie1", "value1"); + std.log("cookie1 value is: " + cookie.get("cookie1")); + } diff --git a/lib/libvmod_cookie/vmod_cookie.c b/lib/libvmod_cookie/vmod_cookie.c new file mode 100644 index 000000000..4187a1fb1 --- /dev/null +++ b/lib/libvmod_cookie/vmod_cookie.c @@ -0,0 +1,522 @@ +/*- + * Copyright (c) 2012-2020 Varnish Software + * + * Author: Lasse Karstensen + * Author: Lasse Karstensen + * Author: Dridi Boukelmoune + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * Cookie VMOD that simplifies handling of the Cookie request header. + */ + +#include "config.h" + +#include +#include +#include +#include +#include + +#include + +#include +#include + +#include "vcc_if.h" + +#define VRE_MAX_GROUPS 8 + +enum filter_action { + blacklist, + whitelist +}; + +static pthread_mutex_t mtx = PTHREAD_MUTEX_INITIALIZER; + +struct cookie { + unsigned magic; +#define VMOD_COOKIE_ENTRY_MAGIC 0x3BB41543 + char *name; + char *value; + VTAILQ_ENTRY(cookie) list; +}; + +/* A structure to represent both whitelists and blacklists */ +struct matchlist { + char *name; + VTAILQ_ENTRY(matchlist) list; +}; + +struct vmod_cookie { + unsigned magic; +#define VMOD_COOKIE_MAGIC 0x4EE5FB2E + VTAILQ_HEAD(, cookie) cookielist; +}; + +static void +cobj_free(void *p) +{ + struct vmod_cookie *vcp; + + CAST_OBJ_NOTNULL(vcp, p, VMOD_COOKIE_MAGIC); + FREE_OBJ(vcp); +} + +static struct vmod_cookie * +cobj_get(struct vmod_priv *priv) +{ + struct vmod_cookie *vcp; + + if (priv->priv == NULL) { + ALLOC_OBJ(vcp, VMOD_COOKIE_MAGIC); + AN(vcp); + VTAILQ_INIT(&vcp->cookielist); + priv->priv = vcp; + priv->free = cobj_free; + } else + CAST_OBJ_NOTNULL(vcp, priv->priv, VMOD_COOKIE_MAGIC); + + return (vcp); +} + +VCL_VOID +vmod_parse(VRT_CTX, struct vmod_priv *priv, VCL_STRING cookieheader) +{ + struct vmod_cookie *vcp = cobj_get(priv); + char *name, *value; + const char *p, *sep; + int i = 0; + + if (cookieheader == NULL || *cookieheader == '\0') { + VSLb(ctx->vsl, SLT_Debug, "cookie: nothing to parse"); + return; + } + + /* If called twice during the same request, clean out old state. */ + if (!VTAILQ_EMPTY(&vcp->cookielist)) + vmod_clean(ctx, priv); + + p = cookieheader; + while (*p != '\0') { + while (isspace(*p)) + p++; + sep = strchr(p, '='); + if (sep == NULL) + break; + name = strndup(p, pdiff(p, sep)); + p = sep + 1; + + sep = p; + while (*sep != '\0' && *sep != ';') + sep++; + value = strndup(p, pdiff(p, sep)); + + vmod_set(ctx, priv, name, value); + free(name); + free(value); + i++; + if (*sep == '\0') + break; + p = sep + 1; + } + + VSLb(ctx->vsl, SLT_Debug, "cookie: parsed %i cookies.", i); +} + +static struct cookie * +find_cookie(struct vmod_cookie *vcp, VCL_STRING name) +{ + struct cookie *cookie; + + VTAILQ_FOREACH(cookie, &vcp->cookielist, list) { + CHECK_OBJ_NOTNULL(cookie, VMOD_COOKIE_ENTRY_MAGIC); + if (!strcmp(cookie->name, name)) + break; + } + return (cookie); +} + +VCL_VOID +vmod_set(VRT_CTX, struct vmod_priv *priv, VCL_STRING name, VCL_STRING value) +{ + struct vmod_cookie *vcp = cobj_get(priv); + struct cookie *cookie; + char *p; + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + CHECK_OBJ_NOTNULL(ctx->ws, WS_MAGIC); + + /* Empty cookies should be ignored. */ + if (name == NULL || *name == '\0') + return; + if (value == NULL || *value == '\0') + return; + + cookie = find_cookie(vcp, name); + if (cookie != NULL) { + p = WS_Printf(ctx->ws, "%s", value); + if (p == NULL) { + VSLb(ctx->vsl, SLT_Error, + "cookie: Workspace overflow in set()"); + } else + cookie->value = p; + return; + } + + cookie = WS_Alloc(ctx->ws, sizeof *cookie); + if (cookie == NULL) { + VSLb(ctx->vsl, SLT_Error, + "cookie: unable to get storage for cookie"); + return; + } + INIT_OBJ(cookie, VMOD_COOKIE_ENTRY_MAGIC); + cookie->name = WS_Printf(ctx->ws, "%s", name); + cookie->value = WS_Printf(ctx->ws, "%s", value); + if (cookie->name == NULL || cookie->value == NULL) { + VSLb(ctx->vsl, SLT_Error, + "cookie: unable to get storage for cookie"); + return; + } + VTAILQ_INSERT_TAIL(&vcp->cookielist, cookie, list); +} + +VCL_BOOL +vmod_isset(VRT_CTX, struct vmod_priv *priv, const char *name) +{ + struct vmod_cookie *vcp = cobj_get(priv); + struct cookie *cookie; + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + if (name == NULL || *name == '\0') + return (0); + + cookie = find_cookie(vcp, name); + return (cookie ? 1 : 0); +} + +VCL_STRING +vmod_get(VRT_CTX, struct vmod_priv *priv, VCL_STRING name) +{ + struct vmod_cookie *vcp = cobj_get(priv); + struct cookie *cookie; + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + if (name == NULL || *name == '\0') + return (NULL); + + cookie = find_cookie(vcp, name); + return (cookie ? cookie->value : NULL); +} + + +static vre_t * +compile_re(VRT_CTX, VCL_STRING expression) { + vre_t *vre; + const char *error; + int erroroffset; + + vre = VRE_compile(expression, 0, &error, &erroroffset); + if (vre == NULL) { + VSLb(ctx->vsl, SLT_Error, + "cookie: PCRE compile error at char %i: %s", + erroroffset, error); + } + return (vre); +} + +static void +free_re(void *priv) +{ + vre_t *vre; + + AN(priv); + vre = priv; + VRE_free(&vre); + AZ(vre); +} + +VCL_STRING +vmod_get_re(VRT_CTX, struct vmod_priv *priv, struct vmod_priv *priv_call, + VCL_STRING expression) +{ + struct vmod_cookie *vcp = cobj_get(priv); + int i, ovector[VRE_MAX_GROUPS]; + struct cookie *cookie = NULL; + struct cookie *current; + vre_t *vre = NULL; + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + if (expression == NULL || *expression == '\0') + return (NULL); + + if (priv_call->priv == NULL) { + AZ(pthread_mutex_lock(&mtx)); + vre = compile_re(ctx, expression); + if (vre == NULL) { + AZ(pthread_mutex_unlock(&mtx)); + return (NULL); + } + + priv_call->priv = vre; + priv_call->free = free_re; + AZ(pthread_mutex_unlock(&mtx)); + } + + VTAILQ_FOREACH(current, &vcp->cookielist, list) { + CHECK_OBJ_NOTNULL(current, VMOD_COOKIE_ENTRY_MAGIC); + VSLb(ctx->vsl, SLT_Debug, "cookie: checking %s", current->name); + i = VRE_exec(vre, current->name, strlen(current->name), 0, 0, + ovector, VRE_MAX_GROUPS, NULL); + if (i < 0) + continue; + + VSLb(ctx->vsl, SLT_Debug, "cookie: %s is a match for regex '%s'", + current->name, expression); + cookie = current; + break; + } + + return (cookie ? cookie->value : NULL); +} + +VCL_VOID +vmod_delete(VRT_CTX, struct vmod_priv *priv, VCL_STRING name) +{ + struct vmod_cookie *vcp = cobj_get(priv); + struct cookie *cookie; + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + if (name == NULL || *name == '\0') + return; + + cookie = find_cookie(vcp, name); + + if (cookie != NULL) + VTAILQ_REMOVE(&vcp->cookielist, cookie, list); +} + +VCL_VOID +vmod_clean(VRT_CTX, struct vmod_priv *priv) +{ + struct vmod_cookie *vcp = cobj_get(priv); + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + AN(vcp); + VTAILQ_INIT(&vcp->cookielist); +} + +static void +filter_cookies(struct vmod_priv *priv, VCL_STRING list_s, + enum filter_action mode) +{ + struct cookie *cookieptr, *safeptr; + struct vmod_cookie *vcp = cobj_get(priv); + struct matchlist *mlentry, *mlsafe; + char const *p = list_s, *q; + int matched = 0; + VTAILQ_HEAD(, matchlist) matchlist_head; + + VTAILQ_INIT(&matchlist_head); + + /* Parse the supplied list. */ + while (p && *p != '\0') { + while (isspace(*p)) + p++; + if (*p == '\0') + break; + + q = p; + while (*q != '\0' && *q != ',') + q++; + + if (q == p) { + p++; + continue; + } + + /* XXX: can we reserve/release lumps of txt instead of + * malloc/free? + */ + mlentry = malloc(sizeof *mlentry); + AN(mlentry); + mlentry->name = strndup(p, q - p); + AN(mlentry->name); + + VTAILQ_INSERT_TAIL(&matchlist_head, mlentry, list); + + p = q; + if (*p != '\0') + p++; + } + + /* Filter existing cookies that either aren't in the whitelist or + * are in the blacklist (depending on the filter_action) */ + VTAILQ_FOREACH_SAFE(cookieptr, &vcp->cookielist, list, safeptr) { + CHECK_OBJ_NOTNULL(cookieptr, VMOD_COOKIE_ENTRY_MAGIC); + matched = 0; + + VTAILQ_FOREACH(mlentry, &matchlist_head, list) { + if (strcmp(cookieptr->name, mlentry->name) == 0) { + matched = 1; + break; + } + } + if (matched != mode) + VTAILQ_REMOVE(&vcp->cookielist, cookieptr, list); + } + + VTAILQ_FOREACH_SAFE(mlentry, &matchlist_head, list, mlsafe) { + VTAILQ_REMOVE(&matchlist_head, mlentry, list); + free(mlentry->name); + free(mlentry); + } +} + +VCL_VOID +vmod_keep(VRT_CTX, struct vmod_priv *priv, VCL_STRING whitelist_s) +{ + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + filter_cookies(priv, whitelist_s, whitelist); +} + + +VCL_VOID +vmod_filter(VRT_CTX, struct vmod_priv *priv, VCL_STRING blacklist_s) +{ + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + filter_cookies(priv, blacklist_s, blacklist); +} + +static VCL_VOID +re_filter(VRT_CTX, struct vmod_priv *priv, struct vmod_priv *priv_call, + VCL_STRING expression, enum filter_action mode) +{ + struct vmod_cookie *vcp = cobj_get(priv); + struct cookie *current, *safeptr; + int i, ovector[VRE_MAX_GROUPS]; + vre_t *vre; + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + if (priv_call->priv == NULL) { + AZ(pthread_mutex_lock(&mtx)); + vre = compile_re(ctx, expression); + if (vre == NULL) { + AZ(pthread_mutex_unlock(&mtx)); + return; // Not much else to do, error already logged. + } + + priv_call->priv = vre; + priv_call->free = free_re; + AZ(pthread_mutex_unlock(&mtx)); + } + + VTAILQ_FOREACH_SAFE(current, &vcp->cookielist, list, safeptr) { + CHECK_OBJ_NOTNULL(current, VMOD_COOKIE_ENTRY_MAGIC); + + i = VRE_exec(priv_call->priv, current->name, + strlen(current->name), 0, 0, ovector, VRE_MAX_GROUPS, NULL); + + switch (mode) { + case blacklist: + if (i < 0) + continue; + VSLb(ctx->vsl, SLT_Debug, + "Removing matching cookie %s (value: %s)", + current->name, current->value); + VTAILQ_REMOVE(&vcp->cookielist, current, list); + break; + case whitelist: + if (i >= 0) { + VSLb(ctx->vsl, SLT_Debug, + "Cookie %s matches expression '%s'", + current->name, expression); + continue; + } + + VSLb(ctx->vsl, SLT_Debug, + "Removing cookie %s (value: %s)", + current->name, current->value); + VTAILQ_REMOVE(&vcp->cookielist, current, list); + break; + default: + WRONG("invalid mode"); + } + } +} + + +VCL_VOID +vmod_keep_re(VRT_CTX, struct vmod_priv *priv, struct vmod_priv *priv_call, + VCL_STRING expression) +{ + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + re_filter(ctx, priv, priv_call, expression, whitelist); +} + + +VCL_VOID +vmod_filter_re(VRT_CTX, struct vmod_priv *priv, struct vmod_priv *priv_call, + VCL_STRING expression) +{ + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + re_filter(ctx, priv, priv_call, expression, blacklist); +} + + +VCL_STRING +vmod_get_string(VRT_CTX, struct vmod_priv *priv) +{ + struct cookie *curr; + struct vsb output[1]; + char *res; + struct vmod_cookie *vcp = cobj_get(priv); + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + CHECK_OBJ_NOTNULL(ctx->ws, WS_MAGIC); + WS_VSB_new(output, ctx->ws); + + VTAILQ_FOREACH(curr, &vcp->cookielist, list) { + CHECK_OBJ_NOTNULL(curr, VMOD_COOKIE_ENTRY_MAGIC); + AN(curr->name); + AN(curr->value); + VSB_printf(output, "%s%s=%s;", + (curr == VTAILQ_FIRST(&vcp->cookielist)) ? "" : " ", + curr->name, curr->value); + } + res = WS_VSB_finish(output, ctx->ws, NULL); + if (res == NULL) + VSLb(ctx->vsl, SLT_Error, "cookie: Workspace overflow"); + return (res); +} + +VCL_STRING +vmod_format_rfc1123(VRT_CTX, VCL_TIME ts, VCL_DURATION duration) +{ + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + return (VRT_TIME_string(ctx, ts + duration)); +} diff --git a/man/Makefile.am b/man/Makefile.am index 20b17dce9..c341ddc87 100644 --- a/man/Makefile.am +++ b/man/Makefile.am @@ -15,6 +15,7 @@ dist_man_MANS = \ varnishtest.1 \ vtc.7 \ varnishtop.1 \ + vmod_cookie.3 \ vmod_directors.3 \ vmod_purge.3 \ vmod_std.3 \ @@ -91,6 +92,9 @@ varnishhist.1: \ $(top_builddir)/doc/sphinx/include/varnishhist_synopsis.rst $(BUILD_MAN) $(top_srcdir)/doc/sphinx/reference/varnishhist.rst $@ +vmod_cookie.3: $(top_builddir)/lib/libvmod_cookie/vmod_cookie.man.rst + $(BUILD_MAN) $? $@ + vmod_directors.3: $(top_builddir)/lib/libvmod_directors/vmod_directors.man.rst $(BUILD_MAN) $? $@ From dridi.boukelmoune at gmail.com Mon Feb 17 18:54:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 17 Feb 2020 18:54:07 +0000 (UTC) Subject: [master] 2caefd586 Add test missing from the previous commit Message-ID: <20200217185407.4F65B6E76E@lists.varnish-cache.org> commit 2caefd58610392a72b56b9ba4a4d727bd9dc5702 Author: Dridi Boukelmoune Date: Mon Feb 17 19:51:00 2020 +0100 Add test missing from the previous commit Apologies, I wrote this one to have proper coverage of priv_call->free functions, and the assertions free_re contains. I forgot to add it to the commit... diff --git a/lib/libvmod_cookie/tests/cookie_v00000.vtc b/lib/libvmod_cookie/tests/cookie_v00000.vtc new file mode 100644 index 000000000..9c5f097f7 --- /dev/null +++ b/lib/libvmod_cookie/tests/cookie_v00000.vtc @@ -0,0 +1,35 @@ +varnishtest "Test priv_call discard" + +varnish v1 -cliok "param.set thread_pools 1" +varnish v1 -vcl { + import cookie; + backend be none; + sub vcl_recv { return (synth(200)); } + sub vcl_synth { + cookie.set("cookie1", "value1"); + cookie.set("cookie2", "value2"); + cookie.set("cookie3", "value3"); + + cookie.filter_re("noop"); + cookie.keep_re("cook"); + set resp.http.get = cookie.get_re("cookie"); + } +} -start + +client c1 { + txreq -url "/" + rxresp + expect resp.http.get == value1 +} -run + +# load a new VCL to discard the first one +varnish v1 -vcl {backend be none;} +varnish v1 -cliok {vcl.discard vcl1} + +# tickle the CLI to trigger the effective VCL discard +varnish v1 -cliok ping +varnish v1 -expect n_vcl_avail == 1 +varnish v1 -expect n_vcl_discard == 0 + +# tickle again, check we survived discard-time assertions +varnish v1 -cliok ping From dridi.boukelmoune at gmail.com Tue Feb 18 05:37:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 18 Feb 2020 05:37:06 +0000 (UTC) Subject: [master] 8ab29a0d5 Polish Message-ID: <20200218053706.B6EB910D057@lists.varnish-cache.org> commit 8ab29a0d5690ac3c15a74d3d9865584cd02208b5 Author: Dridi Boukelmoune Date: Tue Feb 18 06:34:25 2020 +0100 Polish diff --git a/lib/libvmod_cookie/vmod_cookie.c b/lib/libvmod_cookie/vmod_cookie.c index 4187a1fb1..aaf5c1363 100644 --- a/lib/libvmod_cookie/vmod_cookie.c +++ b/lib/libvmod_cookie/vmod_cookie.c @@ -492,8 +492,8 @@ vmod_get_string(VRT_CTX, struct vmod_priv *priv) { struct cookie *curr; struct vsb output[1]; - char *res; struct vmod_cookie *vcp = cobj_get(priv); + const char *sep = "", *res; CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); CHECK_OBJ_NOTNULL(ctx->ws, WS_MAGIC); @@ -503,9 +503,8 @@ vmod_get_string(VRT_CTX, struct vmod_priv *priv) CHECK_OBJ_NOTNULL(curr, VMOD_COOKIE_ENTRY_MAGIC); AN(curr->name); AN(curr->value); - VSB_printf(output, "%s%s=%s;", - (curr == VTAILQ_FIRST(&vcp->cookielist)) ? "" : " ", - curr->name, curr->value); + VSB_printf(output, "%s%s=%s;", sep, curr->name, curr->value); + sep = " "; } res = WS_VSB_finish(output, ctx->ws, NULL); if (res == NULL) From dridi.boukelmoune at gmail.com Tue Feb 18 09:50:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 18 Feb 2020 09:50:06 +0000 (UTC) Subject: [master] 044809d62 Whitespace OCD Message-ID: <20200218095006.6260911281D@lists.varnish-cache.org> commit 044809d62e88d3390ec179c67ebb3d66e6e32fd4 Author: Dridi Boukelmoune Date: Tue Feb 18 10:48:34 2020 +0100 Whitespace OCD diff --git a/lib/libvmod_cookie/vmod_cookie.c b/lib/libvmod_cookie/vmod_cookie.c index aaf5c1363..078aa6d37 100644 --- a/lib/libvmod_cookie/vmod_cookie.c +++ b/lib/libvmod_cookie/vmod_cookie.c @@ -230,7 +230,8 @@ vmod_get(VRT_CTX, struct vmod_priv *priv, VCL_STRING name) static vre_t * -compile_re(VRT_CTX, VCL_STRING expression) { +compile_re(VRT_CTX, VCL_STRING expression) +{ vre_t *vre; const char *error; int erroroffset; @@ -411,7 +412,7 @@ vmod_filter(VRT_CTX, struct vmod_priv *priv, VCL_STRING blacklist_s) static VCL_VOID re_filter(VRT_CTX, struct vmod_priv *priv, struct vmod_priv *priv_call, - VCL_STRING expression, enum filter_action mode) + VCL_STRING expression, enum filter_action mode) { struct vmod_cookie *vcp = cobj_get(priv); struct cookie *current, *safeptr; From dridi.boukelmoune at gmail.com Tue Feb 18 13:41:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 18 Feb 2020 13:41:06 +0000 (UTC) Subject: [master] 686b530d2 Avoid a potential PRIV_TASK collision in vmod_debug Message-ID: <20200218134107.0397E117046@lists.varnish-cache.org> commit 686b530d222465e2b12d5f874bb39a2634be5b46 Author: Dridi Boukelmoune Date: Tue Feb 18 14:22:46 2020 +0100 Avoid a potential PRIV_TASK collision in vmod_debug You never know what the next test case will look like... diff --git a/lib/libvmod_debug/vmod.vcc b/lib/libvmod_debug/vmod.vcc index 5fe13d252..89cc648e4 100644 --- a/lib/libvmod_debug/vmod.vcc +++ b/lib/libvmod_debug/vmod.vcc @@ -269,12 +269,12 @@ Set the client socket' send buffer size to *sndbuf*. The previous, desired and actual values appear in the logs. Not currently implemented for backend transactions. -$Function VOID store_ip(PRIV_TASK, IP) +$Function VOID store_ip(IP) Store an IP address to be later found by ``debug.get_ip()`` in the same transaction. -$Function IP get_ip(PRIV_TASK) +$Function IP get_ip() Get the IP address previously stored by ``debug.store_ip()`` in the same transaction. diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c index ddd3b1f1d..12929bbd3 100644 --- a/lib/libvmod_debug/vmod_debug.c +++ b/lib/libvmod_debug/vmod_debug.c @@ -922,29 +922,37 @@ xyzzy_sndbuf(VRT_CTX, VCL_BYTES arg) } VCL_VOID -xyzzy_store_ip(VRT_CTX, struct vmod_priv *priv, VCL_IP ip) +xyzzy_store_ip(VRT_CTX, VCL_IP ip) { + struct vmod_priv *priv; CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - AN(priv); + + priv = VRT_priv_task(ctx, xyzzy_store_ip); + if (priv == NULL) { + VRT_fail(ctx, "no priv task - out of ws?"); + return; + } + AZ(priv->free); assert(VSA_Sane(ip)); - priv->priv = TRUST_ME(ip); } VCL_IP -xyzzy_get_ip(VRT_CTX, struct vmod_priv *priv) +xyzzy_get_ip(VRT_CTX) { + struct vmod_priv *priv; VCL_IP ip; CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + + priv = VRT_priv_task(ctx, xyzzy_store_ip); AN(priv); AZ(priv->free); ip = priv->priv; assert(VSA_Sane(ip)); - return (ip); } From dridi.boukelmoune at gmail.com Tue Feb 18 16:03:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 18 Feb 2020 16:03:06 +0000 (UTC) Subject: [master] 7351edca9 Forgot missing casts Message-ID: <20200218160306.E6FDB11AD61@lists.varnish-cache.org> commit 7351edca9532e393fdab485c0d97794c18553127 Author: Dridi Boukelmoune Date: Tue Feb 18 17:02:13 2020 +0100 Forgot missing casts diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c index 12929bbd3..65836903c 100644 --- a/lib/libvmod_debug/vmod_debug.c +++ b/lib/libvmod_debug/vmod_debug.c @@ -928,7 +928,7 @@ xyzzy_store_ip(VRT_CTX, VCL_IP ip) CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - priv = VRT_priv_task(ctx, xyzzy_store_ip); + priv = VRT_priv_task(ctx, (void *)xyzzy_store_ip); if (priv == NULL) { VRT_fail(ctx, "no priv task - out of ws?"); return; @@ -947,7 +947,7 @@ xyzzy_get_ip(VRT_CTX) CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - priv = VRT_priv_task(ctx, xyzzy_store_ip); + priv = VRT_priv_task(ctx, (void *)xyzzy_store_ip); AN(priv); AZ(priv->free); From gquintard at users.noreply.github.com Tue Feb 18 16:49:06 2020 From: gquintard at users.noreply.github.com (guillaume quintard) Date: Tue, 18 Feb 2020 16:49:06 +0000 (UTC) Subject: [master] 537ddcdc3 VMOD cookie doc grammar OCD. Message-ID: <20200218164906.9C3756E6B@lists.varnish-cache.org> commit 537ddcdc34ec243313f3e2ef06418acb673c3840 Author: Geoff Simmons Date: Tue Feb 18 17:33:00 2020 +0100 VMOD cookie doc grammar OCD. diff --git a/lib/libvmod_cookie/vmod.vcc b/lib/libvmod_cookie/vmod.vcc index 3cbdfe261..f4263aebd 100644 --- a/lib/libvmod_cookie/vmod.vcc +++ b/lib/libvmod_cookie/vmod.vcc @@ -3,7 +3,7 @@ $Module cookie 3 "Varnish Cookie Module" DESCRIPTION =========== -Handle HTTP cookies easier in Varnish VCL. +Handle HTTP cookies more easily in Varnish VCL. Parses a cookie header into an internal data store, where per-cookie get/set/delete functions are available. A filter_except() method removes all From dridi.boukelmoune at gmail.com Wed Feb 19 08:05:08 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 19 Feb 2020 08:05:08 +0000 (UTC) Subject: [master] 144df88ca SPDX-License-Identifier in vmod_cookie.c Message-ID: <20200219080508.DC9A510A9B5@lists.varnish-cache.org> commit 144df88cabd97bb1b728cffd4fe3e496b151bf72 Author: Dridi Boukelmoune Date: Wed Feb 19 09:03:01 2020 +0100 SPDX-License-Identifier in vmod_cookie.c Refs #3184 diff --git a/lib/libvmod_cookie/vmod_cookie.c b/lib/libvmod_cookie/vmod_cookie.c index 078aa6d37..d4b71cdd7 100644 --- a/lib/libvmod_cookie/vmod_cookie.c +++ b/lib/libvmod_cookie/vmod_cookie.c @@ -5,6 +5,8 @@ * Author: Lasse Karstensen * Author: Dridi Boukelmoune * + * SPDX-License-Identifier: BSD-2-Clause + * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: From phk at FreeBSD.org Wed Feb 19 08:47:07 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 19 Feb 2020 08:47:07 +0000 (UTC) Subject: [master] b797261cd Polish a bit Message-ID: <20200219084707.1F2B610C917@lists.varnish-cache.org> commit b797261cdd40ac99514b1569208d4126fc4a564e Author: Poul-Henning Kamp Date: Wed Feb 19 08:31:55 2020 +0000 Polish a bit diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index 9549f12b8..8a417fb14 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -471,18 +471,18 @@ SES_Wait(struct sess *sp, const struct transport *xp) /* * Put struct waited on the workspace. Make sure that the - * workspace can hold enough space for the largest of struct - * waited and pool_task, as pool_task will be needed when coming + * workspace can hold enough space for both struct waited + * and pool_task, as pool_task will be needed when coming * off the waiter again. */ - u = sizeof (struct waited); - if (sizeof (struct pool_task) > u) - u = sizeof (struct pool_task); - if (!WS_ReserveSize(sp->ws, u)) { + u = WS_ReserveAll(sp->ws); + if (u < sizeof (struct waited) || u < sizeof(struct pool_task)) { + WS_MarkOverflow(sp->ws); SES_Delete(sp, SC_OVERLOAD, NAN); return; } - wp = (void*)sp->ws->f; + + wp = (void*)WS_Front(sp->ws); INIT_OBJ(wp, WAITED_MAGIC); wp->fd = sp->fd; wp->priv1 = sp; From phk at FreeBSD.org Wed Feb 19 08:47:07 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 19 Feb 2020 08:47:07 +0000 (UTC) Subject: [master] cd5eb5506 Make this test do a little bit more work than just not panic Message-ID: <20200219084707.3F6C310C91A@lists.varnish-cache.org> commit cd5eb55061ca288910e5c4be1b723fcb5b8ee4df Author: Poul-Henning Kamp Date: Wed Feb 19 08:32:14 2020 +0000 Make this test do a little bit more work than just not panic diff --git a/bin/varnishtest/tests/r01768.vtc b/bin/varnishtest/tests/r01768.vtc index 716701ae8..2d04f0d60 100644 --- a/bin/varnishtest/tests/r01768.vtc +++ b/bin/varnishtest/tests/r01768.vtc @@ -6,14 +6,17 @@ server s1 { } -start varnish v1 -vcl+backend { - sub vcl_recv { + sub vcl_deliver { if (req.http.foo_bar == req.http.foo-bar) { - set req.http.foo_bar = "xxx"; + set resp.http.foo = "xxx"; + } else { + set resp.http.foo = "yyy"; } } } -start client c1 { - txreq + txreq -hdr "foo_bar: 1" -hdr "foo-bar: 2" rxresp + expect resp.http.foo == yyy } -run From phk at FreeBSD.org Wed Feb 19 08:47:07 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 19 Feb 2020 08:47:07 +0000 (UTC) Subject: [master] ce7bc7ead Always put macros like this in (...), you never know where they'll end up. Message-ID: <20200219084707.648AC10C91D@lists.varnish-cache.org> commit ce7bc7ead74bbd66cce6f455a688b18fbe21b1db Author: Poul-Henning Kamp Date: Wed Feb 19 08:45:12 2020 +0000 Always put macros like this in (...), you never know where they'll end up. diff --git a/bin/varnishd/proxy/cache_proxy_proto.c b/bin/varnishd/proxy/cache_proxy_proto.c index 3cc0f48a0..a9b267458 100644 --- a/bin/varnishd/proxy/cache_proxy_proto.c +++ b/bin/varnishd/proxy/cache_proxy_proto.c @@ -728,9 +728,9 @@ VPX_Format_Proxy(struct vsb *vsb, int version, } #define PXY_BUFSZ \ - sizeof(vpx1_sig) + 4 /* TCPx */ + \ + (sizeof(vpx1_sig) + 4 /* TCPx */ + \ 2 * VTCP_ADDRBUFSIZE + 2 * VTCP_PORTBUFSIZE + \ - 6 /* spaces, CRLF */ + 16 /* safety */ + 6 /* spaces, CRLF */ + 16 /* safety */ ) int VPX_Send_Proxy(int fd, int version, const struct sess *sp) From phk at FreeBSD.org Wed Feb 19 08:47:07 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 19 Feb 2020 08:47:07 +0000 (UTC) Subject: [master] 2034138ed Remove mostly harmless casts to (void*) Message-ID: <20200219084707.89B9E10C923@lists.varnish-cache.org> commit 2034138ed25036df7c95ac784ece9981ebc596c5 Author: Poul-Henning Kamp Date: Wed Feb 19 08:45:59 2020 +0000 Remove mostly harmless casts to (void*) diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c index 65836903c..4c8dac699 100644 --- a/lib/libvmod_debug/vmod_debug.c +++ b/lib/libvmod_debug/vmod_debug.c @@ -928,7 +928,7 @@ xyzzy_store_ip(VRT_CTX, VCL_IP ip) CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - priv = VRT_priv_task(ctx, (void *)xyzzy_store_ip); + priv = VRT_priv_task(ctx, xyzzy_store_ip); if (priv == NULL) { VRT_fail(ctx, "no priv task - out of ws?"); return; @@ -947,7 +947,7 @@ xyzzy_get_ip(VRT_CTX) CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - priv = VRT_priv_task(ctx, (void *)xyzzy_store_ip); + priv = VRT_priv_task(ctx, xyzzy_store_ip); AN(priv); AZ(priv->free); @@ -1110,7 +1110,7 @@ xyzzy_fail_rollback(VRT_CTX) CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - p = VRT_priv_task(ctx, (void *)xyzzy_fail_rollback); + p = VRT_priv_task(ctx, xyzzy_fail_rollback); if (p == NULL) { VRT_fail(ctx, "no priv task - out of ws?"); return; @@ -1133,7 +1133,7 @@ xyzzy_ok_rollback(VRT_CTX) CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - p = VRT_priv_task(ctx, (void *)xyzzy_fail_rollback); + p = VRT_priv_task(ctx, xyzzy_fail_rollback); if (p == NULL) { VRT_fail(ctx, "no priv task - out of ws?"); return; From dridi at varni.sh Wed Feb 19 08:56:02 2020 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 19 Feb 2020 08:56:02 +0000 Subject: [master] 2034138ed Remove mostly harmless casts to (void*) In-Reply-To: <20200219084707.89B9E10C923@lists.varnish-cache.org> References: <20200219084707.89B9E10C923@lists.varnish-cache.org> Message-ID: On Wed, Feb 19, 2020 at 8:47 AM Poul-Henning Kamp wrote: > > > commit 2034138ed25036df7c95ac784ece9981ebc596c5 > Author: Poul-Henning Kamp > Date: Wed Feb 19 08:45:59 2020 +0000 > > Remove mostly harmless casts to (void*) We will need to revert this, pedantic suncc will complain on vtest. > diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c > index 65836903c..4c8dac699 100644 > --- a/lib/libvmod_debug/vmod_debug.c > +++ b/lib/libvmod_debug/vmod_debug.c > @@ -928,7 +928,7 @@ xyzzy_store_ip(VRT_CTX, VCL_IP ip) > > CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); > > - priv = VRT_priv_task(ctx, (void *)xyzzy_store_ip); > + priv = VRT_priv_task(ctx, xyzzy_store_ip); > if (priv == NULL) { > VRT_fail(ctx, "no priv task - out of ws?"); > return; > @@ -947,7 +947,7 @@ xyzzy_get_ip(VRT_CTX) > > CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); > > - priv = VRT_priv_task(ctx, (void *)xyzzy_store_ip); > + priv = VRT_priv_task(ctx, xyzzy_store_ip); > AN(priv); > AZ(priv->free); > > @@ -1110,7 +1110,7 @@ xyzzy_fail_rollback(VRT_CTX) > > CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); > > - p = VRT_priv_task(ctx, (void *)xyzzy_fail_rollback); > + p = VRT_priv_task(ctx, xyzzy_fail_rollback); > if (p == NULL) { > VRT_fail(ctx, "no priv task - out of ws?"); > return; > @@ -1133,7 +1133,7 @@ xyzzy_ok_rollback(VRT_CTX) > > CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); > > - p = VRT_priv_task(ctx, (void *)xyzzy_fail_rollback); > + p = VRT_priv_task(ctx, xyzzy_fail_rollback); > if (p == NULL) { > VRT_fail(ctx, "no priv task - out of ws?"); > return; > _______________________________________________ > varnish-commit mailing list > varnish-commit at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-commit From dridi.boukelmoune at gmail.com Wed Feb 19 09:05:08 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 19 Feb 2020 09:05:08 +0000 (UTC) Subject: [master] 820939d48 Revert "Remove mostly harmless casts to (void*)" Message-ID: <20200219090509.017CD10D514@lists.varnish-cache.org> commit 820939d484a6b291a91c708185b4dd6fa804c399 Author: Dridi Boukelmoune Date: Wed Feb 19 10:03:12 2020 +0100 Revert "Remove mostly harmless casts to (void*)" This reverts commit 2034138ed25036df7c95ac784ece9981ebc596c5. We already know that SunCC will complain about those. Alternatively we could use static variables addresses as cookies, but do we need extra symbols? diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c index 4c8dac699..65836903c 100644 --- a/lib/libvmod_debug/vmod_debug.c +++ b/lib/libvmod_debug/vmod_debug.c @@ -928,7 +928,7 @@ xyzzy_store_ip(VRT_CTX, VCL_IP ip) CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - priv = VRT_priv_task(ctx, xyzzy_store_ip); + priv = VRT_priv_task(ctx, (void *)xyzzy_store_ip); if (priv == NULL) { VRT_fail(ctx, "no priv task - out of ws?"); return; @@ -947,7 +947,7 @@ xyzzy_get_ip(VRT_CTX) CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - priv = VRT_priv_task(ctx, xyzzy_store_ip); + priv = VRT_priv_task(ctx, (void *)xyzzy_store_ip); AN(priv); AZ(priv->free); @@ -1110,7 +1110,7 @@ xyzzy_fail_rollback(VRT_CTX) CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - p = VRT_priv_task(ctx, xyzzy_fail_rollback); + p = VRT_priv_task(ctx, (void *)xyzzy_fail_rollback); if (p == NULL) { VRT_fail(ctx, "no priv task - out of ws?"); return; @@ -1133,7 +1133,7 @@ xyzzy_ok_rollback(VRT_CTX) CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - p = VRT_priv_task(ctx, xyzzy_fail_rollback); + p = VRT_priv_task(ctx, (void *)xyzzy_fail_rollback); if (p == NULL) { VRT_fail(ctx, "no priv task - out of ws?"); return; From phk at FreeBSD.org Wed Feb 19 09:19:08 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 19 Feb 2020 09:19:08 +0000 (UTC) Subject: [master] 8e94f399d Fix this properly, now that I've had a chance to dive into the C-spec to understand what the SunOS compiler was whining about. Message-ID: <20200219091908.34EAA10DADA@lists.varnish-cache.org> commit 8e94f399dd2bf38da086ad161515a86998e1e559 Author: Poul-Henning Kamp Date: Wed Feb 19 09:16:26 2020 +0000 Fix this properly, now that I've had a chance to dive into the C-spec to understand what the SunOS compiler was whining about. Strictly speaking it has a point, code is different from data is different from const data, so mixing pointers the way we did was not kosher. We probably should have made the identifying argument a uintptr_t instead of const void* to begin with to deal with this. diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c index 65836903c..45234400c 100644 --- a/lib/libvmod_debug/vmod_debug.c +++ b/lib/libvmod_debug/vmod_debug.c @@ -62,6 +62,8 @@ static pthread_mutex_t vsc_mtx = PTHREAD_MUTEX_INITIALIZER; static struct vsc_seg *vsc_seg = NULL; static struct VSC_debug *vsc = NULL; static int loads; +static const int store_ip_token; +static const int fail_rollback_token; /**********************************************************************/ @@ -928,7 +930,7 @@ xyzzy_store_ip(VRT_CTX, VCL_IP ip) CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - priv = VRT_priv_task(ctx, (void *)xyzzy_store_ip); + priv = VRT_priv_task(ctx, &store_ip_token); if (priv == NULL) { VRT_fail(ctx, "no priv task - out of ws?"); return; @@ -947,7 +949,7 @@ xyzzy_get_ip(VRT_CTX) CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - priv = VRT_priv_task(ctx, (void *)xyzzy_store_ip); + priv = VRT_priv_task(ctx, &store_ip_token); AN(priv); AZ(priv->free); @@ -1110,7 +1112,7 @@ xyzzy_fail_rollback(VRT_CTX) CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - p = VRT_priv_task(ctx, (void *)xyzzy_fail_rollback); + p = VRT_priv_task(ctx, &fail_rollback_token); if (p == NULL) { VRT_fail(ctx, "no priv task - out of ws?"); return; @@ -1133,7 +1135,7 @@ xyzzy_ok_rollback(VRT_CTX) CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - p = VRT_priv_task(ctx, (void *)xyzzy_fail_rollback); + p = VRT_priv_task(ctx, &fail_rollback_token); if (p == NULL) { VRT_fail(ctx, "no priv task - out of ws?"); return; From dridi.boukelmoune at gmail.com Wed Feb 19 11:07:09 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 19 Feb 2020 11:07:09 +0000 (UTC) Subject: [master] e5c94183d Rename trunk release notes to 6.3 Message-ID: <20200219110709.8C5D610FD2B@lists.varnish-cache.org> commit e5c94183d2c45a5d9a6e2917dd0d9988fc8e23c7 Author: Dridi Boukelmoune Date: Wed Feb 19 11:59:46 2020 +0100 Rename trunk release notes to 6.3 They are however far from complete, and that's an UNDERSTATEMENT. diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-6.3.rst similarity index 95% rename from doc/sphinx/whats-new/changes-trunk.rst rename to doc/sphinx/whats-new/changes-6.3.rst index 7c26ae580..377b73d8f 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-6.3.rst @@ -2,14 +2,14 @@ updates for changes in the development branch. For changes in the released versions of Varnish, see:** :ref:`whats-new-index` -.. _whatsnew_changes_CURRENT: +.. _whatsnew_changes_6.3: -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -Changes in Varnish **$NEXT_RELEASE** -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%% +Changes in Varnish 6.3 +%%%%%%%%%%%%%%%%%%%%%% For information about updating your current Varnish deployment to the -new version, see :ref:`whatsnew_upgrading_CURRENT`. +new version, see :ref:`whatsnew_upgrading_6.3`. A more detailed and technical account of changes in Varnish, with links to issues that have been fixed and pull requests that have been diff --git a/doc/sphinx/whats-new/index.rst b/doc/sphinx/whats-new/index.rst index 510027c08..50084dd3f 100644 --- a/doc/sphinx/whats-new/index.rst +++ b/doc/sphinx/whats-new/index.rst @@ -8,19 +8,14 @@ This section describes the changes and improvements between different versions of Varnish, and what upgrading between the different versions entail. -Varnish **$NEXT_RELEASE** -------------------------- - -**Note: These are working documents for a future release, with running -updates for changes in the development branch. For changes in the -released versions of Varnish, see the chapters listed below.** - +Varnish 6.3 +----------- .. toctree:: :maxdepth: 2 - changes-trunk - upgrading-trunk + changes-6.3 + upgrading-6.3 Varnish 6.2 ----------- diff --git a/doc/sphinx/whats-new/upgrading-trunk.rst b/doc/sphinx/whats-new/upgrading-6.3.rst similarity index 86% rename from doc/sphinx/whats-new/upgrading-trunk.rst rename to doc/sphinx/whats-new/upgrading-6.3.rst index 6143fde99..dfd4f3510 100644 --- a/doc/sphinx/whats-new/upgrading-trunk.rst +++ b/doc/sphinx/whats-new/upgrading-6.3.rst @@ -2,11 +2,11 @@ updates for changes in the development branch. For changes in the released versions of Varnish, see:** :ref:`whats-new-index` -.. _whatsnew_upgrading_CURRENT: +.. _whatsnew_upgrading_6.3: -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -Upgrading to Varnish **$NEXT_RELEASE** -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%% +Upgrading to Varnish 6.3 +%%%%%%%%%%%%%%%%%%%%%%%% **XXX: how to upgrade from previous deployments to this version. Limited to work that has to be done for an upgrade, new From dridi.boukelmoune at gmail.com Wed Feb 19 11:07:09 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 19 Feb 2020 11:07:09 +0000 (UTC) Subject: [master] da9368cb8 Start skeleton release notes for the next version. Message-ID: <20200219110709.A1E2910FD2E@lists.varnish-cache.org> commit da9368cb81106b05d8937c2b520ef4dd2ad441ca Author: Geoff Simmons Date: Tue Sep 25 16:31:17 2018 +0200 Start skeleton release notes for the next version. Restructured so that: * 'Upgrading' is limited to work that has to be done to upgrade from a current deployment to the new version. * 'Changes' is a comprehensive, user-level description of changes and new features. Conflicts: doc/sphinx/whats-new/index.rst diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst new file mode 100644 index 000000000..fbda9d6b9 --- /dev/null +++ b/doc/sphinx/whats-new/changes-trunk.rst @@ -0,0 +1,73 @@ +**Note: This is a working document for a future release, with running +updates for changes in the development branch. For changes in the +released versions of Varnish, see:** :ref:`whats-new-index` + +.. _whatsnew_changes_CURRENT: + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +Changes in Varnish **$NEXT_RELEASE** +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +For information about updating your current Varnish deployment to the +new version, see :ref:`whatsnew_upgrading_CURRENT`. + +A more detailed and technical account of changes in Varnish, with +links to issues that have been fixed and pull requests that have been +merged, may be found in the `change log`_. + +.. _change log: https://github.com/varnishcache/varnish-cache/blob/master/doc/changes.rst + +varnishd +======== + +Parameters +~~~~~~~~~~ + +**XXX changes in -p parameters** + +Other changes in varnishd +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Changes to VCL +============== + +VCL variables +~~~~~~~~~~~~~ + +**XXX new, deprecated or removed variables, or changed semantics** + +Other changes to VCL +~~~~~~~~~~~~~~~~~~~~ + +VMODs +===== + +**XXX changes in the bundled VMODs** + +varnishlog +========== + +**XXX changes concerning varnishlog(1) and/or vsl(7)** + +varnishadm +========== + +**XXX changes concerning varnishadm(1) and/or varnish-cli(7)** + +varnishstat +=========== + +**XXX changes concerning varnishstat(1) and/or varnish-counters(7)** + +varnishtest +=========== + +**XXX changes concerning varnishtest(1) and/or vtc(7)** + +Changes for developers and VMOD authors +======================================= + +**XXX changes concerning VRT, the public APIs, source code organization, +builds etc.** + +*eof* diff --git a/doc/sphinx/whats-new/index.rst b/doc/sphinx/whats-new/index.rst index 50084dd3f..ad1c6bbcd 100644 --- a/doc/sphinx/whats-new/index.rst +++ b/doc/sphinx/whats-new/index.rst @@ -8,6 +8,19 @@ This section describes the changes and improvements between different versions of Varnish, and what upgrading between the different versions entail. +Varnish **$NEXT_RELEASE** +------------------------- + +**Note: These are working documents for a future release, with running +updates for changes in the development branch. For changes in the +released versions of Varnish, see the chapters listed below.** + +.. toctree:: + :maxdepth: 2 + + changes-trunk + upgrading-trunk + Varnish 6.3 ----------- diff --git a/doc/sphinx/whats-new/upgrading-trunk.rst b/doc/sphinx/whats-new/upgrading-trunk.rst new file mode 100644 index 000000000..6143fde99 --- /dev/null +++ b/doc/sphinx/whats-new/upgrading-trunk.rst @@ -0,0 +1,33 @@ +**Note: This is a working document for a future release, with running +updates for changes in the development branch. For changes in the +released versions of Varnish, see:** :ref:`whats-new-index` + +.. _whatsnew_upgrading_CURRENT: + +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +Upgrading to Varnish **$NEXT_RELEASE** +%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% + +**XXX: how to upgrade from previous deployments to this +version. Limited to work that has to be done for an upgrade, new +features are listed in "Changes". Explicitly mention what does *not* +have to be changed, especially in VCL. May include, but is not limited +to:** + +* Elements of VCL that have been removed or are deprecated, or whose + semantics have changed. + +* -p parameters that have been removed or are deprecated, or whose + semantics have changed. + +* Changes in the CLI. + +* Changes in the output or interpretation of stats or the log, including + changes affecting varnishncsa/-hist/-top. + +* Changes that may be necessary in VTCs or in the use of varnishtest. + +* Changes in public APIs that may require changes in VMODs or VAPI/VUT + clients. + +*eof* From phk at FreeBSD.org Wed Feb 19 13:29:06 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 19 Feb 2020 13:29:06 +0000 (UTC) Subject: [master] 8c5d1e729 Try to contain pipe-magic in pipe code. Message-ID: <20200219132906.F19BE113581@lists.varnish-cache.org> commit 8c5d1e7298d3bd79b1af924ead1a4dd5d83be2a0 Author: Poul-Henning Kamp Date: Wed Feb 19 10:47:43 2020 +0000 Try to contain pipe-magic in pipe code. diff --git a/bin/varnishd/cache/cache_backend.c b/bin/varnishd/cache/cache_backend.c index 81ab3c5fa..1ce50846c 100644 --- a/bin/varnishd/cache/cache_backend.c +++ b/bin/varnishd/cache/cache_backend.c @@ -288,7 +288,7 @@ vbe_dir_gethdrs(VRT_CTX, VCL_BACKEND d) extrachance = 0; i = V1F_SendReq(wrk, bo, &bo->acct.bereq_hdrbytes, - &bo->acct.bereq_bodybytes, 0); + &bo->acct.bereq_bodybytes); if (PFD_State(pfd) != PFD_STATE_USED) { if (VTP_Wait(wrk, pfd, VTIM_real() + @@ -374,7 +374,7 @@ vbe_dir_http1pipe(VRT_CTX, VCL_BACKEND d) } else { CHECK_OBJ_NOTNULL(ctx->bo->htc, HTTP_CONN_MAGIC); i = V1F_SendReq(ctx->req->wrk, ctx->bo, - &v1a.bereq, &v1a.out, 1); + &v1a.bereq, &v1a.out); VSLb_ts_req(ctx->req, "Pipe", W_TIM_real(ctx->req->wrk)); if (i == 0) V1P_Process(ctx->req, *PFD_Fd(pfd), &v1a); diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index 7cff23876..a7dbb42e3 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -740,6 +740,9 @@ cnt_pipe(struct worker *wrk, struct req *req) AZ(bo->req); bo->req = req; bo->wrk = wrk; + /* Unless cached, reqbody is not our job */ + if (req->req_body_status != REQ_BODY_CACHED) + req->req_body_status = REQ_BODY_NONE; SES_Close(req->sp, VDI_Http1Pipe(req, bo)); nxt = REQ_FSM_DONE; V1P_Leave(); diff --git a/bin/varnishd/http1/cache_http1.h b/bin/varnishd/http1/cache_http1.h index 4e66938bb..2e8b03ac3 100644 --- a/bin/varnishd/http1/cache_http1.h +++ b/bin/varnishd/http1/cache_http1.h @@ -33,7 +33,7 @@ struct VSC_vbe; /* cache_http1_fetch.c [V1F] */ int V1F_SendReq(struct worker *, struct busyobj *, uint64_t *ctr_hdrbytes, - uint64_t *ctr_bodybytes, int onlycached); + uint64_t *ctr_bodybytes); int V1F_FetchRespHdr(struct busyobj *); int V1F_Setup_Fetch(struct vfp_ctx *vfc, struct http_conn *htc); diff --git a/bin/varnishd/http1/cache_http1_fetch.c b/bin/varnishd/http1/cache_http1_fetch.c index 50edabb83..ee7a9fec1 100644 --- a/bin/varnishd/http1/cache_http1_fetch.c +++ b/bin/varnishd/http1/cache_http1_fetch.c @@ -71,7 +71,7 @@ vbf_iter_req_body(void *priv, unsigned flush, const void *ptr, ssize_t l) int V1F_SendReq(struct worker *wrk, struct busyobj *bo, uint64_t *ctr_hdrbytes, - uint64_t *ctr_bodybytes, int onlycached) + uint64_t *ctr_bodybytes) { struct http *hp; enum sess_close sc; @@ -114,7 +114,7 @@ V1F_SendReq(struct worker *wrk, struct busyobj *bo, uint64_t *ctr_hdrbytes, if (do_chunked) V1L_EndChunk(wrk); } else if (bo->req != NULL && - (bo->req->req_body_status == REQ_BODY_CACHED || !onlycached)) { + bo->req->req_body_status != REQ_BODY_NONE) { if (do_chunked) V1L_Chunked(wrk); i = VRB_Iterate(wrk, bo->vsl, bo->req, vbf_iter_req_body, bo); From phk at FreeBSD.org Wed Feb 19 14:16:06 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 19 Feb 2020 14:16:06 +0000 (UTC) Subject: [master] e9af8be89 Add documenting asserts Message-ID: <20200219141606.4E508114585@lists.varnish-cache.org> commit e9af8be89906411acb0c48a89295222e65937ba0 Author: Poul-Henning Kamp Date: Wed Feb 19 14:15:04 2020 +0000 Add documenting asserts diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index 9f94a546d..4253e21bf 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -250,6 +250,9 @@ vbf_stp_mkbereq(struct worker *wrk, struct busyobj *bo) HSH_Ref(bo->bereq_body); bo->req = NULL; ObjSetState(bo->wrk, bo->fetch_objcore, BOS_REQ_DONE); + } else if (bo->req->req_body_status != REQ_BODY_WITH_LEN && + bo->req->req_body_status != REQ_BODY_WITHOUT_LEN) { + WRONG("Bad req_body_status"); } return (F_STP_STARTFETCH); } diff --git a/bin/varnishd/http1/cache_http1_fetch.c b/bin/varnishd/http1/cache_http1_fetch.c index ee7a9fec1..79b6ceb35 100644 --- a/bin/varnishd/http1/cache_http1_fetch.c +++ b/bin/varnishd/http1/cache_http1_fetch.c @@ -107,6 +107,7 @@ V1F_SendReq(struct worker *wrk, struct busyobj *bo, uint64_t *ctr_hdrbytes, i = 0; if (bo->bereq_body != NULL) { + AZ(bo->req); if (do_chunked) V1L_Chunked(wrk); (void)ObjIterate(bo->wrk, bo->bereq_body, From phk at FreeBSD.org Wed Feb 19 14:55:06 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 19 Feb 2020 14:55:06 +0000 (UTC) Subject: [master] ba8e58b49 Avoid spreading req.body status logic all over the place Message-ID: <20200219145506.DB85B1152FB@lists.varnish-cache.org> commit ba8e58b494b7bfee3ae754f3c6916e568648e5e3 Author: Poul-Henning Kamp Date: Wed Feb 19 14:54:31 2020 +0000 Avoid spreading req.body status logic all over the place diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index b1f8bdc40..0d66cea58 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -389,7 +389,6 @@ struct busyobj { * is recycled. */ unsigned retries; - enum req_body_state_e initial_req_body_status; struct req *req; struct sess *sp; struct worker *wrk; @@ -406,6 +405,8 @@ struct busyobj { struct objcore *stale_oc; struct objcore *fetch_objcore; + const char *no_retry; + struct http_conn *htc; struct pool_task fetch_task; diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index 4253e21bf..06bd94ddc 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -240,7 +240,6 @@ vbf_stp_mkbereq(struct worker *wrk, struct busyobj *bo) bo->ws_bo = WS_Snapshot(bo->ws); HTTP_Clone(bo->bereq, bo->bereq0); - bo->initial_req_body_status = bo->req->req_body_status; if (bo->req->req_body_status == REQ_BODY_NONE) { bo->req = NULL; ObjSetState(bo->wrk, bo->fetch_objcore, BOS_REQ_DONE); @@ -270,12 +269,9 @@ vbf_stp_retry(struct worker *wrk, struct busyobj *bo) assert(bo->fetch_objcore->boc->state <= BOS_REQ_DONE); - if (bo->fetch_objcore->boc->state == BOS_REQ_DONE && - bo->initial_req_body_status != REQ_BODY_NONE && - bo->bereq_body == NULL) { - /* We have already released the req and there was a - * request body that was not cached. Too late to retry. */ - VSLb(bo->vsl, SLT_Error, "req.body already consumed"); + if (bo->no_retry != NULL) { + VSLb(bo->vsl, SLT_Error, + "Retry not possible, %s", bo->no_retry); return (F_STP_FAIL); } diff --git a/bin/varnishd/http1/cache_http1_fetch.c b/bin/varnishd/http1/cache_http1_fetch.c index 79b6ceb35..64547f8e3 100644 --- a/bin/varnishd/http1/cache_http1_fetch.c +++ b/bin/varnishd/http1/cache_http1_fetch.c @@ -120,6 +120,9 @@ V1F_SendReq(struct worker *wrk, struct busyobj *bo, uint64_t *ctr_hdrbytes, V1L_Chunked(wrk); i = VRB_Iterate(wrk, bo->vsl, bo->req, vbf_iter_req_body, bo); + if (bo->req->req_body_status != REQ_BODY_CACHED) + bo->no_retry = "req.body not cached"; + if (bo->req->req_body_status == REQ_BODY_FAIL) { /* * XXX: (#2332) We should test to see if the backend From dridi.boukelmoune at gmail.com Thu Feb 20 11:10:09 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 20 Feb 2020 11:10:09 +0000 (UTC) Subject: [master] b974d76c5 varnishstat.rst: Mention default log level in curses mode Message-ID: <20200220111009.8A8E3108940@lists.varnish-cache.org> commit b974d76c5d4516c5c7ebc1eca66d620aa3b4370c Author: Klemens Nanni Date: Wed Feb 19 17:36:45 2020 +0100 varnishstat.rst: Mention default log level in curses mode The Keybindings section is the only place mentioning differnt log levels and one might assume that `-1` or `-j` behave the same as the default ncurses interface. The latter however does not show all counters by default as the single action flags do, hence `varnishstat -f '*overload*'` will not show `MAIN.sc_overload` by default, with `-1` however it does. Fix grammar while here. diff --git a/doc/sphinx/reference/varnishstat.rst b/doc/sphinx/reference/varnishstat.rst index 26d1b9836..ecc26526c 100644 --- a/doc/sphinx/reference/varnishstat.rst +++ b/doc/sphinx/reference/varnishstat.rst @@ -30,7 +30,7 @@ The following options are available: CURSES MODE =========== -When neither -1, -j or -x options are given, the application starts up +When neither -1, -j nor -x options are given, the application starts up in curses mode. This shows a continuously updated view of the counter values, along with their description. @@ -41,6 +41,8 @@ The center area shows a list of counter values. The bottom area shows the description of the currently selected counter. +On startup, only counters at INFO level are shown. + Columns ------- From dridi.boukelmoune at gmail.com Thu Feb 20 12:17:05 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 20 Feb 2020 12:17:05 +0000 (UTC) Subject: [master] 62434adfd Polish vtc(7) gzip bits Message-ID: <20200220121705.D8BB9109E66@lists.varnish-cache.org> commit 62434adfd6913827585fabc074ed8749f31b8943 Author: Dridi Boukelmoune Date: Thu Feb 20 11:46:09 2020 +0100 Polish vtc(7) gzip bits diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c index 382a55c5a..85441f25f 100644 --- a/bin/varnishtest/vtc_http.c +++ b/bin/varnishtest/vtc_http.c @@ -938,18 +938,18 @@ http_tx_parse_args(char * const *av, struct vtclog *vl, struct http *hp, * Generate and input a body that is NUMBER bytes-long. * * \-gziplevel NUMBER - * Set the gzip level (call it before any of the other gzip - * switches). + * Set the gzip level (call it before any of the other gzip + * switches). * * \-gzipresidual NUMBER * Add extra gzip bits. You should never need it. * * \-gzipbody STRING - * Zip STRING and send it as body. + * Gzip STRING and send it as body. * * \-gziplen NUMBER - * Combine -body and -gzipbody: create a body of length NUMBER, - * zip it and send as body. + * Combine -bodylen and -gzipbody: generate a string of length + * NUMBER, gzip it and send as body. */ /********************************************************************** diff --git a/bin/varnishtest/vtc_http2.c b/bin/varnishtest/vtc_http2.c index ab78ed162..b47295af1 100644 --- a/bin/varnishtest/vtc_http2.c +++ b/bin/varnishtest/vtc_http2.c @@ -1349,14 +1349,14 @@ cmd_sendhex(CMD_ARGS) * Same as ``-body`` but content is read from FILE. * * \-bodylen INT (txreq, txresp) - * Do the same thing as ``-body`` but generate an string of INT length + * Do the same thing as ``-body`` but generate a string of INT length * for you. * * \-gzipbody STRING (txreq, txresp) * Gzip STRING and send it as body. * * \-gziplen NUMBER (txreq, txresp) - * Combine -body and -gzipbody: create a body of length NUMBER, + * Combine -bodylen and -gzipbody: generate a string of length NUMBER, * gzip it and send as body. * * \-nostrend (txreq, txresp) From dridi.boukelmoune at gmail.com Thu Feb 20 12:17:05 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 20 Feb 2020 12:17:05 +0000 (UTC) Subject: [master] a553431cd Initial release notes for Varnish 6.3.0 Message-ID: <20200220121705.F136D109E69@lists.varnish-cache.org> commit a553431cd11b8e9436faa647f404885c640656da Author: Dridi Boukelmoune Date: Thu Feb 20 12:04:31 2020 +0100 Initial release notes for Varnish 6.3.0 This is a result of a commit review between 6.2.0 and 6.3.0 and while it should in theory be comprehensive, it is not. The upgrade notes in particular are developer-centric, which may be a sign that none of the changes require attention during an upgrade it could as well be an oversight. There is one item that I identified that should be mentioned in the upgrade guide. I invite @nigoroll to give a shot at the VCL auto state. In the absence of a fair warning, I extended WS_Reserve's grace period. I omitted the -rcvbuf action on purpose, for the same reason why it is not documented in vtc(7) on purpose: it's currently used in conjunction with vmod_debug and we don't ship it. While this is a milestone, I don't consider this work to be done so I didn't remove the notes saying so. We should also probably partially pick 53b4d2f58202 on the 6.3 branch. diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c index 687d15d2d..39a91992f 100644 --- a/bin/varnishd/cache/cache_ws.c +++ b/bin/varnishd/cache/cache_ws.c @@ -275,7 +275,7 @@ WS_ReserveSize(struct ws *ws, unsigned bytes) return (pdiff(ws->f, ws->r)); } -/* REL_20200315 remove */ +/* REL_20200915 remove */ unsigned WS_Reserve(struct ws *ws, unsigned bytes) { diff --git a/doc/sphinx/whats-new/changes-6.3.rst b/doc/sphinx/whats-new/changes-6.3.rst index 377b73d8f..d73a8c930 100644 --- a/doc/sphinx/whats-new/changes-6.3.rst +++ b/doc/sphinx/whats-new/changes-6.3.rst @@ -23,28 +23,82 @@ varnishd Parameters ~~~~~~~~~~ -**XXX changes in -p parameters** +A new :ref:`ref_param_pipe_sess_max` parameter allows to limit the number of +concurrent pipe transactions. The default value is zero and means unlimited, +for backwards compatibility. Other changes in varnishd ~~~~~~~~~~~~~~~~~~~~~~~~~ +The transferred bytes accounting for HTTP/2 sessions is more accurate: +``ReqAcct`` log records no longer report a full delivery if a stream did +not complete. + +The meaning of VCL temperature changed for the ``auto`` state: it used to +automatically cool down a VCL transitioning from active to available, but +that VCL would remain ``cold``. It now works in both directions, and such a +cold VCL keeps the ``auto`` state and may be used or labelled immediately +without an explicit change of state to ``warm``. + +As a result, a VCL with the ``cold`` state will no longer warm up +automatically. + +The management of counters, and in particular dynamic counters (for example +appearing or disappearing when a VCL is loaded or discarded), has seen +significant performance improvements and setups involving a large amount of +backends should be more responsive. + Changes to VCL ============== VCL variables ~~~~~~~~~~~~~ -**XXX new, deprecated or removed variables, or changed semantics** +The :ref:`ref_param_timeout_idle` parameter can be overriden in VCL using the +``sess.timeout_idle`` variable. Other changes to VCL ~~~~~~~~~~~~~~~~~~~~ -**TODO: return (error);** +A new ``error`` transition to ``vcl_backend_error`` allows to purposely move +to that subroutine. It is similar to the ``synth`` transition and can +optionally take arguments. The three following statements are equivalent:: + + return (error); + return (error(503)); + return (error(503, "Service Unavailable")); + +The ``error`` transition is available in :ref:`vcl_backend_fetch` and +:ref:`vcl_backend_response`. Using an explicit ``error`` transition in +``vcl_backend_fetch`` does not increment the ``MAIN.fetch_failed`` counter. + +It is possible to import the same VMOD twice, as long as the two imports are +identical and wouldn't otherwise conflict. This allows for example included +VCL files to import the VMODs they need without preventing the including VCL +to also perform the same import. + +Similarly, it is now possible to include a VMOD under a different name:: + + import directors as dir; + + sub vcl_init { + new rr = dir.round_robin(); + } + +This can be useful for VMODs with a long name, or VMODs that could use a +more expressive name in VCL code. + +The built-in VCL turns the ``Host`` header lowercase in ``vcl_recv`` to +improve its hashing later in ``vcl_hash`` since domain names are case +insensitive. VMODs ===== -**XXX changes in the bundled VMODs** +``std.ip()`` now takes an optional ``STRING`` argument to specify a port +number or service name. + +See: :ref:`std.ip()` vsl-query(7) ============ @@ -122,25 +176,53 @@ will operate as if the ``or`` operator was used to join all the queries. A new ``-Q`` option allows you to read the query from a file instead. It can also be used multiple times and adds up to any ``-q`` option specified. -varnishadm -========== +Similar to ``-c`` or ``-b`` for client or backend transactions, +``varnishncsa(1)`` can take a ``-E`` option to include ESI transactions. -**XXX changes concerning varnishadm(1) and/or varnish-cli(7)** +``BackendStart`` log records are no longer used, but newer versions of log +utilities should still recognize deprecated records. It remains possible +to read logs written to a file with an older version of ``varnishlog(1)``, +and that backward compatibility officially goes as far as Varnish 6.0.0 +even though it *may* be possible to read logs saved from older releases. + +``Debug`` records are no longer logged by default and can be removed from +the :ref:`ref_param_vsl_mask` parameter to appear in the logs. Since such +records are not meant for production they are only automatically enabled +by ``varnishtest(1)``. varnishstat =========== -**XXX changes concerning varnishstat(1) and/or varnish-counters(7)** +A new ``MAIN.n_pipe`` gauge keeps track of the number of ongoing pipe +transactions. + +A new ``MAIN.pipe_limited`` counter keeps track of how many times a +transaction failed to turn into a pipe because of the +:ref:`ref_param_pipe_sess_max` parameter. varnishtest =========== -**XXX changes concerning varnishtest(1) and/or vtc(7)** +A ``client`` can now use the ``-method`` action for ``txreq`` commands to +specify the request method. This used to be done with ``-req`` which remains +as an alias for compatibility. + +A ``client`` or ``server`` may use the ``-bodyfrom`` action for respectively +``txreq`` or ``txresp`` commands to send a body from a file. + +An HTTP/2 ``client`` or ``server`` can work with gzip content encoding and has +access to ``-gzipbody`` and ``-gziplen``. Changes for developers and VMOD authors ======================================= -**XXX changes concerning VRT, the public APIs, source code organization, -builds etc.** +The most notable change for VMOD developers is the deprecation of string lists +in favor of strands. + +As usual, new functions were added to VRT, and others were changed or removed. +See ``vrt.h`` for a list of changes since the 6.2.0 release. + +We continue to remove functions from VRT that weren't meant to be used by VMOD +authors and were only part of the VMOD infrastructure code. *eof* diff --git a/doc/sphinx/whats-new/upgrading-6.3.rst b/doc/sphinx/whats-new/upgrading-6.3.rst index dfd4f3510..df269e077 100644 --- a/doc/sphinx/whats-new/upgrading-6.3.rst +++ b/doc/sphinx/whats-new/upgrading-6.3.rst @@ -14,20 +14,42 @@ features are listed in "Changes". Explicitly mention what does *not* have to be changed, especially in VCL. May include, but is not limited to:** -* Elements of VCL that have been removed or are deprecated, or whose - semantics have changed. +TODO: a word on VCL temperature and the ``auto`` change. -* -p parameters that have been removed or are deprecated, or whose - semantics have changed. +For developers and authors of VMODs and API clients +=================================================== -* Changes in the CLI. +The Python 2 EOL is approaching and our build system now favors Python 3. In +the 2020-03-15 release we plan to only support Python 3. -* Changes in the output or interpretation of stats or the log, including - changes affecting varnishncsa/-hist/-top. +The "vararg" ``VCL_STRING_LIST`` type is superseded by the array-based +``VCL_STRANDS`` type. The deprecated string list will eventually be removed +entirely and VMOD authors are strongly encouraged to convert to strands. +VRT functions working with string list arguments now take strands. -* Changes that may be necessary in VTCs or in the use of varnishtest. +More functions such as ``VRT_Vmod_Init()`` and ``VRT_Vmod_Unload()`` from +the VRT namespace moved away to the Varnish Private Interface (VPI). Such +functions were never intended for VMODs in the first place. -* Changes in public APIs that may require changes in VMODs or VAPI/VUT - clients. +The functions ``VRT_ref_vcl()`` and ``VRT_rel_vcl()`` were renamed to +respectively ``VRT_VCL_Prevent_Discard()`` and ``VRT_VCL_Allow_Discard()``. + +Some functions taking ``VCL_IP`` arguments now take a ``VRT_CTX`` in order +to fail in the presence of an invalid IP address. + +See ``vrt.h`` for a list of changes since the 6.2.0 release. + +We sometimes use Coccinelle_ to automate C code refactoring throughout the +code base. Our collection of semantic patches may be used by VMOD and API +clients authors and are available in the Varnish source tree in the +``tools/coccinelle/`` directory. + +.. _Coccinelle: http://coccinelle.lip6.fr/ + +The ``WS_Reserve()`` function is deprecated and replaced by two functions +``WS_ReserveAll()`` and ``WS_ReserveSize()`` to avoid ambiguous situations. +Its removal is planned for the 2020-09-15 release. + +A ``ws_reserve.cocci`` semantic patch can help with the transition. *eof* From dridi.boukelmoune at gmail.com Thu Feb 20 16:00:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 20 Feb 2020 16:00:07 +0000 (UTC) Subject: [master] b0e6629ae Revert "Fix test temporarily" Message-ID: <20200220160007.95ECB10F6DD@lists.varnish-cache.org> commit b0e6629ae87db8ce41df02b101ed4c9e66b75190 Author: Dridi Boukelmoune Date: Thu Feb 20 16:05:05 2020 +0100 Revert "Fix test temporarily" This reverts commit 54e7d45b7553791ea64fcf8376e23afd1034bc8e. However instead of reverting to the previous domain, I picked a different one that I hope will not turn out controversial. diff --git a/bin/varnishtest/tests/v00016.vtc b/bin/varnishtest/tests/v00016.vtc index 10c80fbf3..7a0252320 100644 --- a/bin/varnishtest/tests/v00016.vtc +++ b/bin/varnishtest/tests/v00016.vtc @@ -85,7 +85,7 @@ varnish v1 -errvcl {Regexp compilation error:} { varnish v1 -errvcl {resolves to too many addresses} { backend b { .host = "127.0.0.1"; } sub vcl_recv { - if (remote.ip == "www.varnish-cache.org") {} + if (remote.ip == "fastly.com") {} } } From dridi.boukelmoune at gmail.com Thu Feb 20 16:01:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 20 Feb 2020 16:01:07 +0000 (UTC) Subject: [6.3] a20afca54 Rename trunk release notes to 6.3 Message-ID: <20200220160107.21FDA10F89E@lists.varnish-cache.org> commit a20afca5405a32e0ff2b983aeea086bb8920c4f8 Author: Dridi Boukelmoune Date: Wed Feb 19 11:59:46 2020 +0100 Rename trunk release notes to 6.3 They are however far from complete, and that's an UNDERSTATEMENT. diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-6.3.rst similarity index 95% rename from doc/sphinx/whats-new/changes-trunk.rst rename to doc/sphinx/whats-new/changes-6.3.rst index 7c26ae580..377b73d8f 100644 --- a/doc/sphinx/whats-new/changes-trunk.rst +++ b/doc/sphinx/whats-new/changes-6.3.rst @@ -2,14 +2,14 @@ updates for changes in the development branch. For changes in the released versions of Varnish, see:** :ref:`whats-new-index` -.. _whatsnew_changes_CURRENT: +.. _whatsnew_changes_6.3: -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -Changes in Varnish **$NEXT_RELEASE** -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%% +Changes in Varnish 6.3 +%%%%%%%%%%%%%%%%%%%%%% For information about updating your current Varnish deployment to the -new version, see :ref:`whatsnew_upgrading_CURRENT`. +new version, see :ref:`whatsnew_upgrading_6.3`. A more detailed and technical account of changes in Varnish, with links to issues that have been fixed and pull requests that have been diff --git a/doc/sphinx/whats-new/index.rst b/doc/sphinx/whats-new/index.rst index 510027c08..50084dd3f 100644 --- a/doc/sphinx/whats-new/index.rst +++ b/doc/sphinx/whats-new/index.rst @@ -8,19 +8,14 @@ This section describes the changes and improvements between different versions of Varnish, and what upgrading between the different versions entail. -Varnish **$NEXT_RELEASE** -------------------------- - -**Note: These are working documents for a future release, with running -updates for changes in the development branch. For changes in the -released versions of Varnish, see the chapters listed below.** - +Varnish 6.3 +----------- .. toctree:: :maxdepth: 2 - changes-trunk - upgrading-trunk + changes-6.3 + upgrading-6.3 Varnish 6.2 ----------- diff --git a/doc/sphinx/whats-new/upgrading-trunk.rst b/doc/sphinx/whats-new/upgrading-6.3.rst similarity index 86% rename from doc/sphinx/whats-new/upgrading-trunk.rst rename to doc/sphinx/whats-new/upgrading-6.3.rst index 6143fde99..dfd4f3510 100644 --- a/doc/sphinx/whats-new/upgrading-trunk.rst +++ b/doc/sphinx/whats-new/upgrading-6.3.rst @@ -2,11 +2,11 @@ updates for changes in the development branch. For changes in the released versions of Varnish, see:** :ref:`whats-new-index` -.. _whatsnew_upgrading_CURRENT: +.. _whatsnew_upgrading_6.3: -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% -Upgrading to Varnish **$NEXT_RELEASE** -%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% +%%%%%%%%%%%%%%%%%%%%%%%% +Upgrading to Varnish 6.3 +%%%%%%%%%%%%%%%%%%%%%%%% **XXX: how to upgrade from previous deployments to this version. Limited to work that has to be done for an upgrade, new From dridi.boukelmoune at gmail.com Thu Feb 20 16:01:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 20 Feb 2020 16:01:07 +0000 (UTC) Subject: [6.3] 8bf140bb0 Initial release notes for Varnish 6.3.0 Message-ID: <20200220160107.40F8210F8A1@lists.varnish-cache.org> commit 8bf140bb04e6c1fda67b957db8091d2996e8f47a Author: Dridi Boukelmoune Date: Thu Feb 20 12:04:31 2020 +0100 Initial release notes for Varnish 6.3.0 This is a result of a commit review between 6.2.0 and 6.3.0 and while it should in theory be comprehensive, it is not. The upgrade notes in particular are developer-centric, which may be a sign that none of the changes require attention during an upgrade it could as well be an oversight. There is one item that I identified that should be mentioned in the upgrade guide. I invite @nigoroll to give a shot at the VCL auto state. In the absence of a fair warning, I extended WS_Reserve's grace period. I omitted the -rcvbuf action on purpose, for the same reason why it is not documented in vtc(7) on purpose: it's currently used in conjunction with vmod_debug and we don't ship it. While this is a milestone, I don't consider this work to be done so I didn't remove the notes saying so. We should also probably partially pick 53b4d2f58202 on the 6.3 branch. diff --git a/bin/varnishd/cache/cache_ws.c b/bin/varnishd/cache/cache_ws.c index 799bc0aa3..834158e13 100644 --- a/bin/varnishd/cache/cache_ws.c +++ b/bin/varnishd/cache/cache_ws.c @@ -273,7 +273,7 @@ WS_ReserveSize(struct ws *ws, unsigned bytes) return (pdiff(ws->f, ws->r)); } -/* REL_20200315 remove */ +/* REL_20200915 remove */ unsigned WS_Reserve(struct ws *ws, unsigned bytes) { diff --git a/doc/sphinx/whats-new/changes-6.3.rst b/doc/sphinx/whats-new/changes-6.3.rst index 377b73d8f..d73a8c930 100644 --- a/doc/sphinx/whats-new/changes-6.3.rst +++ b/doc/sphinx/whats-new/changes-6.3.rst @@ -23,28 +23,82 @@ varnishd Parameters ~~~~~~~~~~ -**XXX changes in -p parameters** +A new :ref:`ref_param_pipe_sess_max` parameter allows to limit the number of +concurrent pipe transactions. The default value is zero and means unlimited, +for backwards compatibility. Other changes in varnishd ~~~~~~~~~~~~~~~~~~~~~~~~~ +The transferred bytes accounting for HTTP/2 sessions is more accurate: +``ReqAcct`` log records no longer report a full delivery if a stream did +not complete. + +The meaning of VCL temperature changed for the ``auto`` state: it used to +automatically cool down a VCL transitioning from active to available, but +that VCL would remain ``cold``. It now works in both directions, and such a +cold VCL keeps the ``auto`` state and may be used or labelled immediately +without an explicit change of state to ``warm``. + +As a result, a VCL with the ``cold`` state will no longer warm up +automatically. + +The management of counters, and in particular dynamic counters (for example +appearing or disappearing when a VCL is loaded or discarded), has seen +significant performance improvements and setups involving a large amount of +backends should be more responsive. + Changes to VCL ============== VCL variables ~~~~~~~~~~~~~ -**XXX new, deprecated or removed variables, or changed semantics** +The :ref:`ref_param_timeout_idle` parameter can be overriden in VCL using the +``sess.timeout_idle`` variable. Other changes to VCL ~~~~~~~~~~~~~~~~~~~~ -**TODO: return (error);** +A new ``error`` transition to ``vcl_backend_error`` allows to purposely move +to that subroutine. It is similar to the ``synth`` transition and can +optionally take arguments. The three following statements are equivalent:: + + return (error); + return (error(503)); + return (error(503, "Service Unavailable")); + +The ``error`` transition is available in :ref:`vcl_backend_fetch` and +:ref:`vcl_backend_response`. Using an explicit ``error`` transition in +``vcl_backend_fetch`` does not increment the ``MAIN.fetch_failed`` counter. + +It is possible to import the same VMOD twice, as long as the two imports are +identical and wouldn't otherwise conflict. This allows for example included +VCL files to import the VMODs they need without preventing the including VCL +to also perform the same import. + +Similarly, it is now possible to include a VMOD under a different name:: + + import directors as dir; + + sub vcl_init { + new rr = dir.round_robin(); + } + +This can be useful for VMODs with a long name, or VMODs that could use a +more expressive name in VCL code. + +The built-in VCL turns the ``Host`` header lowercase in ``vcl_recv`` to +improve its hashing later in ``vcl_hash`` since domain names are case +insensitive. VMODs ===== -**XXX changes in the bundled VMODs** +``std.ip()`` now takes an optional ``STRING`` argument to specify a port +number or service name. + +See: :ref:`std.ip()` vsl-query(7) ============ @@ -122,25 +176,53 @@ will operate as if the ``or`` operator was used to join all the queries. A new ``-Q`` option allows you to read the query from a file instead. It can also be used multiple times and adds up to any ``-q`` option specified. -varnishadm -========== +Similar to ``-c`` or ``-b`` for client or backend transactions, +``varnishncsa(1)`` can take a ``-E`` option to include ESI transactions. -**XXX changes concerning varnishadm(1) and/or varnish-cli(7)** +``BackendStart`` log records are no longer used, but newer versions of log +utilities should still recognize deprecated records. It remains possible +to read logs written to a file with an older version of ``varnishlog(1)``, +and that backward compatibility officially goes as far as Varnish 6.0.0 +even though it *may* be possible to read logs saved from older releases. + +``Debug`` records are no longer logged by default and can be removed from +the :ref:`ref_param_vsl_mask` parameter to appear in the logs. Since such +records are not meant for production they are only automatically enabled +by ``varnishtest(1)``. varnishstat =========== -**XXX changes concerning varnishstat(1) and/or varnish-counters(7)** +A new ``MAIN.n_pipe`` gauge keeps track of the number of ongoing pipe +transactions. + +A new ``MAIN.pipe_limited`` counter keeps track of how many times a +transaction failed to turn into a pipe because of the +:ref:`ref_param_pipe_sess_max` parameter. varnishtest =========== -**XXX changes concerning varnishtest(1) and/or vtc(7)** +A ``client`` can now use the ``-method`` action for ``txreq`` commands to +specify the request method. This used to be done with ``-req`` which remains +as an alias for compatibility. + +A ``client`` or ``server`` may use the ``-bodyfrom`` action for respectively +``txreq`` or ``txresp`` commands to send a body from a file. + +An HTTP/2 ``client`` or ``server`` can work with gzip content encoding and has +access to ``-gzipbody`` and ``-gziplen``. Changes for developers and VMOD authors ======================================= -**XXX changes concerning VRT, the public APIs, source code organization, -builds etc.** +The most notable change for VMOD developers is the deprecation of string lists +in favor of strands. + +As usual, new functions were added to VRT, and others were changed or removed. +See ``vrt.h`` for a list of changes since the 6.2.0 release. + +We continue to remove functions from VRT that weren't meant to be used by VMOD +authors and were only part of the VMOD infrastructure code. *eof* diff --git a/doc/sphinx/whats-new/upgrading-6.3.rst b/doc/sphinx/whats-new/upgrading-6.3.rst index dfd4f3510..df269e077 100644 --- a/doc/sphinx/whats-new/upgrading-6.3.rst +++ b/doc/sphinx/whats-new/upgrading-6.3.rst @@ -14,20 +14,42 @@ features are listed in "Changes". Explicitly mention what does *not* have to be changed, especially in VCL. May include, but is not limited to:** -* Elements of VCL that have been removed or are deprecated, or whose - semantics have changed. +TODO: a word on VCL temperature and the ``auto`` change. -* -p parameters that have been removed or are deprecated, or whose - semantics have changed. +For developers and authors of VMODs and API clients +=================================================== -* Changes in the CLI. +The Python 2 EOL is approaching and our build system now favors Python 3. In +the 2020-03-15 release we plan to only support Python 3. -* Changes in the output or interpretation of stats or the log, including - changes affecting varnishncsa/-hist/-top. +The "vararg" ``VCL_STRING_LIST`` type is superseded by the array-based +``VCL_STRANDS`` type. The deprecated string list will eventually be removed +entirely and VMOD authors are strongly encouraged to convert to strands. +VRT functions working with string list arguments now take strands. -* Changes that may be necessary in VTCs or in the use of varnishtest. +More functions such as ``VRT_Vmod_Init()`` and ``VRT_Vmod_Unload()`` from +the VRT namespace moved away to the Varnish Private Interface (VPI). Such +functions were never intended for VMODs in the first place. -* Changes in public APIs that may require changes in VMODs or VAPI/VUT - clients. +The functions ``VRT_ref_vcl()`` and ``VRT_rel_vcl()`` were renamed to +respectively ``VRT_VCL_Prevent_Discard()`` and ``VRT_VCL_Allow_Discard()``. + +Some functions taking ``VCL_IP`` arguments now take a ``VRT_CTX`` in order +to fail in the presence of an invalid IP address. + +See ``vrt.h`` for a list of changes since the 6.2.0 release. + +We sometimes use Coccinelle_ to automate C code refactoring throughout the +code base. Our collection of semantic patches may be used by VMOD and API +clients authors and are available in the Varnish source tree in the +``tools/coccinelle/`` directory. + +.. _Coccinelle: http://coccinelle.lip6.fr/ + +The ``WS_Reserve()`` function is deprecated and replaced by two functions +``WS_ReserveAll()`` and ``WS_ReserveSize()`` to avoid ambiguous situations. +Its removal is planned for the 2020-09-15 release. + +A ``ws_reserve.cocci`` semantic patch can help with the transition. *eof* From dridi.boukelmoune at gmail.com Thu Feb 20 16:01:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 20 Feb 2020 16:01:07 +0000 (UTC) Subject: [6.3] 93231dde2 Revert "Fix test temporarily" Message-ID: <20200220160107.5FD0610F8A5@lists.varnish-cache.org> commit 93231dde2dd461c5960fc32351c99199feaca235 Author: Dridi Boukelmoune Date: Thu Feb 20 16:05:05 2020 +0100 Revert "Fix test temporarily" This reverts commit 54e7d45b7553791ea64fcf8376e23afd1034bc8e. However instead of reverting to the previous domain, I picked a different one that I hope will not turn out controversial. diff --git a/bin/varnishtest/tests/v00016.vtc b/bin/varnishtest/tests/v00016.vtc index 56ba92c4f..78f15cbe9 100644 --- a/bin/varnishtest/tests/v00016.vtc +++ b/bin/varnishtest/tests/v00016.vtc @@ -85,7 +85,7 @@ varnish v1 -errvcl {Regexp compilation error:} { varnish v1 -errvcl {resolves to too many addresses} { backend b { .host = "127.0.0.1"; } sub vcl_recv { - if (remote.ip == "www.varnish-cache.org") {} + if (remote.ip == "fastly.com") {} } } From dridi.boukelmoune at gmail.com Thu Feb 20 20:31:11 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 20 Feb 2020 20:31:11 +0000 (UTC) Subject: [master] 7ac98ce6c Get the correct error string for EAI_SYSTEM Message-ID: <20200220203111.6A167114D95@lists.varnish-cache.org> commit 7ac98ce6c82089f4d59b378184218031d0f40c40 Author: Dridi Boukelmoune Date: Thu Feb 20 20:46:21 2020 +0100 Get the correct error string for EAI_SYSTEM diff --git a/lib/libvarnish/vss.c b/lib/libvarnish/vss.c index 24056c4da..9feb7c584 100644 --- a/lib/libvarnish/vss.c +++ b/lib/libvarnish/vss.c @@ -131,7 +131,9 @@ vss_resolve(const char *addr, const char *def_port, int family, int socktype, ret = getaddrinfo(hp, def_port, &hints, res); free(p); - if (ret != 0) + if (ret == EAI_SYSTEM) + *errp = vstrerror(errno); + else if (ret != 0) *errp = gai_strerror(ret); return (ret); From dridi.boukelmoune at gmail.com Thu Feb 20 20:31:11 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 20 Feb 2020 20:31:11 +0000 (UTC) Subject: [master] f4fdc0238 Allocate rss on the stack Message-ID: <20200220203111.7EB03114D97@lists.varnish-cache.org> commit f4fdc0238c0cc44e38d0215dabe98c32a2c06c78 Author: Dridi Boukelmoune Date: Thu Feb 20 20:46:56 2020 +0100 Allocate rss on the stack diff --git a/lib/libvcc/vcc_utils.c b/lib/libvcc/vcc_utils.c index ea38f69f3..efe8f4861 100644 --- a/lib/libvcc/vcc_utils.c +++ b/lib/libvcc/vcc_utils.c @@ -199,7 +199,7 @@ Resolve_Sockaddr(struct vcc *tl, const char *errid) { int error; - struct rss *rss; + struct rss rss[1]; const char *err; *ipv4 = NULL; @@ -207,8 +207,7 @@ Resolve_Sockaddr(struct vcc *tl, if (p_ascii != NULL) *p_ascii = NULL; - ALLOC_OBJ(rss, RSS_MAGIC); - AN(rss); + INIT_OBJ(rss, RSS_MAGIC); rss->vsb = VSB_new_auto(); AN(rss->vsb); @@ -224,7 +223,7 @@ Resolve_Sockaddr(struct vcc *tl, free(rss->vsa4); free(rss->vsa6); VSB_destroy(&rss->vsb); - FREE_OBJ(rss); + ZERO_OBJ(rss, sizeof rss); return; } AZ(error); @@ -255,7 +254,7 @@ Resolve_Sockaddr(struct vcc *tl, vcc_ErrWhere(tl, t_err); } VSB_destroy(&rss->vsb); - FREE_OBJ(rss); + ZERO_OBJ(rss, sizeof rss); } /* From phk at FreeBSD.org Fri Feb 21 05:22:09 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 21 Feb 2020 05:22:09 +0000 (UTC) Subject: [master] dac871dc3 Always initialize req->req_body_status Message-ID: <20200221052209.3CD5A62C70@lists.varnish-cache.org> commit dac871dc33ef656c7c81fc988f2bd55b9f6722fa Author: Poul-Henning Kamp Date: Wed Feb 19 18:31:31 2020 +0000 Always initialize req->req_body_status This worked only because REQ_BODY_INIT = 0 I hate C enums. diff --git a/bin/varnishd/cache/cache_req.c b/bin/varnishd/cache/cache_req.c index 61c0182e8..305cea641 100644 --- a/bin/varnishd/cache/cache_req.c +++ b/bin/varnishd/cache/cache_req.c @@ -94,6 +94,7 @@ Req_New(const struct worker *wrk, struct sess *sp) AN(req); req->magic = REQ_MAGIC; req->sp = sp; + req->req_body_status = REQ_BODY_INIT; e = (char*)req + sz; p = (char*)(req + 1); From phk at FreeBSD.org Fri Feb 21 05:22:09 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 21 Feb 2020 05:22:09 +0000 (UTC) Subject: [master] 03af8e7b9 Use bo->no_retry to prevent automatic retry on dead connections. Message-ID: <20200221052209.50D7062C73@lists.varnish-cache.org> commit 03af8e7b91b5fa47d39759fba7f737f0148307b7 Author: Poul-Henning Kamp Date: Wed Feb 19 18:47:01 2020 +0000 Use bo->no_retry to prevent automatic retry on dead connections. diff --git a/bin/varnishd/cache/cache_backend.c b/bin/varnishd/cache/cache_backend.c index 1ce50846c..0a685a4f3 100644 --- a/bin/varnishd/cache/cache_backend.c +++ b/bin/varnishd/cache/cache_backend.c @@ -319,9 +319,7 @@ vbe_dir_gethdrs(VRT_CTX, VCL_BACKEND d) AZ(bo->htc); if (i < 0 || extrachance == 0) break; - if (bo->req != NULL && - bo->req->req_body_status != REQ_BODY_NONE && - bo->req->req_body_status != REQ_BODY_CACHED) + if (bo->no_retry != NULL) break; VSC_C_main->backend_retry++; } while (extrachance--); From phk at FreeBSD.org Fri Feb 21 05:22:09 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 21 Feb 2020 05:22:09 +0000 (UTC) Subject: [master] b6765cc4b We know the length of cached req.body. so chunked never happens. Message-ID: <20200221052209.6AA3162C76@lists.varnish-cache.org> commit b6765cc4bfe66d1038da063e4ee667c21c623ed9 Author: Poul-Henning Kamp Date: Fri Feb 21 05:15:53 2020 +0000 We know the length of cached req.body. so chunked never happens. diff --git a/bin/varnishd/http1/cache_http1_fetch.c b/bin/varnishd/http1/cache_http1_fetch.c index 64547f8e3..075a412d6 100644 --- a/bin/varnishd/http1/cache_http1_fetch.c +++ b/bin/varnishd/http1/cache_http1_fetch.c @@ -108,12 +108,9 @@ V1F_SendReq(struct worker *wrk, struct busyobj *bo, uint64_t *ctr_hdrbytes, if (bo->bereq_body != NULL) { AZ(bo->req); - if (do_chunked) - V1L_Chunked(wrk); + AZ(do_chunked); (void)ObjIterate(bo->wrk, bo->bereq_body, bo, vbf_iter_req_body, 0); - if (do_chunked) - V1L_EndChunk(wrk); } else if (bo->req != NULL && bo->req->req_body_status != REQ_BODY_NONE) { if (do_chunked) From phk at FreeBSD.org Fri Feb 21 08:34:06 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 21 Feb 2020 08:34:06 +0000 (UTC) Subject: [master] 8ebcfc4bb Inline RFC2616_Response_Body() in V1F, it is not transport agnostic. Message-ID: <20200221083406.AD564100487@lists.varnish-cache.org> commit 8ebcfc4bb27fda161fc26ca02638f26091c356fa Author: Poul-Henning Kamp Date: Fri Feb 21 05:35:36 2020 +0000 Inline RFC2616_Response_Body() in V1F, it is not transport agnostic. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 0d66cea58..e2274dd92 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -812,7 +812,6 @@ unsigned RFC2616_Req_Gzip(const struct http *); int RFC2616_Do_Cond(const struct req *sp); void RFC2616_Weaken_Etag(struct http *hp); void RFC2616_Vary_AE(struct http *hp); -void RFC2616_Response_Body(const struct worker *, const struct busyobj *); /* * A normal pointer difference is signed, but we never want a negative value diff --git a/bin/varnishd/cache/cache_rfc2616.c b/bin/varnishd/cache/cache_rfc2616.c index 77f30f92c..14e655c49 100644 --- a/bin/varnishd/cache/cache_rfc2616.c +++ b/bin/varnishd/cache/cache_rfc2616.c @@ -340,68 +340,3 @@ RFC2616_Vary_AE(struct http *hp) http_SetHeader(hp, "Vary: Accept-Encoding"); } } - -/*--------------------------------------------------------------------*/ - -void -RFC2616_Response_Body(const struct worker *wrk, const struct busyobj *bo) -{ - - CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); - CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); - - /* - * Figure out how the fetch is supposed to happen, before the - * headers are adultered by VCL - */ - if (!strcasecmp(http_GetMethod(bo->bereq), "head")) { - /* - * A HEAD request can never have a body in the reply, - * no matter what the headers might say. - * [RFC7231 4.3.2 p25] - */ - wrk->stats->fetch_head++; - bo->htc->body_status = BS_NONE; - } else if (http_GetStatus(bo->beresp) <= 199) { - /* - * 1xx responses never have a body. - * [RFC7230 3.3.2 p31] - * ... but we should never see them. - */ - wrk->stats->fetch_1xx++; - bo->htc->body_status = BS_ERROR; - } else if (http_IsStatus(bo->beresp, 204)) { - /* - * 204 is "No Content", obviously don't expect a body. - * [RFC7230 3.3.1 p29 and 3.3.2 p31] - */ - wrk->stats->fetch_204++; - if ((http_GetHdr(bo->beresp, H_Content_Length, NULL) && - bo->htc->content_length != 0) || - http_GetHdr(bo->beresp, H_Transfer_Encoding, NULL)) - bo->htc->body_status = BS_ERROR; - else - bo->htc->body_status = BS_NONE; - } else if (http_IsStatus(bo->beresp, 304)) { - /* - * 304 is "Not Modified" it has no body. - * [RFC7230 3.3 p28] - */ - wrk->stats->fetch_304++; - bo->htc->body_status = BS_NONE; - } else if (bo->htc->body_status == BS_CHUNKED) { - wrk->stats->fetch_chunked++; - } else if (bo->htc->body_status == BS_LENGTH) { - assert(bo->htc->content_length > 0); - wrk->stats->fetch_length++; - } else if (bo->htc->body_status == BS_EOF) { - wrk->stats->fetch_eof++; - } else if (bo->htc->body_status == BS_ERROR) { - wrk->stats->fetch_bad++; - } else if (bo->htc->body_status == BS_NONE) { - wrk->stats->fetch_none++; - } else { - WRONG("wrong bodystatus"); - } -} - diff --git a/bin/varnishd/http1/cache_http1_fetch.c b/bin/varnishd/http1/cache_http1_fetch.c index 075a412d6..4c5b593a1 100644 --- a/bin/varnishd/http1/cache_http1_fetch.c +++ b/bin/varnishd/http1/cache_http1_fetch.c @@ -237,7 +237,60 @@ V1F_FetchRespHdr(struct busyobj *bo) } htc->doclose = http_DoConnection(hp); - RFC2616_Response_Body(bo->wrk, bo); + + /* + * Figure out how the fetch is supposed to happen, before the + * headers are adultered by VCL + */ + if (!strcasecmp(http_GetMethod(bo->bereq), "head")) { + /* + * A HEAD request can never have a body in the reply, + * no matter what the headers might say. + * [RFC7231 4.3.2 p25] + */ + bo->wrk->stats->fetch_head++; + bo->htc->body_status = BS_NONE; + } else if (http_GetStatus(bo->beresp) <= 199) { + /* + * 1xx responses never have a body. + * [RFC7230 3.3.2 p31] + * ... but we should never see them. + */ + bo->wrk->stats->fetch_1xx++; + bo->htc->body_status = BS_ERROR; + } else if (http_IsStatus(bo->beresp, 204)) { + /* + * 204 is "No Content", obviously don't expect a body. + * [RFC7230 3.3.1 p29 and 3.3.2 p31] + */ + bo->wrk->stats->fetch_204++; + if ((http_GetHdr(bo->beresp, H_Content_Length, NULL) && + bo->htc->content_length != 0) || + http_GetHdr(bo->beresp, H_Transfer_Encoding, NULL)) + bo->htc->body_status = BS_ERROR; + else + bo->htc->body_status = BS_NONE; + } else if (http_IsStatus(bo->beresp, 304)) { + /* + * 304 is "Not Modified" it has no body. + * [RFC7230 3.3 p28] + */ + bo->wrk->stats->fetch_304++; + bo->htc->body_status = BS_NONE; + } else if (bo->htc->body_status == BS_CHUNKED) { + bo->wrk->stats->fetch_chunked++; + } else if (bo->htc->body_status == BS_LENGTH) { + assert(bo->htc->content_length > 0); + bo->wrk->stats->fetch_length++; + } else if (bo->htc->body_status == BS_EOF) { + bo->wrk->stats->fetch_eof++; + } else if (bo->htc->body_status == BS_ERROR) { + bo->wrk->stats->fetch_bad++; + } else if (bo->htc->body_status == BS_NONE) { + bo->wrk->stats->fetch_none++; + } else { + WRONG("wrong bodystatus"); + } assert(bo->vfc->resp == bo->beresp); if (bo->htc->body_status != BS_NONE && From phk at FreeBSD.org Fri Feb 21 08:34:06 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 21 Feb 2020 08:34:06 +0000 (UTC) Subject: [master] acd0cd4a8 Convert body_status from enum to const struct * Message-ID: <20200221083406.C606910048A@lists.varnish-cache.org> commit acd0cd4a81e93ab76162178ddb41acd4e0bf4018 Author: Poul-Henning Kamp Date: Fri Feb 21 06:55:58 2020 +0000 Convert body_status from enum to const struct * diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index e2274dd92..2b36acc0b 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -57,11 +57,18 @@ /*--------------------------------------------------------------------*/ -enum body_status { -#define BODYSTATUS(U,l) BS_##U, -#include "tbl/body_status.h" +struct body_status { + const char *name; + int nbr; + int avail; + int length_known; }; +#define BODYSTATUS(U, l, n, a, k) extern const struct body_status BS_##U[1]; +#include "tbl/body_status.h" + +typedef const struct body_status *body_status_t; + /*--------------------------------------------------------------------*/ enum req_body_state_e { diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index 06bd94ddc..df00367e0 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -640,7 +640,7 @@ vbf_stp_fetch(struct worker *wrk, struct busyobj *bo) } VSLb(bo->vsl, SLT_Fetch_Body, "%u %s %s", - bo->htc->body_status, body_status_2str(bo->htc->body_status), + bo->htc->body_status->nbr, bo->htc->body_status->name, bo->do_stream ? "stream" : "-"); if (bo->htc->body_status != BS_NONE) { diff --git a/bin/varnishd/cache/cache_http.c b/bin/varnishd/cache/cache_http.c index 2297121e0..91069f103 100644 --- a/bin/varnishd/cache/cache_http.c +++ b/bin/varnishd/cache/cache_http.c @@ -40,6 +40,16 @@ #include "vct.h" #include "vtim.h" +#define BODYSTATUS(U, l, n, a, k) \ + const struct body_status BS_##U[1] = {{ \ + .name = #l, \ + .nbr = n, \ + .avail = a, \ + .length_known = k \ + }}; +#include "tbl/body_status.h" + + #define HTTPH(a, b, c) char b[] = "*" a ":"; #include "tbl/http_headers.h" diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c index d5031b131..81987f89e 100644 --- a/bin/varnishd/cache/cache_panic.c +++ b/bin/varnishd/cache/cache_panic.c @@ -71,19 +71,6 @@ static void pan_req(struct vsb *, const struct req *); /*--------------------------------------------------------------------*/ -const char * -body_status_2str(enum body_status e) -{ - switch (e) { -#define BODYSTATUS(U,l) case BS_##U: return (#l); -#include "tbl/body_status.h" - default: - return ("?"); - } -} - -/*--------------------------------------------------------------------*/ - static const char * reqbody_status_2str(enum req_body_state_e e) { @@ -204,7 +191,7 @@ pan_htc(struct vsb *vsb, const struct http_conn *htc) VSB_printf(vsb, "content_length = %jd,\n", (intmax_t)htc->content_length); VSB_printf(vsb, "body_status = %s,\n", - body_status_2str(htc->body_status)); + htc->body_status ? htc->body_status->name : "NULL"); VSB_printf(vsb, "first_byte_timeout = %f,\n", htc->first_byte_timeout); VSB_printf(vsb, "between_bytes_timeout = %f,\n", diff --git a/bin/varnishd/cache/cache_varnishd.h b/bin/varnishd/cache/cache_varnishd.h index 73c758f23..e7fc7515f 100644 --- a/bin/varnishd/cache/cache_varnishd.h +++ b/bin/varnishd/cache/cache_varnishd.h @@ -67,7 +67,7 @@ struct http_conn { int *rfd; enum sess_close doclose; - enum body_status body_status; + body_status_t body_status; struct ws *ws; char *rxbuf_b; char *rxbuf_e; @@ -335,7 +335,6 @@ void ObjUnsubscribeEvents(uintptr_t *); /* cache_panic.c */ void PAN_Init(void); int PAN_already(struct vsb *, const void *); -const char *body_status_2str(enum body_status e); const char *sess_close_2str(enum sess_close sc, int want_desc); /* cache_pool.c */ diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c index 81ab2e3f6..bbce1b4fe 100644 --- a/bin/varnishd/http1/cache_http1_fsm.c +++ b/bin/varnishd/http1/cache_http1_fsm.c @@ -151,16 +151,9 @@ http1_req_body(struct req *req) { CHECK_OBJ_NOTNULL(req, REQ_MAGIC); - switch (req->htc->body_status) { - case BS_EOF: - case BS_LENGTH: - case BS_CHUNKED: - if (V1F_Setup_Fetch(req->vfc, req->htc) != 0) - req->req_body_status = REQ_BODY_FAIL; - break; - default: - break; - } + if (req->htc->body_status->avail && + V1F_Setup_Fetch(req->vfc, req->htc) != 0) + req->req_body_status = REQ_BODY_FAIL; } static void @@ -282,22 +275,16 @@ http1_dissect(struct worker *wrk, struct req *req) assert (req->req_body_status == REQ_BODY_INIT); - switch (req->htc->body_status) { - case BS_CHUNKED: + if (req->htc->body_status == BS_CHUNKED) req->req_body_status = REQ_BODY_WITHOUT_LEN; - break; - case BS_LENGTH: + else if (req->htc->body_status == BS_LENGTH) req->req_body_status = REQ_BODY_WITH_LEN; - break; - case BS_NONE: + else if (req->htc->body_status == BS_NONE) req->req_body_status = REQ_BODY_NONE; - break; - case BS_EOF: + else if (req->htc->body_status == BS_EOF) req->req_body_status = REQ_BODY_WITHOUT_LEN; - break; - default: + else WRONG("Unknown req_body_status situation"); - } return (0); } diff --git a/bin/varnishd/http1/cache_http1_proto.c b/bin/varnishd/http1/cache_http1_proto.c index 935e70e47..f30c02561 100644 --- a/bin/varnishd/http1/cache_http1_proto.c +++ b/bin/varnishd/http1/cache_http1_proto.c @@ -301,7 +301,7 @@ http1_splitline(struct http *hp, struct http_conn *htc, const int *hf, /*--------------------------------------------------------------------*/ -static enum body_status +static body_status_t http1_body_status(const struct http *hp, struct http_conn *htc, int request) { ssize_t cl; diff --git a/bin/varnishd/http1/cache_http1_vfp.c b/bin/varnishd/http1/cache_http1_vfp.c index 30d879ec2..7e4782a15 100644 --- a/bin/varnishd/http1/cache_http1_vfp.c +++ b/bin/varnishd/http1/cache_http1_vfp.c @@ -265,31 +265,26 @@ V1F_Setup_Fetch(struct vfp_ctx *vfc, struct http_conn *htc) CHECK_OBJ_NOTNULL(vfc, VFP_CTX_MAGIC); CHECK_OBJ_NOTNULL(htc, HTTP_CONN_MAGIC); - switch (htc->body_status) { - case BS_EOF: + if (htc->body_status == BS_EOF) { assert(htc->content_length == -1); vfe = VFP_Push(vfc, &v1f_eof); if (vfe == NULL) return (ENOSPC); vfe->priv2 = 0; - break; - case BS_LENGTH: + } else if (htc->body_status == BS_LENGTH) { assert(htc->content_length > 0); vfe = VFP_Push(vfc, &v1f_straight); if (vfe == NULL) return (ENOSPC); vfe->priv2 = htc->content_length; - break; - case BS_CHUNKED: + } else if (htc->body_status == BS_CHUNKED) { assert(htc->content_length == -1); vfe = VFP_Push(vfc, &v1f_chunked); if (vfe == NULL) return (ENOSPC); vfe->priv2 = -1; - break; - default: + } else { WRONG("Wrong body_status"); - break; } vfe->priv1 = htc; return (0); diff --git a/include/tbl/body_status.h b/include/tbl/body_status.h index 4ab48f1a3..878248d72 100644 --- a/include/tbl/body_status.h +++ b/include/tbl/body_status.h @@ -32,11 +32,12 @@ /*lint -save -e525 -e539 */ -BODYSTATUS(NONE, none) -BODYSTATUS(ERROR, error) -BODYSTATUS(CHUNKED, chunked) -BODYSTATUS(LENGTH, length) -BODYSTATUS(EOF, eof) +/* Upper lower nbr, avail len_known */ +BODYSTATUS(NONE, none, 0, 0, 1) +BODYSTATUS(ERROR, error, 1, 0, 0) +BODYSTATUS(CHUNKED, chunked, 2, 1, 0) +BODYSTATUS(LENGTH, length, 3, 1, 1) +BODYSTATUS(EOF, eof, 4, 1, 0) #undef BODYSTATUS /*lint -restore */ From phk at FreeBSD.org Fri Feb 21 08:34:06 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 21 Feb 2020 08:34:06 +0000 (UTC) Subject: [master] 034abbfd6 Rename REQ_BODY_FAIL to REQ_BODY_ERROR in preparation for merge with BS_* Message-ID: <20200221083406.E521510048D@lists.varnish-cache.org> commit 034abbfd6e314e10b51386f2605bbd5a4c087403 Author: Poul-Henning Kamp Date: Fri Feb 21 07:14:18 2020 +0000 Rename REQ_BODY_FAIL to REQ_BODY_ERROR in preparation for merge with BS_* diff --git a/bin/varnishd/cache/cache_req_body.c b/bin/varnishd/cache/cache_req_body.c index 2987a2bb2..4e12d8e06 100644 --- a/bin/varnishd/cache/cache_req_body.c +++ b/bin/varnishd/cache/cache_req_body.c @@ -77,7 +77,7 @@ vrb_pull(struct req *req, ssize_t maxsize, objiterate_f *func, void *priv) req->storage = NULL; if (STV_NewObject(req->wrk, req->body_oc, stv, 8) == 0) { - req->req_body_status = REQ_BODY_FAIL; + req->req_body_status = REQ_BODY_ERROR; HSH_DerefBoc(req->wrk, req->body_oc); AZ(HSH_DerefObjCore(req->wrk, &req->body_oc, 0)); (void)VFP_Error(vfc, "Object allocation failed:" @@ -88,7 +88,7 @@ vrb_pull(struct req *req, ssize_t maxsize, objiterate_f *func, void *priv) vfc->oc = req->body_oc; if (VFP_Open(vfc) < 0) { - req->req_body_status = REQ_BODY_FAIL; + req->req_body_status = REQ_BODY_ERROR; HSH_DerefBoc(req->wrk, req->body_oc); AZ(HSH_DerefObjCore(req->wrk, &req->body_oc, 0)); return (-1); @@ -136,7 +136,7 @@ vrb_pull(struct req *req, ssize_t maxsize, objiterate_f *func, void *priv) HSH_DerefBoc(req->wrk, req->body_oc); AZ(HSH_DerefObjCore(req->wrk, &req->body_oc, 0)); if (vfps != VFP_END) { - req->req_body_status = REQ_BODY_FAIL; + req->req_body_status = REQ_BODY_ERROR; if (r == 0) r = -1; } @@ -148,7 +148,7 @@ vrb_pull(struct req *req, ssize_t maxsize, objiterate_f *func, void *priv) HSH_DerefBoc(req->wrk, req->body_oc); if (vfps != VFP_END) { - req->req_body_status = REQ_BODY_FAIL; + req->req_body_status = REQ_BODY_ERROR; AZ(HSH_DerefObjCore(req->wrk, &req->body_oc, 0)); return (-1); } @@ -204,7 +204,7 @@ VRB_Iterate(struct worker *wrk, struct vsl_log *vsl, VSLb(vsl, SLT_VCL_Error, "Uncached req.body can only be consumed once."); return (-1); - case REQ_BODY_FAIL: + case REQ_BODY_ERROR: VSLb(vsl, SLT_FetchError, "Had failed reading req.body before."); return (-1); @@ -307,7 +307,7 @@ VRB_Cache(struct req *req, ssize_t maxsize) case REQ_BODY_CACHED: AZ(ObjGetU64(req->wrk, req->body_oc, OA_LEN, &u)); return (u); - case REQ_BODY_FAIL: + case REQ_BODY_ERROR: return (-1); case REQ_BODY_NONE: return (0); @@ -319,7 +319,7 @@ VRB_Cache(struct req *req, ssize_t maxsize) } if (req->htc->content_length > maxsize) { - req->req_body_status = REQ_BODY_FAIL; + req->req_body_status = REQ_BODY_ERROR; (void)VFP_Error(req->vfc, "Request body too big to cache"); return (-1); } diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index a7dbb42e3..1deb2595c 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -877,7 +877,7 @@ cnt_recv(struct worker *wrk, struct req *req) cnt_recv_prep(req, ci); - if (req->req_body_status == REQ_BODY_FAIL) { + if (req->req_body_status == REQ_BODY_ERROR) { req->doclose = SC_OVERLOAD; return (REQ_FSM_DONE); } @@ -899,7 +899,7 @@ cnt_recv(struct worker *wrk, struct req *req) } /* Attempts to cache req.body may fail */ - if (req->req_body_status == REQ_BODY_FAIL) { + if (req->req_body_status == REQ_BODY_ERROR) { req->doclose = SC_RX_BODY; return (REQ_FSM_DONE); } diff --git a/bin/varnishd/http1/cache_http1_fetch.c b/bin/varnishd/http1/cache_http1_fetch.c index 4c5b593a1..4cecb3148 100644 --- a/bin/varnishd/http1/cache_http1_fetch.c +++ b/bin/varnishd/http1/cache_http1_fetch.c @@ -120,7 +120,7 @@ V1F_SendReq(struct worker *wrk, struct busyobj *bo, uint64_t *ctr_hdrbytes, if (bo->req->req_body_status != REQ_BODY_CACHED) bo->no_retry = "req.body not cached"; - if (bo->req->req_body_status == REQ_BODY_FAIL) { + if (bo->req->req_body_status == REQ_BODY_ERROR) { /* * XXX: (#2332) We should test to see if the backend * XXX: sent us some headers explaining why. diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c index bbce1b4fe..73ea12ce0 100644 --- a/bin/varnishd/http1/cache_http1_fsm.c +++ b/bin/varnishd/http1/cache_http1_fsm.c @@ -153,7 +153,7 @@ http1_req_body(struct req *req) CHECK_OBJ_NOTNULL(req, REQ_MAGIC); if (req->htc->body_status->avail && V1F_Setup_Fetch(req->vfc, req->htc) != 0) - req->req_body_status = REQ_BODY_FAIL; + req->req_body_status = REQ_BODY_ERROR; } static void diff --git a/include/tbl/req_body.h b/include/tbl/req_body.h index 9effc0707..93c46c0bc 100644 --- a/include/tbl/req_body.h +++ b/include/tbl/req_body.h @@ -37,7 +37,7 @@ REQ_BODY(WITH_LEN) /* states >= TAKEN imply that no body is to be read */ REQ_BODY(TAKEN) REQ_BODY(CACHED) -REQ_BODY(FAIL) +REQ_BODY(ERROR) REQ_BODY(NONE) #undef REQ_BODY From phk at FreeBSD.org Fri Feb 21 08:34:07 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 21 Feb 2020 08:34:07 +0000 (UTC) Subject: [master] 255a789da Rename REQ_BODY_WITH_LEN to REQ_BODY_LENGTH in preparation for merge with BS_* Message-ID: <20200221083407.13D41100492@lists.varnish-cache.org> commit 255a789da1d53544703cb79193d4614031497087 Author: Poul-Henning Kamp Date: Fri Feb 21 07:17:46 2020 +0000 Rename REQ_BODY_WITH_LEN to REQ_BODY_LENGTH in preparation for merge with BS_* diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index df00367e0..0ac2b8338 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -249,7 +249,7 @@ vbf_stp_mkbereq(struct worker *wrk, struct busyobj *bo) HSH_Ref(bo->bereq_body); bo->req = NULL; ObjSetState(bo->wrk, bo->fetch_objcore, BOS_REQ_DONE); - } else if (bo->req->req_body_status != REQ_BODY_WITH_LEN && + } else if (bo->req->req_body_status != REQ_BODY_LENGTH && bo->req->req_body_status != REQ_BODY_WITHOUT_LEN) { WRONG("Bad req_body_status"); } diff --git a/bin/varnishd/cache/cache_req_body.c b/bin/varnishd/cache/cache_req_body.c index 4e12d8e06..08db8fdb6 100644 --- a/bin/varnishd/cache/cache_req_body.c +++ b/bin/varnishd/cache/cache_req_body.c @@ -197,7 +197,7 @@ VRB_Iterate(struct worker *wrk, struct vsl_log *vsl, return (0); case REQ_BODY_NONE: return (0); - case REQ_BODY_WITH_LEN: + case REQ_BODY_LENGTH: case REQ_BODY_WITHOUT_LEN: break; case REQ_BODY_TAKEN: @@ -212,7 +212,7 @@ VRB_Iterate(struct worker *wrk, struct vsl_log *vsl, WRONG("Wrong req_body_status in VRB_Iterate()"); } Lck_Lock(&req->sp->mtx); - if (req->req_body_status == REQ_BODY_WITH_LEN || + if (req->req_body_status == REQ_BODY_LENGTH || req->req_body_status == REQ_BODY_WITHOUT_LEN) { req->req_body_status = REQ_BODY_TAKEN; i = 0; @@ -254,7 +254,7 @@ VRB_Ignore(struct req *req) if (req->doclose) return (0); - if (req->req_body_status == REQ_BODY_WITH_LEN || + if (req->req_body_status == REQ_BODY_LENGTH || req->req_body_status == REQ_BODY_WITHOUT_LEN) (void)VRB_Iterate(req->wrk, req->vsl, req, httpq_req_body_discard, NULL); return (0); @@ -312,7 +312,7 @@ VRB_Cache(struct req *req, ssize_t maxsize) case REQ_BODY_NONE: return (0); case REQ_BODY_WITHOUT_LEN: - case REQ_BODY_WITH_LEN: + case REQ_BODY_LENGTH: break; default: WRONG("Wrong req_body_status in VRB_Cache()"); diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c index 73ea12ce0..6332aca96 100644 --- a/bin/varnishd/http1/cache_http1_fsm.c +++ b/bin/varnishd/http1/cache_http1_fsm.c @@ -278,7 +278,7 @@ http1_dissect(struct worker *wrk, struct req *req) if (req->htc->body_status == BS_CHUNKED) req->req_body_status = REQ_BODY_WITHOUT_LEN; else if (req->htc->body_status == BS_LENGTH) - req->req_body_status = REQ_BODY_WITH_LEN; + req->req_body_status = REQ_BODY_LENGTH; else if (req->htc->body_status == BS_NONE) req->req_body_status = REQ_BODY_NONE; else if (req->htc->body_status == BS_EOF) diff --git a/bin/varnishd/http2/cache_http2_proto.c b/bin/varnishd/http2/cache_http2_proto.c index e90dfa87f..183a327cf 100644 --- a/bin/varnishd/http2/cache_http2_proto.c +++ b/bin/varnishd/http2/cache_http2_proto.c @@ -583,7 +583,7 @@ h2_end_headers(struct worker *wrk, struct h2_sess *h2, if (!http_GetHdr(req->http, H_Content_Length, NULL)) req->req_body_status = REQ_BODY_WITHOUT_LEN; else - req->req_body_status = REQ_BODY_WITH_LEN; + req->req_body_status = REQ_BODY_LENGTH; } else { assert (req->req_body_status == REQ_BODY_NONE); if (http_GetContentLength(req->http) > 0) diff --git a/include/tbl/req_body.h b/include/tbl/req_body.h index 93c46c0bc..6386f4e82 100644 --- a/include/tbl/req_body.h +++ b/include/tbl/req_body.h @@ -33,7 +33,7 @@ REQ_BODY(INIT) REQ_BODY(WITHOUT_LEN) -REQ_BODY(WITH_LEN) +REQ_BODY(LENGTH) /* states >= TAKEN imply that no body is to be read */ REQ_BODY(TAKEN) REQ_BODY(CACHED) From phk at FreeBSD.org Fri Feb 21 08:34:07 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Fri, 21 Feb 2020 08:34:07 +0000 (UTC) Subject: [master] ccc1339e6 Collapse body_status and req_body_status into one type. Message-ID: <20200221083407.2E3BE100495@lists.varnish-cache.org> commit ccc1339e6628d6e545b8504d0a230a11f261e930 Author: Poul-Henning Kamp Date: Fri Feb 21 08:28:54 2020 +0000 Collapse body_status and req_body_status into one type. A lot of this is s/REQ_BODY_/BS_/g, followed by a reduction pass to go from testing on specific status values to attributes of the status value. For instance the req.body caching code does not need to know if how the transport sees this, chunked vs. eof for instance, it just needs to know that there is a body and that we do not know the length of it yet. The transports own the "vocabulary", because internally H1 needs to know the difference between BS_CHUNKED and BS_EOF in order to setup the VFPs. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 2b36acc0b..e661804e7 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -71,13 +71,6 @@ typedef const struct body_status *body_status_t; /*--------------------------------------------------------------------*/ -enum req_body_state_e { -#define REQ_BODY(U) REQ_BODY_##U, -#include "tbl/req_body.h" -}; - -/*--------------------------------------------------------------------*/ - enum sess_close { SC_NULL = 0, #define SESS_CLOSE(nm, stat, err, desc) SC_##nm, @@ -464,7 +457,7 @@ struct req { #define REQ_MAGIC 0x2751aaa1 enum req_step req_step; - volatile enum req_body_state_e req_body_status; + body_status_t req_body_status; enum sess_close doclose; unsigned restarts; unsigned esi_level; diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c index 301d8f6ed..e8d51637f 100644 --- a/bin/varnishd/cache/cache_esi_deliver.c +++ b/bin/varnishd/cache/cache_esi_deliver.c @@ -171,7 +171,7 @@ ved_include(struct req *preq, const char *src, const char *host, /* Client content already taken care of */ http_Unset(req->http, H_Content_Length); - req->req_body_status = REQ_BODY_NONE; + req->req_body_status = BS_NONE; AZ(req->vcl); AN(req->top); diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index 0ac2b8338..613327c2d 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -240,18 +240,15 @@ vbf_stp_mkbereq(struct worker *wrk, struct busyobj *bo) bo->ws_bo = WS_Snapshot(bo->ws); HTTP_Clone(bo->bereq, bo->bereq0); - if (bo->req->req_body_status == REQ_BODY_NONE) { + if (bo->req->req_body_status->avail == 0) { bo->req = NULL; ObjSetState(bo->wrk, bo->fetch_objcore, BOS_REQ_DONE); - } else if (bo->req->req_body_status == REQ_BODY_CACHED) { + } else if (bo->req->req_body_status == BS_CACHED) { AN(bo->req->body_oc); bo->bereq_body = bo->req->body_oc; HSH_Ref(bo->bereq_body); bo->req = NULL; ObjSetState(bo->wrk, bo->fetch_objcore, BOS_REQ_DONE); - } else if (bo->req->req_body_status != REQ_BODY_LENGTH && - bo->req->req_body_status != REQ_BODY_WITHOUT_LEN) { - WRONG("Bad req_body_status"); } return (F_STP_STARTFETCH); } diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c index 81987f89e..502fcaae0 100644 --- a/bin/varnishd/cache/cache_panic.c +++ b/bin/varnishd/cache/cache_panic.c @@ -71,19 +71,6 @@ static void pan_req(struct vsb *, const struct req *); /*--------------------------------------------------------------------*/ -static const char * -reqbody_status_2str(enum req_body_state_e e) -{ - switch (e) { -#define REQ_BODY(U) case REQ_BODY_##U: return("R_BODY_" #U); -#include "tbl/req_body.h" - default: - return ("?"); - } -} - -/*--------------------------------------------------------------------*/ - static const char * boc_state_2str(enum boc_state_e e) { @@ -524,7 +511,7 @@ pan_req(struct vsb *vsb, const struct req *req) VSB_printf(vsb, "step = 0x%x,\n", req->req_step); VSB_printf(vsb, "req_body = %s,\n", - reqbody_status_2str(req->req_body_status)); + req->req_body_status ? req->req_body_status->name : "NULL"); if (req->err_code) VSB_printf(vsb, diff --git a/bin/varnishd/cache/cache_req.c b/bin/varnishd/cache/cache_req.c index 305cea641..122dfdb6d 100644 --- a/bin/varnishd/cache/cache_req.c +++ b/bin/varnishd/cache/cache_req.c @@ -94,7 +94,6 @@ Req_New(const struct worker *wrk, struct sess *sp) AN(req); req->magic = REQ_MAGIC; req->sp = sp; - req->req_body_status = REQ_BODY_INIT; e = (char*)req + sz; p = (char*)(req + 1); @@ -242,7 +241,7 @@ Req_Cleanup(struct sess *sp, struct worker *wrk, struct req *req) req->t_first = NAN; req->t_prev = NAN; req->t_req = NAN; - req->req_body_status = REQ_BODY_INIT; + req->req_body_status = NULL; req->hash_always_miss = 0; req->hash_ignore_busy = 0; diff --git a/bin/varnishd/cache/cache_req_body.c b/bin/varnishd/cache/cache_req_body.c index 08db8fdb6..966526e22 100644 --- a/bin/varnishd/cache/cache_req_body.c +++ b/bin/varnishd/cache/cache_req_body.c @@ -77,7 +77,7 @@ vrb_pull(struct req *req, ssize_t maxsize, objiterate_f *func, void *priv) req->storage = NULL; if (STV_NewObject(req->wrk, req->body_oc, stv, 8) == 0) { - req->req_body_status = REQ_BODY_ERROR; + req->req_body_status = BS_ERROR; HSH_DerefBoc(req->wrk, req->body_oc); AZ(HSH_DerefObjCore(req->wrk, &req->body_oc, 0)); (void)VFP_Error(vfc, "Object allocation failed:" @@ -88,7 +88,7 @@ vrb_pull(struct req *req, ssize_t maxsize, objiterate_f *func, void *priv) vfc->oc = req->body_oc; if (VFP_Open(vfc) < 0) { - req->req_body_status = REQ_BODY_ERROR; + req->req_body_status = BS_ERROR; HSH_DerefBoc(req->wrk, req->body_oc); AZ(HSH_DerefObjCore(req->wrk, &req->body_oc, 0)); return (-1); @@ -136,7 +136,7 @@ vrb_pull(struct req *req, ssize_t maxsize, objiterate_f *func, void *priv) HSH_DerefBoc(req->wrk, req->body_oc); AZ(HSH_DerefObjCore(req->wrk, &req->body_oc, 0)); if (vfps != VFP_END) { - req->req_body_status = REQ_BODY_ERROR; + req->req_body_status = BS_ERROR; if (r == 0) r = -1; } @@ -148,7 +148,7 @@ vrb_pull(struct req *req, ssize_t maxsize, objiterate_f *func, void *priv) HSH_DerefBoc(req->wrk, req->body_oc); if (vfps != VFP_END) { - req->req_body_status = REQ_BODY_ERROR; + req->req_body_status = BS_ERROR; AZ(HSH_DerefObjCore(req->wrk, &req->body_oc, 0)); return (-1); } @@ -167,7 +167,7 @@ vrb_pull(struct req *req, ssize_t maxsize, objiterate_f *func, void *priv) (uintmax_t)req_bodybytes); } - req->req_body_status = REQ_BODY_CACHED; + req->req_body_status = BS_CACHED; return (req_bodybytes); } @@ -189,32 +189,27 @@ VRB_Iterate(struct worker *wrk, struct vsl_log *vsl, CHECK_OBJ_NOTNULL(req, REQ_MAGIC); AN(func); - switch (req->req_body_status) { - case REQ_BODY_CACHED: + if (req->req_body_status == BS_CACHED) { AN(req->body_oc); if (ObjIterate(wrk, req->body_oc, priv, func, 0)) return (-1); return (0); - case REQ_BODY_NONE: + } + if (req->req_body_status == BS_NONE) return (0); - case REQ_BODY_LENGTH: - case REQ_BODY_WITHOUT_LEN: - break; - case REQ_BODY_TAKEN: + if (req->req_body_status == BS_TAKEN) { VSLb(vsl, SLT_VCL_Error, "Uncached req.body can only be consumed once."); return (-1); - case REQ_BODY_ERROR: + } + if (req->req_body_status == BS_ERROR) { VSLb(vsl, SLT_FetchError, "Had failed reading req.body before."); return (-1); - default: - WRONG("Wrong req_body_status in VRB_Iterate()"); } Lck_Lock(&req->sp->mtx); - if (req->req_body_status == REQ_BODY_LENGTH || - req->req_body_status == REQ_BODY_WITHOUT_LEN) { - req->req_body_status = REQ_BODY_TAKEN; + if (req->req_body_status->avail > 0) { + req->req_body_status = BS_TAKEN; i = 0; } else i = -1; @@ -254,9 +249,9 @@ VRB_Ignore(struct req *req) if (req->doclose) return (0); - if (req->req_body_status == REQ_BODY_LENGTH || - req->req_body_status == REQ_BODY_WITHOUT_LEN) - (void)VRB_Iterate(req->wrk, req->vsl, req, httpq_req_body_discard, NULL); + if (req->req_body_status->avail > 0) + (void)VRB_Iterate(req->wrk, req->vsl, req, + httpq_req_body_discard, NULL); return (0); } @@ -292,6 +287,7 @@ VRB_Cache(struct req *req, ssize_t maxsize) uint64_t u; CHECK_OBJ_NOTNULL(req, REQ_MAGIC); + assert (req->req_step == R_STP_RECV); assert(maxsize >= 0); /* @@ -299,27 +295,22 @@ VRB_Cache(struct req *req, ssize_t maxsize) * where we know we will have no competition or conflicts for the * updates to req.http.* etc. */ - if (req->restarts > 0 && req->req_body_status != REQ_BODY_CACHED) + if (req->restarts > 0 && req->req_body_status != BS_CACHED) { + VSLb(req->vsl, SLT_VCL_Error, + "req.body must be cached before restarts"); return (-1); + } - assert (req->req_step == R_STP_RECV); - switch (req->req_body_status) { - case REQ_BODY_CACHED: + if (req->req_body_status == BS_CACHED) { AZ(ObjGetU64(req->wrk, req->body_oc, OA_LEN, &u)); return (u); - case REQ_BODY_ERROR: - return (-1); - case REQ_BODY_NONE: - return (0); - case REQ_BODY_WITHOUT_LEN: - case REQ_BODY_LENGTH: - break; - default: - WRONG("Wrong req_body_status in VRB_Cache()"); } + if (req->req_body_status->avail <= 0) + return (req->req_body_status->avail); + if (req->htc->content_length > maxsize) { - req->req_body_status = REQ_BODY_ERROR; + req->req_body_status = BS_ERROR; (void)VFP_Error(req->vfc, "Request body too big to cache"); return (-1); } diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index 1deb2595c..3829b6292 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -66,7 +66,7 @@ cnt_transport(struct worker *wrk, struct req *req) CHECK_OBJ_NOTNULL(req, REQ_MAGIC); CHECK_OBJ_NOTNULL(req->http, HTTP_MAGIC); CHECK_OBJ_NOTNULL(req->transport, TRANSPORT_MAGIC); - assert(req->req_body_status != REQ_BODY_INIT); + AN(req->req_body_status); if (http_GetHdr(req->http, H_Expect, &p)) { if (strcasecmp(p, "100-continue")) { @@ -89,7 +89,7 @@ cnt_transport(struct worker *wrk, struct req *req) return (REQ_FSM_DONE); } - if (req->req_body_status < REQ_BODY_TAKEN) { + if (req->req_body_status->avail == 1) { AN(req->transport->req_body != NULL); VFP_Setup(req->vfc, wrk); req->vfc->resp = req->http; // XXX @@ -741,8 +741,8 @@ cnt_pipe(struct worker *wrk, struct req *req) bo->req = req; bo->wrk = wrk; /* Unless cached, reqbody is not our job */ - if (req->req_body_status != REQ_BODY_CACHED) - req->req_body_status = REQ_BODY_NONE; + if (req->req_body_status != BS_CACHED) + req->req_body_status = BS_NONE; SES_Close(req->sp, VDI_Http1Pipe(req, bo)); nxt = REQ_FSM_DONE; V1P_Leave(); @@ -877,7 +877,7 @@ cnt_recv(struct worker *wrk, struct req *req) cnt_recv_prep(req, ci); - if (req->req_body_status == REQ_BODY_ERROR) { + if (req->req_body_status == BS_ERROR) { req->doclose = SC_OVERLOAD; return (REQ_FSM_DONE); } @@ -899,7 +899,7 @@ cnt_recv(struct worker *wrk, struct req *req) } /* Attempts to cache req.body may fail */ - if (req->req_body_status == REQ_BODY_ERROR) { + if (req->req_body_status == BS_ERROR) { req->doclose = SC_RX_BODY; return (REQ_FSM_DONE); } diff --git a/bin/varnishd/http1/cache_http1_fetch.c b/bin/varnishd/http1/cache_http1_fetch.c index 4cecb3148..ad70c6868 100644 --- a/bin/varnishd/http1/cache_http1_fetch.c +++ b/bin/varnishd/http1/cache_http1_fetch.c @@ -91,8 +91,7 @@ V1F_SendReq(struct worker *wrk, struct busyobj *bo, uint64_t *ctr_hdrbytes, assert(*htc->rfd > 0); hp = bo->bereq; - if (bo->req != NULL && - bo->req->req_body_status == REQ_BODY_WITHOUT_LEN) { + if (bo->req != NULL && !bo->req->req_body_status->length_known) { http_PrintfHeader(hp, "Transfer-Encoding: chunked"); do_chunked = 1; } @@ -112,15 +111,15 @@ V1F_SendReq(struct worker *wrk, struct busyobj *bo, uint64_t *ctr_hdrbytes, (void)ObjIterate(bo->wrk, bo->bereq_body, bo, vbf_iter_req_body, 0); } else if (bo->req != NULL && - bo->req->req_body_status != REQ_BODY_NONE) { + bo->req->req_body_status != BS_NONE) { if (do_chunked) V1L_Chunked(wrk); i = VRB_Iterate(wrk, bo->vsl, bo->req, vbf_iter_req_body, bo); - if (bo->req->req_body_status != REQ_BODY_CACHED) + if (bo->req->req_body_status != BS_CACHED) bo->no_retry = "req.body not cached"; - if (bo->req->req_body_status == REQ_BODY_ERROR) { + if (bo->req->req_body_status == BS_ERROR) { /* * XXX: (#2332) We should test to see if the backend * XXX: sent us some headers explaining why. diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c index 6332aca96..67ab7016b 100644 --- a/bin/varnishd/http1/cache_http1_fsm.c +++ b/bin/varnishd/http1/cache_http1_fsm.c @@ -151,9 +151,8 @@ http1_req_body(struct req *req) { CHECK_OBJ_NOTNULL(req, REQ_MAGIC); - if (req->htc->body_status->avail && - V1F_Setup_Fetch(req->vfc, req->htc) != 0) - req->req_body_status = REQ_BODY_ERROR; + if (V1F_Setup_Fetch(req->vfc, req->htc) != 0) + req->req_body_status = BS_ERROR; } static void @@ -273,18 +272,8 @@ http1_dissect(struct worker *wrk, struct req *req) return (-1); } - assert (req->req_body_status == REQ_BODY_INIT); - - if (req->htc->body_status == BS_CHUNKED) - req->req_body_status = REQ_BODY_WITHOUT_LEN; - else if (req->htc->body_status == BS_LENGTH) - req->req_body_status = REQ_BODY_LENGTH; - else if (req->htc->body_status == BS_NONE) - req->req_body_status = REQ_BODY_NONE; - else if (req->htc->body_status == BS_EOF) - req->req_body_status = REQ_BODY_WITHOUT_LEN; - else - WRONG("Unknown req_body_status situation"); + AZ(req->req_body_status); + req->req_body_status = req->htc->body_status; return (0); } diff --git a/bin/varnishd/http2/cache_http2_proto.c b/bin/varnishd/http2/cache_http2_proto.c index 183a327cf..317b3287b 100644 --- a/bin/varnishd/http2/cache_http2_proto.c +++ b/bin/varnishd/http2/cache_http2_proto.c @@ -559,11 +559,6 @@ h2_end_headers(struct worker *wrk, struct h2_sess *h2, assert(r2->state == H2_S_OPEN); h2e = h2h_decode_fini(h2); h2->new_req = NULL; - if (r2->req->req_body_status == REQ_BODY_NONE) { - /* REQ_BODY_NONE implies one of the frames in the - * header block contained END_STREAM */ - r2->state = H2_S_CLOS_REM; - } if (h2e != NULL) { Lck_Lock(&h2->sess->mtx); VSLb(h2->vsl, SLT_Debug, "HPACK/FINI %s", h2e->name); @@ -579,13 +574,15 @@ h2_end_headers(struct worker *wrk, struct h2_sess *h2, // XXX: Have I mentioned H/2 Is hodge-podge ? http_CollectHdrSep(req->http, H_Cookie, "; "); // rfc7540,l,3114,3120 - if (req->req_body_status == REQ_BODY_INIT) { + if (req->req_body_status == NULL) { if (!http_GetHdr(req->http, H_Content_Length, NULL)) - req->req_body_status = REQ_BODY_WITHOUT_LEN; + req->req_body_status = BS_EOF; else - req->req_body_status = REQ_BODY_LENGTH; + req->req_body_status = BS_LENGTH; } else { - assert (req->req_body_status == REQ_BODY_NONE); + /* A HEADER frame contained END_STREAM */ + assert (req->req_body_status == BS_NONE); + r2->state = H2_S_CLOS_REM; if (http_GetContentLength(req->http) > 0) return (H2CE_PROTOCOL_ERROR); //rfc7540,l,1838,1840 } @@ -694,7 +691,7 @@ h2_rx_headers(struct worker *wrk, struct h2_sess *h2, struct h2_req *r2) } if (h2->rxf_flags & H2FF_HEADERS_END_STREAM) - req->req_body_status = REQ_BODY_NONE; + req->req_body_status = BS_NONE; if (h2->rxf_flags & H2FF_HEADERS_END_HEADERS) return (h2_end_headers(wrk, h2, req, r2)); diff --git a/include/Makefile.am b/include/Makefile.am index d31231ddd..9186df797 100644 --- a/include/Makefile.am +++ b/include/Makefile.am @@ -26,7 +26,6 @@ nobase_pkginclude_HEADERS = \ tbl/oc_exp_flags.h \ tbl/oc_flags.h \ tbl/params.h \ - tbl/req_body.h \ tbl/req_flags.h \ tbl/sess_attr.h \ tbl/sess_close.h \ diff --git a/include/tbl/body_status.h b/include/tbl/body_status.h index 878248d72..108345c6c 100644 --- a/include/tbl/body_status.h +++ b/include/tbl/body_status.h @@ -34,10 +34,12 @@ /* Upper lower nbr, avail len_known */ BODYSTATUS(NONE, none, 0, 0, 1) -BODYSTATUS(ERROR, error, 1, 0, 0) +BODYSTATUS(ERROR, error, 1, -1, 0) BODYSTATUS(CHUNKED, chunked, 2, 1, 0) BODYSTATUS(LENGTH, length, 3, 1, 1) BODYSTATUS(EOF, eof, 4, 1, 0) +BODYSTATUS(TAKEN, taken, 5, 0, 0) +BODYSTATUS(CACHED, cached, 6, 2, 1) #undef BODYSTATUS /*lint -restore */ diff --git a/include/tbl/req_body.h b/include/tbl/req_body.h deleted file mode 100644 index 6386f4e82..000000000 --- a/include/tbl/req_body.h +++ /dev/null @@ -1,44 +0,0 @@ -/*- - * Copyright (c) 2013-2015 Varnish Software AS - * All rights reserved. - * - * Author: Poul-Henning Kamp - * - * SPDX-License-Identifier: BSD-2-Clause - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * 1. Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * 2. Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in the - * documentation and/or other materials provided with the distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND - * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE - * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE - * ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE - * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL - * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS - * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) - * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT - * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY - * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF - * SUCH DAMAGE. - * - */ - -/*lint -save -e525 -e539 */ - -REQ_BODY(INIT) -REQ_BODY(WITHOUT_LEN) -REQ_BODY(LENGTH) -/* states >= TAKEN imply that no body is to be read */ -REQ_BODY(TAKEN) -REQ_BODY(CACHED) -REQ_BODY(ERROR) -REQ_BODY(NONE) -#undef REQ_BODY - -/*lint -restore */ From gquintard at users.noreply.github.com Fri Feb 21 16:07:07 2020 From: gquintard at users.noreply.github.com (guillaume quintard) Date: Fri, 21 Feb 2020 16:07:07 +0000 (UTC) Subject: [master] 898e78038 Fix formatting on 'ESI on invalid XML' Message-ID: <20200221160707.6907E1091A9@lists.varnish-cache.org> commit 898e78038fc0e925f344701ad1fd8135589e23ff Author: Napsty Date: Fri Feb 21 07:30:55 2020 +0100 Fix formatting on 'ESI on invalid XML' diff --git a/doc/sphinx/users-guide/esi.rst b/doc/sphinx/users-guide/esi.rst index b85d18f23..87c8c6366 100644 --- a/doc/sphinx/users-guide/esi.rst +++ b/doc/sphinx/users-guide/esi.rst @@ -107,7 +107,7 @@ ESI on invalid XML The ESI parser expects the XML to be reasonably well formed, but this may fail if you are ESI including non-XML files. You can -make the ESI parser disrecard anything but ESI tags by setting: +make the ESI parser disrecard anything but ESI tags by setting:: param.set feature +esi_ignore_other_elements From guillaume at varnish-software.com Fri Feb 21 16:28:06 2020 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Fri, 21 Feb 2020 16:28:06 +0000 (UTC) Subject: [master] b0d1a40f3 add VSB_QUOTE_GLOB Message-ID: <20200221162806.8C37910995F@lists.varnish-cache.org> commit b0d1a40f326f07824af91a2b548d3e2e8e8828f3 Author: Guillaume Quintard Date: Wed Feb 12 01:07:53 2020 -0500 add VSB_QUOTE_GLOB diff --git a/include/vsb.h b/include/vsb.h index 806719b70..a8fe4f6e5 100644 --- a/include/vsb.h +++ b/include/vsb.h @@ -84,6 +84,7 @@ void VSB_destroy(struct vsb **); #define VSB_QUOTE_CSTR 8 #define VSB_QUOTE_UNSAFE 16 #define VSB_QUOTE_ESCHEX 32 +#define VSB_QUOTE_GLOB 64 void VSB_quote_pfx(struct vsb *, const char*, const void *, int len, int how); void VSB_quote(struct vsb *, const void *, int len, int how); diff --git a/lib/libvarnish/Makefile.am b/lib/libvarnish/Makefile.am index bb27d4e1e..6545d0b1b 100644 --- a/lib/libvarnish/Makefile.am +++ b/lib/libvarnish/Makefile.am @@ -43,7 +43,7 @@ libvarnish_a_SOURCES = \ vtim.c \ vus.c -TESTS = vjsn_test vnum_c_test binheap +TESTS = vjsn_test vnum_c_test binheap vsb_test noinst_PROGRAMS = ${TESTS} @@ -57,3 +57,6 @@ vnum_c_test_LDADD = ${LIBM} libvarnish.a @SAN_LDFLAGS@ vjsn_test_SOURCES = vjsn.c vjsn_test_CFLAGS = -DVJSN_TEST @SAN_CFLAGS@ vjsn_test_LDADD = libvarnish.a @SAN_LDFLAGS@ + +vsb_test_SOURCES = vsb_test.c vsb.c vas.c +vsb_test_LDADD = libvarnish.a diff --git a/lib/libvarnish/vsb.c b/lib/libvarnish/vsb.c index 0762c175e..8964971f2 100644 --- a/lib/libvarnish/vsb.c +++ b/lib/libvarnish/vsb.c @@ -563,7 +563,7 @@ VSB_quote_pfx(struct vsb *s, const char *pfx, const void *v, int len, int how) nl = 0; switch (*q) { case '?': - if (how & VSB_QUOTE_CSTR) + if (how & (VSB_QUOTE_CSTR | VSB_QUOTE_GLOB)) (void)VSB_putc(s, '\\'); (void)VSB_putc(s, *q); break; @@ -571,6 +571,10 @@ VSB_quote_pfx(struct vsb *s, const char *pfx, const void *v, int len, int how) (void)VSB_putc(s, *q); break; case '\\': + if (!(how & (VSB_QUOTE_UNSAFE))) + (void)VSB_putc(s, '\\'); + (void)VSB_putc(s, *q); + break; case '"': if (!(how & VSB_QUOTE_UNSAFE)) (void)VSB_putc(s, '\\'); @@ -582,18 +586,42 @@ VSB_quote_pfx(struct vsb *s, const char *pfx, const void *v, int len, int how) } else if (how & (VSB_QUOTE_NONL|VSB_QUOTE_UNSAFE)) { (void)VSB_printf(s, "\n"); nl = 1; + } else if (how & VSB_QUOTE_GLOB) { + (void)VSB_printf(s, "\\\\n"); } else { (void)VSB_printf(s, "\\n"); } break; case '\r': - (void)VSB_cat(s, "\\r"); + if (how & VSB_QUOTE_GLOB) + (void)VSB_cat(s, "\\\\r"); + else + (void)VSB_cat(s, "\\r"); break; case '\t': - (void)VSB_cat(s, "\\t"); + if (how & VSB_QUOTE_GLOB) + (void)VSB_cat(s, "\\\\t"); + else + (void)VSB_cat(s, "\\t"); break; case '\v': - (void)VSB_cat(s, "\\v"); + if (how & VSB_QUOTE_GLOB) + (void)VSB_cat(s, "\\\\v"); + else + (void)VSB_cat(s, "\\v"); + break; + case '[': + /* FALLTHROUGH */ + case ']': + /* FALLTHROUGH */ + case '{': + /* FALLTHROUGH */ + case '}': + /* FALLTHROUGH */ + case '*': + if (how & VSB_QUOTE_GLOB) + (void)VSB_putc(s, '\\'); + (void)VSB_putc(s, *q); break; default: /* XXX: Implement VSB_QUOTE_JSON */ diff --git a/lib/libvarnish/vsb_test.c b/lib/libvarnish/vsb_test.c new file mode 100644 index 000000000..5de87118a --- /dev/null +++ b/lib/libvarnish/vsb_test.c @@ -0,0 +1,49 @@ +#include +#include + +#include "vdef.h" +#include "vas.h" +#include "vsb.h" + +struct tc { + int how; + const char *in; + const char *out; +}; + +struct tc tcs[] = { + { VSB_QUOTE_GLOB, "abcdefghijklmnopqrstvwxyz", "abcdefghijklmnopqrstvwxyz" }, + { VSB_QUOTE_GLOB, "ABCDEFGHIJKLMNOPQRSTVWXYZ", "ABCDEFGHIJKLMNOPQRSTVWXYZ" }, + { VSB_QUOTE_GLOB, "01234567789", "01234567789"}, + { VSB_QUOTE_GLOB, "abcde[f-g]{h,i,j}\\l?*xyz", "abcde\\[f-g\\]\\{h,i,j\\}\\\\l\\?\\*xyz"}, + { VSB_QUOTE_GLOB, "0123\t \"\r\v\n'", "0123\\\\t \\\"\\\\r\\\\v\\\\n'"}, + {0, NULL, NULL} +}; + +int main(int argc, char *argv[]) +{ + int err = 0; + struct tc *tc; + struct vsb *vsb; + + (void)argc; + (void)argv; + vsb = VSB_new_auto(); + AN(vsb); + + for (tc = tcs; tc->in; tc++) { + VSB_quote(vsb, tc->in, -1, tc->how); + VSB_finish(vsb); + + printf("%s -> %s", tc->in, VSB_data(vsb)); + if (strcmp(VSB_data(vsb), tc->out)) { + printf(", but should have been %s", tc->out); + err = 1; + } + printf("\n"); + VSB_clear(vsb); + } + VSB_delete(vsb); + printf("error is %i\n", err); + return (err); +} From guillaume at varnish-software.com Fri Feb 21 16:28:06 2020 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Fri, 21 Feb 2020 16:28:06 +0000 (UTC) Subject: [master] cee00761c typo Message-ID: <20200221162806.A831C109962@lists.varnish-cache.org> commit cee00761cb4fa31167e800a4db332796d9e06eff Author: Guillaume Quintard Date: Fri Feb 21 08:27:11 2020 -0800 typo diff --git a/doc/sphinx/users-guide/esi.rst b/doc/sphinx/users-guide/esi.rst index 87c8c6366..c8cabeef5 100644 --- a/doc/sphinx/users-guide/esi.rst +++ b/doc/sphinx/users-guide/esi.rst @@ -107,7 +107,7 @@ ESI on invalid XML The ESI parser expects the XML to be reasonably well formed, but this may fail if you are ESI including non-XML files. You can -make the ESI parser disrecard anything but ESI tags by setting:: +make the ESI parser disregard anything but ESI tags by setting:: param.set feature +esi_ignore_other_elements From dridi.boukelmoune at gmail.com Mon Feb 24 07:15:09 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 24 Feb 2020 07:15:09 +0000 (UTC) Subject: [master] cccb533f2 Drop explicit FALLTHROUGHs for consecutive cases Message-ID: <20200224071509.CF48710E499@lists.varnish-cache.org> commit cccb533f218eddef728eca48e415f2036543c953 Author: Dridi Boukelmoune Date: Mon Feb 24 08:13:38 2020 +0100 Drop explicit FALLTHROUGHs for consecutive cases diff --git a/lib/libvarnish/vsb.c b/lib/libvarnish/vsb.c index 8964971f2..e0b48c50e 100644 --- a/lib/libvarnish/vsb.c +++ b/lib/libvarnish/vsb.c @@ -611,13 +611,9 @@ VSB_quote_pfx(struct vsb *s, const char *pfx, const void *v, int len, int how) (void)VSB_cat(s, "\\v"); break; case '[': - /* FALLTHROUGH */ case ']': - /* FALLTHROUGH */ case '{': - /* FALLTHROUGH */ case '}': - /* FALLTHROUGH */ case '*': if (how & VSB_QUOTE_GLOB) (void)VSB_putc(s, '\\'); From phk at FreeBSD.org Mon Feb 24 08:42:05 2020 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 24 Feb 2020 08:42:05 +0000 (UTC) Subject: [master] 2b127d43c Polish this to bring it closer to style(9) and how the rest of our test-programs are structured. Message-ID: <20200224084205.D792310FFF5@lists.varnish-cache.org> commit 2b127d43c7b67fee4c92ac44f240db9008bd63cb Author: Poul-Henning Kamp Date: Mon Feb 24 08:40:11 2020 +0000 Polish this to bring it closer to style(9) and how the rest of our test-programs are structured. diff --git a/lib/libvarnish/Makefile.am b/lib/libvarnish/Makefile.am index 6545d0b1b..da5821c7e 100644 --- a/lib/libvarnish/Makefile.am +++ b/lib/libvarnish/Makefile.am @@ -59,4 +59,5 @@ vjsn_test_CFLAGS = -DVJSN_TEST @SAN_CFLAGS@ vjsn_test_LDADD = libvarnish.a @SAN_LDFLAGS@ vsb_test_SOURCES = vsb_test.c vsb.c vas.c +vsb_test_CFLAGS = -DVSB_TEST vsb_test_LDADD = libvarnish.a diff --git a/lib/libvarnish/vsb_test.c b/lib/libvarnish/vsb_test.c index 5de87118a..c6e4866ff 100644 --- a/lib/libvarnish/vsb_test.c +++ b/lib/libvarnish/vsb_test.c @@ -1,3 +1,6 @@ + +#ifdef VSB_TEST + #include #include @@ -11,16 +14,34 @@ struct tc { const char *out; }; -struct tc tcs[] = { - { VSB_QUOTE_GLOB, "abcdefghijklmnopqrstvwxyz", "abcdefghijklmnopqrstvwxyz" }, - { VSB_QUOTE_GLOB, "ABCDEFGHIJKLMNOPQRSTVWXYZ", "ABCDEFGHIJKLMNOPQRSTVWXYZ" }, - { VSB_QUOTE_GLOB, "01234567789", "01234567789"}, - { VSB_QUOTE_GLOB, "abcde[f-g]{h,i,j}\\l?*xyz", "abcde\\[f-g\\]\\{h,i,j\\}\\\\l\\?\\*xyz"}, - { VSB_QUOTE_GLOB, "0123\t \"\r\v\n'", "0123\\\\t \\\"\\\\r\\\\v\\\\n'"}, - {0, NULL, NULL} +static struct tc tcs[] = { + { + VSB_QUOTE_GLOB, + "abcdefghijklmnopqrstvwxyz", + "abcdefghijklmnopqrstvwxyz" + }, { + VSB_QUOTE_GLOB, + "ABCDEFGHIJKLMNOPQRSTVWXYZ", + "ABCDEFGHIJKLMNOPQRSTVWXYZ" + }, { + VSB_QUOTE_GLOB, + "01234567789", + "01234567789" + }, { + VSB_QUOTE_GLOB, + "abcde[f-g]{h,i,j}\\l?*xyz", + "abcde\\[f-g\\]\\{h,i,j\\}\\\\l\\?\\*xyz" + }, { + VSB_QUOTE_GLOB, + "0123\t \"\r\v\n'", + "0123\\\\t \\\"\\\\r\\\\v\\\\n'" + }, { + 0, NULL, NULL + } }; -int main(int argc, char *argv[]) +int +main(int argc, char *argv[]) { int err = 0; struct tc *tc; @@ -33,7 +54,7 @@ int main(int argc, char *argv[]) for (tc = tcs; tc->in; tc++) { VSB_quote(vsb, tc->in, -1, tc->how); - VSB_finish(vsb); + assert(VSB_finish(vsb) == 0); printf("%s -> %s", tc->in, VSB_data(vsb)); if (strcmp(VSB_data(vsb), tc->out)) { @@ -47,3 +68,5 @@ int main(int argc, char *argv[]) printf("error is %i\n", err); return (err); } + +#endif /* VSB_TEST */ From nils.goroll at uplex.de Mon Feb 24 11:20:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 24 Feb 2020 11:20:06 +0000 (UTC) Subject: [master] bcd7d07aa upgrading 6.3: vcl auto state details Message-ID: <20200224112006.96D85113361@lists.varnish-cache.org> commit bcd7d07aab281536dfa4f9f7e91877ad15a030d4 Author: Nils Goroll Date: Mon Feb 24 12:19:27 2020 +0100 upgrading 6.3: vcl auto state details diff --git a/doc/sphinx/whats-new/upgrading-6.3.rst b/doc/sphinx/whats-new/upgrading-6.3.rst index df269e077..b813b9c44 100644 --- a/doc/sphinx/whats-new/upgrading-6.3.rst +++ b/doc/sphinx/whats-new/upgrading-6.3.rst @@ -14,7 +14,27 @@ features are listed in "Changes". Explicitly mention what does *not* have to be changed, especially in VCL. May include, but is not limited to:** -TODO: a word on VCL temperature and the ``auto`` change. +For users of many and/or labeled VCLs +===================================== + +Users of the advanced mechanics behind the ``vcl.state`` CLI command +(most likely used via ``varnishadm``) should be aware of the following +changes, which may require adjustments to (or, more likely, allow for +simplifications of) scripts/programs interfacing with varnish: + +The VCL ``auto`` state has been streamlined. Conceptually, it used to +be a variant of the ``warm`` state which would automatically cool +the vcl. Yet, cooling did not only transition the temperature, but +also the state, so ``auto`` only worked one way - except that +``vcl.use`` or moving a label (by labeling another vcl) would also set +``auto``, so a manual warm/cold setting would get lost. + +Now the ``auto`` state will remain no matter the actual temperature or +labeling, so when a vcl needs to implicitly change temperature (due to +being used or being labeled), an ``auto`` vcl will remain ``auto``, +and a ``cold`` / ``warm`` vcl will change state, but never become +``auto`` implicitly. + For developers and authors of VMODs and API clients =================================================== From dridi.boukelmoune at gmail.com Mon Feb 24 19:49:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 24 Feb 2020 19:49:06 +0000 (UTC) Subject: [master] 79cf4bac4 Make v56.vtc easier to debug Message-ID: <20200224194907.10DB398CC@lists.varnish-cache.org> commit 79cf4bac4b0f138ab747125be6a6e4755d9ef0ae Author: Dridi Boukelmoune Date: Mon Feb 24 20:25:29 2020 +0100 Make v56.vtc easier to debug In case of failure, I can see which client failed more easily. diff --git a/bin/varnishtest/tests/v00056.vtc b/bin/varnishtest/tests/v00056.vtc index 7d3d50733..b3c559f7a 100644 --- a/bin/varnishtest/tests/v00056.vtc +++ b/bin/varnishtest/tests/v00056.vtc @@ -54,16 +54,21 @@ client c1 { txreq rxresp expect resp.status == 500 +} -run - barrier b1 sync +barrier b1 sync +barrier b2 sync - barrier b2 sync +client c2 { txreq rxresp expect resp.status == 500 +} -run - barrier b3 sync - barrier b4 sync +barrier b3 sync +barrier b4 sync + +client c3 { txreq rxresp expect resp.status == 200 From dridi.boukelmoune at gmail.com Mon Feb 24 19:49:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 24 Feb 2020 19:49:07 +0000 (UTC) Subject: [master] 57fe09e3d Kill a dead branch when building a probe request Message-ID: <20200224194907.24F3C98CE@lists.varnish-cache.org> commit 57fe09e3d3b387129a1ac6a9ab67bba8e411e75b Author: Dridi Boukelmoune Date: Mon Feb 24 20:29:44 2020 +0100 Kill a dead branch when building a probe request We have two fallbacks in the absence of a .host_header field in the backend, one of which is systematic. We can build a non .request probe request with a single printf statement. Please note that when the .host field is in the form "$addr $port" we use that as the probe's host header. This behavior is not introduced by this change but ultimately we might want to do something about it and for example capture the first token from hosthdr with a txt. diff --git a/bin/varnishd/cache/cache_backend_probe.c b/bin/varnishd/cache/cache_backend_probe.c index 252ae7ecb..23721fb94 100644 --- a/bin/varnishd/cache/cache_backend_probe.c +++ b/bin/varnishd/cache/cache_backend_probe.c @@ -586,12 +586,14 @@ vbp_build_req(struct vbp_target *vt, const struct vrt_backend_probe *vbp, if (vbp->request != NULL) { VSB_cat(vsb, vbp->request); } else { - VSB_printf(vsb, "GET %s HTTP/1.1\r\n", - vbp->url != NULL ? vbp->url : "/"); - if (be->hosthdr != NULL) - VSB_printf(vsb, "Host: %s\r\n", be->hosthdr); - VSB_cat(vsb, "Connection: close\r\n"); - VSB_cat(vsb, "\r\n"); + AN(be->hosthdr); + VSB_printf(vsb, + "GET %s HTTP/1.1\r\n" + "Host: %s\r\n" + "Connection: close\r\n" + "\r\n", + vbp->url != NULL ? vbp->url : "/", + be->hosthdr); } AZ(VSB_finish(vsb)); vt->req = strdup(VSB_data(vsb)); From dridi.boukelmoune at gmail.com Mon Feb 24 21:48:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 24 Feb 2020 21:48:07 +0000 (UTC) Subject: [master] 67000057e VSB test droppings Message-ID: <20200224214807.1660362802@lists.varnish-cache.org> commit 67000057eef17d1ae5787f851d242b017742bdf6 Author: Dridi Boukelmoune Date: Mon Feb 24 22:47:32 2020 +0100 VSB test droppings diff --git a/.gitignore b/.gitignore index 2fe1a1b6d..5872c9a6b 100644 --- a/.gitignore +++ b/.gitignore @@ -117,6 +117,7 @@ cscope.*out /lib/libvarnish/binheap /lib/libvarnish/vjsn_test /lib/libvarnish/vnum_c_test +/lib/libvarnish/vsb_test /lib/libvarnishapi/vsl_glob_test /lib/libvarnishapi/vsl_glob_test_coverage From dridi.boukelmoune at gmail.com Mon Feb 24 21:56:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 24 Feb 2020 21:56:07 +0000 (UTC) Subject: [master] 55bd09a41 No the default vsl_space is not 1G for 32bit builds Message-ID: <20200224215608.1304262D03@lists.varnish-cache.org> commit 55bd09a41846a0cfea0a93362801af4522a6c865 Author: Dridi Boukelmoune Date: Mon Feb 24 22:54:27 2020 +0100 No the default vsl_space is not 1G for 32bit builds Better diff with the --word-diff option. diff --git a/doc/sphinx/reference/varnishd.rst b/doc/sphinx/reference/varnishd.rst index 98c5ff0db..dd01fbeba 100644 --- a/doc/sphinx/reference/varnishd.rst +++ b/doc/sphinx/reference/varnishd.rst @@ -506,15 +506,16 @@ flags are: Default Value Exceptions on 32 bit Systems ------------------------------------------ -Be aware that on 32 bit systems, certain default values are reduced -relative to the values listed below, in order to conserve VM space: +Be aware that on 32 bit systems, certain default or maximum values are +reduced relative to the values listed below, in order to conserve VM +space: * workspace_client: 24k * workspace_backend: 20k * http_resp_size: 8k * http_req_size: 12k * gzip_buffer: 4k -* vsl_space: 1G +* vsl_space: 1G (maximum) * thread_pool_stack: 52k .. _List of Parameters: From nils.goroll at uplex.de Wed Feb 26 13:08:09 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 26 Feb 2020 13:08:09 +0000 (UTC) Subject: [master] aaaf7b383 Fixup private rfc2616_time function Message-ID: <20200226130810.0314EA1B24@lists.varnish-cache.org> commit aaaf7b383339b18f368935ff3fd38a6575c15666 Author: Martin Blix Grydeland Date: Tue Feb 25 15:22:36 2020 +0100 Fixup private rfc2616_time function Change the return value to unsigned, to match with the expected data type where it is used. Handle very large numbers consistently. Currently it was converting from unsigned long to int, which would throw away the most significant bits. Now overly large integers will be capped at UINT_MAX. Implement the "allow and ignore decimal point" behaviour that the Age header parsing incorporated in rfc2616_time(). This way we will allow a decimal points also in max-age and stale-while-revalidate parsing of Cache-Control directives. diff --git a/bin/varnishd/cache/cache_rfc2616.c b/bin/varnishd/cache/cache_rfc2616.c index 14e655c49..22263a7bf 100644 --- a/bin/varnishd/cache/cache_rfc2616.c +++ b/bin/varnishd/cache/cache_rfc2616.c @@ -64,17 +64,22 @@ * */ -static inline int +static inline unsigned rfc2616_time(const char *p) { char *ep; - int val; + unsigned long val; if (*p == '-') return (0); val = strtoul(p, &ep, 10); - for (; *ep != '\0' && vct_issp(*ep); ep++) - continue; - if (*ep == '\0' || *ep == ',') + if (val > UINT_MAX) + return (UINT_MAX); + while (vct_issp(*ep)) + ep++; + /* We accept ',' as an end character because we may be parsing a + * multi-element Cache-Control part. We accept '.' to be future + * compatble with fractional seconds. */ + if (*ep == '\0' || *ep == ',' || *ep == '.') return (val); return (0); } From nils.goroll at uplex.de Wed Feb 26 13:08:10 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 26 Feb 2020 13:08:10 +0000 (UTC) Subject: [master] b610e9a0b Use rfc2616_time() to parse Age headers Message-ID: <20200226130810.17E2CA1B29@lists.varnish-cache.org> commit b610e9a0b954fe963a86b95b44ee30089a9333d3 Author: Martin Blix Grydeland Date: Tue Feb 25 15:29:58 2020 +0100 Use rfc2616_time() to parse Age headers One time element function to rule and parse them all. diff --git a/bin/varnishd/cache/cache_rfc2616.c b/bin/varnishd/cache/cache_rfc2616.c index 22263a7bf..781a108cc 100644 --- a/bin/varnishd/cache/cache_rfc2616.c +++ b/bin/varnishd/cache/cache_rfc2616.c @@ -117,12 +117,7 @@ RFC2616_Ttl(struct busyobj *bo, vtim_real now, vtim_real *t_origin, */ if (http_GetHdr(hp, H_Age, &p)) { - /* - * We deliberately run with partial results, rather than - * reject the Age: header outright. This will be future - * compatible with fractional seconds. - */ - age = strtoul(p, NULL, 10); + age = rfc2616_time(p); *t_origin -= age; } From nils.goroll at uplex.de Wed Feb 26 13:08:10 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 26 Feb 2020 13:08:10 +0000 (UTC) Subject: [master] cff50548b Keep Age information on passes Message-ID: <20200226130810.3DDA9A1B2D@lists.varnish-cache.org> commit cff50548b9633b676d71fc82c9103b368807e9a3 Author: Martin Blix Grydeland Date: Fri Feb 21 16:40:38 2020 +0100 Keep Age information on passes When doing a pass, we would remove the Age header from the backend, and create a new one based on the time the fetch was initiated. This creates problems when calculating the time to live in downstream caches (browser cache or layered varnishes). With this patch, the RFC_2616_Ttl calculation routine is run also for passes, where the t_origin field of the object is adjusted for an incoming Age header. This makes sure that the Age header generated during delivery is correct. The rest of the Ttl calculation is skipped for passes, including the logging of SLT_TTL "RFC". Fixes: varnishcache/varnish-cache#3221 diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index 613327c2d..7bc595613 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -415,21 +415,12 @@ vbf_stp_startfetch(struct worker *wrk, struct busyobj *bo) http_CollectHdr(bo->beresp, H_Cache_Control); http_CollectHdr(bo->beresp, H_Vary); - if (bo->fetch_objcore->flags & OC_F_PRIVATE) { - /* private objects have negative TTL */ - bo->fetch_objcore->t_origin = now; - bo->fetch_objcore->ttl = -1.; - bo->fetch_objcore->grace = 0; - bo->fetch_objcore->keep = 0; - } else { - /* What does RFC2616 think about TTL ? */ - RFC2616_Ttl(bo, now, - &bo->fetch_objcore->t_origin, - &bo->fetch_objcore->ttl, - &bo->fetch_objcore->grace, - &bo->fetch_objcore->keep - ); - } + /* What does RFC2616 think about TTL ? */ + RFC2616_Ttl(bo, now, + &bo->fetch_objcore->t_origin, + &bo->fetch_objcore->ttl, + &bo->fetch_objcore->grace, + &bo->fetch_objcore->keep); AZ(bo->do_esi); AZ(bo->was_304); diff --git a/bin/varnishd/cache/cache_rfc2616.c b/bin/varnishd/cache/cache_rfc2616.c index 781a108cc..f00b37ebb 100644 --- a/bin/varnishd/cache/cache_rfc2616.c +++ b/bin/varnishd/cache/cache_rfc2616.c @@ -94,6 +94,7 @@ RFC2616_Ttl(struct busyobj *bo, vtim_real now, vtim_real *t_origin, const struct http *hp; CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); + CHECK_OBJ_NOTNULL(bo->fetch_objcore, OBJCORE_MAGIC); assert(now != 0.0 && !isnan(now)); AN(t_origin); AN(ttl); @@ -121,6 +122,19 @@ RFC2616_Ttl(struct busyobj *bo, vtim_real now, vtim_real *t_origin, *t_origin -= age; } + if (bo->fetch_objcore->flags & OC_F_PRIVATE) { + /* Pass object. Halt the processing here, keeping only the + * parsed value of t_origin, as that will be needed to + * synthesize a correct Age header in delivery. The + * SLT_TTL log tag at the end of this function is + * deliberetaly skipped to avoid confusion when reading + * the log.*/ + *ttl = -1; + *grace = 0; + *keep = 0; + return; + } + if (http_GetHdr(hp, H_Expires, &p)) h_expires = VTIM_parse(p); diff --git a/bin/varnishtest/tests/r03221.vtc b/bin/varnishtest/tests/r03221.vtc new file mode 100644 index 000000000..8fab00691 --- /dev/null +++ b/bin/varnishtest/tests/r03221.vtc @@ -0,0 +1,27 @@ +varnishtest "Handling of Age when return(pass)" + +server s1 { + rxreq + txresp -hdr "cache-control: max-age=2" -hdr "age: 1" + + rxreq + txresp -hdr "cache-control: max-age=2" -hdr "age: 1" +} -start + +varnish v1 -vcl+backend { + sub vcl_recv { + if (req.url == "/pass") { + return(pass); + } + } +} -start + +client c1 { + txreq + rxresp + expect resp.http.age == 1 + + txreq -url /pass + rxresp + expect resp.http.age == 1 +} -run From nils.goroll at uplex.de Wed Feb 26 14:05:06 2020 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 26 Feb 2020 14:05:06 +0000 (UTC) Subject: [master] d52425ff8 varnishadm: do not ignore SIGTERM/SIGINT for interactive&pass mode Message-ID: <20200226140506.5B3BEA3510@lists.varnish-cache.org> commit d52425ff8d8a8528cf43233779ae7ab7efd193b0 Author: Nils Goroll Date: Wed Feb 26 15:04:18 2020 +0100 varnishadm: do not ignore SIGTERM/SIGINT for interactive&pass mode fixes #3229 diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c index ba8797c7c..f93ace25d 100644 --- a/bin/varnishadm/varnishadm.c +++ b/bin/varnishadm/varnishadm.c @@ -467,9 +467,6 @@ main(int argc, char * const *argv) argc -= optind; argv += optind; - VSIG_Arm_int(); - VSIG_Arm_term(); - if (T_arg != NULL) { if (n_arg != NULL) usage(1); @@ -482,9 +479,14 @@ main(int argc, char * const *argv) if (sock < 0) exit(2); - if (argc > 0) + if (argc > 0) { + VSIG_Arm_int(); + VSIG_Arm_term(); do_args(sock, argc, argv); - else if (isatty(0)) + NEEDLESS(exit(0)); + } + + if (isatty(0)) interactive(sock); else pass(sock); From dridi.boukelmoune at gmail.com Wed Feb 26 18:34:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 26 Feb 2020 18:34:06 +0000 (UTC) Subject: [master] 30c5e8d81 varnish.m4: Fix documentation mistake Message-ID: <20200226183406.92DD4A96F9@lists.varnish-cache.org> commit 30c5e8d8196cf4e9c4051b3164ef17b822bcb560 Author: Jordan Christiansen Date: Wed Feb 26 11:47:55 2020 -0600 varnish.m4: Fix documentation mistake diff --git a/varnish.m4 b/varnish.m4 index a7ef062ab..29eabeb88 100644 --- a/varnish.m4 +++ b/varnish.m4 @@ -429,7 +429,7 @@ clean-vsc-$1: # to declare sets of counters, but does not associates them automatically # with their respective VMODs: # -# VARNISH_UTILITIES([foo bar]) +# VARNISH_COUNTERS([foo bar]) # # Two build rules will be available for use in Makefile.am for the counters # foo and bar: From dridi.boukelmoune at gmail.com Wed Feb 26 18:34:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 26 Feb 2020 18:34:06 +0000 (UTC) Subject: [master] e191c23f1 Also fix a typo Message-ID: <20200226183406.A7CB2A96FC@lists.varnish-cache.org> commit e191c23f1f2d042d67578e90baf9c7ec17ef9f24 Author: Jordan Christiansen Date: Wed Feb 26 12:17:42 2020 -0600 Also fix a typo diff --git a/varnish.m4 b/varnish.m4 index 29eabeb88..3f01d3301 100644 --- a/varnish.m4 +++ b/varnish.m4 @@ -437,7 +437,7 @@ clean-vsc-$1: # @BUILD_VSC_FOO@ # @BUILD_VSC_BAR@ # -# They take care of turning VSC_foo.vsc and VCS_bar.vcs into C code and +# They take care of turning VSC_foo.vsc and VCS_bar.vsc into C code and # RST documentation. # # Just like the vcc_*_if.[ch] files, you need to manually add the generated From dridi.boukelmoune at gmail.com Wed Feb 26 18:37:06 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 26 Feb 2020 18:37:06 +0000 (UTC) Subject: [master] e2ef73d50 Typo Message-ID: <20200226183706.EF286A9B03@lists.varnish-cache.org> commit e2ef73d50d634317c6cb07fe8f367dee5cbf3c57 Author: Dridi Boukelmoune Date: Wed Feb 26 19:35:27 2020 +0100 Typo diff --git a/varnish.m4 b/varnish.m4 index 3f01d3301..36e24188d 100644 --- a/varnish.m4 +++ b/varnish.m4 @@ -426,7 +426,7 @@ clean-vsc-$1: # # In order to manipulate custom counters that tools like varnishstat can # report, it is possible to do that via a VMOD. This macro allows you -# to declare sets of counters, but does not associates them automatically +# to declare sets of counters, but does not associate them automatically # with their respective VMODs: # # VARNISH_COUNTERS([foo bar]) From dridi.boukelmoune at gmail.com Thu Feb 27 08:39:07 2020 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 27 Feb 2020 08:39:07 +0000 (UTC) Subject: [master] 4356e2bf3 Sync VSC documentation with reality in varnish.m4 Message-ID: <20200227083908.1959264874@lists.varnish-cache.org> commit 4356e2bf3b687f14af73840958e1d0c29c4b6729 Author: Dridi Boukelmoune Date: Thu Feb 27 09:36:38 2020 +0100 Sync VSC documentation with reality in varnish.m4 This was documented following the naming convention in tree, but implemented differently. In order not to change the behavior, only the documentation is updated. Closes #3232 diff --git a/varnish.m4 b/varnish.m4 index 36e24188d..1c10e7659 100644 --- a/varnish.m4 +++ b/varnish.m4 @@ -437,8 +437,8 @@ clean-vsc-$1: # @BUILD_VSC_FOO@ # @BUILD_VSC_BAR@ # -# They take care of turning VSC_foo.vsc and VCS_bar.vsc into C code and -# RST documentation. +# They take care of turning foo.vsc and bar.vsc into C code and RST +# documentation. # # Just like the vcc_*_if.[ch] files, you need to manually add the generated # sources to the appropriate VMODs: @@ -454,6 +454,11 @@ clean-vsc-$1: # # .. include:: VSC_foo.rst # +# Doing so, you can add the generated RST as a dependency of the manual in +# Makefile.am: +# +# vmod_baz.3: VSC_foo.rst +# # That should be all you need to do to start implementing custom counters. # AC_DEFUN([VARNISH_COUNTERS], [