From nils.goroll at uplex.de Thu Nov 1 11:07:11 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 1 Nov 2018 11:07:11 +0000 (UTC) Subject: [master] ee33662a1 do not pass a NULL pointer from strerror() to vsnprintf via VSL Message-ID: <20181101110711.22DFBA1900@lists.varnish-cache.org> commit ee33662a162cbc4fcc7fb8a93d143f85b7786eae Author: Nils Goroll Date: Thu Nov 1 11:45:11 2018 +0100 do not pass a NULL pointer from strerror() to vsnprintf via VSL (at leat on solaris) strerror() itself may fail for an out-of-memory condition (because the localization code contains memory allocations). In order to handle this situation, we need to save the original errno because strerror() may also set errno. This issue exists in many more places all over the code, but in the pool_breed case we likely failed pthread_create for an out-of-memory condition, and in the panic handler we want to make sure that we trip no follow-up panic under any circumstances. In general, while fixing all strerror() calls would unnecessarily complicate the code, doing so should be justified for these cases. Fixes #2815 diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c index 8282ed814..13223c29d 100644 --- a/bin/varnishd/cache/cache_panic.c +++ b/bin/varnishd/cache/cache_panic.c @@ -666,8 +666,12 @@ pan_ic(const char *func, const char *file, int line, const char *cond, pan_backtrace(pan_vsb); - if (err) - VSB_printf(pan_vsb, "errno = %d (%s)\n", err, strerror(err)); + if (err) { + q = strerror(err); + if (q == NULL) + q = "(strerror failed)"; + VSB_printf(pan_vsb, "errno = %d (%s)\n", err, q); + } q = THR_GetName(); if (q != NULL) diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c index 4d546ffd3..a4d9a33ad 100644 --- a/bin/varnishd/cache/cache_wrk.c +++ b/bin/varnishd/cache/cache_wrk.c @@ -437,6 +437,8 @@ pool_breed(struct pool *qp) pthread_t tp; pthread_attr_t tp_attr; struct pool_info *pi; + const char *strerr; + int err; AZ(pthread_attr_init(&tp_attr)); AZ(pthread_attr_setdetachstate(&tp_attr, PTHREAD_CREATE_DETACHED)); @@ -452,8 +454,12 @@ pool_breed(struct pool *qp) pi->qp = qp; if (pthread_create(&tp, &tp_attr, pool_thread, pi)) { + err = errno; + strerr = strerror(errno); + if (strerr == NULL) + strerr = "(strerror failed)"; VSL(SLT_Debug, 0, "Create worker thread failed %d %s", - errno, strerror(errno)); + err, strerr); Lck_Lock(&pool_mtx); VSC_C_main->threads_failed++; Lck_Unlock(&pool_mtx); From phk at phk.freebsd.dk Thu Nov 1 13:33:24 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Thu, 01 Nov 2018 13:33:24 +0000 Subject: [master] ee33662a1 do not pass a NULL pointer from strerror() to vsnprintf via VSL In-Reply-To: <20181101110711.22DFBA1900@lists.varnish-cache.org> References: <20181101110711.22DFBA1900@lists.varnish-cache.org> Message-ID: <2794.1541079204@critter.freebsd.dk> -------- In message <20181101110711.22DFBA1900 at lists.varnish-cache.org>, Nils Goroll wri tes: >commit ee33662a162cbc4fcc7fb8a93d143f85b7786eae >Author: Nils Goroll >Date: Thu Nov 1 11:45:11 2018 +0100 > > do not pass a NULL pointer from strerror() to vsnprintf via VSL > > (at leat on solaris) strerror() itself may fail for an out-of-memory > condition (because the localization code contains memory > allocations). In order to handle this situation, we need to save the > original errno because strerror() may also set errno. > > This issue exists in many more places all over the code, but in the > pool_breed case we likely failed pthread_create for an out-of-memory > condition, and in the panic handler we want to make sure that we > trip no follow-up panic under any circumstances. > > In general, while fixing all strerror() calls would unnecessarily > complicate the code, doing so should be justified for these cases. > > Fixes #2815 I suggest adding a wrapper: const char * VSOMETHING_strerror(int e) { const char *p; p = strerror(e); if (p != NULL) return (p) // XXX: try strerror_p(3) if it exists return ("strerror(3) returned NULL"); } Rather than polute all the code... -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at FreeBSD.org Thu Nov 1 16:03:09 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Thu, 1 Nov 2018 16:03:09 +0000 (UTC) Subject: [master] 53a5e2561 Fix for 32 bit archs Message-ID: <20181101160309.5ED04A7903@lists.varnish-cache.org> commit 53a5e2561e750e393d3423d08f6aaf571b90e4b9 Author: Poul-Henning Kamp Date: Thu Nov 1 16:02:30 2018 +0000 Fix for 32 bit archs diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c index 3a76f6c7e..c04c926d2 100644 --- a/lib/libvmod_debug/vmod_debug.c +++ b/lib/libvmod_debug/vmod_debug.c @@ -586,8 +586,8 @@ xyzzy_priv_perf(VRT_CTX, VCL_INT size, VCL_INT rounds) d = (t1 - t0) * 1e9 / size / rounds; VSLb(ctx->vsl, SLT_Debug, - "perf size %ld rounds %ld time %.9fns check %ld", - size, rounds, d, check); + "perf size %jd rounds %jd time %.9fns check %jd", + (intmax_t)size, (intmax_t)rounds, d, (uintmax_t)check); return (d); } From nils.goroll at uplex.de Thu Nov 1 16:44:08 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 1 Nov 2018 16:44:08 +0000 (UTC) Subject: [master] eb28fe773 more 32bit fixes Message-ID: <20181101164408.31C6AA874D@lists.varnish-cache.org> commit eb28fe7733eee2fe365ad50365dd6c0f63509fb5 Author: Nils Goroll Date: Thu Nov 1 17:39:45 2018 +0100 more 32bit fixes diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c index c04c926d2..6816451aa 100644 --- a/lib/libvmod_debug/vmod_debug.c +++ b/lib/libvmod_debug/vmod_debug.c @@ -565,7 +565,7 @@ xyzzy_priv_perf(VRT_CTX, VCL_INT size, VCL_INT rounds) CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); for (s = 1; s <= size; s++) { - p = VRT_priv_task(ctx, (void *)s); + p = VRT_priv_task(ctx, (void *)(uintptr_t)s); if (p == NULL) { VRT_fail(ctx, "no priv task - out of ws?"); return (-1.0); @@ -576,7 +576,7 @@ xyzzy_priv_perf(VRT_CTX, VCL_INT size, VCL_INT rounds) t0 = VTIM_mono(); for (r = 0; r < rounds; r++) { for (s = 1; s <= size; s++) { - p = VRT_priv_task(ctx, (void *)s); + p = VRT_priv_task(ctx, (void *)(uintptr_t)s); check += (uintptr_t)p->priv; p->priv = (void *)(uintptr_t)(s * rounds + r); } From fgsch at lodoss.net Fri Nov 2 12:05:13 2018 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Fri, 2 Nov 2018 12:05:13 +0000 (UTC) Subject: [master] 0a833fedc Add a debug flag to keep VCL C and so files around Message-ID: <20181102120513.9B12965613@lists.varnish-cache.org> commit 0a833fedce97aedd66321c5176b0eaecb19130f3 Author: Federico G. Schwindt Date: Fri Nov 2 12:02:38 2018 +0000 Add a debug flag to keep VCL C and so files around Enabled when using varnishtest -L. diff --git a/bin/varnishd/mgt/mgt_vcc.c b/bin/varnishd/mgt/mgt_vcc.c index 9777584f8..2b476307b 100644 --- a/bin/varnishd/mgt/mgt_vcc.c +++ b/bin/varnishd/mgt/mgt_vcc.c @@ -380,7 +380,8 @@ mgt_VccCompile(struct cli *cli, struct vclprog *vcl, const char *vclname, } AZ(fclose(fcs)); - (void)unlink(vp.csrcfile); + if (!MGT_DO_DEBUG(DBG_VCL_KEEP)) + (void)unlink(vp.csrcfile); free(vp.csrcfile); free(vp.dir); diff --git a/bin/varnishd/mgt/mgt_vcl.c b/bin/varnishd/mgt/mgt_vcl.c index 62fe45061..0edc68168 100644 --- a/bin/varnishd/mgt/mgt_vcl.c +++ b/bin/varnishd/mgt/mgt_vcl.c @@ -257,7 +257,8 @@ mgt_vcl_del(struct vclprog *vp) if (vp->state != VCL_STATE_LABEL) vcl_count--; if (vp->fname != NULL) { - AZ(unlink(vp->fname)); + if (!MGT_DO_DEBUG(DBG_VCL_KEEP)) + AZ(unlink(vp->fname)); p = strrchr(vp->fname, '/'); AN(p); *p = '\0'; diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c index ac7e17adb..0c7b3f5a3 100644 --- a/bin/varnishtest/vtc_varnish.c +++ b/bin/varnishtest/vtc_varnish.c @@ -413,8 +413,9 @@ varnish_launch(struct varnish *v) if (vtc_witness) VSB_cat(vsb, " -p debug=+witness"); if (leave_temp) { - VSB_cat(vsb, " -p debug=+vsm_keep"); + VSB_cat(vsb, " -p debug=+vcl_keep"); VSB_cat(vsb, " -p debug=+vmod_so_keep"); + VSB_cat(vsb, " -p debug=+vsm_keep"); } VSB_printf(vsb, " -l 2m"); VSB_printf(vsb, " -p auto_restart=off"); diff --git a/include/tbl/debug_bits.h b/include/tbl/debug_bits.h index 4ac3ef60f..8bf099d5d 100644 --- a/include/tbl/debug_bits.h +++ b/include/tbl/debug_bits.h @@ -50,6 +50,7 @@ DEBUG_BIT(H2_NOCHECK, h2_nocheck, "Disable various H2 checks") DEBUG_BIT(VMOD_SO_KEEP, vmod_so_keep, "Keep copied VMOD libraries") DEBUG_BIT(PROCESSORS, processors, "Fetch/Deliver processors") DEBUG_BIT(PROTOCOL, protocol, "Protocol debugging") +DEBUG_BIT(VCL_KEEP, vcl_keep, "Keep VCL C and so files") #undef DEBUG_BIT /*lint -restore */ From dridi.boukelmoune at gmail.com Fri Nov 2 16:56:07 2018 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 2 Nov 2018 16:56:07 +0000 (UTC) Subject: [master] 2f07246f2 Elevate privileges to acquire the CLI file Message-ID: <20181102165607.407CA97729@lists.varnish-cache.org> commit 2f07246f2dee2d90a0f79b8df1af68db56209e5b Author: Dridi Boukelmoune Date: Fri Nov 2 17:54:48 2018 +0100 Elevate privileges to acquire the CLI file diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c index 237fb0f7f..efca546cd 100644 --- a/bin/varnishd/mgt/mgt_main.c +++ b/bin/varnishd/mgt/mgt_main.c @@ -622,10 +622,12 @@ main(int argc, char * const *argv) case 'I': if (I_fd >= 0) ARGV_ERR("\tOnly one -I allowed\n"); + VJ_master(JAIL_MASTER_FILE); I_fd = open(optarg, O_RDONLY); if (I_fd < 0) ARGV_ERR("\tCant open %s: %s\n", optarg, strerror(errno)); + VJ_master(JAIL_MASTER_LOW); break; case 'l': av = VAV_Parse(optarg, NULL, ARGV_COMMA); From hermunn at varnish-software.com Sun Nov 4 16:44:10 2018 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Sun, 4 Nov 2018 16:44:10 +0000 (UTC) Subject: [6.0] 1fade8fd6 Change log entries for 6.0.2 Message-ID: <20181104164410.4FB02AEA0C@lists.varnish-cache.org> commit 1fade8fd6633cfc51a84c65b71a9581df23780df Author: P?l Hermunn Johansen Date: Sun Nov 4 17:43:01 2018 +0100 Change log entries for 6.0.2 These are the most significant changes for 6.0.2, as far as I can see. diff --git a/doc/changes.rst b/doc/changes.rst index ce5859699..faeaaaa84 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -26,6 +26,100 @@ http://varnish-cache.org/docs/trunk/whats-new/index.html and via individual releases. These documents are updated as part of the release process. +================================ +Varnish Cache 6.0.2 (unreleased) +================================ + +* Fix and test objhead refcount for hit-for-pass (2654_, 2754_, 2760_) + +* Allow a string argument to return(fail("Because!")); (2694_) + +* Improve VCC error messages (2696_) + +* Fix obj.hits in vcl_hit (2746_) + +* Improvements to how PRIV_TASK and PRIV_TOP are initialized (2708_, + 2749_) + +* fixed ``varnishhist`` display error (2780_) + +* In ``Error: out of workspace`` log entries, the workspace name is + now reported in lowercase + +* Adjust code generator python tools to python 3 and prefer python 3 + over python 2 where available + +* Clear the IMS object attribute when copying from a stale object + (2763_) + +* Implement and test ECMA-48 "REP" sequence to fix test case + u00008.vtc on some newer platforms. (2668_) + +* Don't mess with C-L when responding to HEAD (2744_) + +* Align handling of STRINGS derived types (2745_) + +* Fix some stats metrics (vsc) which were wrongly marked as _gauge_ + +* Varnishhist: Ignore non-positive values when accumulating (2773_) + +* Fix production of VTC documentation (2777_) + +* Fix ``varnishd -I`` (2782_) + +* Fix ``varnishstat -f`` in curses mode (interactively, without + ``-1``, 2787_) + +* Handle an out-of-workspace condition in HTTP/2 delivery more + gracefully (2589_) + +* Added a thread pool watchdog which will restart the worker process + if scheduling tasks onto worker threads appears stuck. The new + parameter ``thread_pool_watchdog`` configures it. (2418_, 2794_) + +* Clarify and test object slimming for hfp+hfm. (2768_) + +* Allow PRIORITY frames on closed streams (2775_) + +* Hardening of the h2_frame_f callbacks (2781_) + +* Added a JSON section to varnish-cli(7) (2783_) + +* Improved varnish log client performance (2788_) + +* Change nanosecond precision timestamps into microseconds (2792_) + +* Only dlclose() Vmods after all "fini" processing (2800_) + +.. _2418: https://github.com/varnishcache/varnish-cache/issues/2418 +.. _2589: https://github.com/varnishcache/varnish-cache/issues/2589 +.. _2654: https://github.com/varnishcache/varnish-cache/issues/2654 +.. _2663: https://github.com/varnishcache/varnish-cache/pull/2663 +.. _2668: https://github.com/varnishcache/varnish-cache/issues/2668 +.. _2694: https://github.com/varnishcache/varnish-cache/issues/2694 +.. _2696: https://github.com/varnishcache/varnish-cache/issues/2696 +.. _2708: https://github.com/varnishcache/varnish-cache/issues/2708 +.. _2744: https://github.com/varnishcache/varnish-cache/pull/2744 +.. _2745: https://github.com/varnishcache/varnish-cache/issues/2745 +.. _2746: https://github.com/varnishcache/varnish-cache/issues/2746 +.. _2749: https://github.com/varnishcache/varnish-cache/issues/2749 +.. _2754: https://github.com/varnishcache/varnish-cache/issues/2754 +.. _2760: https://github.com/varnishcache/varnish-cache/pull/2760 +.. _2763: https://github.com/varnishcache/varnish-cache/issues/2763 +.. _2768: https://github.com/varnishcache/varnish-cache/issues/2768 +.. _2773: https://github.com/varnishcache/varnish-cache/issues/2773 +.. _2775: https://github.com/varnishcache/varnish-cache/issues/2775 +.. _2777: https://github.com/varnishcache/varnish-cache/issues/2777 +.. _2780: https://github.com/varnishcache/varnish-cache/issues/2780 +.. _2781: https://github.com/varnishcache/varnish-cache/pull/2781 +.. _2782: https://github.com/varnishcache/varnish-cache/issues/2782 +.. _2783: https://github.com/varnishcache/varnish-cache/issues/2783 +.. _2787: https://github.com/varnishcache/varnish-cache/issues/2787 +.. _2788: https://github.com/varnishcache/varnish-cache/issues/2788 +.. _2792: https://github.com/varnishcache/varnish-cache/pull/2792 +.. _2794: https://github.com/varnishcache/varnish-cache/issues/2794 +.. _2800: https://github.com/varnishcache/varnish-cache/issues/2800 + ================================ Varnish Cache 6.0.1 (2018-08-29) From phk at FreeBSD.org Mon Nov 5 07:37:16 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 5 Nov 2018 07:37:16 +0000 (UTC) Subject: [master] 53db0bd28 FlexeLinting Message-ID: <20181105073716.89F1B633E1@lists.varnish-cache.org> commit 53db0bd28d0f4fb25aedbb1d452d0585b10afbe3 Author: Poul-Henning Kamp Date: Mon Nov 5 07:36:27 2018 +0000 FlexeLinting diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c index 6816451aa..61029ad94 100644 --- a/lib/libvmod_debug/vmod_debug.c +++ b/lib/libvmod_debug/vmod_debug.c @@ -577,16 +577,17 @@ xyzzy_priv_perf(VRT_CTX, VCL_INT size, VCL_INT rounds) for (r = 0; r < rounds; r++) { for (s = 1; s <= size; s++) { p = VRT_priv_task(ctx, (void *)(uintptr_t)s); + AN(p); check += (uintptr_t)p->priv; p->priv = (void *)(uintptr_t)(s * rounds + r); } } t1 = VTIM_mono(); - d = (t1 - t0) * 1e9 / size / rounds; + d = (t1 - t0) * 1e9 / ((double)size * (double)rounds); VSLb(ctx->vsl, SLT_Debug, - "perf size %jd rounds %jd time %.9fns check %jd", + "perf size %jd rounds %jd time %.1fns check %jd", (intmax_t)size, (intmax_t)rounds, d, (uintmax_t)check); return (d); From phk at FreeBSD.org Mon Nov 5 09:15:13 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 5 Nov 2018 09:15:13 +0000 (UTC) Subject: [master] a48b80a2b FlexeLinting Message-ID: <20181105091513.4408A6524D@lists.varnish-cache.org> commit a48b80a2b1da5d578f24d48b5b8cfb2ab59f1501 Author: Poul-Henning Kamp Date: Mon Nov 5 07:56:54 2018 +0000 FlexeLinting diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c index bcf996f94..586f7cec1 100644 --- a/bin/varnishncsa/varnishncsa.c +++ b/bin/varnishncsa/varnishncsa.c @@ -39,8 +39,6 @@ #include "config.h" -#define _WITH_GETLINE - #include #include #include @@ -612,7 +610,7 @@ parse_x_format(char *buf) VUT_Error(vut, 1, "Unknown log tag: %s", buf); assert(slt >= 0); - addf_vsl(slt, lval, r); + addf_vsl((enum VSL_tag_e)slt, lval, r); return; } VUT_Error(vut, 1, "Unknown formatting extension: %s", buf); @@ -872,7 +870,7 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], void *priv) { struct VSL_transaction *t; - unsigned tag; + enum VSL_tag_e tag; const char *b, *e, *p; struct watch *w; int i, skip, be_mark; From phk at FreeBSD.org Mon Nov 5 09:15:13 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 5 Nov 2018 09:15:13 +0000 (UTC) Subject: [master] 619eb8519 Simplify frag_fields() and add usage asserts. Message-ID: <20181105091513.4E6336524F@lists.varnish-cache.org> commit 619eb85194e86bc884a66eaf4e7bc189e4d13def Author: Poul-Henning Kamp Date: Mon Nov 5 09:06:51 2018 +0000 Simplify frag_fields() and add usage asserts. diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c index 586f7cec1..6c859358e 100644 --- a/bin/varnishncsa/varnishncsa.c +++ b/bin/varnishncsa/varnishncsa.c @@ -776,33 +776,31 @@ frag_fields(int force, const char *b, const char *e, ...) AN(e); va_start(ap, e); - field = va_arg(ap, int); - frag = va_arg(ap, struct fragment *); - for (n = 1, q = b; q < e; n++) { - /* caller must sort the fields, or this loop will not work: */ - assert(field >= n); - AN(frag); - - p = q; - /* Skip WS */ - while (p < e && isspace(*p)) - p++; - q = p; - /* Skip non-WS */ - while (q < e && !isspace(*q)) - q++; - - if (field == n) { - if (frag->gen != CTX.gen || force) { - /* We only grab the same matching field once */ - frag->gen = CTX.gen; - frag->b = p; - frag->e = q; - } - field = va_arg(ap, int); - if (field == 0) - break; - frag = va_arg(ap, struct fragment *); + n = 0; + while (1) { + field = va_arg(ap, int); + frag = va_arg(ap, struct fragment *); + if (field == 0) { + AZ(frag); + break; + } + p = q = NULL; + while (n < field) { + while (b < e && isspace(*b)) + b++; + p = b; + while (b < e && !isspace(*b)) + b++; + q = b; + n++; + } + assert(p != NULL && q != NULL); + assert(p < e && q > p); + if (frag->gen != CTX.gen || force) { + /* We only grab the same matching field once */ + frag->gen = CTX.gen; + frag->b = p; + frag->e = q; } } va_end(ap); From phk at FreeBSD.org Mon Nov 5 09:15:13 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 5 Nov 2018 09:15:13 +0000 (UTC) Subject: [master] 439a5dfe6 Add missing continue Message-ID: <20181105091513.7A4C565251@lists.varnish-cache.org> commit 439a5dfe69c287ca124c98fe46c32a87ee40d499 Author: Poul-Henning Kamp Date: Mon Nov 5 09:12:58 2018 +0000 Add missing continue diff --git a/bin/varnishtest/vtc_process.c b/bin/varnishtest/vtc_process.c index dd6880e37..c6a5db295 100644 --- a/bin/varnishtest/vtc_process.c +++ b/bin/varnishtest/vtc_process.c @@ -1068,6 +1068,7 @@ cmd_process(CMD_ARGS) term_resize(p, lin, col); AZ(pthread_mutex_unlock(&p->mtx)); process_winsz(p, p->fd_term); + continue; } if (!strcmp(*av, "-write")) { process_write(p, av[1]); @@ -1086,8 +1087,7 @@ cmd_process(CMD_ARGS) continue; } if (**av == '-' || spec_set) - vtc_fatal(p->vl, "Unknown process argument: %s", - *av); + vtc_fatal(p->vl, "Unknown process argument: %s", *av); REPLACE(p->spec, *av); spec_set = 1; } From phk at FreeBSD.org Mon Nov 5 09:15:13 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 5 Nov 2018 09:15:13 +0000 (UTC) Subject: [master] 023682a78 Run a varnishnsca via process and dump the screen Message-ID: <20181105091513.B3DDB65254@lists.varnish-cache.org> commit 023682a784973d48486b5217da928c46d6a1e878 Author: Poul-Henning Kamp Date: Mon Nov 5 09:13:53 2018 +0000 Run a varnishnsca via process and dump the screen diff --git a/bin/varnishtest/tests/u00003.vtc b/bin/varnishtest/tests/u00003.vtc index e18d9b912..9dd5585cb 100644 --- a/bin/varnishtest/tests/u00003.vtc +++ b/bin/varnishtest/tests/u00003.vtc @@ -29,6 +29,8 @@ shell { -w ${tmpdir}/ncsa.log } +process p1 -winsz 25 132 {varnishncsa -n ${v1_name}} -start + delay 1 client c1 { @@ -146,3 +148,9 @@ req (\d+) rxreq \5 - - -$} \ {varnishncsa -n ${v1_name} -d -F "%{VSL:Begin}x \ %{VSL:Begin[2]}x %{VSL:Timestamp:Resp}x \ %{VSL:Timestamp:Resp[2]}x %{VSL:Timestamp:foo}x"} + +process p1 -stop -screen-dump +process p1 -expect-text 1 0 {"GET http://127.0.0.1/1?foo=bar HTTP/1.1" 200 100 "-" "-"} +process p1 -expect-text 1 0 { - user [} +process p1 -expect-text 2 0 {"GET http://127.0.0.1/1?foo=bar HTTP/1.1" 404 248 "-" "-"} +process p1 -expect-text 3 0 {"GET http://127.0.0.1/2 HTTP/1.1" - - "-" "-"} From phk at FreeBSD.org Mon Nov 5 11:45:13 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 5 Nov 2018 11:45:13 +0000 (UTC) Subject: [master] c4481ca5a Forgot about jails and IP#s Message-ID: <20181105114513.A017B94AE9@lists.varnish-cache.org> commit c4481ca5aa4b54a113facb2b09a40ece4de963f5 Author: Poul-Henning Kamp Date: Mon Nov 5 11:44:23 2018 +0000 Forgot about jails and IP#s diff --git a/bin/varnishtest/tests/u00003.vtc b/bin/varnishtest/tests/u00003.vtc index 9dd5585cb..bc86232bb 100644 --- a/bin/varnishtest/tests/u00003.vtc +++ b/bin/varnishtest/tests/u00003.vtc @@ -150,7 +150,7 @@ req (\d+) rxreq \5 - - -$} \ %{VSL:Timestamp:Resp[2]}x %{VSL:Timestamp:foo}x"} process p1 -stop -screen-dump -process p1 -expect-text 1 0 {"GET http://127.0.0.1/1?foo=bar HTTP/1.1" 200 100 "-" "-"} +process p1 -expect-text 1 0 {/1?foo=bar HTTP/1.1" 200 100 "-" "-"} process p1 -expect-text 1 0 { - user [} -process p1 -expect-text 2 0 {"GET http://127.0.0.1/1?foo=bar HTTP/1.1" 404 248 "-" "-"} -process p1 -expect-text 3 0 {"GET http://127.0.0.1/2 HTTP/1.1" - - "-" "-"} +process p1 -expect-text 2 0 {/1?foo=bar HTTP/1.1" 404 248 "-" "-"} +process p1 -expect-text 3 0 {/2 HTTP/1.1" - - "-" "-"} From nils.goroll at uplex.de Mon Nov 5 11:48:09 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 5 Nov 2018 11:48:09 +0000 (UTC) Subject: [master] 7a13ff561 fix VRT_priv_task for calls from vcl_pipe {} and test for it Message-ID: <20181105114809.CD37194E12@lists.varnish-cache.org> commit 7a13ff561c43391c82f82dd90a250f2b9f88a3d3 Author: Nils Goroll Date: Mon Nov 5 12:46:38 2018 +0100 fix VRT_priv_task for calls from vcl_pipe {} and test for it Fixes #2820 diff --git a/bin/varnishd/cache/cache_vrt_priv.c b/bin/varnishd/cache/cache_vrt_priv.c index 754605146..6b3b455fe 100644 --- a/bin/varnishd/cache/cache_vrt_priv.c +++ b/bin/varnishd/cache/cache_vrt_priv.c @@ -35,6 +35,7 @@ #include #include "cache_varnishd.h" +#include "vcl.h" struct vrt_priv { unsigned magic; @@ -127,7 +128,9 @@ VRT_priv_task(VRT_CTX, const void *vmod_id) struct vmod_priv *vp; CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - assert(ctx->req == NULL || ctx->bo == NULL); + assert(ctx->req == NULL || ctx->bo == NULL || + ctx->method == VCL_MET_PIPE); + if (ctx->req) { CHECK_OBJ_NOTNULL(ctx->req, REQ_MAGIC); id = (uintptr_t)ctx->req; diff --git a/bin/varnishtest/tests/v00041.vtc b/bin/varnishtest/tests/v00041.vtc index 533f7d246..e6cfc3c8e 100644 --- a/bin/varnishtest/tests/v00041.vtc +++ b/bin/varnishtest/tests/v00041.vtc @@ -6,6 +6,11 @@ server s1 { rxreq txresp + rxreq + txresp + expect_close + accept + rxreq txresp } -start @@ -29,6 +34,15 @@ varnish v1 -arg "-p debug=+vclrel -p workspace_client=1m" -vcl+backend { if (req.url == "/perf") { return (synth(200)); } + if (req.url == "/pipe") { + return (pipe); + } + debug.test_priv_task(req.url); + set req.http.x0 = debug.test_priv_task(); + debug.test_priv_task("bazz"); + } + + sub vcl_pipe { debug.test_priv_task(req.url); set req.http.x0 = debug.test_priv_task(); debug.test_priv_task("bazz"); @@ -125,6 +139,9 @@ client c1 { txreq -url /perf rxresp + + txreq -url /pipe + rxresp } -run shell "echo 'vcl 4.0; backend foo { .host = \"${s1_addr}\"; .port = \"${s1_port}\"; }' > ${tmpdir}/_b00014.vcl" From nils.goroll at uplex.de Mon Nov 5 15:02:11 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 5 Nov 2018 15:02:11 +0000 (UTC) Subject: [master] ba2bd1b47 add timestamps to the std.ip test Message-ID: <20181105150211.86C649C014@lists.varnish-cache.org> commit ba2bd1b472023d2b461450b536a79b04c17f1de4 Author: Nils Goroll Date: Mon Nov 5 15:57:14 2018 +0100 add timestamps to the std.ip test diff --git a/bin/varnishtest/tests/m00011.vtc b/bin/varnishtest/tests/m00011.vtc index cd4980665..95517c7c8 100644 --- a/bin/varnishtest/tests/m00011.vtc +++ b/bin/varnishtest/tests/m00011.vtc @@ -9,14 +9,23 @@ varnish v1 -vcl+backend { import std; sub vcl_deliver { + std.timestamp("t0"); set resp.http.foo0 = std.ip("8.8.8.*", client.ip); + std.timestamp("8.8.8.*, client.ip"); set resp.http.foo1 = std.ip("9.9.9.*", server.ip); + std.timestamp("9.9.9.*, server.ip"); set resp.http.foo2 = std.ip("1.2.3.*", "127.0.0.2"); + std.timestamp("1.2.3.*"); set resp.http.foo3 = std.ip("1.2.3.5", "127.0.0.3"); + std.timestamp("1.2.3.5"); set resp.http.foo4 = std.ip("2001:db8::", "[::1]"); + std.timestamp("2001:db8::"); set resp.http.foo5 = std.ip("2001::db8::", "[::1]"); + std.timestamp("2001::db8::"); set resp.http.foo6 = std.ip("localhost", "0.0.0.0", resolve = false); + std.timestamp("localhost, resolve = false"); set resp.http.foo7 = std.ip("1.2.3.4", "0.0.0.0", resolve = false); + std.timestamp("1.2.3.4, resolve = false"); } } -start From nils.goroll at uplex.de Mon Nov 5 17:04:11 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 5 Nov 2018 17:04:11 +0000 (UTC) Subject: [master] 6db410864 manual whitespace fixes Message-ID: <20181105170412.0E782A1A6E@lists.varnish-cache.org> commit 6db410864ff05146e573b57ed27a8be8bee09fb3 Author: Nils Goroll Date: Mon Nov 5 17:50:41 2018 +0100 manual whitespace fixes Ref: #2812 diff --git a/include/vtree.h b/include/vtree.h index 064ce00b4..4f53c7551 100644 --- a/include/vtree.h +++ b/include/vtree.h @@ -305,7 +305,7 @@ struct name { \ #define VRBT_BLACK 0 #define VRBT_RED 1 -#define VRBT_ENTRY(type) \ +#define VRBT_ENTRY(type) \ struct { \ struct type *rbe_left; /* left element */ \ struct type *rbe_right; /* right element */ \ @@ -318,7 +318,7 @@ struct { \ #define VRBT_PARENT(elm, field) (elm)->field.rbe_parent #define VRBT_COLOR(elm, field) (elm)->field.rbe_color #define VRBT_ROOT(head) (head)->rbh_root -#define VRBT_EMPTY(head) (VRBT_ROOT(head) == NULL) +#define VRBT_EMPTY(head) (VRBT_ROOT(head) == NULL) #define VRBT_SET(elm, parent, field) do { \ VRBT_PARENT(elm, field) = parent; \ @@ -349,7 +349,7 @@ struct { \ } else \ (head)->rbh_root = (tmp); \ VRBT_LEFT(tmp, field) = (elm); \ - VRBT_PARENT(elm, field) = (tmp); \ + VRBT_PARENT(elm, field) = (tmp); \ VRBT_AUGMENT(tmp); \ if ((VRBT_PARENT(tmp, field))) \ VRBT_AUGMENT(VRBT_PARENT(tmp, field)); \ @@ -369,16 +369,16 @@ struct { \ } else \ (head)->rbh_root = (tmp); \ VRBT_RIGHT(tmp, field) = (elm); \ - VRBT_PARENT(elm, field) = (tmp); \ + VRBT_PARENT(elm, field) = (tmp); \ VRBT_AUGMENT(tmp); \ if ((VRBT_PARENT(tmp, field))) \ VRBT_AUGMENT(VRBT_PARENT(tmp, field)); \ } while (/*CONSTCOND*/ 0) /* Generates prototypes and inline functions */ -#define VRBT_PROTOTYPE(name, type, field, cmp) \ +#define VRBT_PROTOTYPE(name, type, field, cmp) \ VRBT_PROTOTYPE_INTERNAL(name, type, field, cmp,) -#define VRBT_PROTOTYPE_STATIC(name, type, field, cmp) \ +#define VRBT_PROTOTYPE_STATIC(name, type, field, cmp) \ VRBT_PROTOTYPE_INTERNAL(name, type, field, cmp, v_unused_ static) #define VRBT_PROTOTYPE_INTERNAL(name, type, field, cmp, attr) \ /*lint -esym(528, name##_VRBT_*) */ \ @@ -396,9 +396,9 @@ attr struct type *name##_VRBT_MINMAX(const struct name *, int); \ /* Main rb operation. * Moves node close to the key of elm to top */ -#define VRBT_GENERATE(name, type, field, cmp) \ +#define VRBT_GENERATE(name, type, field, cmp) \ VRBT_GENERATE_INTERNAL(name, type, field, cmp,) -#define VRBT_GENERATE_STATIC(name, type, field, cmp) \ +#define VRBT_GENERATE_STATIC(name, type, field, cmp) \ VRBT_GENERATE_INTERNAL(name, type, field, cmp, v_unused_ static) #define VRBT_GENERATE_INTERNAL(name, type, field, cmp, attr) \ attr void \ @@ -410,13 +410,13 @@ name##_VRBT_INSERT_COLOR(struct name *head, struct type *elm) \ gparent = VRBT_PARENT(parent, field); \ if (parent == VRBT_LEFT(gparent, field)) { \ tmp = VRBT_RIGHT(gparent, field); \ - if (tmp && VRBT_COLOR(tmp, field) == VRBT_RED) { \ + if (tmp && VRBT_COLOR(tmp, field) == VRBT_RED) {\ VRBT_COLOR(tmp, field) = VRBT_BLACK; \ VRBT_SET_BLACKRED(parent, gparent, field);\ elm = gparent; \ continue; \ } \ - if (VRBT_RIGHT(parent, field) == elm) { \ + if (VRBT_RIGHT(parent, field) == elm) { \ VRBT_ROTATE_LEFT(head, parent, tmp, field);\ tmp = parent; \ parent = elm; \ @@ -425,8 +425,8 @@ name##_VRBT_INSERT_COLOR(struct name *head, struct type *elm) \ VRBT_SET_BLACKRED(parent, gparent, field); \ VRBT_ROTATE_RIGHT(head, gparent, tmp, field); \ } else { \ - tmp = VRBT_LEFT(gparent, field); \ - if (tmp && VRBT_COLOR(tmp, field) == VRBT_RED) { \ + tmp = VRBT_LEFT(gparent, field); \ + if (tmp && VRBT_COLOR(tmp, field) == VRBT_RED) {\ VRBT_COLOR(tmp, field) = VRBT_BLACK; \ VRBT_SET_BLACKRED(parent, gparent, field);\ elm = gparent; \ @@ -453,11 +453,11 @@ name##_VRBT_REMOVE_COLOR(struct name *head, struct type *parent, struct type *el elm != VRBT_ROOT(head)) { \ AN(parent); \ if (VRBT_LEFT(parent, field) == elm) { \ - tmp = VRBT_RIGHT(parent, field); \ - if (VRBT_COLOR(tmp, field) == VRBT_RED) { \ + tmp = VRBT_RIGHT(parent, field); \ + if (VRBT_COLOR(tmp, field) == VRBT_RED) { \ VRBT_SET_BLACKRED(tmp, parent, field); \ VRBT_ROTATE_LEFT(head, parent, tmp, field);\ - tmp = VRBT_RIGHT(parent, field); \ + tmp = VRBT_RIGHT(parent, field); \ } \ if ((VRBT_LEFT(tmp, field) == NULL || \ VRBT_COLOR(VRBT_LEFT(tmp, field), field) == VRBT_BLACK) &&\ @@ -475,7 +475,7 @@ name##_VRBT_REMOVE_COLOR(struct name *head, struct type *parent, struct type *el VRBT_COLOR(oleft, field) = VRBT_BLACK;\ VRBT_COLOR(tmp, field) = VRBT_RED;\ VRBT_ROTATE_RIGHT(head, tmp, oleft, field);\ - tmp = VRBT_RIGHT(parent, field); \ + tmp = VRBT_RIGHT(parent, field);\ } \ VRBT_COLOR(tmp, field) = VRBT_COLOR(parent, field);\ VRBT_COLOR(parent, field) = VRBT_BLACK; \ @@ -487,7 +487,7 @@ name##_VRBT_REMOVE_COLOR(struct name *head, struct type *parent, struct type *el } \ } else { \ tmp = VRBT_LEFT(parent, field); \ - if (VRBT_COLOR(tmp, field) == VRBT_RED) { \ + if (VRBT_COLOR(tmp, field) == VRBT_RED) { \ VRBT_SET_BLACKRED(tmp, parent, field); \ VRBT_ROTATE_RIGHT(head, parent, tmp, field);\ tmp = VRBT_LEFT(parent, field); \ @@ -531,7 +531,7 @@ name##_VRBT_REMOVE(struct name *head, struct type *elm) \ int color; \ if (VRBT_LEFT(elm, field) == NULL) \ child = VRBT_RIGHT(elm, field); \ - else if (VRBT_RIGHT(elm, field) == NULL) \ + else if (VRBT_RIGHT(elm, field) == NULL) \ child = VRBT_LEFT(elm, field); \ else { \ struct type *left; \ @@ -550,7 +550,7 @@ name##_VRBT_REMOVE(struct name *head, struct type *elm) \ VRBT_RIGHT(parent, field) = child; \ VRBT_AUGMENT(parent); \ } else \ - VRBT_ROOT(head) = child; \ + VRBT_ROOT(head) = child; \ if (VRBT_PARENT(elm, field) == old) \ parent = elm; \ (elm)->field = (old)->field; \ @@ -562,14 +562,14 @@ name##_VRBT_REMOVE(struct name *head, struct type *elm) \ VRBT_AUGMENT(VRBT_PARENT(old, field)); \ } else \ VRBT_ROOT(head) = elm; \ - VRBT_PARENT(VRBT_LEFT(old, field), field) = elm; \ + VRBT_PARENT(VRBT_LEFT(old, field), field) = elm; \ if (VRBT_RIGHT(old, field)) \ - VRBT_PARENT(VRBT_RIGHT(old, field), field) = elm; \ + VRBT_PARENT(VRBT_RIGHT(old, field), field) = elm;\ if (parent) { \ left = parent; \ do { \ VRBT_AUGMENT(left); \ - } while ((left = VRBT_PARENT(left, field)) != NULL); \ + } while ((left = VRBT_PARENT(left, field)) != NULL);\ } \ goto color; \ } \ @@ -584,7 +584,7 @@ name##_VRBT_REMOVE(struct name *head, struct type *elm) \ VRBT_RIGHT(parent, field) = child; \ VRBT_AUGMENT(parent); \ } else \ - VRBT_ROOT(head) = child; \ + VRBT_ROOT(head) = child; \ color: \ if (color == VRBT_BLACK) { \ name##_VRBT_REMOVE_COLOR(head, parent, child); \ @@ -615,7 +615,7 @@ name##_VRBT_INSERT(struct name *head, struct type *elm) \ if (comp < 0) \ VRBT_LEFT(parent, field) = elm; \ else \ - VRBT_RIGHT(parent, field) = elm; \ + VRBT_RIGHT(parent, field) = elm; \ VRBT_AUGMENT(parent); \ } else \ VRBT_ROOT(head) = elm; \ @@ -694,7 +694,7 @@ name##_VRBT_PREV(struct type *elm) \ elm = VRBT_RIGHT(elm, field); \ } else { \ if (VRBT_PARENT(elm, field) && \ - (elm == VRBT_RIGHT(VRBT_PARENT(elm, field), field))) \ + (elm == VRBT_RIGHT(VRBT_PARENT(elm, field), field)))\ elm = VRBT_PARENT(elm, field); \ else { \ while (VRBT_PARENT(elm, field) && \ @@ -730,11 +730,11 @@ name##_VRBT_MINMAX(const struct name *head, int val) \ #define VRBT_NFIND(name, x, y) name##_VRBT_NFIND(x, y) #define VRBT_NEXT(name, x, y) name##_VRBT_NEXT(y) #define VRBT_PREV(name, x, y) name##_VRBT_PREV(y) -#define VRBT_MIN(name, x) name##_VRBT_MINMAX(x, VRBT_NEGINF) -#define VRBT_MAX(name, x) name##_VRBT_MINMAX(x, VRBT_INF) +#define VRBT_MIN(name, x) name##_VRBT_MINMAX(x, VRBT_NEGINF) +#define VRBT_MAX(name, x) name##_VRBT_MINMAX(x, VRBT_INF) #define VRBT_FOREACH(x, name, head) \ - for ((x) = VRBT_MIN(name, head); \ + for ((x) = VRBT_MIN(name, head); \ (x) != NULL; \ (x) = name##_VRBT_NEXT(x)) @@ -744,12 +744,12 @@ name##_VRBT_MINMAX(const struct name *head, int val) \ (x) = (y)) #define VRBT_FOREACH_SAFE(x, name, head, y) \ - for ((x) = VRBT_MIN(name, head); \ + for ((x) = VRBT_MIN(name, head); \ ((x) != NULL) && ((y) = name##_VRBT_NEXT(x), (x) != NULL); \ (x) = (y)) #define VRBT_FOREACH_REVERSE(x, name, head) \ - for ((x) = VRBT_MAX(name, head); \ + for ((x) = VRBT_MAX(name, head); \ (x) != NULL; \ (x) = name##_VRBT_PREV(x)) @@ -759,7 +759,7 @@ name##_VRBT_MINMAX(const struct name *head, int val) \ (x) = (y)) #define VRBT_FOREACH_REVERSE_SAFE(x, name, head, y) \ - for ((x) = VRBT_MAX(name, head); \ + for ((x) = VRBT_MAX(name, head); \ ((x) != NULL) && ((y) = name##_VRBT_PREV(x), (x) != NULL); \ (x) = (y)) From nils.goroll at uplex.de Mon Nov 5 17:04:11 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 5 Nov 2018 17:04:11 +0000 (UTC) Subject: [master] 08d8a294a Rename VRB_ -> VRBT_ for red/black tree macros Message-ID: <20181105170412.034DCA1A6D@lists.varnish-cache.org> commit 08d8a294a8970def12fa044cfb074ce59306f8b3 Author: Nils Goroll Date: Mon Nov 5 17:47:24 2018 +0100 Rename VRB_ -> VRBT_ for red/black tree macros sed 's:VRB_:VRBT_:g' -i "$FILES_CHANGED[@]" Closes #2812 diff --git a/bin/varnishtop/flint.lnt b/bin/varnishtop/flint.lnt index 696ce2d6f..20b82091e 100644 --- a/bin/varnishtop/flint.lnt +++ b/bin/varnishtop/flint.lnt @@ -8,8 +8,8 @@ -e732 // Loss of sign (arg. no. 2) (int to unsigned -e713 // Loss of precision (assignment) (unsigned long long to long long) --sem(t_order_VRB_INSERT, custodial(2)) --sem(t_key_VRB_INSERT, custodial(2)) +-sem(t_order_VRBT_INSERT, custodial(2)) +-sem(t_key_VRBT_INSERT, custodial(2)) /////////////////////////////////////////////////////////////////////// // Varnishstat specific diff --git a/bin/varnishtop/varnishtop.c b/bin/varnishtop/varnishtop.c index afe0dad92..2d72eadfe 100644 --- a/bin/varnishtop/varnishtop.c +++ b/bin/varnishtop/varnishtop.c @@ -70,8 +70,8 @@ struct top { char *rec_buf; int clen; unsigned hash; - VRB_ENTRY(top) e_order; - VRB_ENTRY(top) e_key; + VRBT_ENTRY(top) e_order; + VRBT_ENTRY(top) e_key; double count; }; @@ -85,8 +85,8 @@ static const char *ident; static volatile sig_atomic_t quit = 0; -static VRB_HEAD(t_order, top) h_order = VRB_INITIALIZER(&h_order); -static VRB_HEAD(t_key, top) h_key = VRB_INITIALIZER(&h_key); +static VRBT_HEAD(t_order, top) h_order = VRBT_INITIALIZER(&h_order); +static VRBT_HEAD(t_key, top) h_key = VRBT_INITIALIZER(&h_key); static inline int cmp_key(const struct top *a, const struct top *b) @@ -110,10 +110,10 @@ cmp_order(const struct top *a, const struct top *b) return (cmp_key(a, b)); } -VRB_PROTOTYPE_STATIC(t_order, top, e_order, cmp_order) -VRB_GENERATE_STATIC(t_order, top, e_order, cmp_order) -VRB_PROTOTYPE_STATIC(t_key, top, e_key, cmp_key) -VRB_GENERATE_STATIC(t_key, top, e_key, cmp_key) +VRBT_PROTOTYPE_STATIC(t_order, top, e_order, cmp_order) +VRBT_GENERATE_STATIC(t_order, top, e_order, cmp_order) +VRBT_PROTOTYPE_STATIC(t_key, top, e_key, cmp_key) +VRBT_GENERATE_STATIC(t_key, top, e_key, cmp_key) static int v_matchproto_(VSLQ_dispatch_f) accumulate(struct VSL_data *vsl, struct VSL_transaction * const pt[], @@ -155,12 +155,12 @@ accumulate(struct VSL_data *vsl, struct VSL_transaction * const pt[], t.rec_data = VSL_CDATA(tr->c->rec.ptr); AZ(pthread_mutex_lock(&mtx)); - tp = VRB_FIND(t_key, &h_key, &t); + tp = VRBT_FIND(t_key, &h_key, &t); if (tp) { - VRB_REMOVE(t_order, &h_order, tp); + VRBT_REMOVE(t_order, &h_order, tp); tp->count += 1.0; /* Reinsert to rebalance */ - VRB_INSERT(t_order, &h_order, tp); + VRBT_INSERT(t_order, &h_order, tp); } else { ntop++; tp = calloc(1, sizeof *tp); @@ -172,8 +172,8 @@ accumulate(struct VSL_data *vsl, struct VSL_transaction * const pt[], tp->rec_buf = strdup(t.rec_data); tp->rec_data = tp->rec_buf; AN(tp->rec_data); - VRB_INSERT(t_key, &h_key, tp); - VRB_INSERT(t_order, &h_order, tp); + VRBT_INSERT(t_key, &h_key, tp); + VRBT_INSERT(t_order, &h_order, tp); } AZ(pthread_mutex_unlock(&mtx)); @@ -226,8 +226,8 @@ update(int p) else AC(mvprintw(0, len - 1, "%s", q)); AC(mvprintw(0, 0, "list length %u", ntop)); - for (tp = VRB_MIN(t_order, &h_order); tp != NULL; tp = tp2) { - tp2 = VRB_NEXT(t_order, &h_order, tp); + for (tp = VRBT_MIN(t_order, &h_order); tp != NULL; tp = tp2) { + tp2 = VRBT_NEXT(t_order, &h_order, tp); if (++l < LINES) { len = tp->clen; @@ -243,8 +243,8 @@ update(int p) continue; tp->count += (1.0/3.0 - tp->count) / (double)n; if (tp->count * 10 < t || l > LINES * 10) { - VRB_REMOVE(t_key, &h_key, tp); - VRB_REMOVE(t_order, &h_order, tp); + VRBT_REMOVE(t_key, &h_key, tp); + VRBT_REMOVE(t_order, &h_order, tp); free(tp->rec_buf); free(tp); ntop--; @@ -316,8 +316,8 @@ static void dump(void) { struct top *tp, *tp2; - for (tp = VRB_MIN(t_order, &h_order); tp != NULL; tp = tp2) { - tp2 = VRB_NEXT(t_order, &h_order, tp); + for (tp = VRBT_MIN(t_order, &h_order); tp != NULL; tp = tp2) { + tp2 = VRBT_NEXT(t_order, &h_order, tp); printf("%9.2f %s %*.*s\n", tp->count, VSL_tags[tp->tag], tp->clen, tp->clen, tp->rec_data); diff --git a/flint.lnt b/flint.lnt index 0a9d6e9f0..7c7736d3d 100644 --- a/flint.lnt +++ b/flint.lnt @@ -142,9 +142,9 @@ /////////////////////////////////////////////////////////////////////// // --emacro(801, VRB_*) // goto considered bad --esym(534, *_VRB_REMOVE) // ignore retval --esym(534, *_VRB_INSERT) // ignore retval +-emacro(801, VRBT_*) // goto considered bad +-esym(534, *_VRBT_REMOVE) // ignore retval +-esym(534, *_VRBT_INSERT) // ignore retval /////////////////////////////////////////////////////////////////////// // diff --git a/include/vtree.h b/include/vtree.h index 649e8221d..064ce00b4 100644 --- a/include/vtree.h +++ b/include/vtree.h @@ -291,21 +291,21 @@ void name##_VSPLAY_MINMAX(struct name *head, int __comp) \ (x) = VSPLAY_NEXT(name, head, x)) /* Macros that define a red-black tree */ -#define VRB_HEAD(name, type) \ +#define VRBT_HEAD(name, type) \ struct name { \ struct type *rbh_root; /* root of the tree */ \ } -#define VRB_INITIALIZER(root) \ +#define VRBT_INITIALIZER(root) \ { NULL } -#define VRB_INIT(root) do { \ +#define VRBT_INIT(root) do { \ (root)->rbh_root = NULL; \ } while (/*CONSTCOND*/ 0) -#define VRB_BLACK 0 -#define VRB_RED 1 -#define VRB_ENTRY(type) \ +#define VRBT_BLACK 0 +#define VRBT_RED 1 +#define VRBT_ENTRY(type) \ struct { \ struct type *rbe_left; /* left element */ \ struct type *rbe_right; /* right element */ \ @@ -313,328 +313,328 @@ struct { \ int rbe_color; /* node color */ \ } -#define VRB_LEFT(elm, field) (elm)->field.rbe_left -#define VRB_RIGHT(elm, field) (elm)->field.rbe_right -#define VRB_PARENT(elm, field) (elm)->field.rbe_parent -#define VRB_COLOR(elm, field) (elm)->field.rbe_color -#define VRB_ROOT(head) (head)->rbh_root -#define VRB_EMPTY(head) (VRB_ROOT(head) == NULL) - -#define VRB_SET(elm, parent, field) do { \ - VRB_PARENT(elm, field) = parent; \ - VRB_LEFT(elm, field) = VRB_RIGHT(elm, field) = NULL; \ - VRB_COLOR(elm, field) = VRB_RED; \ +#define VRBT_LEFT(elm, field) (elm)->field.rbe_left +#define VRBT_RIGHT(elm, field) (elm)->field.rbe_right +#define VRBT_PARENT(elm, field) (elm)->field.rbe_parent +#define VRBT_COLOR(elm, field) (elm)->field.rbe_color +#define VRBT_ROOT(head) (head)->rbh_root +#define VRBT_EMPTY(head) (VRBT_ROOT(head) == NULL) + +#define VRBT_SET(elm, parent, field) do { \ + VRBT_PARENT(elm, field) = parent; \ + VRBT_LEFT(elm, field) = VRBT_RIGHT(elm, field) = NULL; \ + VRBT_COLOR(elm, field) = VRBT_RED; \ } while (/*CONSTCOND*/ 0) -#define VRB_SET_BLACKRED(black, red, field) do { \ - VRB_COLOR(black, field) = VRB_BLACK; \ - VRB_COLOR(red, field) = VRB_RED; \ +#define VRBT_SET_BLACKRED(black, red, field) do { \ + VRBT_COLOR(black, field) = VRBT_BLACK; \ + VRBT_COLOR(red, field) = VRBT_RED; \ } while (/*CONSTCOND*/ 0) -#ifndef VRB_AUGMENT -#define VRB_AUGMENT(x) do {} while (0) +#ifndef VRBT_AUGMENT +#define VRBT_AUGMENT(x) do {} while (0) #endif -#define VRB_ROTATE_LEFT(head, elm, tmp, field) do { \ - (tmp) = VRB_RIGHT(elm, field); \ - if ((VRB_RIGHT(elm, field) = VRB_LEFT(tmp, field)) != NULL) { \ - VRB_PARENT(VRB_LEFT(tmp, field), field) = (elm); \ +#define VRBT_ROTATE_LEFT(head, elm, tmp, field) do { \ + (tmp) = VRBT_RIGHT(elm, field); \ + if ((VRBT_RIGHT(elm, field) = VRBT_LEFT(tmp, field)) != NULL) { \ + VRBT_PARENT(VRBT_LEFT(tmp, field), field) = (elm); \ } \ - VRB_AUGMENT(elm); \ - if ((VRB_PARENT(tmp, field) = VRB_PARENT(elm, field)) != NULL) {\ - if ((elm) == VRB_LEFT(VRB_PARENT(elm, field), field)) \ - VRB_LEFT(VRB_PARENT(elm, field), field) = (tmp);\ + VRBT_AUGMENT(elm); \ + if ((VRBT_PARENT(tmp, field) = VRBT_PARENT(elm, field)) != NULL) {\ + if ((elm) == VRBT_LEFT(VRBT_PARENT(elm, field), field)) \ + VRBT_LEFT(VRBT_PARENT(elm, field), field) = (tmp);\ else \ - VRB_RIGHT(VRB_PARENT(elm, field), field) = (tmp);\ + VRBT_RIGHT(VRBT_PARENT(elm, field), field) = (tmp);\ } else \ (head)->rbh_root = (tmp); \ - VRB_LEFT(tmp, field) = (elm); \ - VRB_PARENT(elm, field) = (tmp); \ - VRB_AUGMENT(tmp); \ - if ((VRB_PARENT(tmp, field))) \ - VRB_AUGMENT(VRB_PARENT(tmp, field)); \ + VRBT_LEFT(tmp, field) = (elm); \ + VRBT_PARENT(elm, field) = (tmp); \ + VRBT_AUGMENT(tmp); \ + if ((VRBT_PARENT(tmp, field))) \ + VRBT_AUGMENT(VRBT_PARENT(tmp, field)); \ } while (/*CONSTCOND*/ 0) -#define VRB_ROTATE_RIGHT(head, elm, tmp, field) do { \ - (tmp) = VRB_LEFT(elm, field); \ - if ((VRB_LEFT(elm, field) = VRB_RIGHT(tmp, field)) != NULL) { \ - VRB_PARENT(VRB_RIGHT(tmp, field), field) = (elm); \ +#define VRBT_ROTATE_RIGHT(head, elm, tmp, field) do { \ + (tmp) = VRBT_LEFT(elm, field); \ + if ((VRBT_LEFT(elm, field) = VRBT_RIGHT(tmp, field)) != NULL) { \ + VRBT_PARENT(VRBT_RIGHT(tmp, field), field) = (elm); \ } \ - VRB_AUGMENT(elm); \ - if ((VRB_PARENT(tmp, field) = VRB_PARENT(elm, field)) != NULL) {\ - if ((elm) == VRB_LEFT(VRB_PARENT(elm, field), field)) \ - VRB_LEFT(VRB_PARENT(elm, field), field) = (tmp);\ + VRBT_AUGMENT(elm); \ + if ((VRBT_PARENT(tmp, field) = VRBT_PARENT(elm, field)) != NULL) {\ + if ((elm) == VRBT_LEFT(VRBT_PARENT(elm, field), field)) \ + VRBT_LEFT(VRBT_PARENT(elm, field), field) = (tmp);\ else \ - VRB_RIGHT(VRB_PARENT(elm, field), field) = (tmp);\ + VRBT_RIGHT(VRBT_PARENT(elm, field), field) = (tmp);\ } else \ (head)->rbh_root = (tmp); \ - VRB_RIGHT(tmp, field) = (elm); \ - VRB_PARENT(elm, field) = (tmp); \ - VRB_AUGMENT(tmp); \ - if ((VRB_PARENT(tmp, field))) \ - VRB_AUGMENT(VRB_PARENT(tmp, field)); \ + VRBT_RIGHT(tmp, field) = (elm); \ + VRBT_PARENT(elm, field) = (tmp); \ + VRBT_AUGMENT(tmp); \ + if ((VRBT_PARENT(tmp, field))) \ + VRBT_AUGMENT(VRBT_PARENT(tmp, field)); \ } while (/*CONSTCOND*/ 0) /* Generates prototypes and inline functions */ -#define VRB_PROTOTYPE(name, type, field, cmp) \ - VRB_PROTOTYPE_INTERNAL(name, type, field, cmp,) -#define VRB_PROTOTYPE_STATIC(name, type, field, cmp) \ - VRB_PROTOTYPE_INTERNAL(name, type, field, cmp, v_unused_ static) -#define VRB_PROTOTYPE_INTERNAL(name, type, field, cmp, attr) \ -/*lint -esym(528, name##_VRB_*) */ \ -attr void name##_VRB_INSERT_COLOR(struct name *, struct type *); \ -attr void name##_VRB_REMOVE_COLOR(struct name *, struct type *, struct type *);\ -attr struct type *name##_VRB_REMOVE(struct name *, struct type *); \ -attr struct type *name##_VRB_INSERT(struct name *, struct type *); \ -attr struct type *name##_VRB_FIND(const struct name *, const struct type *); \ -attr struct type *name##_VRB_NFIND(const struct name *, const struct type *); \ -attr struct type *name##_VRB_NEXT(struct type *); \ -attr struct type *name##_VRB_PREV(struct type *); \ -attr struct type *name##_VRB_MINMAX(const struct name *, int); \ +#define VRBT_PROTOTYPE(name, type, field, cmp) \ + VRBT_PROTOTYPE_INTERNAL(name, type, field, cmp,) +#define VRBT_PROTOTYPE_STATIC(name, type, field, cmp) \ + VRBT_PROTOTYPE_INTERNAL(name, type, field, cmp, v_unused_ static) +#define VRBT_PROTOTYPE_INTERNAL(name, type, field, cmp, attr) \ +/*lint -esym(528, name##_VRBT_*) */ \ +attr void name##_VRBT_INSERT_COLOR(struct name *, struct type *); \ +attr void name##_VRBT_REMOVE_COLOR(struct name *, struct type *, struct type *);\ +attr struct type *name##_VRBT_REMOVE(struct name *, struct type *); \ +attr struct type *name##_VRBT_INSERT(struct name *, struct type *); \ +attr struct type *name##_VRBT_FIND(const struct name *, const struct type *); \ +attr struct type *name##_VRBT_NFIND(const struct name *, const struct type *); \ +attr struct type *name##_VRBT_NEXT(struct type *); \ +attr struct type *name##_VRBT_PREV(struct type *); \ +attr struct type *name##_VRBT_MINMAX(const struct name *, int); \ \ /* Main rb operation. * Moves node close to the key of elm to top */ -#define VRB_GENERATE(name, type, field, cmp) \ - VRB_GENERATE_INTERNAL(name, type, field, cmp,) -#define VRB_GENERATE_STATIC(name, type, field, cmp) \ - VRB_GENERATE_INTERNAL(name, type, field, cmp, v_unused_ static) -#define VRB_GENERATE_INTERNAL(name, type, field, cmp, attr) \ +#define VRBT_GENERATE(name, type, field, cmp) \ + VRBT_GENERATE_INTERNAL(name, type, field, cmp,) +#define VRBT_GENERATE_STATIC(name, type, field, cmp) \ + VRBT_GENERATE_INTERNAL(name, type, field, cmp, v_unused_ static) +#define VRBT_GENERATE_INTERNAL(name, type, field, cmp, attr) \ attr void \ -name##_VRB_INSERT_COLOR(struct name *head, struct type *elm) \ +name##_VRBT_INSERT_COLOR(struct name *head, struct type *elm) \ { \ struct type *parent, *gparent, *tmp; \ - while ((parent = VRB_PARENT(elm, field)) != NULL && \ - VRB_COLOR(parent, field) == VRB_RED) { \ - gparent = VRB_PARENT(parent, field); \ - if (parent == VRB_LEFT(gparent, field)) { \ - tmp = VRB_RIGHT(gparent, field); \ - if (tmp && VRB_COLOR(tmp, field) == VRB_RED) { \ - VRB_COLOR(tmp, field) = VRB_BLACK; \ - VRB_SET_BLACKRED(parent, gparent, field);\ + while ((parent = VRBT_PARENT(elm, field)) != NULL && \ + VRBT_COLOR(parent, field) == VRBT_RED) { \ + gparent = VRBT_PARENT(parent, field); \ + if (parent == VRBT_LEFT(gparent, field)) { \ + tmp = VRBT_RIGHT(gparent, field); \ + if (tmp && VRBT_COLOR(tmp, field) == VRBT_RED) { \ + VRBT_COLOR(tmp, field) = VRBT_BLACK; \ + VRBT_SET_BLACKRED(parent, gparent, field);\ elm = gparent; \ continue; \ } \ - if (VRB_RIGHT(parent, field) == elm) { \ - VRB_ROTATE_LEFT(head, parent, tmp, field);\ + if (VRBT_RIGHT(parent, field) == elm) { \ + VRBT_ROTATE_LEFT(head, parent, tmp, field);\ tmp = parent; \ parent = elm; \ elm = tmp; \ } \ - VRB_SET_BLACKRED(parent, gparent, field); \ - VRB_ROTATE_RIGHT(head, gparent, tmp, field); \ + VRBT_SET_BLACKRED(parent, gparent, field); \ + VRBT_ROTATE_RIGHT(head, gparent, tmp, field); \ } else { \ - tmp = VRB_LEFT(gparent, field); \ - if (tmp && VRB_COLOR(tmp, field) == VRB_RED) { \ - VRB_COLOR(tmp, field) = VRB_BLACK; \ - VRB_SET_BLACKRED(parent, gparent, field);\ + tmp = VRBT_LEFT(gparent, field); \ + if (tmp && VRBT_COLOR(tmp, field) == VRBT_RED) { \ + VRBT_COLOR(tmp, field) = VRBT_BLACK; \ + VRBT_SET_BLACKRED(parent, gparent, field);\ elm = gparent; \ continue; \ } \ - if (VRB_LEFT(parent, field) == elm) { \ - VRB_ROTATE_RIGHT(head, parent, tmp, field);\ + if (VRBT_LEFT(parent, field) == elm) { \ + VRBT_ROTATE_RIGHT(head, parent, tmp, field);\ tmp = parent; \ parent = elm; \ elm = tmp; \ } \ - VRB_SET_BLACKRED(parent, gparent, field); \ - VRB_ROTATE_LEFT(head, gparent, tmp, field); \ + VRBT_SET_BLACKRED(parent, gparent, field); \ + VRBT_ROTATE_LEFT(head, gparent, tmp, field); \ } \ } \ - VRB_COLOR(head->rbh_root, field) = VRB_BLACK; \ + VRBT_COLOR(head->rbh_root, field) = VRBT_BLACK; \ } \ \ attr void \ -name##_VRB_REMOVE_COLOR(struct name *head, struct type *parent, struct type *elm) \ +name##_VRBT_REMOVE_COLOR(struct name *head, struct type *parent, struct type *elm) \ { \ struct type *tmp; \ - while ((elm == NULL || VRB_COLOR(elm, field) == VRB_BLACK) && \ - elm != VRB_ROOT(head)) { \ + while ((elm == NULL || VRBT_COLOR(elm, field) == VRBT_BLACK) && \ + elm != VRBT_ROOT(head)) { \ AN(parent); \ - if (VRB_LEFT(parent, field) == elm) { \ - tmp = VRB_RIGHT(parent, field); \ - if (VRB_COLOR(tmp, field) == VRB_RED) { \ - VRB_SET_BLACKRED(tmp, parent, field); \ - VRB_ROTATE_LEFT(head, parent, tmp, field);\ - tmp = VRB_RIGHT(parent, field); \ + if (VRBT_LEFT(parent, field) == elm) { \ + tmp = VRBT_RIGHT(parent, field); \ + if (VRBT_COLOR(tmp, field) == VRBT_RED) { \ + VRBT_SET_BLACKRED(tmp, parent, field); \ + VRBT_ROTATE_LEFT(head, parent, tmp, field);\ + tmp = VRBT_RIGHT(parent, field); \ } \ - if ((VRB_LEFT(tmp, field) == NULL || \ - VRB_COLOR(VRB_LEFT(tmp, field), field) == VRB_BLACK) &&\ - (VRB_RIGHT(tmp, field) == NULL || \ - VRB_COLOR(VRB_RIGHT(tmp, field), field) == VRB_BLACK)) {\ - VRB_COLOR(tmp, field) = VRB_RED; \ + if ((VRBT_LEFT(tmp, field) == NULL || \ + VRBT_COLOR(VRBT_LEFT(tmp, field), field) == VRBT_BLACK) &&\ + (VRBT_RIGHT(tmp, field) == NULL || \ + VRBT_COLOR(VRBT_RIGHT(tmp, field), field) == VRBT_BLACK)) {\ + VRBT_COLOR(tmp, field) = VRBT_RED; \ elm = parent; \ - parent = VRB_PARENT(elm, field); \ + parent = VRBT_PARENT(elm, field); \ } else { \ - if (VRB_RIGHT(tmp, field) == NULL || \ - VRB_COLOR(VRB_RIGHT(tmp, field), field) == VRB_BLACK) {\ + if (VRBT_RIGHT(tmp, field) == NULL || \ + VRBT_COLOR(VRBT_RIGHT(tmp, field), field) == VRBT_BLACK) {\ struct type *oleft; \ - if ((oleft = VRB_LEFT(tmp, field)) \ + if ((oleft = VRBT_LEFT(tmp, field)) \ != NULL) \ - VRB_COLOR(oleft, field) = VRB_BLACK;\ - VRB_COLOR(tmp, field) = VRB_RED;\ - VRB_ROTATE_RIGHT(head, tmp, oleft, field);\ - tmp = VRB_RIGHT(parent, field); \ + VRBT_COLOR(oleft, field) = VRBT_BLACK;\ + VRBT_COLOR(tmp, field) = VRBT_RED;\ + VRBT_ROTATE_RIGHT(head, tmp, oleft, field);\ + tmp = VRBT_RIGHT(parent, field); \ } \ - VRB_COLOR(tmp, field) = VRB_COLOR(parent, field);\ - VRB_COLOR(parent, field) = VRB_BLACK; \ - if (VRB_RIGHT(tmp, field)) \ - VRB_COLOR(VRB_RIGHT(tmp, field), field) = VRB_BLACK;\ - VRB_ROTATE_LEFT(head, parent, tmp, field);\ - elm = VRB_ROOT(head); \ + VRBT_COLOR(tmp, field) = VRBT_COLOR(parent, field);\ + VRBT_COLOR(parent, field) = VRBT_BLACK; \ + if (VRBT_RIGHT(tmp, field)) \ + VRBT_COLOR(VRBT_RIGHT(tmp, field), field) = VRBT_BLACK;\ + VRBT_ROTATE_LEFT(head, parent, tmp, field);\ + elm = VRBT_ROOT(head); \ break; \ } \ } else { \ - tmp = VRB_LEFT(parent, field); \ - if (VRB_COLOR(tmp, field) == VRB_RED) { \ - VRB_SET_BLACKRED(tmp, parent, field); \ - VRB_ROTATE_RIGHT(head, parent, tmp, field);\ - tmp = VRB_LEFT(parent, field); \ + tmp = VRBT_LEFT(parent, field); \ + if (VRBT_COLOR(tmp, field) == VRBT_RED) { \ + VRBT_SET_BLACKRED(tmp, parent, field); \ + VRBT_ROTATE_RIGHT(head, parent, tmp, field);\ + tmp = VRBT_LEFT(parent, field); \ } \ - if ((VRB_LEFT(tmp, field) == NULL || \ - VRB_COLOR(VRB_LEFT(tmp, field), field) == VRB_BLACK) &&\ - (VRB_RIGHT(tmp, field) == NULL || \ - VRB_COLOR(VRB_RIGHT(tmp, field), field) == VRB_BLACK)) {\ - VRB_COLOR(tmp, field) = VRB_RED; \ + if ((VRBT_LEFT(tmp, field) == NULL || \ + VRBT_COLOR(VRBT_LEFT(tmp, field), field) == VRBT_BLACK) &&\ + (VRBT_RIGHT(tmp, field) == NULL || \ + VRBT_COLOR(VRBT_RIGHT(tmp, field), field) == VRBT_BLACK)) {\ + VRBT_COLOR(tmp, field) = VRBT_RED; \ elm = parent; \ - parent = VRB_PARENT(elm, field); \ + parent = VRBT_PARENT(elm, field); \ } else { \ - if (VRB_LEFT(tmp, field) == NULL || \ - VRB_COLOR(VRB_LEFT(tmp, field), field) == VRB_BLACK) {\ + if (VRBT_LEFT(tmp, field) == NULL || \ + VRBT_COLOR(VRBT_LEFT(tmp, field), field) == VRBT_BLACK) {\ struct type *oright; \ - if ((oright = VRB_RIGHT(tmp, field)) \ + if ((oright = VRBT_RIGHT(tmp, field)) \ != NULL) \ - VRB_COLOR(oright, field) = VRB_BLACK;\ - VRB_COLOR(tmp, field) = VRB_RED;\ - VRB_ROTATE_LEFT(head, tmp, oright, field);\ - tmp = VRB_LEFT(parent, field); \ + VRBT_COLOR(oright, field) = VRBT_BLACK;\ + VRBT_COLOR(tmp, field) = VRBT_RED;\ + VRBT_ROTATE_LEFT(head, tmp, oright, field);\ + tmp = VRBT_LEFT(parent, field); \ } \ - VRB_COLOR(tmp, field) = VRB_COLOR(parent, field);\ - VRB_COLOR(parent, field) = VRB_BLACK; \ - if (VRB_LEFT(tmp, field)) \ - VRB_COLOR(VRB_LEFT(tmp, field), field) = VRB_BLACK;\ - VRB_ROTATE_RIGHT(head, parent, tmp, field);\ - elm = VRB_ROOT(head); \ + VRBT_COLOR(tmp, field) = VRBT_COLOR(parent, field);\ + VRBT_COLOR(parent, field) = VRBT_BLACK; \ + if (VRBT_LEFT(tmp, field)) \ + VRBT_COLOR(VRBT_LEFT(tmp, field), field) = VRBT_BLACK;\ + VRBT_ROTATE_RIGHT(head, parent, tmp, field);\ + elm = VRBT_ROOT(head); \ break; \ } \ } \ } \ if (elm) \ - VRB_COLOR(elm, field) = VRB_BLACK; \ + VRBT_COLOR(elm, field) = VRBT_BLACK; \ } \ \ attr struct type * \ -name##_VRB_REMOVE(struct name *head, struct type *elm) \ +name##_VRBT_REMOVE(struct name *head, struct type *elm) \ { \ struct type *child, *parent, *old = elm; \ int color; \ - if (VRB_LEFT(elm, field) == NULL) \ - child = VRB_RIGHT(elm, field); \ - else if (VRB_RIGHT(elm, field) == NULL) \ - child = VRB_LEFT(elm, field); \ + if (VRBT_LEFT(elm, field) == NULL) \ + child = VRBT_RIGHT(elm, field); \ + else if (VRBT_RIGHT(elm, field) == NULL) \ + child = VRBT_LEFT(elm, field); \ else { \ struct type *left; \ - elm = VRB_RIGHT(elm, field); \ - while ((left = VRB_LEFT(elm, field)) != NULL) \ + elm = VRBT_RIGHT(elm, field); \ + while ((left = VRBT_LEFT(elm, field)) != NULL) \ elm = left; \ - child = VRB_RIGHT(elm, field); \ - parent = VRB_PARENT(elm, field); \ - color = VRB_COLOR(elm, field); \ + child = VRBT_RIGHT(elm, field); \ + parent = VRBT_PARENT(elm, field); \ + color = VRBT_COLOR(elm, field); \ if (child) \ - VRB_PARENT(child, field) = parent; \ + VRBT_PARENT(child, field) = parent; \ if (parent) { \ - if (VRB_LEFT(parent, field) == elm) \ - VRB_LEFT(parent, field) = child; \ + if (VRBT_LEFT(parent, field) == elm) \ + VRBT_LEFT(parent, field) = child; \ else \ - VRB_RIGHT(parent, field) = child; \ - VRB_AUGMENT(parent); \ + VRBT_RIGHT(parent, field) = child; \ + VRBT_AUGMENT(parent); \ } else \ - VRB_ROOT(head) = child; \ - if (VRB_PARENT(elm, field) == old) \ + VRBT_ROOT(head) = child; \ + if (VRBT_PARENT(elm, field) == old) \ parent = elm; \ (elm)->field = (old)->field; \ - if (VRB_PARENT(old, field)) { \ - if (VRB_LEFT(VRB_PARENT(old, field), field) == old)\ - VRB_LEFT(VRB_PARENT(old, field), field) = elm;\ + if (VRBT_PARENT(old, field)) { \ + if (VRBT_LEFT(VRBT_PARENT(old, field), field) == old)\ + VRBT_LEFT(VRBT_PARENT(old, field), field) = elm;\ else \ - VRB_RIGHT(VRB_PARENT(old, field), field) = elm;\ - VRB_AUGMENT(VRB_PARENT(old, field)); \ + VRBT_RIGHT(VRBT_PARENT(old, field), field) = elm;\ + VRBT_AUGMENT(VRBT_PARENT(old, field)); \ } else \ - VRB_ROOT(head) = elm; \ - VRB_PARENT(VRB_LEFT(old, field), field) = elm; \ - if (VRB_RIGHT(old, field)) \ - VRB_PARENT(VRB_RIGHT(old, field), field) = elm; \ + VRBT_ROOT(head) = elm; \ + VRBT_PARENT(VRBT_LEFT(old, field), field) = elm; \ + if (VRBT_RIGHT(old, field)) \ + VRBT_PARENT(VRBT_RIGHT(old, field), field) = elm; \ if (parent) { \ left = parent; \ do { \ - VRB_AUGMENT(left); \ - } while ((left = VRB_PARENT(left, field)) != NULL); \ + VRBT_AUGMENT(left); \ + } while ((left = VRBT_PARENT(left, field)) != NULL); \ } \ goto color; \ } \ - parent = VRB_PARENT(elm, field); \ - color = VRB_COLOR(elm, field); \ + parent = VRBT_PARENT(elm, field); \ + color = VRBT_COLOR(elm, field); \ if (child) \ - VRB_PARENT(child, field) = parent; \ + VRBT_PARENT(child, field) = parent; \ if (parent) { \ - if (VRB_LEFT(parent, field) == elm) \ - VRB_LEFT(parent, field) = child; \ + if (VRBT_LEFT(parent, field) == elm) \ + VRBT_LEFT(parent, field) = child; \ else \ - VRB_RIGHT(parent, field) = child; \ - VRB_AUGMENT(parent); \ + VRBT_RIGHT(parent, field) = child; \ + VRBT_AUGMENT(parent); \ } else \ - VRB_ROOT(head) = child; \ + VRBT_ROOT(head) = child; \ color: \ - if (color == VRB_BLACK) { \ - name##_VRB_REMOVE_COLOR(head, parent, child); \ + if (color == VRBT_BLACK) { \ + name##_VRBT_REMOVE_COLOR(head, parent, child); \ } \ return (old); \ } \ \ /* Inserts a node into the RB tree */ \ attr struct type * \ -name##_VRB_INSERT(struct name *head, struct type *elm) \ +name##_VRBT_INSERT(struct name *head, struct type *elm) \ { \ struct type *tmp; \ struct type *parent = NULL; \ int comp = 0; \ - tmp = VRB_ROOT(head); \ + tmp = VRBT_ROOT(head); \ while (tmp) { \ parent = tmp; \ comp = (cmp)(elm, parent); \ if (comp < 0) \ - tmp = VRB_LEFT(tmp, field); \ + tmp = VRBT_LEFT(tmp, field); \ else if (comp > 0) \ - tmp = VRB_RIGHT(tmp, field); \ + tmp = VRBT_RIGHT(tmp, field); \ else \ return (tmp); \ } \ - VRB_SET(elm, parent, field); \ + VRBT_SET(elm, parent, field); \ if (parent != NULL) { \ if (comp < 0) \ - VRB_LEFT(parent, field) = elm; \ + VRBT_LEFT(parent, field) = elm; \ else \ - VRB_RIGHT(parent, field) = elm; \ - VRB_AUGMENT(parent); \ + VRBT_RIGHT(parent, field) = elm; \ + VRBT_AUGMENT(parent); \ } else \ - VRB_ROOT(head) = elm; \ - name##_VRB_INSERT_COLOR(head, elm); \ + VRBT_ROOT(head) = elm; \ + name##_VRBT_INSERT_COLOR(head, elm); \ return (NULL); \ } \ \ /* Finds the node with the same key as elm */ \ attr struct type * \ -name##_VRB_FIND(const struct name *head, const struct type *elm) \ +name##_VRBT_FIND(const struct name *head, const struct type *elm) \ { \ - struct type *tmp = VRB_ROOT(head); \ + struct type *tmp = VRBT_ROOT(head); \ int comp; \ while (tmp) { \ comp = cmp(elm, tmp); \ if (comp < 0) \ - tmp = VRB_LEFT(tmp, field); \ + tmp = VRBT_LEFT(tmp, field); \ else if (comp > 0) \ - tmp = VRB_RIGHT(tmp, field); \ + tmp = VRBT_RIGHT(tmp, field); \ else \ return (tmp); \ } \ @@ -643,19 +643,19 @@ name##_VRB_FIND(const struct name *head, const struct type *elm) \ \ /* Finds the first node greater than or equal to the search key */ \ attr struct type * \ -name##_VRB_NFIND(const struct name *head, const struct type *elm) \ +name##_VRBT_NFIND(const struct name *head, const struct type *elm) \ { \ - struct type *tmp = VRB_ROOT(head); \ + struct type *tmp = VRBT_ROOT(head); \ struct type *res = NULL; \ int comp; \ while (tmp) { \ comp = cmp(elm, tmp); \ if (comp < 0) { \ res = tmp; \ - tmp = VRB_LEFT(tmp, field); \ + tmp = VRBT_LEFT(tmp, field); \ } \ else if (comp > 0) \ - tmp = VRB_RIGHT(tmp, field); \ + tmp = VRBT_RIGHT(tmp, field); \ else \ return (tmp); \ } \ @@ -664,21 +664,21 @@ name##_VRB_NFIND(const struct name *head, const struct type *elm) \ \ /* ARGSUSED */ \ attr struct type * \ -name##_VRB_NEXT(struct type *elm) \ +name##_VRBT_NEXT(struct type *elm) \ { \ - if (VRB_RIGHT(elm, field)) { \ - elm = VRB_RIGHT(elm, field); \ - while (VRB_LEFT(elm, field)) \ - elm = VRB_LEFT(elm, field); \ + if (VRBT_RIGHT(elm, field)) { \ + elm = VRBT_RIGHT(elm, field); \ + while (VRBT_LEFT(elm, field)) \ + elm = VRBT_LEFT(elm, field); \ } else { \ - if (VRB_PARENT(elm, field) && \ - (elm == VRB_LEFT(VRB_PARENT(elm, field), field))) \ - elm = VRB_PARENT(elm, field); \ + if (VRBT_PARENT(elm, field) && \ + (elm == VRBT_LEFT(VRBT_PARENT(elm, field), field))) \ + elm = VRBT_PARENT(elm, field); \ else { \ - while (VRB_PARENT(elm, field) && \ - (elm == VRB_RIGHT(VRB_PARENT(elm, field), field)))\ - elm = VRB_PARENT(elm, field); \ - elm = VRB_PARENT(elm, field); \ + while (VRBT_PARENT(elm, field) && \ + (elm == VRBT_RIGHT(VRBT_PARENT(elm, field), field)))\ + elm = VRBT_PARENT(elm, field); \ + elm = VRBT_PARENT(elm, field); \ } \ } \ return (elm); \ @@ -686,81 +686,81 @@ name##_VRB_NEXT(struct type *elm) \ \ /* ARGSUSED */ \ attr struct type * \ -name##_VRB_PREV(struct type *elm) \ +name##_VRBT_PREV(struct type *elm) \ { \ - if (VRB_LEFT(elm, field)) { \ - elm = VRB_LEFT(elm, field); \ - while (VRB_RIGHT(elm, field)) \ - elm = VRB_RIGHT(elm, field); \ + if (VRBT_LEFT(elm, field)) { \ + elm = VRBT_LEFT(elm, field); \ + while (VRBT_RIGHT(elm, field)) \ + elm = VRBT_RIGHT(elm, field); \ } else { \ - if (VRB_PARENT(elm, field) && \ - (elm == VRB_RIGHT(VRB_PARENT(elm, field), field))) \ - elm = VRB_PARENT(elm, field); \ + if (VRBT_PARENT(elm, field) && \ + (elm == VRBT_RIGHT(VRBT_PARENT(elm, field), field))) \ + elm = VRBT_PARENT(elm, field); \ else { \ - while (VRB_PARENT(elm, field) && \ - (elm == VRB_LEFT(VRB_PARENT(elm, field), field)))\ - elm = VRB_PARENT(elm, field); \ - elm = VRB_PARENT(elm, field); \ + while (VRBT_PARENT(elm, field) && \ + (elm == VRBT_LEFT(VRBT_PARENT(elm, field), field)))\ + elm = VRBT_PARENT(elm, field); \ + elm = VRBT_PARENT(elm, field); \ } \ } \ return (elm); \ } \ \ attr struct type * \ -name##_VRB_MINMAX(const struct name *head, int val) \ +name##_VRBT_MINMAX(const struct name *head, int val) \ { \ - struct type *tmp = VRB_ROOT(head); \ + struct type *tmp = VRBT_ROOT(head); \ struct type *parent = NULL; \ while (tmp) { \ parent = tmp; \ if (val < 0) \ - tmp = VRB_LEFT(tmp, field); \ + tmp = VRBT_LEFT(tmp, field); \ else \ - tmp = VRB_RIGHT(tmp, field); \ + tmp = VRBT_RIGHT(tmp, field); \ } \ return (parent); \ } -#define VRB_NEGINF -1 -#define VRB_INF 1 +#define VRBT_NEGINF -1 +#define VRBT_INF 1 -#define VRB_INSERT(name, x, y) name##_VRB_INSERT(x, y) -#define VRB_REMOVE(name, x, y) name##_VRB_REMOVE(x, y) -#define VRB_FIND(name, x, y) name##_VRB_FIND(x, y) -#define VRB_NFIND(name, x, y) name##_VRB_NFIND(x, y) -#define VRB_NEXT(name, x, y) name##_VRB_NEXT(y) -#define VRB_PREV(name, x, y) name##_VRB_PREV(y) -#define VRB_MIN(name, x) name##_VRB_MINMAX(x, VRB_NEGINF) -#define VRB_MAX(name, x) name##_VRB_MINMAX(x, VRB_INF) +#define VRBT_INSERT(name, x, y) name##_VRBT_INSERT(x, y) +#define VRBT_REMOVE(name, x, y) name##_VRBT_REMOVE(x, y) +#define VRBT_FIND(name, x, y) name##_VRBT_FIND(x, y) +#define VRBT_NFIND(name, x, y) name##_VRBT_NFIND(x, y) +#define VRBT_NEXT(name, x, y) name##_VRBT_NEXT(y) +#define VRBT_PREV(name, x, y) name##_VRBT_PREV(y) +#define VRBT_MIN(name, x) name##_VRBT_MINMAX(x, VRBT_NEGINF) +#define VRBT_MAX(name, x) name##_VRBT_MINMAX(x, VRBT_INF) -#define VRB_FOREACH(x, name, head) \ - for ((x) = VRB_MIN(name, head); \ +#define VRBT_FOREACH(x, name, head) \ + for ((x) = VRBT_MIN(name, head); \ (x) != NULL; \ - (x) = name##_VRB_NEXT(x)) + (x) = name##_VRBT_NEXT(x)) -#define VRB_FOREACH_FROM(x, name, y) \ +#define VRBT_FOREACH_FROM(x, name, y) \ for ((x) = (y); \ - ((x) != NULL) && ((y) = name##_VRB_NEXT(x), (x) != NULL); \ + ((x) != NULL) && ((y) = name##_VRBT_NEXT(x), (x) != NULL); \ (x) = (y)) -#define VRB_FOREACH_SAFE(x, name, head, y) \ - for ((x) = VRB_MIN(name, head); \ - ((x) != NULL) && ((y) = name##_VRB_NEXT(x), (x) != NULL); \ +#define VRBT_FOREACH_SAFE(x, name, head, y) \ + for ((x) = VRBT_MIN(name, head); \ + ((x) != NULL) && ((y) = name##_VRBT_NEXT(x), (x) != NULL); \ (x) = (y)) -#define VRB_FOREACH_REVERSE(x, name, head) \ - for ((x) = VRB_MAX(name, head); \ +#define VRBT_FOREACH_REVERSE(x, name, head) \ + for ((x) = VRBT_MAX(name, head); \ (x) != NULL; \ - (x) = name##_VRB_PREV(x)) + (x) = name##_VRBT_PREV(x)) -#define VRB_FOREACH_REVERSE_FROM(x, name, y) \ +#define VRBT_FOREACH_REVERSE_FROM(x, name, y) \ for ((x) = (y); \ - ((x) != NULL) && ((y) = name##_VRB_PREV(x), (x) != NULL); \ + ((x) != NULL) && ((y) = name##_VRBT_PREV(x), (x) != NULL); \ (x) = (y)) -#define VRB_FOREACH_REVERSE_SAFE(x, name, head, y) \ - for ((x) = VRB_MAX(name, head); \ - ((x) != NULL) && ((y) = name##_VRB_PREV(x), (x) != NULL); \ +#define VRBT_FOREACH_REVERSE_SAFE(x, name, head, y) \ + for ((x) = VRBT_MAX(name, head); \ + ((x) != NULL) && ((y) = name##_VRBT_PREV(x), (x) != NULL); \ (x) = (y)) #endif /* _VTREE_H_ */ diff --git a/lib/libvarnishapi/vsl_dispatch.c b/lib/libvarnishapi/vsl_dispatch.c index 6e786d4f7..fa397ae0d 100644 --- a/lib/libvarnishapi/vsl_dispatch.c +++ b/lib/libvarnishapi/vsl_dispatch.c @@ -135,9 +135,9 @@ struct vslc_vtx { struct vtx_key { unsigned vxid; - VRB_ENTRY(vtx_key) entry; + VRBT_ENTRY(vtx_key) entry; }; -VRB_HEAD(vtx_tree, vtx_key); +VRBT_HEAD(vtx_tree, vtx_key); struct vtx { struct vtx_key key; @@ -222,8 +222,8 @@ vtx_keycmp(const struct vtx_key *a, const struct vtx_key *b) return (0); } -VRB_PROTOTYPE_STATIC(vtx_tree, vtx_key, entry, vtx_keycmp) -VRB_GENERATE_STATIC(vtx_tree, vtx_key, entry, vtx_keycmp) +VRBT_PROTOTYPE_STATIC(vtx_tree, vtx_key, entry, vtx_keycmp) +VRBT_GENERATE_STATIC(vtx_tree, vtx_key, entry, vtx_keycmp) static enum vsl_status v_matchproto_(vslc_next_f) vslc_raw_next(const struct VSL_cursor *cursor) @@ -549,7 +549,7 @@ vtx_retire(struct VSLQ *vslq, struct vtx **pvtx) AZ(vtx->n_child); AZ(vtx->n_descend); vtx->n_childready = 0; - AN(VRB_REMOVE(vtx_tree, &vslq->tree, &vtx->key)); + AN(VRBT_REMOVE(vtx_tree, &vslq->tree, &vtx->key)); vtx->key.vxid = 0; vtx->flags = 0; @@ -594,7 +594,7 @@ vtx_lookup(const struct VSLQ *vslq, unsigned vxid) AN(vslq); lkey.vxid = vxid; - key = VRB_FIND(vtx_tree, &vslq->tree, &lkey); + key = VRBT_FIND(vtx_tree, &vslq->tree, &lkey); if (key == NULL) return (NULL); CAST_OBJ_NOTNULL(vtx, (void *)key, VTX_MAGIC); @@ -611,7 +611,7 @@ vtx_add(struct VSLQ *vslq, unsigned vxid) vtx = vtx_new(vslq); AN(vtx); vtx->key.vxid = vxid; - AZ(VRB_INSERT(vtx_tree, &vslq->tree, &vtx->key)); + AZ(VRBT_INSERT(vtx_tree, &vslq->tree, &vtx->key)); VTAILQ_INSERT_TAIL(&vslq->incomplete, vtx, list_vtx); vslq->n_outstanding++; return (vtx); @@ -1080,7 +1080,7 @@ VSLQ_New(struct VSL_data *vsl, struct VSL_cursor **cp, vslq->query = query; /* Setup normal mode */ - VRB_INIT(&vslq->tree); + VRBT_INIT(&vslq->tree); VTAILQ_INIT(&vslq->ready); VTAILQ_INIT(&vslq->incomplete); VTAILQ_INIT(&vslq->shmrefs); From nils.goroll at uplex.de Mon Nov 5 17:16:12 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 5 Nov 2018 17:16:12 +0000 (UTC) Subject: [master] 25349b1ac manage dynamic privs as a red/black tree Message-ID: <20181105171612.D1373A211F@lists.varnish-cache.org> commit 25349b1ac78e8eed1e2da98ed941ad058232019b Author: Nils Goroll Date: Tue Oct 30 18:49:18 2018 +0100 manage dynamic privs as a red/black tree O(n) does not scale for a high number of dynamic privs, so trade some workspace for O(lg n) diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index ce39e683d..2b13aa9ce 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -49,6 +49,7 @@ #include "miniobj.h" #include "vas.h" #include "vqueue.h" +#include "vtree.h" #include "vapi/vsl_int.h" @@ -198,10 +199,12 @@ struct vxid_pool { /*--------------------------------------------------------------------*/ +VRBT_HEAD(vrt_priv_tree,vrt_priv); + struct vrt_privs { unsigned magic; -#define VRT_PRIVS_MAGIC 0x03ba7501 - VTAILQ_HEAD(,vrt_priv) privs; +#define VRT_PRIVS_MAGIC 0x03ba7502 + struct vrt_priv_tree privs; }; /* Worker pool stuff -------------------------------------------------*/ diff --git a/bin/varnishd/cache/cache_vrt_priv.c b/bin/varnishd/cache/cache_vrt_priv.c index 6b3b455fe..704a6f7a6 100644 --- a/bin/varnishd/cache/cache_vrt_priv.c +++ b/bin/varnishd/cache/cache_vrt_priv.c @@ -40,7 +40,7 @@ struct vrt_priv { unsigned magic; #define VRT_PRIV_MAGIC 0x24157a52 - VTAILQ_ENTRY(vrt_priv) list; + VRBT_ENTRY(vrt_priv) entry; struct vmod_priv priv[1]; const struct vcl *vcl; uintptr_t id; // = scope / vrt_privs @@ -49,6 +49,10 @@ struct vrt_priv { struct vrt_privs cli_task_privs[1]; +static inline int vrt_priv_dyncmp(const struct vrt_priv *, + const struct vrt_priv *); +VRBT_PROTOTYPE(vrt_priv_tree, vrt_priv, entry, vrt_priv_dyncmp); + /*-------------------------------------------------------------------- */ @@ -63,7 +67,7 @@ pan_privs(struct vsb *vsb, const struct vrt_privs *privs) VSB_indent(vsb, 2); PAN_CheckMagic(vsb, privs, VRT_PRIVS_MAGIC); if (privs->magic == VRT_PRIVS_MAGIC) { - VTAILQ_FOREACH(vp, &privs->privs, list) { + VRBT_FOREACH(vp, vrt_priv_tree, &privs->privs) { PAN_CheckMagic(vsb, vp, VRT_PRIV_MAGIC); VSB_printf(vsb, "priv {p %p l %d f %p} vcl %p id %jx vmod %jx\n", @@ -89,26 +93,40 @@ VRTPRIV_init(struct vrt_privs *privs) { INIT_OBJ(privs, VRT_PRIVS_MAGIC); - VTAILQ_INIT(&privs->privs); + VRBT_INIT(&privs->privs); +} + +static inline int +vrt_priv_dyncmp(const struct vrt_priv *vp1, const struct vrt_priv *vp2) +{ + if (vp1->vmod_id < vp2->vmod_id) + return (-1); + if (vp1->vmod_id > vp2->vmod_id) + return (1); + return (0); } +VRBT_GENERATE(vrt_priv_tree, vrt_priv, entry, vrt_priv_dyncmp); + static struct vmod_priv * vrt_priv_dynamic(const struct vcl *vcl, struct ws *ws, struct vrt_privs *vps, uintptr_t id, uintptr_t vmod_id) { struct vrt_priv *vp; + const struct vrt_priv needle = {.vmod_id = vmod_id}; CHECK_OBJ_NOTNULL(vps, VRT_PRIVS_MAGIC); AN(vmod_id); - VTAILQ_FOREACH(vp, &vps->privs, list) { - CHECK_OBJ_NOTNULL(vp, VRT_PRIV_MAGIC); - if (vp->vmod_id == vmod_id) { - assert(vp->vcl == vcl); - assert(vp->id == id); - return (vp->priv); - } + vp = VRBT_FIND(vrt_priv_tree, &vps->privs, &needle); + if (vp) { + CHECK_OBJ(vp, VRT_PRIV_MAGIC); + assert(vp->vmod_id == vmod_id); + assert(vp->vcl == vcl); + assert(vp->id == id); + return (vp->priv); } + vp = WS_Alloc(ws, sizeof *vp); if (vp == NULL) return (NULL); @@ -116,7 +134,7 @@ vrt_priv_dynamic(const struct vcl *vcl, struct ws *ws, vp->vcl = vcl; vp->id = id; vp->vmod_id = vmod_id; - VTAILQ_INSERT_TAIL(&vps->privs, vp, list); + VRBT_INSERT(vrt_priv_tree, &vps->privs, vp); return (vp->priv); } @@ -197,10 +215,8 @@ VCL_TaskLeave(const struct vcl *vcl, struct vrt_privs *privs) AN(vcl); CHECK_OBJ_NOTNULL(privs, VRT_PRIVS_MAGIC); - VTAILQ_FOREACH_SAFE(vp, &privs->privs, list, vp1) { - VTAILQ_REMOVE(&privs->privs, vp, list); + VRBT_FOREACH_SAFE(vp, vrt_priv_tree, &privs->privs, vp1) { VRT_priv_fini(vp->priv); } INIT_OBJ(privs, 0); } - From nils.goroll at uplex.de Mon Nov 5 17:16:12 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 5 Nov 2018 17:16:12 +0000 (UTC) Subject: [master] c1d5934b7 extrend the trivial VRT_priv_task benchmark Message-ID: <20181105171612.EC76EA2122@lists.varnish-cache.org> commit c1d5934b7d86bdc0f0105a153c300c9e9217d625 Author: Nils Goroll Date: Wed Oct 31 12:35:55 2018 +0100 extrend the trivial VRT_priv_task benchmark diff --git a/bin/varnishtest/tests/v00041.vtc b/bin/varnishtest/tests/v00041.vtc index e6cfc3c8e..09977744a 100644 --- a/bin/varnishtest/tests/v00041.vtc +++ b/bin/varnishtest/tests/v00041.vtc @@ -49,12 +49,11 @@ varnish v1 -arg "-p debug=+vclrel -p workspace_client=1m" -vcl+backend { } sub vcl_synth { - std.log("discard 100 " + debug.priv_perf(100)); + std.log("discard 1000 " + debug.priv_perf(1000)); std.log("perf 1 " + debug.priv_perf(1)); std.log("perf 10 " + debug.priv_perf(10)); std.log("perf 100 " + debug.priv_perf(100)); - ## unbearable with the list implementation - ok with rb tree - # std.log("perf 1000 " + debug.priv_perf(1000)); + std.log("perf 1000 " + debug.priv_perf(1000)); return (deliver); } From nils.goroll at uplex.de Mon Nov 5 17:32:09 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 5 Nov 2018 17:32:09 +0000 (UTC) Subject: [master] 8125596d4 slim down struct vrt_priv Message-ID: <20181105173209.43350A290E@lists.varnish-cache.org> commit 8125596d417bb2629ae146a953862b04ebf55bcd Author: Nils Goroll Date: Mon Nov 5 18:24:35 2018 +0100 slim down struct vrt_priv The vcl and id members served no purpose except for debugging. With this, we save 2 * sizeof(uintptr_t), which is exactly the additional footprint introduced with red/black trees. So the net footprint of a struct vrt_priv is now back to what it had been before #2813. For debugging, it should also be trivial to induce the context from the address of the tree root. diff --git a/bin/varnishd/cache/cache_vrt_priv.c b/bin/varnishd/cache/cache_vrt_priv.c index 704a6f7a6..b76755e0e 100644 --- a/bin/varnishd/cache/cache_vrt_priv.c +++ b/bin/varnishd/cache/cache_vrt_priv.c @@ -42,8 +42,6 @@ struct vrt_priv { #define VRT_PRIV_MAGIC 0x24157a52 VRBT_ENTRY(vrt_priv) entry; struct vmod_priv priv[1]; - const struct vcl *vcl; - uintptr_t id; // = scope / vrt_privs uintptr_t vmod_id; }; @@ -70,12 +68,10 @@ pan_privs(struct vsb *vsb, const struct vrt_privs *privs) VRBT_FOREACH(vp, vrt_priv_tree, &privs->privs) { PAN_CheckMagic(vsb, vp, VRT_PRIV_MAGIC); VSB_printf(vsb, - "priv {p %p l %d f %p} vcl %p id %jx vmod %jx\n", + "priv {p %p l %d f %p} vmod %jx\n", vp->priv->priv, vp->priv->len, vp->priv->free, - vp->vcl, - (uintmax_t)vp->id, (uintmax_t)vp->vmod_id ); } @@ -109,8 +105,7 @@ vrt_priv_dyncmp(const struct vrt_priv *vp1, const struct vrt_priv *vp2) VRBT_GENERATE(vrt_priv_tree, vrt_priv, entry, vrt_priv_dyncmp); static struct vmod_priv * -vrt_priv_dynamic(const struct vcl *vcl, struct ws *ws, - struct vrt_privs *vps, uintptr_t id, uintptr_t vmod_id) +vrt_priv_dynamic(struct ws *ws, struct vrt_privs *vps, uintptr_t vmod_id) { struct vrt_priv *vp; const struct vrt_priv needle = {.vmod_id = vmod_id}; @@ -122,8 +117,6 @@ vrt_priv_dynamic(const struct vcl *vcl, struct ws *ws, if (vp) { CHECK_OBJ(vp, VRT_PRIV_MAGIC); assert(vp->vmod_id == vmod_id); - assert(vp->vcl == vcl); - assert(vp->id == id); return (vp->priv); } @@ -131,8 +124,6 @@ vrt_priv_dynamic(const struct vcl *vcl, struct ws *ws, if (vp == NULL) return (NULL); INIT_OBJ(vp, VRT_PRIV_MAGIC); - vp->vcl = vcl; - vp->id = id; vp->vmod_id = vmod_id; VRBT_INSERT(vrt_priv_tree, &vps->privs, vp); return (vp->priv); @@ -141,7 +132,6 @@ vrt_priv_dynamic(const struct vcl *vcl, struct ws *ws, struct vmod_priv * VRT_priv_task(VRT_CTX, const void *vmod_id) { - uintptr_t id; struct vrt_privs *vps; struct vmod_priv *vp; @@ -151,36 +141,31 @@ VRT_priv_task(VRT_CTX, const void *vmod_id) if (ctx->req) { CHECK_OBJ_NOTNULL(ctx->req, REQ_MAGIC); - id = (uintptr_t)ctx->req; CAST_OBJ_NOTNULL(vps, ctx->req->privs, VRT_PRIVS_MAGIC); } else if (ctx->bo) { CHECK_OBJ_NOTNULL(ctx->bo, BUSYOBJ_MAGIC); - id = (uintptr_t)ctx->bo; CAST_OBJ_NOTNULL(vps, ctx->bo->privs, VRT_PRIVS_MAGIC); } else { ASSERT_CLI(); - id = (uintptr_t)cli_task_privs; CAST_OBJ_NOTNULL(vps, cli_task_privs, VRT_PRIVS_MAGIC); } - vp = vrt_priv_dynamic(ctx->vcl, ctx->ws, vps, id, (uintptr_t)vmod_id); + vp = vrt_priv_dynamic(ctx->ws, vps, (uintptr_t)vmod_id); return (vp); } struct vmod_priv * VRT_priv_top(VRT_CTX, const void *vmod_id) { - uintptr_t id; struct vrt_privs *vps; CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); if (ctx->req) { CHECK_OBJ_NOTNULL(ctx->req, REQ_MAGIC); CHECK_OBJ_NOTNULL(ctx->req->top, REQ_MAGIC); - id = (uintptr_t)&ctx->req->top->top; CAST_OBJ_NOTNULL(vps, ctx->req->top->privs, VRT_PRIVS_MAGIC); - return (vrt_priv_dynamic(ctx->vcl, ctx->req->top->ws, - vps, id, (uintptr_t)vmod_id)); + return (vrt_priv_dynamic(ctx->req->top->ws, vps, + (uintptr_t)vmod_id)); } else WRONG("PRIV_TOP is only accessible in client VCL context"); NEEDLESS(return NULL); From nils.goroll at uplex.de Mon Nov 5 17:47:08 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 5 Nov 2018 17:47:08 +0000 (UTC) Subject: [master] a1eba5840 avoid empty statement from VRBT macros Message-ID: <20181105174708.545A1A2F0E@lists.varnish-cache.org> commit a1eba58403e8fba298a9ce8b5befaf78ddf3a241 Author: Nils Goroll Date: Mon Nov 5 18:42:20 2018 +0100 avoid empty statement from VRBT macros seen by sun cc as: "cache/cache_vrt_priv.c", line 54: syntax error: empty declaration (E_EMPTY_DECLARATION) "cache/cache_vrt_priv.c", line 109: syntax error: empty declaration (E_EMPTY_DECLARATION) diff --git a/bin/varnishd/cache/cache_vrt_priv.c b/bin/varnishd/cache/cache_vrt_priv.c index b76755e0e..531718e1d 100644 --- a/bin/varnishd/cache/cache_vrt_priv.c +++ b/bin/varnishd/cache/cache_vrt_priv.c @@ -49,7 +49,7 @@ struct vrt_privs cli_task_privs[1]; static inline int vrt_priv_dyncmp(const struct vrt_priv *, const struct vrt_priv *); -VRBT_PROTOTYPE(vrt_priv_tree, vrt_priv, entry, vrt_priv_dyncmp); +VRBT_PROTOTYPE(vrt_priv_tree, vrt_priv, entry, vrt_priv_dyncmp) /*-------------------------------------------------------------------- */ @@ -102,7 +102,7 @@ vrt_priv_dyncmp(const struct vrt_priv *vp1, const struct vrt_priv *vp2) return (0); } -VRBT_GENERATE(vrt_priv_tree, vrt_priv, entry, vrt_priv_dyncmp); +VRBT_GENERATE(vrt_priv_tree, vrt_priv, entry, vrt_priv_dyncmp) static struct vmod_priv * vrt_priv_dynamic(struct ws *ws, struct vrt_privs *vps, uintptr_t vmod_id) From nils.goroll at uplex.de Mon Nov 5 17:47:08 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 5 Nov 2018 17:47:08 +0000 (UTC) Subject: [master] 84434327f switch to static VRBT functions Message-ID: <20181105174708.614E8A2F21@lists.varnish-cache.org> commit 84434327fe2964e0ee054311c38b6aa73b3bd7e8 Author: Nils Goroll Date: Mon Nov 5 18:43:25 2018 +0100 switch to static VRBT functions This may reduce our binary size. diff --git a/bin/varnishd/cache/cache_vrt_priv.c b/bin/varnishd/cache/cache_vrt_priv.c index 531718e1d..0719e203f 100644 --- a/bin/varnishd/cache/cache_vrt_priv.c +++ b/bin/varnishd/cache/cache_vrt_priv.c @@ -49,7 +49,7 @@ struct vrt_privs cli_task_privs[1]; static inline int vrt_priv_dyncmp(const struct vrt_priv *, const struct vrt_priv *); -VRBT_PROTOTYPE(vrt_priv_tree, vrt_priv, entry, vrt_priv_dyncmp) +VRBT_PROTOTYPE_STATIC(vrt_priv_tree, vrt_priv, entry, vrt_priv_dyncmp) /*-------------------------------------------------------------------- */ @@ -102,7 +102,7 @@ vrt_priv_dyncmp(const struct vrt_priv *vp1, const struct vrt_priv *vp2) return (0); } -VRBT_GENERATE(vrt_priv_tree, vrt_priv, entry, vrt_priv_dyncmp) +VRBT_GENERATE_STATIC(vrt_priv_tree, vrt_priv, entry, vrt_priv_dyncmp) static struct vmod_priv * vrt_priv_dynamic(struct ws *ws, struct vrt_privs *vps, uintptr_t vmod_id) From nils.goroll at uplex.de Mon Nov 5 18:01:03 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 5 Nov 2018 18:01:03 +0000 (UTC) Subject: [master] ec0edf5e6 Revert "do not pass a NULL pointer from strerror() to vsnprintf via VSL" Message-ID: <20181105180103.6168CA3743@lists.varnish-cache.org> commit ec0edf5e679ea5557e633fe2d737bd46e1683543 Author: Nils Goroll Date: Thu Nov 1 15:10:22 2018 +0100 Revert "do not pass a NULL pointer from strerror() to vsnprintf via VSL" This reverts commit ee33662a162cbc4fcc7fb8a93d143f85b7786eae. diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c index 13223c29d..8282ed814 100644 --- a/bin/varnishd/cache/cache_panic.c +++ b/bin/varnishd/cache/cache_panic.c @@ -666,12 +666,8 @@ pan_ic(const char *func, const char *file, int line, const char *cond, pan_backtrace(pan_vsb); - if (err) { - q = strerror(err); - if (q == NULL) - q = "(strerror failed)"; - VSB_printf(pan_vsb, "errno = %d (%s)\n", err, q); - } + if (err) + VSB_printf(pan_vsb, "errno = %d (%s)\n", err, strerror(err)); q = THR_GetName(); if (q != NULL) diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c index a4d9a33ad..4d546ffd3 100644 --- a/bin/varnishd/cache/cache_wrk.c +++ b/bin/varnishd/cache/cache_wrk.c @@ -437,8 +437,6 @@ pool_breed(struct pool *qp) pthread_t tp; pthread_attr_t tp_attr; struct pool_info *pi; - const char *strerr; - int err; AZ(pthread_attr_init(&tp_attr)); AZ(pthread_attr_setdetachstate(&tp_attr, PTHREAD_CREATE_DETACHED)); @@ -454,12 +452,8 @@ pool_breed(struct pool *qp) pi->qp = qp; if (pthread_create(&tp, &tp_attr, pool_thread, pi)) { - err = errno; - strerr = strerror(errno); - if (strerr == NULL) - strerr = "(strerror failed)"; VSL(SLT_Debug, 0, "Create worker thread failed %d %s", - err, strerr); + errno, strerror(errno)); Lck_Lock(&pool_mtx); VSC_C_main->threads_failed++; Lck_Unlock(&pool_mtx); From nils.goroll at uplex.de Mon Nov 5 18:01:05 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 5 Nov 2018 18:01:05 +0000 (UTC) Subject: [master] 43935f471 wrap errno.h / strerror() in a v* variant (no inline) Message-ID: <20181105180105.6C023A37C7@lists.varnish-cache.org> commit 43935f471e968a9bf38189f916a8d6c2c858c097 Author: Nils Goroll Date: Mon Nov 5 18:53:06 2018 +0100 wrap errno.h / strerror() in a v* variant (no inline) (at leat on solaris) strerror() itself may fail for an out-of-memory condition (because the localization code contains memory allocations). In order to handle this situation, we need to save the original errno because strerror() may also set errno. Fixes #2815 diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c index 69f319509..d909ebd24 100644 --- a/bin/varnishadm/varnishadm.c +++ b/bin/varnishadm/varnishadm.c @@ -49,7 +49,6 @@ # error missing readline.h - this should have got caught in configure #endif -#include #include #include #include diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c index c0d44850f..bc3c1360b 100644 --- a/bin/varnishd/cache/cache_acceptor.c +++ b/bin/varnishd/cache/cache_acceptor.c @@ -33,7 +33,6 @@ #include "config.h" -#include #include #include #include @@ -526,7 +525,7 @@ vca_accept_task(struct worker *wrk, void *arg) VSL(SLT_SessError, 0, "%s %s %s %d %d %s", wa.acceptlsock->name, laddr, lport, - ls->sock, i, strerror(i)); + ls->sock, i, vstrerror(i)); (void)Pool_TrySumstat(wrk); continue; } @@ -650,12 +649,12 @@ ccf_start(struct cli *cli, const char * const *av, void *priv) if (i) VSL(SLT_Error, 0, "Kernel TCP Fast Open: sock=%d, ret=%d %s", - ls->sock, i, strerror(errno)); + ls->sock, i, vstrerror(errno)); } if (listen(ls->sock, cache_param->listen_depth)) { VCLI_SetResult(cli, CLIS_CANT); VCLI_Out(cli, "Listen failed on socket '%s': %s", - ls->endpoint, strerror(errno)); + ls->endpoint, vstrerror(errno)); return; } vca_tcp_opt_set(ls->sock, ls->uds, 1); @@ -665,7 +664,7 @@ ccf_start(struct cli *cli, const char * const *av, void *priv) if (i) VSL(SLT_Error, 0, "Kernel filtering: sock=%d, ret=%d %s", - ls->sock, i, strerror(errno)); + ls->sock, i, vstrerror(errno)); } } diff --git a/bin/varnishd/cache/cache_backend.c b/bin/varnishd/cache/cache_backend.c index c714af99f..f1426f618 100644 --- a/bin/varnishd/cache/cache_backend.c +++ b/bin/varnishd/cache/cache_backend.c @@ -33,7 +33,6 @@ #include "config.h" #include -#include #include "cache_varnishd.h" @@ -154,7 +153,7 @@ vbe_dir_getfd(struct worker *wrk, struct backend *bp, struct busyobj *bo, VBE_Connect_Error(bp->vsc, err); VSLb(bo->vsl, SLT_FetchError, "backend %s: fail errno %d (%s)", - VRT_BACKEND_string(bp->director), err, strerror(err)); + VRT_BACKEND_string(bp->director), err, vstrerror(err)); VSC_C_main->backend_fail++; bo->htc = NULL; return (NULL); diff --git a/bin/varnishd/cache/cache_backend_probe.c b/bin/varnishd/cache/cache_backend_probe.c index 899847c81..1537c9de7 100644 --- a/bin/varnishd/cache/cache_backend_probe.c +++ b/bin/varnishd/cache/cache_backend_probe.c @@ -39,7 +39,6 @@ #include #include -#include #include #include @@ -236,11 +235,11 @@ vbp_write(struct vbp_target *vt, int *sock, const void *buf, size_t len) if (i < 0) { vt->err_xmit |= 1; bprintf(vt->resp_buf, "Write error %d (%s)", - errno, strerror(errno)); + errno, vstrerror(errno)); } else { bprintf(vt->resp_buf, "Short write (%d/%zu) error %d (%s)", - i, len, errno, strerror(errno)); + i, len, errno, vstrerror(errno)); } VTCP_close(sock); return (-1); @@ -289,7 +288,7 @@ vbp_poke(struct vbp_target *vt) s = VTP_Open(vt->tcp_pool, t_end - t_now, (const void **)&sa, &err); if (s < 0) { - bprintf(vt->resp_buf, "Open error %d (%s)", err, strerror(err)); + bprintf(vt->resp_buf, "Open error %d (%s)", err, vstrerror(err)); Lck_Lock(&vbp_mtx); if (vt->backend) VBE_Connect_Error(vt->backend->vsc, err); @@ -355,7 +354,7 @@ vbp_poke(struct vbp_target *vt) if (i == 0) { vt->err_recv |= 1; bprintf(vt->resp_buf, "Poll error %d (%s)", - errno, strerror(errno)); + errno, vstrerror(errno)); VTCP_close(&s); return; } @@ -374,7 +373,7 @@ vbp_poke(struct vbp_target *vt) if (i <= 0) { if (i < 0) bprintf(vt->resp_buf, "Read error %d (%s)", - errno, strerror(errno)); + errno, vstrerror(errno)); break; } rlen += i; diff --git a/bin/varnishd/cache/cache_esi_fetch.c b/bin/varnishd/cache/cache_esi_fetch.c index b1f993dbf..81ad5535f 100644 --- a/bin/varnishd/cache/cache_esi_fetch.c +++ b/bin/varnishd/cache/cache_esi_fetch.c @@ -30,7 +30,6 @@ #include "config.h" -#include #include #include "cache_varnishd.h" diff --git a/bin/varnishd/cache/cache_lck.c b/bin/varnishd/cache/cache_lck.c index 1e6e689ac..30cbbab4a 100644 --- a/bin/varnishd/cache/cache_lck.c +++ b/bin/varnishd/cache/cache_lck.c @@ -37,7 +37,6 @@ #include "cache_varnishd.h" -#include #include #include diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c index 8282ed814..be6dd70f6 100644 --- a/bin/varnishd/cache/cache_panic.c +++ b/bin/varnishd/cache/cache_panic.c @@ -30,7 +30,6 @@ #include "config.h" #include -#include #include #include #include @@ -667,7 +666,7 @@ pan_ic(const char *func, const char *file, int line, const char *cond, pan_backtrace(pan_vsb); if (err) - VSB_printf(pan_vsb, "errno = %d (%s)\n", err, strerror(err)); + VSB_printf(pan_vsb, "errno = %d (%s)\n", err, vstrerror(err)); q = THR_GetName(); if (q != NULL) diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index fede8e0ae..4d074747b 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -38,7 +38,6 @@ #include "cache_varnishd.h" -#include #include #include diff --git a/bin/varnishd/cache/cache_shmlog.c b/bin/varnishd/cache/cache_shmlog.c index e4db8158f..d57949a95 100644 --- a/bin/varnishd/cache/cache_shmlog.c +++ b/bin/varnishd/cache/cache_shmlog.c @@ -31,7 +31,6 @@ #include "cache_varnishd.h" -#include #include #include diff --git a/bin/varnishd/cache/cache_tcp_pool.c b/bin/varnishd/cache/cache_tcp_pool.c index 5a04eb5aa..59725b142 100644 --- a/bin/varnishd/cache/cache_tcp_pool.c +++ b/bin/varnishd/cache/cache_tcp_pool.c @@ -31,7 +31,6 @@ #include "config.h" -#include #include #include "cache_varnishd.h" diff --git a/bin/varnishd/cache/cache_vcl.c b/bin/varnishd/cache/cache_vcl.c index 17179fec0..02af16289 100644 --- a/bin/varnishd/cache/cache_vcl.c +++ b/bin/varnishd/cache/cache_vcl.c @@ -30,7 +30,6 @@ #include "config.h" -#include #include #include #include diff --git a/bin/varnishd/cache/cache_vrt.c b/bin/varnishd/cache/cache_vrt.c index 58186b2eb..0c28ba29e 100644 --- a/bin/varnishd/cache/cache_vrt.c +++ b/bin/varnishd/cache/cache_vrt.c @@ -659,7 +659,7 @@ VRT_synth_page(VRT_CTX, const char *str, ...) p = "(null)"; if (VSB_cat(vsb, p)) { VRT_fail(ctx, "synthetic(): %s", - strerror(VSB_error(vsb))); + vstrerror(VSB_error(vsb))); break; } p = va_arg(ap, const char *); diff --git a/bin/varnishd/cache/cache_vrt_filter.c b/bin/varnishd/cache/cache_vrt_filter.c index 23eb919a3..b74ceb318 100644 --- a/bin/varnishd/cache/cache_vrt_filter.c +++ b/bin/varnishd/cache/cache_vrt_filter.c @@ -30,7 +30,6 @@ #include "config.h" -#include #include #include #include diff --git a/bin/varnishd/cache/cache_vrt_vcl.c b/bin/varnishd/cache/cache_vrt_vcl.c index 62f42ff45..4147e211c 100644 --- a/bin/varnishd/cache/cache_vrt_vcl.c +++ b/bin/varnishd/cache/cache_vrt_vcl.c @@ -30,7 +30,6 @@ #include "config.h" -#include #include #include diff --git a/bin/varnishd/cache/cache_wrk.c b/bin/varnishd/cache/cache_wrk.c index 4d546ffd3..48672a9ea 100644 --- a/bin/varnishd/cache/cache_wrk.c +++ b/bin/varnishd/cache/cache_wrk.c @@ -52,7 +52,6 @@ #include "config.h" -#include #include #include "cache_varnishd.h" @@ -453,7 +452,7 @@ pool_breed(struct pool *qp) if (pthread_create(&tp, &tp_attr, pool_thread, pi)) { VSL(SLT_Debug, 0, "Create worker thread failed %d %s", - errno, strerror(errno)); + errno, vstrerror(errno)); Lck_Lock(&pool_mtx); VSC_C_main->threads_failed++; Lck_Unlock(&pool_mtx); diff --git a/bin/varnishd/common/common_vsc.c b/bin/varnishd/common/common_vsc.c index 7beb9036d..ea1d0049c 100644 --- a/bin/varnishd/common/common_vsc.c +++ b/bin/varnishd/common/common_vsc.c @@ -29,7 +29,6 @@ #include "config.h" -#include #include #include #include diff --git a/bin/varnishd/common/common_vsmw.c b/bin/varnishd/common/common_vsmw.c index 339394b58..4e54a4aea 100644 --- a/bin/varnishd/common/common_vsmw.c +++ b/bin/varnishd/common/common_vsmw.c @@ -34,7 +34,6 @@ #include "config.h" -#include #include #include #include diff --git a/bin/varnishd/http1/cache_http1_fetch.c b/bin/varnishd/http1/cache_http1_fetch.c index b8e3e876f..d8318b444 100644 --- a/bin/varnishd/http1/cache_http1_fetch.c +++ b/bin/varnishd/http1/cache_http1_fetch.c @@ -32,7 +32,6 @@ #include "cache/cache_varnishd.h" #include "cache/cache_filter.h" -#include #include #include @@ -115,7 +114,7 @@ V1F_SendReq(struct worker *wrk, struct busyobj *bo, uint64_t *ctr_hdrbytes, assert(i < 0); VSLb(bo->vsl, SLT_FetchError, "req.body read error: %d (%s)", - errno, strerror(errno)); + errno, vstrerror(errno)); bo->req->doclose = SC_RX_BODY; } if (do_chunked) @@ -134,7 +133,7 @@ V1F_SendReq(struct worker *wrk, struct busyobj *bo, uint64_t *ctr_hdrbytes, if (j != 0 || i < 0) { VSLb(bo->vsl, SLT_FetchError, "backend write error: %d (%s)", - errno, strerror(errno)); + errno, vstrerror(errno)); VSLb_ts_busyobj(bo, "Bereq", W_TIM_real(wrk)); htc->doclose = SC_TX_ERROR; return (-1); diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c index b339339f5..a6693f662 100644 --- a/bin/varnishd/http1/cache_http1_fsm.c +++ b/bin/varnishd/http1/cache_http1_fsm.c @@ -35,7 +35,6 @@ #include "cache/cache_varnishd.h" -#include #include #include diff --git a/bin/varnishd/http1/cache_http1_line.c b/bin/varnishd/http1/cache_http1_line.c index a7fc994fe..c8352af54 100644 --- a/bin/varnishd/http1/cache_http1_line.c +++ b/bin/varnishd/http1/cache_http1_line.c @@ -42,7 +42,6 @@ #include #include "cache/cache_varnishd.h" -#include #include #include "cache_http1.h" @@ -232,7 +231,7 @@ V1L_Flush(const struct worker *wrk) v1l->werr++; VSLb(v1l->vsl, SLT_Debug, "Write error, retval = %zd, len = %zd, errno = %s", - i, v1l->liov, strerror(errno)); + i, v1l->liov, vstrerror(errno)); } } v1l->liov = 0; diff --git a/bin/varnishd/http1/cache_http1_vfp.c b/bin/varnishd/http1/cache_http1_vfp.c index 36228cc44..736febf85 100644 --- a/bin/varnishd/http1/cache_http1_vfp.c +++ b/bin/varnishd/http1/cache_http1_vfp.c @@ -35,7 +35,6 @@ #include "config.h" -#include #include #include "cache/cache_varnishd.h" @@ -77,7 +76,7 @@ v1f_read(const struct vfp_ctx *vc, struct http_conn *htc, void *d, ssize_t len) if (i < 0) { // XXX: VTCP_Assert(i); // but also: EAGAIN VSLb(vc->wrk->vsl, SLT_FetchError, - "%s", strerror(errno)); + "%s", vstrerror(errno)); return (i); } } diff --git a/bin/varnishd/http2/cache_http2_session.c b/bin/varnishd/http2/cache_http2_session.c index ce114d390..932a5b93f 100644 --- a/bin/varnishd/http2/cache_http2_session.c +++ b/bin/varnishd/http2/cache_http2_session.c @@ -31,7 +31,6 @@ #include "cache/cache_varnishd.h" -#include #include #include "cache/cache_transport.h" @@ -263,7 +262,7 @@ h2_ou_session(struct worker *wrk, struct h2_sess *h2, sz = write(h2->sess->fd, h2_resp_101, strlen(h2_resp_101)); if (sz != strlen(h2_resp_101)) { VSLb(h2->vsl, SLT_Debug, "H2: Upgrade: Error writing 101" - " response: %s\n", strerror(errno)); + " response: %s\n", vstrerror(errno)); return (h2_ou_rel(wrk, req)); } diff --git a/bin/varnishd/mgt/mgt_acceptor.c b/bin/varnishd/mgt/mgt_acceptor.c index fbf7f2619..cd2794612 100644 --- a/bin/varnishd/mgt/mgt_acceptor.c +++ b/bin/varnishd/mgt/mgt_acceptor.c @@ -35,7 +35,6 @@ #include #include #include -#include #include #include #include @@ -133,7 +132,7 @@ MAC_reopen_sockets(void) fail = err; MGT_Complain(C_ERR, "Could not reopen listen socket %s: %s", - ls->endpoint, strerror(err)); + ls->endpoint, vstrerror(err)); } return fail; } @@ -167,7 +166,7 @@ mk_listen_sock(const struct listen_arg *la, const struct suckaddr *sa) FREE_OBJ(ls); if (fail != EAFNOSUPPORT) ARGV_ERR("Could not get socket %s: %s\n", - la->endpoint, strerror(fail)); + la->endpoint, vstrerror(fail)); return(NULL); } return(ls); @@ -330,7 +329,7 @@ MAC_Arg(const char *spec) val); if (errno) ARGV_ERR("Cannot parse mode sub-arg %s in -a: " - "%s\n", val, strerror(errno)); + "%s\n", val, vstrerror(errno)); if (m <= 0 || m > 0777) ARGV_ERR("Mode sub-arg %s out of range in -a\n", val); diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c index 16050b33a..8617bfdaf 100644 --- a/bin/varnishd/mgt/mgt_child.c +++ b/bin/varnishd/mgt/mgt_child.c @@ -34,7 +34,6 @@ #include #include -#include #include #include #include @@ -607,7 +606,7 @@ MCH_Cli_Fail(void) " killed it.", (intmax_t)child_pid); else MGT_Complain(C_ERR, "Failed to kill child with PID %jd: %s", - (intmax_t)child_pid, strerror(errno)); + (intmax_t)child_pid, vstrerror(errno)); } /*===================================================================== diff --git a/bin/varnishd/mgt/mgt_cli.c b/bin/varnishd/mgt/mgt_cli.c index 3677eb620..ed38683de 100644 --- a/bin/varnishd/mgt/mgt_cli.c +++ b/bin/varnishd/mgt/mgt_cli.c @@ -34,7 +34,6 @@ #include #include -#include #include #include #include @@ -284,7 +283,7 @@ mcf_auth(struct cli *cli, const char *const *av, void *priv) fd = open(secret_file, O_RDONLY); if (fd < 0) { VCLI_Out(cli, "Cannot open secret file (%s)\n", - strerror(errno)); + vstrerror(errno)); VCLI_SetResult(cli, CLIS_CANT); VJ_master(JAIL_MASTER_LOW); return; @@ -593,7 +592,7 @@ Marg_connect(const struct vev *e, int what) M_fd = VTCP_connected(M_fd); if (M_fd < 0) { MGT_Complain(C_INFO, "Could not connect to CLI-master: %s", - strerror(errno)); + vstrerror(errno)); ma = VTAILQ_FIRST(&m_addr_list); AN(ma); VTAILQ_REMOVE(&m_addr_list, ma, list); diff --git a/bin/varnishd/mgt/mgt_jail.c b/bin/varnishd/mgt/mgt_jail.c index d8d7dc993..a46a366f8 100644 --- a/bin/varnishd/mgt/mgt_jail.c +++ b/bin/varnishd/mgt/mgt_jail.c @@ -31,7 +31,6 @@ #include "config.h" -#include #include #include #include @@ -150,18 +149,18 @@ VJ_make_workdir(const char *dname) VJ_master(JAIL_MASTER_FILE); if (mkdir(dname, 0755) < 0 && errno != EEXIST) ARGV_ERR("Cannot create working directory '%s': %s\n", - dname, strerror(errno)); + dname, vstrerror(errno)); } if (chdir(dname) < 0) ARGV_ERR("Cannot change to working directory '%s': %s\n", - dname, strerror(errno)); + dname, vstrerror(errno)); i = open("_.testfile", O_RDWR|O_CREAT|O_EXCL, 0600); if (i < 0) ARGV_ERR("Cannot create test-file in %s (%s)\n" "Check permissions (or delete old directory)\n", - dname, strerror(errno)); + dname, vstrerror(errno)); closefd(&i); AZ(unlink("_.testfile")); VJ_master(JAIL_MASTER_LOW); @@ -184,11 +183,11 @@ VJ_make_subdir(const char *dname, const char *what, struct vsb *vsb) if (vsb != NULL) { VSB_printf(vsb, "Cannot create %s directory '%s': %s\n", - what, dname, strerror(e)); + what, dname, vstrerror(e)); } else { MGT_Complain(C_ERR, "Cannot create %s directory '%s': %s", - what, dname, strerror(e)); + what, dname, vstrerror(e)); } return (1); } diff --git a/bin/varnishd/mgt/mgt_jail_solaris.c b/bin/varnishd/mgt/mgt_jail_solaris.c index dc1921297..f594eaede 100644 --- a/bin/varnishd/mgt/mgt_jail_solaris.c +++ b/bin/varnishd/mgt/mgt_jail_solaris.c @@ -207,7 +207,6 @@ #ifdef HAVE_SETPPRIV -#include #include #include #include @@ -395,7 +394,7 @@ vjs_setup(enum jail_gen_e jge) MGT_Complain(C_SECURITY, "Solaris Jail warning: " " vjs_setup - priv_allocset failed: errno=%d (%s)", - errno, strerror(errno)); + errno, vstrerror(errno)); return; } @@ -459,7 +458,7 @@ vjs_waive(enum jail_gen_e jge) MGT_Complain(C_SECURITY, "Solaris Jail warning: " " vjs_waive - priv_allocset failed: errno=%d (%s)", - errno, strerror(errno)); + errno, vstrerror(errno)); return; } diff --git a/bin/varnishd/mgt/mgt_jail_unix.c b/bin/varnishd/mgt/mgt_jail_unix.c index f91cf24b9..cc3aeb199 100644 --- a/bin/varnishd/mgt/mgt_jail_unix.c +++ b/bin/varnishd/mgt/mgt_jail_unix.c @@ -30,7 +30,6 @@ #include "config.h" -#include #include #include #include @@ -253,11 +252,11 @@ vju_make_subdir(const char *dname, const char *what, struct vsb *vsb) if (vsb != NULL) { VSB_printf(vsb, "Cannot create %s directory '%s': %s\n", - what, dname, strerror(e)); + what, dname, vstrerror(e)); } else { MGT_Complain(C_ERR, "Cannot create %s directory '%s': %s", - what, dname, strerror(e)); + what, dname, vstrerror(e)); } return (1); } @@ -277,7 +276,7 @@ vju_make_workdir(const char *dname, const char *what, struct vsb *vsb) if (mkdir(dname, 0755) < 0 && errno != EEXIST) { MGT_Complain(C_ERR, "Cannot create working directory '%s': %s", - dname, strerror(errno)); + dname, vstrerror(errno)); return (1); } AZ(chown(dname, -1, vju_gid)); diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c index efca546cd..d3fe23fa1 100644 --- a/bin/varnishd/mgt/mgt_main.c +++ b/bin/varnishd/mgt/mgt_main.c @@ -32,7 +32,6 @@ #include "config.h" #include -#include #include #include #include @@ -226,7 +225,7 @@ make_secret(const char *dirname) fdo = open(fn, O_RDWR|O_CREAT|O_TRUNC, 0640); if (fdo < 0) ARGV_ERR("Cannot create secret-file in %s (%s)\n", - dirname, strerror(errno)); + dirname, vstrerror(errno)); for (i = 0; i < 256; i++) { AZ(VRND_RandomCrypto(&b, 1)); @@ -319,7 +318,7 @@ mgt_eric(void) switch (fork()) { case -1: - fprintf(stderr, "Fork() failed: %s\n", strerror(errno)); + fprintf(stderr, "Fork() failed: %s\n", vstrerror(errno)); exit(-1); case 0: closefd(&eric_pipes[0]); @@ -419,7 +418,7 @@ mgt_f_read(const char *fn) VFIL_setpath(&vcl_path, mgt_vcl_path); if (VFIL_searchpath(vcl_path, NULL, &f, fn, &fnp) || f == NULL) { ARGV_ERR("Cannot read -f file '%s' (%s)\n", - fnp != NULL ? fnp : fn, strerror(errno)); + fnp != NULL ? fnp : fn, vstrerror(errno)); } free(fa->farg); fa->farg = fnp; @@ -626,7 +625,7 @@ main(int argc, char * const *argv) I_fd = open(optarg, O_RDONLY); if (I_fd < 0) ARGV_ERR("\tCant open %s: %s\n", - optarg, strerror(errno)); + optarg, vstrerror(errno)); VJ_master(JAIL_MASTER_LOW); break; case 'l': @@ -732,13 +731,13 @@ main(int argc, char * const *argv) o = open(S_arg, O_RDONLY, 0); if (o < 0) ARGV_ERR("Cannot open -S file (%s): %s\n", - S_arg, strerror(errno)); + S_arg, vstrerror(errno)); closefd(&o); VJ_master(JAIL_MASTER_LOW); } if (VIN_n_Arg(n_arg, &dirname) != 0) - ARGV_ERR("Invalid instance (-n) name: %s\n", strerror(errno)); + ARGV_ERR("Invalid instance (-n) name: %s\n", vstrerror(errno)); if (i_arg == NULL || *i_arg == '\0') i_arg = mgt_HostName(); @@ -750,12 +749,12 @@ main(int argc, char * const *argv) if (VJ_make_workdir(dirname)) ARGV_ERR("Cannot create working directory (%s): %s\n", - dirname, strerror(errno)); + dirname, vstrerror(errno)); if (VJ_make_subdir("vmod_cache", "VMOD cache", NULL)) { ARGV_ERR( "Cannot create vmod directory (%s/vmod_cache): %s\n", - dirname, strerror(errno)); + dirname, vstrerror(errno)); } vsb = VSB_new_auto(); @@ -769,7 +768,7 @@ main(int argc, char * const *argv) (intmax_t)pid); if (pfh1 == NULL) ARGV_ERR("Could not open pid-file (%s): %s\n", - VSB_data(vsb), strerror(errno)); + VSB_data(vsb), vstrerror(errno)); VSB_destroy(&vsb); if (P_arg) { pfh2 = VPF_Open(P_arg, 0644, &pid); @@ -778,7 +777,7 @@ main(int argc, char * const *argv) (intmax_t)pid); if (pfh2 == NULL) ARGV_ERR("Could not open pid-file (%s): %s\n", - P_arg, strerror(errno)); + P_arg, vstrerror(errno)); } VJ_master(JAIL_MASTER_LOW); diff --git a/bin/varnishd/mgt/mgt_shmem.c b/bin/varnishd/mgt/mgt_shmem.c index ddfa6504a..90e966f0e 100644 --- a/bin/varnishd/mgt/mgt_shmem.c +++ b/bin/varnishd/mgt/mgt_shmem.c @@ -32,7 +32,6 @@ #include -#include #include #include #include diff --git a/bin/varnishd/mgt/mgt_vcc.c b/bin/varnishd/mgt/mgt_vcc.c index 2b476307b..dabf750a3 100644 --- a/bin/varnishd/mgt/mgt_vcc.c +++ b/bin/varnishd/mgt/mgt_vcc.c @@ -31,7 +31,6 @@ #include "config.h" -#include #include #include #include @@ -212,13 +211,13 @@ mgt_vcc_touchfile(const char *fn, struct vsb *sb) i = open(fn, O_WRONLY|O_CREAT|O_TRUNC, 0640); if (i < 0) { - VSB_printf(sb, "Failed to create %s: %s", fn, strerror(errno)); + VSB_printf(sb, "Failed to create %s: %s", fn, vstrerror(errno)); return (2); } if (fchown(i, mgt_param.uid, mgt_param.gid) != 0) if (geteuid() == 0) VSB_printf(sb, "Failed to change owner on %s: %s\n", - fn, strerror(errno)); + fn, vstrerror(errno)); closefd(&i); return (0); } diff --git a/bin/varnishd/mgt/mgt_vcl.c b/bin/varnishd/mgt/mgt_vcl.c index 0edc68168..f1ce50f5d 100644 --- a/bin/varnishd/mgt/mgt_vcl.c +++ b/bin/varnishd/mgt/mgt_vcl.c @@ -31,7 +31,6 @@ #include "config.h" -#include #include #include #include @@ -319,13 +318,13 @@ mgt_vcl_cache_vmod(const char *nm, const char *fm, const char *to) return (0); if (fo < 0) { fprintf(stderr, "Creating copy of vmod %s: %s\n", - nm, strerror(errno)); + nm, vstrerror(errno)); return (1); } fi = open(fm, O_RDONLY); if (fi < 0) { fprintf(stderr, "Opening vmod %s from %s: %s\n", - nm, fm, strerror(errno)); + nm, fm, vstrerror(errno)); AZ(unlink(to)); closefd(&fo); return (1); @@ -336,7 +335,7 @@ mgt_vcl_cache_vmod(const char *nm, const char *fm, const char *to) break; if (sz < 0 || sz != write(fo, buf, sz)) { fprintf(stderr, "Copying vmod %s: %s\n", - nm, strerror(errno)); + nm, vstrerror(errno)); AZ(unlink(to)); ret = 1; break; diff --git a/bin/varnishd/storage/mgt_storage_persistent.c b/bin/varnishd/storage/mgt_storage_persistent.c index b15be3f5c..e2fbf1c0f 100644 --- a/bin/varnishd/storage/mgt_storage_persistent.c +++ b/bin/varnishd/storage/mgt_storage_persistent.c @@ -40,7 +40,6 @@ #include -#include #include #include @@ -205,7 +204,7 @@ smp_mgt_init(struct stevedore *parent, int ac, char * const *av) if (sc->base == MAP_FAILED) ARGV_ERR("(-spersistent) failed to mmap (%s)\n", - strerror(errno)); + vstrerror(errno)); if (target != NULL && sc->base != target) fprintf(stderr, "WARNING: Persistent silo lost to ASLR %s\n", sc->filename); diff --git a/bin/varnishd/storage/stevedore_utils.c b/bin/varnishd/storage/stevedore_utils.c index 80e8fd8eb..926c6b5a9 100644 --- a/bin/varnishd/storage/stevedore_utils.c +++ b/bin/varnishd/storage/stevedore_utils.c @@ -33,7 +33,6 @@ #include -#include #include #include #include @@ -102,7 +101,7 @@ STV_GetFile(const char *fn, int *fdp, const char **fnp, const char *ctx) fd = mkstemp(buf); if (fd < 0) ARGV_ERR("(%s) \"%s\" mkstemp(%s) failed (%s)\n", - ctx, fn, buf, strerror(errno)); + ctx, fn, buf, vstrerror(errno)); AZ(unlink(buf)); *fnp = strdup(buf); AN(*fnp); @@ -111,7 +110,7 @@ STV_GetFile(const char *fn, int *fdp, const char **fnp, const char *ctx) fd = open(fn, O_RDWR | O_LARGEFILE); if (fd < 0) ARGV_ERR("(%s) \"%s\" could not open (%s)\n", - ctx, fn, strerror(errno)); + ctx, fn, vstrerror(errno)); *fnp = fn; retval = 0; } else diff --git a/bin/varnishd/storage/storage_file.c b/bin/varnishd/storage/storage_file.c index c2cb479ff..78588ccab 100644 --- a/bin/varnishd/storage/storage_file.c +++ b/bin/varnishd/storage/storage_file.c @@ -36,7 +36,6 @@ #include -#include #include #include @@ -162,7 +161,7 @@ smf_init(struct stevedore *parent, int ac, char * const *av) MCH_Fd_Inherit(sc->fd, "storage_file"); sc->filesize = STV_FileSize(sc->fd, size, &sc->pagesize, "-sfile"); if (VFIL_allocate(sc->fd, (off_t)sc->filesize, 0)) - ARGV_ERR("(-sfile) allocation error: %s\n", strerror(errno)); + ARGV_ERR("(-sfile) allocation error: %s\n", vstrerror(errno)); } /*-------------------------------------------------------------------- diff --git a/bin/varnishd/waiter/cache_waiter_epoll.c b/bin/varnishd/waiter/cache_waiter_epoll.c index d0a4c690d..f7dcc7c3f 100644 --- a/bin/varnishd/waiter/cache_waiter_epoll.c +++ b/bin/varnishd/waiter/cache_waiter_epoll.c @@ -39,7 +39,6 @@ #include -#include #include "cache/cache_varnishd.h" diff --git a/bin/varnishd/waiter/cache_waiter_ports.c b/bin/varnishd/waiter/cache_waiter_ports.c index 4846488c5..5d34a5539 100644 --- a/bin/varnishd/waiter/cache_waiter_ports.c +++ b/bin/varnishd/waiter/cache_waiter_ports.c @@ -68,7 +68,6 @@ #include -#include #include #include #include diff --git a/bin/varnishhist/varnishhist.c b/bin/varnishhist/varnishhist.c index 21ca64666..626a80714 100644 --- a/bin/varnishhist/varnishhist.c +++ b/bin/varnishhist/varnishhist.c @@ -33,7 +33,6 @@ #include "config.h" -#include #include #include #include diff --git a/bin/varnishlog/varnishlog.c b/bin/varnishlog/varnishlog.c index e3e2e4576..8abab3037 100644 --- a/bin/varnishlog/varnishlog.c +++ b/bin/varnishlog/varnishlog.c @@ -37,7 +37,6 @@ #include #include #include -#include #include #define VOPT_DEFINITION diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c index 6c859358e..f3ed20a78 100644 --- a/bin/varnishncsa/varnishncsa.c +++ b/bin/varnishncsa/varnishncsa.c @@ -43,7 +43,6 @@ #include #include #include -#include #include #include #include diff --git a/bin/varnishtest/vtc_barrier.c b/bin/varnishtest/vtc_barrier.c index 7f2f0c1ce..7a7ac31f5 100644 --- a/bin/varnishtest/vtc_barrier.c +++ b/bin/varnishtest/vtc_barrier.c @@ -28,7 +28,6 @@ #include "config.h" -#include #include #include #include diff --git a/bin/varnishtest/vtc_client.c b/bin/varnishtest/vtc_client.c index 5c714de0b..6966e462b 100644 --- a/bin/varnishtest/vtc_client.c +++ b/bin/varnishtest/vtc_client.c @@ -31,7 +31,6 @@ #include #include -#include #include #include #include diff --git a/bin/varnishtest/vtc_haproxy.c b/bin/varnishtest/vtc_haproxy.c index 157d0e67c..9d12ed671 100644 --- a/bin/varnishtest/vtc_haproxy.c +++ b/bin/varnishtest/vtc_haproxy.c @@ -28,7 +28,6 @@ #include "config.h" -#include #include #include #include diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c index 03aa7f7e5..e01ebc181 100644 --- a/bin/varnishtest/vtc_http.c +++ b/bin/varnishtest/vtc_http.c @@ -30,7 +30,6 @@ #include -#include #include #include #include diff --git a/bin/varnishtest/vtc_http2.c b/bin/varnishtest/vtc_http2.c index ed72ea37a..e942bab17 100644 --- a/bin/varnishtest/vtc_http2.c +++ b/bin/varnishtest/vtc_http2.c @@ -31,7 +31,6 @@ #include #include -#include #include #include #include diff --git a/bin/varnishtest/vtc_log.c b/bin/varnishtest/vtc_log.c index 1e6e321e8..60f0576b8 100644 --- a/bin/varnishtest/vtc_log.c +++ b/bin/varnishtest/vtc_log.c @@ -28,7 +28,6 @@ #include "config.h" -#include #include #include #include diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c index 01ae08ba7..4b7fe2a4a 100644 --- a/bin/varnishtest/vtc_main.c +++ b/bin/varnishtest/vtc_main.c @@ -33,7 +33,6 @@ #include #include -#include #include #include #include diff --git a/bin/varnishtest/vtc_misc.c b/bin/varnishtest/vtc_misc.c index 9fac1360a..e496de120 100644 --- a/bin/varnishtest/vtc_misc.c +++ b/bin/varnishtest/vtc_misc.c @@ -30,7 +30,6 @@ #include -#include #include #include #include diff --git a/bin/varnishtest/vtc_process.c b/bin/varnishtest/vtc_process.c index c6a5db295..2b1534be1 100644 --- a/bin/varnishtest/vtc_process.c +++ b/bin/varnishtest/vtc_process.c @@ -34,7 +34,6 @@ #include // Linux: struct winsize #include -#include #include #include #include diff --git a/bin/varnishtest/vtc_server.c b/bin/varnishtest/vtc_server.c index d156185d9..1a7c06f87 100644 --- a/bin/varnishtest/vtc_server.c +++ b/bin/varnishtest/vtc_server.c @@ -32,7 +32,6 @@ #include #include -#include #include #include #include diff --git a/bin/varnishtest/vtc_subr.c b/bin/varnishtest/vtc_subr.c index b3584a964..0e5922978 100644 --- a/bin/varnishtest/vtc_subr.c +++ b/bin/varnishtest/vtc_subr.c @@ -29,7 +29,6 @@ #include "config.h" #include -#include #include #include #include diff --git a/bin/varnishtest/vtc_syslog.c b/bin/varnishtest/vtc_syslog.c index f91014c9a..6318b6e39 100644 --- a/bin/varnishtest/vtc_syslog.c +++ b/bin/varnishtest/vtc_syslog.c @@ -31,7 +31,6 @@ #include #include -#include #include #include #include diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c index 0c7b3f5a3..4aa4680a5 100644 --- a/bin/varnishtest/vtc_varnish.c +++ b/bin/varnishtest/vtc_varnish.c @@ -31,7 +31,6 @@ #include #include -#include #include #include #include diff --git a/bin/varnishtop/varnishtop.c b/bin/varnishtop/varnishtop.c index 2d72eadfe..f91e3cd01 100644 --- a/bin/varnishtop/varnishtop.c +++ b/bin/varnishtop/varnishtop.c @@ -34,7 +34,6 @@ #include "config.h" #include -#include #include #include #include diff --git a/include/Makefile.am b/include/Makefile.am index 98050fd78..802231dce 100644 --- a/include/Makefile.am +++ b/include/Makefile.am @@ -60,6 +60,7 @@ nobase_pkginclude_HEADERS += \ vbm.h \ vcl.h \ vcs.h \ + verrno.h \ vmod_abi.h \ vqueue.h \ vre.h \ diff --git a/include/vas.h b/include/vas.h index 11c00692d..841bd309a 100644 --- a/include/vas.h +++ b/include/vas.h @@ -38,6 +38,8 @@ #ifndef VAS_H_INCLUDED #define VAS_H_INCLUDED +#include "verrno.h" + enum vas_e { VAS_WRONG, VAS_MISSING, diff --git a/include/verrno.h b/include/verrno.h new file mode 100644 index 000000000..f2ab427ed --- /dev/null +++ b/include/verrno.h @@ -0,0 +1,12 @@ +/*- + * Written by Nils Goroll based upon a draft by Poul-Henning Kamp + * + * This file is in the public domain. + * + * trivial strerror() wrapper never returning NULL + */ + +#include +#include + +const char * vstrerror(int e); diff --git a/lib/libvarnish/Makefile.am b/lib/libvarnish/Makefile.am index 2f4d3eb95..271b13c43 100644 --- a/lib/libvarnish/Makefile.am +++ b/lib/libvarnish/Makefile.am @@ -20,6 +20,7 @@ libvarnish_a_SOURCES = \ vcli_proto.c \ vcli_serve.c \ vct.c \ + verrno.c \ version.c \ vev.c \ vfil.c \ diff --git a/lib/libvarnish/vas.c b/lib/libvarnish/vas.c index 8380df999..3a9193268 100644 --- a/lib/libvarnish/vas.c +++ b/lib/libvarnish/vas.c @@ -31,7 +31,6 @@ #include "config.h" -#include #include #include #include diff --git a/lib/libvarnish/vcli_proto.c b/lib/libvarnish/vcli_proto.c index 789c08d93..d914f0879 100644 --- a/lib/libvarnish/vcli_proto.c +++ b/lib/libvarnish/vcli_proto.c @@ -31,7 +31,6 @@ #include #include -#include #include #include #include diff --git a/lib/libvarnish/vcli_serve.c b/lib/libvarnish/vcli_serve.c index fc8578c50..e5cf0c411 100644 --- a/lib/libvarnish/vcli_serve.c +++ b/lib/libvarnish/vcli_serve.c @@ -33,7 +33,6 @@ #include #include -#include #include #include #include diff --git a/lib/libvarnish/verrno.c b/lib/libvarnish/verrno.c new file mode 100644 index 000000000..2e377f7bd --- /dev/null +++ b/lib/libvarnish/verrno.c @@ -0,0 +1,25 @@ +/*- + * Written by Nils Goroll based upon a draft by Poul-Henning Kamp + * + * This file is in the public domain. + * + * trivial strerror() wrapper never returning NULL + */ + +#include "config.h" + +#include "verrno.h" + +const char * +vstrerror(int e) +{ + const char *p; + int oerrno = errno; + + p = strerror(e); + if (p != NULL) + return (p); + + errno = oerrno; + return ("strerror(3) returned NULL"); +} diff --git a/lib/libvarnish/vev.c b/lib/libvarnish/vev.c index 0254a9cf7..6e4e48c7d 100644 --- a/lib/libvarnish/vev.c +++ b/lib/libvarnish/vev.c @@ -30,7 +30,6 @@ #include "config.h" #include -#include #include #include #include diff --git a/lib/libvarnish/vfil.c b/lib/libvarnish/vfil.c index 345096a7c..4dfc30faf 100644 --- a/lib/libvarnish/vfil.c +++ b/lib/libvarnish/vfil.c @@ -32,7 +32,6 @@ #include #include #include -#include #include #include #include diff --git a/lib/libvarnish/vfl.c b/lib/libvarnish/vfl.c index 9af496a91..fa7804569 100644 --- a/lib/libvarnish/vfl.c +++ b/lib/libvarnish/vfl.c @@ -32,13 +32,13 @@ #include #include -#include #include #include #include #include #include "vfl.h" +#include "verrno.h" /* * Reliably open and lock a file. diff --git a/lib/libvarnish/vin.c b/lib/libvarnish/vin.c index 6dac9d211..6c098afb7 100644 --- a/lib/libvarnish/vin.c +++ b/lib/libvarnish/vin.c @@ -30,7 +30,6 @@ #include "config.h" -#include #include #include #include diff --git a/lib/libvarnish/vjsn.c b/lib/libvarnish/vjsn.c index 4ba0cd4da..61280496a 100644 --- a/lib/libvarnish/vjsn.c +++ b/lib/libvarnish/vjsn.c @@ -28,7 +28,6 @@ #include "config.h" -#include #include #include #include diff --git a/lib/libvarnish/vpf.c b/lib/libvarnish/vpf.c index cf326fb02..0c1bb1f2d 100644 --- a/lib/libvarnish/vpf.c +++ b/lib/libvarnish/vpf.c @@ -32,7 +32,6 @@ #include #include -#include #include #include #include diff --git a/lib/libvarnish/vsb.c b/lib/libvarnish/vsb.c index e1e6d1445..0e92b1771 100644 --- a/lib/libvarnish/vsb.c +++ b/lib/libvarnish/vsb.c @@ -30,7 +30,6 @@ __FBSDID("$FreeBSD: head/sys/kern/subr_vsb.c 222004 2011-05-17 06:36:32Z phk $") #include "config.h" #include -#include #include #include #include diff --git a/lib/libvarnish/vsub.c b/lib/libvarnish/vsub.c index cafbe4086..06457992c 100644 --- a/lib/libvarnish/vsub.c +++ b/lib/libvarnish/vsub.c @@ -33,7 +33,6 @@ #include -#include #include #include // Solaris closefrom(3c) #include diff --git a/lib/libvarnish/vtcp.c b/lib/libvarnish/vtcp.c index 93c2fce4a..3294bbf08 100644 --- a/lib/libvarnish/vtcp.c +++ b/lib/libvarnish/vtcp.c @@ -39,7 +39,6 @@ #include #include -#include #include #include #include diff --git a/lib/libvarnish/vus.c b/lib/libvarnish/vus.c index f233c5f8f..ba8a3d507 100644 --- a/lib/libvarnish/vus.c +++ b/lib/libvarnish/vus.c @@ -30,7 +30,6 @@ #include #include -#include #include #include #include diff --git a/lib/libvarnishapi/daemon.c b/lib/libvarnishapi/daemon.c index bd723517f..20d9ebc2c 100644 --- a/lib/libvarnishapi/daemon.c +++ b/lib/libvarnishapi/daemon.c @@ -32,7 +32,6 @@ //lint -e{766} #include "config.h" -#include #include #include #include diff --git a/lib/libvarnishapi/vsl.c b/lib/libvarnishapi/vsl.c index 970c1750e..1818767a2 100644 --- a/lib/libvarnishapi/vsl.c +++ b/lib/libvarnishapi/vsl.c @@ -30,7 +30,6 @@ #include "config.h" -#include #include #include #include diff --git a/lib/libvarnishapi/vsl_cursor.c b/lib/libvarnishapi/vsl_cursor.c index 8559c071a..652751c52 100644 --- a/lib/libvarnishapi/vsl_cursor.c +++ b/lib/libvarnishapi/vsl_cursor.c @@ -31,7 +31,6 @@ #include "config.h" -#include #include #include #include diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c index 36d737775..7380bbae3 100644 --- a/lib/libvarnishapi/vsm.c +++ b/lib/libvarnishapi/vsm.c @@ -33,7 +33,6 @@ #include #include -#include #include #include #include diff --git a/lib/libvarnishapi/vut.c b/lib/libvarnishapi/vut.c index 9c02bf407..9c767bfe7 100644 --- a/lib/libvarnishapi/vut.c +++ b/lib/libvarnishapi/vut.c @@ -37,7 +37,6 @@ #include #include #include -#include #include #include #include /* for MUSL */ diff --git a/lib/libvcc/vcc_compile.c b/lib/libvcc/vcc_compile.c index bcfaadcab..adfee9579 100644 --- a/lib/libvcc/vcc_compile.c +++ b/lib/libvcc/vcc_compile.c @@ -52,7 +52,6 @@ #include "config.h" -#include #include #include #include diff --git a/lib/libvcc/vcc_utils.c b/lib/libvcc/vcc_utils.c index b28f1628f..e92be266f 100644 --- a/lib/libvcc/vcc_utils.c +++ b/lib/libvcc/vcc_utils.c @@ -36,7 +36,6 @@ #include #include #include -#include #include "vcc_compile.h" diff --git a/lib/libvmod_blob/base64.c b/lib/libvmod_blob/base64.c index 3bea94871..62ad71e39 100644 --- a/lib/libvmod_blob/base64.c +++ b/lib/libvmod_blob/base64.c @@ -27,7 +27,6 @@ */ #include "config.h" -#include #include "vdef.h" #include "vrt.h" diff --git a/lib/libvmod_blob/hex.c b/lib/libvmod_blob/hex.c index dfdbb3e9e..2306a2fec 100644 --- a/lib/libvmod_blob/hex.c +++ b/lib/libvmod_blob/hex.c @@ -28,7 +28,6 @@ #include "config.h" #include -#include #include "hex.h" diff --git a/lib/libvmod_blob/id.c b/lib/libvmod_blob/id.c index 1164e983d..3fc8d5280 100644 --- a/lib/libvmod_blob/id.c +++ b/lib/libvmod_blob/id.c @@ -28,7 +28,6 @@ #include "config.h" #include -#include #include "vdef.h" #include "vrt.h" diff --git a/lib/libvmod_blob/url.c b/lib/libvmod_blob/url.c index acb55a6bb..1713bb5ab 100644 --- a/lib/libvmod_blob/url.c +++ b/lib/libvmod_blob/url.c @@ -27,7 +27,6 @@ */ #include "config.h" -#include #include "hex.h" diff --git a/lib/libvmod_blob/vmod_blob.c b/lib/libvmod_blob/vmod_blob.c index 013533ae9..1b675abb2 100644 --- a/lib/libvmod_blob/vmod_blob.c +++ b/lib/libvmod_blob/vmod_blob.c @@ -27,7 +27,6 @@ */ #include "config.h" -#include #include #include "cache/cache.h" diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c index 61029ad94..8851ae6e4 100644 --- a/lib/libvmod_debug/vmod_debug.c +++ b/lib/libvmod_debug/vmod_debug.c @@ -28,7 +28,6 @@ #include "config.h" -#include #include #include #include diff --git a/lib/libvmod_debug/vmod_debug_dyn.c b/lib/libvmod_debug/vmod_debug_dyn.c index cd0b7b6a3..c7026b4ea 100644 --- a/lib/libvmod_debug/vmod_debug_dyn.c +++ b/lib/libvmod_debug/vmod_debug_dyn.c @@ -35,7 +35,6 @@ #include #include #include -#include #include "cache/cache.h" diff --git a/lib/libvmod_unix/cred_compat.h b/lib/libvmod_unix/cred_compat.h index cd4fd7437..fe3a5b9ff 100644 --- a/lib/libvmod_unix/cred_compat.h +++ b/lib/libvmod_unix/cred_compat.h @@ -27,7 +27,6 @@ #include #include -#include #if defined(HAVE_GETPEEREID) #include diff --git a/lib/libvmod_vtc/vmod_vtc.c b/lib/libvmod_vtc/vmod_vtc.c index 6b6501adf..c2d4f383c 100644 --- a/lib/libvmod_vtc/vmod_vtc.c +++ b/lib/libvmod_vtc/vmod_vtc.c @@ -29,7 +29,6 @@ #include "config.h" -#include #include #include #include From fgsch at lodoss.net Mon Nov 5 19:53:15 2018 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Mon, 5 Nov 2018 19:53:15 +0000 (UTC) Subject: [master] 539e41ec2 Polish Message-ID: <20181105195315.42204A5E01@lists.varnish-cache.org> commit 539e41ec2db997c56a604d85e96e346af41a31eb Author: Federico G. Schwindt Date: Mon Nov 5 19:51:39 2018 +0000 Polish diff --git a/include/verrno.h b/include/verrno.h index f2ab427ed..f457aa2c7 100644 --- a/include/verrno.h +++ b/include/verrno.h @@ -7,6 +7,5 @@ */ #include -#include const char * vstrerror(int e); diff --git a/lib/libvarnish/verrno.c b/lib/libvarnish/verrno.c index 2e377f7bd..924010215 100644 --- a/lib/libvarnish/verrno.c +++ b/lib/libvarnish/verrno.c @@ -8,6 +8,8 @@ #include "config.h" +#include + #include "verrno.h" const char * From phk at FreeBSD.org Tue Nov 6 08:20:14 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 6 Nov 2018 08:20:14 +0000 (UTC) Subject: [master] 3e0cdae03 Decouple v_printflike_() from the OS #includes Message-ID: <20181106082014.B1B22BFED9@lists.varnish-cache.org> commit 3e0cdae03c55f9209744610b42e195d45c2b943b Author: Poul-Henning Kamp Date: Tue Nov 6 08:19:20 2018 +0000 Decouple v_printflike_() from the OS #includes diff --git a/bin/varnishd/cache/cache_acceptor.c b/bin/varnishd/cache/cache_acceptor.c index bc3c1360b..3874ac70d 100644 --- a/bin/varnishd/cache/cache_acceptor.c +++ b/bin/varnishd/cache/cache_acceptor.c @@ -37,7 +37,6 @@ #include #include #include -#include #include "cache_varnishd.h" diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c index 8617bfdaf..99485af83 100644 --- a/bin/varnishd/mgt/mgt_child.c +++ b/bin/varnishd/mgt/mgt_child.c @@ -34,7 +34,6 @@ #include #include -#include #include #include #include From phk at FreeBSD.org Tue Nov 6 08:41:19 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 6 Nov 2018 08:41:19 +0000 (UTC) Subject: [master] 3ef62a623 Move static asserts to vas.h, and decouple from OS #includes Message-ID: <20181106084119.323ED4711@lists.varnish-cache.org> commit 3ef62a623a1e8ebfd4eb1dfd53410445fb468c2a Author: Poul-Henning Kamp Date: Tue Nov 6 08:39:59 2018 +0000 Move static asserts to vas.h, and decouple from OS #includes diff --git a/flint.lnt b/flint.lnt index 7c7736d3d..a20d04af2 100644 --- a/flint.lnt +++ b/flint.lnt @@ -112,6 +112,9 @@ -emacro(527, WRONG) // unreachable code -emacro(774, VALID_OBJ) // boolean always true +-emacro(506, v_static_assert) // Constant value Boolean +-esym(751, __vassert_*) // local typedef '___' (___) not referenced + /////////////////////////////////////////////////////////////////////// // Places where we use x<<0 for reasons of symmetry diff --git a/include/vas.h b/include/vas.h index 841bd309a..bd51d9211 100644 --- a/include/vas.h +++ b/include/vas.h @@ -90,4 +90,26 @@ do { \ "", VAS_INCOMPLETE); \ } while (0) +/* + * Most of this nightmare is stolen from FreeBSD's + */ +#ifndef __has_extension +# define __has_extension(x) 0 +#endif + +#if __has_extension(c_static_assert) +# define v_static_assert _Static_assert +#elif __GNUC_PREREQ__(4,6) && !defined(__cplusplus) +# define v_static_assert _Static_assert +#else +# if defined(__COUNTER__) +# define v_static_assert(x, y) __v_static_assert(x, __COUNTER__) +# else +# define v_static_assert(x, y) __v_static_assert(x, __LINE__) +# endif +# define __v_static_assert(x, y) ___v_static_assert(x, y) +# define ___v_static_assert(x, y) \ + typedef char __vassert_## y[(x) ? 1 : -1] v_unused_ +#endif + #endif diff --git a/include/vdef.h b/include/vdef.h index 3a5e40329..df63b245f 100644 --- a/include/vdef.h +++ b/include/vdef.h @@ -73,9 +73,7 @@ # endif #endif -#ifdef __printflike -# define v_printflike_(f,a) __printflike(f,a) -#elif __GNUC_PREREQ__(2, 95) || defined(__INTEL_COMPILER) +#if __GNUC_PREREQ__(2, 95) || defined(__INTEL_COMPILER) # define v_printflike_(f,a) __attribute__((format(printf, f, a))) #else # define v_printflike_(f,a) @@ -139,33 +137,6 @@ # define v_unused_ #endif -/* - * Most of this nightmare is stolen from FreeBSD's - */ -#ifndef __has_extension -# define __has_extension(x) 0 -#endif - -#if defined(_Static_assert) - /* Nothing, somebody already did this for us */ -#elif __has_extension(c_static_assert) - /* Nothing, we should be fine */ -#elif (defined(__cplusplus) && __cplusplus >= 201103L) || \ - __has_extension(cxx_static_assert) -# define _Static_assert(x, y) static_assert(x, y) -#elif __GNUC_PREREQ__(4,6) && !defined(__cplusplus) - /* Nothing, gcc 4.6 and higher has _Static_assert built-in */ -#else -# if defined(__COUNTER__) -# define _Static_assert(x, y) __Static_assert(x, __COUNTER__) -# else -# define _Static_assert(x, y) __Static_assert(x, __LINE__) -# endif -# define __Static_assert(x, y) ___Static_assert(x, y) -# define ___Static_assert(x, y) \ - typedef char __assert_## y[(x) ? 1 : -1] v_unused_ -#endif - /* VTIM API overhaul WIP */ typedef double vtim_mono; typedef double vtim_real; diff --git a/lib/libvcc/vsctool.py b/lib/libvcc/vsctool.py index f1e0d02ab..d93871b73 100644 --- a/lib/libvcc/vsctool.py +++ b/lib/libvcc/vsctool.py @@ -219,7 +219,7 @@ class CounterSet(object): def emit_c_paranoia(self, fo): '''Emit asserts to make sure compiler gets same byte index''' fo.write("#define PARANOIA(a,n)\t\t\t\t\\\n") - fo.write(" _Static_assert(\t\t\t\t\\\n") + fo.write(" v_static_assert(\t\t\t\t\\\n") fo.write("\toffsetof(" + self.struct + ", a) == n,\t\\\n") fo.write("\t\"VSC element '\" #a \"' at wrong offset\")\n\n") From phk at FreeBSD.org Tue Nov 6 09:24:11 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 6 Nov 2018 09:24:11 +0000 (UTC) Subject: [master] a2392eb72 Eliminate pointless return value Message-ID: <20181106092411.6FB8554DD@lists.varnish-cache.org> commit a2392eb721e850e7205fc077e56b3675673f74b2 Author: Poul-Henning Kamp Date: Tue Nov 6 08:53:27 2018 +0000 Eliminate pointless return value diff --git a/bin/varnishd/http2/cache_http2_deliver.c b/bin/varnishd/http2/cache_http2_deliver.c index 50f9dd760..20a300811 100644 --- a/bin/varnishd/http2/cache_http2_deliver.c +++ b/bin/varnishd/http2/cache_http2_deliver.c @@ -171,7 +171,7 @@ h2_minimal_response(struct req *req, uint16_t status) return (0); } -static int +static void h2_enc_len(struct vsb *vsb, unsigned bits, unsigned val, uint8_t b0) { assert(bits < 8); @@ -186,7 +186,6 @@ h2_enc_len(struct vsb *vsb, unsigned bits, unsigned val, uint8_t b0) } } VSB_putc(vsb, (uint8_t)val); - return (0); } /* From phk at FreeBSD.org Tue Nov 6 09:24:11 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 6 Nov 2018 09:24:11 +0000 (UTC) Subject: [master] cd178217e Make this test un-racy. Message-ID: <20181106092411.8C7A854E0@lists.varnish-cache.org> commit cd178217e1397110e5401ae07e03493e463d3b69 Author: Poul-Henning Kamp Date: Tue Nov 6 09:23:07 2018 +0000 Make this test un-racy. diff --git a/bin/varnishtest/tests/r01821.vtc b/bin/varnishtest/tests/r01821.vtc index b07799325..a7943e1b0 100644 --- a/bin/varnishtest/tests/r01821.vtc +++ b/bin/varnishtest/tests/r01821.vtc @@ -2,9 +2,13 @@ varnishtest "Slim down hit-for-miss / hit-for-miss objects" # see also #2768 -server s1 -repeat 2 { +server s1 { rxreq - txresp -bodylen 65535 + expect req.url == "/hfm" + txresp -hdr "HFM: True" -bodylen 65530 + rxreq + expect req.url == "/hfp" + txresp -hdr "HFP: True" -bodylen 65550 } -start varnish v1 -arg "-s Transient=default" -vcl+backend { @@ -25,16 +29,15 @@ logexpect l1 -v v1 -g raw { client c1 { txreq -url "/hfm" rxresp -} -start - -client c2 { + expect resp.status == 200 + expect resp.http.hfm == True txreq -url "/hfp" rxresp + expect resp.status == 200 + expect resp.http.hfp == True } -run -client c1 -wait - logexpect l1 -wait varnish v1 -expect SM?.Transient.c_bytes > 131072 -varnish v1 -expect SM?.Transient.g_bytes < 65536 +varnish v1 -expect SM?.Transient.g_bytes < 2000 From nils.goroll at uplex.de Tue Nov 6 10:17:13 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 6 Nov 2018 10:17:13 +0000 (UTC) Subject: [master] 13015f599 For warmup/rampup, only consider healthy backends Message-ID: <20181106101713.EDC1A6894@lists.varnish-cache.org> commit 13015f59968438afb5ac99a511dd654205089a30 Author: Nils Goroll Date: Tue Nov 6 10:56:44 2018 +0100 For warmup/rampup, only consider healthy backends For healthy == IGNORE, we return before the call to shard_next(), so at this point we do want to consider health state - and it makes absolutely no sense to give up a healthy backend in favor of an un- healthy one anyway. Fixes #2823 diff --git a/bin/varnishtest/tests/d00023.vtc b/bin/varnishtest/tests/d00023.vtc index 36ebe8e78..12c494f56 100644 --- a/bin/varnishtest/tests/d00023.vtc +++ b/bin/varnishtest/tests/d00023.vtc @@ -13,6 +13,9 @@ server s3 { rxreq expect req.http.healthy == "true" txresp -body "xiuFi3Pe" + rxreq + expect req.http.healthy == "true" + txresp -body "xiuFi3Pe" } -start varnish v1 -vcl+backend { @@ -29,8 +32,16 @@ varnish v1 -vcl+backend { } sub vcl_recv { - set req.backend_hint = vd.backend(by=KEY, - key=1756955383); + if (req.url == "/warmup") { + # key for which the order is s3->s1->s2 + # we do not want to hit s1/s2 because they are down + set req.backend_hint = vd.backend(by=KEY, + key=192192123, warmup=1.0); + } else { + # key for which the order is s1->s2->s3 + set req.backend_hint = vd.backend(by=KEY, + key=1756955383); + } set req.http.healthy = std.healthy(req.backend_hint); return(pass); } @@ -50,4 +61,7 @@ client c1 { txreq rxresp expect resp.body == "xiuFi3Pe" + txreq -url "/warmup" + rxresp + expect resp.body == "xiuFi3Pe" } -run diff --git a/lib/libvmod_directors/shard_dir.c b/lib/libvmod_directors/shard_dir.c index bfc86f45a..acee05780 100644 --- a/lib/libvmod_directors/shard_dir.c +++ b/lib/libvmod_directors/shard_dir.c @@ -401,7 +401,7 @@ sharddir_pick_be(VRT_CTX, struct sharddir *shardd, /* short path for cases we dont want ramup/warmup or can't */ if (alt > 0 || healthy == IGNORE || (!rampup && warmup == 0) || - shard_next(&state, 0, 0) == -1) + shard_next(&state, 0, 1) == -1) goto ok; assert(alt == 0); From nils.goroll at uplex.de Tue Nov 6 10:26:08 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 6 Nov 2018 10:26:08 +0000 (UTC) Subject: [master] dbc956693 default thread_pool_watchdog == default cli_timeout Message-ID: <20181106102608.B41626CF7@lists.varnish-cache.org> commit dbc956693df2e8c45e2505330e2eeac681b5e6d2 Author: Nils Goroll Date: Tue Oct 9 14:15:22 2018 +0200 default thread_pool_watchdog == default cli_timeout Avoid (potential) issues due to the monotonic clock not advancing for ~10 seconds such as for virtual machine snapshots or migrations. In such a scenario, we would kill a child after cli_timeout anyway, so using the same timeout by default for the watchdog also should reduce the additional risk implied by it. Ref #2814 diff --git a/bin/varnishd/mgt/mgt_pool.c b/bin/varnishd/mgt/mgt_pool.c index 25b6efba0..98ad42c3c 100644 --- a/bin/varnishd/mgt/mgt_pool.c +++ b/bin/varnishd/mgt/mgt_pool.c @@ -156,7 +156,7 @@ struct parspec WRK_parspec[] = { "If no queued work have been released for this long," " the worker process panics itself.", EXPERIMENTAL, - "10", "seconds" }, + "60", "seconds" }, { "thread_pool_destroy_delay", tweak_timeout, &mgt_param.wthread_destroy_delay, "0.01", NULL, diff --git a/include/tbl/params.h b/include/tbl/params.h index f1703613c..d9f27c99e 100644 --- a/include/tbl/params.h +++ b/include/tbl/params.h @@ -1202,7 +1202,7 @@ PARAM( /* typ */ timeout, /* min */ "0.1", /* max */ NULL, - /* default */ "10.000", + /* default */ "60.000", /* units */ "seconds", /* flags */ EXPERIMENTAL, /* s-text */ From nils.goroll at uplex.de Tue Nov 6 10:51:13 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 6 Nov 2018 10:51:13 +0000 (UTC) Subject: [master] ee9e0f6e7 promote vtree.h to vmod include Message-ID: <20181106105113.ECA56766D@lists.varnish-cache.org> commit ee9e0f6e7259580f17bcba83158c2cc9b79c13f3 Author: Nils Goroll Date: Tue Nov 6 11:47:11 2018 +0100 promote vtree.h to vmod include Now that we include it in cache.h, it is required by vmods Ref #2813 diff --git a/include/Makefile.am b/include/Makefile.am index 802231dce..1479a6384 100644 --- a/include/Makefile.am +++ b/include/Makefile.am @@ -72,6 +72,7 @@ nobase_pkginclude_HEADERS += \ vsha256.h \ vtcp.h \ vtim.h \ + vtree.h \ vrnd.h # Private headers @@ -99,7 +100,6 @@ nobase_noinst_HEADERS = \ vsub.h \ vss.h \ vtcp.h \ - vtree.h \ vus.h ## keep in sync with lib/libvcc/Makefile.am From nils.goroll at uplex.de Tue Nov 6 10:57:16 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 6 Nov 2018 10:57:16 +0000 (UTC) Subject: [master] b07b701a4 lost commit from ee9e0f6e7259580f17bcba83158c2cc9b79c13f3 Message-ID: <20181106105716.E5B257A6F@lists.varnish-cache.org> commit b07b701a43d58d70b9b84260148938c3c83192d0 Author: Nils Goroll Date: Tue Nov 6 11:54:50 2018 +0100 lost commit from ee9e0f6e7259580f17bcba83158c2cc9b79c13f3 diff --git a/tools/include_wash.py b/tools/include_wash.py index 4df6eeaf3..973d5d6ad 100644 --- a/tools/include_wash.py +++ b/tools/include_wash.py @@ -33,7 +33,7 @@ def check(fn): if i in l: for i in ("stddef.h", "stdint.h", "vrt.h", "math.h", "pthread.h", "stdarg.h", "sys/types.h", - "vdef.h", "miniobj.h", "vas.h", "vqueue.h"): + "vdef.h", "miniobj.h", "vas.h", "vqueue.h", "vtree.h"): if i in l: print(fn, i + " included with cache.h") From nils.goroll at uplex.de Tue Nov 6 11:26:11 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 6 Nov 2018 11:26:11 +0000 (UTC) Subject: [master] ca9d07195 changelog tlc Message-ID: <20181106112611.794D69460@lists.varnish-cache.org> commit ca9d07195b1d3389aa89ebd78a5af5713b3329a6 Author: Nils Goroll Date: Tue Nov 6 12:25:00 2018 +0100 changelog tlc will continue looking backwards from 79687f1344387a26d5e3b0a3b0454ac917db76de diff --git a/doc/changes.rst b/doc/changes.rst index d817aa794..82bdc813b 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -67,6 +67,47 @@ Varnish Cache trunk (ongoing) * Changed ``ExpKill`` log tags to emit microsecond-precision timestamps instead of nanoseconds (2792_) +* Included ``vtree.h`` in the distribution for vmods and + renamed the red/black tree macros from ``VRB_*`` to ``VRBT_*`` + to disambiguate from the acronym for Varnish Request Body. + + Changed the internal organisation of dynamic PRIVs (``PRIV_TASK``, + ``PRIV_TOP`` from a list to a red/black tree) for performance. + (2813_) + +* Changed the default of the ``thread_pool_watchdog`` parameter + to 60 seconds to match the ``cli_timeout`` default + +* Fix warmup/rampup of the shard director (2823_) + +* Added ``vstrerror()`` as a safe wrapper for ``strerror()`` to avoid + a NULL pointer dereference under rare conditions where the latter + could return NULL. (2815_) + +* Fix VRT_priv_task for calls from vcl_pipe {} (2820_) + +* For ``varnishtest -L``, also keep VCL C source files + +* Fix assinging == (2809_) + +* Fix vmod object constructor documentation in the ``vmodtool.py`` - + generated RST files + +* Vmod developers are advised that anything returned by a vmod + function/method is assumed to be immutable. In other words, a vmod + `must not` modify any data which was previously returned. + +* VSB quoted output has been unified to three-digit octal, + VSB_QUOTE_ESCHEX has been added to prefer hex over octal quoting + +* Varnish-based tools using the VUT interface should now consider + using the ``VUT_Usage()`` function for consistency + +.. _2809: https://github.com/varnishcache/varnish-cache/issues/2809 +.. _2820: https://github.com/varnishcache/varnish-cache/issues/2820 +.. _2815: https://github.com/varnishcache/varnish-cache/issues/2815 +.. _2823: https://github.com/varnishcache/varnish-cache/issues/2823 +.. _2813: https://github.com/varnishcache/varnish-cache/pull/2813 .. _2792: https://github.com/varnishcache/varnish-cache/pull/2792 .. _2783: https://github.com/varnishcache/varnish-cache/pull/2783 .. _2780: https://github.com/varnishcache/varnish-cache/issues/2780 From phk at FreeBSD.org Tue Nov 6 19:36:12 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 6 Nov 2018 19:36:12 +0000 (UTC) Subject: [master] a9aa47768 Cut this test down to a reasonable size for the VTEST boxes Message-ID: <20181106193612.757FB952F6@lists.varnish-cache.org> commit a9aa477687c6939011e31801f01efa6df9530ec4 Author: Poul-Henning Kamp Date: Tue Nov 6 19:35:14 2018 +0000 Cut this test down to a reasonable size for the VTEST boxes diff --git a/bin/varnishtest/tests/v00041.vtc b/bin/varnishtest/tests/v00041.vtc index 09977744a..7d14b388f 100644 --- a/bin/varnishtest/tests/v00041.vtc +++ b/bin/varnishtest/tests/v00041.vtc @@ -53,7 +53,7 @@ varnish v1 -arg "-p debug=+vclrel -p workspace_client=1m" -vcl+backend { std.log("perf 1 " + debug.priv_perf(1)); std.log("perf 10 " + debug.priv_perf(10)); std.log("perf 100 " + debug.priv_perf(100)); - std.log("perf 1000 " + debug.priv_perf(1000)); + // std.log("perf 1000 " + debug.priv_perf(1000)); return (deliver); } From hermunn at varnish-software.com Wed Nov 7 09:13:18 2018 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Wed, 7 Nov 2018 09:13:18 +0000 (UTC) Subject: [6.1] 2f2387038 Fix gensequences for BusyBox awk Message-ID: <20181107091318.F3356ACEF1@lists.varnish-cache.org> commit 2f2387038a7b9aca4c31c0f839cda3b7ab3391c0 Author: Guillaume Quintard Date: Thu Apr 26 21:59:41 2018 +0200 Fix gensequences for BusyBox awk I never thought that I'd have to fix a string concatenation problem in a BusyBox awk program to generate VT100 code in a container, but here we are: echo | awk 'END {print "foo" "" ++a, "foo" ++a}' should output "foo0 foo1", and for all the ?awk I tested, it does, except for BusyBox awk who thought funny to output "0 foo1", breaking the teken_state.h file. diff --git a/bin/varnishtest/gensequences b/bin/varnishtest/gensequences index 4337186b8..18a68b43b 100644 --- a/bin/varnishtest/gensequences +++ b/bin/varnishtest/gensequences @@ -70,7 +70,7 @@ while (getline > 0) { l_prefix_parent[n] = prefix; l_prefix_suffix[n] = sequence[i]; if (!l_prefix_name[n]) - l_prefix_name[n] = "teken_state_" ++npr; + l_prefix_name[n] = "teken_state_" "" ++npr; prefix = n; } From hermunn at varnish-software.com Wed Nov 7 10:04:05 2018 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Wed, 7 Nov 2018 10:04:05 +0000 (UTC) Subject: [6.0] 2f883b152 fix VRT_priv_task for calls from vcl_pipe {} and test for it Message-ID: <20181107100405.366DCADF59@lists.varnish-cache.org> commit 2f883b15279818abaafda32d122f98a8f64df6d3 Author: Nils Goroll Date: Mon Nov 5 12:46:38 2018 +0100 fix VRT_priv_task for calls from vcl_pipe {} and test for it Fixes #2820 Conflicts: bin/varnishtest/tests/v00041.vtc diff --git a/bin/varnishd/cache/cache_vrt_priv.c b/bin/varnishd/cache/cache_vrt_priv.c index e4dd16479..52fd156b7 100644 --- a/bin/varnishd/cache/cache_vrt_priv.c +++ b/bin/varnishd/cache/cache_vrt_priv.c @@ -35,6 +35,7 @@ #include #include "cache_varnishd.h" +#include "vcl.h" struct vrt_priv { unsigned magic; @@ -125,7 +126,9 @@ VRT_priv_task(VRT_CTX, const void *vmod_id) struct vmod_priv *vp; CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); - assert(ctx->req == NULL || ctx->bo == NULL); + assert(ctx->req == NULL || ctx->bo == NULL || + ctx->method == VCL_MET_PIPE); + if (ctx->req) { CHECK_OBJ_NOTNULL(ctx->req, REQ_MAGIC); id = (uintptr_t)ctx->req; diff --git a/bin/varnishtest/tests/v00041.vtc b/bin/varnishtest/tests/v00041.vtc index a67928ca9..3f86c9268 100644 --- a/bin/varnishtest/tests/v00041.vtc +++ b/bin/varnishtest/tests/v00041.vtc @@ -6,6 +6,11 @@ server s1 { rxreq txresp + rxreq + txresp + expect_close + accept + rxreq txresp } -start @@ -26,6 +31,18 @@ varnish v1 -arg "-p debug=+vclrel" -vcl+backend { } sub vcl_recv { + if (req.url == "/perf") { + return (synth(200)); + } + if (req.url == "/pipe") { + return (pipe); + } + debug.test_priv_task(req.url); + set req.http.x0 = debug.test_priv_task(); + debug.test_priv_task("bazz"); + } + + sub vcl_pipe { debug.test_priv_task(req.url); set req.http.x0 = debug.test_priv_task(); debug.test_priv_task("bazz"); @@ -109,6 +126,12 @@ client c1 { expect resp.http.bx0 == "b /snafu" expect resp.http.bx1 == "b /snafu" expect resp.http.bo1 == "b /snafu" + + txreq -url /perf + rxresp + + txreq -url /pipe + rxresp } -run shell "echo 'vcl 4.0; backend foo { .host = \"${s1_addr}\"; .port = \"${s1_port}\"; }' > ${tmpdir}/_b00014.vcl" diff --git a/doc/changes.rst b/doc/changes.rst index faeaaaa84..5f8d3b90c 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -91,6 +91,8 @@ Varnish Cache 6.0.2 (unreleased) * Only dlclose() Vmods after all "fini" processing (2800_) +* Fix VRT_priv_task for calls from vcl_pipe {} and test for it (2820_) + .. _2418: https://github.com/varnishcache/varnish-cache/issues/2418 .. _2589: https://github.com/varnishcache/varnish-cache/issues/2589 .. _2654: https://github.com/varnishcache/varnish-cache/issues/2654 @@ -119,6 +121,7 @@ Varnish Cache 6.0.2 (unreleased) .. _2792: https://github.com/varnishcache/varnish-cache/pull/2792 .. _2794: https://github.com/varnishcache/varnish-cache/issues/2794 .. _2800: https://github.com/varnishcache/varnish-cache/issues/2800 +.. _2820: https://github.com/varnishcache/varnish-cache/issues/2820 ================================ From hermunn at varnish-software.com Wed Nov 7 10:04:05 2018 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Wed, 7 Nov 2018 10:04:05 +0000 (UTC) Subject: [6.0] 1262e2533 Fix gensequences for BusyBox awk Message-ID: <20181107100405.6743EADF5C@lists.varnish-cache.org> commit 1262e2533a943b58a479374ee98509d354d2be7d Author: Guillaume Quintard Date: Thu Apr 26 21:59:41 2018 +0200 Fix gensequences for BusyBox awk I never thought that I'd have to fix a string concatenation problem in a BusyBox awk program to generate VT100 code in a container, but here we are: echo | awk 'END {print "foo" "" ++a, "foo" ++a}' should output "foo0 foo1", and for all the ?awk I tested, it does, except for BusyBox awk who thought funny to output "0 foo1", breaking the teken_state.h file. diff --git a/bin/varnishtest/gensequences b/bin/varnishtest/gensequences index 4337186b8..18a68b43b 100644 --- a/bin/varnishtest/gensequences +++ b/bin/varnishtest/gensequences @@ -70,7 +70,7 @@ while (getline > 0) { l_prefix_parent[n] = prefix; l_prefix_suffix[n] = sequence[i]; if (!l_prefix_name[n]) - l_prefix_name[n] = "teken_state_" ++npr; + l_prefix_name[n] = "teken_state_" "" ++npr; prefix = n; } From hermunn at varnish-software.com Wed Nov 7 10:04:05 2018 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Wed, 7 Nov 2018 10:04:05 +0000 (UTC) Subject: [6.0] 33c07e1cc default thread_pool_watchdog == default cli_timeout Message-ID: <20181107100405.BD4DCADF61@lists.varnish-cache.org> commit 33c07e1ccb6b26e230bdcee76223353fcf340c2a Author: Nils Goroll Date: Tue Oct 9 14:15:22 2018 +0200 default thread_pool_watchdog == default cli_timeout Avoid (potential) issues due to the monotonic clock not advancing for ~10 seconds such as for virtual machine snapshots or migrations. In such a scenario, we would kill a child after cli_timeout anyway, so using the same timeout by default for the watchdog also should reduce the additional risk implied by it. Ref #2814 diff --git a/bin/varnishd/mgt/mgt_pool.c b/bin/varnishd/mgt/mgt_pool.c index 25b6efba0..98ad42c3c 100644 --- a/bin/varnishd/mgt/mgt_pool.c +++ b/bin/varnishd/mgt/mgt_pool.c @@ -156,7 +156,7 @@ struct parspec WRK_parspec[] = { "If no queued work have been released for this long," " the worker process panics itself.", EXPERIMENTAL, - "10", "seconds" }, + "60", "seconds" }, { "thread_pool_destroy_delay", tweak_timeout, &mgt_param.wthread_destroy_delay, "0.01", NULL, diff --git a/include/tbl/params.h b/include/tbl/params.h index 42083aa5e..71d425195 100644 --- a/include/tbl/params.h +++ b/include/tbl/params.h @@ -1201,7 +1201,7 @@ PARAM( /* typ */ timeout, /* min */ "0.1", /* max */ NULL, - /* default */ "10.000", + /* default */ "60.000", /* units */ "seconds", /* flags */ EXPERIMENTAL, /* s-text */ From hermunn at varnish-software.com Wed Nov 7 10:04:06 2018 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Wed, 7 Nov 2018 10:04:06 +0000 (UTC) Subject: [6.0] 177e40e82 For warmup/rampup, only consider healthy backends Message-ID: <20181107100406.50D42ADF65@lists.varnish-cache.org> commit 177e40e82030425357897eed249fd9118994011f Author: Nils Goroll Date: Tue Nov 6 10:56:44 2018 +0100 For warmup/rampup, only consider healthy backends For healthy == IGNORE, we return before the call to shard_next(), so at this point we do want to consider health state - and it makes absolutely no sense to give up a healthy backend in favor of an un- healthy one anyway. Fixes #2823 diff --git a/bin/varnishtest/tests/d00023.vtc b/bin/varnishtest/tests/d00023.vtc index 36ebe8e78..12c494f56 100644 --- a/bin/varnishtest/tests/d00023.vtc +++ b/bin/varnishtest/tests/d00023.vtc @@ -13,6 +13,9 @@ server s3 { rxreq expect req.http.healthy == "true" txresp -body "xiuFi3Pe" + rxreq + expect req.http.healthy == "true" + txresp -body "xiuFi3Pe" } -start varnish v1 -vcl+backend { @@ -29,8 +32,16 @@ varnish v1 -vcl+backend { } sub vcl_recv { - set req.backend_hint = vd.backend(by=KEY, - key=1756955383); + if (req.url == "/warmup") { + # key for which the order is s3->s1->s2 + # we do not want to hit s1/s2 because they are down + set req.backend_hint = vd.backend(by=KEY, + key=192192123, warmup=1.0); + } else { + # key for which the order is s1->s2->s3 + set req.backend_hint = vd.backend(by=KEY, + key=1756955383); + } set req.http.healthy = std.healthy(req.backend_hint); return(pass); } @@ -50,4 +61,7 @@ client c1 { txreq rxresp expect resp.body == "xiuFi3Pe" + txreq -url "/warmup" + rxresp + expect resp.body == "xiuFi3Pe" } -run diff --git a/doc/changes.rst b/doc/changes.rst index 5f8d3b90c..6f98e77a3 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -93,6 +93,9 @@ Varnish Cache 6.0.2 (unreleased) * Fix VRT_priv_task for calls from vcl_pipe {} and test for it (2820_) +* Shard director: For warmup/rampup, only consider healthy backends + (2823_) + .. _2418: https://github.com/varnishcache/varnish-cache/issues/2418 .. _2589: https://github.com/varnishcache/varnish-cache/issues/2589 .. _2654: https://github.com/varnishcache/varnish-cache/issues/2654 @@ -122,6 +125,7 @@ Varnish Cache 6.0.2 (unreleased) .. _2794: https://github.com/varnishcache/varnish-cache/issues/2794 .. _2800: https://github.com/varnishcache/varnish-cache/issues/2800 .. _2820: https://github.com/varnishcache/varnish-cache/issues/2820 +.. _2823: https://github.com/varnishcache/varnish-cache/issues/2823 ================================ diff --git a/lib/libvmod_directors/shard_dir.c b/lib/libvmod_directors/shard_dir.c index a83fc44dc..01f044fff 100644 --- a/lib/libvmod_directors/shard_dir.c +++ b/lib/libvmod_directors/shard_dir.c @@ -394,7 +394,7 @@ sharddir_pick_be(VRT_CTX, struct sharddir *shardd, /* short path for cases we dont want ramup/warmup or can't */ if (alt > 0 || healthy == IGNORE || (!rampup && warmup == 0) || - shard_next(&state, 0, 0) == -1) + shard_next(&state, 0, 1) == -1) goto ok; assert(alt == 0); From hermunn at varnish-software.com Wed Nov 7 10:04:06 2018 From: hermunn at varnish-software.com (Pål Hermunn Johansen) Date: Wed, 7 Nov 2018 10:04:06 +0000 (UTC) Subject: [6.0] 0458b54db Prepare for 6.0.2 Message-ID: <20181107100406.93106ADF7A@lists.varnish-cache.org> commit 0458b54db26cfbea79af45ca5c4767c7c2925a91 Author: P?l Hermunn Johansen Date: Tue Nov 6 15:27:42 2018 +0100 Prepare for 6.0.2 diff --git a/configure.ac b/configure.ac index 8794dfaee..615dcd9d1 100644 --- a/configure.ac +++ b/configure.ac @@ -2,7 +2,7 @@ AC_PREREQ(2.59) AC_COPYRIGHT([Copyright (c) 2006 Verdens Gang AS Copyright (c) 2006-2018 Varnish Software]) AC_REVISION([$Id$]) -AC_INIT([Varnish], [6.0.1], [varnish-dev at varnish-cache.org]) +AC_INIT([Varnish], [6.0.2], [varnish-dev at varnish-cache.org]) AC_CONFIG_SRCDIR(include/miniobj.h) AC_CONFIG_HEADERS([config.h]) AC_CONFIG_MACRO_DIR([m4]) diff --git a/doc/changes.rst b/doc/changes.rst index 6f98e77a3..624d4a43c 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -27,7 +27,7 @@ individual releases. These documents are updated as part of the release process. ================================ -Varnish Cache 6.0.2 (unreleased) +Varnish Cache 6.0.2 (2018-11-07) ================================ * Fix and test objhead refcount for hit-for-pass (2654_, 2754_, 2760_) From hermunn at varnish-software.com Wed Nov 7 12:36:12 2018 From: hermunn at varnish-software.com (hermunn) Date: Wed, 7 Nov 2018 12:36:12 +0000 (UTC) Subject: [master] 746f83945 fix typo Message-ID: <20181107123613.0A129B2E09@lists.varnish-cache.org> commit 746f839454cac878aae0803b5d33a48b7f0ef7f1 Author: Valentin Matei Date: Wed Nov 7 13:19:20 2018 +0200 fix typo diff --git a/doc/sphinx/whats-new/changes-6.1.rst b/doc/sphinx/whats-new/changes-6.1.rst index 3ceffc093..f9579b76c 100644 --- a/doc/sphinx/whats-new/changes-6.1.rst +++ b/doc/sphinx/whats-new/changes-6.1.rst @@ -3,7 +3,7 @@ Changes in Varnish 6.1 ====================== -This is release is a maintenance release, so while there are many actual +This release is a maintenance release, so while there are many actual changes, and of course many bugfixes, they should not have little to no impact on running Varnish installations. From phk at FreeBSD.org Wed Nov 7 14:09:19 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 7 Nov 2018 14:09:19 +0000 (UTC) Subject: [master] 27ffa6a27 Re-add fix from 9f50c4bd12303674890ca742ce6b9592a29e3ba9, which was lost in f2abaf8698a34a44d8a80d80756b0f843560e31d. Message-ID: <20181107140919.4D899BF8AE@lists.varnish-cache.org> commit 27ffa6a27fe66951a2af9f2dfd7a7153eac2c7e6 Author: Poul-Henning Kamp Date: Wed Nov 7 14:07:58 2018 +0000 Re-add fix from 9f50c4bd12303674890ca742ce6b9592a29e3ba9, which was lost in f2abaf8698a34a44d8a80d80756b0f843560e31d. diff --git a/bin/varnishtest/gensequences b/bin/varnishtest/gensequences index 4337186b8..18a68b43b 100644 --- a/bin/varnishtest/gensequences +++ b/bin/varnishtest/gensequences @@ -70,7 +70,7 @@ while (getline > 0) { l_prefix_parent[n] = prefix; l_prefix_suffix[n] = sequence[i]; if (!l_prefix_name[n]) - l_prefix_name[n] = "teken_state_" ++npr; + l_prefix_name[n] = "teken_state_" "" ++npr; prefix = n; } From phk at FreeBSD.org Mon Nov 12 10:14:13 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 12 Nov 2018 10:14:13 +0000 (UTC) Subject: [master] 55995ab0f Renovate libvarnisapi signal handling. Message-ID: <20181112101413.CE27B97793@lists.varnish-cache.org> commit 55995ab0f96c458b1b585afe49c3af737e363205 Author: Poul-Henning Kamp Date: Mon Nov 12 10:10:28 2018 +0000 Renovate libvarnisapi signal handling. Introduce VSIG which simply counts number of signals, which everybody else can then look at. If you use libvarnishapi and want your own signal handler, you must increment these VSIG_{int|hup|term|usr1} counters yourself, for libvarnishapi to work correctly. The old VUT_Signal()/VUT_Signaled() API is retained and should be deprecated in a future version. Inspired by: #2818 diff --git a/bin/varnishhist/varnishhist.c b/bin/varnishhist/varnishhist.c index 626a80714..6d441ca6f 100644 --- a/bin/varnishhist/varnishhist.c +++ b/bin/varnishhist/varnishhist.c @@ -371,14 +371,6 @@ accumulate(struct VSL_data *vsl, struct VSL_transaction * const pt[], return (0); } -static int v_matchproto_(VUT_cb_f) -sighup(struct VUT *v) -{ - assert(v == vut); - quit = 1; - return (1); -} - static void * v_matchproto_(pthread_t) do_curses(void *arg) { @@ -392,7 +384,7 @@ do_curses(void *arg) intrflush(stdscr, FALSE); curs_set(0); erase(); - while (!quit) { + while (!quit && !vut->last_sighup) { AZ(pthread_mutex_lock(&mtx)); update(); AZ(pthread_mutex_unlock(&mtx)); @@ -480,14 +472,6 @@ profile_error(const char *s) exit(1); } -static void -vut_sighandler(int sig) -{ - - if (vut != NULL) - VUT_Signaled(vut, sig); -} - int main(int argc, char **argv) { @@ -638,7 +622,6 @@ main(int argc, char **argv) log_ten = log(10.0); - VUT_Signal(vut_sighandler); VUT_Setup(vut); if (vut->vsm) ident = VSM_Dup(vut->vsm, "Arg", "-i"); @@ -647,7 +630,6 @@ main(int argc, char **argv) AZ(pthread_create(&thr, NULL, do_curses, NULL)); vut->dispatch_f = accumulate; vut->dispatch_priv = NULL; - vut->sighup_f = sighup; (void)VUT_Main(vut); end_of_file = 1; AZ(pthread_join(thr, NULL)); diff --git a/bin/varnishlog/varnishlog.c b/bin/varnishlog/varnishlog.c index 8abab3037..09f8983d5 100644 --- a/bin/varnishlog/varnishlog.c +++ b/bin/varnishlog/varnishlog.c @@ -32,6 +32,7 @@ #include "config.h" +#include #include #include #include @@ -102,21 +103,6 @@ flushout(struct VUT *v) return (0); } -static int v_matchproto_(VUT_cb_f) -sighup(struct VUT *v) -{ - assert(v == vut); - return (1); -} - -static void -vut_sighandler(int sig) -{ - - if (vut != NULL) - VUT_Signaled(vut, sig); -} - int main(int argc, char * const *argv) { @@ -160,7 +146,6 @@ main(int argc, char * const *argv) vut->dispatch_f = VSL_PrintTransactions; else vut->dispatch_f = VSL_WriteTransactions; - vut->sighup_f = sighup; if (LOG.w_arg) { openout(LOG.a_opt); AN(LOG.fo); @@ -170,7 +155,6 @@ main(int argc, char * const *argv) LOG.fo = stdout; vut->idle_f = flushout; - VUT_Signal(vut_sighandler); VUT_Setup(vut); (void)VUT_Main(vut); VUT_Fini(&vut); diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c index f3ed20a78..ca470e2a4 100644 --- a/bin/varnishncsa/varnishncsa.c +++ b/bin/varnishncsa/varnishncsa.c @@ -1050,21 +1050,6 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], return (0); } -static int v_matchproto_(VUT_cb_f) -sighup(struct VUT *v) -{ - assert(v == vut); - return (1); -} - -static void -vut_sighandler(int sig) -{ - - if (vut != NULL) - VUT_Signaled(vut, sig); -} - static char * read_format(const char *formatfile) { @@ -1175,7 +1160,6 @@ main(int argc, char * const *argv) /* Setup output */ vut->dispatch_f = dispatch_f; vut->dispatch_priv = NULL; - vut->sighup_f = sighup; if (CTX.w_arg) { openout(CTX.a_opt); AN(CTX.fo); @@ -1185,7 +1169,6 @@ main(int argc, char * const *argv) CTX.fo = stdout; vut->idle_f = flushout; - VUT_Signal(vut_sighandler); VUT_Setup(vut); (void)VUT_Main(vut); VUT_Fini(&vut); diff --git a/bin/varnishtop/varnishtop.c b/bin/varnishtop/varnishtop.c index f91e3cd01..1fc844f27 100644 --- a/bin/varnishtop/varnishtop.c +++ b/bin/varnishtop/varnishtop.c @@ -182,22 +182,6 @@ accumulate(struct VSL_data *vsl, struct VSL_transaction * const pt[], return (0); } -static int v_matchproto_(VUT_cb_f) -sighup(struct VUT *v) -{ - assert(v == vut); - quit = 1; - return (1); -} - -static void -vut_sighandler(int sig) -{ - - if (vut != NULL) - VUT_Signaled(vut, sig); -} - static void update(int p) { @@ -273,7 +257,7 @@ do_curses(void *arg) (void)curs_set(0); AC(erase()); timeout(1000); - while (!quit) { + while (!quit && !vut->last_sighup) { AZ(pthread_mutex_lock(&mtx)); update(period); AZ(pthread_mutex_unlock(&mtx)); @@ -364,7 +348,6 @@ main(int argc, char **argv) if (optind != argc) VUT_Usage(vut, &vopt_spec, 1); - VUT_Signal(vut_sighandler); VUT_Setup(vut); if (vut->vsm) ident = VSM_Dup(vut->vsm, "Arg", "-i"); @@ -372,7 +355,6 @@ main(int argc, char **argv) ident = strdup(""); vut->dispatch_f = accumulate; vut->dispatch_priv = NULL; - vut->sighup_f = sighup; if (once) { (void)VUT_Main(vut); dump(); diff --git a/include/Makefile.am b/include/Makefile.am index 1479a6384..50142a64b 100644 --- a/include/Makefile.am +++ b/include/Makefile.am @@ -39,13 +39,15 @@ nobase_pkginclude_HEADERS = \ tbl/vhp_static.h \ tbl/vrt_stv_var.h \ tbl/vsc_levels.h \ + tbl/vsig_list.h \ tbl/vsl_tags.h \ tbl/vsl_tags_http.h \ tbl/waiters.h \ - vapi/vsm.h \ vapi/vsc.h \ + vapi/vsig.h \ vapi/vsl.h \ vapi/vsl_int.h \ + vapi/vsm.h \ vapi/voptget.h \ vapi/vapi_options.h \ vcli.h \ diff --git a/include/tbl/vsig_list.h b/include/tbl/vsig_list.h new file mode 100644 index 000000000..f7a05d60c --- /dev/null +++ b/include/tbl/vsig_list.h @@ -0,0 +1,40 @@ +/*- + * Copyright (c) 2018 Varnish Software AS + * All rights reserved. + * + * Author: Poul-Henning Kamp + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * NB: Changing the list of signals requires updating libvarnishapi.map + * + */ + +/*lint -save -e525 -e539 */ + +VSIG_SIGNAL(INT, int) +VSIG_SIGNAL(HUP, hup) +VSIG_SIGNAL(TERM, term) +VSIG_SIGNAL(USR1, usr1) +#undef VSIG_SIGNAL + +/*lint -restore */ diff --git a/include/vapi/vsig.h b/include/vapi/vsig.h new file mode 100644 index 000000000..1556ae779 --- /dev/null +++ b/include/vapi/vsig.h @@ -0,0 +1,40 @@ +/*- + * Copyright (c) 2018 Varnish Software AS + * All rights reserved. + * + * Author: Poul-Henning Kamp + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + */ + +#ifndef VAPI_VSIG_H_INCLUDED +#define VAPI_VSIG_H_INCLUDED + +#define VSIG_SIGNAL(UPPER, lower) \ + extern sig_atomic_t VSIG_##lower; \ + void VSIG_Got_##lower(int sig); \ + void VSIG_Arm_##lower(void); + +#include "tbl/vsig_list.h" + +#endif /* VAPI_VSC_H_INCLUDED */ diff --git a/include/vut.h b/include/vut.h index 57eb7c02a..8a35b0267 100644 --- a/include/vut.h +++ b/include/vut.h @@ -56,9 +56,9 @@ struct VUT { struct VSL_data *vsl; struct vsm *vsm; struct VSLQ *vslq; - int sighup; - int sigint; - int sigusr1; + + sig_atomic_t last_sighup; + sig_atomic_t last_sigusr1; /* Callback functions */ VUT_cb_f *idle_f; diff --git a/lib/libvarnishapi/Makefile.am b/lib/libvarnishapi/Makefile.am index 5e13de019..77e96d666 100644 --- a/lib/libvarnishapi/Makefile.am +++ b/lib/libvarnishapi/Makefile.am @@ -31,18 +31,19 @@ libvarnishapi_la_SOURCES = \ ../libvarnish/vtim.c \ ../libvarnish/vnum.c \ ../libvarnish/vsha256.c \ - vsm.c \ + vsc.c \ + vsig.c \ + vsl.c \ vsl_arg.c \ vsl_cursor.c \ vsl_dispatch.c \ vsl_query.c \ - vsl.c \ - vsc.c \ + vsm.c \ vut.c \ vxp.c \ - vxp_parse.c \ - vxp_lexer.c \ vxp_fixed_token.c \ + vxp_lexer.c \ + vxp_parse.c \ libvarnishapi.map if ! HAVE_DAEMON diff --git a/lib/libvarnishapi/libvarnishapi.map b/lib/libvarnishapi/libvarnishapi.map index 1c8a6ff2b..addfb3077 100644 --- a/lib/libvarnishapi/libvarnishapi.map +++ b/lib/libvarnishapi/libvarnishapi.map @@ -148,3 +148,22 @@ LIBVARNISHAPI_2.1 { local: *; }; + +LIBVARNISHAPI_2.2 { + global: + # vsig.c + VSIG_int; + VSIG_Got_int; + VSIG_Arm_int; + VSIG_hup; + VSIG_Got_hup; + VSIG_Arm_hup; + VSIG_term; + VSIG_Got_term; + VSIG_Arm_term; + VSIG_usr1; + VSIG_Got_usr1; + VSIG_Arm_usr1; + local: + *; +}; diff --git a/lib/libvarnishapi/vsig.c b/lib/libvarnishapi/vsig.c new file mode 100644 index 000000000..924973e3c --- /dev/null +++ b/lib/libvarnishapi/vsig.c @@ -0,0 +1,56 @@ +/*- + * Copyright (c) 2018 Varnish Software AS + * All rights reserved. + * + * Author: Poul-Henning Kamp + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * Common functions for the utilities + */ + +#include "config.h" + +#include + +#include "vapi/vsig.h" + +#define VSIG_SIGNAL(UPPER, lower) \ + sig_atomic_t VSIG_##lower; +#include "tbl/vsig_list.h" + +#define VSIG_SIGNAL(UPPER, lower) \ + void \ + VSIG_Got_##lower(int sig) \ + { \ + (void)sig; \ + VSIG_##lower++; \ + } +#include "tbl/vsig_list.h" + +#define VSIG_SIGNAL(UPPER, lower) \ + void \ + VSIG_Arm_##lower(void) \ + { \ + (void)signal(SIG##UPPER, VSIG_Got_##lower); \ + } +#include "tbl/vsig_list.h" diff --git a/lib/libvarnishapi/vut.c b/lib/libvarnishapi/vut.c index 9c767bfe7..61c200147 100644 --- a/lib/libvarnishapi/vut.c +++ b/lib/libvarnishapi/vut.c @@ -54,6 +54,7 @@ #include "vut.h" #include "vapi/voptget.h" +#include "vapi/vsig.h" static int vut_synopsis(const struct vopt_spec *); static int vut_options(const struct vopt_spec *); @@ -206,6 +207,11 @@ VUT_Init(const char *progname, int argc, char * const *argv, AN(argv); AN(voc); + VSIG_Arm_hup(); + VSIG_Arm_int(); + VSIG_Arm_term(); + VSIG_Arm_usr1(); + ALLOC_OBJ(vut, VUT_MAGIC); AN(vut); @@ -243,9 +249,9 @@ VUT_Signaled(struct VUT *vut, int sig) { CHECK_OBJ_NOTNULL(vut, VUT_MAGIC); - vut->sighup |= (int)(sig == SIGHUP); - vut->sigint |= (int)(sig == SIGINT || sig == SIGTERM); - vut->sigusr1 |= (int)(sig == SIGUSR1); +#define VSIG_SIGNAL(UPPER, lower) \ + VSIG_##lower += (int)(sig == SIG##UPPER); +#include "tbl/vsig_list.h" } void @@ -353,18 +359,21 @@ VUT_Main(struct VUT *vut) CHECK_OBJ_NOTNULL(vut, VUT_MAGIC); AN(vut->vslq); - while (!vut->sigint) { - if (vut->sighup && vut->sighup_f) { + while (!VSIG_int && !VSIG_term) { + if (VSIG_hup != vut->last_sighup) { /* sighup callback */ - vut->sighup = 0; - i = vut->sighup_f(vut); + vut->last_sighup = VSIG_hup; + if (vut->sighup_f != NULL) + i = vut->sighup_f(vut); + else + i = 1; if (i) break; } - if (vut->sigusr1) { + if (VSIG_usr1 != vut->last_sigusr1) { /* Flush and report any incomplete records */ - vut->sigusr1 = 0; + vut->last_sigusr1 = VSIG_usr1; (void)VSLQ_Flush(vut->vslq, vut_dispatch, vut); } @@ -412,7 +421,7 @@ VUT_Main(struct VUT *vut) do i = VSLQ_Dispatch(vut->vslq, vut_dispatch, vut); - while (i == vsl_more && !vut->sighup && !vut->sigusr1); + while (i == vsl_more && !VSIG_hup && !VSIG_usr1); if (i == vsl_more) continue; From dridi at varni.sh Mon Nov 12 14:06:20 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 12 Nov 2018 15:06:20 +0100 Subject: [master] 55995ab0f Renovate libvarnisapi signal handling. In-Reply-To: <20181112101413.CE27B97793@lists.varnish-cache.org> References: <20181112101413.CE27B97793@lists.varnish-cache.org> Message-ID: > The old VUT_Signal()/VUT_Signaled() API is retained and should be > deprecated in a future version. Should we have a dummy vut akin to vmod-debug to extend our coverage of libvarnishapi? I haven't reviewed this yet, but that would allow test coverage of this kind of changes and help a lot when we consider bringing the improvement to the 6.0 branch. (And also make sure we don't break the existing feature while it's still present and then deprecated.) My intuition tells me that this one will be a no-no and may even require an soname bump. Dridi From nils.goroll at uplex.de Mon Nov 12 15:24:12 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 12 Nov 2018 15:24:12 +0000 (UTC) Subject: [master] 1babec2d7 polish: avoid duplicate zero initialization Message-ID: <20181112152412.2BACBA41BB@lists.varnish-cache.org> commit 1babec2d7ce65d05e2d1b35d9c9f6f5845732f66 Author: Nils Goroll Date: Mon Nov 12 16:19:35 2018 +0100 polish: avoid duplicate zero initialization ALLOC_OBJ calls memset(0) already diff --git a/lib/libvmod_blob/vmod_blob.c b/lib/libvmod_blob/vmod_blob.c index 1b675abb2..06f29f664 100644 --- a/lib/libvmod_blob/vmod_blob.c +++ b/lib/libvmod_blob/vmod_blob.c @@ -199,15 +199,12 @@ vmod_blob__init(VRT_CTX, struct vmod_blob_blob **blobp, const char *vcl_name, ALLOC_OBJ(b, VMOD_BLOB_MAGIC); AN(b); *blobp = b; - b->blob.free = NULL; AZ(pthread_mutex_init(&b->lock, NULL)); len = decode_l(dec, strings); - if (len == 0) { - b->blob.len = 0; - b->blob.priv = NULL; + if (len == 0) return; - } + assert(len > 0); b->blob.priv = malloc(len); From nils.goroll at uplex.de Mon Nov 12 16:40:16 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 12 Nov 2018 16:40:16 +0000 (UTC) Subject: [master] 01198002d polish blob.same and blob.equal Message-ID: <20181112164016.98C39A5B53@lists.varnish-cache.org> commit 01198002d79fd79218d515bae94e99aa210c8004 Author: Nils Goroll Date: Mon Nov 12 17:37:24 2018 +0100 polish blob.same and blob.equal If two blob pointers compare equal, the blobs are both the same and equal - both being NULL is implied. diff --git a/lib/libvmod_blob/vmod_blob.c b/lib/libvmod_blob/vmod_blob.c index 06f29f664..28e486856 100644 --- a/lib/libvmod_blob/vmod_blob.c +++ b/lib/libvmod_blob/vmod_blob.c @@ -483,7 +483,7 @@ vmod_same(VRT_CTX, VCL_BLOB b1, VCL_BLOB b2) { (void) ctx; - if (b1 == NULL && b2 == NULL) + if (b1 == b2) return 1; if (b1 == NULL || b2 == NULL) return 0; @@ -495,7 +495,7 @@ vmod_equal(VRT_CTX, VCL_BLOB b1, VCL_BLOB b2) { (void) ctx; - if (b1 == NULL && b2 == NULL) + if (b1 == b2) return 1; if (b1 == NULL || b2 == NULL) return 0; From nils.goroll at uplex.de Tue Nov 13 11:38:11 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 13 Nov 2018 11:38:11 +0000 (UTC) Subject: [master] c59c73b1f Improve argstruct test coverage Message-ID: <20181113113811.7F3F0624C9@lists.varnish-cache.org> commit c59c73b1f1a01622402d72cfffccbee15fc1bedb Author: Nils Goroll Date: Tue Nov 13 12:35:53 2018 +0100 Improve argstruct test coverage Related to #2810 diff --git a/bin/varnishtest/tests/m00019.vtc b/bin/varnishtest/tests/m00019.vtc index b2c9d666f..edc0e3ab7 100644 --- a/bin/varnishtest/tests/m00019.vtc +++ b/bin/varnishtest/tests/m00019.vtc @@ -15,6 +15,10 @@ varnish v1 -vcl+backend { new obj3 = debug.obj(string="s_two", number=two); new obj4 = debug.obj(number=three, string="s_three"); new obj5 = debug.obj(number=three); + + new oo0 = debug.obj_opt(); + new oo1 = debug.obj_opt(s="string"); + new oo2 = debug.obj_opt(b=true); } sub vcl_deliver { @@ -32,6 +36,10 @@ varnish v1 -vcl+backend { set resp.http.obj3 = obj3.string() + ", " + obj3.number(); set resp.http.obj4 = obj4.string() + ", " + obj4.number(); set resp.http.obj5 = obj5.string() + ", " + obj5.number(); + + set resp.http.oo0 = oo0.meth_opt(s="s1"); + set resp.http.oo1 = oo1.meth_opt(b=false); + set resp.http.oo2 = oo2.meth_opt(); } } -start @@ -53,6 +61,11 @@ client c1 { expect resp.http.obj3 == "s_two, two" expect resp.http.obj4 == "s_three, three" expect resp.http.obj5 == "default, three" + + expect resp.http.oo0 == "obj oo0 obj_s *undef* obj_b *undef* met_s s1 met_b *undef*" + expect resp.http.oo1 == "obj oo1 obj_s string obj_b *undef* met_s *undef* met_b false" + expect resp.http.oo2 == "obj oo2 obj_s *undef* obj_b true met_s *undef* met_b *undef*" + } -run delay .1 diff --git a/lib/libvmod_debug/vmod.vcc b/lib/libvmod_debug/vmod.vcc index ad24840da..3e0cc4faa 100644 --- a/lib/libvmod_debug/vmod.vcc +++ b/lib/libvmod_debug/vmod.vcc @@ -212,3 +212,13 @@ to seconds - iow the value given as seconds is actually the duration in nanoseconds. For comparable results, a higher size run should called first and discarded. + +$Object obj_opt(PRIV_CALL, PRIV_VCL, PRIV_TASK, + [STRING s], [BOOL b]) + +Test object constructor with all the fancy stuff. + +$Method STRING .meth_opt(PRIV_CALL, PRIV_VCL, PRIV_TASK, + [STRING s], [BOOL b]) + +Test object method with all the fancy stuff. diff --git a/lib/libvmod_debug/vmod_debug_obj.c b/lib/libvmod_debug/vmod_debug_obj.c index fcf74a740..9038bcb81 100644 --- a/lib/libvmod_debug/vmod_debug_obj.c +++ b/lib/libvmod_debug/vmod_debug_obj.c @@ -152,3 +152,94 @@ xyzzy_obj_test_priv_top(VRT_CTX, (void)o; return (xyzzy_test_priv_top(ctx, priv, s)); } + +/* ---------------------------------------------------------------------------- + * obj_opt (optional arguments and privs) + */ +struct xyzzy_debug_obj_opt { + unsigned magic; +#define VMOD_DEBUG_OBJ_OPT_MAGIC 0xccbd9b78 + char *name; + struct xyzzy_obj_opt_meth_opt_arg args; + void *freeptr; +}; + +VCL_VOID v_matchproto_() +xyzzy_obj_opt__init(VRT_CTX, + struct xyzzy_debug_obj_opt **op, const char *vcl_name, + struct xyzzy_obj_opt__init_arg *args) +{ + struct xyzzy_debug_obj_opt *o; + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + AN(args); + + AN(args->arg1); // priv_call + AN(args->arg2); // priv_vcl + AN(args->arg3); // priv_task + assert(args->arg1 != args->arg2); + assert(args->arg2 != args->arg3); + + if (args->valid_s) + AN(args->s); + + AN(op); + AZ(*op); + ALLOC_OBJ(o, VMOD_DEBUG_OBJ_OPT_MAGIC); + AN(o); + *op = o; + REPLACE(o->name, vcl_name); + memcpy(&o->args, args, sizeof o->args); + if (args->valid_s) { + REPLACE(o->freeptr, args->s); + o->args.s = o->freeptr; + } +} + +VCL_VOID v_matchproto_() +xyzzy_obj_opt__fini(struct xyzzy_debug_obj_opt **op) +{ + struct xyzzy_debug_obj_opt *o; + + AN(op); + if (*op == NULL) + return; /* init has failed */ + + TAKE_OBJ_NOTNULL(o, op, VMOD_DEBUG_OBJ_OPT_MAGIC); + + REPLACE(o->name, NULL); + if (o->freeptr) { + AN(o->args.valid_s); + REPLACE(o->freeptr, NULL); + } + FREE_OBJ(o); + AZ(o); +} + +VCL_STRING v_matchproto_() +xyzzy_obj_opt_meth_opt(VRT_CTX, + struct xyzzy_debug_obj_opt *o, + struct xyzzy_obj_opt_meth_opt_arg *args) +{ + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + CHECK_OBJ_NOTNULL(o, VMOD_DEBUG_OBJ_OPT_MAGIC); + + AN(args); + AN(args->arg1); // priv_call + AN(args->arg2); // priv_vcl + AN(args->arg3); // priv_task + assert(args->arg1 != args->arg2); + assert(args->arg2 != args->arg3); + + return (VRT_CollectString(ctx, + "obj ", o->name, + " obj_s ", (o->args.valid_s ? o->args.s : "*undef*"), + " obj_b ", (o->args.valid_b + ? (o->args.b ? "true" : "false" ) + : "*undef*"), + " met_s ", (args->valid_s ? args->s : "*undef*"), + " met_b ", (args->valid_b + ? (args->b ? "true" : "false" ) + : "*undef*"), + vrt_magic_string_end)); +} From nils.goroll at uplex.de Tue Nov 13 17:00:19 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 13 Nov 2018 17:00:19 +0000 (UTC) Subject: [master] a50d90d53 improve busyobj panic Message-ID: <20181113170019.CA3E094F02@lists.varnish-cache.org> commit a50d90d53b1080ca0a69f853570b9658f5c54698 Author: Nils Goroll Date: Tue Nov 13 17:43:41 2018 +0100 improve busyobj panic For backend requests, we do not have the thread local storage of the client thread available, so trq.req will normally be (nil). Dump the busyobj back pointers to req/sess/wrk to maximize the usefulness of panics. Also reorder the panic by struct busyobj and add some more members which may or may not be useful (but when we find out, it is likely too late). Also move the vmod list dump to the top level. Noticed when trying to understand if #2667 was also a stack overflow in the context of #2817 diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c index be6dd70f6..74c1973dd 100644 --- a/bin/varnishd/cache/cache_panic.c +++ b/bin/varnishd/cache/cache_panic.c @@ -60,6 +60,7 @@ static struct vsb pan_vsb_storage, *pan_vsb; static pthread_mutex_t panicstr_mtx; static void pan_sess(struct vsb *, const struct sess *); +static void pan_req(struct vsb *, const struct req *); /*--------------------------------------------------------------------*/ @@ -399,8 +400,39 @@ pan_busyobj(struct vsb *vsb, const struct busyobj *bo) return; VSB_indent(vsb, 2); PAN_CheckMagic(vsb, bo, BUSYOBJ_MAGIC); + VSB_printf(vsb, "end = %p,\n", bo->end); + VSB_printf(vsb, "retries = %d,\n", bo->retries); + + if (bo->req != NULL) + pan_req(vsb, bo->req); + if (bo->sp != NULL) + pan_sess(vsb, bo->sp); + if (bo->wrk != NULL) + pan_wrk(vsb, bo->wrk); + + if (bo->vfc != NULL) + pan_vfp(vsb, bo->vfc); + if (bo->filter_list != NULL) + VSB_printf(vsb, "filter_list = \"%s\",\n", bo->filter_list); + pan_ws(vsb, bo->ws); - VSB_printf(vsb, "retries = %d, ", bo->retries); + VSB_printf(vsb, "ws_bo = %p,\n", (void *)bo->ws_bo); + + // bereq0 left out + if (bo->bereq != NULL && bo->bereq->ws != NULL) + pan_http(vsb, "bereq", bo->bereq); + if (bo->beresp != NULL && bo->beresp->ws != NULL) + pan_http(vsb, "beresp", bo->beresp); + if (bo->stale_oc) + pan_objcore(vsb, "stale_oc", bo->stale_oc); + if (bo->fetch_objcore) + pan_objcore(vsb, "fetch", bo->fetch_objcore); + + if (VALID_OBJ(bo->htc, HTTP_CONN_MAGIC)) + pan_htc(vsb, bo->htc); + + // fetch_task left out + VSB_printf(vsb, "flags = {"); p = ""; /*lint -save -esym(438,p) -e539 */ @@ -410,27 +442,14 @@ pan_busyobj(struct vsb *vsb, const struct busyobj *bo) /*lint -restore */ VSB_printf(vsb, "},\n"); - if (VALID_OBJ(bo->htc, HTTP_CONN_MAGIC)) - pan_htc(vsb, bo->htc); - - if (bo->vfc) - pan_vfp(vsb, bo->vfc); + // timeouts/timers/acct/storage left out VDI_Panic(bo->director_req, vsb, "director_req"); if (bo->director_resp == bo->director_req) VSB_printf(vsb, "director_resp = director_req,\n"); else VDI_Panic(bo->director_resp, vsb, "director_resp"); - if (bo->bereq != NULL && bo->bereq->ws != NULL) - pan_http(vsb, "bereq", bo->bereq); - if (bo->beresp != NULL && bo->beresp->ws != NULL) - pan_http(vsb, "beresp", bo->beresp); - if (bo->fetch_objcore) - pan_objcore(vsb, "fetch", bo->fetch_objcore); - if (bo->stale_oc) - pan_objcore(vsb, "ims", bo->stale_oc); VCL_Panic(vsb, bo->vcl); - VMOD_Panic(vsb); VSB_indent(vsb, -2); VSB_printf(vsb, "},\n"); } @@ -495,7 +514,6 @@ pan_req(struct vsb *vsb, const struct req *req) pan_http(vsb, "resp", req->resp); VCL_Panic(vsb, req->vcl); - VMOD_Panic(vsb); if (req->body_oc != NULL) pan_objcore(vsb, "BODY", req->body_oc); @@ -683,6 +701,7 @@ pan_ic(const char *func, const char *file, int line, const char *cond, pan_busyobj(pan_vsb, bo); if (bo != NULL) VSL_Flush(bo->vsl, 0); + VMOD_Panic(pan_vsb); } else { VSB_printf(pan_vsb, "Feature short panic supressed details.\n"); From fgsch at lodoss.net Tue Nov 13 21:49:09 2018 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Tue, 13 Nov 2018 21:49:09 +0000 (UTC) Subject: [master] 1d327ef32 Make VSM_Attach interruptible Message-ID: <20181113214909.D6E87A0D83@lists.varnish-cache.org> commit 1d327ef32b2a452d9ae782839f48617eb1bb4db1 Author: Federico G. Schwindt Date: Tue Nov 13 12:59:35 2018 +0900 Make VSM_Attach interruptible Fixes #2657. diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c index d909ebd24..b7b82cbf5 100644 --- a/bin/varnishadm/varnishadm.c +++ b/bin/varnishadm/varnishadm.c @@ -59,6 +59,7 @@ #include "vdef.h" +#include "vapi/vsig.h" #include "vapi/vsm.h" #include "vas.h" #include "vcli.h" @@ -472,6 +473,9 @@ main(int argc, char * const *argv) argc -= optind; argv += optind; + VSIG_Arm_int(); + VSIG_Arm_term(); + if (T_arg != NULL) { if (n_arg != NULL) usage(1); diff --git a/bin/varnishtest/tests/u00003.vtc b/bin/varnishtest/tests/u00003.vtc index bc86232bb..30dc846e1 100644 --- a/bin/varnishtest/tests/u00003.vtc +++ b/bin/varnishtest/tests/u00003.vtc @@ -154,3 +154,8 @@ process p1 -expect-text 1 0 {/1?foo=bar HTTP/1.1" 200 100 "-" "-"} process p1 -expect-text 1 0 { - user [} process p1 -expect-text 2 0 {/1?foo=bar HTTP/1.1" 404 248 "-" "-"} process p1 -expect-text 3 0 {/2 HTTP/1.1" - - "-" "-"} + +process p2 {varnishncsa -t 5 -n nonexistent} -start +delay 1 +process p2 -expect-exit 1 -kill INT -wait +shell {grep -q "VSM: Attach interrupted" ${p2_err}} diff --git a/lib/libvarnishapi/vsm.c b/lib/libvarnishapi/vsm.c index 7380bbae3..f10c77e36 100644 --- a/lib/libvarnishapi/vsm.c +++ b/lib/libvarnishapi/vsm.c @@ -55,6 +55,7 @@ #include "vqueue.h" #include "vtim.h" +#include "vapi/vsig.h" #include "vapi/vsm.h" #ifndef MAP_HASSEMAPHORE @@ -676,7 +677,7 @@ VSM_Attach(struct vsm *vd, int progress) } AZ(vd->attached); - while (1) { + while (!VSIG_int && !VSIG_term) { u = VSM_Status(vd); VSM_ResetError(vd); if (u & VSM_MGT_RUNNING) { @@ -695,6 +696,7 @@ VSM_Attach(struct vsm *vd, int progress) (void)write(progress, ".", 1); VTIM_sleep(.25); } + return (vsm_diag(vd, "Attach interrupted")); } /*--------------------------------------------------------------------*/ From fgsch at lodoss.net Tue Nov 13 21:56:06 2018 From: fgsch at lodoss.net (Federico G. Schwindt) Date: Tue, 13 Nov 2018 21:56:06 +0000 (UTC) Subject: [master] 3e3990d7d Add missing include Message-ID: <20181113215606.E9A7DA1110@lists.varnish-cache.org> commit 3e3990d7d6feb4ad5315634128f6244788e9fa03 Author: Federico G. Schwindt Date: Wed Nov 14 06:55:09 2018 +0900 Add missing include diff --git a/bin/varnishadm/varnishadm.c b/bin/varnishadm/varnishadm.c index b7b82cbf5..16393ecde 100644 --- a/bin/varnishadm/varnishadm.c +++ b/bin/varnishadm/varnishadm.c @@ -52,6 +52,7 @@ #include #include #include +#include #include #include #include From phk at FreeBSD.org Wed Nov 14 08:36:12 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 14 Nov 2018 08:36:12 +0000 (UTC) Subject: [master] 57cd74cf5 Make FlexeLint a little bit happier Message-ID: <20181114083612.BFA21AEF00@lists.varnish-cache.org> commit 57cd74cf51b63a54df68600a573eb0e20e2a6c7a Author: Poul-Henning Kamp Date: Wed Nov 14 08:35:15 2018 +0000 Make FlexeLint a little bit happier diff --git a/lib/libvmod_debug/vmod_debug_obj.c b/lib/libvmod_debug/vmod_debug_obj.c index 9038bcb81..3ecf33d0c 100644 --- a/lib/libvmod_debug/vmod_debug_obj.c +++ b/lib/libvmod_debug/vmod_debug_obj.c @@ -183,6 +183,9 @@ xyzzy_obj_opt__init(VRT_CTX, if (args->valid_s) AN(args->s); + if (args->valid_b) + AN(args->b); + AN(op); AZ(*op); ALLOC_OBJ(o, VMOD_DEBUG_OBJ_OPT_MAGIC); From nils.goroll at uplex.de Wed Nov 14 10:35:48 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 14 Nov 2018 11:35:48 +0100 Subject: [master] 57cd74cf5 Make FlexeLint a little bit happier In-Reply-To: <20181114083612.BFA21AEF00@lists.varnish-cache.org> References: <20181114083612.BFA21AEF00@lists.varnish-cache.org> Message-ID: On 14/11/2018 09:36, Poul-Henning Kamp wrote: > + if (args->valid_b) > + AN(args->b); this is wrong semantically: If we have a boolean argument, that does not imply that the boolean is true. In contrast, if we have a string argument, it needs to be a non-NULL pointer -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg tel +49 40 28805731 mob +49 170 2723133 fax +49 40 42949753 xmpp://slink at jabber.int.uplex.de/ http://uplex.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From phk at FreeBSD.org Wed Nov 14 11:03:14 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 14 Nov 2018 11:03:14 +0000 (UTC) Subject: [master] 4924c7908 Try to straighten out resp.body processing. Message-ID: <20181114110314.3C759B1C5C@lists.varnish-cache.org> commit 4924c7908f5cc7b726d1f3c12804ab14fdd5b8db Author: Poul-Henning Kamp Date: Wed Nov 14 11:01:58 2018 +0000 Try to straighten out resp.body processing. diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index 961a3f953..74515e7b6 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -349,73 +349,66 @@ cnt_transmit(struct worker *wrk, struct req *req) CHECK_OBJ_NOTNULL(req->transport, TRANSPORT_MAGIC); CHECK_OBJ_NOTNULL(req->objcore, OBJCORE_MAGIC); AZ(req->stale_oc); + AZ(req->res_mode); - /* Grab a ref to the bo if there is one */ + /* Grab a ref to the bo if there is one (=streaming) */ boc = HSH_RefBoc(req->objcore); - clval = http_GetContentLength(req->resp); + /* RFC 7230, 3.3.3 */ + status = http_GetStatus(req->resp); + head = !strcmp(req->http0->hd[HTTP_HDR_METHOD].b, "HEAD"); + err = 0; + if (boc != NULL) req->resp_len = clval; else req->resp_len = ObjGetLen(req->wrk, req->objcore); - req->res_mode = 0; - - /* RFC 7230, 3.3.3 */ - status = http_GetStatus(req->resp); - head = !strcmp(req->http0->hd[HTTP_HDR_METHOD].b, "HEAD"); - if (head) { - if (req->objcore->flags & OC_F_PASS) - sendbody = -1; - else - sendbody = 0; - } else if (status < 200 || status == 204 || status == 304) { - req->resp_len = -1; - sendbody = 0; - } else { + if (head || status < 200 || status == 204 || status == 304) + sendbody = 0; /* rfc7230,l,1748,1752 */ + else sendbody = 1; - } - err = 0; - if (sendbody >= 0) { - if (!req->disable_esi && req->resp_len != 0 && - ObjHasAttr(wrk, req->objcore, OA_ESIDATA) && - VDP_Push(req, &VDP_esi, NULL) < 0) - err++; - - if (cache_param->http_gzip_support && - ObjCheckFlag(req->wrk, req->objcore, OF_GZIPED) && - !RFC2616_Req_Gzip(req->http) && - VDP_Push(req, &VDP_gunzip, NULL) < 0) - err++; - - if (cache_param->http_range_support && - http_IsStatus(req->resp, 200)) { - http_ForceHeader(req->resp, H_Accept_Ranges, "bytes"); - if (sendbody && http_GetHdr(req->http, H_Range, &r)) - VRG_dorange(req, r); - } - } + if (!req->disable_esi && req->resp_len != 0 && + ObjHasAttr(wrk, req->objcore, OA_ESIDATA) && + VDP_Push(req, &VDP_esi, NULL) < 0) + err++; - if (sendbody < 0 || head) { - /* Don't touch HEAD C-L */ - sendbody = 0; - } else if (clval >= 0 && clval == req->resp_len) { - /* Reuse C-L header */ - } else { - http_Unset(req->resp, H_Content_Length); - if (req->resp_len >= 0 && sendbody) - http_PrintfHeader(req->resp, - "Content-Length: %jd", req->resp_len); + if (cache_param->http_gzip_support && + ObjCheckFlag(req->wrk, req->objcore, OF_GZIPED) && + !RFC2616_Req_Gzip(req->http) && + VDP_Push(req, &VDP_gunzip, NULL) < 0) + err++; + + if (cache_param->http_range_support && status == 200) { + http_ForceHeader(req->resp, H_Accept_Ranges, "bytes"); + if (http_GetHdr(req->http, H_Range, &r)) + VRG_dorange(req, r); } - if (err == 0) { + if (err) { + VSLb(req->vsl, SLT_Error, "Failure to push processors"); + req->doclose = SC_OVERLOAD; + } else { + if (status < 200 || status == 204) { + // rfc7230,l,1691,1695 + http_Unset(req->resp, H_Content_Length); + } else if (status == 304) { + // rfc7230,l,1675,1677 + http_Unset(req->resp, H_Content_Length); + } else if (clval >= 0 && clval == req->resp_len) { + /* Reuse C-L header */ + } else if (head && req->objcore->flags & OC_F_PASS) { + /* Don't touch C-L header */ + } else { + http_Unset(req->resp, H_Content_Length); + if (req->resp_len >= 0) + http_PrintfHeader(req->resp, + "Content-Length: %jd", req->resp_len); + } if (req->resp_len == 0) sendbody = 0; req->transport->deliver(req, boc, sendbody); - } else { - VSLb(req->vsl, SLT_Error, "Failure to push processors"); - req->doclose = SC_OVERLOAD; } VSLb_ts_req(req, "Resp", W_TIM_real(wrk)); @@ -436,6 +429,7 @@ cnt_transmit(struct worker *wrk, struct req *req) (void)HSH_DerefObjCore(wrk, &req->objcore, HSH_RUSH_POLICY); http_Teardown(req->resp); + req->res_mode = 0; return (REQ_FSM_DONE); } diff --git a/bin/varnishtest/tests/b00065.vtc b/bin/varnishtest/tests/b00065.vtc index 5eb86d8d9..daf030659 100644 --- a/bin/varnishtest/tests/b00065.vtc +++ b/bin/varnishtest/tests/b00065.vtc @@ -1,20 +1,33 @@ -varnishtest "Test that C-L on HEAD request aren't nuked" +varnishtest "Check that HEAD+pass returns Content-Length if backend provides it" server s1 { + rxreq + expect req.method == "GET" + txresp -bodylen 5 rxreq expect req.method == "HEAD" - txresp -nolen -hdr "content-length: 17" + txresp -nolen -hdr "Content-Length: 6" } -start varnish v1 -vcl+backend { - sub vcl_backend_fetch { - set bereq.method = "HEAD"; + sub vcl_recv { + if (req.url == "/2") { + return (pass); + } + } + sub vcl_backend_response { + set beresp.do_stream = false; } } -start client c1 { - txreq -req HEAD -hdr "connection: close" - rxresphdrs - expect resp.http.content-length == 17 - expect_close + txreq -req HEAD + rxresp -no_obj + expect resp.http.content-length == 5 +} -run + +client c1 { + txreq -req HEAD -url /2 + rxresp -no_obj + expect resp.http.content-length == 6 } -run diff --git a/bin/varnishtest/tests/r00730.vtc b/bin/varnishtest/tests/r00730.vtc deleted file mode 100644 index e28c8c912..000000000 --- a/bin/varnishtest/tests/r00730.vtc +++ /dev/null @@ -1,31 +0,0 @@ -varnishtest "Check that HEAD returns Content-Length if backend provides it" - -server s1 { - rxreq - txresp -bodylen 5 - rxreq - txresp -nolen -hdr "Content-Length: 6" -} -start - -varnish v1 -vcl+backend { - sub vcl_recv { - if (req.url == "/2") { - return (pass); - } - } - sub vcl_backend_response { - set beresp.do_stream = false; - } -} -start - -client c1 { - txreq -req HEAD - rxresp -no_obj - expect resp.http.content-length == 5 -} -run - -client c1 { - txreq -req HEAD -url /2 - rxresp -no_obj - expect resp.http.content-length == 6 -} -run From phk at FreeBSD.org Wed Nov 14 11:06:08 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Wed, 14 Nov 2018 11:06:08 +0000 (UTC) Subject: [master] 77d4c56c0 Get the RFC comment right Message-ID: <20181114110608.DB955B1E8C@lists.varnish-cache.org> commit 77d4c56c0b7c2d35db2a7c2eeace449fa16a6874 Author: Poul-Henning Kamp Date: Wed Nov 14 11:05:50 2018 +0000 Get the RFC comment right diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index 74515e7b6..e8c2e542b 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -364,10 +364,12 @@ cnt_transmit(struct worker *wrk, struct req *req) else req->resp_len = ObjGetLen(req->wrk, req->objcore); - if (head || status < 200 || status == 204 || status == 304) - sendbody = 0; /* rfc7230,l,1748,1752 */ - else + if (head || status < 200 || status == 204 || status == 304) { + // rfc7230,l,1748,1752 + sendbody = 0; + } else { sendbody = 1; + } if (!req->disable_esi && req->resp_len != 0 && ObjHasAttr(wrk, req->objcore, OA_ESIDATA) && From nils.goroll at uplex.de Wed Nov 14 11:46:08 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 14 Nov 2018 11:46:08 +0000 (UTC) Subject: [master] b2d2b6fe9 Revert "Make FlexeLint a little bit happier" Message-ID: <20181114114608.BE6BCB2BED@lists.varnish-cache.org> commit b2d2b6fe93b694ed5250b5d7af05f5a314f830d5 Author: Nils Goroll Date: Wed Nov 14 12:43:41 2018 +0100 Revert "Make FlexeLint a little bit happier" The fact that a boolean argument is present does not imply that it's true. This reverts commit 57cd74cf51b63a54df68600a573eb0e20e2a6c7a. I promise to look after this once I got my Flexelint config back, which, unfortunately, I lost during a reinstall. Yes, I do have a backup, but cannot access it easily at the moment. diff --git a/lib/libvmod_debug/vmod_debug_obj.c b/lib/libvmod_debug/vmod_debug_obj.c index 3ecf33d0c..9038bcb81 100644 --- a/lib/libvmod_debug/vmod_debug_obj.c +++ b/lib/libvmod_debug/vmod_debug_obj.c @@ -183,9 +183,6 @@ xyzzy_obj_opt__init(VRT_CTX, if (args->valid_s) AN(args->s); - if (args->valid_b) - AN(args->b); - AN(op); AZ(*op); ALLOC_OBJ(o, VMOD_DEBUG_OBJ_OPT_MAGIC); From nils.goroll at uplex.de Thu Nov 15 15:31:16 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 15 Nov 2018 15:31:16 +0000 (UTC) Subject: [master] 5765c9f8f work around older libedit which fails to append rl_completion_append_character Message-ID: <20181115153116.EA8BFAE7C1@lists.varnish-cache.org> commit 5765c9f8f257f830044466d5d55fe40c3cda760c Author: Nils Goroll Date: Thu Nov 15 16:25:04 2018 +0100 work around older libedit which fails to append rl_completion_append_character I failed to find any issue in varnish and failed to find a workaround for the code, so change the test to add an additional space after the tab-completion. If anyone has a better solution, please go head. Otherwise I feel this is not worth any more time. The test change, though not being ideal, should not cause any harm at least. Closes #2833 diff --git a/bin/varnishtest/tests/u00011.vtc b/bin/varnishtest/tests/u00011.vtc index d6e8098dd..b53a9ba7b 100644 --- a/bin/varnishtest/tests/u00011.vtc +++ b/bin/varnishtest/tests/u00011.vtc @@ -26,7 +26,7 @@ process p1 -write "vcl.li\t\r" process p1 -expect-text 0 1 "active auto/warm" -process p1 -write "vcl.s\t\th\tvcl1\r" +process p1 -write "vcl.s\t\th\t vcl1\r" process p1 -expect-text 0 1 "backend s1" From nils.goroll at uplex.de Sat Nov 17 11:25:16 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Sat, 17 Nov 2018 11:25:16 +0000 (UTC) Subject: [master] 423452539 vtc: array not needed for token expansion Message-ID: <20181117112516.7B81277B2@lists.varnish-cache.org> commit 4234525392f871b98d89f05e23202855828d3abe Author: Nils Goroll Date: Sat Nov 17 12:15:44 2018 +0100 vtc: array not needed for token expansion diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c index 66a2f97f2..2ddb98cd3 100644 --- a/bin/varnishtest/vtc.c +++ b/bin/varnishtest/vtc.c @@ -302,7 +302,7 @@ parse_string(const char *spec, const struct cmds *cmd, void *priv, struct vtclog *vl) { char *token_s[MAX_TOKENS], *token_e[MAX_TOKENS]; - struct vsb *token_exp[MAX_TOKENS]; + struct vsb *token_exp; char *e, *p, *q, *f, *buf; int nest_brace; int tn; @@ -401,14 +401,13 @@ parse_string(const char *spec, const struct cmds *cmd, void *priv, assert(tn < MAX_TOKENS); token_s[tn] = NULL; for (tn = 0; token_s[tn] != NULL; tn++) { - token_exp[tn] = NULL; AN(token_e[tn]); /*lint !e771 */ *token_e[tn] = '\0'; /*lint !e771 */ if (NULL != strstr(token_s[tn], "${")) { - token_exp[tn] = macro_expand(vl, token_s[tn]); + token_exp = macro_expand(vl, token_s[tn]); if (vtc_error) return; - token_s[tn] = VSB_data(token_exp[tn]); + token_s[tn] = VSB_data(token_exp); token_e[tn] = strchr(token_s[tn], '\0'); } } From nils.goroll at uplex.de Sat Nov 17 11:25:16 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Sat, 17 Nov 2018 11:25:16 +0000 (UTC) Subject: [master] 1c1a72af3 vtc: zero the token arrays for each command Message-ID: <20181117112516.92AFE77B7@lists.varnish-cache.org> commit 1c1a72af37c46bc5ba6e5335ad742e541ce268dd Author: Nils Goroll Date: Sat Nov 17 12:17:18 2018 +0100 vtc: zero the token arrays for each command arguments could leak from one vtc command to the next where NULL arguments do not necessarily denote the end of arguments, eg. in varnish_expect diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c index 2ddb98cd3..8a9057333 100644 --- a/bin/varnishtest/vtc.c +++ b/bin/varnishtest/vtc.c @@ -339,6 +339,8 @@ parse_string(const char *spec, const struct cmds *cmd, void *priv, vtc_log(vl, 2, "=== %.*s", (int)(q - p), p); /* First content on line, collect tokens */ + memset(token_s, 0, sizeof token_s); + memset(token_e, 0, sizeof token_e); tn = 0; f = p; while (p < e) { From nils.goroll at uplex.de Sat Nov 17 14:08:13 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Sat, 17 Nov 2018 14:08:13 +0000 (UTC) Subject: [master] eaeac2062 fix vcl temperature after failing warmup event Message-ID: <20181117140813.5452061E1E@lists.varnish-cache.org> commit eaeac20620384802478b8f4fba41e13697a0b3ca Author: Nils Goroll Date: Sat Nov 17 15:06:41 2018 +0100 fix vcl temperature after failing warmup event Fixes #2835 diff --git a/bin/varnishd/cache/cache_vcl.c b/bin/varnishd/cache/cache_vcl.c index 02af16289..fd9deade4 100644 --- a/bin/varnishd/cache/cache_vcl.c +++ b/bin/varnishd/cache/cache_vcl.c @@ -507,10 +507,12 @@ vcl_set_state(VRT_CTX, const char *state) else { vcl->temp = VCL_TEMP_WARM; i = vcl_send_event(ctx, VCL_EVENT_WARM); - if (i == 0) + if (i == 0) { vcl_BackendEvent(vcl, VCL_EVENT_WARM); - else - AZ(vcl->conf->event_vcl(ctx, VCL_EVENT_COLD)); + break; + } + AZ(vcl->conf->event_vcl(ctx, VCL_EVENT_COLD)); + vcl->temp = VCL_TEMP_COLD; } break; default: diff --git a/bin/varnishtest/tests/v00003.vtc b/bin/varnishtest/tests/v00003.vtc index ff218f28c..6236f28f1 100644 --- a/bin/varnishtest/tests/v00003.vtc +++ b/bin/varnishtest/tests/v00003.vtc @@ -88,6 +88,8 @@ varnish v1 -expect !VBE.vcl1.default.happy varnish v1 -cliok "param.set max_esi_depth 42" varnish v1 -clierr 300 "vcl.state vcl1 warm" +varnish v1 -cliexpect "available *cold/cold *[0-9]+ *vcl1\\s+active *warm/warm *[0-9]+ *vcl2" "vcl.list" + # A warm-up failure can also fail a child start varnish v1 -cliok stop varnish v1 -cliok "vcl.state vcl1 warm" From nils.goroll at uplex.de Sat Nov 17 18:07:09 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Sat, 17 Nov 2018 18:07:09 +0000 (UTC) Subject: [master] 3a2a7a33e centralize updating the vcl->state in vcl_set_state Message-ID: <20181117180709.D26F56EBB3@lists.varnish-cache.org> commit 3a2a7a33ef6bb7aa2c2f36379130269c93ccef0e Author: Nils Goroll Date: Sat Nov 17 14:13:51 2018 +0100 centralize updating the vcl->state in vcl_set_state Allow calling it for this purpose only (stop asserting if the temperature has not changed) diff --git a/bin/varnishd/cache/cache_vcl.c b/bin/varnishd/cache/cache_vcl.c index fd9deade4..a5a33cf9b 100644 --- a/bin/varnishd/cache/cache_vcl.c +++ b/bin/varnishd/cache/cache_vcl.c @@ -479,9 +479,9 @@ vcl_set_state(VRT_CTX, const char *state) switch (state[0]) { case '0': - assert(vcl->temp != VCL_TEMP_COLD); + if (vcl->temp == VCL_TEMP_COLD) + break; if (vcl->busy == 0 && VCL_WARM(vcl)) { - vcl->temp = VTAILQ_EMPTY(&vcl->ref_list) ? VCL_TEMP_COLD : VCL_TEMP_COOLING; AZ(vcl_send_event(ctx, VCL_EVENT_COLD)); @@ -495,7 +495,8 @@ vcl_set_state(VRT_CTX, const char *state) vcl->temp = VCL_TEMP_COOLING; break; case '1': - assert(vcl->temp != VCL_TEMP_WARM); + if (vcl->temp == VCL_TEMP_WARM) + break; /* The warm VCL hasn't seen a cold event yet */ if (vcl->temp == VCL_TEMP_BUSY) vcl->temp = VCL_TEMP_WARM; @@ -518,6 +519,8 @@ vcl_set_state(VRT_CTX, const char *state) default: WRONG("Wrong enum state"); } + if (i == 0 && state[1]) + bprintf(vcl->state, "%s", state + 1); AZ(errno=pthread_rwlock_unlock(&vcl->temp_rwl)); return (i); @@ -588,7 +591,6 @@ vcl_load(struct cli *cli, struct vrt_ctx *ctx, vcl_cancel_load(ctx, cli, name, "warmup"); return; } - bprintf(vcl->state, "%s", state + 1); VCLI_Out(cli, "Loaded \"%s\" as \"%s\"", fn , name); VTAILQ_INSERT_TAIL(&vcl_head, vcl, list); Lck_Lock(&vcl_mtx); @@ -741,9 +743,7 @@ vcl_cli_state(struct cli *cli, const char * const *av, void *priv) ctx = vcl_get_ctx(0, 1); ctx->vcl = vcl_find(av[2]); AN(ctx->vcl); // MGT ensures this - if (vcl_set_state(ctx, av[3]) == 0) { - bprintf(ctx->vcl->state, "%s", av[3] + 1); - } else { + if (vcl_set_state(ctx, av[3])) { AZ(VSB_finish(ctx->msg)); VCLI_SetResult(cli, CLIS_CANT); VCLI_Out(cli, "Failed ", ctx->vcl->loaded_name, From nils.goroll at uplex.de Sat Nov 17 18:07:09 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Sat, 17 Nov 2018 18:07:09 +0000 (UTC) Subject: [master] 5aeec6ddf centralize vcl state symbols / strings in the manager Message-ID: <20181117180709.C108F6EBB1@lists.varnish-cache.org> commit 5aeec6ddfdbf684d5e2db94bc600632aea2c8129 Author: Nils Goroll Date: Wed Nov 14 18:37:27 2018 +0100 centralize vcl state symbols / strings in the manager The actual benefit of this will be in follow-up commits diff --git a/bin/varnishd/mgt/mgt_vcl.c b/bin/varnishd/mgt/mgt_vcl.c index f1ce50f5d..7d8acb572 100644 --- a/bin/varnishd/mgt/mgt_vcl.c +++ b/bin/varnishd/mgt/mgt_vcl.c @@ -47,10 +47,12 @@ #include "vev.h" #include "vtim.h" -static const char * const VCL_STATE_COLD = "cold"; -static const char * const VCL_STATE_WARM = "warm"; -static const char * const VCL_STATE_AUTO = "auto"; +#define VCL_STATE(sym, str) \ + static const char * const VCL_STATE_ ## sym = str; +#include "tbl/vcl_states.h" + static const char * const VCL_STATE_LABEL = "label"; + static int vcl_count; struct vclprog; @@ -106,6 +108,18 @@ static int mgt_vcl_setstate(struct cli *, struct vclprog *, const char *); /*--------------------------------------------------------------------*/ +static const char * +mcf_vcl_parse_state(const char *s) +{ + if (s == NULL || *s == '\0') + return (NULL); +#define VCL_STATE(sym, str) \ + if (!strcmp(s, VCL_STATE_ ## sym)) \ + return (VCL_STATE_ ## sym); +#include "tbl/vcl_states.h" + return (NULL); +} + static struct vclprog * mcf_vcl_byname(const char *name) { @@ -171,7 +185,7 @@ mcf_find_no_vcl(struct cli *cli, const char *name) static int mcf_is_label(const struct vclprog *vp) { - return (!strcmp(vp->state, VCL_STATE_LABEL)); + return (vp->state == VCL_STATE_LABEL); } /*--------------------------------------------------------------------*/ @@ -480,13 +494,10 @@ mgt_new_vcl(struct cli *cli, const char *vclname, const char *vclsrc, if (state == NULL) state = VCL_STATE_AUTO; - else if (!strcmp(state, VCL_STATE_AUTO)) - state = VCL_STATE_AUTO; - else if (!strcmp(state, VCL_STATE_COLD)) - state = VCL_STATE_COLD; - else if (!strcmp(state, VCL_STATE_WARM)) - state = VCL_STATE_WARM; - else { + else + state = mcf_vcl_parse_state(state); + + if (state == NULL) { VCLI_Out(cli, "State must be one of auto, cold or warm."); VCLI_SetResult(cli, CLIS_PARAM); return; @@ -526,7 +537,7 @@ mgt_new_vcl(struct cli *cli, const char *vclname, const char *vclsrc, } free(p); - if (vp->warm && !strcmp(vp->state, "auto")) + if (vp->warm && vp->state == VCL_STATE_AUTO) vp->go_cold = VTIM_mono(); } @@ -645,6 +656,7 @@ mcf_vcl_load(struct cli *cli, const char * const *av, void *priv) static void v_matchproto_(cli_func_t) mcf_vcl_state(struct cli *cli, const char * const *av, void *priv) { + const char *state; struct vclprog *vp; (void)priv; @@ -657,22 +669,30 @@ mcf_vcl_state(struct cli *cli, const char * const *av, void *priv) VCLI_SetResult(cli, CLIS_PARAM); return; } + + state = mcf_vcl_parse_state(av[3]); + if (state == NULL) { + VCLI_Out(cli, "State must be one of auto, cold or warm."); + VCLI_SetResult(cli, CLIS_PARAM); + return; + } + if (!VTAILQ_EMPTY(&vp->dto)) { - AN(strcmp(vp->state, "cold")); - if (!strcmp(av[3], "cold")) { + assert(vp->state != VCL_STATE_COLD); + if (state == VCL_STATE_COLD) { VCLI_Out(cli, "A labeled VCL cannot be set cold"); VCLI_SetResult(cli, CLIS_CANT); return; } } - if (!strcmp(vp->state, av[3])) + if (vp->state == state) return; - if (!strcmp(av[3], VCL_STATE_AUTO)) { + if (state == VCL_STATE_AUTO) { vp->state = VCL_STATE_AUTO; (void)mgt_vcl_setstate(cli, vp, VCL_STATE_AUTO); - } else if (!strcmp(av[3], VCL_STATE_COLD)) { + } else if (state == VCL_STATE_COLD) { if (vp == active_vcl) { VCLI_Out(cli, "Cannot set the active VCL cold."); VCLI_SetResult(cli, CLIS_CANT); @@ -680,7 +700,7 @@ mcf_vcl_state(struct cli *cli, const char * const *av, void *priv) } vp->state = VCL_STATE_AUTO; (void)mgt_vcl_setstate(cli, vp, VCL_STATE_COLD); - } else if (!strcmp(av[3], VCL_STATE_WARM)) { + } else if (state == VCL_STATE_WARM) { if (mgt_vcl_setstate(cli, vp, VCL_STATE_WARM) == 0) vp->state = VCL_STATE_WARM; } else { diff --git a/include/tbl/vcl_states.h b/include/tbl/vcl_states.h new file mode 100644 index 000000000..6fe74dad1 --- /dev/null +++ b/include/tbl/vcl_states.h @@ -0,0 +1,8 @@ +/*lint -save -e525 -e539 */ +VCL_STATE(COLD, "cold") +VCL_STATE(WARM, "warm") +VCL_STATE(AUTO, "auto") +#undef VCL_STATE +// LABEL is private to mgt_vcl.c + +/*lint -restore */ From nils.goroll at uplex.de Sat Nov 17 18:43:06 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Sat, 17 Nov 2018 18:43:06 +0000 (UTC) Subject: [master] c90f8fd65 forgotten file in dist Message-ID: <20181117184306.9EC2C92B53@lists.varnish-cache.org> commit c90f8fd654e5c9ffaad1e4fd53bac9b7d05a7c69 Author: Nils Goroll Date: Sat Nov 17 19:41:44 2018 +0100 forgotten file in dist diff --git a/include/Makefile.am b/include/Makefile.am index 50142a64b..63757e48c 100644 --- a/include/Makefile.am +++ b/include/Makefile.am @@ -32,6 +32,7 @@ nobase_pkginclude_HEADERS = \ tbl/symbol_kind.h \ tbl/vcc_types.h \ tbl/vcl_returns.h \ + tbl/vcl_states.h \ tbl/vhd_fsm.h \ tbl/vhd_fsm_funcs.h \ tbl/vhd_return.h \ From daghf at varnish-software.com Mon Nov 19 10:30:17 2018 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Mon, 19 Nov 2018 10:30:17 +0000 (UTC) Subject: [master] 6045eaaa6 Add error handling for STV_NewObject() in vrb_pull Message-ID: <20181119103017.0CA0CA1749@lists.varnish-cache.org> commit 6045eaaa6a2f0a51f667d4c2cf23cd7fb43abe9f Author: Dag Haavi Finstad Date: Wed Nov 14 13:39:46 2018 +0100 Add error handling for STV_NewObject() in vrb_pull Fixes: #2831 diff --git a/bin/varnishd/cache/cache_req_body.c b/bin/varnishd/cache/cache_req_body.c index 86d1dd4bf..488815d8a 100644 --- a/bin/varnishd/cache/cache_req_body.c +++ b/bin/varnishd/cache/cache_req_body.c @@ -73,7 +73,14 @@ vrb_pull(struct req *req, ssize_t maxsize, objiterate_f *func, void *priv) req->storage = NULL; - XXXAN(STV_NewObject(req->wrk, req->body_oc, stv, 8)); + if (STV_NewObject(req->wrk, req->body_oc, stv, 8) == 0) { + req->req_body_status = REQ_BODY_FAIL; + HSH_DerefBoc(req->wrk, req->body_oc); + AZ(HSH_DerefObjCore(req->wrk, &req->body_oc, 0)); + (void)VFP_Error(vfc, "Object allocation failed:" + " Ran out of space in %s", stv->vclname); + return (-1); + } vfc->oc = req->body_oc; diff --git a/bin/varnishtest/tests/r02831.vtc b/bin/varnishtest/tests/r02831.vtc new file mode 100644 index 000000000..0a7e85239 --- /dev/null +++ b/bin/varnishtest/tests/r02831.vtc @@ -0,0 +1,57 @@ +varnishtest "#2831: Out of storage in cache_req_body" + +server s1 { + rxreq + expect req.url == "/obj1" + txresp -bodylen 1048400 +} -start + +varnish v1 \ + -arg "-p nuke_limit=0" \ + -arg "-sTransient=default,1m" \ + -syntax 4.0 \ + -vcl+backend { + import std; + sub vcl_recv { + if (req.method == "POST") { + std.cache_req_body(1KB); + } + } + sub vcl_backend_response { + set beresp.do_stream = false; + set beresp.storage = storage.Transient; + # Unset Date header to not change the object sizes + unset beresp.http.Date; + } +} -start + +varnish v1 -cliok "param.set debug +syncvsl" + +delay .1 + +client c1 { + # Fill transient + txreq -url "/obj1" + rxresp + expect resp.status == 200 +} -run + +delay .1 + +varnish v1 -expect SM?.Transient.g_bytes > 1048400 +varnish v1 -expect SM?.Transient.g_space < 100 + +client c1 { + # No space for caching this req.body + txreq -req "POST" -body "foobar" + delay 1 +} -run + +varnish v1 -expect SMA.Transient.c_fail == 1 + +client c1 { + # Check that Varnish is still alive + txreq -url "/obj1" + rxresp + expect resp.status == 200 +} -run From daghf at varnish-software.com Mon Nov 19 12:15:12 2018 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Mon, 19 Nov 2018 12:15:12 +0000 (UTC) Subject: [master] 9e155bbab Fix test case: 'SMA'->'SM?' Message-ID: <20181119121512.3744AA3BCE@lists.varnish-cache.org> commit 9e155bbab0949fb6d4c3cc023c75de5bf5ddb1fb Author: Dag Haavi Finstad Date: Mon Nov 19 13:14:27 2018 +0100 Fix test case: 'SMA'->'SM?' diff --git a/bin/varnishtest/tests/r02831.vtc b/bin/varnishtest/tests/r02831.vtc index 0a7e85239..6f5e87ac5 100644 --- a/bin/varnishtest/tests/r02831.vtc +++ b/bin/varnishtest/tests/r02831.vtc @@ -47,7 +47,7 @@ client c1 { delay 1 } -run -varnish v1 -expect SMA.Transient.c_fail == 1 +varnish v1 -expect SM?.Transient.c_fail == 1 client c1 { # Check that Varnish is still alive From nils.goroll at uplex.de Mon Nov 19 13:29:08 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 19 Nov 2018 13:29:08 +0000 (UTC) Subject: [master] 518f46bbf minor vtim-ification Message-ID: <20181119132908.E513AA5525@lists.varnish-cache.org> commit 518f46bbfd01b68907aa730b1112ab2e188a10cd Author: Nils Goroll Date: Mon Nov 19 14:28:14 2018 +0100 minor vtim-ification diff --git a/lib/libvarnish/vnum.c b/lib/libvarnish/vnum.c index 6caf931a1..b619199c6 100644 --- a/lib/libvarnish/vnum.c +++ b/lib/libvarnish/vnum.c @@ -110,11 +110,12 @@ VNUM(const char *p) /**********************************************************************/ -double +vtim_dur VNUM_duration(const char *p) { const char *t; - double r, sc = 1.0; + vtim_dur r; + double sc = 1.0; if (p == NULL) return (nan("")); From nils.goroll at uplex.de Mon Nov 19 14:14:14 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 19 Nov 2018 14:14:14 +0000 (UTC) Subject: [master] dc1df48ff VIP16: retire parameters aliases Message-ID: <20181119141414.A4D7EA65D7@lists.varnish-cache.org> commit dc1df48ffe9c3f5ba9bc61cf959d23f233a7a605 Author: Nils Goroll Date: Mon Nov 19 15:08:36 2018 +0100 VIP16: retire parameters aliases Ref: #2830 diff --git a/bin/varnishd/mgt/mgt_param_tbl.c b/bin/varnishd/mgt/mgt_param_tbl.c index 2bd7bb92e..46f462b14 100644 --- a/bin/varnishd/mgt/mgt_param_tbl.c +++ b/bin/varnishd/mgt/mgt_param_tbl.c @@ -55,12 +55,6 @@ struct parspec mgt_parspec[] = { "and %o will be replaced with the output file name.", MUST_RELOAD, VCC_CC , NULL }, - { "vcl_dir", tweak_string, &mgt_vcl_path, - NULL, NULL, - "Old name for vcl_path, use that instead.", - 0, - VARNISH_VCL_DIR, - NULL }, { "vcl_path", tweak_string, &mgt_vcl_path, NULL, NULL, "Directory (or colon separated list of directories) " @@ -73,12 +67,6 @@ struct parspec mgt_parspec[] = { 0, VARNISH_VCL_DIR, NULL }, - { "vmod_dir", tweak_string, &mgt_vmod_path, - NULL, NULL, - "Old name for vmod_path, use that instead.", - 0, - VARNISH_VMOD_DIR, - NULL }, { "vmod_path", tweak_string, &mgt_vmod_path, NULL, NULL, "Directory (or colon separated list of directories) " @@ -146,12 +134,6 @@ struct parspec mgt_parspec[] = { MEMPOOL_TEXT, 0, "10,100,10", ""}, - { "shm_reclen", tweak_vsl_reclen, &mgt_param.vsl_reclen, - "16b", NULL, - "Old name for vsl_reclen, use that instead.", - 0, - "255b", - "bytes" }, { NULL, NULL, NULL } }; diff --git a/bin/varnishd/mgt/mgt_param_tweak.c b/bin/varnishd/mgt/mgt_param_tweak.c index 01cab5f6b..6900f4e5f 100644 --- a/bin/varnishd/mgt/mgt_param_tweak.c +++ b/bin/varnishd/mgt/mgt_param_tweak.c @@ -340,7 +340,6 @@ tweak_vsl_buffer(struct vsb *vsb, const struct parspec *par, const char *arg) return (-1); *d1 = dest; MCF_ParamConf(MCF_MAXIMUM, "vsl_reclen", "%u", *d1 - 12); - MCF_ParamConf(MCF_MAXIMUM, "shm_reclen", "%u", *d1 - 12); return (0); } diff --git a/bin/varnishtest/tests/m00003.vtc b/bin/varnishtest/tests/m00003.vtc index f0b312053..e2f6dccab 100644 --- a/bin/varnishtest/tests/m00003.vtc +++ b/bin/varnishtest/tests/m00003.vtc @@ -18,7 +18,7 @@ varnish v1 -errvcl {Could not load VMOD std} { import std; } -varnish v1 -cliok "param.set vmod_dir ${topbuild}/lib/libvmod_std/.libs/" +varnish v1 -cliok "param.set vmod_path ${topbuild}/lib/libvmod_std/.libs/" varnish v1 -vcl+backend { import std; diff --git a/bin/varnishtest/tests/r02307.vtc b/bin/varnishtest/tests/r02307.vtc deleted file mode 100644 index a86fe6d2f..000000000 --- a/bin/varnishtest/tests/r02307.vtc +++ /dev/null @@ -1,4 +0,0 @@ -varnishtest "shm_reclen alias for vsl_reclen" - -varnish v1 -cli "param.set shm_reclen 42" -varnish v1 -cliexpect 42b "param.show vsl_reclen" diff --git a/doc/changes.rst b/doc/changes.rst index 82bdc813b..a3d500a28 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -103,6 +103,11 @@ Varnish Cache trunk (ongoing) * Varnish-based tools using the VUT interface should now consider using the ``VUT_Usage()`` function for consistency +* retired long deprecated parameters (VIP16_). Replacement mapping is: + ``shm_reclen`` -> ``vsl_reclen`` + ``vcl_dir`` -> ``vcl_path`` + ``vmod_dir`` -> ``vmod_path`` + .. _2809: https://github.com/varnishcache/varnish-cache/issues/2809 .. _2820: https://github.com/varnishcache/varnish-cache/issues/2820 .. _2815: https://github.com/varnishcache/varnish-cache/issues/2815 @@ -117,6 +122,7 @@ Varnish Cache trunk (ongoing) .. _2418: https://github.com/varnishcache/varnish-cache/issues/2418 .. _2788: https://github.com/varnishcache/varnish-cache/issues/2788 .. _2790: https://github.com/varnishcache/varnish-cache/issues/2790 +.. _VIP16: https://github.com/varnishcache/varnish-cache/wiki/VIP16%3A-Retire-parameters-aliases ================================ Varnish Cache 6.1.0 (2018-09-17) diff --git a/include/tbl/params.h b/include/tbl/params.h index d9f27c99e..c60b48e2b 100644 --- a/include/tbl/params.h +++ b/include/tbl/params.h @@ -1040,23 +1040,6 @@ PARAM( ) #undef XYZZY -#if 0 -/* actual location mgt_param_tbl.c */ -PARAM( - /* name */ shm_reclen, - /* typ */ vsl_reclen, - /* min */ "16b", - /* max */ NULL, - /* default */ "255b", - /* units */ "bytes", - /* flags */ 0, - /* s-text */ - "Old name for vsl_reclen, use that instead.", - /* l-text */ "", - /* func */ NULL -) -#endif - PARAM( /* name */ shortlived, /* typ */ timeout, @@ -1585,39 +1568,6 @@ PARAM( /* func */ NULL ) -#if 0 -/* actual location mgt_param_tbl.c */ -PARAM( - /* name */ vcl_dir, - /* typ */ string, - /* min */ NULL, - /* max */ NULL, - /* default */ "/opt/varnish/etc/varnish", - /* units */ NULL, - /* flags */ 0, - /* s-text */ - "Directory from which relative VCL filenames (vcl.load and " - "include) are opened.", - /* l-text */ "", - /* func */ NULL -) - -/* actual location mgt_param_tbl.c */ -PARAM( - /* name */ vmod_dir, - /* typ */ string, - /* min */ NULL, - /* max */ NULL, - /* default */ "/opt/varnish/lib/varnish/vmods", - /* units */ NULL, - /* flags */ 0, - /* s-text */ - "Directory where Varnish modules are to be found.", - /* l-text */ "", - /* func */ NULL -) -#endif - PARAM( /* name */ vsl_buffer, /* typ */ vsl_buffer, From phk at FreeBSD.org Mon Nov 19 22:15:16 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 19 Nov 2018 22:15:16 +0000 (UTC) Subject: [master] ce4c3214c Clarify reasoning with comment Message-ID: <20181119221516.0EB5BB27D0@lists.varnish-cache.org> commit ce4c3214c01da83481f54ac3d31ac001cc061028 Author: Poul-Henning Kamp Date: Mon Nov 19 11:44:12 2018 +0000 Clarify reasoning with comment diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index e8c2e542b..0b4141606 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -366,7 +366,7 @@ cnt_transmit(struct worker *wrk, struct req *req) if (head || status < 200 || status == 204 || status == 304) { // rfc7230,l,1748,1752 - sendbody = 0; + sendbody = 0; } else { sendbody = 1; } @@ -401,7 +401,14 @@ cnt_transmit(struct worker *wrk, struct req *req) } else if (clval >= 0 && clval == req->resp_len) { /* Reuse C-L header */ } else if (head && req->objcore->flags & OC_F_PASS) { - /* Don't touch C-L header */ + /* + * Don't touch C-L header (debatable) + * + * The only way to do it correctly would be to GET + * to the backend, and discard the body once the + * filters have had a chance to chew on it, but that + * would negate the "pass for huge objects" use case. + */ } else { http_Unset(req->resp, H_Content_Length); if (req->resp_len >= 0) From phk at FreeBSD.org Mon Nov 19 22:15:16 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 19 Nov 2018 22:15:16 +0000 (UTC) Subject: [master] fa31620e2 Allow VDP->init to say "no thanks, but do go on delivering." Message-ID: <20181119221516.21977B27D2@lists.varnish-cache.org> commit fa31620e2facb1e44f8379dd2c00c477889ba3e3 Author: Poul-Henning Kamp Date: Mon Nov 19 12:21:26 2018 +0000 Allow VDP->init to say "no thanks, but do go on delivering." diff --git a/bin/varnishd/cache/cache_deliver_proc.c b/bin/varnishd/cache/cache_deliver_proc.c index 0e7602a10..c549f3be3 100644 --- a/bin/varnishd/cache/cache_deliver_proc.c +++ b/bin/varnishd/cache/cache_deliver_proc.c @@ -78,6 +78,7 @@ VDP_Push(struct req *req, const struct vdp *vdp, void *priv) { struct vdp_entry *vdpe; struct vdp_ctx *vdc; + uintptr_t sn; CHECK_OBJ_NOTNULL(req, REQ_MAGIC); vdc = req->vdc; @@ -91,6 +92,7 @@ VDP_Push(struct req *req, const struct vdp *vdp, void *priv) if (DO_DEBUG(DBG_PROCESSORS)) VSLb(req->vsl, SLT_Debug, "VDP_push(%s)", vdp->name); + sn = WS_Snapshot(req->ws); vdpe = WS_Alloc(req->ws, sizeof *vdpe); if (vdpe == NULL) { AZ(vdc->retval); @@ -106,6 +108,12 @@ VDP_Push(struct req *req, const struct vdp *vdp, void *priv) AZ(vdc->retval); if (vdpe->vdp->init != NULL) vdc->retval = vdpe->vdp->init(req, &vdpe->priv); + if (vdc->retval > 0) { + VTAILQ_REMOVE(&vdc->vdp, vdpe, list); + vdc->nxt = VTAILQ_FIRST(&vdc->vdp); + WS_Reset(req->ws, sn); + vdc->retval = 0; + } return (vdc->retval); } diff --git a/bin/varnishd/cache/cache_filter.h b/bin/varnishd/cache/cache_filter.h index 79097f99d..57415e1af 100644 --- a/bin/varnishd/cache/cache_filter.h +++ b/bin/varnishd/cache/cache_filter.h @@ -99,6 +99,13 @@ enum vdp_action { }; typedef int vdp_init_f(struct req *, void **priv); +/* + * Return value: + * negative: Error - abandon delivery + * zero: OK + * positive: Don't push this VDP anyway + */ + typedef int vdp_fini_f(struct req *, void **priv); typedef int vdp_bytes_f(struct req *, enum vdp_action, void **priv, const void *ptr, ssize_t len); From phk at FreeBSD.org Mon Nov 19 22:15:16 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 19 Nov 2018 22:15:16 +0000 (UTC) Subject: [master] 721289a34 Rearrange range code to ponder life in VDP->init() Message-ID: <20181119221516.42B23B27D6@lists.varnish-cache.org> commit 721289a34cd3a94f8d3d0dd3e3b9869561fb6a9d Author: Poul-Henning Kamp Date: Mon Nov 19 20:57:47 2018 +0000 Rearrange range code to ponder life in VDP->init() diff --git a/bin/varnishd/cache/cache_range.c b/bin/varnishd/cache/cache_range.c index 8bcec96d4..37e4d84e7 100644 --- a/bin/varnishd/cache/cache_range.c +++ b/bin/varnishd/cache/cache_range.c @@ -89,16 +89,10 @@ vrg_range_bytes(struct req *req, enum vdp_action act, void **priv, vrg_priv->range_off >= vrg_priv->range_high ? 1 : 0); } -static const struct vdp vrg_vdp = { - .name = "RNG", - .bytes = vrg_range_bytes, - .fini = vrg_range_fini, -}; - /*--------------------------------------------------------------------*/ static const char * -vrg_dorange(struct req *req, const char *r) +vrg_dorange(struct req *req, const char *r, void **priv) { ssize_t low, high, has_low, has_high, t; struct vrg_priv *vrg_priv; @@ -181,35 +175,47 @@ vrg_dorange(struct req *req, const char *r) vrg_priv->range_off = 0; vrg_priv->range_low = low; vrg_priv->range_high = high + 1; - if (VDP_Push(req, &vrg_vdp, vrg_priv)) - return ("WS too small"); + *priv = vrg_priv; http_PutResponse(req->resp, "HTTP/1.1", 206, NULL); return (NULL); } +static int v_matchproto_(vdp_init_f) +vrg_range_init(struct req *req, void **priv) +{ + const char *r; + const char *err; + + assert(http_GetHdr(req->http, H_Range, &r)); + err = vrg_dorange(req, r, priv); + if (err == NULL) + return (*priv == NULL ? 1 : 0); + + VSLb(req->vsl, SLT_Debug, "RANGE_FAIL %s", err); + if (req->resp_len >= 0) + http_PrintfHeader(req->resp, + "Content-Range: bytes */%jd", + (intmax_t)req->resp_len); + http_PutResponse(req->resp, "HTTP/1.1", 416, NULL); + /* + * XXX: We ought to produce a body explaining things. + * XXX: That really calls for us to hit vcl_synth{} + */ + req->resp_len = 0; + return (1); +} + +static const struct vdp vrg_vdp = { + .name = "range", + .init = vrg_range_init, + .bytes = vrg_range_bytes, + .fini = vrg_range_fini, +}; + void VRG_dorange(struct req *req, const char *r) { - const char *err; - CHECK_OBJ_NOTNULL(req, REQ_MAGIC); - CHECK_OBJ_NOTNULL(req->objcore, OBJCORE_MAGIC); - assert(http_IsStatus(req->resp, 200)); - - /* We must snapshot the length if we're streaming from the backend */ - - err = vrg_dorange(req, r); - if (err != NULL) { - VSLb(req->vsl, SLT_Debug, "RANGE_FAIL %s", err); - if (req->resp_len >= 0) - http_PrintfHeader(req->resp, - "Content-Range: bytes */%jd", - (intmax_t)req->resp_len); - http_PutResponse(req->resp, "HTTP/1.1", 416, NULL); - /* - * XXX: We ought to produce a body explaining things. - * XXX: That really calls for us to hit vcl_synth{} - */ - req->resp_len = 0; - } + (void)r; + AZ(VDP_Push(req, &vrg_vdp, NULL)); } From phk at FreeBSD.org Mon Nov 19 22:15:16 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 19 Nov 2018 22:15:16 +0000 (UTC) Subject: [master] b0bc73e42 Use vsl_catchup instead of delay .1 Message-ID: <20181119221516.5C6B3B27DB@lists.varnish-cache.org> commit b0bc73e4269165d337ef5fc5004df0c1f8447ced Author: Poul-Henning Kamp Date: Mon Nov 19 21:20:00 2018 +0000 Use vsl_catchup instead of delay .1 diff --git a/bin/varnishtest/tests/c00034.vtc b/bin/varnishtest/tests/c00034.vtc index 1344fe2a6..3fbe5fefd 100644 --- a/bin/varnishtest/tests/c00034.vtc +++ b/bin/varnishtest/tests/c00034.vtc @@ -22,6 +22,8 @@ client c1 { varnish v1 -expect s_resp_bodybytes == 100 +varnish v1 -vsl_catchup + varnish v1 -cliok "param.set http_range_support on" client c1 { @@ -72,7 +74,8 @@ client c1 { } -run varnish v1 -expect s_resp_bodybytes == 100 -delay .1 + +varnish v1 -vsl_catchup client c1 { # Valid range requests @@ -122,6 +125,8 @@ client c1 { varnish v1 -expect s_resp_bodybytes == 501 +varnish v1 -vsl_catchup + # Test Range streaming with streaming objects with C-L server s1 { @@ -174,7 +179,7 @@ client c1 { expect_close } -run -delay .1 +varnish v1 -vsl_catchup client c1 { # Closed C-L because we cannot use C-L From phk at FreeBSD.org Mon Nov 19 22:15:16 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 19 Nov 2018 22:15:16 +0000 (UTC) Subject: [master] 45c767cce Don't reset the workspace on pointless VDP_Push(), things might still have been happening on the workspace (See: VRG) Message-ID: <20181119221516.798D6B27E0@lists.varnish-cache.org> commit 45c767cce08971793c631312f307d67b4418aa02 Author: Poul-Henning Kamp Date: Mon Nov 19 21:26:48 2018 +0000 Don't reset the workspace on pointless VDP_Push(), things might still have been happening on the workspace (See: VRG) diff --git a/bin/varnishd/cache/cache_deliver_proc.c b/bin/varnishd/cache/cache_deliver_proc.c index c549f3be3..8c4d3b538 100644 --- a/bin/varnishd/cache/cache_deliver_proc.c +++ b/bin/varnishd/cache/cache_deliver_proc.c @@ -78,7 +78,6 @@ VDP_Push(struct req *req, const struct vdp *vdp, void *priv) { struct vdp_entry *vdpe; struct vdp_ctx *vdc; - uintptr_t sn; CHECK_OBJ_NOTNULL(req, REQ_MAGIC); vdc = req->vdc; @@ -92,7 +91,6 @@ VDP_Push(struct req *req, const struct vdp *vdp, void *priv) if (DO_DEBUG(DBG_PROCESSORS)) VSLb(req->vsl, SLT_Debug, "VDP_push(%s)", vdp->name); - sn = WS_Snapshot(req->ws); vdpe = WS_Alloc(req->ws, sizeof *vdpe); if (vdpe == NULL) { AZ(vdc->retval); @@ -111,7 +109,6 @@ VDP_Push(struct req *req, const struct vdp *vdp, void *priv) if (vdc->retval > 0) { VTAILQ_REMOVE(&vdc->vdp, vdpe, list); vdc->nxt = VTAILQ_FIRST(&vdc->vdp); - WS_Reset(req->ws, sn); vdc->retval = 0; } return (vdc->retval); From phk at FreeBSD.org Mon Nov 19 22:15:16 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 19 Nov 2018 22:15:16 +0000 (UTC) Subject: [master] 704a87000 Engage the resp.filters variable Message-ID: <20181119221516.91920B27E6@lists.varnish-cache.org> commit 704a87000bb2acdf3e923f6560fd0b99965e6c1f Author: Poul-Henning Kamp Date: Mon Nov 19 22:03:12 2018 +0000 Engage the resp.filters variable diff --git a/bin/varnishd/cache/cache_range.c b/bin/varnishd/cache/cache_range.c index 37e4d84e7..c59fe857f 100644 --- a/bin/varnishd/cache/cache_range.c +++ b/bin/varnishd/cache/cache_range.c @@ -205,17 +205,9 @@ vrg_range_init(struct req *req, void **priv) return (1); } -static const struct vdp vrg_vdp = { +const struct vdp VDP_range = { .name = "range", .init = vrg_range_init, .bytes = vrg_range_bytes, .fini = vrg_range_fini, }; - -void -VRG_dorange(struct req *req, const char *r) -{ - - (void)r; - AZ(VDP_Push(req, &vrg_vdp, NULL)); -} diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index 0b4141606..ae807f739 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -339,9 +339,8 @@ static enum req_fsm_nxt cnt_transmit(struct worker *wrk, struct req *req) { struct boc *boc; - const char *r; uint16_t status; - int err, sendbody, head; + int sendbody, head; intmax_t clval; CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); @@ -357,7 +356,6 @@ cnt_transmit(struct worker *wrk, struct req *req) /* RFC 7230, 3.3.3 */ status = http_GetStatus(req->resp); head = !strcmp(req->http0->hd[HTTP_HDR_METHOD].b, "HEAD"); - err = 0; if (boc != NULL) req->resp_len = clval; @@ -371,27 +369,16 @@ cnt_transmit(struct worker *wrk, struct req *req) sendbody = 1; } - if (!req->disable_esi && req->resp_len != 0 && - ObjHasAttr(wrk, req->objcore, OA_ESIDATA) && - VDP_Push(req, &VDP_esi, NULL) < 0) - err++; - - if (cache_param->http_gzip_support && - ObjCheckFlag(req->wrk, req->objcore, OF_GZIPED) && - !RFC2616_Req_Gzip(req->http) && - VDP_Push(req, &VDP_gunzip, NULL) < 0) - err++; - - if (cache_param->http_range_support && status == 200) { - http_ForceHeader(req->resp, H_Accept_Ranges, "bytes"); - if (http_GetHdr(req->http, H_Range, &r)) - VRG_dorange(req, r); - } - - if (err) { + if (req->filter_list == NULL) + req->filter_list = resp_Get_Filter_List(req); + if (req->filter_list == NULL || + VCL_StackVDP(req, req->vcl, req->filter_list)) { VSLb(req->vsl, SLT_Error, "Failure to push processors"); req->doclose = SC_OVERLOAD; } else { + if (cache_param->http_range_support && status == 200) + http_ForceHeader(req->resp, H_Accept_Ranges, "bytes"); + if (status < 200 || status == 204) { // rfc7230,l,1691,1695 http_Unset(req->resp, H_Content_Length); @@ -417,8 +404,8 @@ cnt_transmit(struct worker *wrk, struct req *req) } if (req->resp_len == 0) sendbody = 0; - req->transport->deliver(req, boc, sendbody); } + req->transport->deliver(req, boc, sendbody); VSLb_ts_req(req, "Resp", W_TIM_real(wrk)); @@ -438,6 +425,7 @@ cnt_transmit(struct worker *wrk, struct req *req) (void)HSH_DerefObjCore(wrk, &req->objcore, HSH_RUSH_POLICY); http_Teardown(req->resp); + req->filter_list = NULL; req->res_mode = 0; return (REQ_FSM_DONE); } diff --git a/bin/varnishd/cache/cache_varnishd.h b/bin/varnishd/cache/cache_varnishd.h index 791fa30b8..08caa721b 100644 --- a/bin/varnishd/cache/cache_varnishd.h +++ b/bin/varnishd/cache/cache_varnishd.h @@ -188,6 +188,7 @@ int VDP_DeliverObj(struct req *req); extern const struct vdp VDP_gunzip; extern const struct vdp VDP_esi; +extern const struct vdp VDP_range; /* cache_expire.c */ void EXP_Init(void); @@ -290,8 +291,6 @@ typedef void obj_event_f(struct worker *, void *priv, struct objcore *, uintptr_t ObjSubscribeEvents(obj_event_f *, void *, unsigned mask); void ObjUnsubscribeEvents(uintptr_t *); - - /* cache_panic.c */ void PAN_Init(void); int PAN_already(struct vsb *, const void *); @@ -308,9 +307,6 @@ int Pool_TrySumstat(const struct worker *wrk); void Pool_PurgeStat(unsigned nobj); int Pool_Task_Any(struct pool_task *task, enum task_prio prio); -/* cache_range.c [VRG] */ -void VRG_dorange(struct req *req, const char *r); - /* cache_req.c */ struct req *Req_New(const struct worker *, struct sess *); void Req_Release(struct req *); @@ -405,11 +401,15 @@ int VCL_IterDirector(struct cli *, const char *, vcl_be_func *, void *); /* cache_vcl_vrt.c */ void VCL_VRT_Init(void); -int VCL_StackVFP(struct vfp_ctx *, const struct vcl *, const char *); /* cache_vrt.c */ void pan_privs(struct vsb *, const struct vrt_privs *); +/* cache_vrt_filter.c */ +int VCL_StackVFP(struct vfp_ctx *, const struct vcl *, const char *); +int VCL_StackVDP(struct req *, const struct vcl *, const char *); +const char *resp_Get_Filter_List(struct req *req); + /* cache_vrt_priv.c */ extern struct vrt_privs cli_task_privs[1]; diff --git a/bin/varnishd/cache/cache_vrt_filter.c b/bin/varnishd/cache/cache_vrt_filter.c index b74ceb318..82540cc0d 100644 --- a/bin/varnishd/cache/cache_vrt_filter.c +++ b/bin/varnishd/cache/cache_vrt_filter.c @@ -190,6 +190,7 @@ VCL_StackVFP(struct vfp_ctx *vc, const struct vcl *vcl, const char *fl) { const struct vfilter *vp; + AN(fl); VSLb(vc->wrk->vsl, SLT_Filters, "%s", fl); while (1) { @@ -203,6 +204,27 @@ VCL_StackVFP(struct vfp_ctx *vc, const struct vcl *vcl, const char *fl) } } +int +VCL_StackVDP(struct req *req, const struct vcl *vcl, const char *fl) +{ + const struct vfilter *vp; + + AN(fl); + VSLb(req->vsl, SLT_Filters, "%s", fl); + while (1) { + vp = vcl_filter_list_iter(&vdp_filters, &vcl->vfps, &fl); + if (vp == NULL) + return (0); + if (vp == vfilter_error) { + VSLb(req->vsl, SLT_Error, + "Filter '...%s' not found", fl); + return (-1); + } + if (VDP_Push(req, vp->vdp, NULL)) + return (-1); + } +} + void VCL_VRT_Init(void) { @@ -213,15 +235,16 @@ VCL_VRT_Init(void) VRT_AddVFP(NULL, &VFP_esi_gzip); VRT_AddVDP(NULL, &VDP_esi); VRT_AddVDP(NULL, &VDP_gunzip); + VRT_AddVDP(NULL, &VDP_range); } /*-------------------------------------------------------------------- */ -typedef void filter_list_t(const void *, struct vsb *vsb); +typedef void filter_list_t(void *, struct vsb *vsb); static const char * -filter_on_ws(struct ws *ws, filter_list_t *func, const void *arg) +filter_on_ws(struct ws *ws, filter_list_t *func, void *arg) { unsigned u; struct vsb vsb[1]; @@ -253,7 +276,7 @@ filter_on_ws(struct ws *ws, filter_list_t *func, const void *arg) */ static void v_matchproto_(filter_list_t) -vbf_default_filter_list(const void *arg, struct vsb *vsb) +vbf_default_filter_list(void *arg, struct vsb *vsb) { const struct busyobj *bo; const char *p; @@ -328,15 +351,29 @@ VBF_Get_Filter_List(struct busyobj *bo) */ static void v_matchproto_(filter_list_t) -resp_default_filter_list(const void *arg, struct vsb *vsb) +resp_default_filter_list(void *arg, struct vsb *vsb) { - const struct req *req; + struct req *req; + const char *r; CAST_OBJ_NOTNULL(req, arg, REQ_MAGIC); - (void)vsb; + + if (!req->disable_esi && req->resp_len != 0 && + ObjHasAttr(req->wrk, req->objcore, OA_ESIDATA)) + VSB_cat(vsb, " esi"); + + if (cache_param->http_gzip_support && + ObjCheckFlag(req->wrk, req->objcore, OF_GZIPED) && + !RFC2616_Req_Gzip(req->http)) + VSB_cat(vsb, " gunzip"); + + if (cache_param->http_range_support && + http_GetStatus(req->resp) == 200 && + http_GetHdr(req->http, H_Range, &r)) + VSB_cat(vsb, " range"); } -static const char * +const char * resp_Get_Filter_List(struct req *req) { CHECK_OBJ_NOTNULL(req, REQ_MAGIC); diff --git a/bin/varnishtest/tests/c00071.vtc b/bin/varnishtest/tests/c00071.vtc index 6cff35d87..7e3c156ee 100644 --- a/bin/varnishtest/tests/c00071.vtc +++ b/bin/varnishtest/tests/c00071.vtc @@ -39,6 +39,8 @@ client c1 { expect resp.http.x-of == } -run +varnish v1 -vsl_catchup + client c2 { txreq -url /baz rxresp diff --git a/bin/varnishtest/tests/g00003.vtc b/bin/varnishtest/tests/g00003.vtc index 773917cba..34744bb98 100644 --- a/bin/varnishtest/tests/g00003.vtc +++ b/bin/varnishtest/tests/g00003.vtc @@ -48,7 +48,6 @@ varnish v1 -cliok "param.set http_gzip_support true" -vcl+backend { } sub vcl_deliver { set resp.http.filters = resp.filters; - set resp.filters = ""; } } -start @@ -57,18 +56,30 @@ client c1 { rxresp expect resp.http.content-encoding == expect resp.bodylen == 41 +} -run + +varnish v1 -vsl_catchup +client c1 { txreq -url /bar -hdr "Accept-Encoding: gzip" rxresp expect resp.http.content-encoding == expect resp.bodylen == 42 +} -run + +varnish v1 -vsl_catchup +client c1 { txreq -url /foobar -hdr "Accept-Encoding: gzip" rxresp expect resp.http.content-encoding == "gzip" gunzip expect resp.bodylen == 43 +} -run +varnish v1 -vsl_catchup + +client c1 { txreq -url /foobar rxresp expect resp.http.content-encoding == From phk at FreeBSD.org Mon Nov 19 22:15:16 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 19 Nov 2018 22:15:16 +0000 (UTC) Subject: [master] b31c69c84 Doc-fix Message-ID: <20181119221516.A978CB27EA@lists.varnish-cache.org> commit b31c69c8435b68af4d4b9f4c6d0edc07d28a4d6a Author: Poul-Henning Kamp Date: Mon Nov 19 22:14:26 2018 +0000 Doc-fix diff --git a/doc/sphinx/reference/vcl_var.rst b/doc/sphinx/reference/vcl_var.rst index b269aa029..c0007a761 100644 --- a/doc/sphinx/reference/vcl_var.rst +++ b/doc/sphinx/reference/vcl_var.rst @@ -1155,7 +1155,7 @@ resp.filters Writable from: vcl_deliver, vcl_synth - List of VFP filters the resp.body will be pulled through. + List of VDP filters the resp.body will be pushed through. Special variables ~~~~~~~~~~~~~~~~~ From nils.goroll at uplex.de Wed Nov 21 18:14:08 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 21 Nov 2018 18:14:08 +0000 (UTC) Subject: [master] 23e05af40 gc stray comment Message-ID: <20181121181408.56FB0634B4@lists.varnish-cache.org> commit 23e05af40e20e612d5c1badca522a0027e9233a7 Author: Nils Goroll Date: Wed Nov 21 19:12:47 2018 +0100 gc stray comment Ref 3c0b8768afda0cc657fb902d1d390bdd0e8b7ce5 diff --git a/bin/varnishd/cache/cache_tcp_pool.c b/bin/varnishd/cache/cache_tcp_pool.c index 59725b142..10c24e0da 100644 --- a/bin/varnishd/cache/cache_tcp_pool.c +++ b/bin/varnishd/cache/cache_tcp_pool.c @@ -412,7 +412,6 @@ VCP_Open(struct conn_pool *cp, double tmo, const void **privp, int *err) h = 0; - /* stats access unprotected */ switch (errno) { case EACCES: case EPERM: From nils.goroll at uplex.de Thu Nov 22 12:29:14 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 22 Nov 2018 12:29:14 +0000 (UTC) Subject: [master] 8e2c82c00 assert that -I file processing has finished and closed the fd Message-ID: <20181122122914.8F5B310FE04@lists.varnish-cache.org> commit 8e2c82c006ebec1c3c438695e190daacc229919c Author: Nils Goroll Date: Thu Nov 22 10:28:43 2018 +0100 assert that -I file processing has finished and closed the fd diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c index d3fe23fa1..ceebbf92c 100644 --- a/bin/varnishd/mgt/mgt_main.c +++ b/bin/varnishd/mgt/mgt_main.c @@ -871,6 +871,7 @@ main(int argc, char * const *argv) "VEV_Once() = %d", o); } } + assert(I_fd == -1); if (!d_flag && !mgt_has_vcl() && !novcl) MGT_Complain(C_ERR, "No VCL loaded yet"); From nils.goroll at uplex.de Thu Nov 22 12:29:14 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 22 Nov 2018 12:29:14 +0000 (UTC) Subject: [master] 76773e7c5 fix insignificant off-by-one in child file descriptor close code Message-ID: <20181122122914.9C4C010FE06@lists.varnish-cache.org> commit 76773e7c51c44817de3a65a132806e5b525689b9 Author: Nils Goroll Date: Thu Nov 22 13:21:24 2018 +0100 fix insignificant off-by-one in child file descriptor close code The name CLOSE_FD_UP_TO implies that the value is included. Also this is the semantics of MCH_TrackHighFd() and relevant for #define CLOSE_FD_UP_TO mgt_max_fd diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c index 99485af83..5072db159 100644 --- a/bin/varnishd/mgt/mgt_child.c +++ b/bin/varnishd/mgt/mgt_child.c @@ -342,7 +342,7 @@ mgt_launch_child(struct cli *cli) */ closelog(); - for (i = STDERR_FILENO + 1; i < CLOSE_FD_UP_TO; i++) { + for (i = STDERR_FILENO + 1; i <= CLOSE_FD_UP_TO; i++) { if (vbit_test(fd_map, i)) continue; if (close(i) == 0) From nils.goroll at uplex.de Thu Nov 22 12:29:14 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 22 Nov 2018 12:29:14 +0000 (UTC) Subject: [master] 1be6ad1d5 Gain more confidence that our file descriptor tracking works Message-ID: <20181122122914.B211C10FE09@lists.varnish-cache.org> commit 1be6ad1d5a807038adcaa39168ce69a5a76bc8e1 Author: Nils Goroll Date: Thu Nov 22 13:14:19 2018 +0100 Gain more confidence that our file descriptor tracking works Side note #define CLOSE_FD_UP_TO mgt_max_fd exposes that we are currently missing to track some fds. diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c index 5072db159..f6c9c810d 100644 --- a/bin/varnishd/mgt/mgt_child.c +++ b/bin/varnishd/mgt/mgt_child.c @@ -181,11 +181,15 @@ mch_cli_panic_clear(struct cli *cli, const char * const *av, void *priv) * This is likely to a bit on the low side, as libc and other libraries * has a tendency to cache file descriptors (syslog, resolver, etc.) * so we add a margin of 100 fds. + * + * for added safety, we check that we see no file descriptor open for + * another margin above the limit for which we close by design */ static int mgt_max_fd; #define CLOSE_FD_UP_TO (mgt_max_fd + 100) +#define CHECK_FD_UP_TO (CLOSE_FD_UP_TO + 100) void MCH_TrackHighFd(int fd) @@ -348,6 +352,10 @@ mgt_launch_child(struct cli *cli) if (close(i) == 0) VFIL_null_fd(i); } + for (i = CLOSE_FD_UP_TO + 1; i <= CHECK_FD_UP_TO; i++) { + assert(close(i) == -1); + assert(errno == EBADF); + } mgt_ProcTitle("Child"); From nils.goroll at uplex.de Thu Nov 22 12:45:24 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 22 Nov 2018 12:45:24 +0000 (UTC) Subject: [master] 6bead571d fix the missing high fd tracking Message-ID: <20181122124524.36ADD11061C@lists.varnish-cache.org> commit 6bead571da6a2027a518af81e14d87a247f38156 Author: Nils Goroll Date: Thu Nov 22 13:44:50 2018 +0100 fix the missing high fd tracking diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c index f6c9c810d..6d3d7a09e 100644 --- a/bin/varnishd/mgt/mgt_child.c +++ b/bin/varnishd/mgt/mgt_child.c @@ -189,6 +189,8 @@ mch_cli_panic_clear(struct cli *cli, const char * const *av, void *priv) static int mgt_max_fd; #define CLOSE_FD_UP_TO (mgt_max_fd + 100) +// XXX should work now - engage? +//#define CLOSE_FD_UP_TO mgt_max_fd #define CHECK_FD_UP_TO (CLOSE_FD_UP_TO + 100) void @@ -213,6 +215,7 @@ MCH_Fd_Inherit(int fd, const char *what) { assert(fd >= 0); + MCH_TrackHighFd(fd); if (fd_map == NULL) fd_map = vbit_new(128); AN(fd_map); From nils.goroll at uplex.de Thu Nov 22 16:26:08 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 22 Nov 2018 16:26:08 +0000 (UTC) Subject: [master] f3e9c54cf Fix regression from 6bead571da6a2027a518af81e14d87a247f38156 Message-ID: <20181122162608.6FC8811441A@lists.varnish-cache.org> commit f3e9c54cf7a1ecc7aab1bffc15219849174c6352 Author: Nils Goroll Date: Thu Nov 22 17:22:19 2018 +0100 Fix regression from 6bead571da6a2027a518af81e14d87a247f38156 Seen once, and failed to reprocuce (yet), needs to be understood better from vmod_blobdigest tests/usage.vtc: **** v1 1.8 CLI RX|No panic to clear *** v1 1.8 debug|Info: manager stopping child *** v1 1.8 debug|Debug: Stopping Child **** v1 1.9 vsl| 0 CLI - EOF on CLI connection, worker stops *** v1 2.8 debug|Info: Child (174227) ended *** v1 2.8 debug|Info: Child (174227) said Child dies *** v1 2.8 debug|Debug: Child cleanup complete *** v1 2.8 debug|Assert error in MCH_TrackHighFd(), mgt/mgt_child.c line 203: *** v1 2.8 debug| Condition(fd > 0) not true. **** v1 2.9 STDOUT poll 0x10 ** v1 2.9 WAIT4 pid=174215 status=0x0086 (user 0.914949 sys 0.567445) * v1 2.9 Expected exit: 0x0 signal: 0 core: 0 ---- v1 2.9 Bad exit status: 0x0086 exit 0x0 signal 6 core 128 * top 2.9 failure during reset diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c index 6d3d7a09e..e6fd7c514 100644 --- a/bin/varnishd/mgt/mgt_child.c +++ b/bin/varnishd/mgt/mgt_child.c @@ -215,7 +215,9 @@ MCH_Fd_Inherit(int fd, const char *what) { assert(fd >= 0); - MCH_TrackHighFd(fd); + // XXX why? + if (fd > 0) + MCH_TrackHighFd(fd); if (fd_map == NULL) fd_map = vbit_new(128); AN(fd_map); From dridi.boukelmoune at gmail.com Sat Nov 24 00:25:10 2018 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Sat, 24 Nov 2018 00:25:10 +0000 (UTC) Subject: [master] 2bd3cea06 More double to vtim_* conversions Message-ID: <20181124002510.9F7CBA94D4@lists.varnish-cache.org> commit 2bd3cea06afbff0aeeb00d04dbbdda5faf4779d1 Author: Dridi Boukelmoune Date: Sat Nov 24 01:07:09 2018 +0100 More double to vtim_* conversions This may break out of tree code not respecting include order of vdef.h first (or via cache/cache.h). It's trivial to fix and forces consumers to follow the tracks. Refs #2791 diff --git a/bin/varnishd/cache/cache_panic.c b/bin/varnishd/cache/cache_panic.c index 74c1973dd..c1da769d8 100644 --- a/bin/varnishd/cache/cache_panic.c +++ b/bin/varnishd/cache/cache_panic.c @@ -34,9 +34,6 @@ #include #include -#include "vtim.h" -#include "vcs.h" - #include "cache_varnishd.h" #include "cache_transport.h" @@ -46,6 +43,8 @@ #include "storage/storage.h" #include "vcli_serve.h" +#include "vtim.h" +#include "vcs.h" /* * The panic string is constructed in memory, then copied to the diff --git a/bin/varnishd/cache/cache_session.c b/bin/varnishd/cache/cache_session.c index 4d074747b..c85331285 100644 --- a/bin/varnishd/cache/cache_session.c +++ b/bin/varnishd/cache/cache_session.c @@ -254,8 +254,8 @@ enum htc_status_e HTC_RxStuff(struct http_conn *htc, htc_complete_f *func, double *t1, double *t2, double ti, double tn, int maxbytes) { - double tmo; - double now; + vtim_dur tmo; + vtim_real now; enum htc_status_e hs; ssize_t z; @@ -380,7 +380,7 @@ SES_New(struct pool *pp) */ static void v_matchproto_(waiter_handle_f) -ses_handle(struct waited *wp, enum wait_event ev, double now) +ses_handle(struct waited *wp, enum wait_event ev, vtim_real now) { struct sess *sp; struct pool *pp; @@ -517,7 +517,7 @@ SES_Close(struct sess *sp, enum sess_close reason) */ void -SES_Delete(struct sess *sp, enum sess_close reason, double now) +SES_Delete(struct sess *sp, enum sess_close reason, vtim_real now) { CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); diff --git a/bin/varnishd/cache/cache_varnishd.h b/bin/varnishd/cache/cache_varnishd.h index 08caa721b..a5a3d185b 100644 --- a/bin/varnishd/cache/cache_varnishd.h +++ b/bin/varnishd/cache/cache_varnishd.h @@ -75,8 +75,8 @@ struct http_conn { void *priv; /* Timeouts */ - double first_byte_timeout; - double between_bytes_timeout; + vtim_dur first_byte_timeout; + vtim_dur between_bytes_timeout; }; typedef enum htc_status_e htc_complete_f(struct http_conn *); diff --git a/bin/varnishtest/vtc_haproxy.c b/bin/varnishtest/vtc_haproxy.c index 9d12ed671..846b5c8a8 100644 --- a/bin/varnishtest/vtc_haproxy.c +++ b/bin/varnishtest/vtc_haproxy.c @@ -105,7 +105,7 @@ struct haproxy_cli { size_t rxbuf_sz; char *rxbuf; - double timeout; + vtim_dur timeout; }; /********************************************************************** @@ -113,7 +113,7 @@ struct haproxy_cli { */ static int -haproxy_cli_tcp_connect(struct vtclog *vl, const char *addr, double tmo, +haproxy_cli_tcp_connect(struct vtclog *vl, const char *addr, vtim_dur tmo, const char **errp) { int fd; diff --git a/bin/varnishtest/vtc_logexp.c b/bin/varnishtest/vtc_logexp.c index 4735472e2..43e300acf 100644 --- a/bin/varnishtest/vtc_logexp.c +++ b/bin/varnishtest/vtc_logexp.c @@ -116,11 +116,12 @@ #include "vapi/vsm.h" #include "vapi/vsl.h" -#include "vtim.h" -#include "vre.h" #include "vtc.h" +#include "vtim.h" +#include "vre.h" + #define LE_ANY (-1) #define LE_LAST (-2) diff --git a/bin/varnishtest/vtc_syslog.c b/bin/varnishtest/vtc_syslog.c index 6318b6e39..e415d8c92 100644 --- a/bin/varnishtest/vtc_syslog.c +++ b/bin/varnishtest/vtc_syslog.c @@ -36,13 +36,13 @@ #include #include +#include "vtc.h" + #include "vsa.h" #include "vss.h" #include "vtcp.h" #include "vre.h" -#include "vtc.h" - struct syslog_srv { unsigned magic; #define SYSLOG_SRV_MAGIC 0xbf28a692 @@ -62,7 +62,7 @@ struct syslog_srv { ssize_t rxbuf_left; size_t rxbuf_sz; char *rxbuf; - double timeout; + vtim_dur timeout; }; static pthread_mutex_t syslog_mtx; diff --git a/include/vrt.h b/include/vrt.h index 8aabdbf83..19bdb7b08 100644 --- a/include/vrt.h +++ b/include/vrt.h @@ -158,7 +158,7 @@ typedef const struct vmod_priv * VCL_BLOB; typedef const char * VCL_BODY; typedef unsigned VCL_BOOL; typedef int64_t VCL_BYTES; -typedef double VCL_DURATION; +typedef vtim_dur VCL_DURATION; typedef const char * VCL_ENUM; typedef const struct gethdr_s * VCL_HEADER; typedef struct http * VCL_HTTP; @@ -170,7 +170,7 @@ typedef double VCL_REAL; typedef const struct stevedore * VCL_STEVEDORE; typedef const struct strands * VCL_STRANDS; typedef const char * VCL_STRING; -typedef double VCL_TIME; +typedef vtim_real VCL_TIME; typedef struct vcl * VCL_VCL; typedef void VCL_VOID; @@ -213,7 +213,7 @@ struct vrt_ctx { VCL_HTTP http_bereq; VCL_HTTP http_beresp; - double now; + vtim_real now; /* * method specific argument: @@ -276,9 +276,9 @@ extern const void * const vrt_magic_string_unset; rigid char *port; \ rigid char *path; \ rigid char *hosthdr; \ - double connect_timeout; \ - double first_byte_timeout; \ - double between_bytes_timeout; \ + vtim_dur connect_timeout; \ + vtim_dur first_byte_timeout; \ + vtim_dur between_bytes_timeout; \ unsigned max_connections; \ unsigned proxy_header; @@ -307,8 +307,8 @@ struct vrt_backend { }; #define VRT_BACKEND_PROBE_FIELDS(rigid) \ - double timeout; \ - double interval; \ + vtim_dur timeout; \ + vtim_dur interval; \ unsigned exp_status; \ unsigned window; \ unsigned threshold; \ diff --git a/include/vtcp.h b/include/vtcp.h index 293759392..29f83cb75 100644 --- a/include/vtcp.h +++ b/include/vtcp.h @@ -55,13 +55,13 @@ void VTCP_name(const struct suckaddr *addr, char *abuf, unsigned alen, char *pbuf, unsigned plen); int VTCP_connected(int s); int VTCP_connect(const struct suckaddr *name, int msec); -int VTCP_open(const char *addr, const char *def_port, double timeout, +int VTCP_open(const char *addr, const char *def_port, vtim_dur timeout, const char **err); void VTCP_close(int *s); int VTCP_bind(const struct suckaddr *addr, const char **errp); int VTCP_listen(const struct suckaddr *addr, int depth, const char **errp); int VTCP_listen_on(const char *addr, const char *def_port, int depth, const char **errp); -void VTCP_set_read_timeout(int s, double seconds); -int VTCP_read(int fd, void *ptr, size_t len, double tmo); +void VTCP_set_read_timeout(int s, vtim_dur seconds); +int VTCP_read(int fd, void *ptr, size_t len, vtim_dur tmo); // #endif diff --git a/include/vtim.h b/include/vtim.h index 9c2e6ccfc..1968bc568 100644 --- a/include/vtim.h +++ b/include/vtim.h @@ -33,8 +33,8 @@ extern unsigned VTIM_postel; #define VTIM_FORMAT_SIZE 30 void VTIM_format(double t, char *p); double VTIM_parse(const char *p); -double VTIM_mono(void); -double VTIM_real(void); -void VTIM_sleep(double t); -struct timespec VTIM_timespec(double t); -struct timeval VTIM_timeval(double t); +vtim_mono VTIM_mono(void); +vtim_real VTIM_real(void); +void VTIM_sleep(vtim_dur t); +struct timespec VTIM_timespec(vtim_dur t); +struct timeval VTIM_timeval(vtim_dur t); diff --git a/lib/libvarnish/vev.c b/lib/libvarnish/vev.c index 6e4e48c7d..e6459bc7d 100644 --- a/lib/libvarnish/vev.c +++ b/lib/libvarnish/vev.c @@ -353,7 +353,7 @@ VEV_Loop(struct vev_root *evb) /*--------------------------------------------------------------------*/ static int -vev_sched_timeout(struct vev_root *evb, struct vev *e, double t) +vev_sched_timeout(struct vev_root *evb, struct vev *e, vtim_mono t) { int i; diff --git a/lib/libvarnish/vtcp.c b/lib/libvarnish/vtcp.c index 3294bbf08..972595ea0 100644 --- a/lib/libvarnish/vtcp.c +++ b/lib/libvarnish/vtcp.c @@ -51,6 +51,7 @@ #include "vsa.h" #include "vss.h" #include "vtcp.h" +#include "vtim.h" /*--------------------------------------------------------------------*/ static void @@ -345,12 +346,10 @@ VTCP_close(int *s) } void -VTCP_set_read_timeout(int s, double seconds) +VTCP_set_read_timeout(int s, vtim_dur seconds) { #ifdef SO_RCVTIMEO_WORKS - struct timeval timeout; - timeout.tv_sec = (int)floor(seconds); - timeout.tv_usec = (int)(1e6 * (seconds - timeout.tv_sec)); + struct timeval timeout = VTIM_timeval(seconds); /* * Solaris bug (present at least in snv_151 and older): If this fails * with EINVAL, the socket is half-closed (SS_CANTSENDMORE) and the @@ -371,13 +370,14 @@ VTCP_set_read_timeout(int s, double seconds) static int v_matchproto_(vss_resolved_f) vtcp_open_callback(void *priv, const struct suckaddr *sa) { + /* XXX: vtim_dur? */ double *p = priv; return (VTCP_connect(sa, (int)floor(*p * 1e3))); } int -VTCP_open(const char *addr, const char *def_port, double timeout, +VTCP_open(const char *addr, const char *def_port, vtim_dur timeout, const char **errp) { int error; @@ -592,7 +592,7 @@ VTCP_Check(int a) */ int -VTCP_read(int fd, void *ptr, size_t len, double tmo) +VTCP_read(int fd, void *ptr, size_t len, vtim_dur tmo) { struct pollfd pfd[1]; int i, j; From daghf at varnish-software.com Mon Nov 26 12:58:08 2018 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Mon, 26 Nov 2018 12:58:08 +0000 (UTC) Subject: [master] 05f3ee5ef Fix VRT_fail for 'if'/'elseif' conditional expressions Message-ID: <20181126125808.2EF1E9E191@lists.varnish-cache.org> commit 05f3ee5ef2cc08ec6363a5604ca07d8f223ce1aa Author: Dag Haavi Finstad Date: Thu Nov 22 15:43:14 2018 +0100 Fix VRT_fail for 'if'/'elseif' conditional expressions This adds a ctx->handling test at the beginning of any compound statement, to catch the cases where VRT_fail was invoked as part of an if test condition. Fixes: #2840 diff --git a/bin/varnishtest/tests/v00051.vtc b/bin/varnishtest/tests/v00051.vtc index 664af71f7..a12d23ade 100644 --- a/bin/varnishtest/tests/v00051.vtc +++ b/bin/varnishtest/tests/v00051.vtc @@ -425,3 +425,32 @@ varnish v1 -expect sc_vcl_failure == 10 logexpect l1 -wait +####################################################################### +# Fail in vmod call used in an if test + +varnish v1 -vcl+backend { + import debug; + sub vcl_recv { + if (debug.fail2()) { + return (hash); + } + } +} + +logexpect l1 -v v1 -g raw { + expect * 1033 VCL_call "RECV" + expect 0 1033 VCL_Error "Forced failure" + expect 0 1033 VCL_return "fail" +} -start + +client c1 { + txreq + rxresp + expect resp.status == 503 + expect resp.reason == "VCL failed" +} -run + +varnish v1 -expect vcl_fail == 14 +varnish v1 -expect sc_vcl_failure == 11 + +logexpect l1 -wait diff --git a/lib/libvcc/vcc_parse.c b/lib/libvcc/vcc_parse.c index ee8830859..2b5075358 100644 --- a/lib/libvcc/vcc_parse.c +++ b/lib/libvcc/vcc_parse.c @@ -149,6 +149,7 @@ vcc_Compound(struct vcc *tl) Fb(tl, 1, "{\n"); tl->indent += INDENT; C(tl, ";"); + Fb(tl, 1, "if (*ctx->handling) return;\n"); while (1) { ERRCHK(tl); t = tl->t; diff --git a/lib/libvmod_debug/vmod.vcc b/lib/libvmod_debug/vmod.vcc index 3e0cc4faa..590971aa1 100644 --- a/lib/libvmod_debug/vmod.vcc +++ b/lib/libvmod_debug/vmod.vcc @@ -130,6 +130,10 @@ $Function VOID fail() Function to fail vcl code. (See also: RFC748) +$Function BOOL fail2() + +Function to fail vcl code. Always returns true. + $Object dyn(STRING addr, STRING port, PROBE probe=0) Dynamically create a single-backend director, addr and port must not be empty. diff --git a/lib/libvmod_debug/vmod_debug.c b/lib/libvmod_debug/vmod_debug.c index 8851ae6e4..474bc064f 100644 --- a/lib/libvmod_debug/vmod_debug.c +++ b/lib/libvmod_debug/vmod_debug.c @@ -252,6 +252,14 @@ xyzzy_fail(VRT_CTX) VRT_fail(ctx, "Forced failure"); } +VCL_BOOL v_matchproto_(td_debug_fail2) +xyzzy_fail2(VRT_CTX) +{ + + VRT_fail(ctx, "Forced failure"); + return (1); +} + static void v_matchproto_(vmod_priv_free_f) priv_vcl_free(void *priv) { From daghf at varnish-software.com Mon Nov 26 13:36:08 2018 From: daghf at varnish-software.com (Dag Haavi Finstad) Date: Mon, 26 Nov 2018 13:36:08 +0000 (UTC) Subject: [master] b2b04a617 Add VSL rate limiting Message-ID: <20181126133608.44B82A0261@lists.varnish-cache.org> commit b2b04a617d83a834e8739ddde897ef7211945c8a Author: Dag Haavi Finstad Date: Tue Nov 20 11:16:51 2018 +0100 Add VSL rate limiting This adds rate limiting to varnishncsa and varnishlog. Rate limiting is done on a per-transaction basis, respective to the grouping mode selected. I.e. for -g request the limit will apply on a per-request basis, -g raw on a per-record basis, etc. Rate limit is specified as -R [/duration]. Default period if not specified is seconds ('s'). diff --git a/bin/varnishlog/varnishlog_options.h b/bin/varnishlog/varnishlog_options.h index 417b76432..ea0b4c698 100644 --- a/bin/varnishlog/varnishlog_options.h +++ b/bin/varnishlog/varnishlog_options.h @@ -70,6 +70,7 @@ VUT_OPT_n VUT_GLOBAL_OPT_P VUT_OPT_q VUT_OPT_r +VSL_OPT_R VUT_OPT_t VSL_OPT_T VSL_OPT_v diff --git a/bin/varnishncsa/varnishncsa_options.h b/bin/varnishncsa/varnishncsa_options.h index 16bcdb1cd..73765083f 100644 --- a/bin/varnishncsa/varnishncsa_options.h +++ b/bin/varnishncsa/varnishncsa_options.h @@ -88,6 +88,7 @@ VUT_OPT_n VUT_GLOBAL_OPT_P VUT_OPT_q VUT_OPT_r +VSL_OPT_R VUT_OPT_t VUT_GLOBAL_OPT_V NCSA_OPT_w diff --git a/include/vapi/vapi_options.h b/include/vapi/vapi_options.h index 3d95c3123..a357abd28 100644 --- a/include/vapi/vapi_options.h +++ b/include/vapi/vapi_options.h @@ -77,6 +77,22 @@ " running queries. Defaults to 1000 transactions." \ ) +#define VSL_OPT_R \ + VOPT("R:", "[-R ]", "Output rate limit", \ + "Restrict the output to the specified limit." \ + " Transactions exceeding the limit will be suppressed." \ + " The limit is specified as the maximum number of" \ + " transactions (with respect to the chosen grouping" \ + " method) and an optional time period. If no duration" \ + " is specified, a default of ``s`` is used. The duration" \ + " field can be formatted as in VCL (e.g. ``-R 10/2m``) or" \ + " as a simple time period without the prefix (e.g." \ + " ``-R 5/m``)." \ + " When in ``-g raw`` grouping mode, this setting can" \ + " not be used alongside ``-i``, ``-I``, ``-x`` or " \ + "``-X``, and we advise using ``-q`` instead." \ + ) + #define VSL_OPT_T \ VOPT("T:", "[-T ]", "Transaction end timeout", \ "Sets the transaction timeout in seconds. This defines the" \ diff --git a/lib/libvarnishapi/vsl_api.h b/lib/libvarnishapi/vsl_api.h index 0e265c0b1..8817aa8fe 100644 --- a/lib/libvarnishapi/vsl_api.h +++ b/lib/libvarnishapi/vsl_api.h @@ -84,6 +84,8 @@ struct VSL_data { int c_opt; int C_opt; int L_opt; + int R_opt_l; + vtim_dur R_opt_p; double T_opt; int v_opt; }; diff --git a/lib/libvarnishapi/vsl_arg.c b/lib/libvarnishapi/vsl_arg.c index b4bef5adb..2929b4ccf 100644 --- a/lib/libvarnishapi/vsl_arg.c +++ b/lib/libvarnishapi/vsl_arg.c @@ -300,6 +300,41 @@ vsl_IX_arg(struct VSL_data *vsl, int opt, const char *arg) return (1); } +static int +vsl_R_arg(struct VSL_data *vsl, const char *arg) +{ + char buf[32] = ""; + char *p; + long l; + + AN(arg); + CHECK_OBJ_NOTNULL(vsl, VSL_MAGIC); + + l = strtol(arg, &p, 0); + if (l <= 0 || l > INT_MAX) + return (vsl_diag(vsl, "-R: Range error")); + vsl->R_opt_l = l; + assert(p != arg); + AN(p); + if (*p == '\0') { + vsl->R_opt_p = 1.0; + return (1); + } + if (*p != '/' || p[1] == '\0') + return (vsl_diag(vsl, "-R: Syntax error")); + p++; + if (strlen(p) > sizeof(buf) - 2) + return (vsl_diag(vsl, "-R: Syntax error")); + if (!isdigit(*p)) + strcat(buf, "1"); + strcat(buf, p); + vsl->R_opt_p = VNUM_duration(buf); + if (isnan(vsl->R_opt_p) || vsl->R_opt_p <= 0.0) + return (vsl_diag(vsl, + "-R: Syntax error: Invalid duration")); + return (1); +} + int VSL_Arg(struct VSL_data *vsl, int opt, const char *arg) { @@ -334,6 +369,8 @@ VSL_Arg(struct VSL_data *vsl, int opt, const char *arg) return (vsl_diag(vsl, "-L: Range error")); vsl->L_opt = (int)l; return (1); + case 'R': + return (vsl_R_arg(vsl, arg)); case 'T': AN(arg); d = VNUM(arg); diff --git a/lib/libvarnishapi/vsl_dispatch.c b/lib/libvarnishapi/vsl_dispatch.c index fa397ae0d..57c45168f 100644 --- a/lib/libvarnishapi/vsl_dispatch.c +++ b/lib/libvarnishapi/vsl_dispatch.c @@ -194,6 +194,10 @@ struct VSLQ { VTAILQ_HEAD(,vtx) cache; unsigned n_cache; + /* Rate limiting */ + double credits; + vtim_mono last_use; + /* Raw mode */ struct { struct vslc_raw c; @@ -908,10 +912,33 @@ vtx_force(struct VSLQ *vslq, struct vtx *vtx, const char *reason) AN(vtx->flags & VTX_F_COMPLETE); } +static int +vslq_ratelimit(struct VSLQ *vslq) +{ + vtim_mono now; + vtim_dur delta; + + CHECK_OBJ_NOTNULL(vslq, VSLQ_MAGIC); + CHECK_OBJ_NOTNULL(vslq->vsl, VSL_MAGIC); + + now = VTIM_mono(); + delta = now - vslq->last_use; + vslq->credits += (delta / vslq->vsl->R_opt_p) * vslq->vsl->R_opt_l; + if (vslq->credits > vslq->vsl->R_opt_l) + vslq->credits = vslq->vsl->R_opt_l; + vslq->last_use = now; + + if (vslq->credits < 1.0) + return (0); + + vslq->credits -= 1.0; + return (1); +} + /* Build transaction array, do the query and callback. Returns 0 or the return value from func */ static int -vslq_callback(const struct VSLQ *vslq, struct vtx *vtx, VSLQ_dispatch_f *func, +vslq_callback(struct VSLQ *vslq, struct vtx *vtx, VSLQ_dispatch_f *func, void *priv) { unsigned n = vtx->n_descend + 1; @@ -973,6 +1000,9 @@ vslq_callback(const struct VSLQ *vslq, struct vtx *vtx, VSLQ_dispatch_f *func, if (vslq->query != NULL && !vslq_runquery(vslq->query, ptrans)) return (0); + if (vslq->vsl->R_opt_l != 0 && !vslq_ratelimit(vslq)) + return (0); + /* Callback */ return ((func)(vslq->vsl, ptrans, priv)); } @@ -1078,6 +1108,10 @@ VSLQ_New(struct VSL_data *vsl, struct VSL_cursor **cp, } vslq->grouping = grouping; vslq->query = query; + if (vslq->vsl->R_opt_l != 0) { + vslq->last_use = VTIM_mono(); + vslq->credits = 1; + } /* Setup normal mode */ VRBT_INIT(&vslq->tree); @@ -1195,6 +1229,9 @@ vslq_raw(struct VSLQ *vslq, VSLQ_dispatch_f *func, void *priv) !vslq_runquery(vslq->query, vslq->raw.ptrans)) return (r); + if (vslq->vsl->R_opt_l != 0 && !vslq_ratelimit(vslq)) + return (r); + i = (func)(vslq->vsl, vslq->raw.ptrans, priv); if (i) return (i); From phk at FreeBSD.org Tue Nov 27 16:17:11 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 27 Nov 2018 16:17:11 +0000 (UTC) Subject: [master] 80f09e4f7 More use of tokenlist Message-ID: <20181127161712.1752F9557D@lists.varnish-cache.org> commit 80f09e4f73ff772ac92ed9ad74f738537b98bee1 Author: Poul-Henning Kamp Date: Tue Nov 27 14:05:09 2018 +0000 More use of tokenlist diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py index 5061be2d9..19d8b1fa9 100755 --- a/lib/libvcc/vmodtool.py +++ b/lib/libvcc/vmodtool.py @@ -591,10 +591,17 @@ class stanza(object): class s_module(stanza): + + ''' $Module modname man_section description ... ''' + def parse(self): + if len(self.toks) < 4: + self.syntax() a = self.line[1].split(None, 2) - self.vcc.modname = a[0] - self.vcc.mansection = a[1] + self.vcc.modname = self.toks[1] + self.vcc.mansection = self.toks[2] + # XXX: Find better solution for moddesc + self.vcc.moddesc = " ".join(self.toks[2:]) self.vcc.moddesc = a[2] self.rstlbl = "vmod_%s(%s)" % ( self.vcc.modname, @@ -678,17 +685,31 @@ class s_prefix(stanza): class s_synopsis(stanza): + + ''' $Synopsis [auto|manual] ''' + def parse(self): - if self.line[1] not in ('auto', 'manual'): + if len(self.toks) != 2: + self.syntax() + valid = { + 'auto': True, + 'manual': False, + } + self.vcc.auto_synopsis = vald.get(self.toks[1]) + if self.vcc.auto_synopsis is None: err("Valid Synopsis values are 'auto' or 'manual', got '%s'\n" % - self.line[1]) - self.vcc.auto_synopsis = self.line[1] == 'auto' + self.toks[1]) self.vcc.contents.append(self) class s_event(stanza): + + ''' $Event function_name ''' + def parse(self): - self.event_func = self.line[1] + if len(self.toks) != 2: + self.syntax() + self.event_func = self.toks[1] self.vcc.contents.append(self) def rstfile(self, fo, man): From phk at FreeBSD.org Tue Nov 27 16:17:11 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 27 Nov 2018 16:17:11 +0000 (UTC) Subject: [master] 24d6007d8 Add a real tokenizer (only little use yet) Message-ID: <20181127161712.085B695542@lists.varnish-cache.org> commit 24d6007d8117dc94e472d7fcaa76a9c2e3cce2a2 Author: Poul-Henning Kamp Date: Tue Nov 27 13:21:32 2018 +0000 Add a real tokenizer (only little use yet) Make reporting syntax errors easier. diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py index 7db51c668..5061be2d9 100755 --- a/lib/libvcc/vmodtool.py +++ b/lib/libvcc/vmodtool.py @@ -529,7 +529,8 @@ class ProtoType(object): class stanza(object): - def __init__(self, l0, doc, vcc): + def __init__(self, toks, l0, doc, vcc): + self.toks = toks self.line = l0 while doc and doc[0] == '': doc.pop(0) @@ -545,6 +546,12 @@ class stanza(object): def dump(self): print(type(self), self.line) + def syntax(self): + err("Syntax error.\n" + + "\tShould be: " + self.__doc__.strip() + "\n" + + "\tIs: " + " ".join(self.toks) + "\n", + warn=False) + def rstfile(self, fo, man): if self.rstlbl is not None: fo.write(".. _" + self.rstlbl + ":\n\n") @@ -642,17 +649,31 @@ class s_module(stanza): class s_abi(stanza): + + ''' $ABI [strict|vrt] ''' + def parse(self): - if self.line[1] not in ('strict', 'vrt'): + if len(self.toks) != 2: + self.syntax() + valid = { + 'strict': True, + 'vrt': False, + } + self.vcc.strict_abi = valid.get(self.toks[1]) + if self.vcc.strict_abi is None: err("Valid ABI types are 'strict' or 'vrt', got '%s'\n" % - self.line[1]) - self.vcc.strict_abi = self.line[1] == 'strict' + self.toks[1]) self.vcc.contents.append(self) class s_prefix(stanza): + + ''' $Prefix symbol ''' + def parse(self): - self.vcc.sympfx = self.line[1] + "_" + if len(self.toks) != 2: + self.syntax() + self.vcc.sympfx = self.toks[1] + "_" self.vcc.contents.append(self) @@ -834,6 +855,43 @@ DISPATCH = { "Synopsis": s_synopsis, } +def tokenize(str, seps=None, quotes=None): + if seps is None: + seps = "[](){},=" + if quotes is None: + quotes = '"' + "'" + quote = None + out = [] + i = 0 + inside = False + while i < len(str): + c = str[i] + # print("T", [c], quote, inside, i) + i += 1 + if quote is not None and c == quote: + inside = False + quote = None + out[-1] += c + elif quote is not None: + out[-1] += c + elif c.isspace(): + inside = False + elif seps.find(c) >= 0: + inside = False + out.append(c) + elif quotes.find(c) >= 0: + quote = c + out.append(c) + elif inside: + out[-1] += c + else: + out.append(c) + inside = True + #print("TOK", [str]) + #for i in out: + # print("\t", [i]) + return out + class vcc(object): def __init__(self, inputvcc, rstdir, outputprefix): @@ -866,12 +924,13 @@ class vcc(object): self.copyright = s.pop(0).strip() while s: ss = re.split('\n([^\t ])', s.pop(0), maxsplit=1) + toks = tokenize(ss[0]) c = ss[0].split() d = "".join(ss[1:]) - m = DISPATCH.get(c[0]) + m = DISPATCH.get(toks[0]) if m is None: err("Unknown stanze $%s" % ss[:i]) - m([c[0], " ".join(c[1:])], d.split('\n'), self) + m(toks, [c[0], " ".join(c[1:])], d.split('\n'), self) inputline = None def rst_copyright(self, fo): From dridi at varni.sh Tue Nov 27 16:22:08 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 27 Nov 2018 17:22:08 +0100 Subject: [master] 80f09e4f7 More use of tokenlist In-Reply-To: <20181127161712.1752F9557D@lists.varnish-cache.org> References: <20181127161712.1752F9557D@lists.varnish-cache.org> Message-ID: On Tue, Nov 27, 2018 at 5:17 PM Poul-Henning Kamp wrote: > > > commit 80f09e4f73ff772ac92ed9ad74f738537b98bee1 > Author: Poul-Henning Kamp > Date: Tue Nov 27 14:05:09 2018 +0000 > > More use of tokenlist > > diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py > index 5061be2d9..19d8b1fa9 100755 > --- a/lib/libvcc/vmodtool.py > +++ b/lib/libvcc/vmodtool.py > @@ -591,10 +591,17 @@ class stanza(object): > > > class s_module(stanza): > + > + ''' $Module modname man_section description ... ''' > + > def parse(self): > + if len(self.toks) < 4: > + self.syntax() > a = self.line[1].split(None, 2) > - self.vcc.modname = a[0] > - self.vcc.mansection = a[1] > + self.vcc.modname = self.toks[1] > + self.vcc.mansection = self.toks[2] > + # XXX: Find better solution for moddesc > + self.vcc.moddesc = " ".join(self.toks[2:]) If the tokenizer supports quoting, require that $Module stanzas look like: $Module foo 3 "Foo module for Varnish" Breaking change though. > self.vcc.moddesc = a[2] > self.rstlbl = "vmod_%s(%s)" % ( > self.vcc.modname, > @@ -678,17 +685,31 @@ class s_prefix(stanza): > > > class s_synopsis(stanza): > + > + ''' $Synopsis [auto|manual] ''' > + > def parse(self): > - if self.line[1] not in ('auto', 'manual'): > + if len(self.toks) != 2: > + self.syntax() > + valid = { > + 'auto': True, > + 'manual': False, > + } > + self.vcc.auto_synopsis = vald.get(self.toks[1]) > + if self.vcc.auto_synopsis is None: > err("Valid Synopsis values are 'auto' or 'manual', got '%s'\n" % > - self.line[1]) > - self.vcc.auto_synopsis = self.line[1] == 'auto' > + self.toks[1]) > self.vcc.contents.append(self) > > > class s_event(stanza): > + > + ''' $Event function_name ''' > + > def parse(self): > - self.event_func = self.line[1] > + if len(self.toks) != 2: > + self.syntax() > + self.event_func = self.toks[1] > self.vcc.contents.append(self) > > def rstfile(self, fo, man): > _______________________________________________ > varnish-commit mailing list > varnish-commit at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-commit From nils.goroll at uplex.de Tue Nov 27 16:35:11 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 27 Nov 2018 16:35:11 +0000 (UTC) Subject: [master] 7395ecae8 fix typo Message-ID: <20181127163511.9820F95F41@lists.varnish-cache.org> commit 7395ecae8fdfaefca9ee5438d35ea6101c7a9f56 Author: Nils Goroll Date: Tue Nov 27 17:34:50 2018 +0100 fix typo diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py index 19d8b1fa9..dd0e9f4dc 100755 --- a/lib/libvcc/vmodtool.py +++ b/lib/libvcc/vmodtool.py @@ -695,7 +695,7 @@ class s_synopsis(stanza): 'auto': True, 'manual': False, } - self.vcc.auto_synopsis = vald.get(self.toks[1]) + self.vcc.auto_synopsis = valid.get(self.toks[1]) if self.vcc.auto_synopsis is None: err("Valid Synopsis values are 'auto' or 'manual', got '%s'\n" % self.toks[1]) From phk at FreeBSD.org Tue Nov 27 23:32:08 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 27 Nov 2018 23:32:08 +0000 (UTC) Subject: [master] 0c74d1b99 Implement a better solution: Prefer a single quoted token for $Module description, and suggest people to do it that way. Message-ID: <20181127233208.BFD24A1394@lists.varnish-cache.org> commit 0c74d1b99f1f43fc99725fe14322760afa7858ee Author: Poul-Henning Kamp Date: Tue Nov 27 22:17:39 2018 +0000 Implement a better solution: Prefer a single quoted token for $Module description, and suggest people to do it that way. diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py index dd0e9f4dc..0184e5f24 100755 --- a/lib/libvcc/vmodtool.py +++ b/lib/libvcc/vmodtool.py @@ -120,6 +120,15 @@ CTYPES.update(PRIVS) ####################################################################### +def is_quoted(str): + return len(str) > 2 and str[0] == str[-1] and str[0] in ('"', "'") + +def unquote(str): + assert is_quoted(str) + return str[1:-1] + +####################################################################### + def write_file_warning(fo, a, b, c): fo.write(a + "\n") @@ -600,9 +609,11 @@ class s_module(stanza): a = self.line[1].split(None, 2) self.vcc.modname = self.toks[1] self.vcc.mansection = self.toks[2] - # XXX: Find better solution for moddesc - self.vcc.moddesc = " ".join(self.toks[2:]) - self.vcc.moddesc = a[2] + if len(self.toks) == 4 and is_quoted(self.toks[3]): + self.vcc.moddesc = unquote(self.toks[3]) + else: + print("\nNOTICE: Please put $Module description in quotes.\n") + self.vcc.moddesc = " ".join(self.toks[3:]) self.rstlbl = "vmod_%s(%s)" % ( self.vcc.modname, self.vcc.mansection diff --git a/lib/libvmod_blob/vmod.vcc b/lib/libvmod_blob/vmod.vcc index a6514dd2e..1ea3716d4 100644 --- a/lib/libvmod_blob/vmod.vcc +++ b/lib/libvmod_blob/vmod.vcc @@ -6,7 +6,7 @@ # Geoffrey Simmons # -$Module blob 3 utilities for the VCL blob type +$Module blob 3 "Utilities for the VCL blob type" $ABI strict diff --git a/lib/libvmod_debug/vmod.vcc b/lib/libvmod_debug/vmod.vcc index 590971aa1..c4c221a3f 100644 --- a/lib/libvmod_debug/vmod.vcc +++ b/lib/libvmod_debug/vmod.vcc @@ -26,7 +26,7 @@ # SUCH DAMAGE. # -$Module debug 3 Development, test and debug +$Module debug 3 "Development, test and debug" $ABI strict $Prefix xyzzy DESCRIPTION diff --git a/lib/libvmod_directors/vmod.vcc b/lib/libvmod_directors/vmod.vcc index 8fecc08e7..54b111fba 100644 --- a/lib/libvmod_directors/vmod.vcc +++ b/lib/libvmod_directors/vmod.vcc @@ -32,7 +32,7 @@ # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. -$Module directors 3 Varnish Directors Module +$Module directors 3 "Varnish Directors Module" $ABI strict DESCRIPTION diff --git a/lib/libvmod_proxy/vmod.vcc b/lib/libvmod_proxy/vmod.vcc index d986b5295..34b1199b2 100644 --- a/lib/libvmod_proxy/vmod.vcc +++ b/lib/libvmod_proxy/vmod.vcc @@ -25,7 +25,7 @@ # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. -$Module proxy 3 Varnish Module to extract TLV attributes from PROXYv2 +$Module proxy 3 "Varnish Module to extract TLV attributes from PROXYv2" $ABI strict DESCRIPTION diff --git a/lib/libvmod_purge/vmod.vcc b/lib/libvmod_purge/vmod.vcc index 25a9cb801..e158aac69 100644 --- a/lib/libvmod_purge/vmod.vcc +++ b/lib/libvmod_purge/vmod.vcc @@ -25,7 +25,7 @@ # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. -$Module purge 3 Varnish Purge Module +$Module purge 3 "Varnish Purge Module" $ABI strict DESCRIPTION diff --git a/lib/libvmod_std/vmod.vcc b/lib/libvmod_std/vmod.vcc index c30172ed1..fc5ef3736 100644 --- a/lib/libvmod_std/vmod.vcc +++ b/lib/libvmod_std/vmod.vcc @@ -25,7 +25,7 @@ # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. -$Module std 3 Varnish Standard Module +$Module std 3 "Varnish Standard Module" $ABI strict DESCRIPTION diff --git a/lib/libvmod_unix/vmod.vcc b/lib/libvmod_unix/vmod.vcc index e70a092d3..e227e9f99 100644 --- a/lib/libvmod_unix/vmod.vcc +++ b/lib/libvmod_unix/vmod.vcc @@ -5,7 +5,7 @@ # Authors: Geoffrey Simmons # -$Module unix 3 utilities for Unix domain sockets +$Module unix 3 "Utilities for Unix domain sockets" $ABI strict diff --git a/lib/libvmod_vtc/vmod.vcc b/lib/libvmod_vtc/vmod.vcc index c521f37da..94f8f75b6 100644 --- a/lib/libvmod_vtc/vmod.vcc +++ b/lib/libvmod_vtc/vmod.vcc @@ -25,7 +25,7 @@ # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. -$Module vtc 3 Utility module for varnishtest +$Module vtc 3 "Utility module for varnishtest" $ABI strict DESCRIPTION From phk at FreeBSD.org Tue Nov 27 23:32:08 2018 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 27 Nov 2018 23:32:08 +0000 (UTC) Subject: [master] 02186b8d3 Ditch the old prototype tokenizer for the new. Message-ID: <20181127233208.D49DDA1397@lists.varnish-cache.org> commit 02186b8d3870997311bb2c33e07a084c986abf32 Author: Poul-Henning Kamp Date: Tue Nov 27 22:56:20 2018 +0000 Ditch the old prototype tokenizer for the new. Other polish. diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py index 0184e5f24..5ad48c6ac 100755 --- a/lib/libvcc/vmodtool.py +++ b/lib/libvcc/vmodtool.py @@ -235,10 +235,8 @@ class CType(object): self.spec = [] while True: x = wl.pop(0) - if x[0] == '"' and x[-1] == '"': - x = x[1:-1] - elif x[0] == "'" and x[-1] == "'": - x = x[1:-1] + if is_quoted(x): + x = unquote(x) assert x self.spec.append(x) enums[x] = True @@ -268,6 +266,9 @@ class CType(object): class arg(CType): + + ''' Parse front of word list into argument ''' + def __init__(self, wl, argnames, enums, end): super(arg, self).__init__(wl, enums) @@ -289,10 +290,8 @@ class arg(CType): x = wl.pop(0) if self.vt == "ENUM": - if x[0] == '"' and x[-1] == '"': - x = x[1:-1] - elif x[0] == "'" and x[-1] == "'": - x = x[1:-1] + if is_quoted(x): + x = unquote(x) self.defval = x def json(self, jl): @@ -305,60 +304,13 @@ class arg(CType): ####################################################################### -def lex(l): - wl = [] - s = 0 - assert l - for i in range(len(l)): - c = l[i] - - if s == 0 and re.match('[0-9a-zA-Z_.-]', c): - wl.append(c) - s = 3 - continue - - if s == 3: - if re.match('[0-9a-zA-Z_.-]', c): - wl[-1] += c - continue - s = 0 - - if s == 0 and c in (' ', '\t', '\n', '\r'): - continue - - if s == 0 and c in ('[', '(', '{', '}', ')', ']', ',', '='): - wl.append(c) - elif s == 0 and c in ('"', "'"): - sep = c - s = 1 - wl.append(c) - elif s == 1: - if c == '\\': - s = 2 - else: - wl[-1] += c - if c == sep: - s = 0 - elif s == 2: - wl[-1] += c - s = 1 - else: - err("Syntax error at char %d '%s'" % (i, c), warn=False) - - if s != 0: - err("Syntax error at char %d '%s'" % (i, c), warn=False) - return wl - -####################################################################### - - class ProtoType(object): def __init__(self, st, retval=True, prefix=""): self.st = st self.obj = None self.args = [] self.argstruct = False - wl = lex(st.line[1]) + wl = self.st.toks[1:] if retval: self.retval = CType(wl, st.vcc.enums) @@ -606,7 +558,6 @@ class s_module(stanza): def parse(self): if len(self.toks) < 4: self.syntax() - a = self.line[1].split(None, 2) self.vcc.modname = self.toks[1] self.vcc.mansection = self.toks[2] if len(self.toks) == 4 and is_quoted(self.toks[3]): @@ -887,43 +838,6 @@ DISPATCH = { "Synopsis": s_synopsis, } -def tokenize(str, seps=None, quotes=None): - if seps is None: - seps = "[](){},=" - if quotes is None: - quotes = '"' + "'" - quote = None - out = [] - i = 0 - inside = False - while i < len(str): - c = str[i] - # print("T", [c], quote, inside, i) - i += 1 - if quote is not None and c == quote: - inside = False - quote = None - out[-1] += c - elif quote is not None: - out[-1] += c - elif c.isspace(): - inside = False - elif seps.find(c) >= 0: - inside = False - out.append(c) - elif quotes.find(c) >= 0: - quote = c - out.append(c) - elif inside: - out[-1] += c - else: - out.append(c) - inside = True - #print("TOK", [str]) - #for i in out: - # print("\t", [i]) - return out - class vcc(object): def __init__(self, inputvcc, rstdir, outputprefix): @@ -956,7 +870,8 @@ class vcc(object): self.copyright = s.pop(0).strip() while s: ss = re.split('\n([^\t ])', s.pop(0), maxsplit=1) - toks = tokenize(ss[0]) + toks = self.tokenize(ss[0]) + inputline = '$' + ' '.join(toks) c = ss[0].split() d = "".join(ss[1:]) m = DISPATCH.get(toks[0]) @@ -965,6 +880,44 @@ class vcc(object): m(toks, [c[0], " ".join(c[1:])], d.split('\n'), self) inputline = None + def tokenize(self, str, seps=None, quotes=None): + if seps is None: + seps = "[](){},=" + if quotes is None: + quotes = '"' + "'" + quote = None + out = [] + i = 0 + inside = False + while i < len(str): + c = str[i] + # print("T", [c], quote, inside, i) + i += 1 + if quote is not None and c == quote: + inside = False + quote = None + out[-1] += c + elif quote is not None: + out[-1] += c + elif c.isspace(): + inside = False + elif seps.find(c) >= 0: + inside = False + out.append(c) + elif quotes.find(c) >= 0: + quote = c + out.append(c) + elif inside: + out[-1] += c + else: + out.append(c) + inside = True + #print("TOK", [str]) + #for i in out: + # print("\t", [i]) + return out + + def rst_copyright(self, fo): write_rst_hdr(fo, "COPYRIGHT", "=") fo.write("\n::\n\n") From phk at phk.freebsd.dk Tue Nov 27 23:33:38 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 27 Nov 2018 23:33:38 +0000 Subject: [master] 7395ecae8 fix typo In-Reply-To: <20181127163511.9820F95F41@lists.varnish-cache.org> References: <20181127163511.9820F95F41@lists.varnish-cache.org> Message-ID: <1979.1543361618@critter.freebsd.dk> -------- In message <20181127163511.9820F95F41 at lists.varnish-cache.org>, Nils Goroll wri tes: > >commit 7395ecae8fdfaefca9ee5438d35ea6101c7a9f56 >Author: Nils Goroll >Date: Tue Nov 27 17:34:50 2018 +0100 > > fix typo Sorry. We should try to get more of vmodtool covered in the tre, and probably find a way to monitor it (a'la GCOV) -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From phk at phk.freebsd.dk Tue Nov 27 23:21:24 2018 From: phk at phk.freebsd.dk (Poul-Henning Kamp) Date: Tue, 27 Nov 2018 23:21:24 +0000 Subject: [master] 7395ecae8 fix typo In-Reply-To: <20181127163511.9820F95F41@lists.varnish-cache.org> References: <20181127163511.9820F95F41@lists.varnish-cache.org> Message-ID: <23028.1543360884@critter.freebsd.dk> -------- In message <20181127163511.9820F95F41 at lists.varnish-cache.org>, Nils Goroll wri tes: >commit 7395ecae8fdfaefca9ee5438d35ea6101c7a9f56 >Author: Nils Goroll >Date: Tue Nov 27 17:34:50 2018 +0100 > > fix typo Sorry. We should consider getting better coverage of vmodtool in the tree. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk at FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From nils.goroll at uplex.de Wed Nov 28 13:09:11 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 28 Nov 2018 13:09:11 +0000 (UTC) Subject: [master] cca34010d Test $Synopsis stanza Message-ID: <20181128130911.BD60D903F@lists.varnish-cache.org> commit cca34010d11950b04cae41ccdd2cd94ed4ae86a6 Author: Nils Goroll Date: Wed Nov 28 13:53:23 2018 +0100 Test $Synopsis stanza this would have caught 7395ecae8fdfaefca9ee5438d35ea6101c7a9f56 diff --git a/lib/libvmod_debug/vmod.vcc b/lib/libvmod_debug/vmod.vcc index c4c221a3f..e3fb3cf12 100644 --- a/lib/libvmod_debug/vmod.vcc +++ b/lib/libvmod_debug/vmod.vcc @@ -29,6 +29,7 @@ $Module debug 3 "Development, test and debug" $ABI strict $Prefix xyzzy +$Synopsis auto DESCRIPTION =========== From nils.goroll at uplex.de Wed Nov 28 13:09:11 2018 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 28 Nov 2018 13:09:11 +0000 (UTC) Subject: [master] b2283944e fix vmodtool error message Message-ID: <20181128130911.B423B903E@lists.varnish-cache.org> commit b2283944ef9aeac00e6042c7bb1b918fb25eed29 Author: Nils Goroll Date: Wed Nov 28 13:51:10 2018 +0100 fix vmodtool error message diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py index 5ad48c6ac..d0955f518 100755 --- a/lib/libvcc/vmodtool.py +++ b/lib/libvcc/vmodtool.py @@ -876,7 +876,7 @@ class vcc(object): d = "".join(ss[1:]) m = DISPATCH.get(toks[0]) if m is None: - err("Unknown stanze $%s" % ss[:i]) + err("Unknown stanza $%s" % toks[0], warn=False) m(toks, [c[0], " ".join(c[1:])], d.split('\n'), self) inputline = None