From nils.goroll at uplex.de Mon Feb 3 14:59:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 3 Feb 2025 14:59:06 +0000 (UTC) Subject: [master] 1d2175b5a doc: Mention noexec and workdir Message-ID: <20250203145906.2A658108FCA@lists.varnish-cache.org> commit 1d2175b5af7e3fe5f4436f0ee143dbe5451f7058 Author: Nils Goroll Date: Mon Feb 3 15:54:12 2025 +0100 doc: Mention noexec and workdir Ref #3943 diff --git a/doc/sphinx/installation/platformnotes.rst b/doc/sphinx/installation/platformnotes.rst index d9a77ad08..eee80f795 100644 --- a/doc/sphinx/installation/platformnotes.rst +++ b/doc/sphinx/installation/platformnotes.rst @@ -35,6 +35,12 @@ Otherwise, consider creating a ``tmpfs`` mountpoint at *workdir*, or configure Note: Very valid reasons exist for *not* following this recommendation, if you know what you are doing. +workdir can not be mounted ``noexec`` +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Varnish compiles VCL to a shared object and needs to load it at runtime. So the +workdie can not reside on a file system mounted with ``noexec``. + Lift locked memory limits ~~~~~~~~~~~~~~~~~~~~~~~~~ From dridi.boukelmoune at gmail.com Mon Feb 3 15:18:09 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 3 Feb 2025 15:18:09 +0000 (UTC) Subject: [master] 367c9f045 doc: s/workdie/workdir/ Message-ID: <20250203151809.4CEA5109B3F@lists.varnish-cache.org> commit 367c9f045e5ebc026aaf53c9b8a156fab77d1e4c Author: Dridi Boukelmoune Date: Mon Feb 3 16:16:50 2025 +0100 doc: s/workdie/workdir/ diff --git a/doc/sphinx/installation/platformnotes.rst b/doc/sphinx/installation/platformnotes.rst index eee80f795..012095fa3 100644 --- a/doc/sphinx/installation/platformnotes.rst +++ b/doc/sphinx/installation/platformnotes.rst @@ -39,7 +39,7 @@ workdir can not be mounted ``noexec`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Varnish compiles VCL to a shared object and needs to load it at runtime. So the -workdie can not reside on a file system mounted with ``noexec``. +*workdir* can not reside on a file system mounted with ``noexec``. Lift locked memory limits ~~~~~~~~~~~~~~~~~~~~~~~~~ From dridi.boukelmoune at gmail.com Mon Feb 3 15:41:05 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 3 Feb 2025 15:41:05 +0000 (UTC) Subject: [master] 4b37eae1e vjsn: Update test corpus script to Python 3 Message-ID: <20250203154105.15BAC10D8EE@lists.varnish-cache.org> commit 4b37eae1e241dc794a65d25405368a2659870cc3 Author: Dridi Boukelmoune Date: Mon Jan 13 15:45:16 2025 +0100 vjsn: Update test corpus script to Python 3 diff --git a/lib/libvarnish/vjsn.c b/lib/libvarnish/vjsn.c index fc37b4095..a72f66acd 100644 --- a/lib/libvarnish/vjsn.c +++ b/lib/libvarnish/vjsn.c @@ -563,7 +563,7 @@ VJSN_TYPES def emit(fin): if fin in skip: return - x = bytearray(open(fin).read()) + x = bytearray(open(fin, 'rb').read()) if 0 in x: return if len(x) > 1000: From dridi.boukelmoune at gmail.com Mon Feb 3 15:41:05 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Mon, 3 Feb 2025 15:41:05 +0000 (UTC) Subject: [master] d01b358d0 vjsn: Polish test corpus script Message-ID: <20250203154105.2DEFF10D8F1@lists.varnish-cache.org> commit d01b358d0f97b279a2ebaa059fa4f4f778688f27 Author: Dridi Boukelmoune Date: Mon Jan 13 15:51:26 2025 +0100 vjsn: Polish test corpus script diff --git a/lib/libvarnish/vjsn.c b/lib/libvarnish/vjsn.c index a72f66acd..0ea056f18 100644 --- a/lib/libvarnish/vjsn.c +++ b/lib/libvarnish/vjsn.c @@ -576,15 +576,15 @@ VJSN_TYPES t = '\t"' print(t + '",') - print("const char *good[] = {") + print("static const char *good[] = {") l = list(glob.glob("y_*")) l.sort() for f in l: emit(f) print("\tNULL") - print("};") + print("};\n") - print("const char *bad[] = {") + print("static const char *bad[] = {") l = list(glob.glob("n_*")) l.sort() for f in l: @@ -728,6 +728,7 @@ static const char *good[] = { "\x20\x5b\x5d\x20", NULL }; + static const char *bad[] = { "\x5b\x31\x20\x74\x72\x75\x65\x5d", "\x5b\x61\xe5\x5d", From dridi.boukelmoune at gmail.com Tue Feb 4 07:27:09 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 4 Feb 2025 07:27:09 +0000 (UTC) Subject: [master] db5314ef4 vcl: Grant req.hash access to more subroutines Message-ID: <20250204072709.8872B103301@lists.varnish-cache.org> commit db5314ef439756b77ea72ea6f15323d54c2aaf2d Author: Dridi Boukelmoune Date: Tue Feb 4 08:22:42 2025 +0100 vcl: Grant req.hash access to more subroutines diff --git a/doc/sphinx/reference/vcl_var.rst b/doc/sphinx/reference/vcl_var.rst index 450c64fc7..08d1e7966 100644 --- a/doc/sphinx/reference/vcl_var.rst +++ b/doc/sphinx/reference/vcl_var.rst @@ -288,7 +288,7 @@ req.hash Type: BLOB - Readable from: vcl_hit, vcl_miss, vcl_pass, vcl_purge, vcl_deliver + Readable from: vcl_hit, vcl_miss, vcl_pass, vcl_purge, vcl_deliver, vcl_synth, vcl_pipe The hash key of this request. From dridi.boukelmoune at gmail.com Tue Feb 4 16:42:09 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 4 Feb 2025 16:42:09 +0000 (UTC) Subject: [master] 380ea7aeb coverity: Tentatively update github action Message-ID: <20250204164209.6BCDF11C3A0@lists.varnish-cache.org> commit 380ea7aebed6366af2c265ca57b0498ddc4f7f84 Author: Dridi Boukelmoune Date: Tue Feb 4 17:40:50 2025 +0100 coverity: Tentatively update github action I have no idea how to test a YAML change without commiting it first. diff --git a/.github/workflows/coverity.yml b/.github/workflows/coverity.yml index 6843d91bd..c86322600 100644 --- a/.github/workflows/coverity.yml +++ b/.github/workflows/coverity.yml @@ -28,7 +28,7 @@ jobs: - uses: actions/checkout at v2 - run: ./autogen.sh - run: ./configure --with-persistent-storage - - uses: vapier/coverity-scan-action at v0 + - uses: vapier/coverity-scan-action at v1.8.0 with: project: varnish email: varnish-dev at varnish-cache.org From nils.goroll at uplex.de Tue Feb 4 17:39:07 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 4 Feb 2025 17:39:07 +0000 (UTC) Subject: [master] c63bd5088 vdire: avoid accidental use-after-free Message-ID: <20250204173907.BA13211E2CD@lists.varnish-cache.org> commit c63bd508841d3781dc99dc3109b331fffc240021 Author: Nils Goroll Date: Tue Feb 4 18:37:41 2025 +0100 vdire: avoid accidental use-after-free Spotted by Coverity Pointed out by Dridi Coverity CID#1641631 Fixup to 586d0c6d23e1b35d44ede2a90b70e76c4df461ae Ref #4142 diff --git a/bin/varnishd/cache/cache_vcl.c b/bin/varnishd/cache/cache_vcl.c index 2140bc664..5a82baf21 100644 --- a/bin/varnishd/cache/cache_vcl.c +++ b/bin/varnishd/cache/cache_vcl.c @@ -414,7 +414,7 @@ vdire_end_iter(struct vdire *vdire) { struct vcldir_head resigning = VTAILQ_HEAD_INITIALIZER(resigning); const struct vcltemp *temp = NULL; - struct vcldir *vdir; + struct vcldir *vdir, *next; unsigned n; CHECK_OBJ_NOTNULL(vdire, VDIRE_MAGIC); @@ -432,7 +432,7 @@ vdire_end_iter(struct vdire *vdire) } Lck_Unlock(vdire->mtx); - VTAILQ_FOREACH(vdir, &resigning, resigning_list) + VTAILQ_FOREACH_SAFE(vdir, &resigning, resigning_list, next) vcldir_retire(vdir, temp); } From nils.goroll at uplex.de Wed Feb 5 08:43:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 5 Feb 2025 08:43:06 +0000 (UTC) Subject: [master] 6dd8f6d0e params: Add tweak_unit_orzero which allows 0 in addition to the uint range Message-ID: <20250205084306.A1E5C115F4F@lists.varnish-cache.org> commit 6dd8f6d0e85aa0052b7f08ec1641dbf50a2bf76d Author: Nils Goroll Date: Wed Feb 5 09:04:07 2025 +0100 params: Add tweak_unit_orzero which allows 0 in addition to the uint range diff --git a/bin/varnishd/mgt/mgt_param.h b/bin/varnishd/mgt/mgt_param.h index c88aef212..063ae7032 100644 --- a/bin/varnishd/mgt/mgt_param.h +++ b/bin/varnishd/mgt/mgt_param.h @@ -86,6 +86,7 @@ tweak_t tweak_thread_pool_min; tweak_t tweak_thread_pool_max; tweak_t tweak_timeout; tweak_t tweak_uint; +tweak_t tweak_uint_orzero; tweak_t tweak_vcc_feature; tweak_t tweak_vsl_buffer; tweak_t tweak_vsl_mask; diff --git a/bin/varnishd/mgt/mgt_param_tweak.c b/bin/varnishd/mgt/mgt_param_tweak.c index be0af0b62..644dbab44 100644 --- a/bin/varnishd/mgt/mgt_param_tweak.c +++ b/bin/varnishd/mgt/mgt_param_tweak.c @@ -293,6 +293,21 @@ tweak_uint(struct vsb *vsb, const struct parspec *par, const char *arg) par->dyn_min_reason, par->dyn_max_reason)); } +int v_matchproto_(tweak_t) +tweak_uint_orzero(struct vsb *vsb, const struct parspec *par, const char *arg) +{ + volatile unsigned *dest; + + dest = par->priv; + if (arg != NULL && arg != JSON_FMT && ! strcmp(arg, "0")) { + VSB_cat(vsb, "0"); + *dest = 0; + return (0); + } + return (tweak_generic_uint(vsb, dest, arg, par->min, par->max, + par->dyn_min_reason, par->dyn_max_reason)); +} + /*--------------------------------------------------------------------*/ static void From nils.goroll at uplex.de Wed Feb 5 08:43:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 5 Feb 2025 08:43:06 +0000 (UTC) Subject: [master] 749a2c3fc params: Add http_req_overflow_status parameter Message-ID: <20250205084306.C0D15115F54@lists.varnish-cache.org> commit 749a2c3fcb417563fe3c1e076f6c78349e869aa1 Author: Thibaut Artis Date: Mon Jul 29 11:36:20 2024 +0200 params: Add http_req_overflow_status parameter Adds a Varnish parameter for whether and which HTTP response code should be sent in case of a http_req_size overflow. The default value (0) keeps the old behavior which silently closes the connection. Committer comment: This has been significantly edited since the original Author's commit. Resolves #2735 diff --git a/bin/varnishd/common/common_param.h b/bin/varnishd/common/common_param.h index b26440f45..60c4d2f71 100644 --- a/bin/varnishd/common/common_param.h +++ b/bin/varnishd/common/common_param.h @@ -118,6 +118,7 @@ struct params { #define ptyp_thread_pool_min unsigned #define ptyp_timeout vtim_dur #define ptyp_uint unsigned +#define ptyp_uint_orzero unsigned #define ptyp_vcc_feature vcc_feature_t #define ptyp_vsl_buffer unsigned #define ptyp_vsl_mask vsl_mask_t diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c index e4fd595f4..976307863 100644 --- a/bin/varnishd/http1/cache_http1_fsm.c +++ b/bin/varnishd/http1/cache_http1_fsm.c @@ -327,6 +327,10 @@ HTTP1_Session(struct worker *wrk, struct req *req) cache_param->http_req_size); assert(!WS_IsReserved(req->htc->ws)); if (hs < HTC_S_EMPTY) { + if (hs == HTC_S_OVERFLOW && cache_param->http_req_overflow_status != 0) { + (void)req->transport->minimal_response(req, + cache_param->http_req_overflow_status); + } req->acct.req_hdrbytes += req->htc->rxbuf_e - req->htc->rxbuf_b; Req_AcctLogCharge(wrk->stats, req); diff --git a/bin/varnishtest/tests/c00039.vtc b/bin/varnishtest/tests/c00039.vtc index c7350219d..1c3b2a9da 100644 --- a/bin/varnishtest/tests/c00039.vtc +++ b/bin/varnishtest/tests/c00039.vtc @@ -62,3 +62,20 @@ client c1 { send "1...5: ..0....5....0....5....\r\n\r\n" expect_close } -run + +varnish v1 -cliok "param.set http_req_overflow_status 414" +client c1 { + # Each line is 32 except last, which is 33. Total: 32 * 7 + 33 == 257 + send "GET /..... HTTP/1.1\r\nHost: foo\r\n" + send "1...5: ..0....5....0....5....0\r\n" + send "1...5: ..0....5....0....5....0\r\n" + send "1...5: ..0....5....0....5....0\r\n" + send "1...5: ..0....5....0....5....0\r\n" + send "1...5: ..0....5....0....5....0\r\n" + send "1...5: ..0....5....0....5....0\r\n" + send "1...5: ..0....5....0....5....\r\n\r\n" + rxresp + expect resp.status == 414 +} -run + +varnish v1 -clierr 106 "param.set http_req_overflow_status 200" diff --git a/include/tbl/params.h b/include/tbl/params.h index cf3429921..7b20e86b2 100644 --- a/include/tbl/params.h +++ b/include/tbl/params.h @@ -649,6 +649,23 @@ PARAM_SIMPLE( "the initial SETTINGS frame." ) +PARAM_SIMPLE( + /* name */ http_req_overflow_status, + /* type */ uint_orzero, + /* min */ "400", + /* max */ "499", + /* def */ "0", + /* units */ "HTTP status code or 0 to disable", + /* descr */ + "HTTP status code to be returned if http_req_size is exceeded. " + "The default value of 0 closes the connection silently without " + "sending a HTTP response.\n" + "Note that there is no standard HTTP status which exactly matches " + "the implementation of http_req_size. 414 applies to the URL only, " + "while 413 applies to the request body. 400 is probably the least " + "incorrect alternative value to sending no response at all (0)." +) + PARAM_SIMPLE( /* name */ http_resp_hdr_len, /* type */ bytes_u, From nils.goroll at uplex.de Wed Feb 5 09:14:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 5 Feb 2025 09:14:05 +0000 (UTC) Subject: [master] b0dcf6b90 V1L: Make workspace rollback optional Message-ID: <20250205091405.566AA1176B9@lists.varnish-cache.org> commit b0dcf6b901413d78f93d2aad9f92b0327d02c466 Author: Nils Goroll Date: Wed Feb 5 10:07:42 2025 +0100 V1L: Make workspace rollback optional When we changed V1L to also be able to use other workspaces than the thread workspace (aws), we (me) overlooked that the rollback in V1L_Close() can only work with the aws and only makes sense for the aws. So to support new use cases where V1L outlives the worker thread, we need to make the rollback optional. As long as the new use case is the exception, specifically preventing it without changing the rest of the API seems to be the least intrusive. Ref 8c872002c75c9dd38c5f114720dd5f893449a03c Ref febfce2ad87d218c6345e343fd712442e86b6c15 Spotted by ws_emu aka workspace SAN https://app.circleci.com/pipelines/github/varnishcache/varnish-cache/6655/workflows/cb155a57-c1f7-4bc1-885c-0a8432613319/jobs/86172 diff --git a/bin/varnishd/http1/cache_http1.h b/bin/varnishd/http1/cache_http1.h index 85203295f..d9b8f4405 100644 --- a/bin/varnishd/http1/cache_http1.h +++ b/bin/varnishd/http1/cache_http1.h @@ -63,6 +63,7 @@ void V1L_Chunked(struct v1l *v1l); void V1L_EndChunk(struct v1l *v1l); struct v1l * V1L_Open(struct ws *, int *fd, struct vsl_log *, vtim_real deadline, unsigned niov); +void V1L_NoRollback(struct v1l *v1l); stream_close_t V1L_Flush(struct v1l *v1l); stream_close_t V1L_Close(struct v1l **v1lp, uint64_t *cnt); size_t V1L_Write(struct v1l *v1l, const void *ptr, ssize_t len); diff --git a/bin/varnishd/http1/cache_http1_line.c b/bin/varnishd/http1/cache_http1_line.c index d19fb4f08..abf853ae2 100644 --- a/bin/varnishd/http1/cache_http1_line.c +++ b/bin/varnishd/http1/cache_http1_line.c @@ -126,6 +126,14 @@ V1L_Open(struct ws *ws, int *fd, struct vsl_log *vsl, return (v1l); } +void +V1L_NoRollback(struct v1l *v1l) +{ + + CHECK_OBJ_NOTNULL(v1l, V1L_MAGIC); + v1l->ws_snap = 0; +} + stream_close_t V1L_Close(struct v1l **v1lp, uint64_t *cnt) { @@ -145,7 +153,8 @@ V1L_Close(struct v1l **v1lp, uint64_t *cnt) ws = v1l->ws; ws_snap = v1l->ws_snap; ZERO_OBJ(v1l, sizeof *v1l); - WS_Rollback(ws, ws_snap); + if (ws_snap != 0) + WS_Rollback(ws, ws_snap); return (sc); } diff --git a/vmod/vmod_debug_transport_reembarking_http1.c b/vmod/vmod_debug_transport_reembarking_http1.c index 32e176ab3..0c5dcffaf 100644 --- a/vmod/vmod_debug_transport_reembarking_http1.c +++ b/vmod/vmod_debug_transport_reembarking_http1.c @@ -89,6 +89,9 @@ dbg_deliver(struct req *req, int sendbody) return (VTR_D_DONE); } + // Do not roll back req->ws upon V1L_Close() + V1L_NoRollback(v1l); + if (sendbody) { if (!http_GetHdr(req->resp, H_Content_Length, NULL)) { if (req->http->protover == 11) { From dridi.boukelmoune at gmail.com Wed Feb 5 16:56:06 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 5 Feb 2025 16:56:06 +0000 (UTC) Subject: [master] a5a3c14d8 cocci: Patch to migrate away from usleep() Message-ID: <20250205165606.862AB126818@lists.varnish-cache.org> commit a5a3c14d88f9ad7e169afaa5a2f51b32b764186e Author: Dridi Boukelmoune Date: Wed Feb 5 17:37:27 2025 +0100 cocci: Patch to migrate away from usleep() diff --git a/tools/coccinelle/usleep.cocci b/tools/coccinelle/usleep.cocci new file mode 100644 index 000000000..d8d245bd5 --- /dev/null +++ b/tools/coccinelle/usleep.cocci @@ -0,0 +1,45 @@ +/* This patch turns hard-coded usleep() usage into a VTIM_sleep() equivalent. + * + * Unfortunately, if a file contains the same usleep() call twice, each call + * site will be substituted twice. + * + * This code: + * + * usleep(10000); + * usleep(10000); + * + * Will turn into this: + * + * VTIM_sleep(0.01)VTIM_sleep(0.01); + * VTIM_sleep(0.01)VTIM_sleep(0.01); + * + * Fortunately, it does not compile and cannot go unnoticed. + */ + + at found@ +constant int usec; +@@ + +usleep(usec) + + at script:python conv@ +usec << found.usec; +sec; +@@ + +coccinelle.sec = cocci.make_expr("{:g}".format(int(usec) / 1000000.0)) + + at uncast@ +constant int found.usec; +@@ + +- (void)usleep(usec) ++ usleep(usec) + + at replace@ +constant int found.usec; +expression conv.sec; +@@ + +- usleep(usec) +++ VTIM_sleep(sec) From dridi.boukelmoune at gmail.com Wed Feb 5 16:56:06 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 5 Feb 2025 16:56:06 +0000 (UTC) Subject: [master] dd65da676 varnishd: Apply usleep.cocci Message-ID: <20250205165606.A01BE12681B@lists.varnish-cache.org> commit dd65da6765105762e6d4e19b3586f65d0a678c6e Author: Dridi Boukelmoune Date: Wed Feb 5 17:45:39 2025 +0100 varnishd: Apply usleep.cocci diff --git a/bin/varnishd/cache/cache_conn_pool.c b/bin/varnishd/cache/cache_conn_pool.c index f510e3b68..300ba66da 100644 --- a/bin/varnishd/cache/cache_conn_pool.c +++ b/bin/varnishd/cache/cache_conn_pool.c @@ -380,7 +380,7 @@ VCP_Recycle(const struct worker *wrk, struct pfd **pfdp) * have N+1 backend connections rather than N, which is * entirely harmless. */ - (void)usleep(10000); + VTIM_sleep(0.01); } } diff --git a/bin/varnishd/cache/cache_lck.c b/bin/varnishd/cache/cache_lck.c index 029ecc8c9..b4582eee9 100644 --- a/bin/varnishd/cache/cache_lck.c +++ b/bin/varnishd/cache/cache_lck.c @@ -268,7 +268,7 @@ Lck_CondWaitUntil(pthread_cond_t *cond, struct lock *lck, vtim_real when) * 20220329 /phk */ if (errno == EINVAL) { - usleep(100); + VTIM_sleep(0.0001); errno = pthread_cond_timedwait(cond, &ilck->mtx, &ts); } #endif diff --git a/bin/varnishd/cache/cache_pool.c b/bin/varnishd/cache/cache_pool.c index 53d00a5b8..d3642c942 100644 --- a/bin/varnishd/cache/cache_pool.c +++ b/bin/varnishd/cache/cache_pool.c @@ -166,7 +166,7 @@ pool_mkpool(unsigned pool_no) PTOK(pthread_create(&pp->herder_thr, NULL, pool_herder, pp)); while (VTAILQ_EMPTY(&pp->idle_queue)) - (void)usleep(10000); + VTIM_sleep(0.01); SES_NewPool(pp, pool_no); VCA_NewPool(pp); @@ -282,5 +282,5 @@ Pool_Init(void) Lck_New(&pool_mtx, lck_wq); PTOK(pthread_create(&thr_pool_herder, NULL, pool_poolherder, NULL)); while (!VSC_C_main->pools) - (void)usleep(10000); + VTIM_sleep(0.01); } diff --git a/bin/varnishd/cache/cache_vrt_vcl.c b/bin/varnishd/cache/cache_vrt_vcl.c index b021e2280..b4e77d158 100644 --- a/bin/varnishd/cache/cache_vrt_vcl.c +++ b/bin/varnishd/cache/cache_vrt_vcl.c @@ -83,7 +83,7 @@ VCL_Refresh(struct vcl **vcc) { while (vcl_active == NULL) - (void)usleep(100000); + VTIM_sleep(0.1); ASSERT_VCL_ACTIVE(); if (*vcc == vcl_active) diff --git a/bin/varnishd/http1/cache_http1_pipe.c b/bin/varnishd/http1/cache_http1_pipe.c index e361ba020..32d7471fe 100644 --- a/bin/varnishd/http1/cache_http1_pipe.c +++ b/bin/varnishd/http1/cache_http1_pipe.c @@ -63,7 +63,7 @@ rdf(int fd0, int fd1, uint64_t *pcnt) return (1); *pcnt += j; if (i != j) - (void)usleep(100000); /* XXX hack */ + VTIM_sleep(0.1); /* XXX hack */ } return (0); } diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c index eadd65f0c..58f0e80b7 100644 --- a/bin/varnishd/mgt/mgt_child.c +++ b/bin/varnishd/mgt/mgt_child.c @@ -555,7 +555,7 @@ mgt_reap_child(void) r = waitpid(child_pid, &status, WNOHANG); if (r == child_pid) break; - (void)usleep(100000); + VTIM_sleep(0.1); } if (r == 0) { VSB_printf(vsb, "Child (%jd) not dying (waitpid = %jd)," diff --git a/bin/varnishd/waiter/cache_waiter_poll.c b/bin/varnishd/waiter/cache_waiter_poll.c index e9c017bf2..26d1c8739 100644 --- a/bin/varnishd/waiter/cache_waiter_poll.c +++ b/bin/varnishd/waiter/cache_waiter_poll.c @@ -259,7 +259,7 @@ vwp_fini(struct waiter *w) CAST_OBJ_NOTNULL(vwp, w->priv, VWP_MAGIC); vp = NULL; while (vwp->hpoll > 1) - (void)usleep(100000); + VTIM_sleep(0.1); // XXX: set write pipe blocking assert(write(vwp->pipes[1], &vp, sizeof vp) == sizeof vp); PTOK(pthread_join(vwp->thread, &vp)); From dridi.boukelmoune at gmail.com Wed Feb 5 16:56:06 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 5 Feb 2025 16:56:06 +0000 (UTC) Subject: [master] 3dbca974b varnishtest: Apply usleep.cocci Message-ID: <20250205165606.BEBC812681F@lists.varnish-cache.org> commit 3dbca974bde396fd61d57beb7dcb6c091d10eeed Author: Dridi Boukelmoune Date: Wed Feb 5 17:52:56 2025 +0100 varnishtest: Apply usleep.cocci diff --git a/bin/varnishtest/vtc_haproxy.c b/bin/varnishtest/vtc_haproxy.c index e4150bf01..432493ae6 100644 --- a/bin/varnishtest/vtc_haproxy.c +++ b/bin/varnishtest/vtc_haproxy.c @@ -824,7 +824,7 @@ haproxy_wait(struct haproxy *h) vtc_log(h->vl, 4, "Kill(%d)=%d: %s", sig, i, strerror(errno)); } - usleep(100000); + VTIM_sleep(0.1); if (++n == 20) { switch (sig) { case SIGINT: sig = SIGTERM ; break; diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c index 352054a59..983576896 100644 --- a/bin/varnishtest/vtc_http.c +++ b/bin/varnishtest/vtc_http.c @@ -1682,7 +1682,7 @@ cmd_http_txpri(CMD_ARGS) if (l != 18) vtc_log(vl, hp->fatal, "Write failed: (%zd vs %zd) %s", l, sizeof(PREFACE), strerror(errno)); - usleep(10000); + VTIM_sleep(0.01); l = write(hp->sess->fd, PREFACE + 18, sizeof(PREFACE) - 18); if (l != sizeof(PREFACE) - 18) vtc_log(vl, hp->fatal, "Write failed: (%zd vs %zd) %s", diff --git a/bin/varnishtest/vtc_process.c b/bin/varnishtest/vtc_process.c index baf196257..e9417450c 100644 --- a/bin/varnishtest/vtc_process.c +++ b/bin/varnishtest/vtc_process.c @@ -309,7 +309,7 @@ term_expect_text(struct process *pp, vtc_fatal(pp->vl, "YYY %d nlin %d", y, pp->nlin); x = strtoul(col, NULL, 0); for(l = 0; l <= 10 && x > pp->ncol; l++) // wait for screen change - usleep(100000); + VTIM_sleep(0.1); if (x < 0 || x > pp->ncol) vtc_fatal(pp->vl, "XXX %d ncol %d", x, pp->ncol); l = strlen(pat); @@ -344,7 +344,7 @@ term_expect_cursor(const struct process *pp, const char *lin, const char *col) vtc_fatal(pp->vl, "YYY %d nlin %d", y, pp->nlin); x = strtoul(col, NULL, 0); for(l = 0; l < 10 && x > pp->ncol; l++) // wait for screen change - usleep(100000); + VTIM_sleep(0.1); if (x < 0 || x > pp->ncol) vtc_fatal(pp->vl, "XXX %d ncol %d", x, pp->ncol); if (y != 0 && (y-1) != pos->tp_row) @@ -374,7 +374,7 @@ term_match_text(struct process *pp, vtc_fatal(pp->vl, "YYY %zd nlin %d", y, pp->nlin); x = strtoul(col, NULL, 0); for(l = 0; l < 10 && x > pp->ncol; l++) // wait for screen change - usleep(100000); + VTIM_sleep(0.1); if (x < 0 || x > pp->ncol) vtc_fatal(pp->vl, "XXX %zd ncol %d", x, pp->ncol); @@ -1142,7 +1142,7 @@ cmd_process(CMD_ARGS) v = p->stdout_bytes; PTOK(pthread_mutex_unlock(&p->mtx)); vtc_log(p->vl, 4, "Have %ju bytes", v); - usleep(500000); + VTIM_sleep(0.5); } while(v < u); continue; } diff --git a/bin/varnishtest/vtc_varnish.c b/bin/varnishtest/vtc_varnish.c index d72242196..12a67b4ff 100644 --- a/bin/varnishtest/vtc_varnish.c +++ b/bin/varnishtest/vtc_varnish.c @@ -126,7 +126,7 @@ varnish_fatal_cleanup(const struct varnish *v) } if (n == 1) - usleep(10000); + VTIM_sleep(0.01); } while (n > 0); } @@ -193,7 +193,7 @@ wait_stopped(const struct varnish *v) } free(r); r = NULL; - (void)usleep(200000); + VTIM_sleep(0.2); } } /********************************************************************** @@ -227,7 +227,7 @@ wait_running(const struct varnish *v) } free(r); r = NULL; - (void)usleep(200000); + VTIM_sleep(0.2); } } From dridi.boukelmoune at gmail.com Wed Feb 5 16:56:06 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 5 Feb 2025 16:56:06 +0000 (UTC) Subject: [master] 87a9412aa vsc: Apply usleep.cocci Message-ID: <20250205165606.DC603126823@lists.varnish-cache.org> commit 87a9412aa6c2a7e739540f31672e32ef67a477e6 Author: Dridi Boukelmoune Date: Wed Feb 5 17:53:24 2025 +0100 vsc: Apply usleep.cocci diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c index 668b0b676..2e2f2e4d8 100644 --- a/lib/libvarnishapi/vsc.c +++ b/lib/libvarnishapi/vsc.c @@ -402,7 +402,7 @@ vsc_map_seg(const struct vsc *vsc, struct vsm *vsm, struct vsc_seg *sp) * operations, so there is no reason to allow a long retry * time. */ for (retry = 10; retry > 0 && head->ready == 0; retry--) - usleep(10000); + VTIM_sleep(0.01); if (head->ready == 0) { AZ(VSM_Unmap(vsm, sp->fantom)); From nils.goroll at uplex.de Wed Feb 5 19:58:46 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 5 Feb 2025 20:58:46 +0100 Subject: [master] 87a9412aa vsc: Apply usleep.cocci In-Reply-To: <20250205165606.DC603126823@lists.varnish-cache.org> References: <20250205165606.DC603126823@lists.varnish-cache.org> Message-ID: Hm. this broke all builds Also, what do we gain other than having to convert to a timespec for each sleep? -- Nils Goroll (he/him) ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg tel +49 40 28805731 mob +49 170 2723133 fax +49 40 42949753 xmpp://slink at jabber.int.uplex.de/ http://uplex.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_0x1DCD8F57A3868BD7.asc Type: application/pgp-keys Size: 3943 bytes Desc: OpenPGP public key URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From dridi.boukelmoune at gmail.com Wed Feb 5 20:29:05 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 5 Feb 2025 20:29:05 +0000 (UTC) Subject: [master] 94ac5214e varnishd: Missing vtim.h include Message-ID: <20250205202905.EA5E210548D@lists.varnish-cache.org> commit 94ac5214efe7aa0e824d3e47f7e787d71f3f9603 Author: Dridi Boukelmoune Date: Wed Feb 5 21:04:11 2025 +0100 varnishd: Missing vtim.h include Refs dd65da6765105762e6d4e19b3586f65d0a678c6e diff --git a/bin/varnishd/cache/cache_pool.c b/bin/varnishd/cache/cache_pool.c index d3642c942..9933365c9 100644 --- a/bin/varnishd/cache/cache_pool.c +++ b/bin/varnishd/cache/cache_pool.c @@ -42,6 +42,8 @@ #include "cache_varnishd.h" #include "cache_pool.h" +#include "vtim.h" + static pthread_t thr_pool_herder; static struct lock wstat_mtx; From dridi.boukelmoune at gmail.com Wed Feb 5 20:29:06 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 5 Feb 2025 20:29:06 +0000 (UTC) Subject: [master] cf9c01586 varnishtest: Missing vtim.h include Message-ID: <20250205202906.0F25A105490@lists.varnish-cache.org> commit cf9c01586442a705737120eb524717c8c189127c Author: Dridi Boukelmoune Date: Wed Feb 5 21:04:35 2025 +0100 varnishtest: Missing vtim.h include Refs 3dbca974bde396fd61d57beb7dcb6c091d10eeed diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c index 983576896..5ad7d9723 100644 --- a/bin/varnishtest/vtc_http.c +++ b/bin/varnishtest/vtc_http.c @@ -47,6 +47,7 @@ #include "vnum.h" #include "vrnd.h" #include "vtcp.h" +#include "vtim.h" #include "hpack.h" extern const struct cmds http_cmds[]; diff --git a/bin/varnishtest/vtc_process.c b/bin/varnishtest/vtc_process.c index e9417450c..50dc1fea7 100644 --- a/bin/varnishtest/vtc_process.c +++ b/bin/varnishtest/vtc_process.c @@ -55,6 +55,7 @@ #include "vlu.h" #include "vsb.h" #include "vsub.h" +#include "vtim.h" #include "teken.h" From dridi.boukelmoune at gmail.com Wed Feb 5 20:29:06 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Wed, 5 Feb 2025 20:29:06 +0000 (UTC) Subject: [master] 28830b0bc vsc: Missing vtim.h include Message-ID: <20250205202906.2EC7D105493@lists.varnish-cache.org> commit 28830b0bcbd780587a8809211ba332ec47e87d1f Author: Dridi Boukelmoune Date: Wed Feb 5 21:04:53 2025 +0100 vsc: Missing vtim.h include Refs 87a9412aa6c2a7e739540f31672e32ef67a477e6 diff --git a/lib/libvarnishapi/vsc.c b/lib/libvarnishapi/vsc.c index 2e2f2e4d8..bb9ac5a03 100644 --- a/lib/libvarnishapi/vsc.c +++ b/lib/libvarnishapi/vsc.c @@ -48,6 +48,7 @@ #include "vjsn.h" #include "vsb.h" #include "vsc_priv.h" +#include "vtim.h" #include "vapi/vsc.h" #include "vapi/vsm.h" From dridi.boukelmoune at gmail.com Thu Feb 6 10:00:10 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 6 Feb 2025 10:00:10 +0000 (UTC) Subject: [master] a3e3c4231 cocci: Work around usleep.cocci shortcoming Message-ID: <20250206100011.042291245EA@lists.varnish-cache.org> commit a3e3c423164858f4b6e7b60d08939ae6505287fb Author: Dridi Boukelmoune Date: Thu Feb 6 10:56:20 2025 +0100 cocci: Work around usleep.cocci shortcoming Refs coccinelle/coccinelle#390 diff --git a/tools/coccinelle/usleep.cocci b/tools/coccinelle/usleep.cocci index d8d245bd5..1dcda70c0 100644 --- a/tools/coccinelle/usleep.cocci +++ b/tools/coccinelle/usleep.cocci @@ -1,45 +1,33 @@ /* This patch turns hard-coded usleep() usage into a VTIM_sleep() equivalent. - * - * Unfortunately, if a file contains the same usleep() call twice, each call - * site will be substituted twice. - * - * This code: - * - * usleep(10000); - * usleep(10000); - * - * Will turn into this: - * - * VTIM_sleep(0.01)VTIM_sleep(0.01); - * VTIM_sleep(0.01)VTIM_sleep(0.01); - * - * Fortunately, it does not compile and cannot go unnoticed. */ @found@ constant int usec; +position pos; @@ -usleep(usec) +usleep at pos(usec) @script:python conv@ usec << found.usec; +pos << found.pos; sec; @@ coccinelle.sec = cocci.make_expr("{:g}".format(int(usec) / 1000000.0)) - at uncast@ + at replace@ constant int found.usec; +position found.pos; +expression conv.sec; @@ -- (void)usleep(usec) -+ usleep(usec) +- usleep at pos(usec) ++ VTIM_sleep(sec) - at replace@ -constant int found.usec; -expression conv.sec; + at uncast@ +expression e; @@ -- usleep(usec) -++ VTIM_sleep(sec) +- (void) + VTIM_sleep(e) From nils.goroll at uplex.de Thu Feb 6 19:14:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 6 Feb 2025 19:14:06 +0000 (UTC) Subject: [master] b93872a9b v68.vtc: refactor bereq.body test Message-ID: <20250206191406.78615110FEE@lists.varnish-cache.org> commit b93872a9bd93608604f8d3a90e3d5feea060f5bc Author: Nils Goroll Date: Thu Feb 6 19:13:50 2025 +0100 v68.vtc: refactor bereq.body test diff --git a/bin/varnishtest/tests/v00068.vtc b/bin/varnishtest/tests/v00068.vtc index 4b54a5ba0..47b01b71c 100644 --- a/bin/varnishtest/tests/v00068.vtc +++ b/bin/varnishtest/tests/v00068.vtc @@ -1,13 +1,15 @@ -varnishtest "unset bereq.body with cached req body" +varnishtest "unset bereq.body tests" server s1 { rxreq expect req.method == "GET" + expect req.url == "/cached" expect req.http.Content-Length == txresp rxreq expect req.method == "GET" + expect req.url == "/cached" txresp } -start @@ -15,7 +17,9 @@ varnish v1 -vcl+backend { import std; sub vcl_recv { - std.cache_req_body(2KB); + if (req.url == "/cached") { + std.cache_req_body(2KB); + } } sub vcl_backend_fetch { unset bereq.body; @@ -23,11 +27,11 @@ varnish v1 -vcl+backend { } -start client c1 { - txreq -body "fine" + txreq -url "/cached" -body "fine" rxresp expect resp.status == 200 - txreq + txreq -url "/cached" rxresp expect resp.status == 200 } -run From nils.goroll at uplex.de Thu Feb 6 19:14:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 6 Feb 2025 19:14:06 +0000 (UTC) Subject: [master] 2bfaa565f v68.vtc: actually excercise second server request Message-ID: <20250206191406.90CF1110FF1@lists.varnish-cache.org> commit 2bfaa565f4b689ef0e151ade6490586ba215ceec Author: Nils Goroll Date: Thu Feb 6 19:35:46 2025 +0100 v68.vtc: actually excercise second server request IIUC, the intention here was that the second client request without a body should also hit the server, but it did not, because it was cached. diff --git a/bin/varnishtest/tests/v00068.vtc b/bin/varnishtest/tests/v00068.vtc index 47b01b71c..cf85d768d 100644 --- a/bin/varnishtest/tests/v00068.vtc +++ b/bin/varnishtest/tests/v00068.vtc @@ -9,7 +9,8 @@ server s1 { rxreq expect req.method == "GET" - expect req.url == "/cached" + expect req.url == "/cachednobody" + expect req.http.Content-Length == txresp } -start @@ -31,7 +32,7 @@ client c1 { rxresp expect resp.status == 200 - txreq -url "/cached" + txreq -url "/cachednobody" rxresp expect resp.status == 200 } -run From nils.goroll at uplex.de Thu Feb 6 19:14:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 6 Feb 2025 19:14:06 +0000 (UTC) Subject: [master] 67d69d2f8 cache_vrt_var: for unset bereq.body, remove Content-Length always Message-ID: <20250206191406.AD688110FF5@lists.varnish-cache.org> commit 67d69d2f8fae919ce9d3fe8e935e1d51fe846d69 Author: Nils Goroll Date: Thu Feb 6 19:42:22 2025 +0100 cache_vrt_var: for unset bereq.body, remove Content-Length always If there is no request body, we need to remove the Content-Length header. Doing this in VRT_u_bereq_body() is insufficient, because, for a restart on the client side and a rollback on the backend side, the headers get restored. Fixes #4228 diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index 76a819c2c..81608573a 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -410,6 +410,9 @@ vbf_stp_startfetch(struct worker *wrk, struct busyobj *bo) VCL_backend_fetch_method(bo->vcl, wrk, NULL, bo, NULL); + if (bo->bereq_body == NULL && bo->req == NULL) + http_Unset(bo->bereq, H_Content_Length); + if (wrk->vpi->handling == VCL_RET_ABANDON || wrk->vpi->handling == VCL_RET_FAIL) return (F_STP_FAIL); diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c index 5d5be780d..98fbda953 100644 --- a/bin/varnishd/cache/cache_vrt_var.c +++ b/bin/varnishd/cache/cache_vrt_var.c @@ -640,17 +640,15 @@ VRT_u_bereq_body(VRT_CTX) { CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); CHECK_OBJ_NOTNULL(ctx->bo, BUSYOBJ_MAGIC); - if (ctx->bo->bereq_body != NULL) { + + if (ctx->bo->bereq_body != NULL) (void)HSH_DerefObjCore(ctx->bo->wrk, &ctx->bo->bereq_body, 0); - http_Unset(ctx->bo->bereq, H_Content_Length); - } if (ctx->bo->req != NULL) { CHECK_OBJ(ctx->bo->req, REQ_MAGIC); ctx->bo->req = NULL; ObjSetState(ctx->bo->wrk, ctx->bo->fetch_objcore, BOS_REQ_DONE); - http_Unset(ctx->bo->bereq, H_Content_Length); } } diff --git a/bin/varnishtest/tests/v00068.vtc b/bin/varnishtest/tests/v00068.vtc index cf85d768d..07e77b6f5 100644 --- a/bin/varnishtest/tests/v00068.vtc +++ b/bin/varnishtest/tests/v00068.vtc @@ -12,6 +12,18 @@ server s1 { expect req.url == "/cachednobody" expect req.http.Content-Length == txresp + + rxreq + expect req.method == "HEAD" + expect req.url == "/head-restart-get" + expect req.http.Content-Length == + txresp + + rxreq + expect req.method == "GET" + expect req.url == "/head-restart-get" + expect req.http.Content-Length == + txresp } -start varnish v1 -vcl+backend { @@ -21,9 +33,28 @@ varnish v1 -vcl+backend { if (req.url == "/cached") { std.cache_req_body(2KB); } + set req.http.restarts = req.restarts; + } + + sub vcl_hash { + hash_data(req.restarts); } + + sub vcl_deliver { + if (req.url == "/head-restart-get" && req.restarts == 0) { + return (restart); + } + } + sub vcl_backend_fetch { - unset bereq.body; + if (bereq.url == "/head-restart-get" && bereq.http.restarts == "0") { + set bereq.method = "HEAD"; + } + + if (bereq.http.restarts == "0") { + unset bereq.body; + } + return (fetch); } } -start @@ -35,4 +66,8 @@ client c1 { txreq -url "/cachednobody" rxresp expect resp.status == 200 + + txreq -url "/head-restart-get" -body "fine" + rxresp + expect resp.status == 200 } -run From nils.goroll at uplex.de Thu Feb 6 19:14:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 6 Feb 2025 19:14:06 +0000 (UTC) Subject: [master] 455d4a7a6 v68.vtc: Test that the previous scenario also works wrt TE: chunked Message-ID: <20250206191406.CA140110FFA@lists.varnish-cache.org> commit 455d4a7a6e050a5db261f426ccc8503989fdbb5b Author: Nils Goroll Date: Thu Feb 6 19:54:23 2025 +0100 v68.vtc: Test that the previous scenario also works wrt TE: chunked diff --git a/bin/varnishtest/tests/v00068.vtc b/bin/varnishtest/tests/v00068.vtc index 07e77b6f5..2ccf30afe 100644 --- a/bin/varnishtest/tests/v00068.vtc +++ b/bin/varnishtest/tests/v00068.vtc @@ -5,24 +5,42 @@ server s1 { expect req.method == "GET" expect req.url == "/cached" expect req.http.Content-Length == + expect req.http.Transfer-Encoding == txresp rxreq expect req.method == "GET" expect req.url == "/cachednobody" expect req.http.Content-Length == + expect req.http.Transfer-Encoding == txresp rxreq expect req.method == "HEAD" expect req.url == "/head-restart-get" expect req.http.Content-Length == + expect req.http.Transfer-Encoding == txresp rxreq expect req.method == "GET" expect req.url == "/head-restart-get" expect req.http.Content-Length == + expect req.http.Transfer-Encoding == + txresp + + rxreq + expect req.method == "HEAD" + expect req.url == "/head-restart-get-chunked" + expect req.http.Content-Length == + expect req.http.Transfer-Encoding == + txresp + + rxreq + expect req.method == "GET" + expect req.url == "/head-restart-get-chunked" + expect req.http.Content-Length == + expect req.http.Transfer-Encoding == txresp } -start @@ -41,13 +59,13 @@ varnish v1 -vcl+backend { } sub vcl_deliver { - if (req.url == "/head-restart-get" && req.restarts == 0) { + if (req.url ~ "^/head-restart-get" && req.restarts == 0) { return (restart); } } sub vcl_backend_fetch { - if (bereq.url == "/head-restart-get" && bereq.http.restarts == "0") { + if (bereq.url ~ "^/head-restart-get" && bereq.http.restarts == "0") { set bereq.method = "HEAD"; } @@ -70,4 +88,10 @@ client c1 { txreq -url "/head-restart-get" -body "fine" rxresp expect resp.status == 200 + + txreq -url "/head-restart-get-chunked" -hdr "Transfer-encoding: chunked" + chunkedlen 10 + chunkedlen 0 + rxresp + expect resp.status == 200 } -run From nils.goroll at uplex.de Thu Feb 6 19:23:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 6 Feb 2025 19:23:05 +0000 (UTC) Subject: [master] e57b17288 build: limit umem to solaris Message-ID: <20250206192305.4DEAB1119B0@lists.varnish-cache.org> commit e57b17288ca87e7cd6a28d160be207bda7c7096d Author: Nils Goroll Date: Thu Feb 6 20:21:55 2025 +0100 build: limit umem to solaris Fixes #4192 diff --git a/configure.ac b/configure.ac index 3f7b25257..8e683eb02 100644 --- a/configure.ac +++ b/configure.ac @@ -100,7 +100,11 @@ _VARNISH_CHECK_LIB(nsl, getaddrinfo) AC_SUBST(NET_LIBS, "${SOCKET_LIBS} ${NSL_LIBS}") # Userland slab allocator from Solaris, ported to other systems -AC_CHECK_HEADERS([umem.h]) +case $target in +*-*-solaris*) + AC_CHECK_HEADERS([umem.h]) + ;; +esac # More portable vmb.h AC_CHECK_HEADERS([stdatomic.h]) From nils.goroll at uplex.de Fri Feb 7 08:49:12 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 7 Feb 2025 08:49:12 +0000 (UTC) Subject: [master] 40f4366fa doc: Mention known malloc stevedore issues more clearly Message-ID: <20250207084912.AE7591024DA@lists.varnish-cache.org> commit 40f4366fa3045f53ed0b8e0aba4b2f56a9c4136a Author: Nils Goroll Date: Thu Feb 6 21:03:32 2025 +0100 doc: Mention known malloc stevedore issues more clearly Closes #3965 diff --git a/doc/sphinx/installation/platformnotes.rst b/doc/sphinx/installation/platformnotes.rst index 012095fa3..371105ffb 100644 --- a/doc/sphinx/installation/platformnotes.rst +++ b/doc/sphinx/installation/platformnotes.rst @@ -53,6 +53,8 @@ size of *workdir* when varnish is running. See :ref:`ref-vsm` for details. +.. _platform-thp: + Transparent hugepages on Redhat Enterprise Linux 6 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/sphinx/users-guide/sizing-your-cache.rst b/doc/sphinx/users-guide/sizing-your-cache.rst index 20189474c..c3375e87c 100644 --- a/doc/sphinx/users-guide/sizing-your-cache.rst +++ b/doc/sphinx/users-guide/sizing-your-cache.rst @@ -23,10 +23,5 @@ Deciding on cache size can be a tricky task. A few things to consider: evicting objects due to space constraints and you should consider increasing the size of the cache. -Be aware that every object that is stored also carries overhead that -is kept outside the actually storage area. So, even if you specify ``-s -malloc,16G`` Varnish might actually use **double** that. Varnish has a -overhead of about 1KB per object. So, if you have lots of small objects -in your cache the overhead might be significant. - -.. XXX:This seems to contradict the last paragraph in "storage-backends". benc +Also, there are additional considerations for specific storage engines +(stevedores), see :ref:`guide-storage` for details. diff --git a/doc/sphinx/users-guide/storage-backends.rst b/doc/sphinx/users-guide/storage-backends.rst index 36014ff81..0030461d9 100644 --- a/doc/sphinx/users-guide/storage-backends.rst +++ b/doc/sphinx/users-guide/storage-backends.rst @@ -25,6 +25,11 @@ objects, the rule of thumb is *n* x ``transit_buffer``. Storage backends are also called stevedores. +.. _vmods: https://www.varnish-cache.org/vmods + +Besides the built-in storage backends, separately distributed extensions exist, +which can be found on the `vmods`_ page by searching for "stevedore". + default ~~~~~~~ @@ -38,18 +43,13 @@ malloc syntax: malloc[,size] -Malloc is a memory based backend. Each object will be allocated from -memory. If your system runs low on memory swap will be used. - -Be aware that the size limitation only limits the actual storage and that the -approximately 1k of memory per object, used for various internal -structures, is included in the actual storage as well. +Malloc is a virtual memory based storage backend. Each object will be allocated +using whatever ``malloc()`` implementation is in effect. If configured, virtual +memory might get paged in and out to swap space by the operating system. -.. XXX:This seems to contradict the last paragraph in "sizing-your-cache". benc - -The size parameter specifies the maximum amount of memory `varnishd` -will allocate. The size is assumed to be in bytes, unless followed by -one of the following suffixes: +The size parameter specifies the maximum *net* amount of memory `varnishd` will +allocate. The size is assumed to be in bytes, unless followed by one of the +following suffixes: K, k The size is expressed in kibibytes. @@ -61,9 +61,21 @@ one of the following suffixes: The default size is unlimited. -malloc's performance is bound to memory speed so it is very fast. If -the dataset is bigger than available memory performance will -depend on the operating systems ability to page effectively. +The *net* amount of memory comprises object metadata (typically in the order of +the total size of headers), segmented body data and metadata for the storage +engine itself. + +This *net* amount of memory is the sum of all allocation sizes from the +perspective of `varnishd`, but for the actual *gross* amount, two additional +factors need to be considered: `varnishd` also requires memory outside the +storage engine in the order of 1KB per object. And, more importantly, due to +fragmentation, the amount of memory actually used by the malloc implementation +might be substantially higher by a factor of typically **two to four times**. +Specific optimizations like :ref:`platform-thp` can amplify this effect. + +malloc's performance is bound to memory speed, so it is very fast. If +the dataset is bigger than available memory, performance will +depend on the operating system's ability to page effectively. .. _guide-storage_umem: From nils.goroll at uplex.de Fri Feb 7 11:04:09 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 7 Feb 2025 11:04:09 +0000 (UTC) Subject: [master] 1f53374ef vcc: trivial refactor of .authority choice Message-ID: <20250207110409.208CD107B2B@lists.varnish-cache.org> commit 1f53374eff2259896f4cb7915d6bf0d913faffef Author: Nils Goroll Date: Fri Feb 7 10:16:05 2025 +0100 vcc: trivial refactor of .authority choice diff --git a/lib/libvcc/vcc_backend.c b/lib/libvcc/vcc_backend.c index f0641d25a..7e6bf64c6 100644 --- a/lib/libvcc/vcc_backend.c +++ b/lib/libvcc/vcc_backend.c @@ -667,13 +667,16 @@ vcc_ParseHostDef(struct vcc *tl, struct symbol *sym, */ if (via != NULL) { AN(t_host); - Fb(tl, 0, "\t.authority = "); if (t_authority != NULL) - EncToken(tl->fb, t_authority); + t_val = t_authority; else if (t_hosthdr != NULL) - EncToken(tl->fb, t_hosthdr); + t_val = t_hosthdr; else - EncToken(tl->fb, t_host); + t_val = t_host; + p = t_val->dec; + + Fb(tl, 0, "\t.authority = "); + VSB_quote(tl->fb, p, -1, VSB_QUOTE_CSTR); Fb(tl, 0, ",\n"); } From nils.goroll at uplex.de Fri Feb 7 11:04:08 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 7 Feb 2025 11:04:08 +0000 (UTC) Subject: [master] 0e494c67a flint.sh: Consistently pass arguments to flint_skel Message-ID: <20250207110409.10FCC107B27@lists.varnish-cache.org> commit 0e494c67aac4d2c26751f0d35580080cf09ce0ac Author: Nils Goroll Date: Fri Feb 7 10:27:58 2025 +0100 flint.sh: Consistently pass arguments to flint_skel ... to enable the -ok option. This was already the case in one flint.sh script, but not the others. diff --git a/bin/varnishadm/flint.sh b/bin/varnishadm/flint.sh index f4340eb51..69dde629b 100644 --- a/bin/varnishadm/flint.sh +++ b/bin/varnishadm/flint.sh @@ -8,4 +8,4 @@ FLOPS=' *.c ../../lib/libvarnishapi/flint.lnt ../../lib/libvarnishapi/*.c -' ../../tools/flint_skel.sh +' ../../tools/flint_skel.sh $* diff --git a/bin/varnishhist/flint.sh b/bin/varnishhist/flint.sh index f4340eb51..69dde629b 100644 --- a/bin/varnishhist/flint.sh +++ b/bin/varnishhist/flint.sh @@ -8,4 +8,4 @@ FLOPS=' *.c ../../lib/libvarnishapi/flint.lnt ../../lib/libvarnishapi/*.c -' ../../tools/flint_skel.sh +' ../../tools/flint_skel.sh $* diff --git a/bin/varnishlog/flint.sh b/bin/varnishlog/flint.sh index a9039a3c0..6d2df55a3 100644 --- a/bin/varnishlog/flint.sh +++ b/bin/varnishlog/flint.sh @@ -9,4 +9,4 @@ FLOPS=' *.c ../../lib/libvarnishapi/flint.lnt ../../lib/libvarnishapi/*.c -' ../../tools/flint_skel.sh +' ../../tools/flint_skel.sh $* diff --git a/bin/varnishncsa/flint.sh b/bin/varnishncsa/flint.sh index dc8d1adc5..83afec164 100644 --- a/bin/varnishncsa/flint.sh +++ b/bin/varnishncsa/flint.sh @@ -8,4 +8,4 @@ FLOPS=' *.c ../../lib/libvarnishapi/flint.lnt ../../lib/libvarnishapi/*.c -' ../../tools/flint_skel.sh +' ../../tools/flint_skel.sh $* diff --git a/bin/varnishstat/flint.sh b/bin/varnishstat/flint.sh index a7ce10885..87d8d0bc2 100644 --- a/bin/varnishstat/flint.sh +++ b/bin/varnishstat/flint.sh @@ -11,4 +11,4 @@ FLOPS=' ../../lib/libvarnishapi/flint.lnt ../../lib/libvarnishapi/*.c -' ../../tools/flint_skel.sh +' ../../tools/flint_skel.sh $* diff --git a/bin/varnishtest/flint.sh b/bin/varnishtest/flint.sh index 1d1192d60..42519e962 100644 --- a/bin/varnishtest/flint.sh +++ b/bin/varnishtest/flint.sh @@ -10,4 +10,4 @@ FLOPS=' -DTOP_BUILDDIR="foo" -I../../lib/libvgz *.c -' ../../tools/flint_skel.sh +' ../../tools/flint_skel.sh $* diff --git a/bin/varnishtop/flint.sh b/bin/varnishtop/flint.sh index e0503ed90..a3058f857 100644 --- a/bin/varnishtop/flint.sh +++ b/bin/varnishtop/flint.sh @@ -8,4 +8,4 @@ FLOPS=' *.c ../../lib/libvarnishapi/flint.lnt ../../lib/libvarnishapi/*.c -' ../../tools/flint_skel.sh +' ../../tools/flint_skel.sh $* diff --git a/lib/libvarnish/flint.sh b/lib/libvarnish/flint.sh index b0b86276c..819daa989 100644 --- a/lib/libvarnish/flint.sh +++ b/lib/libvarnish/flint.sh @@ -6,4 +6,4 @@ FLOPS=' *.c -' ../../tools/flint_skel.sh +' ../../tools/flint_skel.sh $* diff --git a/lib/libvarnishapi/flint.sh b/lib/libvarnishapi/flint.sh index 285bcf4d6..067b93e58 100644 --- a/lib/libvarnishapi/flint.sh +++ b/lib/libvarnishapi/flint.sh @@ -6,4 +6,4 @@ FLOPS=' *.c -' ../../tools/flint_skel.sh +' ../../tools/flint_skel.sh $* diff --git a/lib/libvcc/flint.sh b/lib/libvcc/flint.sh index 3fc056ecf..da4fe8e99 100644 --- a/lib/libvcc/flint.sh +++ b/lib/libvcc/flint.sh @@ -6,4 +6,4 @@ FLOPS=' *.c -' ../../tools/flint_skel.sh +' ../../tools/flint_skel.sh $* diff --git a/vmod/flint.sh b/vmod/flint.sh index 7ad9deeb0..53f97fd69 100644 --- a/vmod/flint.sh +++ b/vmod/flint.sh @@ -11,5 +11,5 @@ for vmod in vmod_*.vcc ; do echo "=====================" vmod="${vmod#vmod_}" FLOPS="-I../bin/varnishd -I../lib/libvgz vcc_${vmod}_if.c vmod_${vmod}*.c" \ - ../tools/flint_skel.sh + ../tools/flint_skel.sh $* done From nils.goroll at uplex.de Fri Feb 7 11:04:09 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 7 Feb 2025 11:04:09 +0000 (UTC) Subject: [master] cf658eb5f vcc: Ensure the authority does not contain a : Message-ID: <20250207110409.3C410107B2D@lists.varnish-cache.org> commit cf658eb5f181c736a04de9d8ef421a563a9024ec Author: Nils Goroll Date: Fri Feb 7 10:32:35 2025 +0100 vcc: Ensure the authority does not contain a : First part of #3963 diff --git a/bin/varnishtest/tests/c00042.vtc b/bin/varnishtest/tests/c00042.vtc index ed69de824..dc3c5dc12 100644 --- a/bin/varnishtest/tests/c00042.vtc +++ b/bin/varnishtest/tests/c00042.vtc @@ -110,7 +110,7 @@ varnish v1 -vcl { .via = v2; .host = "${s1_addr}"; .port = "${s1_port}"; - .host_header = "host.com"; + .host_header = "host.com:1234"; } sub vcl_recv { diff --git a/doc/sphinx/reference/vcl-backend.rst b/doc/sphinx/reference/vcl-backend.rst index 00904acbf..5d3ddd372 100644 --- a/doc/sphinx/reference/vcl-backend.rst +++ b/doc/sphinx/reference/vcl-backend.rst @@ -219,6 +219,8 @@ The HTTP authority to use when connecting to this backend. If unset, ``.authority = ""`` disables sending an authority. +A colon and anything following (signifying a port) is removed from the authority. + As of this release, the attribute is only used by ``.via`` connections as a ``PP2_TYPE_AUTHORITY`` Type-Length-Value (TLV) in the `PROXY2`_ preamble. diff --git a/lib/libvcc/vcc_backend.c b/lib/libvcc/vcc_backend.c index 7e6bf64c6..31332b4a1 100644 --- a/lib/libvcc/vcc_backend.c +++ b/lib/libvcc/vcc_backend.c @@ -374,8 +374,9 @@ vcc_ParseHostDef(struct vcc *tl, struct symbol *sym, vtim_dur first_byte_timeout = NAN; vtim_dur between_bytes_timeout = NAN; vtim_dur backend_wait_timeout = NAN; - char *p; + char *p, *pp; unsigned u; + int l; if (tl->t->tok == ID && (vcc_IdIs(tl->t, "none") || vcc_IdIs(tl->t, "None"))) { @@ -675,8 +676,11 @@ vcc_ParseHostDef(struct vcc *tl, struct symbol *sym, t_val = t_host; p = t_val->dec; + pp = strchr(p, ':'); + l = (pp == NULL) ? -1 : (int)(pp - p); + Fb(tl, 0, "\t.authority = "); - VSB_quote(tl->fb, p, -1, VSB_QUOTE_CSTR); + VSB_quote(tl->fb, p, l, VSB_QUOTE_CSTR); Fb(tl, 0, ",\n"); } From nils.goroll at uplex.de Fri Feb 7 11:27:07 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 7 Feb 2025 11:27:07 +0000 (UTC) Subject: [master] 027cef367 cache_vrt_var: unset bereq.body should remove Content-Length Message-ID: <20250207112707.23135109105@lists.varnish-cache.org> commit 027cef3676992204d2c01911c59b6245c8db5041 Author: Nils Goroll Date: Fri Feb 7 12:14:09 2025 +0100 cache_vrt_var: unset bereq.body should remove Content-Length Address a valid argument by Dridi in #4228 diff --git a/bin/varnishd/cache/cache_vrt_var.c b/bin/varnishd/cache/cache_vrt_var.c index 98fbda953..778c9a42f 100644 --- a/bin/varnishd/cache/cache_vrt_var.c +++ b/bin/varnishd/cache/cache_vrt_var.c @@ -641,6 +641,8 @@ VRT_u_bereq_body(VRT_CTX) CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); CHECK_OBJ_NOTNULL(ctx->bo, BUSYOBJ_MAGIC); + http_Unset(ctx->bo->bereq, H_Content_Length); + if (ctx->bo->bereq_body != NULL) (void)HSH_DerefObjCore(ctx->bo->wrk, &ctx->bo->bereq_body, 0); From nils.goroll at uplex.de Fri Feb 7 11:27:07 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 7 Feb 2025 11:27:07 +0000 (UTC) Subject: [master] 0cf688b5d cache_fetch: Remove Content-Length when there is no bereq.body (any more) Message-ID: <20250207112707.3A40D109107@lists.varnish-cache.org> commit 0cf688b5dd227dc4902ca3fd36a5fda14f90ea7f Author: Nils Goroll Date: Fri Feb 7 12:17:37 2025 +0100 cache_fetch: Remove Content-Length when there is no bereq.body (any more) Rather than specifically addressing rollbacks, simply remove Content-Length before calling VCL such that it can observe the missing Content-Length if there is no bereq.body. This way, the logic is consistent also for restarts from the client side. Fixes #4228 diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index 81608573a..dc842208b 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -408,11 +408,11 @@ vbf_stp_startfetch(struct worker *wrk, struct busyobj *bo) http_PrintfHeader(bo->bereq, "X-Varnish: %ju", VXID(bo->vsl->wid)); - VCL_backend_fetch_method(bo->vcl, wrk, NULL, bo, NULL); - if (bo->bereq_body == NULL && bo->req == NULL) http_Unset(bo->bereq, H_Content_Length); + VCL_backend_fetch_method(bo->vcl, wrk, NULL, bo, NULL); + if (wrk->vpi->handling == VCL_RET_ABANDON || wrk->vpi->handling == VCL_RET_FAIL) return (F_STP_FAIL); diff --git a/bin/varnishtest/tests/v00068.vtc b/bin/varnishtest/tests/v00068.vtc index 2ccf30afe..cdab5e2dd 100644 --- a/bin/varnishtest/tests/v00068.vtc +++ b/bin/varnishtest/tests/v00068.vtc @@ -72,6 +72,11 @@ varnish v1 -vcl+backend { if (bereq.http.restarts == "0") { unset bereq.body; } + + if (bereq.http.Content-Length) { + return (abandon); + } + return (fetch); } } -start From nils.goroll at uplex.de Fri Feb 7 11:57:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 7 Feb 2025 11:57:06 +0000 (UTC) Subject: [master] c175ad574 VCP: Fix a race when destroying connection pools Message-ID: <20250207115706.8511D10D4D0@lists.varnish-cache.org> commit c175ad5747ad14545c2938c8d3649d224fa3f1e7 Author: Walid Boudebouda Date: Thu Feb 6 19:08:32 2025 +0100 VCP: Fix a race when destroying connection pools Reading n_kill should always be done while holding the connection pool mutex, otherwise a race could occur where the task cleaning the last connection under the lock would set n_kill to 0, and then a context switch would happen before it gets a chance to release the lock. This leads to a panic in vcp_destroy that tries to Lck_Delete a lock that is still being hold. Fixes #4260 diff --git a/bin/varnishd/cache/cache_conn_pool.c b/bin/varnishd/cache/cache_conn_pool.c index 300ba66da..bca79783c 100644 --- a/bin/varnishd/cache/cache_conn_pool.c +++ b/bin/varnishd/cache/cache_conn_pool.c @@ -248,6 +248,7 @@ VCP_Rel(struct conn_pool **cpp) { struct conn_pool *cp; struct pfd *pfd, *pfd2; + int n_kill; TAKE_OBJ_NOTNULL(cp, cpp, CONN_POOL_MAGIC); @@ -270,8 +271,9 @@ VCP_Rel(struct conn_pool **cpp) (void)shutdown(pfd->fd, SHUT_RDWR); cp->n_kill++; } + n_kill = cp->n_kill; Lck_Unlock(&cp->mtx); - if (cp->n_kill == 0) { + if (n_kill == 0) { vcp_destroy(&cp); return; } @@ -289,6 +291,7 @@ VCP_RelPoll(void) { struct vrb dead; struct conn_pool *cp, *cp2; + int n_kill; ASSERT_CLI(); @@ -303,7 +306,10 @@ VCP_RelPoll(void) VRBT_FOREACH_SAFE(cp, vrb, &dead, cp2) { CHECK_OBJ_NOTNULL(cp, CONN_POOL_MAGIC); - if (cp->n_kill > 0) + Lck_Lock(&cp->mtx); + n_kill = cp->n_kill; + Lck_Unlock(&cp->mtx); + if (n_kill > 0) continue; VRBT_REMOVE(vrb, &dead, cp); vcp_destroy(&cp); From nils.goroll at uplex.de Fri Feb 7 12:51:11 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 7 Feb 2025 12:51:11 +0000 (UTC) Subject: [master] d6b1cb997 Revert "vcc: Ensure the authority does not contain a :" Message-ID: <20250207125112.3BB5410F422@lists.varnish-cache.org> commit d6b1cb99759ffe92c460d50a9d3dc5914dfad234 Author: Nils Goroll Date: Fri Feb 7 13:48:11 2025 +0100 Revert "vcc: Ensure the authority does not contain a :" It makes more sense to implement this as part of #4264, because once VSS tells us that the authority is not an ip address, we can just strip the port without having to basically duplicate vss_parse(). This reverts commit cf658eb5f181c736a04de9d8ef421a563a9024ec. Closes #4265 diff --git a/bin/varnishtest/tests/c00042.vtc b/bin/varnishtest/tests/c00042.vtc index dc3c5dc12..ed69de824 100644 --- a/bin/varnishtest/tests/c00042.vtc +++ b/bin/varnishtest/tests/c00042.vtc @@ -110,7 +110,7 @@ varnish v1 -vcl { .via = v2; .host = "${s1_addr}"; .port = "${s1_port}"; - .host_header = "host.com:1234"; + .host_header = "host.com"; } sub vcl_recv { diff --git a/doc/sphinx/reference/vcl-backend.rst b/doc/sphinx/reference/vcl-backend.rst index 5d3ddd372..00904acbf 100644 --- a/doc/sphinx/reference/vcl-backend.rst +++ b/doc/sphinx/reference/vcl-backend.rst @@ -219,8 +219,6 @@ The HTTP authority to use when connecting to this backend. If unset, ``.authority = ""`` disables sending an authority. -A colon and anything following (signifying a port) is removed from the authority. - As of this release, the attribute is only used by ``.via`` connections as a ``PP2_TYPE_AUTHORITY`` Type-Length-Value (TLV) in the `PROXY2`_ preamble. diff --git a/lib/libvcc/vcc_backend.c b/lib/libvcc/vcc_backend.c index 31332b4a1..7e6bf64c6 100644 --- a/lib/libvcc/vcc_backend.c +++ b/lib/libvcc/vcc_backend.c @@ -374,9 +374,8 @@ vcc_ParseHostDef(struct vcc *tl, struct symbol *sym, vtim_dur first_byte_timeout = NAN; vtim_dur between_bytes_timeout = NAN; vtim_dur backend_wait_timeout = NAN; - char *p, *pp; + char *p; unsigned u; - int l; if (tl->t->tok == ID && (vcc_IdIs(tl->t, "none") || vcc_IdIs(tl->t, "None"))) { @@ -676,11 +675,8 @@ vcc_ParseHostDef(struct vcc *tl, struct symbol *sym, t_val = t_host; p = t_val->dec; - pp = strchr(p, ':'); - l = (pp == NULL) ? -1 : (int)(pp - p); - Fb(tl, 0, "\t.authority = "); - VSB_quote(tl->fb, p, l, VSB_QUOTE_CSTR); + VSB_quote(tl->fb, p, -1, VSB_QUOTE_CSTR); Fb(tl, 0, ",\n"); } From nils.goroll at uplex.de Fri Feb 7 21:49:08 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 7 Feb 2025 21:49:08 +0000 (UTC) Subject: [master] 3cf3e0eaf varnishhist: ignore curs_set() errors Message-ID: <20250207214908.D79F712188D@lists.varnish-cache.org> commit 3cf3e0eafef7d5c3c18892e4e02f543d015d59bc Author: Nils Goroll Date: Fri Feb 7 20:30:33 2025 +0100 varnishhist: ignore curs_set() errors Don't abort if the terminal does not support invisible cursors. Fixes #3757 diff --git a/bin/varnishhist/varnishhist.c b/bin/varnishhist/varnishhist.c index 03251fce4..2e7e7cea3 100644 --- a/bin/varnishhist/varnishhist.c +++ b/bin/varnishhist/varnishhist.c @@ -58,8 +58,10 @@ #if 1 #define AC(x) assert((x) != ERR) +#define IC(x) (void)x #else -#define AC(x) x +#define AC(x) (void)x +#define IC(x) (void)x #endif #define HIST_N 2000 /* how far back we remember */ @@ -380,7 +382,7 @@ do_curses(void *arg) AC(noecho()); AC(nonl()); AC(intrflush(stdscr, FALSE)); - AC(curs_set(0)); + IC(curs_set(0)); AC(erase()); while (!VSIG_int && !VSIG_term && !VSIG_hup) { From nils.goroll at uplex.de Fri Feb 7 21:49:08 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 7 Feb 2025 21:49:08 +0000 (UTC) Subject: [master] 7af001469 vcurses: move AC() and IC() macros to vcurses.h Message-ID: <20250207214909.008D012188F@lists.varnish-cache.org> commit 7af00146975db17247b7602b0dfce3ccf8090245 Author: Nils Goroll Date: Fri Feb 7 20:38:08 2025 +0100 vcurses: move AC() and IC() macros to vcurses.h AC() asserts that curses commands do not fail IC() ignores return values (just (void)x) diff --git a/bin/varnishhist/varnishhist.c b/bin/varnishhist/varnishhist.c index 2e7e7cea3..df40ae91b 100644 --- a/bin/varnishhist/varnishhist.c +++ b/bin/varnishhist/varnishhist.c @@ -56,14 +56,6 @@ #include "vut.h" #include "vtim.h" -#if 1 -#define AC(x) assert((x) != ERR) -#define IC(x) (void)x -#else -#define AC(x) (void)x -#define IC(x) (void)x -#endif - #define HIST_N 2000 /* how far back we remember */ #define HIST_RES 100 /* bucket resolution */ diff --git a/bin/varnishtop/varnishtop.c b/bin/varnishtop/varnishtop.c index 4d2c8077a..3022cb025 100644 --- a/bin/varnishtop/varnishtop.c +++ b/bin/varnishtop/varnishtop.c @@ -58,12 +58,6 @@ #include "vut.h" #include "vapi/vsig.h" -#if 0 -#define AC(x) assert((x) != ERR) -#else -#define AC(x) x -#endif - static struct VUT *vut; struct top { diff --git a/include/vcurses.h b/include/vcurses.h index 1bd756661..2ba4cf7e1 100644 --- a/include/vcurses.h +++ b/include/vcurses.h @@ -43,3 +43,11 @@ #else # error "SysV or X/Open-compatible Curses header file required" #endif + +#if 1 +#define AC(x) assert((x) != ERR) +#define IC(x) (void)x +#else +#define AC(x) (void)x +#define IC(x) (void)x +#endif From nils.goroll at uplex.de Fri Feb 7 21:49:09 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 7 Feb 2025 21:49:09 +0000 (UTC) Subject: [master] 2e46df72d vcurses: ignore initsrc return value Message-ID: <20250207214909.1C8B1121894@lists.varnish-cache.org> commit 2e46df72d174d20948ba21f4003a22c878c7db2b Author: Nils Goroll Date: Fri Feb 7 21:36:15 2025 +0100 vcurses: ignore initsrc return value it is of type WINDOW, so we should not use AC() diff --git a/bin/varnishhist/varnishhist.c b/bin/varnishhist/varnishhist.c index df40ae91b..2136b5192 100644 --- a/bin/varnishhist/varnishhist.c +++ b/bin/varnishhist/varnishhist.c @@ -369,7 +369,7 @@ do_curses(void *arg) int ch; (void)arg; - initscr(); + (void)initscr(); AC(raw()); AC(noecho()); AC(nonl()); diff --git a/bin/varnishstat/varnishstat_curses.c b/bin/varnishstat/varnishstat_curses.c index 6938fbdb3..14f5b0cb1 100644 --- a/bin/varnishstat/varnishstat_curses.c +++ b/bin/varnishstat/varnishstat_curses.c @@ -1153,7 +1153,7 @@ do_curses(struct vsm *vsm, struct vsc *vsc) verbosity = VSC_ChangeLevel(NULL, 0); - initscr(); + (void)initscr(); raw(); noecho(); nonl(); From nils.goroll at uplex.de Fri Feb 7 22:37:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 7 Feb 2025 22:37:05 +0000 (UTC) Subject: [master] a39c931f1 vcurses: defuse assertions on *printw() Message-ID: <20250207223705.4FDA5123440@lists.varnish-cache.org> commit a39c931f15c8052b0ea7a54f99bd5ea2f08f46a5 Author: Nils Goroll Date: Fri Feb 7 23:34:02 2025 +0100 vcurses: defuse assertions on *printw() 7af00146975db17247b7602b0dfce3ccf8090245 accidentally enabled previous AC() checks for varnishtop only. *printw() is buggy, see #4266 diff --git a/bin/varnishtop/varnishtop.c b/bin/varnishtop/varnishtop.c index 3022cb025..e9cdae6de 100644 --- a/bin/varnishtop/varnishtop.c +++ b/bin/varnishtop/varnishtop.c @@ -210,16 +210,16 @@ update(unsigned p) q = ident; len = COLS - strlen(q); if (end_of_file) - AC(mvprintw(0, len - (1 + 6), "%s (EOF)", q)); + IC(mvprintw(0, len - (1 + 6), "%s (EOF)", q)); else - AC(mvprintw(0, len - 1, "%s", q)); - AC(mvprintw(0, 0, "list length %u", ntop)); + IC(mvprintw(0, len - 1, "%s", q)); + IC(mvprintw(0, 0, "list length %u", ntop)); for (tp = VRBT_MIN(t_order, &h_order); tp != NULL; tp = tp2) { tp2 = VRBT_NEXT(t_order, &h_order, tp); if (++l < LINES) { len = vmin(tp->clen, COLS - 20); - AC(mvprintw(l, 0, "%9.2f %-*.*s %*.*s\n", + IC(mvprintw(l, 0, "%9.2f %-*.*s %*.*s\n", tp->count, maxfieldlen, maxfieldlen, VSL_tags[tp->tag], len, len, tp->rec_data)); From nils.goroll at uplex.de Fri Feb 7 22:43:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 7 Feb 2025 22:43:05 +0000 (UTC) Subject: [master] c9463f1b3 Fix ubsan complaints when using gcc Message-ID: <20250207224305.5B1FA1238A2@lists.varnish-cache.org> commit c9463f1b354a3a9e22bae5d3bf5a387c38a46898 Author: Steven Wojcik Date: Sun Mar 17 22:40:20 2024 -0400 Fix ubsan complaints when using gcc diff --git a/bin/varnishd/cache/cache_req.c b/bin/varnishd/cache/cache_req.c index 42e2b93b0..7be842d5e 100644 --- a/bin/varnishd/cache/cache_req.c +++ b/bin/varnishd/cache/cache_req.c @@ -173,7 +173,7 @@ Req_New(struct sess *sp) p = (void*)PRNDUP(p + sizeof(*req->vfc)); req->vdc = (void*)p; - memset(req->vdc, 0, sizeof *req->vdc); + ZERO_OBJ(req->vdc, sizeof *req->vdc); p = (void*)PRNDUP(p + sizeof(*req->vdc)); req->htc = (void*)p; diff --git a/bin/varnishtest/vtc.c b/bin/varnishtest/vtc.c index 80f04b501..2ac436700 100644 --- a/bin/varnishtest/vtc.c +++ b/bin/varnishtest/vtc.c @@ -562,6 +562,8 @@ exec_file(const char *fn, const char *script, const char *tmpdir, struct vsb *vsb; const char *p; + AN(tmpdir); + (void)signal(SIGPIPE, SIG_IGN); PTOK(pthread_mutex_init(&vtc_vrnd_mtx, NULL)); From nils.goroll at uplex.de Fri Feb 7 22:43:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 7 Feb 2025 22:43:05 +0000 (UTC) Subject: [master] ed39f3d52 Fix leak in create_bogo_n_arg Message-ID: <20250207224305.710A11238A5@lists.varnish-cache.org> commit ed39f3d523c55385ffec9d2a83250eb013036e6e Author: Steven Wojcik Date: Fri Feb 23 14:02:44 2024 -0500 Fix leak in create_bogo_n_arg When running with ASAN and UBSAN on ubuntu 22 u00000.vtc and m00003.vtc fails from a leak from the return value of create_bogo_n_arg() in mgt_main.c. Freeing the returned value allows tests to pass. diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c index a31ef0506..d67acac69 100644 --- a/bin/varnishd/mgt/mgt_main.c +++ b/bin/varnishd/mgt/mgt_main.c @@ -567,7 +567,7 @@ mgt_process_f_arg(struct cli *cli, unsigned C_flag, void **fap) return (retval); } -static const char * +static char * create_bogo_n_arg(void) { struct vsb *vsb; @@ -842,8 +842,11 @@ main(int argc, char * const *argv) assert(d_flag == 0 || F_flag == 0); - if (C_flag && n_arg == NULL) - n_arg = create_bogo_n_arg(); + p = NULL; + if (C_flag && n_arg == NULL) { + p = create_bogo_n_arg(); + n_arg = p; + } if (S_arg != NULL && !strcmp(S_arg, "none")) { fprintf(stderr, @@ -860,6 +863,8 @@ main(int argc, char * const *argv) workdir = VIN_n_Arg(n_arg); AN(workdir); + if (p != NULL) + free(p); if (i_arg == NULL || *i_arg == '\0') i_arg = mgt_HostName(); From nils.goroll at uplex.de Fri Feb 7 22:43:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 7 Feb 2025 22:43:05 +0000 (UTC) Subject: [master] ae6a9b161 Skip e29.vtc when using sanitizers Message-ID: <20250207224305.8CF691238A9@lists.varnish-cache.org> commit ae6a9b161be535fd6d928554c16d6a62ec40b1c4 Author: Steven Wojcik Date: Sun Mar 17 23:19:30 2024 -0400 Skip e29.vtc when using sanitizers This test fails when compiling varnish with GCC (11.4.0) with ASAN and UBSAN enabled. This test passes when only one of ASAN or UBSAN are enabled. It also passes when both are enabled when using clang. diff --git a/bin/varnishtest/tests/e00029.vtc b/bin/varnishtest/tests/e00029.vtc index 6f57c5110..74c52dfe9 100644 --- a/bin/varnishtest/tests/e00029.vtc +++ b/bin/varnishtest/tests/e00029.vtc @@ -1,5 +1,7 @@ varnishtest "ESI max_esi_depth" +feature !sanitizer + # test that the default stack size is sufficient for hitting # max_esi_depth From nils.goroll at uplex.de Sun Feb 9 14:14:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Sun, 9 Feb 2025 14:14:06 +0000 (UTC) Subject: [master] 4c86b8941 vmod_debug: Add a utility function to log the origin of strands Message-ID: <20250209141406.200FC126EC8@lists.varnish-cache.org> commit 4c86b89419142c8c2ae1bdf8399348d1b5272df6 Author: Nils Goroll Date: Sun Feb 9 14:58:11 2025 +0100 vmod_debug: Add a utility function to log the origin of strands This is implemented as a logging function to not use workspace itself. diff --git a/bin/varnishtest/tests/v00058.vtc b/bin/varnishtest/tests/v00058.vtc index 2aa5ee2b2..c69fcce4d 100644 --- a/bin/varnishtest/tests/v00058.vtc +++ b/bin/varnishtest/tests/v00058.vtc @@ -1,4 +1,4 @@ -varnishtest "Test VRT STRANDS functions" +varnishtest "Test VRT STRANDS related functions" varnish v1 -arg "-i foobar" -vcl { import debug; @@ -85,6 +85,11 @@ varnish v1 -arg "-i foobar" -vcl { req.url + "<-->" + req.url ) ); + debug.log_strands("return_strands", + debug.return_strands( + req.url + "<-->" + req.url + ) + ); } } -start diff --git a/vmod/vmod_debug.c b/vmod/vmod_debug.c index 847477d0a..299b8461d 100644 --- a/vmod/vmod_debug.c +++ b/vmod/vmod_debug.c @@ -1288,3 +1288,77 @@ xyzzy_use_reembarking_http1(VRT_CTX) { debug_transport_reembarking_http1_use(ctx); } + +static int +in_oc(struct worker *wrk, struct objcore *oc, const char *p) +{ + const char *hdrs; + ssize_t len = 0; + + if (oc == NULL) + return (0); + hdrs = ObjGetAttr(wrk, oc, OA_HEADERS, &len); + if (hdrs == NULL) + return (0); + if (p < hdrs) + return (0); + if (p > hdrs + len) + return (0); + return (1); +} + +static const char * +ptr_where(VRT_CTX, const char *p) +{ + struct ws *ws = ctx->ws; + struct ws *aws; + struct worker *wrk; + struct objcore *oc, *stale_oc; + + if (ctx->req != NULL) { + wrk = ctx->req->wrk; + oc = ctx->req->objcore; + stale_oc = ctx->req->stale_oc; + } + else if (ctx->bo != NULL) { + wrk = ctx->bo->wrk; + oc = ctx->bo->fetch_objcore; + stale_oc = ctx->bo->stale_oc; + } + else + WRONG("ctx"); + + AN(wrk); + aws = ctx->req->wrk->aws; + + if (WS_Allocated(ws, p, -1)) + return ("ws"); + if (WS_Allocated(aws, p, -1)) + return ("aws"); + if (in_oc(wrk, oc, p)) + return ("oc"); + if (in_oc(wrk, stale_oc, p)) + return ("stale_oc"); + return ("?"); +} + +VCL_VOID +xyzzy_log_strands(VRT_CTX, VCL_STRING prefix, VCL_STRANDS subject, VCL_INT nn) +{ + int i, n; + + CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); + if (prefix == NULL) + prefix = ""; + AN(subject); + if (nn > INT_MAX) + n = INT_MAX; + else + n = nn; + + for (i = 0; i < subject->n; i++) { + const char *p = subject->p[i]; + mylog(ctx->vsl, SLT_Debug, "%s[%d]: (%s) %p %.*s%s", prefix, i, + ptr_where(ctx, p), p, n, p, strlen(p) > n ? "..." : ""); + } +} diff --git a/vmod/vmod_debug.vcc b/vmod/vmod_debug.vcc index 3d791a6c8..1d531cc8c 100644 --- a/vmod/vmod_debug.vcc +++ b/vmod/vmod_debug.vcc @@ -447,6 +447,25 @@ $Restrict vcl_deliver Switch to the reembarking http1 debug transport. Calling it from any other transport than http1 results in VCL failure. +$Function VOID log_strands(STRING prefix, STRANDS subject, INT n=4) + +For each strands member, emit one ``Debug`` log line as formatted using +``printf()`` with prefix as ``%s``, the strands member index as ``[%d]:``, the +*origin* as ``(%s)``, addresses of each strands as ``%p`` and at most the first +*n* characters of each strand, with truncations marked by "...". + +*origin* can be one of: + +* ``ws`` if from ``ctx->ws`` +* ``aws`` if from ``wrk->aws`` +* ``oc`` if from ``OA_HEADERS`` of ``req->objcore`` or ``bo->fetch_objcore`` +* ``stale_oc`` if from ``OA_HEADERS`` of ``req->stale_oc`` or bo->stale_oc`` +* ``?`` otherwise + +Example:: + + Debug c prefix[0]: (ws) 0x7fe69ef80420 abcd... + DEPRECATED ========== From nils.goroll at uplex.de Sun Feb 9 14:14:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Sun, 9 Feb 2025 14:14:06 +0000 (UTC) Subject: [master] ba46e71df vtc: Exercise test using strings from three different objects Message-ID: <20250209141406.3C776126ECB@lists.varnish-cache.org> commit ba46e71df529f8419a7b0c60a3ee4f4686674b61 Author: Nils Goroll Date: Sun Feb 9 13:59:27 2025 +0100 vtc: Exercise test using strings from three different objects diff --git a/bin/varnishtest/tests/b00092.vtc b/bin/varnishtest/tests/b00092.vtc new file mode 100644 index 000000000..878c68dd6 --- /dev/null +++ b/bin/varnishtest/tests/b00092.vtc @@ -0,0 +1,48 @@ +varnishtest "Use multiple objects for a single request" + +server s0 { + rxreq + txresp -hdr "method: O1_METHOD" + rxreq + txresp -hdr "url: /o2_url" + rxreq + txresp -hdr "val: 3" +} -start + + +# we use req.method and url because they could simply keep a reference to +# the object rather than copying, while, to copy headers, we currently always +# copy anyway to (re)generate the header name (we could avoid that by special +# casing copies from workspaces and objects) + +varnish v1 -vcl+backend { + import debug; + + sub vcl_recv { + return (pass); + } + sub vcl_deliver { + if (req.restarts == 0) { + set req.method = resp.http.method; + debug.log_strands("resp.http.method", resp.http.method); + } else if (req.restarts == 1) { + set req.url = resp.http.url; + debug.log_strands("resp.http.url", resp.http.url); + } else { + set resp.http.method = req.method; + set resp.http.url = req.url; + debug.log_strands("req.method", req.method); + debug.log_strands("req.url", req.url); + return (deliver); + } + return (restart); + } +} -start + +client c1 { + txreq + rxresp + expect resp.http.method == O1_METHOD + expect resp.http.url == /o2_url + expect resp.http.val == 3 +} -run From nils.goroll at uplex.de Sun Feb 9 14:41:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Sun, 9 Feb 2025 14:41:05 +0000 (UTC) Subject: [master] 2c8ec9dc9 Flexelint Message-ID: <20250209144105.E1AF6127D8E@lists.varnish-cache.org> commit 2c8ec9dc9b4f0e8d9162a4a2a7997b2475f3a896 Author: Nils Goroll Date: Sun Feb 9 15:38:44 2025 +0100 Flexelint I had actually run it locally, but only in the vmod directory, and I overlooked that vmod_debug gets linted from bin/varnishd. Ref 4c86b89419142c8c2ae1bdf8399348d1b5272df6 diff --git a/vmod/vmod_debug.c b/vmod/vmod_debug.c index 299b8461d..fd1100451 100644 --- a/vmod/vmod_debug.c +++ b/vmod/vmod_debug.c @@ -1353,12 +1353,14 @@ xyzzy_log_strands(VRT_CTX, VCL_STRING prefix, VCL_STRANDS subject, VCL_INT nn) AN(subject); if (nn > INT_MAX) n = INT_MAX; + if (nn < 0) + n = 0; else n = nn; for (i = 0; i < subject->n; i++) { const char *p = subject->p[i]; mylog(ctx->vsl, SLT_Debug, "%s[%d]: (%s) %p %.*s%s", prefix, i, - ptr_where(ctx, p), p, n, p, strlen(p) > n ? "..." : ""); + ptr_where(ctx, p), p, n, p, strlen(p) > (unsigned)n ? "..." : ""); } } From nils.goroll at uplex.de Sun Feb 9 15:12:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Sun, 9 Feb 2025 15:12:05 +0000 (UTC) Subject: [master] ac3111510 docs: clarify that A-E header needs to be set on bereq.http for pass requests Message-ID: <20250209151205.C5AA764185@lists.varnish-cache.org> commit ac3111510a29564ae2761e9a49c0fac7e5df69b9 Author: Kai Burjack <72760888+kaiburjack at users.noreply.github.com> Date: Sun Feb 9 13:52:56 2025 +0100 docs: clarify that A-E header needs to be set on bereq.http for pass requests diff --git a/doc/sphinx/phk/gzip.rst b/doc/sphinx/phk/gzip.rst index 6a62bf346..e8d997cf0 100644 --- a/doc/sphinx/phk/gzip.rst +++ b/doc/sphinx/phk/gzip.rst @@ -70,11 +70,12 @@ Tuning, tweaking and frobbing In vcl_recv{} you have a chance to modify the client's Accept-Encoding: header before anything else happens. -In vcl_pass{} the clients Accept-Encoding header is copied to the +In vcl_pass{} the client's Accept-Encoding header is copied to the backend request unchanged. -Even if the client does not support gzip, you can force the A-C header -to "gzip" to save bandwidth between the backend and varnish, varnish will -gunzip the object before delivering to the client. +Even if the client does not support gzip, you can force the A-E header +to "gzip" on bereq.http in vcl_backend_fetch{} to save bandwidth between +the backend and varnish. Varnish will gunzip the object before delivering +to the client. In vcl_miss{} you can remove the "Accept-Encoding: gzip" header, if you do not want the backend to gzip this object. From nils.goroll at uplex.de Sun Feb 9 18:41:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Sun, 9 Feb 2025 18:41:06 +0000 (UTC) Subject: [master] e527957bd cache_busyobj: gc a long obsolete peculiarity Message-ID: <20250209184106.1A4A8106528@lists.varnish-cache.org> commit e527957bdaaf918a101b453946842b53ceaffc35 Author: Nils Goroll Date: Sun Feb 9 19:19:50 2025 +0100 cache_busyobj: gc a long obsolete peculiarity The "memset from one member down" is no longer needed since the per worker busyobj cache was removed in ec38fd337091cd5b989f02633dc0596cb2799cee. Since then, we always go through the MPL, which clears out the whole allocation anyway. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 9346f0980..0930fb117 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -383,10 +383,6 @@ struct busyobj { char *end; - /* - * All fields from retries and down are zeroed when the busyobj - * is recycled. - */ unsigned retries; struct req *req; struct sess *sp; diff --git a/bin/varnishd/cache/cache_busyobj.c b/bin/varnishd/cache/cache_busyobj.c index 593bdfe43..75a8c68b3 100644 --- a/bin/varnishd/cache/cache_busyobj.c +++ b/bin/varnishd/cache/cache_busyobj.c @@ -190,8 +190,5 @@ VBO_ReleaseBusyObj(struct worker *wrk, struct busyobj **pbo) WS_Rollback(bo->ws, 0); #endif - memset(&bo->retries, 0, - sizeof *bo - offsetof(struct busyobj, retries)); - vbo_Free(&bo); } From nils.goroll at uplex.de Sun Feb 9 18:41:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Sun, 9 Feb 2025 18:41:06 +0000 (UTC) Subject: [master] 1ad90c3dd cache_busyobj: cleanup Message-ID: <20250209184106.3192410652C@lists.varnish-cache.org> commit 1ad90c3dde15146c7a9042565b221393cb2d8117 Author: Nils Goroll Date: Sun Feb 9 19:28:35 2025 +0100 cache_busyobj: cleanup The two removed functions had just one call site and wrapped trivial code. We had three instances of AZ(bo->htc) in the VBO_ReleaseBusyObj code path. diff --git a/bin/varnishd/cache/cache_busyobj.c b/bin/varnishd/cache/cache_busyobj.c index 75a8c68b3..5293186fe 100644 --- a/bin/varnishd/cache/cache_busyobj.c +++ b/bin/varnishd/cache/cache_busyobj.c @@ -58,29 +58,6 @@ VBO_Init(void) * BusyObj handling */ -static struct busyobj * -vbo_New(void) -{ - struct busyobj *bo; - unsigned sz; - - bo = MPL_Get(vbopool, &sz); - XXXAN(bo); - bo->magic = BUSYOBJ_MAGIC; - bo->end = (char *)bo + sz; - return (bo); -} - -static void -vbo_Free(struct busyobj **bop) -{ - struct busyobj *bo; - - TAKE_OBJ_NOTNULL(bo, bop, BUSYOBJ_MAGIC); - AZ(bo->htc); - MPL_Free(vbopool, bo); -} - struct busyobj * VBO_GetBusyObj(const struct worker *wrk, const struct req *req) { @@ -91,8 +68,10 @@ VBO_GetBusyObj(const struct worker *wrk, const struct req *req) CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); - bo = vbo_New(); - CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); + bo = MPL_Get(vbopool, &sz); + XXXAN(bo); + bo->magic = BUSYOBJ_MAGIC; + bo->end = (char *)bo + sz; p = (void*)(bo + 1); p = (void*)PRNDUP(p); @@ -173,8 +152,6 @@ VBO_ReleaseBusyObj(struct worker *wrk, struct busyobj **pbo) VSL_End(bo->vsl); - AZ(bo->htc); - if (WS_Overflowed(bo->ws)) wrk->stats->ws_backend_overflow++; @@ -190,5 +167,5 @@ VBO_ReleaseBusyObj(struct worker *wrk, struct busyobj **pbo) WS_Rollback(bo->ws, 0); #endif - vbo_Free(&bo); + MPL_Free(vbopool, bo); } From nils.goroll at uplex.de Sun Feb 9 18:41:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Sun, 9 Feb 2025 18:41:06 +0000 (UTC) Subject: [master] 0f935ccec cache_req: make max_restarts consistent Message-ID: <20250209184106.4F9EB106530@lists.varnish-cache.org> commit 0f935ccecfb66c5058099875f281ec4bb41306a7 Author: Nils Goroll Date: Sun Feb 9 18:49:38 2025 +0100 cache_req: make max_restarts consistent assert(req->restarts <= cache_param->max_restarts) could otherwise fail when the max_restarts parameter is reduced. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 0930fb117..1be031eb8 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -466,6 +466,7 @@ struct req { body_status_t req_body_status; stream_close_t doclose; unsigned restarts; + unsigned max_restarts; unsigned esi_level; /* Delivery mode */ diff --git a/bin/varnishd/cache/cache_req.c b/bin/varnishd/cache/cache_req.c index 7be842d5e..ac188560c 100644 --- a/bin/varnishd/cache/cache_req.c +++ b/bin/varnishd/cache/cache_req.c @@ -195,6 +195,7 @@ Req_New(struct sess *sp) req->t_req = NAN; req->req_step = R_STP_TRANSPORT; req->doclose = SC_NULL; + req->max_restarts = cache_param->max_restarts; return (req); } diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index bbcb3824f..89f71f0a9 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -236,7 +236,7 @@ cnt_deliver(struct worker *wrk, struct req *req) req->t_resp = W_TIM_real(wrk); VCL_deliver_method(req->vcl, wrk, req, NULL, NULL); - assert(req->restarts <= cache_param->max_restarts); + assert(req->restarts <= req->max_restarts); if (wrk->vpi->handling != VCL_RET_DELIVER) { HSH_Cancel(wrk, req->objcore, NULL); @@ -356,8 +356,7 @@ cnt_synth(struct worker *wrk, struct req *req) return (REQ_FSM_DONE); } - if (wrk->vpi->handling == VCL_RET_RESTART && - req->restarts > cache_param->max_restarts) + if (wrk->vpi->handling == VCL_RET_RESTART && req->restarts > req->max_restarts) wrk->vpi->handling = VCL_RET_DELIVER; if (wrk->vpi->handling == VCL_RET_RESTART) { @@ -870,7 +869,7 @@ cnt_restart(struct worker *wrk, struct req *req) AZ(req->objcore); AZ(req->stale_oc); - if (++req->restarts > cache_param->max_restarts) { + if (++req->restarts > req->max_restarts) { VSLb(req->vsl, SLT_VCL_Error, "Too many restarts"); req->err_code = 503; req->req_step = R_STP_SYNTH; From nils.goroll at uplex.de Sun Feb 9 18:41:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Sun, 9 Feb 2025 18:41:06 +0000 (UTC) Subject: [master] 1970515d8 cache_fetch: make max_retries consistent during the busyobj lifetime Message-ID: <20250209184106.6B90E10653A@lists.varnish-cache.org> commit 1970515d8a4ae770cb610c98a155d456fe4e42e9 Author: Nils Goroll Date: Sun Feb 9 19:35:09 2025 +0100 cache_fetch: make max_retries consistent during the busyobj lifetime There is no assertion to be hit like in the previous commit, but we might still want to avoid running in a situation where the number of retries "suddenly" changes. For example, we could fail for "too many retries", end up in v_b_e {} and be able to retry from there, if the cache parameter was changed at just the right time. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 1be031eb8..ffa3d8577 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -383,6 +383,7 @@ struct busyobj { char *end; + unsigned max_retries; unsigned retries; struct req *req; struct sess *sp; diff --git a/bin/varnishd/cache/cache_busyobj.c b/bin/varnishd/cache/cache_busyobj.c index 5293186fe..42332ecd7 100644 --- a/bin/varnishd/cache/cache_busyobj.c +++ b/bin/varnishd/cache/cache_busyobj.c @@ -72,6 +72,7 @@ VBO_GetBusyObj(const struct worker *wrk, const struct req *req) XXXAN(bo); bo->magic = BUSYOBJ_MAGIC; bo->end = (char *)bo + sz; + bo->max_retries = cache_param->max_retries; p = (void*)(bo + 1); p = (void*)PRNDUP(p); diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index dc842208b..196817512 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -530,7 +530,7 @@ vbf_stp_startfetch(struct worker *wrk, struct busyobj *bo) bo->htc->doclose = SC_RESP_CLOSE; vbf_cleanup(bo); - if (bo->retries++ < cache_param->max_retries) + if (bo->retries++ < bo->max_retries) return (F_STP_RETRY); VSLb(bo->vsl, SLT_VCL_Error, @@ -995,7 +995,7 @@ vbf_stp_error(struct worker *wrk, struct busyobj *bo) if (wrk->vpi->handling == VCL_RET_RETRY) { VSB_destroy(&synth_body); - if (bo->retries++ < cache_param->max_retries) + if (bo->retries++ < bo->max_retries) return (F_STP_RETRY); VSLb(bo->vsl, SLT_VCL_Error, "Too many retries, failing"); return (F_STP_FAIL); From nils.goroll at uplex.de Mon Feb 10 12:54:09 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 10 Feb 2025 12:54:09 +0000 (UTC) Subject: [master] af18af5b7 vmod_debug: fix glitches Message-ID: <20250210125409.B105010586E@lists.varnish-cache.org> commit af18af5b7405852ec39c20af0161050eb604a013 Author: Nils Goroll Date: Mon Feb 10 13:51:23 2025 +0100 vmod_debug: fix glitches CID 1642724 CID 1642723 Ref 4c86b89419142c8c2ae1bdf8399348d1b5272df6 diff --git a/vmod/vmod_debug.c b/vmod/vmod_debug.c index fd1100451..7b22d87df 100644 --- a/vmod/vmod_debug.c +++ b/vmod/vmod_debug.c @@ -1329,7 +1329,7 @@ ptr_where(VRT_CTX, const char *p) WRONG("ctx"); AN(wrk); - aws = ctx->req->wrk->aws; + aws = wrk->aws; if (WS_Allocated(ws, p, -1)) return ("ws"); @@ -1353,7 +1353,7 @@ xyzzy_log_strands(VRT_CTX, VCL_STRING prefix, VCL_STRANDS subject, VCL_INT nn) AN(subject); if (nn > INT_MAX) n = INT_MAX; - if (nn < 0) + else if (nn < 0) n = 0; else n = nn; From nils.goroll at uplex.de Mon Feb 10 14:42:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 10 Feb 2025 14:42:06 +0000 (UTC) Subject: [master] 77d7786a7 vcurses: add return value assertions using a semantic patch Message-ID: <20250210144206.910E310964A@lists.varnish-cache.org> commit 77d7786a7842874fa2e7e39b68420deb0292647e Author: Nils Goroll Date: Fri Feb 7 21:40:22 2025 +0100 vcurses: add return value assertions using a semantic patch The only relevant change in this patch is the curses.cocci file. All other changes are just the result of applying that patch, with additional manual polishing where coccinelle caused unwanted cstyle changes. We explicitly ignore the return value of some curses functions through the IC() macro: * curs_set is known to be ok to fail for terminals which do have no capability to make the cursor invisible and * all print-related functions, which have the unpleasent bug to return ERR when the cursor ends up outside the screen/window _after_ completing the print function. For most other curses functions, we add AC() to check that they do not return ERR. diff --git a/bin/varnishhist/varnishhist.c b/bin/varnishhist/varnishhist.c index 2136b5192..4d1f1eb90 100644 --- a/bin/varnishhist/varnishhist.c +++ b/bin/varnishhist/varnishhist.c @@ -150,13 +150,13 @@ update(void) (void)mvaddch(LINES - 2, k, '-'); for (i = 0, j = hist_low; i < hist_range; ++i, ++j) { (void)mvaddch(LINES - 2, w * i, '+'); - mvprintw(LINES - 1, w * i, "|1e%d", j); + IC(mvprintw(LINES - 1, w * i, "|1e%d", j)); } if (end_of_file) - mvprintw(0, 0, "%*s", COLS - 1, "EOF"); + IC(mvprintw(0, 0, "%*s", COLS - 1, "EOF")); else - mvprintw(0, 0, "%*s", COLS - 1, ident); + IC(mvprintw(0, 0, "%*s", COLS - 1, ident)); /* count our flock */ memset(bm, 0, sizeof bm); @@ -178,14 +178,15 @@ update(void) if (vsl_t0 > 0) { VTIM_format(vsl_ts, t); - mvprintw(0, 0, "1:%u, n = %u, d = %g @ %s x %g", - scale, nhist, 1e-3 * ms_delay, t, timebend); - } else - mvprintw(0, 0, "1:%u, n = %u, d = %g", - scale, nhist, 1e-3 * ms_delay); + IC(mvprintw(0, 0, "1:%u, n = %u, d = %g @ %s x %g", + scale, nhist, 1e-3 * ms_delay, t, timebend)); + } else { + IC(mvprintw(0, 0, "1:%u, n = %u, d = %g", + scale, nhist, 1e-3 * ms_delay)); + } for (j = 5; j < LINES - 2; j += 5) - mvprintw((LINES - 2) - j, 0, "%u_", j * scale); + IC(mvprintw((LINES - 2) - j, 0, "%u_", j * scale)); /* show them */ for (k = 0; k < n; ++k) { @@ -396,7 +397,7 @@ do_curses(void *arg) #endif case '\014': /* Ctrl-L */ case '\024': /* Ctrl-T */ - redrawwin(stdscr); + AC(redrawwin(stdscr)); AC(refresh()); break; case '\032': /* Ctrl-Z */ diff --git a/bin/varnishstat/varnishstat_curses.c b/bin/varnishstat/varnishstat_curses.c index 14f5b0cb1..6250c8512 100644 --- a/bin/varnishstat/varnishstat_curses.c +++ b/bin/varnishstat/varnishstat_curses.c @@ -327,7 +327,7 @@ destroy_window(WINDOW **w) AN(w); if (*w == NULL) return; - assert(delwin(*w) != ERR); + AC(delwin(*w)); *w = NULL; } @@ -373,28 +373,28 @@ make_windows(void) w_status = newwin(l_status, X, y_status, 0); AN(w_status); - nodelay(w_status, 1); - keypad(w_status, 1); - wnoutrefresh(w_status); + AC(nodelay(w_status, 1)); + AC(keypad(w_status, 1)); + AC(wnoutrefresh(w_status)); w_bar_t = newwin(l_bar_t, X, y_bar_t, 0); AN(w_bar_t); wbkgd(w_bar_t, A_REVERSE); - wnoutrefresh(w_bar_t); + AC(wnoutrefresh(w_bar_t)); w_points = newwin(l_points, X, y_points, 0); AN(w_points); - wnoutrefresh(w_points); + AC(wnoutrefresh(w_points)); w_bar_b = newwin(l_bar_b, X, y_bar_b, 0); AN(w_bar_b); wbkgd(w_bar_b, A_REVERSE); - wnoutrefresh(w_bar_b); + AC(wnoutrefresh(w_bar_b)); if (l_info) { w_info = newwin(l_info, X, y_info, 0); AN(w_info); - wnoutrefresh(w_info); + AC(wnoutrefresh(w_info)); } if (X - COLW_NAME_MIN > N_COL * COLW) @@ -409,9 +409,9 @@ static void print_duration(WINDOW *w, uint64_t t) { - wprintw(w, "%4ju+%02ju:%02ju:%02ju", + IC(wprintw(w, "%4ju+%02ju:%02ju:%02ju", (uintmax_t)t / 86400, (uintmax_t)(t % 86400) / 3600, - (uintmax_t)(t % 3600) / 60, (uintmax_t)t % 60); + (uintmax_t)(t % 3600) / 60, (uintmax_t)t % 60)); } static void @@ -421,7 +421,7 @@ running(WINDOW *w, uint64_t up, int flg) print_duration(w_status, up); } else { wattron(w, A_STANDOUT); - wprintw(w, " Not Running"); + IC(wprintw(w, " Not Running")); wattroff(w, A_STANDOUT); } } @@ -434,32 +434,32 @@ draw_status(void) AN(w_status); - werase(w_status); + AC(werase(w_status)); if (mgt_uptime != NULL) up_mgt = *mgt_uptime; if (main_uptime != NULL) up_chld = *main_uptime; - mvwprintw(w_status, 0, 0, "Uptime mgt: "); + IC(mvwprintw(w_status, 0, 0, "Uptime mgt: ")); running(w_status, up_mgt, VSM_MGT_RUNNING); - mvwprintw(w_status, 1, 0, "Uptime child: "); + IC(mvwprintw(w_status, 1, 0, "Uptime child: ")); running(w_status, up_chld, VSM_WRK_RUNNING); - mvwprintw(w_status, 2, 0, "Press to toggle help screen"); + IC(mvwprintw(w_status, 2, 0, "Press to toggle help screen")); if (VTIM_mono() < notification_eol) mvwaddstr(w_status, 2, 0, notification_message); if (COLS > 70) { - mvwprintw(w_status, 0, getmaxx(w_status) - 37, + IC(mvwprintw(w_status, 0, getmaxx(w_status) - 37, "Hitrate n: %8u %8u %8u", hitrate.hr_10.n, hitrate.hr_100.n, - hitrate.hr_1000.n); - mvwprintw(w_status, 1, getmaxx(w_status) - 37, + hitrate.hr_1000.n)); + IC(mvwprintw(w_status, 1, getmaxx(w_status) - 37, " avg(n): %8.4f %8.4f %8.4f", hitrate.hr_10.acc, - hitrate.hr_100.acc, hitrate.hr_1000.acc); + hitrate.hr_100.acc, hitrate.hr_1000.acc)); } - wnoutrefresh(w_status); + AC(wnoutrefresh(w_status)); } static void @@ -480,11 +480,11 @@ draw_bar_t(void) X = getmaxx(w_bar_t); x = 0; - werase(w_bar_t); + AC(werase(w_bar_t)); if (page_start > 0) - mvwprintw(w_bar_t, 0, x, "^^^"); + IC(mvwprintw(w_bar_t, 0, x, "^^^")); x += 4; - mvwprintw(w_bar_t, 0, x, "%.*s", colw_name - 4, "NAME"); + IC(mvwprintw(w_bar_t, 0, x, "%.*s", colw_name - 4, "NAME")); x += colw_name - 4; col = COL_CUR; while (col < COL_LAST) { @@ -492,22 +492,22 @@ draw_bar_t(void) break; switch (col) { case COL_CUR: - mvwprintw(w_bar_t, 0, x, " %12.12s", "CURRENT"); + IC(mvwprintw(w_bar_t, 0, x, " %12.12s", "CURRENT")); break; case COL_CHG: - mvwprintw(w_bar_t, 0, x, " %12.12s", "CHANGE"); + IC(mvwprintw(w_bar_t, 0, x, " %12.12s", "CHANGE")); break; case COL_AVG: - mvwprintw(w_bar_t, 0, x, " %12.12s", "AVERAGE"); + IC(mvwprintw(w_bar_t, 0, x, " %12.12s", "AVERAGE")); break; case COL_MA10: - mvwprintw(w_bar_t, 0, x, " %12.12s", "AVG_10"); + IC(mvwprintw(w_bar_t, 0, x, " %12.12s", "AVG_10")); break; case COL_MA100: - mvwprintw(w_bar_t, 0, x, " %12.12s", "AVG_100"); + IC(mvwprintw(w_bar_t, 0, x, " %12.12s", "AVG_100")); break; case COL_MA1000: - mvwprintw(w_bar_t, 0, x, " %12.12s", "AVG_1000"); + IC(mvwprintw(w_bar_t, 0, x, " %12.12s", "AVG_1000")); break; default: break; @@ -516,7 +516,7 @@ draw_bar_t(void) col++; } - wnoutrefresh(w_bar_t); + AC(wnoutrefresh(w_bar_t)); } static void @@ -541,28 +541,28 @@ draw_line_default(WINDOW *w, int y, int x, int X, const struct pt *pt) break; switch (col) { case COL_CUR: - mvwprintw(w, y, x, " %12ju", (uintmax_t)pt->cur); + IC(mvwprintw(w, y, x, " %12ju", (uintmax_t)pt->cur)); break; case COL_CHG: if (pt->t_last) - mvwprintw(w, y, x, " %12.2f", pt->chg); + IC(mvwprintw(w, y, x, " %12.2f", pt->chg)); else - mvwprintw(w, y, x, " %12s", ". "); + IC(mvwprintw(w, y, x, " %12s", ". ")); break; case COL_AVG: if (pt->avg) - mvwprintw(w, y, x, " %12.2f", pt->avg); + IC(mvwprintw(w, y, x, " %12.2f", pt->avg)); else - mvwprintw(w, y, x, " %12s", ". "); + IC(mvwprintw(w, y, x, " %12s", ". ")); break; case COL_MA10: - mvwprintw(w, y, x, " %12.2f", pt->ma_10.acc); + IC(mvwprintw(w, y, x, " %12.2f", pt->ma_10.acc)); break; case COL_MA100: - mvwprintw(w, y, x, " %12.2f", pt->ma_100.acc); + IC(mvwprintw(w, y, x, " %12.2f", pt->ma_100.acc)); break; case COL_MA1000: - mvwprintw(w, y, x, " %12.2f", pt->ma_1000.acc); + IC(mvwprintw(w, y, x, " %12.2f", pt->ma_1000.acc)); break; default: break; @@ -593,7 +593,7 @@ print_bytes(WINDOW *w, double val) if (scale) val = scale_bytes(val, &q); - wprintw(w, " %12.2f%c", val, q); + IC(wprintw(w, " %12.2f%c", val, q)); } static void @@ -602,9 +602,9 @@ print_trunc(WINDOW *w, uintmax_t val) if (val > VALUE_MAX) { while (val > VALUE_MAX) val /= 1000; - wprintw(w, " %9ju...", val); + IC(wprintw(w, " %9ju...", val)); } else - wprintw(w, " %12ju", val); + IC(wprintw(w, " %12ju", val)); } static void @@ -639,13 +639,13 @@ draw_line_bytes(WINDOW *w, int y, int x, int X, const struct pt *pt) if (pt->t_last) print_bytes(w, pt->chg); else - wprintw(w, " %12s", ". "); + IC(wprintw(w, " %12s", ". ")); break; case COL_AVG: if (pt->avg) print_bytes(w, pt->avg); else - wprintw(w, " %12s", ". "); + IC(wprintw(w, " %12s", ". ")); break; case COL_MA10: print_bytes(w, pt->ma_10.acc); @@ -684,8 +684,8 @@ draw_line_bitmap(WINDOW *w, int y, int x, int X, const struct pt *pt) case COL_VAL: if (X - x < COLW) return; - mvwprintw(w, y, x, " %10.10jx", - (uintmax_t)((pt->cur >> 24) & 0xffffffffffLL)); + IC(mvwprintw(w, y, x, " %10.10jx", + (uintmax_t)((pt->cur >> 24) & 0xffffffffffLL))); x += COLW; break; case COL_MAP: @@ -728,7 +728,7 @@ draw_line_duration(WINDOW *w, int y, int x, int X, const struct pt *pt) if (scale) print_duration(w, pt->cur); else - wprintw(w, " %12ju", (uintmax_t)pt->cur); + IC(wprintw(w, " %12ju", (uintmax_t)pt->cur)); break; default: break; @@ -747,9 +747,9 @@ draw_line(WINDOW *w, int y, const struct pt *pt) X = getmaxx(w); x = 0; if (strlen(pt->vpt->name) > colw_name) - mvwprintw(w, y, x, "%.*s...", colw_name - 3, pt->vpt->name); + IC(mvwprintw(w, y, x, "%.*s...", colw_name - 3, pt->vpt->name)); else - mvwprintw(w, y, x, "%.*s", colw_name, pt->vpt->name); + IC(mvwprintw(w, y, x, "%.*s", colw_name, pt->vpt->name)); x += colw_name; switch (pt->vpt->format) { @@ -776,9 +776,9 @@ draw_points(void) AN(w_points); - werase(w_points); + AC(werase(w_points)); if (n_ptarray == 0) { - wnoutrefresh(w_points); + AC(wnoutrefresh(w_points)); return; } @@ -799,7 +799,7 @@ draw_points(void) if (n == current) wattroff(w_points, A_BOLD); } - wnoutrefresh(w_points); + AC(wnoutrefresh(w_points)); } static void @@ -818,19 +818,19 @@ draw_help(void) } X = getmaxx(w_points); - werase(w_points); + AC(werase(w_points)); for (y = 0, p = bindings_help + help_line; y < l; y++, p++) { if (**p == '\t') { - mvwprintw(w_points, y, 0, " %.*s", X - 4, *p + 1); + IC(mvwprintw(w_points, y, 0, " %.*s", X - 4, *p + 1)); } else { wattron(w_points, A_BOLD); - mvwprintw(w_points, y, 0, "%.*s", X, *p); + IC(mvwprintw(w_points, y, 0, "%.*s", X, *p)); wattroff(w_points, A_BOLD); } } - wnoutrefresh(w_points); + AC(wnoutrefresh(w_points)); } static void @@ -843,33 +843,33 @@ draw_bar_b(void) x = 0; X = getmaxx(w_bar_b); - werase(w_bar_b); + AC(werase(w_bar_b)); if (page_start + l_points < n_ptarray) - mvwprintw(w_bar_b, 0, x, "vvv"); + IC(mvwprintw(w_bar_b, 0, x, "vvv")); x += 4; if (current < n_ptarray) - mvwprintw(w_bar_b, 0, x, "%s", ptarray[current]->vpt->name); + IC(mvwprintw(w_bar_b, 0, x, "%s", ptarray[current]->vpt->name)); bprintf(buf, "%d-%d/%d", page_start + 1, page_start + l_points < n_ptarray ? page_start + l_points : n_ptarray, n_ptarray); - mvwprintw(w_bar_b, 0, X - strlen(buf), "%s", buf); + IC(mvwprintw(w_bar_b, 0, X - strlen(buf), "%s", buf)); X -= strlen(buf) + 2; if (verbosity != NULL) { - mvwprintw(w_bar_b, 0, X - strlen(verbosity->label), "%s", - verbosity->label); + IC(mvwprintw(w_bar_b, 0, X - strlen(verbosity->label), "%s", + verbosity->label)); X -= strlen(verbosity->label) + 2; } if (!hide_unseen) { - mvwprintw(w_bar_b, 0, X - 6, "%s", "UNSEEN"); + IC(mvwprintw(w_bar_b, 0, X - 6, "%s", "UNSEEN")); X -= 8; } if (raw_vsc) - mvwprintw(w_bar_b, 0, X - 3, "%s", "RAW"); + IC(mvwprintw(w_bar_b, 0, X - 3, "%s", "RAW")); - wnoutrefresh(w_bar_b); + AC(wnoutrefresh(w_bar_b)); } static void @@ -879,15 +879,15 @@ draw_info(void) if (w_info == NULL) return; - werase(w_info); + AC(werase(w_info)); if (current < n_ptarray) { /* XXX: Word wrapping, and overflow handling? */ - mvwprintw(w_info, 0, 0, "%s:", - ptarray[current]->vpt->sdesc); - mvwprintw(w_info, 1, 0, "%s", - ptarray[current]->vpt->ldesc); + IC(mvwprintw(w_info, 0, 0, "%s:", + ptarray[current]->vpt->sdesc)); + IC(mvwprintw(w_info, 1, 0, "%s", + ptarray[current]->vpt->ldesc)); } - wnoutrefresh(w_info); + AC(wnoutrefresh(w_info)); } static void @@ -895,12 +895,12 @@ draw_screen(void) { draw_status(); if (show_help) { - werase(w_bar_t); - werase(w_bar_b); - werase(w_info); - wnoutrefresh(w_bar_t); - wnoutrefresh(w_bar_b); - wnoutrefresh(w_info); + AC(werase(w_bar_t)); + AC(werase(w_bar_b)); + AC(werase(w_info)); + AC(wnoutrefresh(w_bar_t)); + AC(wnoutrefresh(w_bar_b)); + AC(wnoutrefresh(w_info)); draw_help(); } else { draw_bar_t(); @@ -908,7 +908,7 @@ draw_screen(void) draw_bar_b(); draw_info(); } - doupdate(); + AC(doupdate()); redraw = 0; } @@ -1154,13 +1154,13 @@ do_curses(struct vsm *vsm, struct vsc *vsc) verbosity = VSC_ChangeLevel(NULL, 0); (void)initscr(); - raw(); - noecho(); - nonl(); - curs_set(0); + AC(raw()); + AC(noecho()); + AC(nonl()); + IC(curs_set(0)); make_windows(); - doupdate(); + AC(doupdate()); VSC_State(vsc, newpt, delpt, NULL); diff --git a/bin/varnishtop/varnishtop.c b/bin/varnishtop/varnishtop.c index e9cdae6de..1a3289270 100644 --- a/bin/varnishtop/varnishtop.c +++ b/bin/varnishtop/varnishtop.c @@ -257,7 +257,7 @@ do_curses(void *arg) AC(noecho()); AC(nonl()); AC(intrflush(stdscr, FALSE)); - (void)curs_set(0); + IC(curs_set(0)); AC(erase()); timeout(1000); while (!VSIG_int && !VSIG_term && !VSIG_hup) { diff --git a/tools/coccinelle/curses.cocci b/tools/coccinelle/curses.cocci new file mode 100644 index 000000000..47c86ce51 --- /dev/null +++ b/tools/coccinelle/curses.cocci @@ -0,0 +1,45 @@ + +// XXX can we reuse the pattern somehow without inheriting the metavariable? + +@@ +identifier curs =~ "^(curs_set|(w|mv|mvw|vw_)?printw)$"; +expression list EL; +@@ +-curs(EL); ++IC(curs(EL)); + +@@ +identifier curs =~ "^(curs_set|(w|mv|mvw|vw_)?printw)$"; +expression list EL; +@@ +-(void)curs(EL); ++IC(curs(EL)); + +// ensure we undo AC() patching when we move patterns to IC() +@@ +identifier curs =~ "^(curs_set|(w|mv|mvw|vw_)?printw)$"; +expression list EL; +@@ +-AC(curs(EL)); ++IC(curs(EL)); + +@@ +identifier curs =~ "^(endwin|(no)?cbreak|(no)?echo|intrflush|keypad|meta|nodelay|notimeout|(no)?nl|(no)?raw|w?erase|w?clear|w?clrtobot|w?clrtoeol|(w|wnout)?refresh|doupdate|redrawwin|wredrawln|beep|flash|delwin|mv(der)?win|syncok)$"; +expression list EL; +@@ +-curs(EL); ++AC(curs(EL)); + +@@ +identifier curs =~ "^(endwin|(no)?cbreak|(no)?echo|intrflush|keypad|meta|nodelay|notimeout|(no)?nl|(no)?raw|w?erase|w?clear|w?clrtobot|w?clrtoeol|(w|wnout)?refresh|doupdate|redrawwin|wredrawln|beep|flash|delwin|mv(der)?win|syncok)$"; +expression list EL; +@@ +-(void)curs(EL); ++AC(curs(EL)); + +@@ +identifier curs =~ "^(endwin|(no)?cbreak|(no)?echo|intrflush|keypad|meta|nodelay|notimeout|(no)?nl|(no)?raw|w?erase|w?clear|w?clrtobot|w?clrtoeol|(w|wnout)?refresh|doupdate|redrawwin|wredrawln|beep|flash|delwin|mv(der)?win|syncok)$"; +expression list EL; +@@ +-assert(curs(EL) != ERR); ++AC(curs(EL)); From nils.goroll at uplex.de Mon Feb 10 15:02:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 10 Feb 2025 15:02:05 +0000 (UTC) Subject: [master] b8366fe25 vcc_backend: Forbid ip addresses as .authority with .via Message-ID: <20250210150205.DCB6710D378@lists.varnish-cache.org> commit b8366fe25c284884f4e8c2f62c8babe5a8a384f8 Author: Nils Goroll Date: Fri Feb 7 11:38:25 2025 +0100 vcc_backend: Forbid ip addresses as .authority with .via One half of the fix for #3963 diff --git a/bin/varnishtest/tests/c00042.vtc b/bin/varnishtest/tests/c00042.vtc index ed69de824..60fd5c994 100644 --- a/bin/varnishtest/tests/c00042.vtc +++ b/bin/varnishtest/tests/c00042.vtc @@ -47,8 +47,8 @@ varnish v2 -cliok "param.set debug +syncvsl" varnish v1 -vcl { backend v2 { .host = "${v2_addr}"; .port = "${v2_port}"; } - backend s1 { .via = v2; .host = "${s1_addr}"; .port = "${s1_port}"; } - backend s2 { .via = v2; .host = "${s2_addr}"; .port = "${s2_port}"; } + backend s1 { .via = v2; .authority = "1.2.3.4.example.com"; .host = "${s1_addr}"; .port = "${s1_port}"; } + backend s2 { .via = v2; .authority = "s2"; .host = "${s2_addr}"; .port = "${s2_port}"; } sub vcl_recv { if (req.url ~ "^/s1/") { @@ -65,19 +65,19 @@ client c1 { txreq -url /s1/1 rxresp expect resp.status == 200 - expect resp.http.Authority == "${s1_addr}" + expect resp.http.Authority == "1.2.3.4.example.com" expect resp.http.Server == "s1" txreq -url /s2/1 rxresp expect resp.status == 200 - expect resp.http.Authority == "${s2_addr}" + expect resp.http.Authority == "s2" expect resp.http.Server == "s2" txreq -url /s1/2 rxresp expect resp.status == 200 - expect resp.http.Authority == "${s1_addr}" + expect resp.http.Authority == "1.2.3.4.example.com" expect resp.http.Server == "s1" } -run @@ -150,6 +150,27 @@ client c1 { expect resp.http.Authority == "" } -run +varnish v1 -errvcl ".host used as authority can not be an ip address with .via" { + backend v2 { .host = "${v2_addr}"; .port = "${v2_port}"; } + + backend s1 { + .via = v2; + .host = "${s1_addr}"; + .port = "${s1_port}"; + } +} + +varnish v1 -errvcl ".host_header used as authority can not be an ip address with .via" { + backend v2 { .host = "${v2_addr}"; .port = "${v2_port}"; } + + backend s1 { + .via = v2; + .host = "${s1_addr}"; + .host_header = "${s1_addr}"; + .port = "${s1_port}"; + } +} + varnish v1 -errvcl "Cannot set both .via and .path" { backend v2 { .host = "${v2_addr}"; .port = "${v2_port}"; } @@ -164,11 +185,13 @@ varnish v1 -errvcl "Cannot stack .via backends" { backend b { .via = a; + .host_header = "b"; .host = "127.0.0.1"; } backend c { .via = b; + .authority = "c"; .host = "127.0.0.2"; } @@ -180,5 +203,5 @@ varnish v1 -errvcl "Cannot stack .via backends" { # issue #4177: backend named default with .via property varnish v1 -vcl { backend via { .host = "${localhost}"; } - backend default { .via = via; .host = "${localhost}"; } + backend default { .via = via; .authority = "localhost"; .host = "${localhost}"; } } diff --git a/doc/sphinx/reference/vcl-backend.rst b/doc/sphinx/reference/vcl-backend.rst index 00904acbf..24ffc28fc 100644 --- a/doc/sphinx/reference/vcl-backend.rst +++ b/doc/sphinx/reference/vcl-backend.rst @@ -219,6 +219,8 @@ The HTTP authority to use when connecting to this backend. If unset, ``.authority = ""`` disables sending an authority. +``.authority`` can not be an IP address. + As of this release, the attribute is only used by ``.via`` connections as a ``PP2_TYPE_AUTHORITY`` Type-Length-Value (TLV) in the `PROXY2`_ preamble. diff --git a/lib/libvcc/vcc_backend.c b/lib/libvcc/vcc_backend.c index 7e6bf64c6..7b61703f5 100644 --- a/lib/libvcc/vcc_backend.c +++ b/lib/libvcc/vcc_backend.c @@ -36,8 +36,14 @@ #include #include #include +#include +#include +#include + #include "vcc_compile.h" +#include "vsa.h" +#include "vss.h" #include "vus.h" /*-------------------------------------------------------------------- @@ -75,6 +81,18 @@ Emit_Sockaddr(struct vcc *tl, struct vsb *vsb1, const struct token *t_host, VSB_cat(vsb1, "\t.uds_path = (void *) 0,\n"); } +/* + * a string represents an IP address if getaddrinfo(AI_NUMERICHOST) succeeds + */ +static int +isip(const char *addr) +{ + char buf[vsa_suckaddr_len]; + + return (VSS_ResolveOne(buf, addr, "80", AF_UNSPEC, SOCK_STREAM, + AI_NUMERICHOST | AI_NUMERICSERV) != NULL); +} + /* * For UDS, we do not create a VSA. We run the VUS_resolver() checks and, if * it's a path, can be accessed, and is a socket. If so, just emit the path @@ -660,10 +678,12 @@ vcc_ParseHostDef(struct vcc *tl, struct symbol *sym, * * When authority is "", sending the TLV is disabled. * - * Falling back to host may result in an IP address in authority, - * which is an illegal SNI HostName (RFC 4366 ch. 3.1). But we - * document the potential error, rather than try to find out - * whether or not Emit_Sockaddr() had to look up a name. + * authority must be a valid SNI HostName (RFC 4366 ch. 3.1), but the + * RFC does not define what that is and defers to "DNS hostname". + * + * So instead of trying to attempt a solution to that pandora's box, we + * just implement >>Literal IPv4 and IPv6 addresses are not permitted in + * "HostName".<< */ if (via != NULL) { AN(t_host); @@ -675,6 +695,17 @@ vcc_ParseHostDef(struct vcc *tl, struct symbol *sym, t_val = t_host; p = t_val->dec; + if (isip(p)) { + if (t_val == t_authority) + VSB_cat(tl->sb, ".authority can not be an ip address with .via.\n"); + else { + VSB_printf(tl->sb, ".%s used as authority can not be an ip address with .via.\n", + t_val == t_hosthdr ? "host_header" : "host"); + VSB_cat(tl->sb, "Hint: configure .authority explicitly\n"); + } + vcc_ErrWhere(tl, t_val); + } + Fb(tl, 0, "\t.authority = "); VSB_quote(tl->fb, p, -1, VSB_QUOTE_CSTR); Fb(tl, 0, ",\n"); From nils.goroll at uplex.de Mon Feb 10 15:02:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 10 Feb 2025 15:02:05 +0000 (UTC) Subject: [master] 96bc829f3 vcc: Ensure the authority does not contain a : Message-ID: <20250210150206.027F710D37B@lists.varnish-cache.org> commit 96bc829f3c1da738032c3fe0590072728904d834 Author: Nils Goroll Date: Fri Feb 7 10:32:35 2025 +0100 vcc: Ensure the authority does not contain a : Fixes #3963 diff --git a/bin/varnishtest/tests/c00042.vtc b/bin/varnishtest/tests/c00042.vtc index 60fd5c994..6268cded9 100644 --- a/bin/varnishtest/tests/c00042.vtc +++ b/bin/varnishtest/tests/c00042.vtc @@ -110,7 +110,7 @@ varnish v1 -vcl { .via = v2; .host = "${s1_addr}"; .port = "${s1_port}"; - .host_header = "host.com"; + .host_header = "host.com:1234"; } sub vcl_recv { diff --git a/doc/sphinx/reference/vcl-backend.rst b/doc/sphinx/reference/vcl-backend.rst index 24ffc28fc..6729df72b 100644 --- a/doc/sphinx/reference/vcl-backend.rst +++ b/doc/sphinx/reference/vcl-backend.rst @@ -221,6 +221,8 @@ The HTTP authority to use when connecting to this backend. If unset, ``.authority`` can not be an IP address. +A colon and anything following (signifying a port) is removed from the authority. + As of this release, the attribute is only used by ``.via`` connections as a ``PP2_TYPE_AUTHORITY`` Type-Length-Value (TLV) in the `PROXY2`_ preamble. diff --git a/lib/libvcc/vcc_backend.c b/lib/libvcc/vcc_backend.c index 7b61703f5..3d4a1779f 100644 --- a/lib/libvcc/vcc_backend.c +++ b/lib/libvcc/vcc_backend.c @@ -392,8 +392,9 @@ vcc_ParseHostDef(struct vcc *tl, struct symbol *sym, vtim_dur first_byte_timeout = NAN; vtim_dur between_bytes_timeout = NAN; vtim_dur backend_wait_timeout = NAN; - char *p; + char *p, *pp; unsigned u; + int l; if (tl->t->tok == ID && (vcc_IdIs(tl->t, "none") || vcc_IdIs(tl->t, "None"))) { @@ -706,8 +707,11 @@ vcc_ParseHostDef(struct vcc *tl, struct symbol *sym, vcc_ErrWhere(tl, t_val); } + pp = strchr(p, ':'); + l = (pp == NULL) ? -1 : (int)(pp - p); + Fb(tl, 0, "\t.authority = "); - VSB_quote(tl->fb, p, -1, VSB_QUOTE_CSTR); + VSB_quote(tl->fb, p, l, VSB_QUOTE_CSTR); Fb(tl, 0, ",\n"); } From nils.goroll at uplex.de Mon Feb 10 16:29:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 10 Feb 2025 16:29:06 +0000 (UTC) Subject: [master] 8b53dbeda cache_req: Refactor: Use REQ_FLAGS for esi and pipe marker Message-ID: <20250210162906.5E5131106E8@lists.varnish-cache.org> commit 8b53dbeda3fe45621ed47182ba1e4cb40aac2ff2 Author: Nils Goroll Date: Mon Feb 10 17:06:22 2025 +0100 cache_req: Refactor: Use REQ_FLAGS for esi and pipe marker No need to spend 4 bytes on just two bits... Note: RES_ESI could be replaced by "check if the ESI VDP is on the stack", but as long as we have the marker for pipe, we might as well keep it for ESI, too. diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index ffa3d8577..c15f0833c 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -470,11 +470,6 @@ struct req { unsigned max_restarts; unsigned esi_level; - /* Delivery mode */ - unsigned res_mode; -#define RES_ESI (1<<4) -#define RES_PIPE (1<<7) - const struct req_step *req_step; struct reqtop *top; /* esi_level == 0 request */ diff --git a/bin/varnishd/cache/cache_backend.c b/bin/varnishd/cache/cache_backend.c index c4f64f42e..645207b39 100644 --- a/bin/varnishd/cache/cache_backend.c +++ b/bin/varnishd/cache/cache_backend.c @@ -511,7 +511,7 @@ vbe_dir_http1pipe(VRT_CTX, VCL_BACKEND d) v1a.req = ctx->req->acct.req_hdrbytes; ctx->req->acct.req_hdrbytes = 0; - ctx->req->res_mode = RES_PIPE; + ctx->req->res_pipe = 1; retval = SC_TX_ERROR; pfd = vbe_dir_getfd(ctx, ctx->req->wrk, d, bp, 0); diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c index 6a336b549..f739b22c3 100644 --- a/bin/varnishd/cache/cache_esi_deliver.c +++ b/bin/varnishd/cache/cache_esi_deliver.c @@ -290,7 +290,7 @@ ved_vdp_esi_init(VRT_CTX, struct vdp_ctx *vdc, void **priv) *priv = ecx; RFC2616_Weaken_Etag(vdc->hp); - ctx->req->res_mode |= RES_ESI; + ctx->req->res_esi = 1; if (*vdc->clen != 0) *vdc->clen = -1; if (ctx->req->esi_level > 0) { @@ -909,7 +909,7 @@ ved_deliver(struct req *req, int wantbody) INIT_OBJ(ctx, VRT_CTX_MAGIC); VCL_Req2Ctx(ctx, req); - if (ecx->isgzip && i && !(req->res_mode & RES_ESI)) { + if (ecx->isgzip && i && !req->res_esi) { /* A gzipped include which is not ESI processed */ /* OA_GZIPBITS are not valid until BOS_FINISHED */ diff --git a/bin/varnishd/cache/cache_req.c b/bin/varnishd/cache/cache_req.c index ac188560c..0511ae09b 100644 --- a/bin/varnishd/cache/cache_req.c +++ b/bin/varnishd/cache/cache_req.c @@ -55,7 +55,7 @@ Req_AcctLogCharge(struct VSC_main_wrk *ds, struct req *req) a = &req->acct; - if (!IS_NO_VXID(req->vsl->wid) && !(req->res_mode & RES_PIPE)) { + if (!IS_NO_VXID(req->vsl->wid) && !req->res_pipe) { VSLb(req->vsl, SLT_ReqAcct, "%ju %ju %ju %ju %ju %ju", (uintmax_t)a->req_hdrbytes, (uintmax_t)a->req_bodybytes, diff --git a/bin/varnishd/cache/cache_req_fsm.c b/bin/varnishd/cache/cache_req_fsm.c index 89f71f0a9..1004cbc5f 100644 --- a/bin/varnishd/cache/cache_req_fsm.c +++ b/bin/varnishd/cache/cache_req_fsm.c @@ -440,7 +440,7 @@ cnt_transmit(struct worker *wrk, struct req *req) CHECK_OBJ_NOTNULL(req->transport, TRANSPORT_MAGIC); CHECK_OBJ_NOTNULL(req->objcore, OBJCORE_MAGIC); AZ(req->stale_oc); - AZ(req->res_mode); + AZ(req->res_pipe | req->res_esi); AZ(req->boc); req->req_step = R_STP_FINISH; @@ -533,7 +533,8 @@ cnt_finish(struct worker *wrk, struct req *req) http_Teardown(req->resp); req->vdp_filter_list = NULL; - req->res_mode = 0; + req->res_pipe = 0; + req->res_esi = 0; return (REQ_FSM_DONE); } diff --git a/include/tbl/req_flags.h b/include/tbl/req_flags.h index 6a9e6acbb..dde208d16 100644 --- a/include/tbl/req_flags.h +++ b/include/tbl/req_flags.h @@ -41,6 +41,8 @@ REQ_FLAG(waitinglist, 0, 0, "") REQ_FLAG(want100cont, 0, 0, "") REQ_FLAG(late100cont, 0, 0, "") REQ_FLAG(req_reset, 0, 0, "") +REQ_FLAG(res_esi, 0, 0, "") +REQ_FLAG(res_pipe, 0, 0, "") #define REQ_BEREQ_FLAG(lower, vcl_r, vcl_w, doc) \ REQ_FLAG(lower, vcl_r, vcl_w, doc) #include "tbl/req_bereq_flags.h" From nils.goroll at uplex.de Mon Feb 10 16:29:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 10 Feb 2025 16:29:06 +0000 (UTC) Subject: [master] 926e8be7d cache_req: pack struct req better Message-ID: <20250210162906.79F3B1106EC@lists.varnish-cache.org> commit 926e8be7d62d7a12cab93002d47b44c38ed4f23d Author: Nils Goroll Date: Mon Feb 10 17:25:07 2025 +0100 cache_req: pack struct req better diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index c15f0833c..297243328 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -464,19 +464,19 @@ struct req { unsigned magic; #define REQ_MAGIC 0xfb4abf6d + unsigned esi_level; body_status_t req_body_status; stream_close_t doclose; unsigned restarts; unsigned max_restarts; - unsigned esi_level; const struct req_step *req_step; struct reqtop *top; /* esi_level == 0 request */ + uint16_t err_code; #define REQ_FLAG(l, r, w, d) unsigned l:1; #include "tbl/req_flags.h" - uint16_t err_code; const char *err_reason; struct sess *sp; From nils.goroll at uplex.de Mon Feb 10 16:45:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 10 Feb 2025 16:45:06 +0000 (UTC) Subject: [master] 001e77a95 cache_fetch: Pack struct busyobj better Message-ID: <20250210164506.6FE681112BE@lists.varnish-cache.org> commit 001e77a9524c503995fd54f11040baa0b0361c1b Author: Nils Goroll Date: Mon Feb 10 17:43:34 2025 +0100 cache_fetch: Pack struct busyobj better diff --git a/bin/varnishd/cache/cache.h b/bin/varnishd/cache/cache.h index 297243328..82bede0f4 100644 --- a/bin/varnishd/cache/cache.h +++ b/bin/varnishd/cache/cache.h @@ -410,11 +410,16 @@ struct busyobj { struct pool_task fetch_task[1]; + const char *err_reason; + enum director_state_e director_state; + uint16_t err_code; + #define BERESP_FLAG(l, r, w, f, d) unsigned l:1; #define BEREQ_FLAG(l, r, w, d) BERESP_FLAG(l, r, w, 0, d) #include "tbl/bereq_flags.h" #include "tbl/beresp_flags.h" + /* Timeouts */ vtim_dur connect_timeout; vtim_dur first_byte_timeout; @@ -432,7 +437,6 @@ struct busyobj { const struct stevedore *storage; const struct director *director_req; const struct director *director_resp; - enum director_state_e director_state; struct vcl *vcl; struct vsl_log vsl[1]; @@ -440,9 +444,6 @@ struct busyobj { uint8_t digest[DIGEST_LEN]; struct vrt_privs privs[1]; - uint16_t err_code; - const char *err_reason; - const char *client_identity; }; From nils.goroll at uplex.de Mon Feb 10 19:03:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 10 Feb 2025 19:03:05 +0000 (UTC) Subject: [master] 593632401 miniobj.h: Fix ALLOC_FLEX_OBJ Message-ID: <20250210190305.B8D80115F8E@lists.varnish-cache.org> commit 593632401763f463e7d611cc6cc916ccf6208cc4 Author: Nils Goroll Date: Mon Feb 10 19:57:03 2025 +0100 miniobj.h: Fix ALLOC_FLEX_OBJ use of sizeof(base) is actually wrong because of padding. See https://gustedt.wordpress.com/2011/03/14/flexible-array-member/ diff --git a/include/miniobj.h b/include/miniobj.h index 8cb9e293e..ecb341f49 100644 --- a/include/miniobj.h +++ b/include/miniobj.h @@ -38,7 +38,12 @@ } while (0) #define ALLOC_FLEX_OBJ(to, fld, len, type_magic) \ - ALLOC_OBJ_EXTRA(to, (len) * sizeof *((to)->fld), (type_magic)) + do { \ + (to) = calloc(1, offsetof(typeof(*to), fld) + \ + sizeof *(to)->fld * len); \ + if ((to) != NULL) \ + (to)->magic = (type_magic); \ + } while (0) #define FREE_OBJ(to) \ do { \ From nils.goroll at uplex.de Tue Feb 11 13:39:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 11 Feb 2025 13:39:05 +0000 (UTC) Subject: [master] 4fe462a47 Flexelint silencing Message-ID: <20250211133906.03049115F44@lists.varnish-cache.org> commit 4fe462a4794584c43fb26046d966be8cb7cebf13 Author: Nils Goroll Date: Tue Feb 11 14:38:10 2025 +0100 Flexelint silencing Ref 593632401763f463e7d611cc6cc916ccf6208cc4 diff --git a/flint.lnt b/flint.lnt index 65d641676..f8340ea20 100644 --- a/flint.lnt +++ b/flint.lnt @@ -282,6 +282,9 @@ // "miniobj.h" -emacro(774, REPLACE) +// Flexelint does not understand offsetof(typeof(*ptr)) +-emacro(10, ALLOC_FLEX_OBJ) +-emacro(40, ALLOC_FLEX_OBJ) // It is ok to default after handling a few select SLT_* tags From nils.goroll at uplex.de Thu Feb 13 09:37:07 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 13 Feb 2025 09:37:07 +0000 (UTC) Subject: [master] 8bc28ccd3 miniobj: Extract SIZEOF_FLEX_OBJ() macro Message-ID: <20250213093707.70DFC122DD8@lists.varnish-cache.org> commit 8bc28ccd3417092a1197845e5450214a126a82e4 Author: Dridi Boukelmoune Date: Thu Feb 13 08:49:22 2025 +0100 miniobj: Extract SIZEOF_FLEX_OBJ() macro committer edit: Updated Flexelint silencing Picked from #4271 diff --git a/flint.lnt b/flint.lnt index f8340ea20..4cc5cb348 100644 --- a/flint.lnt +++ b/flint.lnt @@ -283,8 +283,8 @@ -emacro(774, REPLACE) // Flexelint does not understand offsetof(typeof(*ptr)) --emacro(10, ALLOC_FLEX_OBJ) --emacro(40, ALLOC_FLEX_OBJ) +-emacro(10, SIZEOF_FLEX_OBJ) +-emacro(40, SIZEOF_FLEX_OBJ) // It is ok to default after handling a few select SLT_* tags diff --git a/include/miniobj.h b/include/miniobj.h index ecb341f49..df5018d96 100644 --- a/include/miniobj.h +++ b/include/miniobj.h @@ -37,10 +37,12 @@ (to)->magic = (type_magic); \ } while (0) +#define SIZEOF_FLEX_OBJ(to, fld, len) \ + (offsetof(typeof(*to), fld) + sizeof *(to)->fld * len) + #define ALLOC_FLEX_OBJ(to, fld, len, type_magic) \ do { \ - (to) = calloc(1, offsetof(typeof(*to), fld) + \ - sizeof *(to)->fld * len); \ + (to) = calloc(1, SIZEOF_FLEX_OBJ(to, fld, len)); \ if ((to) != NULL) \ (to)->magic = (type_magic); \ } while (0) From dridi.boukelmoune at gmail.com Thu Feb 13 11:34:06 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 13 Feb 2025 11:34:06 +0000 (UTC) Subject: [master] eb85bf87c vrt_filter: Minor polish Message-ID: <20250213113406.640E5126919@lists.varnish-cache.org> commit eb85bf87cae0b9feca74a20a1cee2b54a2b06d57 Author: Dridi Boukelmoune Date: Thu Feb 13 12:22:39 2025 +0100 vrt_filter: Minor polish I'm not convinced by req_filter_can(). Filters have no effect on a bereq, beresp or resp without a body, why should it be different for req? We should be consistent here. diff --git a/bin/varnishd/cache/cache_vrt_filter.c b/bin/varnishd/cache/cache_vrt_filter.c index 18077c220..c3309d77d 100644 --- a/bin/varnishd/cache/cache_vrt_filter.c +++ b/bin/varnishd/cache/cache_vrt_filter.c @@ -404,7 +404,7 @@ static const char * bereq_Empty_Filter(struct busyobj *bo) { - (void)bo; + CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); return (""); } @@ -437,6 +437,7 @@ resp_default_filter_list(void *arg, struct vsb *vsb) const char * resp_Get_Filter_List(struct req *req) { + CHECK_OBJ_NOTNULL(req, REQ_MAGIC); return (filter_on_ws(req->ws, resp_default_filter_list, req)); } @@ -445,15 +446,19 @@ static const char * req_Empty_Filter(struct req *req) { - (void)req; + CHECK_OBJ_NOTNULL(req, REQ_MAGIC); return (""); } -/*--------------------------------------------------------------------*/ +/*-------------------------------------------------------------------- + * XXX: why ignore needless filters for everything but req? + */ + static int -req_filter_can(struct req *req) { - CHECK_OBJ_NOTNULL(req, REQ_MAGIC); +req_filter_can(struct req *req) +{ + CHECK_OBJ_NOTNULL(req, REQ_MAGIC); return (req->req_body_status->avail == 1); } From nils.goroll at uplex.de Thu Feb 13 14:49:56 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 13 Feb 2025 15:49:56 +0100 Subject: [master] eb85bf87c vrt_filter: Minor polish In-Reply-To: <20250213113406.640E5126919@lists.varnish-cache.org> References: <20250213113406.640E5126919@lists.varnish-cache.org> Message-ID: On 13.02.25 12:34, Dridi Boukelmoune wrote: > > commit eb85bf87cae0b9feca74a20a1cee2b54a2b06d57 > Author: Dridi Boukelmoune > Date: Thu Feb 13 12:22:39 2025 +0100 > > vrt_filter: Minor polish > > I'm not convinced by req_filter_can(). Filters have no effect on a > bereq, beresp or resp without a body, why should it be different for > req? We should be consistent here. I would hope that your question was answered by the vcl_var.rst addition from 00b190a0aed482a638fdb9f1207598cca3db6c27: The req body (conversely to others) is one where VCL has control over when it gets processed by means of std.cache_req_body(): Filters act on the received body when that function is called, so confusion could be caused if people expect filters to have an effect when set/changed after the req.body is already cached. To implement trailer support, I think we would need generic vcl "send/receive the body _now_" control, and this mechanism would get generalized. > - (void)bo; > + CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); I am not against, but why null/magic check an unused value? Otherwise thank you for the cstyle touchups. Nils -- Nils Goroll (he/him) ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg tel +49 40 28805731 mob +49 170 2723133 fax +49 40 42949753 xmpp://slink at jabber.int.uplex.de/ http://uplex.de/ -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_0x1DCD8F57A3868BD7.asc Type: application/pgp-keys Size: 3943 bytes Desc: OpenPGP public key URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature.asc Type: application/pgp-signature Size: 488 bytes Desc: OpenPGP digital signature URL: From nils.goroll at uplex.de Thu Feb 13 14:52:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 13 Feb 2025 14:52:06 +0000 (UTC) Subject: [master] 0c10a6a52 Polish WITH_PERSISTENT_STORAGE use Message-ID: <20250213145206.8D7E8103938@lists.varnish-cache.org> commit 0c10a6a520f85a8a5e3f3192fa3d06a247c2de4d Author: Nils Goroll Date: Thu Feb 13 15:38:45 2025 +0100 Polish WITH_PERSISTENT_STORAGE use The macro is either defined as 1 or undefined. If undefined, Flexelint complained: Warning 553: Undefined preprocessor variable 'WITH_PERSISTENT_STORAGE', assumed 0 diff --git a/bin/varnishd/cache/cache_main.c b/bin/varnishd/cache/cache_main.c index c71a6a108..c74292bdb 100644 --- a/bin/varnishd/cache/cache_main.c +++ b/bin/varnishd/cache/cache_main.c @@ -482,7 +482,7 @@ child_main(int sigmagic, size_t altstksz) CLI_AddFuncs(debug_cmds); CLI_AddFuncs(child_cmds); -#if WITH_PERSISTENT_STORAGE +#ifdef WITH_PERSISTENT_STORAGE /* Wait for persistent storage to load if asked to */ if (FEATURE(FEATURE_WAIT_SILO)) SMP_Ready(); diff --git a/bin/varnishd/cache/cache_varnishd.h b/bin/varnishd/cache/cache_varnishd.h index 34a616c8d..12b3dd416 100644 --- a/bin/varnishd/cache/cache_varnishd.h +++ b/bin/varnishd/cache/cache_varnishd.h @@ -587,7 +587,7 @@ struct stv_buffer *STV_AllocBuf(struct worker *wrk, const struct stevedore *stv, void STV_FreeBuf(struct worker *wrk, struct stv_buffer **pstvbuf); void *STV_GetBufPtr(struct stv_buffer *stvbuf, size_t *psize); -#if WITH_PERSISTENT_STORAGE +#ifdef WITH_PERSISTENT_STORAGE /* storage_persistent.c */ void SMP_Ready(void); #endif From nils.goroll at uplex.de Thu Feb 13 16:01:07 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 13 Feb 2025 16:01:07 +0000 (UTC) Subject: [master] a0a3c3ea2 Flexelint: consistently define WITH_PERSISTENT_STORAGE Message-ID: <20250213160107.333941060F0@lists.varnish-cache.org> commit a0a3c3ea2d57f6ab993ecc198df439cb14991645 Author: Nils Goroll Date: Thu Feb 13 16:58:47 2025 +0100 Flexelint: consistently define WITH_PERSISTENT_STORAGE if the build was configured without it defined, flexelint would still be run on storage_persistent.c. In order to not complicate the flexelint invocation, just always define it. diff --git a/bin/varnishd/flint.sh b/bin/varnishd/flint.sh index b76ee681b..b26f0f27a 100644 --- a/bin/varnishd/flint.sh +++ b/bin/varnishd/flint.sh @@ -11,6 +11,7 @@ FLOPS=' -DVARNISH_STATE_DIR="foo" -DVARNISH_VMOD_DIR="foo" -DVARNISH_VCL_DIR="foo" + -DWITH_PERSISTENT_STORAGE acceptor/*.c cache/*.c common/*.c diff --git a/bin/varnishd/storage/storage_persistent.c b/bin/varnishd/storage/storage_persistent.c index df2fc9128..f193ae976 100644 --- a/bin/varnishd/storage/storage_persistent.c +++ b/bin/varnishd/storage/storage_persistent.c @@ -61,6 +61,10 @@ static struct VSC_lck *lck_smp; static void smp_init(void); +#ifndef WITH_PERSISTENT_STORAGE +#error "WITH_PERSISTENT_STORAGE must be defined" +#endif + /*--------------------------------------------------------------------*/ /* From nils.goroll at uplex.de Thu Feb 13 16:15:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 13 Feb 2025 16:15:06 +0000 (UTC) Subject: [master] 45cdc60e9 mgt: Simplify key=val argument handling Message-ID: <20250213161506.72D02106AAC@lists.varnish-cache.org> commit 45cdc60e96e9cc2052c523239e99e774baf656f1 Author: Nils Goroll Date: Wed Feb 12 20:39:12 2025 +0100 mgt: Simplify key=val argument handling we add a simple utility function to return the pointer to the thing after "key=" if the first argument begins with "key=". Note that this is no worse than a macro with a "reasonable good" compiler *) and default (-O2) optimization, because the function call gets inlined and the strlen() turned into a constant. 0x00000000000a1bd1 <+49>: lea 0x6ed49(%rip),%r14 # 0x110921 ... 0x00000000000a1c01 <+97>: mov $0x5,%edx 0x00000000000a1c06 <+102>: mov %rbx,%rdi 0x00000000000a1c09 <+105>: mov %r14,%rsi 0x00000000000a1c0c <+108>: call 0x2d360 0x00000000000a1c11 <+113>: lea 0x5(%rbx),%rbp 0x00000000000a1c15 <+117>: test %eax,%eax (gdb) p (const char *)0x110921 $2 = 0x110921 "user=" Replaces #4202 Polishes 4001ea251bac6924c2e86bac3d0f2c8d93c96bd8 *) tested: gcc version 12.2.0 (Debian 12.2.0-14) Debian clang version 14.0.6 diff --git a/bin/varnishd/acceptor/mgt_acceptor_uds.c b/bin/varnishd/acceptor/mgt_acceptor_uds.c index c10fa7af6..51c558129 100644 --- a/bin/varnishd/acceptor/mgt_acceptor_uds.c +++ b/bin/varnishd/acceptor/mgt_acceptor_uds.c @@ -179,10 +179,9 @@ vca_uds_open(char **av, struct listen_arg *la, const char **err) heritage.min_vcl_version = vmax(heritage.min_vcl_version, 41U); for (int i = 0; av[i] != NULL; i++) { - char *eq, *val; - int len; + const char *val; - if ((eq = strchr(av[i], '=')) == NULL) { + if (strchr(av[i], '=') == NULL) { if (xp != NULL) ARGV_ERR("Too many protocol sub-args" " in -a (%s)\n", av[i]); @@ -195,41 +194,35 @@ vca_uds_open(char **av, struct listen_arg *la, const char **err) ARGV_ERR("Invalid sub-arg %s" " in -a\n", av[i]); - val = eq + 1; - len = eq - av[i]; - assert(len >= 0); - if (len == 0) - ARGV_ERR("Invalid sub-arg %s in -a\n", av[i]); - - if (strncmp(av[i], "user", len) == 0) { - if (pwd != NULL) - ARGV_ERR("Too many user sub-args in -a (%s)\n", - av[i]); + val = keyval(av[i], "user="); + if (val != NULL && pwd != NULL) + ARGV_ERR("Too many user sub-args in -a (%s)\n", av[i]); + if (val != NULL) { pwd = getpwnam(val); if (pwd == NULL) ARGV_ERR("Unknown user %s in -a\n", val); continue; } - if (strncmp(av[i], "group", len) == 0) { - if (grp != NULL) - ARGV_ERR("Too many group sub-args in -a (%s)\n", - av[i]); + val = keyval(av[i], "group="); + if (val != NULL && grp != NULL) + ARGV_ERR("Too many group sub-args in -a (%s)\n", av[i]); + if (val != NULL) { grp = getgrnam(val); if (grp == NULL) ARGV_ERR("Unknown group %s in -a\n", val); continue; } - if (strncmp(av[i], "mode", len) == 0) { + val = keyval(av[i], "mode="); + if (val != NULL && mode != 0) + ARGV_ERR("Too many mode sub-args in -a (%s)\n", av[i]); + if (val != NULL && *val == '\0') + ARGV_ERR("Empty mode sub-arg in -a\n"); + if (val != NULL) { long m; char *p; - if (mode != 0) - ARGV_ERR("Too many mode sub-args in -a (%s)\n", - av[i]); - if (*val == '\0') - ARGV_ERR("Empty mode sub-arg in -a\n"); errno = 0; m = strtol(val, &p, 8); if (*p != '\0') diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h index f98862363..172288ab2 100644 --- a/bin/varnishd/mgt/mgt.h +++ b/bin/varnishd/mgt/mgt.h @@ -38,6 +38,7 @@ #include +#include "string.h" #include #include "vdef.h" @@ -64,6 +65,16 @@ extern struct vev_root *mgt_evb; extern unsigned d_flag; extern int exit_status; +/* option=value argument parse helper */ +static inline const char * +keyval(const char *p, const char *name) +{ + size_t l = strlen(name); + if (strncmp(p, name, l)) + return (NULL); + return (p + l); +} + /* builtin_vcl.c */ extern const char * const builtin_vcl; diff --git a/bin/varnishd/mgt/mgt_jail_solaris.c b/bin/varnishd/mgt/mgt_jail_solaris.c index d35d93726..de685f8f8 100644 --- a/bin/varnishd/mgt/mgt_jail_solaris.c +++ b/bin/varnishd/mgt/mgt_jail_solaris.c @@ -323,13 +323,14 @@ static int v_matchproto_(jail_init_f) vjs_init(char **args) { priv_set_t **sets, *permitted, *inheritable, *user = NULL; - const char *e; + const char *e, *val; int vj, vs; if (args != NULL && *args != NULL) { for (;*args != NULL; args++) { - if (!strncmp(*args, "worker=", 7)) { - user = priv_str_to_set((*args) + 7, ",", &e); + val = keyval(*args, "worker="); + if (val != NULL) { + user = priv_str_to_set(val, ",", &e); if (user == NULL) ARGV_ERR( "-jsolaris: parsing worker= " diff --git a/bin/varnishd/mgt/mgt_jail_unix.c b/bin/varnishd/mgt/mgt_jail_unix.c index 66e088253..9ca2f234b 100644 --- a/bin/varnishd/mgt/mgt_jail_unix.c +++ b/bin/varnishd/mgt/mgt_jail_unix.c @@ -117,9 +117,15 @@ vju_getccgid(const char *arg) /********************************************************************** */ +// avoid line breaks +#define JERR(what, val) \ + ARGV_ERR("Unix jail: %s " what " not found.\n", val) + static int v_matchproto_(jail_init_f) vju_init(char **args) { + const char *val; + if (args == NULL) { /* Autoconfig */ if (geteuid() != 0) @@ -132,31 +138,22 @@ vju_init(char **args) ARGV_ERR("Unix Jail: Must be root.\n"); for (;*args != NULL; args++) { - const char * const a_user = "user="; - const size_t l_user = strlen(a_user); - if (!strncmp(*args, a_user, l_user)) { - if (vju_getuid((*args) + l_user)) - ARGV_ERR( - "Unix jail: %s user not found.\n", - (*args) + 5); + val = keyval(*args, "user="); + if (val != NULL) { + if (vju_getuid(val)) + JERR("user", val); continue; } - const char * const a_workuser = "workuser="; - const size_t l_workuser = strlen(a_workuser); - if (!strncmp(*args, a_workuser, l_workuser)) { - if (vju_getwrkuid((*args) + l_workuser)) - ARGV_ERR( - "Unix jail: %s user not found.\n", - (*args) + 9); + val = keyval(*args, "workuser="); + if (val != NULL) { + if (vju_getwrkuid(val)) + JERR("user", val); continue; } - const char * const a_ccgroup = "ccgroup="; - const size_t l_ccgroup = strlen(a_ccgroup); - if (!strncmp(*args, "ccgroup=", l_ccgroup)) { - if (vju_getccgid((*args) + l_ccgroup)) - ARGV_ERR( - "Unix jail: %s group not found.\n", - (*args) + 8); + val = keyval(*args, "ccgroup="); + if (val != NULL) { + if (vju_getccgid(val)) + JERR("group", val); continue; } ARGV_ERR("Unix jail: unknown sub-argument '%s'\n", @@ -164,8 +161,7 @@ vju_init(char **args) } if (vju_user == NULL && vju_getuid(VARNISH_USER)) - ARGV_ERR("Unix jail: %s user not found.\n", - VARNISH_USER); + JERR("user", VARNISH_USER); } AN(vju_user); @@ -187,6 +183,8 @@ vju_init(char **args) return (0); } +#undef JERR + static void v_matchproto_(jail_master_f) vju_master(enum jail_master_e jme) { From nils.goroll at uplex.de Thu Feb 13 20:02:07 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 13 Feb 2025 20:02:07 +0000 (UTC) Subject: [master] 5d97901af mgt_jail: Make error reporting more consistent Message-ID: <20250213200207.169B5111B6F@lists.varnish-cache.org> commit 5d97901af726de70a9532758324d6e4d04406a5b Author: Nils Goroll Date: Thu Feb 13 20:51:24 2025 +0100 mgt_jail: Make error reporting more consistent Fixes #4273 diff --git a/bin/varnishd/common/heritage.h b/bin/varnishd/common/heritage.h index 043343bbc..aedc7c295 100644 --- a/bin/varnishd/common/heritage.h +++ b/bin/varnishd/common/heritage.h @@ -105,13 +105,18 @@ extern struct heritage heritage; /* Really belongs in mgt.h, but storage_file chokes on both */ void MCH_Fd_Inherit(int fd, const char *what); -#define ARGV_ERR(...) \ +#define ARGV_EXIT \ do { \ - fprintf(stderr, "Error: " __VA_ARGS__); \ fprintf(stderr, "(-? gives usage)\n"); \ exit(2); \ } while (0) +#define ARGV_ERR(...) \ + do { \ + fprintf(stderr, "Error: " __VA_ARGS__); \ + ARGV_EXIT; \ + } while (0) + /* cache/cache_main.c */ void child_main(int, size_t); diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c index d67acac69..2fbcfccad 100644 --- a/bin/varnishd/mgt/mgt_main.c +++ b/bin/varnishd/mgt/mgt_main.c @@ -879,8 +879,7 @@ main(int argc, char * const *argv) openlog("varnishd", LOG_PID, LOG_LOCAL0); if (VJ_make_workdir(workdir)) - ARGV_ERR("Cannot create working directory (%s): %s\n", - workdir, VAS_errtxt(errno)); + ARGV_EXIT; VJ_master(JAIL_MASTER_SYSTEM); #ifdef RLIMIT_MEMLOCK From nils.goroll at uplex.de Thu Feb 13 20:02:07 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 13 Feb 2025 20:02:07 +0000 (UTC) Subject: [master] a0f16286f jail_unix: Improve error reporting of failed chown() Message-ID: <20250213200207.30EB7111B72@lists.varnish-cache.org> commit a0f16286fa859d534eb073d9fe550de6532c0b61 Author: Nils Goroll Date: Thu Feb 13 20:57:11 2025 +0100 jail_unix: Improve error reporting of failed chown() Before this patch, configuring an existing read only directory as the working directory would result in an assertion failure: $ mount | grep /tmp/ro swap on /tmp/ro type tmpfs (ro,relatime,inode64) $ sudo /tmp/sbin/varnishd -j unix -n /tmp/ro -a @a -b @b Assert error in vju_make_workdir(), mgt/mgt_jail_unix.c line 277: Condition((chown(dname, -1, vju_gid)) == 0) not true. errno = 30 (Read-only file system) This is now changed to: $ sudo /tmp/sbin/varnishd -j unix -n /tmp/ro -a @a -b @b Error: Cannot change group of working directory '/tmp/ro': Read-only file system (-? gives usage) Note: There are more {f,}chown calls in AZ(), but under the traditional model of a super cow powered uid 0, they should, I think, succeed once chown() of the working directory itself succeeded. diff --git a/bin/varnishd/mgt/mgt_jail_unix.c b/bin/varnishd/mgt/mgt_jail_unix.c index 9ca2f234b..18e2fcb0f 100644 --- a/bin/varnishd/mgt/mgt_jail_unix.c +++ b/bin/varnishd/mgt/mgt_jail_unix.c @@ -274,7 +274,11 @@ vju_make_workdir(const char *dname, const char *what, struct vsb *vsb) return (1); } //lint -e{570} - AZ(chown(dname, -1, vju_gid)); + if (chown(dname, -1, vju_gid)) { + MGT_Complain(C_ERR, "Cannot change group of working directory '%s': %s", + dname, VAS_errtxt(errno)); + return (1); + } AZ(seteuid(vju_uid)); return (0); } From nils.goroll at uplex.de Thu Feb 13 20:27:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 13 Feb 2025 20:27:05 +0000 (UTC) Subject: [master] c0fa7bd35 mgt: Split the printing part from MGT_Complain() into MGT_ComplainVSB() Message-ID: <20250213202705.A25B1112C8A@lists.varnish-cache.org> commit c0fa7bd35d479528dce0a5587c782c0d25886a4a Author: Nils Goroll Date: Thu Feb 13 17:39:34 2025 +0100 mgt: Split the printing part from MGT_Complain() into MGT_ComplainVSB() diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h index 172288ab2..bcdc81d70 100644 --- a/bin/varnishd/mgt/mgt.h +++ b/bin/varnishd/mgt/mgt.h @@ -219,6 +219,7 @@ char *mgt_HostName(void); void mgt_ProcTitle(const char *comp); void mgt_DumpRstVsl(void); struct vsb *mgt_BuildVident(void); +void MGT_ComplainVSB(const char *, const struct vsb *vsb); void MGT_Complain(const char *, const char *, ...) v_printflike_(2, 3); const void *MGT_Pick(const struct choice *, const char *, const char *); char **MGT_NamedArg(const char *, const char **, const char *); diff --git a/bin/varnishd/mgt/mgt_util.c b/bin/varnishd/mgt/mgt_util.c index f43b43b0d..435e60673 100644 --- a/bin/varnishd/mgt/mgt_util.c +++ b/bin/varnishd/mgt/mgt_util.c @@ -144,20 +144,12 @@ const char C_SECURITY[] = "Security:"; const char C_CLI[] = "Cli:"; void -MGT_Complain(const char *loud, const char *fmt, ...) +MGT_ComplainVSB(const char *loud, const struct vsb *vsb) { - va_list ap; - struct vsb *vsb; int sf; if (loud == C_CLI && !mgt_param.syslog_cli_traffic) return; - vsb = VSB_new_auto(); - AN(vsb); - va_start(ap, fmt); - VSB_vprintf(vsb, fmt, ap); - va_end(ap); - AZ(VSB_finish(vsb)); if (loud == C_ERR) sf = LOG_ERR; @@ -177,6 +169,21 @@ MGT_Complain(const char *loud, const char *fmt, ...) if (!MGT_DO_DEBUG(DBG_VTC_MODE)) syslog(sf, "%s", VSB_data(vsb)); +} + +void +MGT_Complain(const char *loud, const char *fmt, ...) +{ + va_list ap; + struct vsb *vsb; + + vsb = VSB_new_auto(); + AN(vsb); + va_start(ap, fmt); + VSB_vprintf(vsb, fmt, ap); + va_end(ap); + AZ(VSB_finish(vsb)); + MGT_ComplainVSB(loud, vsb); VSB_destroy(&vsb); } From nils.goroll at uplex.de Thu Feb 13 20:27:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 13 Feb 2025 20:27:05 +0000 (UTC) Subject: [master] 9a18ce3bd mgt_jail_unix: Dedup generation of error messages Message-ID: <20250213202705.C9F77112C8D@lists.varnish-cache.org> commit 9a18ce3bdd6e06ef3b6d654fd36d23e6ebfe3372 Author: Nils Goroll Date: Thu Feb 13 18:08:14 2025 +0100 mgt_jail_unix: Dedup generation of error messages Instead of duplicating the code for "a vsb is present" (which was actually never the case) and not so, we always write to a vsb and output it in VJ_make_workdir. diff --git a/bin/varnishd/mgt/mgt_jail.c b/bin/varnishd/mgt/mgt_jail.c index 6ba0a8e9a..bc88a8f55 100644 --- a/bin/varnishd/mgt/mgt_jail.c +++ b/bin/varnishd/mgt/mgt_jail.c @@ -143,12 +143,20 @@ VJ_subproc(enum jail_subproc_e jse) int VJ_make_workdir(const char *dname) { + struct vsb *vsb; int i; AN(dname); CHECK_OBJ_NOTNULL(vjt, JAIL_TECH_MAGIC); + if (vjt->make_workdir != NULL) { - i = vjt->make_workdir(dname, NULL, NULL); + vsb = VSB_new_auto(); + AN(vsb); + i = vjt->make_workdir(dname, NULL, vsb); + AZ(VSB_finish(vsb)); + if (VSB_len(vsb) > 0) + MGT_ComplainVSB(i ? C_ERR : C_INFO, vsb); + VSB_destroy(&vsb); if (i) return (i); VJ_master(JAIL_MASTER_FILE); diff --git a/bin/varnishd/mgt/mgt_jail_linux.c b/bin/varnishd/mgt/mgt_jail_linux.c index 19889c2d0..b8f136020 100644 --- a/bin/varnishd/mgt/mgt_jail_linux.c +++ b/bin/varnishd/mgt/mgt_jail_linux.c @@ -39,7 +39,6 @@ #include #include #include -#include #include #include #include @@ -87,22 +86,19 @@ vjl_make_workdir(const char *dname, const char *what, struct vsb *vsb) { struct statfs info; + AN(vsb); if (jail_tech_unix.make_workdir(dname, what, vsb) != 0) return (1); vjl_master(JAIL_MASTER_FILE); if (statfs(dname, &info) != 0) { - if (vsb) - VSB_printf(vsb, "Could not stat working directory '%s': %s (%d)\n", dname, VAS_errtxt(errno), errno); - else - MGT_Complain(C_ERR, "Could not stat working directory '%s': %s (%d)", dname, VAS_errtxt(errno), errno); + VSB_printf(vsb, "Could not stat working directory '%s':" + " %s (%d)\n", dname, VAS_errtxt(errno), errno); return (1); } if (info.f_type != TMPFS_MAGIC) { - if (vsb != NULL) - VSB_printf(vsb, "Working directory not mounted on tmpfs partition\n"); - else - MGT_Complain(C_INFO, "Working directory not mounted on tmpfs partition"); + VSB_printf(vsb, "Working directory not mounted on" + " tmpfs partition\n"); } vjl_master(JAIL_MASTER_LOW); return (0); diff --git a/bin/varnishd/mgt/mgt_jail_unix.c b/bin/varnishd/mgt/mgt_jail_unix.c index 18e2fcb0f..70414c806 100644 --- a/bin/varnishd/mgt/mgt_jail_unix.c +++ b/bin/varnishd/mgt/mgt_jail_unix.c @@ -265,11 +265,11 @@ vju_make_workdir(const char *dname, const char *what, struct vsb *vsb) AN(dname); AZ(what); - AZ(vsb); + AN(vsb); AZ(seteuid(0)); if (mkdir(dname, 0755) < 0 && errno != EEXIST) { - MGT_Complain(C_ERR, "Cannot create working directory '%s': %s", + VSB_printf(vsb, "Cannot create working directory '%s': %s", dname, VAS_errtxt(errno)); return (1); } From nils.goroll at uplex.de Fri Feb 14 08:53:14 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 14 Feb 2025 08:53:14 +0000 (UTC) Subject: [master] 9d19eff59 mgt: Refactor to use MGT_ComplainVSB more Message-ID: <20250214085314.16A77103169@lists.varnish-cache.org> commit 9d19eff59dd0f5c4fe4fa2d8d9f920feb2773b7d Author: Nils Goroll Date: Fri Feb 14 08:54:03 2025 +0100 mgt: Refactor to use MGT_ComplainVSB more diff --git a/bin/varnishd/mgt/mgt_child.c b/bin/varnishd/mgt/mgt_child.c index 58f0e80b7..ae08c54e6 100644 --- a/bin/varnishd/mgt/mgt_child.c +++ b/bin/varnishd/mgt/mgt_child.c @@ -107,13 +107,12 @@ mgt_panic_record(pid_t r) child_panic = VSB_new_auto(); AN(child_panic); VTIM_format(VTIM_real(), time_str); - VSB_printf(child_panic, "Panic at: %s\n", time_str); + VSB_printf(child_panic, "Child (%jd) Panic at: %s\n", + (intmax_t)r, time_str); VSB_quote(child_panic, heritage.panic_str, strnlen(heritage.panic_str, heritage.panic_str_len), VSB_QUOTE_NONL); - AZ(VSB_finish(child_panic)); - MGT_Complain(C_ERR, "Child (%jd) %s", - (intmax_t)r, VSB_data(child_panic)); + MGT_ComplainVSB(C_ERR, child_panic); } static void @@ -592,8 +591,7 @@ mgt_reap_child(void) VSC_C_mgt->child_dump++; } #endif - AZ(VSB_finish(vsb)); - MGT_Complain(status ? C_ERR : C_INFO, "%s", VSB_data(vsb)); + MGT_ComplainVSB(status ? C_ERR : C_INFO, vsb); VSB_destroy(&vsb); /* Dispose of shared memory but evacuate panic messages first */ From nils.goroll at uplex.de Fri Feb 14 08:53:13 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 14 Feb 2025 08:53:13 +0000 (UTC) Subject: [master] 9056935ef mgt_util: simplify MGT_ComplainVSB Message-ID: <20250214085313.F3ACF103166@lists.varnish-cache.org> commit 9056935ef6420cc299cc897146c0256385cc3550 Author: Nils Goroll Date: Fri Feb 14 09:22:02 2025 +0100 mgt_util: simplify MGT_ComplainVSB move VSB_len() check and VSB_finish() to MGT_ComplainVSB because they are needed every time anyway. Leave the VSB cleared for reuse. diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h index bcdc81d70..f868613fc 100644 --- a/bin/varnishd/mgt/mgt.h +++ b/bin/varnishd/mgt/mgt.h @@ -219,7 +219,7 @@ char *mgt_HostName(void); void mgt_ProcTitle(const char *comp); void mgt_DumpRstVsl(void); struct vsb *mgt_BuildVident(void); -void MGT_ComplainVSB(const char *, const struct vsb *vsb); +void MGT_ComplainVSB(const char *, struct vsb *vsb); void MGT_Complain(const char *, const char *, ...) v_printflike_(2, 3); const void *MGT_Pick(const struct choice *, const char *, const char *); char **MGT_NamedArg(const char *, const char **, const char *); diff --git a/bin/varnishd/mgt/mgt_jail.c b/bin/varnishd/mgt/mgt_jail.c index bc88a8f55..b94e89f0a 100644 --- a/bin/varnishd/mgt/mgt_jail.c +++ b/bin/varnishd/mgt/mgt_jail.c @@ -153,9 +153,7 @@ VJ_make_workdir(const char *dname) vsb = VSB_new_auto(); AN(vsb); i = vjt->make_workdir(dname, NULL, vsb); - AZ(VSB_finish(vsb)); - if (VSB_len(vsb) > 0) - MGT_ComplainVSB(i ? C_ERR : C_INFO, vsb); + MGT_ComplainVSB(i ? C_ERR : C_INFO, vsb); VSB_destroy(&vsb); if (i) return (i); diff --git a/bin/varnishd/mgt/mgt_util.c b/bin/varnishd/mgt/mgt_util.c index 435e60673..7e985d30d 100644 --- a/bin/varnishd/mgt/mgt_util.c +++ b/bin/varnishd/mgt/mgt_util.c @@ -144,12 +144,17 @@ const char C_SECURITY[] = "Security:"; const char C_CLI[] = "Cli:"; void -MGT_ComplainVSB(const char *loud, const struct vsb *vsb) +MGT_ComplainVSB(const char *loud, struct vsb *vsb) { int sf; - if (loud == C_CLI && !mgt_param.syslog_cli_traffic) + if (VSB_len(vsb) == 0 || + (loud == C_CLI && !mgt_param.syslog_cli_traffic)) { + VSB_clear(vsb); return; + } + + AZ(VSB_finish(vsb)); if (loud == C_ERR) sf = LOG_ERR; @@ -169,6 +174,7 @@ MGT_ComplainVSB(const char *loud, const struct vsb *vsb) if (!MGT_DO_DEBUG(DBG_VTC_MODE)) syslog(sf, "%s", VSB_data(vsb)); + VSB_clear(vsb); } void @@ -182,7 +188,6 @@ MGT_Complain(const char *loud, const char *fmt, ...) va_start(ap, fmt); VSB_vprintf(vsb, fmt, ap); va_end(ap); - AZ(VSB_finish(vsb)); MGT_ComplainVSB(loud, vsb); VSB_destroy(&vsb); } From nils.goroll at uplex.de Fri Feb 14 08:53:14 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 14 Feb 2025 08:53:14 +0000 (UTC) Subject: [master] 668bde61a mgt_jail: Streamline VJ_make_workdir Message-ID: <20250214085314.70941103171@lists.varnish-cache.org> commit 668bde61ab75e4adf81e8a02c46991fe00d08e6f Author: Nils Goroll Date: Fri Feb 14 09:37:35 2025 +0100 mgt_jail: Streamline VJ_make_workdir return error in the vsb always. diff --git a/bin/varnishd/mgt/mgt.h b/bin/varnishd/mgt/mgt.h index f868613fc..2610aaf75 100644 --- a/bin/varnishd/mgt/mgt.h +++ b/bin/varnishd/mgt/mgt.h @@ -161,7 +161,7 @@ struct jail_tech { void VJ_Init(const char *); void VJ_master(enum jail_master_e); void VJ_subproc(enum jail_subproc_e); -int VJ_make_workdir(const char *); +int VJ_make_workdir(const char *, struct vsb *); int VJ_make_subdir(const char *, const char *, struct vsb *); void VJ_fix_fd(int, enum jail_fixfd_e); void VJ_unlink(const char *, int); diff --git a/bin/varnishd/mgt/mgt_jail.c b/bin/varnishd/mgt/mgt_jail.c index e9957a6bb..6d7971052 100644 --- a/bin/varnishd/mgt/mgt_jail.c +++ b/bin/varnishd/mgt/mgt_jail.c @@ -141,39 +141,42 @@ VJ_subproc(enum jail_subproc_e jse) } int -VJ_make_workdir(const char *dname) +VJ_make_workdir(const char *dname, struct vsb *vsb) { - struct vsb *vsb; int i; AN(dname); + AN(vsb); CHECK_OBJ_NOTNULL(vjt, JAIL_TECH_MAGIC); if (vjt->make_workdir != NULL) { - vsb = VSB_new_auto(); - AN(vsb); i = vjt->make_workdir(dname, NULL, vsb); - MGT_ComplainVSB(i ? C_ERR : C_INFO, vsb); - VSB_destroy(&vsb); if (i) return (i); VJ_master(JAIL_MASTER_FILE); } else { VJ_master(JAIL_MASTER_FILE); - if (mkdir(dname, 0755) < 0 && errno != EEXIST) - ARGV_ERR("Cannot create working directory '%s': %s\n", - dname, VAS_errtxt(errno)); + if (mkdir(dname, 0755) < 0 && errno != EEXIST) { + VSB_printf(vsb, + "Cannot create working directory '%s': %s\n", + dname, VAS_errtxt(errno)); + return (1); + } } - if (chdir(dname) < 0) - ARGV_ERR("Cannot change to working directory '%s': %s\n", + if (chdir(dname) < 0) { + VSB_printf(vsb, "Cannot change to working directory '%s': %s\n", dname, VAS_errtxt(errno)); + return (1); + } i = open("_.testfile", O_RDWR|O_CREAT|O_EXCL, 0600); - if (i < 0) - ARGV_ERR("Cannot create test-file in %s (%s)\n" + if (i < 0) { + VSB_printf(vsb, "Cannot create test-file in %s (%s)\n" "Check permissions (or delete old directory)\n", dname, VAS_errtxt(errno)); + return (1); + } #ifdef ST_NOEXEC struct statvfs vfs[1]; @@ -182,8 +185,9 @@ VJ_make_workdir(const char *dname) if (! fstatvfs(i, vfs) && vfs->f_flag & ST_NOEXEC) { closefd(&i); AZ(unlink("_.testfile")); - ARGV_ERR("Working directory %s (-n argument) " + VSB_printf(vsb, "Working directory %s (-n argument) " "cannot reside on a file system mounted noexec\n", dname); + return (1); } #endif diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c index 47f51ae96..ed955acdb 100644 --- a/bin/varnishd/mgt/mgt_main.c +++ b/bin/varnishd/mgt/mgt_main.c @@ -879,7 +879,11 @@ main(int argc, char * const *argv) openlog("varnishd", LOG_PID, LOG_LOCAL0); - if (VJ_make_workdir(workdir)) + vsb = VSB_new_auto(); + AN(vsb); + o = VJ_make_workdir(workdir, vsb); + MGT_ComplainVSB(o ? C_ERR : C_INFO, vsb); + if (o) ARGV_EXIT; VJ_master(JAIL_MASTER_SYSTEM); @@ -897,8 +901,6 @@ main(int argc, char * const *argv) AZ(system("rm -rf vmod_cache vext_cache worker_tmpdir")); VJ_master(JAIL_MASTER_LOW); - vsb = VSB_new_auto(); - AN(vsb); o = VJ_make_subdir("vmod_cache", "VMOD cache", vsb) || VJ_make_subdir("worker_tmpdir", "TMPDIR for the worker process", vsb) || From nils.goroll at uplex.de Fri Feb 14 08:53:14 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 14 Feb 2025 08:53:14 +0000 (UTC) Subject: [master] 21c603de4 mgt_jail_unix: Dedup generation of error messages Message-ID: <20250214085314.3778810316C@lists.varnish-cache.org> commit 21c603de4009ddfe4187c50ccbb323c8f21496bd Author: Nils Goroll Date: Fri Feb 14 09:11:16 2025 +0100 mgt_jail_unix: Dedup generation of error messages Similar case as 9a18ce3bdd6e06ef3b6d654fd36d23e6ebfe3372, but this time for make_subdir diff --git a/bin/varnishd/mgt/mgt_jail.c b/bin/varnishd/mgt/mgt_jail.c index b94e89f0a..e9957a6bb 100644 --- a/bin/varnishd/mgt/mgt_jail.c +++ b/bin/varnishd/mgt/mgt_jail.c @@ -200,6 +200,7 @@ VJ_make_subdir(const char *dname, const char *what, struct vsb *vsb) AN(dname); AN(what); + AN(vsb); CHECK_OBJ_NOTNULL(vjt, JAIL_TECH_MAGIC); if (vjt->make_subdir != NULL) return (vjt->make_subdir(dname, what, vsb)); @@ -207,15 +208,8 @@ VJ_make_subdir(const char *dname, const char *what, struct vsb *vsb) VJ_master(JAIL_MASTER_FILE); if (mkdir(dname, 0755) < 0 && errno != EEXIST) { e = errno; - if (vsb != NULL) { - VSB_printf(vsb, - "Cannot create %s directory '%s': %s\n", - what, dname, VAS_errtxt(e)); - } else { - MGT_Complain(C_ERR, - "Cannot create %s directory '%s': %s", - what, dname, VAS_errtxt(e)); - } + VSB_printf(vsb, "Cannot create %s directory '%s': %s\n", + what, dname, VAS_errtxt(e)); return (1); } VJ_master(JAIL_MASTER_LOW); diff --git a/bin/varnishd/mgt/mgt_jail_unix.c b/bin/varnishd/mgt/mgt_jail_unix.c index 70414c806..01c6f5e02 100644 --- a/bin/varnishd/mgt/mgt_jail_unix.c +++ b/bin/varnishd/mgt/mgt_jail_unix.c @@ -239,19 +239,13 @@ vju_make_subdir(const char *dname, const char *what, struct vsb *vsb) AN(dname); AN(what); + AN(vsb); AZ(seteuid(0)); if (mkdir(dname, 0755) < 0 && errno != EEXIST) { e = errno; - if (vsb != NULL) { - VSB_printf(vsb, - "Cannot create %s directory '%s': %s\n", - what, dname, VAS_errtxt(e)); - } else { - MGT_Complain(C_ERR, - "Cannot create %s directory '%s': %s", - what, dname, VAS_errtxt(e)); - } + VSB_printf(vsb, "Cannot create %s directory '%s': %s\n", + what, dname, VAS_errtxt(e)); return (1); } AZ(chown(dname, vju_uid, vju_gid)); diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c index 2fbcfccad..47f51ae96 100644 --- a/bin/varnishd/mgt/mgt_main.c +++ b/bin/varnishd/mgt/mgt_main.c @@ -640,6 +640,7 @@ main(int argc, char * const *argv) pid_t pid; struct arg_list *alp; int first_arg = 1; + struct vsb *vsb; if (argc == 2 && !strcmp(argv[1], "--optstring")) { printf("%s\n", opt_spec); @@ -896,25 +897,17 @@ main(int argc, char * const *argv) AZ(system("rm -rf vmod_cache vext_cache worker_tmpdir")); VJ_master(JAIL_MASTER_LOW); - if (VJ_make_subdir("vmod_cache", "VMOD cache", NULL)) { - ARGV_ERR( - "Cannot create vmod directory (%s/vmod_cache): %s\n", - workdir, VAS_errtxt(errno)); - } - - if (arg_list_count("E") && - VJ_make_subdir("vext_cache", "VEXT cache", NULL)) { - ARGV_ERR( - "Cannot create vmod directory (%s/vext_cache): %s\n", - workdir, VAS_errtxt(errno)); - } - - if (VJ_make_subdir("worker_tmpdir", - "TMPDIR for the worker process", NULL)) { - ARGV_ERR( - "Cannot create vmod directory (%s/worker_tmpdir): %s\n", - workdir, VAS_errtxt(errno)); - } + vsb = VSB_new_auto(); + AN(vsb); + o = VJ_make_subdir("vmod_cache", "VMOD cache", vsb) || + VJ_make_subdir("worker_tmpdir", + "TMPDIR for the worker process", vsb) || + (arg_list_count("E") && + VJ_make_subdir("vext_cache", "VEXT cache", vsb)); + MGT_ComplainVSB(o ? C_ERR : C_INFO, vsb); + VSB_destroy(&vsb); + if (o) + ARGV_EXIT; o = open("worker_tmpdir", O_RDONLY); VJ_master(JAIL_MASTER_SYSTEM); From nils.goroll at uplex.de Fri Feb 14 09:23:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 14 Feb 2025 09:23:06 +0000 (UTC) Subject: [master] 47d11712e mgt_util: Do not clear the VSB in MGT_ComplainVSB Message-ID: <20250214092306.EF469104CA4@lists.varnish-cache.org> commit 47d11712e819312e6715687923fa1719cc547cba Author: Nils Goroll Date: Fri Feb 14 10:19:46 2025 +0100 mgt_util: Do not clear the VSB in MGT_ComplainVSB Clearing out the VSB was not a good idea, the panic code keeps the vsb around. Sorry to everyone, and I missed to re-run make check... Fixup to 9056935ef6420cc299cc897146c0256385cc3550 diff --git a/bin/varnishd/mgt/mgt_main.c b/bin/varnishd/mgt/mgt_main.c index ed955acdb..933a9d93a 100644 --- a/bin/varnishd/mgt/mgt_main.c +++ b/bin/varnishd/mgt/mgt_main.c @@ -901,6 +901,7 @@ main(int argc, char * const *argv) AZ(system("rm -rf vmod_cache vext_cache worker_tmpdir")); VJ_master(JAIL_MASTER_LOW); + VSB_clear(vsb); o = VJ_make_subdir("vmod_cache", "VMOD cache", vsb) || VJ_make_subdir("worker_tmpdir", "TMPDIR for the worker process", vsb) || diff --git a/bin/varnishd/mgt/mgt_util.c b/bin/varnishd/mgt/mgt_util.c index 7e985d30d..6ccf67cee 100644 --- a/bin/varnishd/mgt/mgt_util.c +++ b/bin/varnishd/mgt/mgt_util.c @@ -148,13 +148,11 @@ MGT_ComplainVSB(const char *loud, struct vsb *vsb) { int sf; + AZ(VSB_finish(vsb)); + if (VSB_len(vsb) == 0 || - (loud == C_CLI && !mgt_param.syslog_cli_traffic)) { - VSB_clear(vsb); + (loud == C_CLI && !mgt_param.syslog_cli_traffic)) return; - } - - AZ(VSB_finish(vsb)); if (loud == C_ERR) sf = LOG_ERR; @@ -174,7 +172,6 @@ MGT_ComplainVSB(const char *loud, struct vsb *vsb) if (!MGT_DO_DEBUG(DBG_VTC_MODE)) syslog(sf, "%s", VSB_data(vsb)); - VSB_clear(vsb); } void From nils.goroll at uplex.de Fri Feb 14 10:15:11 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 14 Feb 2025 10:15:11 +0000 (UTC) Subject: [master] 522fcc577 mgt_jail: Fail if jail init fails Message-ID: <20250214101511.D8CC31070A6@lists.varnish-cache.org> commit 522fcc5775d53bf02f21f6991760f569a0fe73a6 Author: Nils Goroll Date: Fri Feb 14 10:52:07 2025 +0100 mgt_jail: Fail if jail init fails Previously, we would fail at a later point with a confusing error message: Error: linux jail: Could not disable Transparent Hugepage: Invalid argument (22) Error: Running VCC-compiler failed, signal 11 VCL compilation failed (note: EINVAL was injected for testing) diff --git a/bin/varnishd/mgt/mgt_jail.c b/bin/varnishd/mgt/mgt_jail.c index 6d7971052..62e3967f7 100644 --- a/bin/varnishd/mgt/mgt_jail.c +++ b/bin/varnishd/mgt/mgt_jail.c @@ -109,7 +109,8 @@ VJ_Init(const char *j_arg) ARGV_ERR("-j argument is empty\n"); vjt = MGT_Pick(vj_choice, av[1], "jail"); CHECK_OBJ_NOTNULL(vjt, JAIL_TECH_MAGIC); - (void)vjt->init(av + 2); + if (vjt->init(av + 2)) + ARGV_EXIT; VAV_Free(av); } else { /* From nils.goroll at uplex.de Fri Feb 14 10:27:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 14 Feb 2025 10:27:06 +0000 (UTC) Subject: [master] bf84027da jail_linux: Add THP control Message-ID: <20250214102706.978E910795A@lists.varnish-cache.org> commit bf84027da5f54e5b85bae47b38752722ad21cb48 Author: Thibaut Artis Date: Wed Sep 11 18:02:14 2024 +0200 jail_linux: Add THP control Disabling Transparent Hugepage has often been the solution to solve hard-to-diagnose instability issues and despite improvements in this area compared to the RHEL6 era, our recommandation is still to avoid THP to this day. In addition to refreshing the documentation on this topic, we add thp control to the linux jail Committer edit: - Updated to master - Added "try-disable" option as default - Made "enable" and "disable" options fail - Ensured default gets called for -jlinux - Edited documentation - polished diff --git a/bin/varnishd/mgt/mgt_jail_linux.c b/bin/varnishd/mgt/mgt_jail_linux.c index b8f136020..2dd7f89a2 100644 --- a/bin/varnishd/mgt/mgt_jail_linux.c +++ b/bin/varnishd/mgt/mgt_jail_linux.c @@ -44,12 +44,84 @@ #include #include "mgt/mgt.h" +#include "common/heritage.h" + +static int +vjl_set_thp(const char *arg, struct vsb *vsb) +{ + int r, val, must; + + if (!strcmp(arg, "ignore")) + return (0); + must = 1; + if (!strcmp(arg, "enable")) + val = 0; + else if (!strcmp(arg, "disable")) + val = 1; + else if (!strcmp(arg, "try-disable")) { + arg = "disable"; + val = 1; + must = 0; + } + else { + VSB_printf(vsb, "linux jail: unknown value '%s' for argument" + " transparent_hugepage.", arg); + return (1); + } + r = prctl(PR_SET_THP_DISABLE, val, 0, 0, 0); + if (r) { + VSB_printf(vsb, "linux jail: Could not %s " + "Transparent Hugepage: %s (%d)", + arg, VAS_errtxt(errno), errno); + } + return (r && must); +} static int vjl_init(char **args) { + struct vsb *vsb; + char **unix_args; + const char *val; + int seen = 0, ret = 0; + size_t i; + + vsb = VSB_new_auto(); + AN(vsb); + + if (args == NULL) { + /* Autoconfig */ + AZ(vjl_set_thp("try-disable", vsb)); + MGT_ComplainVSB(C_INFO, vsb); + VSB_destroy(&vsb); + return (jail_tech_unix.init(NULL)); + } + + for (i = 0; args[i] != NULL; i++); + unix_args = calloc(i + 1, sizeof *unix_args); + AN(unix_args); + + for (i = 0; *args != NULL && ret == 0; args++) { + val = keyval(*args, "transparent_hugepage="); + if (val == NULL) { + unix_args[i++] = *args; + continue; + } + + ret |= vjl_set_thp(val, vsb); + seen++; + } + + if (seen == 0) + AZ(vjl_set_thp("try-disable", vsb)); + + MGT_ComplainVSB(ret ? C_ERR : C_INFO, vsb); + VSB_destroy(&vsb); - return jail_tech_unix.init(args); + if (ret == 0) + ret = jail_tech_unix.init(unix_args); + free(unix_args); + return (ret); } static void diff --git a/doc/sphinx/installation/platformnotes.rst b/doc/sphinx/installation/platformnotes.rst index 371105ffb..3c645befc 100644 --- a/doc/sphinx/installation/platformnotes.rst +++ b/doc/sphinx/installation/platformnotes.rst @@ -32,6 +32,10 @@ column, no additional action is necessary. Otherwise, consider creating a ``tmpfs`` mountpoint at *workdir*, or configure *workdir* on an existing ``tmpfs``. +The ``tmpfs`` for *workdir* should be mounted with Transparent Hugepage +disabled. Consider mounting the working directory with the ``huge=never`` mount +option if that is not the default. + Note: Very valid reasons exist for *not* following this recommendation, if you know what you are doing. @@ -55,21 +59,24 @@ See :ref:`ref-vsm` for details. .. _platform-thp: -Transparent hugepages on Redhat Enterprise Linux 6 -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Transparent Hugepage on Linux +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +On certain Linux distributions Transparent Hugepage (THP) kernel support is +enabled by default. This is known to cause instabilities of Varnish. -On RHEL6 Transparent Hugepage kernel support is enabled by default. -This is known to cause sporadic crashes of Varnish. +By default, Varnish tries to disable the THP feature, but does not fail if it +can't. The ``linux`` :ref:`ref-varnishd-opt_j` offers to optionally enable, +disable or ignore THP. -It is recommended to disable transparent hugepages on affected -systems. This can be done with -``echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled`` -(runtime) or by adding "transparent_hugepage=never" to the kernel boot -line in the "/etc/grub.conf" file (persistent). +Alternatively, THP can be disabled system-wide. If Varnish is the only +significant service running on this system, this can be done during runtime +with:: -On Debian/Ubuntu systems running 3.2 kernels the default value is "madvise" and -does not need to be changed. + echo never > /sys/kernel/mm/transparent_hugepage/enabled +The setting can be also be persisted in the bootloader configuration by adding +``transparent_hugepage=never`` to the kernel command line. OpenVZ ~~~~~~ diff --git a/doc/sphinx/reference/varnishd.rst b/doc/sphinx/reference/varnishd.rst index a3c445d14..d93721cae 100644 --- a/doc/sphinx/reference/varnishd.rst +++ b/doc/sphinx/reference/varnishd.rst @@ -454,13 +454,26 @@ specific options. Available jails are: -j solaris,worker=basic --j +-j Default on Linux platforms, it extends the UNIX jail with Linux-specific mechanisms: - It warns when *workdir* is not on a ``tmpfs``. - It tries to keep the process dumpable after dropping privileges. + - It adds control over the transparent hugepage (THP) setting. + + `thp_setting` can take these values: + + - ``ignore``: Do nothing + - ``enable``: Enable THP (see Note below) + - ``disable``: Disable THP + - ``try-disable`` (default): Try to disable, ignore failure (but emit a + warning) + + Note: Technically, ``enable`` is "disable the disable", so it does not + necessarily enable THP. The setting names have been chosen to avoid a + confusing double negation. -j From nils.goroll at uplex.de Fri Feb 14 10:58:08 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 14 Feb 2025 10:58:08 +0000 (UTC) Subject: [master] 707dafd26 b39.vtc: relax timing Message-ID: <20250214105808.4B7C3109241@lists.varnish-cache.org> commit 707dafd26bd050a7d0e3e61d409ef9671a5248e3 Author: Nils Goroll Date: Fri Feb 14 11:54:37 2025 +0100 b39.vtc: relax timing IIUC, the logexpect wants to see a bgfetch triggered by the last c1, which looks like it might not happen if the timing is too tight. http://varnish-cache.org/vtest/attachment_bf84027da5f54e5b85bae47b38752722ad21cb48_SunOS_i86pc_5.11_1.05_suncc12.6%3CBR%3E-m64%3CBR%3EIPv6%3CBR%3Euffix_uplex_b00039.txt diff --git a/bin/varnishtest/tests/b00039.vtc b/bin/varnishtest/tests/b00039.vtc index 49afa4550..d5a9afb27 100644 --- a/bin/varnishtest/tests/b00039.vtc +++ b/bin/varnishtest/tests/b00039.vtc @@ -56,7 +56,7 @@ client c1 { expect resp.http.was-304 == "true" } -run -delay 1 +delay 2 client c1 { txreq From nils.goroll at uplex.de Fri Feb 14 11:06:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 14 Feb 2025 11:06:06 +0000 (UTC) Subject: [master] 4e56fd6ba Remove OBE comment Message-ID: <20250214110606.412D310981C@lists.varnish-cache.org> commit 4e56fd6ba296489fc412cbb304cdbb67310df881 Author: Nils Goroll Date: Fri Feb 14 12:02:38 2025 +0100 Remove OBE comment The canonical way to parametrize filters is to use a PRIV_TASK diff --git a/bin/varnishd/cache/cache_vrt_filter.c b/bin/varnishd/cache/cache_vrt_filter.c index c3309d77d..43ef51362 100644 --- a/bin/varnishd/cache/cache_vrt_filter.c +++ b/bin/varnishd/cache/cache_vrt_filter.c @@ -190,7 +190,6 @@ VRT_RemoveVDP(VRT_CTX, const struct vdp *filter) static const struct vfilter vfilter_error[1] = {{0}}; -// XXX: idea(fgs): Allow filters (...) arguments in the list static const struct vfilter * vcl_filter_list_iter(int want_vfp, const struct vfilter_head *h1, const struct vfilter_head *h2, const char **flp) From nils.goroll at uplex.de Fri Feb 14 11:06:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 14 Feb 2025 11:06:06 +0000 (UTC) Subject: [master] b49013193 answer an XXX comment question Message-ID: <20250214110606.5A25B10981F@lists.varnish-cache.org> commit b4901319364a70afb5990d288218f90f701a5c05 Author: Nils Goroll Date: Fri Feb 14 12:05:16 2025 +0100 answer an XXX comment question Basically copied from a -commit email diff --git a/bin/varnishd/cache/cache_vrt_filter.c b/bin/varnishd/cache/cache_vrt_filter.c index 43ef51362..2ebe3183d 100644 --- a/bin/varnishd/cache/cache_vrt_filter.c +++ b/bin/varnishd/cache/cache_vrt_filter.c @@ -450,7 +450,13 @@ req_Empty_Filter(struct req *req) } /*-------------------------------------------------------------------- - * XXX: why ignore needless filters for everything but req? + * control if "set req.filters" is allowed + * + * req.body (conversely to other body object) is one where VCL has control over + * when it gets processed by means of std.cache_req_body(): Filters act on the + * received body when that function is called, so confusion could be caused if + * people expect filters to have an effect when set/changed after the req.body + * is already cached. */ static int From nils.goroll at uplex.de Fri Feb 14 14:20:09 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 14 Feb 2025 14:20:09 +0000 (UTC) Subject: [master] b82366398 miniobj: polish SIZEOF_FLEX_OBJ Message-ID: <20250214142009.1FD4C113BDD@lists.varnish-cache.org> commit b8236639855c7f77a5b6b4d9d995c0f7aa79e544 Author: Nils Goroll Date: Fri Feb 14 15:19:18 2025 +0100 miniobj: polish SIZEOF_FLEX_OBJ diff --git a/include/miniobj.h b/include/miniobj.h index df5018d96..f794bf389 100644 --- a/include/miniobj.h +++ b/include/miniobj.h @@ -38,7 +38,7 @@ } while (0) #define SIZEOF_FLEX_OBJ(to, fld, len) \ - (offsetof(typeof(*to), fld) + sizeof *(to)->fld * len) + (offsetof(typeof(*to), fld) + sizeof *(to)->fld * (len)) #define ALLOC_FLEX_OBJ(to, fld, len, type_magic) \ do { \ From nils.goroll at uplex.de Fri Feb 14 14:30:08 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 14 Feb 2025 14:30:08 +0000 (UTC) Subject: [master] 6d91e6aff jail_linux: Flexelint Message-ID: <20250214143008.218DD1142D4@lists.varnish-cache.org> commit 6d91e6aff0bb09a42bc858c0af6deb31d2275738 Author: Nils Goroll Date: Fri Feb 14 15:29:00 2025 +0100 jail_linux: Flexelint Ref bf84027da5f54e5b85bae47b38752722ad21cb48 diff --git a/bin/varnishd/mgt/mgt_jail_linux.c b/bin/varnishd/mgt/mgt_jail_linux.c index 2dd7f89a2..ae2201d54 100644 --- a/bin/varnishd/mgt/mgt_jail_linux.c +++ b/bin/varnishd/mgt/mgt_jail_linux.c @@ -44,7 +44,6 @@ #include #include "mgt/mgt.h" -#include "common/heritage.h" static int vjl_set_thp(const char *arg, struct vsb *vsb) @@ -86,6 +85,8 @@ vjl_init(char **args) int seen = 0, ret = 0; size_t i; + (void)args; + vsb = VSB_new_auto(); AN(vsb); @@ -97,11 +98,15 @@ vjl_init(char **args) return (jail_tech_unix.init(NULL)); } - for (i = 0; args[i] != NULL; i++); + i = 0; + while (args[i] != NULL) + i++; + unix_args = calloc(i + 1, sizeof *unix_args); AN(unix_args); - for (i = 0; *args != NULL && ret == 0; args++) { + i = 0; + for (; *args != NULL && ret == 0; args++) { val = keyval(*args, "transparent_hugepage="); if (val == NULL) { unix_args[i++] = *args; From nils.goroll at uplex.de Fri Feb 14 20:08:09 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 14 Feb 2025 20:08:09 +0000 (UTC) Subject: [master] c0e29adca body_status: More accurate comment Message-ID: <20250214200809.6359511FB56@lists.varnish-cache.org> commit c0e29adcaa5befe192a16c8937be1b0d083b5c3f Author: Dridi Boukelmoune Date: Fri Feb 14 16:42:59 2025 +0100 body_status: More accurate comment The body_status can also refer to the client request. diff --git a/include/tbl/body_status.h b/include/tbl/body_status.h index 108345c6c..3130d1432 100644 --- a/include/tbl/body_status.h +++ b/include/tbl/body_status.h @@ -27,7 +27,7 @@ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF * SUCH DAMAGE. * - * Various ways to handle the body coming from the backend. + * Various ways to handle the body coming from a peer. */ /*lint -save -e525 -e539 */ From nils.goroll at uplex.de Fri Feb 14 20:08:09 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 14 Feb 2025 20:08:09 +0000 (UTC) Subject: [master] ecddfd74c Reapply "fetch: A beresp body cannot be BS_TAKEN" Message-ID: <20250214200809.7823B11FB58@lists.varnish-cache.org> commit ecddfd74c8a0e6ff310da9945f5bf49ad5523c18 Author: Dridi Boukelmoune Date: Fri Feb 14 16:45:48 2025 +0100 Reapply "fetch: A beresp body cannot be BS_TAKEN" This reverts commit 8c4ae4f0e1c1990a48884b36e2b3e9acc8d162a1. I dispute the claim from the revert commit that bo->htc->body_status refers to the bereq, since this is managed by the V1F (HTTP/1 Fetch) code. What we fetch on the backend side is a beresp. diff --git a/bin/varnishd/cache/cache_fetch.c b/bin/varnishd/cache/cache_fetch.c index 196817512..2f67446b6 100644 --- a/bin/varnishd/cache/cache_fetch.c +++ b/bin/varnishd/cache/cache_fetch.c @@ -141,10 +141,11 @@ Bereq_Rollback(VRT_CTX) bo = ctx->bo; CHECK_OBJ_NOTNULL(bo, BUSYOBJ_MAGIC); - if (bo->htc != NULL && - bo->htc->body_status != BS_NONE && - bo->htc->body_status != BS_TAKEN) - bo->htc->doclose = SC_RESP_CLOSE; + if (bo->htc != NULL) { + assert(bo->htc->body_status != BS_TAKEN); + if (bo->htc->body_status != BS_NONE) + bo->htc->doclose = SC_RESP_CLOSE; + } vbf_cleanup(bo); VCL_TaskLeave(ctx, bo->privs); From phk at FreeBSD.org Mon Feb 17 13:50:08 2025 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Mon, 17 Feb 2025 13:50:08 +0000 (UTC) Subject: [master] 487efb063 Always "set -e" in "shell" directives. Message-ID: <20250217135008.B9F9910D7EE@lists.varnish-cache.org> commit 487efb063c153b4f80242bdc0f3fff9829e563d2 Author: Poul-Henning Kamp Date: Mon Feb 17 13:48:57 2025 +0000 Always "set -e" in "shell" directives. diff --git a/bin/varnishtest/vtc_misc.c b/bin/varnishtest/vtc_misc.c index 36551af28..8167c23ce 100644 --- a/bin/varnishtest/vtc_misc.c +++ b/bin/varnishtest/vtc_misc.c @@ -170,6 +170,7 @@ cmd_shell_engine(struct vtclog *vl, int ok, const char *cmd, errbuf, erroff); } } + VSB_printf(vsb, "set -e ;"); VSB_printf(vsb, "exec 2>&1 ; %s", cmd); AZ(VSB_finish(vsb)); vtc_dump(vl, 4, "shell_cmd", VSB_data(vsb), -1); From nils.goroll at uplex.de Mon Feb 17 16:07:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 17 Feb 2025 16:07:06 +0000 (UTC) Subject: [master] 7fd4dc5cc cache_req: Allocate topreq only if needed Message-ID: <20250217160706.C5CC81128E4@lists.varnish-cache.org> commit 7fd4dc5cc63f7a8fe929faaa2784b36957fa0836 Author: Nils Goroll Date: Wed Feb 12 20:04:29 2025 +0100 cache_req: Allocate topreq only if needed From a discussion here: https://github.com/varnishcache/varnish-cache/pull/4269#issuecomment-2654577140 diff --git a/bin/varnishd/acceptor/cache_acceptor_tcp.c b/bin/varnishd/acceptor/cache_acceptor_tcp.c index 66f15b8e7..af2ceda90 100644 --- a/bin/varnishd/acceptor/cache_acceptor_tcp.c +++ b/bin/varnishd/acceptor/cache_acceptor_tcp.c @@ -406,7 +406,7 @@ vca_tcp_make_session(struct worker *wrk, void *arg) vca_tcp_sockopt_set(wa->acceptlsock, sp); - req = Req_New(sp); + req = Req_New(sp, NULL); CHECK_OBJ_NOTNULL(req, REQ_MAGIC); req->htc->rfd = &sp->fd; diff --git a/bin/varnishd/acceptor/cache_acceptor_uds.c b/bin/varnishd/acceptor/cache_acceptor_uds.c index b71cd1cc4..75462416b 100644 --- a/bin/varnishd/acceptor/cache_acceptor_uds.c +++ b/bin/varnishd/acceptor/cache_acceptor_uds.c @@ -356,7 +356,7 @@ vca_uds_make_session(struct worker *wrk, void *arg) vca_uds_sockopt_set(wa->acceptlsock, sp); - req = Req_New(sp); + req = Req_New(sp, NULL); CHECK_OBJ_NOTNULL(req, REQ_MAGIC); req->htc->rfd = &sp->fd; diff --git a/bin/varnishd/cache/cache_esi_deliver.c b/bin/varnishd/cache/cache_esi_deliver.c index f739b22c3..ccb44d32d 100644 --- a/bin/varnishd/cache/cache_esi_deliver.c +++ b/bin/varnishd/cache/cache_esi_deliver.c @@ -132,7 +132,7 @@ ved_include(struct req *preq, const char *src, const char *host, return; } - req = Req_New(sp); + req = Req_New(sp, preq); AN(req); THR_SetRequest(req); assert(IS_NO_VXID(req->vsl->wid)); @@ -148,9 +148,6 @@ ved_include(struct req *preq, const char *src, const char *host, VSLb_ts_req(req, "Start", W_TIM_real(wrk)); - memset(req->top, 0, sizeof *req->top); - req->top = preq->top; - HTTP_Setup(req->http, req->ws, req->vsl, SLT_ReqMethod); HTTP_Dup(req->http, preq->http0); @@ -182,7 +179,7 @@ ved_include(struct req *preq, const char *src, const char *host, req->req_body_status = BS_NONE; AZ(req->vcl); - AN(req->top); + assert(req->top == preq->top); if (req->top->vcl0) req->vcl = req->top->vcl0; else diff --git a/bin/varnishd/cache/cache_req.c b/bin/varnishd/cache/cache_req.c index 0511ae09b..dd441b33e 100644 --- a/bin/varnishd/cache/cache_req.c +++ b/bin/varnishd/cache/cache_req.c @@ -123,7 +123,7 @@ Req_LogStart(const struct worker *wrk, struct req *req) */ struct req * -Req_New(struct sess *sp) +Req_New(struct sess *sp, struct req *preq) { struct pool *pp; struct req *req; @@ -134,6 +134,7 @@ Req_New(struct sess *sp) CHECK_OBJ_NOTNULL(sp, SESS_MAGIC); pp = sp->pool; CHECK_OBJ_NOTNULL(pp, POOL_MAGIC); + CHECK_OBJ_ORNULL(preq, REQ_MAGIC); req = MPL_Get(pp->mpl_req, &sz); AN(req); @@ -181,10 +182,14 @@ Req_New(struct sess *sp) req->htc->doclose = SC_NULL; p = (void*)PRNDUP(p + sizeof(*req->htc)); - req->top = (void*)p; - INIT_OBJ(req->top, REQTOP_MAGIC); - req->top->topreq = req; - p = (void*)PRNDUP(p + sizeof(*req->top)); + if (UNLIKELY(preq != NULL)) + req->top = preq->top; + else { + req->top = (void*)p; + INIT_OBJ(req->top, REQTOP_MAGIC); + req->top->topreq = req; + p = (void*)PRNDUP(p + sizeof(*req->top)); + } assert(p < e); diff --git a/bin/varnishd/cache/cache_varnishd.h b/bin/varnishd/cache/cache_varnishd.h index 12b3dd416..e3011d8ca 100644 --- a/bin/varnishd/cache/cache_varnishd.h +++ b/bin/varnishd/cache/cache_varnishd.h @@ -411,7 +411,7 @@ void pan_pool(struct vsb *); int VRG_CheckBo(struct busyobj *); /* cache_req.c */ -struct req *Req_New(struct sess *); +struct req *Req_New(struct sess *, struct req *); void Req_Release(struct req *); void Req_Rollback(VRT_CTX); void Req_Cleanup(struct sess *sp, struct worker *wrk, struct req *req); diff --git a/bin/varnishd/http1/cache_http1_fsm.c b/bin/varnishd/http1/cache_http1_fsm.c index 976307863..ac1dc012c 100644 --- a/bin/varnishd/http1/cache_http1_fsm.c +++ b/bin/varnishd/http1/cache_http1_fsm.c @@ -141,7 +141,7 @@ http1_unwait(struct worker *wrk, void *arg) CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); CAST_OBJ_NOTNULL(sp, arg, SESS_MAGIC); WS_Release(sp->ws, 0); - req = Req_New(sp); + req = Req_New(sp, NULL); CHECK_OBJ_NOTNULL(req, REQ_MAGIC); req->htc->rfd = &sp->fd; HTC_RxInit(req->htc, req->ws); diff --git a/bin/varnishd/http2/cache_http2_proto.c b/bin/varnishd/http2/cache_http2_proto.c index c3800ded8..ba21ea5c6 100644 --- a/bin/varnishd/http2/cache_http2_proto.c +++ b/bin/varnishd/http2/cache_http2_proto.c @@ -150,7 +150,7 @@ h2_new_req(struct h2_sess *h2, unsigned stream, struct req *req) ASSERT_RXTHR(h2); if (req == NULL) - req = Req_New(h2->sess); + req = Req_New(h2->sess, NULL); CHECK_OBJ_NOTNULL(req, REQ_MAGIC); r2 = WS_Alloc(req->ws, sizeof *r2); diff --git a/bin/varnishd/http2/cache_http2_session.c b/bin/varnishd/http2/cache_http2_session.c index 075cba183..46b02dc09 100644 --- a/bin/varnishd/http2/cache_http2_session.c +++ b/bin/varnishd/http2/cache_http2_session.c @@ -132,7 +132,7 @@ h2_init_sess(struct sess *sp, assert(*up == 0); if (srq == NULL) - srq = Req_New(sp); + srq = Req_New(sp, NULL); AN(srq); h2 = h2s; AN(h2); From dridi.boukelmoune at gmail.com Tue Feb 18 09:46:16 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 18 Feb 2025 09:46:16 +0000 (UTC) Subject: [master] 1eeb29114 param: Show "all" or "none" for bits parameters Message-ID: <20250218094616.70F541101C8@lists.varnish-cache.org> commit 1eeb29114641d036f8e7e2243c99f5e30eb580cc Author: Dridi Boukelmoune Date: Tue Feb 4 13:24:39 2025 +0100 param: Show "all" or "none" for bits parameters The value for bits parameters currently follows this syntax: all(,-\w+)* none(,[+]\w+)* When either all or none of the flags are set according to the parameter sign, the value will simply be "all" or "none". The opposite value will respectively be: all,-every,-single,-flag... none,+every,+single,+flag... Considering the number of bits per parameters, we can compute upfront whether all bits are raised and simply print "none" or "all": "none" instead of "all,-every,-single,-flag..." "all" instead of "none,+every,+single,+flag..." Co-authored-by: Nils Goroll diff --git a/bin/varnishd/mgt/mgt_param_tweak.c b/bin/varnishd/mgt/mgt_param_tweak.c index 644dbab44..51c3766d5 100644 --- a/bin/varnishd/mgt/mgt_param_tweak.c +++ b/bin/varnishd/mgt/mgt_param_tweak.c @@ -713,7 +713,7 @@ tweak_generic_bits(struct vsb *vsb, const struct parspec *par, const char *arg, uint8_t *p, unsigned l, const char * const *tags, const char *desc, char sign) { - unsigned j; + unsigned j, all; if (arg != NULL && !strcmp(arg, "default")) { /* XXX: deprecated in favor of param.reset */ @@ -724,10 +724,19 @@ tweak_generic_bits(struct vsb *vsb, const struct parspec *par, const char *arg, if (arg != NULL && arg != JSON_FMT) return (bit_tweak(vsb, p, l, arg, tags, desc, sign)); + all = 1; + for (j = 0; all && j < l; j++) { + if (!bit(p, j, BTST)) + all = 0; + } + if (arg == JSON_FMT) VSB_putc(vsb, '"'); - VSB_cat(vsb, sign == '+' ? "none" : "all"); - for (j = 0; j < l; j++) { + if (all) + VSB_cat(vsb, sign == '+' ? "all" : "none"); + else + VSB_cat(vsb, sign == '+' ? "none" : "all"); + for (j = 0; !all && j < l; j++) { if (bit(p, j, BTST)) VSB_printf(vsb, ",%c%s", sign, tags[j]); } From dridi.boukelmoune at gmail.com Tue Feb 18 09:46:16 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Tue, 18 Feb 2025 09:46:16 +0000 (UTC) Subject: [master] d31b1ff11 param: Tweak all bits parameters with all or none Message-ID: <20250218094616.905121101CB@lists.varnish-cache.org> commit d31b1ff114d84c94a40e18c7f72ecc9665e6b3ba Author: Dridi Boukelmoune Date: Tue Feb 4 13:37:34 2025 +0100 param: Tweak all bits parameters with all or none Until now, only one of the two magic keywords was allowed, depending on the sign. In order to allow the counterparts, the rendering side of the value needs to ignore bits without a tag, which can happen for bits after the last bit once rounded up to 8bit groups. In the vsl_mask case, there are also holes in VSL tags that were maintained for file format compatibility. Co-authored-by: Nils Goroll diff --git a/bin/varnishd/mgt/mgt_param_tweak.c b/bin/varnishd/mgt/mgt_param_tweak.c index 51c3766d5..d3e753f1f 100644 --- a/bin/varnishd/mgt/mgt_param_tweak.c +++ b/bin/varnishd/mgt/mgt_param_tweak.c @@ -652,6 +652,13 @@ bit_clear(uint8_t *p, unsigned l) memset(p, 0, ((size_t)l + 7) >> 3); } +static inline void +bit_set(uint8_t *p, unsigned l) +{ + + memset(p, 255, ((size_t)l + 7) >> 3); +} + /*-------------------------------------------------------------------- */ @@ -676,10 +683,18 @@ bit_tweak(struct vsb *vsb, uint8_t *p, unsigned l, const char *arg, bit_clear(p, l); continue; } + if (sign == '+' && !strcmp(s, "all")) { + bit_set(p, l); + continue; + } if (sign == '-' && !strcmp(s, "all")) { bit_clear(p, l); continue; } + if (sign == '-' && !strcmp(s, "none")) { + bit_set(p, l); + continue; + } if (*s != '-' && *s != '+') { VSB_printf(vsb, "Missing '+' or '-' (%s)\n", s); VAV_Free(av); @@ -726,6 +741,8 @@ tweak_generic_bits(struct vsb *vsb, const struct parspec *par, const char *arg, all = 1; for (j = 0; all && j < l; j++) { + if (tags[j] == NULL) + continue; if (!bit(p, j, BTST)) all = 0; } @@ -737,6 +754,8 @@ tweak_generic_bits(struct vsb *vsb, const struct parspec *par, const char *arg, else VSB_cat(vsb, sign == '+' ? "none" : "all"); for (j = 0; !all && j < l; j++) { + if (tags[j] == NULL) + continue; if (bit(p, j, BTST)) VSB_printf(vsb, ",%c%s", sign, tags[j]); } diff --git a/bin/varnishtest/tests/c00054.vtc b/bin/varnishtest/tests/c00054.vtc index 0ee1c2674..b66639779 100644 --- a/bin/varnishtest/tests/c00054.vtc +++ b/bin/varnishtest/tests/c00054.vtc @@ -9,6 +9,8 @@ varnish v1 -cliok "param.set vsl_mask +WorkThread,+TTL,+Hash" varnish v1 -cliok "param.show vsl_mask" varnish v1 -cliexpect {"value": "none"} "param.set -j feature none" +varnish v1 -cliexpect {"value": "all"} "param.set -j feature all" +varnish v1 -cliexpect {"value": "none"} "param.set -j vsl_mask none" varnish v1 -cliexpect {"value": "all"} "param.set -j vsl_mask all" varnish v1 -cliexpect {"value": "all(,-\w+)+"} "param.reset -j vsl_mask" From nils.goroll at uplex.de Tue Feb 18 09:53:10 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 18 Feb 2025 09:53:10 +0000 (UTC) Subject: [master] 8a23f4651 Constify Message-ID: <20250218095310.D1B6D1108F2@lists.varnish-cache.org> commit 8a23f46513acee9e3278d9da3d06c23f8abb1ab1 Author: Nils Goroll Date: Tue Feb 18 10:52:20 2025 +0100 Constify diff --git a/bin/varnishd/cache/cache_req.c b/bin/varnishd/cache/cache_req.c index dd441b33e..2fa6ecd77 100644 --- a/bin/varnishd/cache/cache_req.c +++ b/bin/varnishd/cache/cache_req.c @@ -123,7 +123,7 @@ Req_LogStart(const struct worker *wrk, struct req *req) */ struct req * -Req_New(struct sess *sp, struct req *preq) +Req_New(struct sess *sp, const struct req *preq) { struct pool *pp; struct req *req; diff --git a/bin/varnishd/cache/cache_varnishd.h b/bin/varnishd/cache/cache_varnishd.h index e3011d8ca..c4dc02a48 100644 --- a/bin/varnishd/cache/cache_varnishd.h +++ b/bin/varnishd/cache/cache_varnishd.h @@ -411,7 +411,7 @@ void pan_pool(struct vsb *); int VRG_CheckBo(struct busyobj *); /* cache_req.c */ -struct req *Req_New(struct sess *, struct req *); +struct req *Req_New(struct sess *, const struct req *); void Req_Release(struct req *); void Req_Rollback(VRT_CTX); void Req_Cleanup(struct sess *sp, struct worker *wrk, struct req *req); From walid.boudebouda at varnish-software.com Fri Feb 14 11:14:46 2025 From: walid.boudebouda at varnish-software.com (Walid Boudebouda) Date: Fri, 14 Feb 2025 12:14:46 +0100 Subject: [master] bf84027da jail_linux: Add THP control In-Reply-To: <20250214102706.978E910795A@lists.varnish-cache.org> References: <20250214102706.978E910795A@lists.varnish-cache.org> Message-ID: I find this a bit confusing, is there an actual difference between setting thp_setting to 'enable' vs 'ignore' ? On Fri, Feb 14, 2025 at 11:27?AM Nils Goroll wrote: > > commit bf84027da5f54e5b85bae47b38752722ad21cb48 > Author: Thibaut Artis > Date: Wed Sep 11 18:02:14 2024 +0200 > > jail_linux: Add THP control > > Disabling Transparent Hugepage has often been the solution to solve > hard-to-diagnose instability issues and despite improvements in this > area > compared to the RHEL6 era, our recommandation is still to avoid THP to > this day. > > In addition to refreshing the documentation on this topic, we add thp > control to > the linux jail > > Committer edit: > > - Updated to master > - Added "try-disable" option as default > - Made "enable" and "disable" options fail > - Ensured default gets called for -jlinux > - Edited documentation > - polished > > diff --git a/bin/varnishd/mgt/mgt_jail_linux.c > b/bin/varnishd/mgt/mgt_jail_linux.c > index b8f136020..2dd7f89a2 100644 > --- a/bin/varnishd/mgt/mgt_jail_linux.c > +++ b/bin/varnishd/mgt/mgt_jail_linux.c > @@ -44,12 +44,84 @@ > #include > > #include "mgt/mgt.h" > +#include "common/heritage.h" > + > +static int > +vjl_set_thp(const char *arg, struct vsb *vsb) > +{ > + int r, val, must; > + > + if (!strcmp(arg, "ignore")) > + return (0); > + must = 1; > + if (!strcmp(arg, "enable")) > + val = 0; > + else if (!strcmp(arg, "disable")) > + val = 1; > + else if (!strcmp(arg, "try-disable")) { > + arg = "disable"; > + val = 1; > + must = 0; > + } > + else { > + VSB_printf(vsb, "linux jail: unknown value '%s' for > argument" > + " transparent_hugepage.", arg); > + return (1); > + } > + r = prctl(PR_SET_THP_DISABLE, val, 0, 0, 0); > + if (r) { > + VSB_printf(vsb, "linux jail: Could not %s " > + "Transparent Hugepage: %s (%d)", > + arg, VAS_errtxt(errno), errno); > + } > + return (r && must); > +} > > static int > vjl_init(char **args) > { > + struct vsb *vsb; > + char **unix_args; > + const char *val; > + int seen = 0, ret = 0; > + size_t i; > + > + vsb = VSB_new_auto(); > + AN(vsb); > + > + if (args == NULL) { > + /* Autoconfig */ > + AZ(vjl_set_thp("try-disable", vsb)); > + MGT_ComplainVSB(C_INFO, vsb); > + VSB_destroy(&vsb); > + return (jail_tech_unix.init(NULL)); > + } > + > + for (i = 0; args[i] != NULL; i++); > + unix_args = calloc(i + 1, sizeof *unix_args); > + AN(unix_args); > + > + for (i = 0; *args != NULL && ret == 0; args++) { > + val = keyval(*args, "transparent_hugepage="); > + if (val == NULL) { > + unix_args[i++] = *args; > + continue; > + } > + > + ret |= vjl_set_thp(val, vsb); > + seen++; > + } > + > + if (seen == 0) > + AZ(vjl_set_thp("try-disable", vsb)); > + > + MGT_ComplainVSB(ret ? C_ERR : C_INFO, vsb); > + VSB_destroy(&vsb); > > - return jail_tech_unix.init(args); > + if (ret == 0) > + ret = jail_tech_unix.init(unix_args); > + free(unix_args); > + return (ret); > } > > static void > diff --git a/doc/sphinx/installation/platformnotes.rst > b/doc/sphinx/installation/platformnotes.rst > index 371105ffb..3c645befc 100644 > --- a/doc/sphinx/installation/platformnotes.rst > +++ b/doc/sphinx/installation/platformnotes.rst > @@ -32,6 +32,10 @@ column, no additional action is necessary. > Otherwise, consider creating a ``tmpfs`` mountpoint at *workdir*, or > configure > *workdir* on an existing ``tmpfs``. > > +The ``tmpfs`` for *workdir* should be mounted with Transparent Hugepage > +disabled. Consider mounting the working directory with the ``huge=never`` > mount > +option if that is not the default. > + > Note: Very valid reasons exist for *not* following this recommendation, > if you > know what you are doing. > > @@ -55,21 +59,24 @@ See :ref:`ref-vsm` for details. > > .. _platform-thp: > > -Transparent hugepages on Redhat Enterprise Linux 6 > -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > +Transparent Hugepage on Linux > +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > + > +On certain Linux distributions Transparent Hugepage (THP) kernel support > is > +enabled by default. This is known to cause instabilities of Varnish. > > -On RHEL6 Transparent Hugepage kernel support is enabled by default. > -This is known to cause sporadic crashes of Varnish. > +By default, Varnish tries to disable the THP feature, but does not fail > if it > +can't. The ``linux`` :ref:`ref-varnishd-opt_j` offers to optionally > enable, > +disable or ignore THP. > > -It is recommended to disable transparent hugepages on affected > -systems. This can be done with > -``echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled`` > -(runtime) or by adding "transparent_hugepage=never" to the kernel boot > -line in the "/etc/grub.conf" file (persistent). > +Alternatively, THP can be disabled system-wide. If Varnish is the only > +significant service running on this system, this can be done during > runtime > +with:: > > -On Debian/Ubuntu systems running 3.2 kernels the default value is > "madvise" and > -does not need to be changed. > + echo never > /sys/kernel/mm/transparent_hugepage/enabled > > +The setting can be also be persisted in the bootloader configuration by > adding > +``transparent_hugepage=never`` to the kernel command line. > > OpenVZ > ~~~~~~ > diff --git a/doc/sphinx/reference/varnishd.rst > b/doc/sphinx/reference/varnishd.rst > index a3c445d14..d93721cae 100644 > --- a/doc/sphinx/reference/varnishd.rst > +++ b/doc/sphinx/reference/varnishd.rst > @@ -454,13 +454,26 @@ specific options. Available jails are: > > -j solaris,worker=basic > > --j > +-j > > Default on Linux platforms, it extends the UNIX jail with > Linux-specific mechanisms: > > - It warns when *workdir* is not on a ``tmpfs``. > - It tries to keep the process dumpable after dropping privileges. > + - It adds control over the transparent hugepage (THP) setting. > + > + `thp_setting` can take these values: > + > + - ``ignore``: Do nothing > + - ``enable``: Enable THP (see Note below) > + - ``disable``: Disable THP > + - ``try-disable`` (default): Try to disable, ignore failure (but emit a > + warning) > + > + Note: Technically, ``enable`` is "disable the disable", so it does not > + necessarily enable THP. The setting names have been chosen to avoid a > + confusing double negation. > > -j > > _______________________________________________ > varnish-commit mailing list > varnish-commit at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-commit > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nils.goroll at uplex.de Wed Feb 19 14:42:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 19 Feb 2025 14:42:06 +0000 (UTC) Subject: [master] 53f14d83b varnishncsa: Add hitmiss hitpass indicators to Varnish:handling Message-ID: <20250219144206.BA6171262D7@lists.varnish-cache.org> commit 53f14d83b39cf2dd33f521bb403a92d0423082e3 Author: Nils Goroll Date: Tue Oct 29 14:06:12 2024 +0100 varnishncsa: Add hitmiss hitpass indicators to Varnish:handling diff --git a/bin/varnishncsa/varnishncsa.c b/bin/varnishncsa/varnishncsa.c index 15e5eb098..ee4a1add2 100644 --- a/bin/varnishncsa/varnishncsa.c +++ b/bin/varnishncsa/varnishncsa.c @@ -1128,6 +1128,14 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], case SLT_RespUnset: process_hdr(FMTPOL_RESP, &CTX.watch_resphdr, b, e, 1); break; + case SLT_HitMiss: + CTX.hitmiss = "miss"; + CTX.handling = "hitmiss"; + break; + case SLT_HitPass: + CTX.hitmiss = "miss"; + CTX.handling = "hitpass"; + break; case SLT_VCL_call: if (!strcasecmp(b, "recv")) { CTX.recv_compl = 1; @@ -1136,10 +1144,10 @@ dispatch_f(struct VSL_data *vsl, struct VSL_transaction * const pt[], } else if (!strcasecmp(b, "hit")) { CTX.hitmiss = "hit"; CTX.handling = "hit"; - } else if (!strcasecmp(b, "miss")) { + } else if (!strcasecmp(b, "miss") && strcmp(CTX.handling, "hitmiss")) { CTX.hitmiss = "miss"; CTX.handling = "miss"; - } else if (!strcasecmp(b, "pass")) { + } else if (!strcasecmp(b, "pass") && strcmp(CTX.handling, "hitpass")) { CTX.hitmiss = "miss"; CTX.handling = "pass"; } else if (!strcasecmp(b, "synth")) { diff --git a/bin/varnishtest/tests/u00002.vtc b/bin/varnishtest/tests/u00002.vtc new file mode 100644 index 000000000..d0976c826 --- /dev/null +++ b/bin/varnishtest/tests/u00002.vtc @@ -0,0 +1,104 @@ +varnishtest "varnishncsa handling" + +server s1 -repeat 4 { + rxreq + txresp +} -start + +varnish v1 -vcl+backend { + backend none none; + + sub vcl_backend_fetch { + # XXX would like to do everything in v_b_e, but + # return(pass(x)) is not supported + if (bereq.url == "/hitpass") { + set bereq.backend = s1; + } else { + set bereq.backend = none; + } + } + + sub vcl_backend_response { + set beresp.ttl = 1y; + return (pass(1y)); + } + + sub vcl_backend_error { + set beresp.status = 200; + set beresp.ttl = 1y; + set beresp.body = bereq.url; + if (bereq.url ~ "^/hitmiss") { + set beresp.uncacheable = true; + } + return (deliver); + } + + sub vcl_miss { + if (req.url == "/hitmisspass" && req.is_hitmiss) { + return (pass); + } + } + + sub vcl_recv { + if (req.url == "/pass") { + return (pass); + } + if (req.url == "/synth") { + return (synth(204)); + } + } +} -start + +client c1 { + # prime + txreq -url "/hitmiss" + rxresp + txreq -url "/hitmisspass" + rxresp + expect resp.status == 200 + txreq -url "/hitpass" + rxresp + expect resp.status == 200 + txreq -url "/hit" + rxresp + expect resp.status == 200 + + # re-check + txreq -url "/hitmiss" + rxresp + expect resp.status == 200 + txreq -url "/hitmisspass" + rxresp + expect resp.status == 200 + txreq -url "/hitpass" + rxresp + expect resp.status == 200 + txreq -url "/hit" + rxresp + expect resp.status == 200 + + # others + txreq -url "/pass" + rxresp + expect resp.status == 200 + txreq -url "/synth" + rxresp + expect resp.status == 204 +} -run + +shell { + cat <expect +miss /hitmiss +miss /hitmisspass +miss /hitpass +miss /hit +hitmiss /hitmiss +pass /hitmisspass +hitpass /hitpass +hit /hit +pass /pass +synth /synth +EOF + varnishncsa -d -n ${v1_name} -F "%{Varnish:handling}x %U" >have + diff -u expect have +} diff --git a/doc/changes.rst b/doc/changes.rst index a14b6ef66..a132c45cf 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -41,6 +41,13 @@ Varnish Cache NEXT (2025-03-15) .. PLEASE keep this roughly in commit order as shown by git-log / tig (new to old) +* The ``hitmiss`` and ``hitpass`` handling indicators have been added to the + ``Varnish:handling`` format of ``varnishncsa``. + +* The scope of VCL variables `req.is_hitmiss` and `req.is_hitpass` is now restricted + to `vcl_miss, vcl_deliver, vcl_pass, vcl_synth` and `vcl_pass, vcl_deliver, vcl_synth` + respectively. + * Two fields have been added to the VMOD data registered with varnish-cache: - ``vcs`` for Version Control System is intended as an identifier from the diff --git a/doc/sphinx/reference/varnishncsa.rst b/doc/sphinx/reference/varnishncsa.rst index f734a3614..82bd4a4cf 100644 --- a/doc/sphinx/reference/varnishncsa.rst +++ b/doc/sphinx/reference/varnishncsa.rst @@ -191,9 +191,9 @@ Supported formatters are: misses. In backend mode, this field is blank. Varnish:handling - In client mode, one of the 'hit', 'miss', 'pass', 'pipe' or 'synth' strings - indicating how the request was handled. In backend mode, this field is - blank. + In client mode, one of the 'hit', 'hitmiss', 'hitpass', 'miss', 'pass', + 'pipe' or 'synth' strings indicating how the request was handled. In + backend mode, this field is blank. Varnish:side Backend or client side. One of two values, 'b' or 'c', depending From dridi.boukelmoune at gmail.com Thu Feb 20 07:28:09 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 20 Feb 2025 07:28:09 +0000 (UTC) Subject: [master] 28497a92e vrg: Remove invalid no-content-range assertion Message-ID: <20250220072809.774AB121FAC@lists.varnish-cache.org> commit 28497a92ec990f8d4b9919c979b7c7e42e2044cf Author: Dridi Boukelmoune Date: Thu Feb 20 08:22:39 2025 +0100 vrg: Remove invalid no-content-range assertion An unknown range unit will result in a -1 content-range length, which is not an error but intentional behavior from http_GetContentRange(). Fixes #4276 Closes #4277 diff --git a/bin/varnishd/cache/cache_range.c b/bin/varnishd/cache/cache_range.c index 8bdd8e8df..2a60fa03b 100644 --- a/bin/varnishd/cache/cache_range.c +++ b/bin/varnishd/cache/cache_range.c @@ -273,7 +273,8 @@ VRG_CheckBo(struct busyobj *bo) } if (crlo < 0 && crhi < 0 && crlen < 0) { - AZ(http_GetHdr(bo->beresp, H_Content_Range, NULL)); + VSLb(bo->vsl, SLT_Debug, + "Missing content-range header or unknown range unit"); return (0); } diff --git a/bin/varnishtest/tests/r04276.vtc b/bin/varnishtest/tests/r04276.vtc new file mode 100644 index 000000000..ad598f98a --- /dev/null +++ b/bin/varnishtest/tests/r04276.vtc @@ -0,0 +1,17 @@ +varnishtest "should not crash" + +server s1 { + rxreq + txresp -status 206 -hdr "Content-Range: fish 0-5/68" -body "F" +} -start + +varnish v1 -vcl+backend { + sub vcl_recv { + return (pass); + } +} -start + +client c1 { + txreq + rxresp +} -run From dridi.boukelmoune at gmail.com Thu Feb 20 07:32:05 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 20 Feb 2025 07:32:05 +0000 (UTC) Subject: [master] 0dba0b426 vtc: Forgot to change submitted test case title Message-ID: <20250220073205.469F212232A@lists.varnish-cache.org> commit 0dba0b42663f9fe741ff70b798d86fe2fae9c1d3 Author: Dridi Boukelmoune Date: Thu Feb 20 08:30:39 2025 +0100 vtc: Forgot to change submitted test case title Refs #4276 diff --git a/bin/varnishtest/tests/r04276.vtc b/bin/varnishtest/tests/r04276.vtc index ad598f98a..d034a0115 100644 --- a/bin/varnishtest/tests/r04276.vtc +++ b/bin/varnishtest/tests/r04276.vtc @@ -1,4 +1,4 @@ -varnishtest "should not crash" +varnishtest "Panic on unknown content-range unit" server s1 { rxreq From dridi.boukelmoune at gmail.com Thu Feb 20 10:41:06 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Thu, 20 Feb 2025 10:41:06 +0000 (UTC) Subject: [master] d0279b4ea vtc: Ignore the new content-range debug log Message-ID: <20250220104106.99F2D64BAE@lists.varnish-cache.org> commit d0279b4ea29afdc0ad650b998e4574b3738c0808 Author: Dridi Boukelmoune Date: Thu Feb 20 11:40:07 2025 +0100 vtc: Ignore the new content-range debug log Refs #4276 diff --git a/bin/varnishtest/tests/r01637.vtc b/bin/varnishtest/tests/r01637.vtc index 28abc41c1..0bc0cc76b 100644 --- a/bin/varnishtest/tests/r01637.vtc +++ b/bin/varnishtest/tests/r01637.vtc @@ -24,7 +24,7 @@ varnish v1 -arg "-sdefault,1M" -arg "-p nuke_limit=0 -p gzip_level=0" \ } } -start -logexpect l1 -v v1 -g vxid -q "vxid == 1004" { +logexpect l1 -v v1 -g vxid -q "vxid == 1004" -x Debug { expect 26 1004 VCL_call {^BACKEND_RESPONSE} expect 0 = BerespHeader {^free:} expect 0 = VCL_return {^deliver} diff --git a/bin/varnishtest/tests/r03502.vtc b/bin/varnishtest/tests/r03502.vtc index e2d30faf5..9076e64df 100644 --- a/bin/varnishtest/tests/r03502.vtc +++ b/bin/varnishtest/tests/r03502.vtc @@ -29,7 +29,7 @@ varnish v1 -arg "-sTransient=default,1M -p debug=+syncvsl -p nuke_limit=0" -vcl+ } } -start -logexpect l1 -v v1 -g vxid -q "vxid == 1004" { +logexpect l1 -v v1 -g vxid -q "vxid == 1004" -x Debug { expect 25 1004 VCL_call {^BACKEND_RESPONSE} expect 0 = BerespHeader {^free:} expect 0 = VCL_return {^deliver} From nils.goroll at uplex.de Thu Feb 20 12:40:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 20 Feb 2025 12:40:06 +0000 (UTC) Subject: [master] abfa8c98f varnishtest: ensure varnishd has a workdir always Message-ID: <20250220124007.00413103FAE@lists.varnish-cache.org> commit abfa8c98fe42b9cbfaba3700f8fe1801ac0dc920 Author: Nils Goroll Date: Wed Feb 19 16:58:55 2025 +0100 varnishtest: ensure varnishd has a workdir always We have at least one in out tests where we start varnishd without a -n argument (e.g. m00003.vtc), but the system wide default might actually not even work. Rather than adding -n specifically, we add a global VARNISH_DEFAULT_N environment variable and simplify the other cases where we now no longer need an -n argument. diff --git a/bin/varnishtest/tests/b00045.vtc b/bin/varnishtest/tests/b00045.vtc index ec8c5cb51..183d30e2f 100644 --- a/bin/varnishtest/tests/b00045.vtc +++ b/bin/varnishtest/tests/b00045.vtc @@ -15,11 +15,11 @@ client c1 { delay .2 shell -err -expect {Error: Could not open pid-file} { - varnishd -P /dev/tty -b None -a :0 -n ${tmpdir} + varnishd -P /dev/tty -b None -a :0 } shell -err -expect {Error: Varnishd is already running} { - varnishd -P ${v1_name}/varnishd.pid -b None -a :0 -n ${tmpdir} + varnishd -P ${v1_name}/varnishd.pid -b None -a :0 } shell -err -expect {Error: Varnishd is already running} { diff --git a/bin/varnishtest/tests/c00003.vtc b/bin/varnishtest/tests/c00003.vtc index 1ad256922..6592e2044 100644 --- a/bin/varnishtest/tests/c00003.vtc +++ b/bin/varnishtest/tests/c00003.vtc @@ -126,10 +126,10 @@ shell -err -expect "Invalid sub-arg =foo" { # All bad listen addresses shell -err -expect "Error: Could not get socket" { - varnishd -F -a "${bad_ip}:0" -b '***' -n ${tmpdir} + varnishd -F -a "${bad_ip}:0" -b '***' } # old style address list shell -err -expect "Unknown protocol" { - varnishd -F -a "${listen_addr},${bad_ip}:0" -b '***' -n ${tmpdir} + varnishd -F -a "${listen_addr},${bad_ip}:0" -b '***' } diff --git a/bin/varnishtest/tests/p00000.vtc b/bin/varnishtest/tests/p00000.vtc index d31177f73..96aadbdfa 100644 --- a/bin/varnishtest/tests/p00000.vtc +++ b/bin/varnishtest/tests/p00000.vtc @@ -3,7 +3,7 @@ varnishtest "Test Basic persistence" feature persistent_storage process p1 -dump { - varnishd -spersistent -b ${localhost} -n ${tmpdir} -a :0 -d 2>&1 + varnishd -spersistent -b ${localhost} -a :0 -d 2>&1 } -start -expect-exit 1 process p1 -expect-text 0 0 {to launch} diff --git a/bin/varnishtest/tests/r02321.vtc b/bin/varnishtest/tests/r02321.vtc index 1596456ff..efde8643c 100644 --- a/bin/varnishtest/tests/r02321.vtc +++ b/bin/varnishtest/tests/r02321.vtc @@ -2,18 +2,15 @@ varnishtest "Storage name collisions" # intentional collision shell -err -expect "Error: (-s main=default,100m) 'main' is already defined" { - exec varnishd -a :0 -f '' -n ${tmpdir} \ - -s main=default,10m -s main=default,100m + exec varnishd -a :0 -f '' -s main=default,10m -s main=default,100m } # pseudo-accidental collision shell -err -expect "Error: (-s s0=default,100m) 's0' is already defined" { - exec varnishd -a :0 -f '' -n ${tmpdir} \ - -s default,10m -s s0=default,100m + exec varnishd -a :0 -f '' -s default,10m -s s0=default,100m } # Transient collision shell -err -expect "Error: (-s Transient=default,100m) 'Transient' is already defined" { - exec varnishd -a :0 -f '' -n ${tmpdir} \ - -s Transient=default,10m -s Transient=default,100m + exec varnishd -a :0 -f '' -s Transient=default,10m -s Transient=default,100m } diff --git a/bin/varnishtest/tests/r02325.vtc b/bin/varnishtest/tests/r02325.vtc index fab645a53..ef00ed4e5 100644 --- a/bin/varnishtest/tests/r02325.vtc +++ b/bin/varnishtest/tests/r02325.vtc @@ -1,8 +1,8 @@ varnishtest "validate storage identifiers" shell -err -expect {Error: invalid -s name "///"=[...]} { - varnishd -a :0 -n ${tmpdir} -F -f '' -s ///=default + varnishd -a :0 -F -f '' -s ///=default } shell -err -expect {Error: Empty named -s argument "foo="} { - varnishd -a :0 -n ${tmpdir} -F -f '' -s foo= + varnishd -a :0 -F -f '' -s foo= } diff --git a/bin/varnishtest/tests/s00003.vtc b/bin/varnishtest/tests/s00003.vtc index b15de012c..5c79e70db 100644 --- a/bin/varnishtest/tests/s00003.vtc +++ b/bin/varnishtest/tests/s00003.vtc @@ -46,14 +46,14 @@ varnish v1 -vsl_catchup varnish v1 -cliok "ban obj.http.date ~ ." process p1 { - varnishd -sTransient=file,${tmpdir}/foo,xxx -b${localhost} -a:0 -n ${tmpdir} 2>&1 + varnishd -sTransient=file,${tmpdir}/foo,xxx -b${localhost} -a:0 2>&1 } -expect-exit 0x2 -dump -start -expect-text 0 0 "Invalid number" -wait -screen_dump process p1 { - varnishd -sTransient=file,${tmpdir}/foo,10M,xxx -b${localhost} -a:0 -n ${tmpdir} 2>&1 + varnishd -sTransient=file,${tmpdir}/foo,10M,xxx -b${localhost} -a:0 2>&1 } -expect-exit 0x2 -dump -start -expect-text 0 0 "granularity" -wait -screen_dump process p1 { - varnishd -sTransient=file,${tmpdir}/foo,10m,,foo -b${localhost} -a:0 -n ${tmpdir} 2>&1 + varnishd -sTransient=file,${tmpdir}/foo,10m,,foo -b${localhost} -a:0 2>&1 } -expect-exit 0x2 -dump -start -expect-text 0 0 "invalid advice" -wait diff --git a/bin/varnishtest/vtc_main.c b/bin/varnishtest/vtc_main.c index b3e58f6b8..9880c7c5e 100644 --- a/bin/varnishtest/vtc_main.c +++ b/bin/varnishtest/vtc_main.c @@ -399,6 +399,7 @@ start_test(void) int p[2], retval; struct vtc_job *jp; char tmpdir[PATH_MAX]; + char default_n[PATH_MAX]; ALLOC_OBJ(jp, JOB_MAGIC); AN(jp); @@ -409,6 +410,10 @@ start_test(void) (unsigned)random()); AZ(mkdir(tmpdir, 0755)); + bprintf(default_n, "%s/default_n", tmpdir); + + AZ(setenv("VARNISH_DEFAULT_N", default_n, 1)); + tp = VTAILQ_FIRST(&tst_head); CHECK_OBJ_NOTNULL(tp, TST_MAGIC); AN(tp->ntodo); From nils.goroll at uplex.de Fri Feb 21 07:57:14 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 21 Feb 2025 07:57:14 +0000 (UTC) Subject: [master] 5ec6f29a0 [doc] add some notes about storage selection Message-ID: <20250221075714.65385106018@lists.varnish-cache.org> commit 5ec6f29a06e9ebe8311f39c9954fd1c7e9afeb1f Author: Guillaume Quintard Date: Thu Feb 20 13:00:57 2025 -0800 [doc] add some notes about storage selection diff --git a/doc/sphinx/reference/varnishd.rst b/doc/sphinx/reference/varnishd.rst index d93721cae..569edc541 100644 --- a/doc/sphinx/reference/varnishd.rst +++ b/doc/sphinx/reference/varnishd.rst @@ -350,6 +350,10 @@ The following hash algorithms are available: Storage Backend --------------- +Multiple storage backend (when the cached and in-flight objects are held) can +be defined, and VCL can pilot which store is used by setting the +`beresp.storage` variable (see `man vcl-var` for more information). + The argument format to define storage backends is: -s <[name]=kind[,options]> diff --git a/doc/sphinx/reference/vcl_var.rst b/doc/sphinx/reference/vcl_var.rst index 08d1e7966..b9ef20e0d 100644 --- a/doc/sphinx/reference/vcl_var.rst +++ b/doc/sphinx/reference/vcl_var.rst @@ -1328,7 +1328,10 @@ beresp.storage Writable from: vcl_backend_response, vcl_backend_error - The storage backend to use to save this object. + The storage backend to use to save this object. If + none is set, Varnish will pick a storage backend in a + round-robin fashion, or the `Transient` backend if + the object is short-lived. beresp.storage_hint ``VCL <= 4.0`` diff --git a/doc/sphinx/users-guide/storage-backends.rst b/doc/sphinx/users-guide/storage-backends.rst index 0030461d9..b1bbdb31c 100644 --- a/doc/sphinx/users-guide/storage-backends.rst +++ b/doc/sphinx/users-guide/storage-backends.rst @@ -28,7 +28,15 @@ Storage backends are also called stevedores. .. _vmods: https://www.varnish-cache.org/vmods Besides the built-in storage backends, separately distributed extensions exist, -which can be found on the `vmods`_ page by searching for "stevedore". + +Storage Selection +~~~~~~~~~~~~~~~~~ + +By default, Varnish will store short-lived and passed objects in a storage +called `Transient`, described below. + +For other objects, it will rotate between all the non-transient storages, +unless the VCL variable `beresp.storage` is explicitly set. default ~~~~~~~ From nils.goroll at uplex.de Fri Feb 21 08:00:07 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 21 Feb 2025 08:00:07 +0000 (UTC) Subject: [master] 37a8bebf0 doc: Minor polish of previous commit Message-ID: <20250221080007.CD46C106368@lists.varnish-cache.org> commit 37a8bebf0f7fb0a68af9669af57ff094b8114ee0 Author: Nils Goroll Date: Fri Feb 21 08:59:22 2025 +0100 doc: Minor polish of previous commit diff --git a/doc/sphinx/reference/varnishd.rst b/doc/sphinx/reference/varnishd.rst index 569edc541..8afe6dde0 100644 --- a/doc/sphinx/reference/varnishd.rst +++ b/doc/sphinx/reference/varnishd.rst @@ -350,7 +350,7 @@ The following hash algorithms are available: Storage Backend --------------- -Multiple storage backend (when the cached and in-flight objects are held) can +Multiple storage backends (where the cached and in-flight objects are held) can be defined, and VCL can pilot which store is used by setting the `beresp.storage` variable (see `man vcl-var` for more information). From nils.goroll at uplex.de Fri Feb 21 12:19:11 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 21 Feb 2025 12:19:11 +0000 (UTC) Subject: [master] 3e2fd5ac1 varnishtest: refactor client_proxy() in preparation of next commit(s) Message-ID: <20250221121911.57D45113FE8@lists.varnish-cache.org> commit 3e2fd5ac1a4a30d8717fbadc920f8d35131c1d82 Author: Nils Goroll Date: Fri Feb 21 11:30:23 2025 +0100 varnishtest: refactor client_proxy() in preparation of next commit(s) diff --git a/bin/varnishtest/vtc_client.c b/bin/varnishtest/vtc_client.c index 6142f78eb..1e52f6cdb 100644 --- a/bin/varnishtest/vtc_client.c +++ b/bin/varnishtest/vtc_client.c @@ -74,26 +74,31 @@ static VTAILQ_HEAD(, client) clients = VTAILQ_HEAD_INITIALIZER(clients); */ static void -client_proxy(struct vtclog *vl, int fd, int version, const char *spec) +client_proxy(struct vtclog *vl, int fd, int version, const char *speca) { const struct suckaddr *sac, *sas; - char *p, *p2; + char *spec, *save, *p; - p = strdup(spec); - AN(p); - p2 = strchr(p, ' '); - AN(p2); - *p2++ = '\0'; + spec = strdup(speca); + AN(spec); + save = NULL; + p = strtok_r(spec, " ", &save); + AN(p); sac = VSS_ResolveOne(NULL, p, NULL, 0, SOCK_STREAM, AI_PASSIVE); if (sac == NULL) vtc_fatal(vl, "Could not resolve client address"); - sas = VSS_ResolveOne(NULL, p2, NULL, 0, SOCK_STREAM, AI_PASSIVE); + + p = strtok_r(NULL, " ", &save); + AN(p); + sas = VSS_ResolveOne(NULL, p, NULL, 0, SOCK_STREAM, AI_PASSIVE); if (sas == NULL) - vtc_fatal(vl, "Could not resolve server address"); + vtc_fatal(vl, "Could not resolve client address"); + + free(spec); + if (vtc_send_proxy(fd, version, sac, sas)) vtc_fatal(vl, "Write failed: %s", strerror(errno)); - free(p); VSA_free(&sac); VSA_free(&sas); } From nils.goroll at uplex.de Fri Feb 21 12:19:11 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 21 Feb 2025 12:19:11 +0000 (UTC) Subject: [master] f2b398f0c varnishtest: Add tlv capabilities to client -proxy Message-ID: <20250221121911.7A039113FEB@lists.varnish-cache.org> commit f2b398f0ce0a3575bb3d3546f49cb40792ca8854 Author: Nils Goroll Date: Fri Feb 21 13:12:07 2025 +0100 varnishtest: Add tlv capabilities to client -proxy diff --git a/bin/varnishtest/vtc.h b/bin/varnishtest/vtc.h index e7513092e..2e5d4161a 100644 --- a/bin/varnishtest/vtc.h +++ b/bin/varnishtest/vtc.h @@ -124,8 +124,9 @@ void vtc_dump(struct vtclog *vl, int lvl, const char *pfx, const char *str, int len); void vtc_hexdump(struct vtclog *, int , const char *, const void *, unsigned); +void vtc_proxy_tlv(struct vtclog *vl, struct vsb *vsb, const char *kva); int vtc_send_proxy(int fd, int version, const struct suckaddr *sac, - const struct suckaddr *sas); + const struct suckaddr *sas, struct vsb *tlb); int exec_file(const char *fn, const char *script, const char *tmpdir, char *logbuf, unsigned loglen); diff --git a/bin/varnishtest/vtc_client.c b/bin/varnishtest/vtc_client.c index 1e52f6cdb..1281cc5a9 100644 --- a/bin/varnishtest/vtc_client.c +++ b/bin/varnishtest/vtc_client.c @@ -78,6 +78,7 @@ client_proxy(struct vtclog *vl, int fd, int version, const char *speca) { const struct suckaddr *sac, *sas; char *spec, *save, *p; + struct vsb *tlv; spec = strdup(speca); AN(spec); @@ -95,9 +96,14 @@ client_proxy(struct vtclog *vl, int fd, int version, const char *speca) if (sas == NULL) vtc_fatal(vl, "Could not resolve client address"); + tlv = VSB_new_auto(); + while ((p = strtok_r(NULL, " ", &save)) != NULL) + vtc_proxy_tlv(vl, tlv, p); + free(spec); - if (vtc_send_proxy(fd, version, sac, sas)) + AZ(VSB_finish(tlv)); + if (vtc_send_proxy(fd, version, sac, sas, tlv)) vtc_fatal(vl, "Write failed: %s", strerror(errno)); VSA_free(&sac); VSA_free(&sas); diff --git a/bin/varnishtest/vtc_http.c b/bin/varnishtest/vtc_http.c index 5ad7d9723..0f57aada6 100644 --- a/bin/varnishtest/vtc_http.c +++ b/bin/varnishtest/vtc_http.c @@ -128,7 +128,14 @@ extern const struct cmds http_cmds[]; * * \-proxy2 STRING (client only) * Use the PROXY protocol version 2 for this connection. STRING - * is of the form "CLIENTIP:PORT SERVERIP:PORT". + * is of the form "CLIENTIP:PORT SERVERIP:PORT [TLV [TLV ... ]]". + * + * TLV is in the form name=val + * + * name: 0xID or alpn, authority, crc32c, noop, unique_id, netns + * val: 0x... or string + * + * ssl is currently not implemented (can be sent as hex) * * SECTION: client-server.spec Specification * diff --git a/bin/varnishtest/vtc_proxy.c b/bin/varnishtest/vtc_proxy.c index ab7167ee8..32983df8c 100644 --- a/bin/varnishtest/vtc_proxy.c +++ b/bin/varnishtest/vtc_proxy.c @@ -30,6 +30,8 @@ #include "config.h" +#include +#include #include #include @@ -48,6 +50,101 @@ static const char vpx2_sig[] = { 'Q', 'U', 'I', 'T', '\n', }; +#define PP2_TYPE_ALPN 0x01 +#define PP2_TYPE_AUTHORITY 0x02 +#define PP2_TYPE_CRC32C 0x03 +#define PP2_TYPE_NOOP 0x04 +#define PP2_TYPE_UNIQUE_ID 0x05 +#define PP2_TYPE_SSL 0x20 +#define PP2_SUBTYPE_SSL_VERSION 0x21 +#define PP2_SUBTYPE_SSL_CN 0x22 +#define PP2_SUBTYPE_SSL_CIPHER 0x23 +#define PP2_SUBTYPE_SSL_SIG_ALG 0x24 +#define PP2_SUBTYPE_SSL_KEY_ALG 0x25 +#define PP2_SUBTYPE_SSL_MAX 0x25 +#define PP2_TYPE_NETNS 0x30 + +struct pp2_type { + const char * name; + uint8_t type; +}; + +/* sorted ! */ +static const struct pp2_type pp2_types[] = { + {"alpn", PP2_TYPE_ALPN}, + {"authority", PP2_TYPE_AUTHORITY}, + {"crc32c", PP2_TYPE_CRC32C}, + {"netns", PP2_TYPE_NETNS}, + {"noop", PP2_TYPE_NOOP}, + {"unique_id", PP2_TYPE_UNIQUE_ID} +}; + +static int +pp2cmp(const void *va, const void *vb) +{ + const struct pp2_type *a = va; + const struct pp2_type *b = vb; + return (strcmp(a->name, b->name)); +} + +void +vtc_proxy_tlv(struct vtclog *vl, struct vsb *vsb, const char *kva) +{ + struct pp2_type *pp2, needle; + char *save = NULL, *kv; + struct vsb *vsb2; + const char *p; + uint16_t le; + ssize_t sz; + + kv = strdup(kva); + AN(kv); + save = NULL; + + p = strtok_r(kv, "=", &save); + AN(p); + if (p[0] == '0' && p[1] == 'x') { + p += 2; + vsb2 = vtc_hex_to_bin(vl, p); + AN(vsb2); + if (VSB_len(vsb2) != 1) + vtc_fatal(vl, "tlv hex type has wrong length"); + VSB_bcat(vsb, VSB_data(vsb2), 1); + VSB_destroy(&vsb2); + } + else { + needle = (typeof(needle)){p, 0}; + pp2 = bsearch(&needle, pp2_types, sizeof pp2_types / sizeof pp2_types[0], + sizeof pp2_types[0], pp2cmp); + if (pp2 == NULL) + vtc_fatal(vl, "tlv type %s not found", p); + VSB_putc(vsb, pp2->type); + } + + p = strtok_r(NULL, "", &save); + if (p == NULL) + vtc_fatal(vl, "tlv value missing"); + if (p[0] == '0' && p[1] == 'x') + vsb2 = vtc_hex_to_bin(vl, p + 2); + else { + vsb2 = VSB_new_auto(); + AN(vsb2); + VSB_cat(vsb2, p); + VSB_finish(vsb2); + } + AN(vsb2); + + sz = VSB_len(vsb2); + assert(sz >= 0); + assert(sz <= UINT16_MAX); + + vbe16enc(&le, (uint16_t)sz); + assert(sizeof(le) == 2); + VSB_bcat(vsb, &le, 2); + VSB_bcat(vsb, VSB_data(vsb2), sz); + VSB_destroy(&vsb2); +} + static void vpx_enc_addr(struct vsb *vsb, int proto, const struct suckaddr *s) { @@ -79,13 +176,14 @@ vpx_enc_port(struct vsb *vsb, const struct suckaddr *s) int vtc_send_proxy(int fd, int version, const struct suckaddr *sac, - const struct suckaddr *sas) + const struct suckaddr *sas, struct vsb *tlv) { struct vsb *vsb; char hc[VTCP_ADDRBUFSIZE]; char pc[VTCP_PORTBUFSIZE]; char hs[VTCP_ADDRBUFSIZE]; char ps[VTCP_PORTBUFSIZE]; + uint16_t le, l; int i; int proto; @@ -99,6 +197,11 @@ vtc_send_proxy(int fd, int version, const struct suckaddr *sac, proto = VSA_Get_Proto(sas); assert(proto == PF_INET6 || proto == PF_INET); + if (tlv == NULL) + l = 0; + else + l = VSB_len(tlv); + if (version == 1) { VSB_bcat(vsb, vpx1_sig, sizeof(vpx1_sig)); if (proto == PF_INET6) @@ -113,13 +216,16 @@ vtc_send_proxy(int fd, int version, const struct suckaddr *sac, VSB_putc(vsb, 0x21); if (proto == PF_INET6) { VSB_putc(vsb, 0x21); - VSB_putc(vsb, 0x00); - VSB_putc(vsb, 0x24); + l += 0x24; } else if (proto == PF_INET) { VSB_putc(vsb, 0x11); - VSB_putc(vsb, 0x00); - VSB_putc(vsb, 0x0c); - } + l += 0x0c; + } else + WRONG("proto"); + + vbe16enc(&le, l); + assert(sizeof(le) == 2); + VSB_bcat(vsb, &le, 2); vpx_enc_addr(vsb, proto, sac); vpx_enc_addr(vsb, proto, sas); vpx_enc_port(vsb, sac); @@ -130,5 +236,12 @@ vtc_send_proxy(int fd, int version, const struct suckaddr *sac, AZ(VSB_finish(vsb)); i = VSB_tofile(vsb, fd); VSB_destroy(&vsb); + if (i != 0 && tlv != NULL) + VSB_destroy(&tlv); + if (i != 0 || tlv == NULL) + return (i); + + i = VSB_tofile(tlv, fd); + VSB_destroy(&tlv); return (i); } From nils.goroll at uplex.de Fri Feb 21 13:48:09 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 21 Feb 2025 13:48:09 +0000 (UTC) Subject: [master] a5f2da706 vtc: cp bin/varnishtest/tests/o0{0,2}001.vtc Message-ID: <20250221134809.E2D1E1174D0@lists.varnish-cache.org> commit a5f2da706a7992b9bbc491d8a6d1c154ac736b6b Author: Nils Goroll Date: Fri Feb 21 14:44:40 2025 +0100 vtc: cp bin/varnishtest/tests/o0{0,2}001.vtc copy as separate commit to improve the git history (show edits) diff --git a/bin/varnishtest/tests/o02001.vtc b/bin/varnishtest/tests/o02001.vtc new file mode 100644 index 000000000..97209254c --- /dev/null +++ b/bin/varnishtest/tests/o02001.vtc @@ -0,0 +1,215 @@ +varnishtest "PROXY v2 test" + +server s1 { + rxreq + expect req.url == "/1" + expect req.http.x-forwarded-for == "${localhost}" + txresp + + rxreq + expect req.url == "/2" + expect req.http.x-forwarded-for == "${localhost}" + txresp + + rxreq + expect req.url == "/3" + expect req.http.x-forwarded-for == "${localhost}" + txresp + + rxreq + expect req.url == "/4" + expect req.http.x-forwarded-for == "${localhost}" + txresp + + rxreq + expect req.url == "/5" + expect req.http.x-forwarded-for == "${localhost}" + txresp + + rxreq + expect req.url == "/6" + expect req.http.x-forwarded-for == "1.2.3.4" + txresp + + rxreq + expect req.url == "/7" + expect req.http.x-forwarded-for == "102:304:506::d0e:f10" + txresp +} -start + +varnish v1 -proto "PROXY" -vcl+backend { + import std; + + acl fwd_client { + "1.2.3.4"; + "102:304:506::d0e:f10"; + } + + acl fwd_server { + "5.6.7.8"; + "8182:8384:8586::8d8e:8f80"; + } + + sub vcl_deliver { + set resp.http.li = local.ip; + set resp.http.lp = std.port(local.ip); + set resp.http.ri = remote.ip; + set resp.http.rp = std.port(remote.ip); + set resp.http.ci = client.ip; + set resp.http.cp = std.port(client.ip); + set resp.http.si = server.ip; + set resp.http.sp = std.port(server.ip); + set resp.http.fc = (client.ip ~ fwd_client); + set resp.http.fs = (server.ip ~ fwd_server); + set resp.http.xport = req.transport; + } +} -start + +logexpect l1 -v v1 -g raw { + expect * 1000 Proxy "2 local local local local" + expect * 1003 ProxyGarbage "PROXY2: bad command \\(2\\)" + expect * 1004 ProxyGarbage "PROXY2: Ignoring UNSPEC\\|UNSPEC addresses" + expect * 1007 ProxyGarbage "PROXY2: Ignoring unsupported protocol \\(0x99\\)" + expect * 1010 ProxyGarbage "PROXY2: Ignoring short IPv4 addresses \\(11\\)" + expect * 1013 ProxyGarbage "PROXY2: Ignoring short IPv6 addresses \\(35\\)" + expect * 1016 Proxy "2 1.2.3.4 2314 5.6.7.8 2828" + expect * 1019 Proxy "2 102:304:506::d0e:f10 2314 8182:8384:8586::8d8e:8f80 2828" + expect * 1022 Begin "^sess 0 PROXY" + expect * 1022 SessClose "^RX_OVERFLOW" +} -start + +client c1 { + # LOCAL command + sendhex "0d 0a 0d 0a 00 0d 0a 51 55 49 54 0a" + sendhex "20 00 00 00" + txreq -url /1 + rxresp + expect resp.status == 200 + expect resp.http.si == "${v1_addr}" + expect resp.http.sp == "${v1_port}" + expect resp.http.ci == "${localhost}" + # Transport is HTTP/1, even though proxy is used + expect resp.http.xport == HTTP/1 +} -run + +delay .1 + +client c1 { + # unknown command + sendhex "0d 0a 0d 0a 00 0d 0a 51 55 49 54 0a" + sendhex "22 00 00 00" + timeout 8 + expect_close +} -run + +delay .1 + +client c1 { + # UNSPEC proto + sendhex "0d 0a 0d 0a 00 0d 0a 51 55 49 54 0a" + sendhex "21 00 00 00" + txreq -url /2 + rxresp + expect resp.status == 200 + expect resp.http.si == "${v1_addr}" + expect resp.http.sp == "${v1_port}" + expect resp.http.ci == "${localhost}" +} -run + +delay .1 + +client c1 { + # unknown proto + sendhex "0d 0a 0d 0a 00 0d 0a 51 55 49 54 0a" + sendhex "21 99 00 00" + txreq -url /3 + rxresp + expect resp.status == 200 + expect resp.http.si == "${v1_addr}" + expect resp.http.sp == "${v1_port}" + expect resp.http.ci == "${localhost}" +} -run + +delay .1 + +client c1 { + # short IPv4 + sendhex "0d 0a 0d 0a 00 0d 0a 51 55 49 54 0a" + sendhex "21 11 00 0b" + sendhex "01 02 03 04 05 06 07 08 09 0a 0b" + txreq -url /4 + rxresp + expect resp.status == 200 + expect resp.http.si == "${v1_addr}" + expect resp.http.sp == "${v1_port}" + expect resp.http.ci == "${localhost}" +} -run + +delay .1 + +client c1 { + # short IPv6 + sendhex "0d 0a 0d 0a 00 0d 0a 51 55 49 54 0a" + sendhex "21 21 00 23" + sendhex "00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f" + sendhex "00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f" + sendhex "01 02 03" + txreq -url /5 + rxresp + expect resp.status == 200 + expect resp.http.fs == false + expect resp.http.fc == false + expect resp.http.si == "${v1_addr}" + expect resp.http.sp == "${v1_port}" + expect resp.http.ci == "${localhost}" +} -run +delay .1 + +# good IPv4 +client c1 -proxy2 "1.2.3.4:2314 5.6.7.8:2828" { + txreq -url /6 + rxresp + expect resp.status == 200 + expect resp.http.fs == true + expect resp.http.fc == true + expect resp.http.ci == "1.2.3.4" + expect resp.http.cp == "2314" + expect resp.http.si == "5.6.7.8" + expect resp.http.sp == "2828" + expect resp.http.li == "${v1_addr}" + expect resp.http.lp == "${v1_port}" + expect resp.http.ri != "1.2.3.4" +} -run + +delay .1 + +# good IPv6 +client c1 \ + -proxy2 "[102:304:506::d0e:f10]:2314 [8182:8384:8586::8d8e:8f80]:2828" { + txreq -url /7 + rxresp + expect resp.status == 200 + expect resp.http.fs == true + expect resp.http.fc == true + expect resp.http.ci == "102:304:506::d0e:f10" + expect resp.http.cp == "2314" + expect resp.http.si == "8182:8384:8586::8d8e:8f80" + expect resp.http.sp == "2828" + expect resp.http.li == "${v1_addr}" + expect resp.http.lp == "${v1_port}" + expect resp.http.ri != "102:304:506::d0e:f10" +} -run + +delay .1 + +client c2 { + # max length with garbage + sendhex "0d 0a 0d 0a 00 0d 0a 51 55 49 54 0a" + # announce 1025 bytes > 1024 implicit limit + sendhex "20 00 04 01" + expect_close +} -run + +delay .1 + +logexpect l1 -wait From nils.goroll at uplex.de Fri Feb 21 13:48:10 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 21 Feb 2025 13:48:10 +0000 (UTC) Subject: [master] 2e3208f52 vtc: Add a variant of o00001.vtc to test proxy + http/2 Message-ID: <20250221134810.23B7B1174D3@lists.varnish-cache.org> commit 2e3208f52950604ea3f8b1522962db64b6933aaf Author: Nils Goroll Date: Fri Feb 21 14:36:38 2025 +0100 vtc: Add a variant of o00001.vtc to test proxy + http/2 diff --git a/bin/varnishtest/tests/o02001.vtc b/bin/varnishtest/tests/o02001.vtc index 97209254c..c3cdddced 100644 --- a/bin/varnishtest/tests/o02001.vtc +++ b/bin/varnishtest/tests/o02001.vtc @@ -1,4 +1,4 @@ -varnishtest "PROXY v2 test" +varnishtest "PROXY v2 test on http2" server s1 { rxreq @@ -37,7 +37,10 @@ server s1 { txresp } -start -varnish v1 -proto "PROXY" -vcl+backend { +varnish v1 -proto "PROXY" \ + -arg "-p feature=+http2" \ + -vcl+backend { + import proxy; import std; acl fwd_client { @@ -61,6 +64,7 @@ varnish v1 -proto "PROXY" -vcl+backend { set resp.http.sp = std.port(server.ip); set resp.http.fc = (client.ip ~ fwd_client); set resp.http.fs = (server.ip ~ fwd_server); + set resp.http.alpn = proxy.alpn(); set resp.http.xport = req.transport; } } -start @@ -82,14 +86,15 @@ client c1 { # LOCAL command sendhex "0d 0a 0d 0a 00 0d 0a 51 55 49 54 0a" sendhex "20 00 00 00" - txreq -url /1 - rxresp - expect resp.status == 200 - expect resp.http.si == "${v1_addr}" - expect resp.http.sp == "${v1_port}" - expect resp.http.ci == "${localhost}" - # Transport is HTTP/1, even though proxy is used - expect resp.http.xport == HTTP/1 + stream 1 { + txreq -url /1 + rxresp + expect resp.status == 200 + expect resp.http.si == "${v1_addr}" + expect resp.http.sp == "${v1_port}" + expect resp.http.ci == "${localhost}" + expect resp.http.xport == "HTTP/2" + } -run } -run delay .1 @@ -108,12 +113,15 @@ client c1 { # UNSPEC proto sendhex "0d 0a 0d 0a 00 0d 0a 51 55 49 54 0a" sendhex "21 00 00 00" - txreq -url /2 - rxresp - expect resp.status == 200 - expect resp.http.si == "${v1_addr}" - expect resp.http.sp == "${v1_port}" - expect resp.http.ci == "${localhost}" + stream 1 { + txreq -url /2 + rxresp + expect resp.status == 200 + expect resp.http.si == "${v1_addr}" + expect resp.http.sp == "${v1_port}" + expect resp.http.ci == "${localhost}" + expect resp.http.xport == "HTTP/2" + } -run } -run delay .1 @@ -122,12 +130,15 @@ client c1 { # unknown proto sendhex "0d 0a 0d 0a 00 0d 0a 51 55 49 54 0a" sendhex "21 99 00 00" - txreq -url /3 - rxresp - expect resp.status == 200 - expect resp.http.si == "${v1_addr}" - expect resp.http.sp == "${v1_port}" - expect resp.http.ci == "${localhost}" + stream 1 { + txreq -url /3 + rxresp + expect resp.status == 200 + expect resp.http.si == "${v1_addr}" + expect resp.http.sp == "${v1_port}" + expect resp.http.ci == "${localhost}" + expect resp.http.xport == "HTTP/2" + } -run } -run delay .1 @@ -137,12 +148,15 @@ client c1 { sendhex "0d 0a 0d 0a 00 0d 0a 51 55 49 54 0a" sendhex "21 11 00 0b" sendhex "01 02 03 04 05 06 07 08 09 0a 0b" - txreq -url /4 - rxresp - expect resp.status == 200 - expect resp.http.si == "${v1_addr}" - expect resp.http.sp == "${v1_port}" - expect resp.http.ci == "${localhost}" + stream 1 { + txreq -url /4 + rxresp + expect resp.status == 200 + expect resp.http.si == "${v1_addr}" + expect resp.http.sp == "${v1_port}" + expect resp.http.ci == "${localhost}" + expect resp.http.xport == "HTTP/2" + } -run } -run delay .1 @@ -154,50 +168,60 @@ client c1 { sendhex "00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f" sendhex "00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f" sendhex "01 02 03" - txreq -url /5 - rxresp - expect resp.status == 200 - expect resp.http.fs == false - expect resp.http.fc == false - expect resp.http.si == "${v1_addr}" - expect resp.http.sp == "${v1_port}" - expect resp.http.ci == "${localhost}" + stream 1 { + txreq -url /5 + rxresp + expect resp.status == 200 + expect resp.http.fs == false + expect resp.http.fc == false + expect resp.http.si == "${v1_addr}" + expect resp.http.sp == "${v1_port}" + expect resp.http.ci == "${localhost}" + } -run } -run delay .1 # good IPv4 -client c1 -proxy2 "1.2.3.4:2314 5.6.7.8:2828" { - txreq -url /6 - rxresp - expect resp.status == 200 - expect resp.http.fs == true - expect resp.http.fc == true - expect resp.http.ci == "1.2.3.4" - expect resp.http.cp == "2314" - expect resp.http.si == "5.6.7.8" - expect resp.http.sp == "2828" - expect resp.http.li == "${v1_addr}" - expect resp.http.lp == "${v1_port}" - expect resp.http.ri != "1.2.3.4" +client c1 -proxy2 "1.2.3.4:2314 5.6.7.8:2828 alpn=h2" { + stream 1 { + txreq -url /6 + rxresp + expect resp.status == 200 + expect resp.http.fs == true + expect resp.http.fc == true + expect resp.http.ci == "1.2.3.4" + expect resp.http.cp == "2314" + expect resp.http.si == "5.6.7.8" + expect resp.http.sp == "2828" + expect resp.http.li == "${v1_addr}" + expect resp.http.lp == "${v1_port}" + expect resp.http.ri != "1.2.3.4" + expect resp.http.alpn == "h2" + expect resp.http.xport == "HTTP/2" + } -run } -run delay .1 # good IPv6 client c1 \ - -proxy2 "[102:304:506::d0e:f10]:2314 [8182:8384:8586::8d8e:8f80]:2828" { - txreq -url /7 - rxresp - expect resp.status == 200 - expect resp.http.fs == true - expect resp.http.fc == true - expect resp.http.ci == "102:304:506::d0e:f10" - expect resp.http.cp == "2314" - expect resp.http.si == "8182:8384:8586::8d8e:8f80" - expect resp.http.sp == "2828" - expect resp.http.li == "${v1_addr}" - expect resp.http.lp == "${v1_port}" - expect resp.http.ri != "102:304:506::d0e:f10" + -proxy2 "[102:304:506::d0e:f10]:2314 [8182:8384:8586::8d8e:8f80]:2828 alpn=h2" { + stream 1 { + txreq -url /7 + rxresp + expect resp.status == 200 + expect resp.http.fs == true + expect resp.http.fc == true + expect resp.http.ci == "102:304:506::d0e:f10" + expect resp.http.cp == "2314" + expect resp.http.si == "8182:8384:8586::8d8e:8f80" + expect resp.http.sp == "2828" + expect resp.http.li == "${v1_addr}" + expect resp.http.lp == "${v1_port}" + expect resp.http.ri != "102:304:506::d0e:f10" + expect resp.http.alpn == "h2" + expect resp.http.xport == "HTTP/2" + } -run } -run delay .1 From nils.goroll at uplex.de Fri Feb 21 13:53:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 21 Feb 2025 13:53:06 +0000 (UTC) Subject: [master] 1efd4f1e4 varnishtest: Flexelint polish Message-ID: <20250221135306.5569C117A74@lists.varnish-cache.org> commit 1efd4f1e43b41893ee1687ee90aac3635cf74f38 Author: Nils Goroll Date: Fri Feb 21 14:52:20 2025 +0100 varnishtest: Flexelint polish Ref f2b398f0ce0a3575bb3d3546f49cb40792ca8854 diff --git a/bin/varnishtest/vtc_proxy.c b/bin/varnishtest/vtc_proxy.c index 32983df8c..351d6c59f 100644 --- a/bin/varnishtest/vtc_proxy.c +++ b/bin/varnishtest/vtc_proxy.c @@ -50,6 +50,7 @@ static const char vpx2_sig[] = { 'Q', 'U', 'I', 'T', '\n', }; +//lint -emacro(750, PP2_) #define PP2_TYPE_ALPN 0x01 #define PP2_TYPE_AUTHORITY 0x02 #define PP2_TYPE_CRC32C 0x03 @@ -99,7 +100,6 @@ vtc_proxy_tlv(struct vtclog *vl, struct vsb *vsb, const char *kva) kv = strdup(kva); AN(kv); - save = NULL; p = strtok_r(kv, "=", &save); AN(p); @@ -130,7 +130,7 @@ vtc_proxy_tlv(struct vtclog *vl, struct vsb *vsb, const char *kva) vsb2 = VSB_new_auto(); AN(vsb2); VSB_cat(vsb2, p); - VSB_finish(vsb2); + AZ(VSB_finish(vsb2)); } AN(vsb2); From dridi.boukelmoune at gmail.com Fri Feb 21 21:27:09 2025 From: dridi.boukelmoune at gmail.com (Dridi Boukelmoune) Date: Fri, 21 Feb 2025 21:27:09 +0000 (UTC) Subject: [master] 5a26bb816 varnishstat: Key binding to reset running averages Message-ID: <20250221212709.A50BE1268D2@lists.varnish-cache.org> commit 5a26bb8160c7099593f4a212b065fd01637f9629 Author: Guillaume Quintard Date: Thu Feb 13 18:38:26 2025 -0800 varnishstat: Key binding to reset running averages I've been running benchmark tests these last few days, and I've been wishing to be able to reset the average calculations without having to restart varnishstat, so here we go. diff --git a/bin/varnishstat/varnishstat_bindings.h b/bin/varnishstat/varnishstat_bindings.h index 565b5cc49..c4a79ad56 100644 --- a/bin/varnishstat/varnishstat_bindings.h +++ b/bin/varnishstat/varnishstat_bindings.h @@ -101,6 +101,9 @@ BINDING(QUIT, "\tQuit.") BINDING_KEY(BINDING_CTRL('t'), "CTRL+T",) BINDING(SAMPLE, "\tSample now.") +BINDING_KEY('0', "0",) +BINDING(RESET_AVERAGES, "\tReset averages.") + BINDING_KEY('+', "+",) BINDING(ACCEL, "\tIncrease refresh interval.") diff --git a/bin/varnishstat/varnishstat_curses.c b/bin/varnishstat/varnishstat_curses.c index 6250c8512..d1a1cb219 100644 --- a/bin/varnishstat/varnishstat_curses.c +++ b/bin/varnishstat/varnishstat_curses.c @@ -129,6 +129,7 @@ static int current = 0; static int rebuild = 0; static int redraw = 0; static int sample = 0; +static int reset_averages = 0; static int scale = 1; static double t_sample = 0.; static double interval = 1.; @@ -262,6 +263,12 @@ sample_points(void) pt->t_last = pt->t_cur; pt->t_cur = VTIM_mono(); + if (reset_averages) { + pt->chg = 0; + pt->ma_10.n = 0; + pt->ma_100.n = 0; + pt->ma_1000.n = 0; + } if (pt->t_last) pt->chg = ((int64_t)pt->cur - (int64_t)pt->last) / (pt->t_cur - pt->t_last); @@ -305,6 +312,11 @@ sample_hitrate(void) ratio = hr / (hr + mr); else ratio = 0; + if (reset_averages) { + hitrate.hr_10.n = 0; + hitrate.hr_100.n = 0; + hitrate.hr_1000.n = 0; + } update_ma(&hitrate.hr_10, ratio); update_ma(&hitrate.hr_100, ratio); update_ma(&hitrate.hr_1000, ratio); @@ -318,6 +330,7 @@ sample_data(void) redraw = 1; sample_points(); sample_hitrate(); + reset_averages = 0; } static void @@ -1006,6 +1019,9 @@ handle_points_keypress(struct vsc *vsc, enum kb_e kb) case KB_SAMPLE: sample = 1; return; + case KB_RESET_AVERAGES: + reset_averages = 1; + return; case KB_QUIT: case KB_SIG_INT: case KB_SIG_TSTP: @@ -1055,6 +1071,7 @@ handle_help_keypress(enum kb_e kb) case KB_VERBOSE: case KB_QUIET: case KB_SAMPLE: + case KB_RESET_AVERAGES: break; case KB_QUIT: case KB_SIG_INT: diff --git a/bin/varnishtest/tests/u00021.vtc b/bin/varnishtest/tests/u00021.vtc new file mode 100644 index 000000000..0bc3ba8a9 --- /dev/null +++ b/bin/varnishtest/tests/u00021.vtc @@ -0,0 +1,23 @@ +varnishtest "varnishstat coverage" + + +server s1 { + rxreq + txresp +} -start + +varnish v1 -vcl+backend {} -start +process p1 -dump {varnishstat -n ${v1_name}} -start +delay 2 + +client c1 -repeat 11 { + txreq + rxresp +} -run + +delay 2 +process p1 -screen_dump +process p1 -write 0 +delay 1 +process p1 -screen_dump +process p1 -expect-text 0 0 "avg(n): 0.0000 0.0000 0.0000" From nils.goroll at uplex.de Sat Feb 22 08:53:08 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Sat, 22 Feb 2025 08:53:08 +0000 (UTC) Subject: [master] 3cd659685 varnishtest: Flexelint silencing was missing a wildcard Message-ID: <20250222085308.7B73B116C47@lists.varnish-cache.org> commit 3cd65968591aee010e7c63df10a64382b7c5d9c7 Author: Nils Goroll Date: Sat Feb 22 09:51:46 2025 +0100 varnishtest: Flexelint silencing was missing a wildcard diff --git a/bin/varnishtest/vtc_proxy.c b/bin/varnishtest/vtc_proxy.c index 351d6c59f..14029671c 100644 --- a/bin/varnishtest/vtc_proxy.c +++ b/bin/varnishtest/vtc_proxy.c @@ -50,7 +50,7 @@ static const char vpx2_sig[] = { 'Q', 'U', 'I', 'T', '\n', }; -//lint -emacro(750, PP2_) +//lint -emacro(750, PP2_*) #define PP2_TYPE_ALPN 0x01 #define PP2_TYPE_AUTHORITY 0x02 #define PP2_TYPE_CRC32C 0x03 From nils.goroll at uplex.de Sun Feb 23 12:36:10 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Sun, 23 Feb 2025 12:36:10 +0000 (UTC) Subject: [master] 04e460016 varnishtest: Flexelint silencing was still wrong Message-ID: <20250223123610.440491014E7@lists.varnish-cache.org> commit 04e4600161b837e8e95d3e8a61d115864c0b086b Author: Nils Goroll Date: Sun Feb 23 13:06:45 2025 +0100 varnishtest: Flexelint silencing was still wrong (sorry) diff --git a/bin/varnishtest/vtc_proxy.c b/bin/varnishtest/vtc_proxy.c index 14029671c..4734ff77b 100644 --- a/bin/varnishtest/vtc_proxy.c +++ b/bin/varnishtest/vtc_proxy.c @@ -50,7 +50,7 @@ static const char vpx2_sig[] = { 'Q', 'U', 'I', 'T', '\n', }; -//lint -emacro(750, PP2_*) +//lint -esym(750, PP2_*) #define PP2_TYPE_ALPN 0x01 #define PP2_TYPE_AUTHORITY 0x02 #define PP2_TYPE_CRC32C 0x03 From nils.goroll at uplex.de Sun Feb 23 12:36:10 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Sun, 23 Feb 2025 12:36:10 +0000 (UTC) Subject: [master] d0fe49799 varnishtest: enable the client to send arbitrary settings Message-ID: <20250223123610.661351014EC@lists.varnish-cache.org> commit d0fe497993c7c451eb88987cafe6947055742895 Author: Nils Goroll Date: Sun Feb 23 13:34:18 2025 +0100 varnishtest: enable the client to send arbitrary settings diff --git a/bin/varnishtest/vtc_http2.c b/bin/varnishtest/vtc_http2.c index 1caeeb5a3..f89bb187e 100644 --- a/bin/varnishtest/vtc_http2.c +++ b/bin/varnishtest/vtc_http2.c @@ -1898,6 +1898,9 @@ cmd_txprio(CMD_ARGS) * \-hdrsize INT * maximum size of the header list authorized * + * \-0xHH[HH] INT + * tx arbitraty settings with tag xx + * * \-ack * set the ack bit */ @@ -1906,7 +1909,8 @@ cmd_txsettings(CMD_ARGS) { struct stream *s, *s2; struct http *hp; - char *p; + char *p, *e; + unsigned long u; uint32_t val = 0; struct frame f; //TODO dynamic alloc @@ -1952,6 +1956,15 @@ cmd_txsettings(CMD_ARGS) PUT_KV(av, vl, framesize, val, 0x5); else if (!strcmp(*av, "-hdrsize")) PUT_KV(av, vl, hdrsize, val, 0x6); + else if (!strncmp(*av, "-0x", 3)) { + p = *av + 3; + errno = 0; + u = strtoul(p, &e, 16); + if (*p == '\0' || *e != '\0' || u > 0xffff || errno != 0) + vtc_fatal(vl, "Invalid settings tag %s", p); + assert(u <= 0xffff); + PUT_KV(av, vl, hdrtbl, val, (uint16_t)u); + } else if (!strcmp(*av, "-ack")) f.flags |= 1; else From nils.goroll at uplex.de Sun Feb 23 17:17:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Sun, 23 Feb 2025 17:17:05 +0000 (UTC) Subject: [master] bd0076f16 cache_http2_hpack: Improve usefulnes of field value check Message-ID: <20250223171705.D0FD8115AE4@lists.varnish-cache.org> commit bd0076f16ee318e1ee0d39f424c14193ece9fd3b Author: Nils Goroll Date: Sun Feb 23 18:08:13 2025 +0100 cache_http2_hpack: Improve usefulnes of field value check diff --git a/bin/varnishd/http2/cache_http2_hpack.c b/bin/varnishd/http2/cache_http2_hpack.c index 694e7fc83..a90e6fde2 100644 --- a/bin/varnishd/http2/cache_http2_hpack.c +++ b/bin/varnishd/http2/cache_http2_hpack.c @@ -106,7 +106,8 @@ h2h_checkhdr(struct vsl_log *vsl, txt nm, txt val) case FLD_VALUE_FIRST: if (vct_issp(*p)) { VSLb(vsl, SLT_BogoHeader, - "Illegal field value start %.*s", l, nm.b); + "Illegal field value 0x%02x start %.*s", + *p, l, nm.b); return (H2SE_PROTOCOL_ERROR); } state = FLD_VALUE; @@ -114,7 +115,8 @@ h2h_checkhdr(struct vsl_log *vsl, txt nm, txt val) case FLD_VALUE: if (!vct_ishdrval(*p)) { VSLb(vsl, SLT_BogoHeader, - "Illegal field value %.*s", l, nm.b); + "Illegal field value 0x%02x %.*s", + *p, l, nm.b); return (H2SE_PROTOCOL_ERROR); } break; @@ -124,7 +126,8 @@ h2h_checkhdr(struct vsl_log *vsl, txt nm, txt val) } if (state == FLD_VALUE && vct_issp(val.e[-1])) { VSLb(vsl, SLT_BogoHeader, - "Illegal field value (end) %.*s", l, nm.b); + "Illegal field value 0x%02x (end) at %.*s", + val.e[-1], l, nm.b); return (H2SE_PROTOCOL_ERROR); } return (0); From nils.goroll at uplex.de Mon Feb 24 11:01:09 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 24 Feb 2025 11:01:09 +0000 (UTC) Subject: [master] d4b5767be http2: Unique names for errors Message-ID: <20250224110109.B59BF1146FE@lists.varnish-cache.org> commit d4b5767be34945c06fdd58861ac138216758e63d Author: Nils Goroll Date: Mon Feb 24 11:59:26 2025 +0100 http2: Unique names for errors Connection and stream errors are not the same thing, so our error reporting should also use unambiguous names. diff --git a/bin/varnishd/http2/cache_http2_proto.c b/bin/varnishd/http2/cache_http2_proto.c index ba21ea5c6..ca8092ee5 100644 --- a/bin/varnishd/http2/cache_http2_proto.c +++ b/bin/varnishd/http2/cache_http2_proto.c @@ -48,9 +48,9 @@ #define H2_CUSTOM_ERRORS #define H2EC1(U,v,g,r,d) \ - const struct h2_error_s H2CE_##U[1] = {{#U,d,v,0,1,g,r}}; + const struct h2_error_s H2CE_##U[1] = {{"H2CE_" #U,d,v,0,1,g,r}}; #define H2EC2(U,v,g,r,d) \ - const struct h2_error_s H2SE_##U[1] = {{#U,d,v,1,0,g,r}}; + const struct h2_error_s H2SE_##U[1] = {{"H2SE_" #U,d,v,1,0,g,r}}; #define H2EC3(U,v,g,r,d) H2EC1(U,v,g,r,d) H2EC2(U,v,g,r,d) #define H2_ERROR(NAME, val, sc, goaway, reason, desc) \ H2EC##sc(NAME, val, goaway, reason, desc) From nils.goroll at uplex.de Mon Feb 24 13:35:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 24 Feb 2025 13:35:05 +0000 (UTC) Subject: [master] 440319deb http2: clear an in-progress request when deleting it Message-ID: <20250224133506.0DEE311AE41@lists.varnish-cache.org> commit 440319deb628ab65a53ebace729aa2579eef9710 Author: Nils Goroll Date: Mon Feb 24 12:56:42 2025 +0100 http2: clear an in-progress request when deleting it otherwise, for the modified vtc, a continuation is still expected when another stream is opened: 1031 SessError c H2: expected continuation but received HEADERS on stream 3 1031 Debug c H2 CLEANUP H2CE_PROTOCOL_ERROR Partly addresses #4283 diff --git a/bin/varnishd/http2/cache_http2_proto.c b/bin/varnishd/http2/cache_http2_proto.c index ca8092ee5..e857203f6 100644 --- a/bin/varnishd/http2/cache_http2_proto.c +++ b/bin/varnishd/http2/cache_http2_proto.c @@ -192,6 +192,8 @@ h2_del_req(struct worker *wrk, struct h2_req *r2) --h2->refcnt; /* XXX: PRIORITY reshuffle */ VTAILQ_REMOVE(&h2->streams, r2, list); + if (r2->req == h2->new_req) + h2->new_req = NULL; Lck_Unlock(&sp->mtx); assert(!WS_IsReserved(r2->req->ws)); diff --git a/bin/varnishtest/tests/t02023.vtc b/bin/varnishtest/tests/t02023.vtc index 13c4cb445..4ca8306c4 100644 --- a/bin/varnishtest/tests/t02023.vtc +++ b/bin/varnishtest/tests/t02023.vtc @@ -130,4 +130,8 @@ client c1 { rxrst expect rst.err == PROTOCOL_ERROR } -run + stream 3 { + txreq + rxresp + } -run } -run From nils.goroll at uplex.de Tue Feb 25 16:04:09 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 25 Feb 2025 16:04:09 +0000 (UTC) Subject: [master] 145b38a8e whitespace Message-ID: <20250225160409.49DCA101D8B@lists.varnish-cache.org> commit 145b38a8e4b8487617ae0ac04e8a2bce59a3dd98 Author: Nils Goroll Date: Tue Feb 25 17:02:45 2025 +0100 whitespace diff --git a/bin/varnishd/cache/cache_ban.h b/bin/varnishd/cache/cache_ban.h index 6512e20f6..d3339eef0 100644 --- a/bin/varnishd/cache/cache_ban.h +++ b/bin/varnishd/cache/cache_ban.h @@ -93,8 +93,8 @@ #define BANS_ARG_REQHTTP 0x19 #define BANS_ARG_OBJHTTP 0x1a #define BANS_ARG_OBJSTATUS 0x1b -#define BANS_ARG_OBJTTL 0x1c -#define BANS_ARG_OBJAGE 0x1d +#define BANS_ARG_OBJTTL 0x1c +#define BANS_ARG_OBJAGE 0x1d #define BANS_ARG_OBJGRACE 0x1e #define BANS_ARG_OBJKEEP 0x1f #define BANS_ARG_LIM (BANS_ARG_OBJKEEP + 1) From nils.goroll at uplex.de Wed Feb 26 09:47:11 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 26 Feb 2025 09:47:11 +0000 (UTC) Subject: [master] afdff6e1b cache_ban: refactor: split out ban_add_spec Message-ID: <20250226094711.6B2FF64870@lists.varnish-cache.org> commit afdff6e1b4a359101251741333002da2fcf270e5 Author: Nils Goroll Date: Wed Feb 26 10:28:05 2025 +0100 cache_ban: refactor: split out ban_add_spec Just moving code, no change diff --git a/bin/varnishd/cache/cache_ban_build.c b/bin/varnishd/cache/cache_ban_build.c index bab9ac79a..302898da5 100644 --- a/bin/varnishd/cache/cache_ban_build.c +++ b/bin/varnishd/cache/cache_ban_build.c @@ -225,6 +225,20 @@ ban_parse_oper(const char *p) /*-------------------------------------------------------------------- * Add a (and'ed) test-condition to a ban */ +static const char * +ban_add_spec(struct ban_proto *bp, const struct pvar *pv, int op, const char *a3) +{ + + assert(! BANS_HAS_ARG2_DOUBLE(pv->tag)); + + ban_add_lump(bp, a3, strlen(a3) + 1); + VSB_putc(bp->vsb, op); + + if (! BANS_HAS_ARG2_SPEC(op)) + return (NULL); + + return (ban_parse_regexp(bp, a3)); +} const char * BAN_AddTest(struct ban_proto *bp, @@ -274,17 +288,8 @@ BAN_AddTest(struct ban_proto *bp, "expected conditional (%s) got \"%s\"", arg_operhelp[BAN_ARGIDX(pv->tag)], a2)); - if ((pv->flag & BANS_FLAG_DURATION) == 0) { - assert(! BANS_HAS_ARG2_DOUBLE(pv->tag)); - - ban_add_lump(bp, a3, strlen(a3) + 1); - VSB_putc(bp->vsb, op); - - if (! BANS_HAS_ARG2_SPEC(op)) - return (NULL); - - return (ban_parse_regexp(bp, a3)); - } + if ((pv->flag & BANS_FLAG_DURATION) == 0) + return (ban_add_spec(bp, pv, op, a3)); assert(pv->flag & BANS_FLAG_DURATION); assert(BANS_HAS_ARG2_DOUBLE(pv->tag)); From nils.goroll at uplex.de Wed Feb 26 09:47:11 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 26 Feb 2025 09:47:11 +0000 (UTC) Subject: [master] 6e3e2f39c cache_ban: refactor: split out ban_add_duration / ban_add_number Message-ID: <20250226094711.8FC1D64874@lists.varnish-cache.org> commit 6e3e2f39c182ef62a22a17153218471abcae0850 Author: Nils Goroll Date: Wed Feb 26 10:34:59 2025 +0100 cache_ban: refactor: split out ban_add_duration / ban_add_number Just moving code, no change Now guess which change is going to be proposed... diff --git a/bin/varnishd/cache/cache_ban_build.c b/bin/varnishd/cache/cache_ban_build.c index 302898da5..c9af918f7 100644 --- a/bin/varnishd/cache/cache_ban_build.c +++ b/bin/varnishd/cache/cache_ban_build.c @@ -240,14 +240,42 @@ ban_add_spec(struct ban_proto *bp, const struct pvar *pv, int op, const char *a3 return (ban_parse_regexp(bp, a3)); } +static const char * +ban_add_double(struct ban_proto *bp, const struct pvar *pv, int op, double darg) +{ + uint64_t dtmp; + uint8_t denc[sizeof darg]; + + assert(BANS_HAS_ARG2_DOUBLE(pv->tag)); + assert(sizeof darg == sizeof dtmp); + assert(sizeof dtmp == sizeof denc); + memcpy(&dtmp, &darg, sizeof dtmp); + vbe64enc(denc, dtmp); + + ban_add_lump(bp, denc, sizeof denc); + VSB_putc(bp->vsb, op); + return (NULL); +} + +static const char * +ban_add_duration(struct ban_proto *bp, const struct pvar *pv, int op, const char *a3) +{ + double darg; + + assert(pv->flag & BANS_FLAG_DURATION); + darg = VNUM_duration(a3); + if (isnan(darg)) { + return (ban_error(bp, + "expected duration [ms|s|m|h|d|w|y] got \"%s\"", a3)); + } + return (ban_add_double(bp, pv, op, darg)); +} + const char * BAN_AddTest(struct ban_proto *bp, const char *a1, const char *a2, const char *a3) { const struct pvar *pv; - double darg; - uint64_t dtmp; - uint8_t denc[sizeof darg]; int op; CHECK_OBJ_NOTNULL(bp, BAN_PROTO_MAGIC); @@ -290,23 +318,8 @@ BAN_AddTest(struct ban_proto *bp, if ((pv->flag & BANS_FLAG_DURATION) == 0) return (ban_add_spec(bp, pv, op, a3)); - - assert(pv->flag & BANS_FLAG_DURATION); - assert(BANS_HAS_ARG2_DOUBLE(pv->tag)); - darg = VNUM_duration(a3); - if (isnan(darg)) { - return (ban_error(bp, - "expected duration [ms|s|m|h|d|w|y] got \"%s\"", a3)); - } - - assert(sizeof darg == sizeof dtmp); - assert(sizeof dtmp == sizeof denc); - memcpy(&dtmp, &darg, sizeof dtmp); - vbe64enc(denc, dtmp); - - ban_add_lump(bp, denc, sizeof denc); - VSB_putc(bp->vsb, op); - return (NULL); + else + return (ban_add_duration(bp, pv, op, a3)); } /*-------------------------------------------------------------------- From nils.goroll at uplex.de Wed Feb 26 10:28:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 26 Feb 2025 10:28:05 +0000 (UTC) Subject: [master] a62cb553d cache_ban: refactor: constify Message-ID: <20250226102805.0BABB10166C@lists.varnish-cache.org> commit a62cb553d3a535c9d13a423c8f3d01366261ab1a Author: Nils Goroll Date: Wed Feb 26 11:27:34 2025 +0100 cache_ban: refactor: constify diff --git a/bin/varnishd/cache/cache_ban_build.c b/bin/varnishd/cache/cache_ban_build.c index c9af918f7..8bb7c41f7 100644 --- a/bin/varnishd/cache/cache_ban_build.c +++ b/bin/varnishd/cache/cache_ban_build.c @@ -241,7 +241,7 @@ ban_add_spec(struct ban_proto *bp, const struct pvar *pv, int op, const char *a3 } static const char * -ban_add_double(struct ban_proto *bp, const struct pvar *pv, int op, double darg) +ban_add_double(const struct ban_proto *bp, const struct pvar *pv, int op, double darg) { uint64_t dtmp; uint8_t denc[sizeof darg]; From nils.goroll at uplex.de Wed Feb 26 13:41:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 26 Feb 2025 13:41:06 +0000 (UTC) Subject: [master] 6d5aa36cc cache_ban: refactor ban_iter Message-ID: <20250226134106.21E51109540@lists.varnish-cache.org> commit 6d5aa36cc13f0d09211ffa74e68d7ca607d921d2 Author: Nils Goroll Date: Wed Feb 26 11:29:07 2025 +0100 cache_ban: refactor ban_iter diff --git a/bin/varnishd/cache/cache_ban.c b/bin/varnishd/cache/cache_ban.c index 0c6a6a2de..d611445d6 100644 --- a/bin/varnishd/cache/cache_ban.c +++ b/bin/varnishd/cache/cache_ban.c @@ -220,6 +220,7 @@ ban_get_lump(const uint8_t **bs) static void ban_iter(const uint8_t **bs, struct ban_test *bt) { + const void *lump; uint64_t dtmp; memset(bt, 0, sizeof *bt); @@ -229,15 +230,14 @@ ban_iter(const uint8_t **bs, struct ban_test *bt) bt->arg1_spec = (const char *)*bs; (*bs) += (*bs)[0] + 2; } + lump = ban_get_lump(bs); + bt->oper = *(*bs)++; if (BANS_HAS_ARG2_DOUBLE(bt->arg1)) { - dtmp = vbe64dec(ban_get_lump(bs)); - bt->oper = *(*bs)++; - + dtmp = vbe64dec(lump); memcpy(&bt->arg2_double, &dtmp, sizeof dtmp); return; } - bt->arg2 = ban_get_lump(bs); - bt->oper = *(*bs)++; + bt->arg2 = lump; if (BANS_HAS_ARG2_SPEC(bt->oper)) bt->arg2_spec = ban_get_lump(bs); } From nils.goroll at uplex.de Thu Feb 27 11:14:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 27 Feb 2025 11:14:05 +0000 (UTC) Subject: [master] 08e778882 cache_http2: Clarification refactor: HEADERS always starts a new stream Message-ID: <20250227111405.A92C8A039E@lists.varnish-cache.org> commit 08e7788820a99875b8493c5a7ceeab97dc85dc6f Author: Nils Goroll Date: Wed Feb 26 18:44:01 2025 +0100 cache_http2: Clarification refactor: HEADERS always starts a new stream The existing code suggested that HEADERS could arrive on an existing stream, which was then checked for its state to be idle. But because we do not support PUSH_PROMISE, the only way to open a new stream is by receiving a HEADERS frame, which opens a new stream (see rfc9113 section 5.1). In our case, the "idle" state for all streams is implicit. Diff is best viewed ignoring whitespace (git log -pb) diff --git a/bin/varnishd/http2/cache_http2_proto.c b/bin/varnishd/http2/cache_http2_proto.c index e857203f6..b3d6e7397 100644 --- a/bin/varnishd/http2/cache_http2_proto.c +++ b/bin/varnishd/http2/cache_http2_proto.c @@ -735,31 +735,30 @@ h2_rx_headers(struct worker *wrk, struct h2_sess *h2, struct h2_req *r2) CHECK_OBJ_NOTNULL(wrk, WORKER_MAGIC); ASSERT_RXTHR(h2); - if (r2 == NULL) { - if (h2->rxf_stream <= h2->highest_stream) { - H2S_Lock_VSLb(h2, SLT_SessError, "H2: new stream ID < highest stream"); - return (H2CE_PROTOCOL_ERROR); // rfc7540,l,1153,1158 - } - /* NB: we don't need to guard the read of h2->open_streams - * because headers are handled sequentially so it cannot - * increase under our feet. - */ - if (h2->open_streams >= - (int)h2->local_settings.max_concurrent_streams) { - H2S_Lock_VSLb(h2, SLT_Debug, - "H2: stream %u: Hit maximum number of " - "concurrent streams", h2->rxf_stream); - return (H2SE_REFUSED_STREAM); // rfc7540,l,1200,1205 - } - h2->highest_stream = h2->rxf_stream; - r2 = h2_new_req(h2, h2->rxf_stream, NULL); - } - CHECK_OBJ_NOTNULL(r2, H2_REQ_MAGIC); - - if (r2->state != H2_S_IDLE) { + if (r2 != NULL) { H2S_Lock_VSLb(h2, SLT_SessError, "H2: rx headers on non-idle stream"); return (H2CE_PROTOCOL_ERROR); // XXX spec ? } + + if (h2->rxf_stream <= h2->highest_stream) { + H2S_Lock_VSLb(h2, SLT_SessError, "H2: new stream ID < highest stream"); + return (H2CE_PROTOCOL_ERROR); // rfc7540,l,1153,1158 + } + /* NB: we don't need to guard the read of h2->open_streams + * because headers are handled sequentially so it cannot + * increase under our feet. + */ + if (h2->open_streams >= + (int)h2->local_settings.max_concurrent_streams) { + H2S_Lock_VSLb(h2, SLT_Debug, + "H2: stream %u: Hit maximum number of " + "concurrent streams", h2->rxf_stream); + return (H2SE_REFUSED_STREAM); // rfc7540,l,1200,1205 + } + h2->highest_stream = h2->rxf_stream; + r2 = h2_new_req(h2, h2->rxf_stream, NULL); + CHECK_OBJ_NOTNULL(r2, H2_REQ_MAGIC); + assert(r2->state == H2_S_IDLE); r2->state = H2_S_OPEN; req = r2->req; From nils.goroll at uplex.de Thu Feb 27 11:14:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 27 Feb 2025 11:14:05 +0000 (UTC) Subject: [master] f25f08a57 http2: Update rfc references Message-ID: <20250227111405.BDFFFA03A2@lists.varnish-cache.org> commit f25f08a57d2a372cca6e7515d8f6f2c1236de867 Author: Nils Goroll Date: Thu Feb 27 12:12:25 2025 +0100 http2: Update rfc references diff --git a/bin/varnishd/http2/cache_http2_proto.c b/bin/varnishd/http2/cache_http2_proto.c index b3d6e7397..3ef036145 100644 --- a/bin/varnishd/http2/cache_http2_proto.c +++ b/bin/varnishd/http2/cache_http2_proto.c @@ -464,6 +464,8 @@ h2_rx_window_update(struct worker *wrk, struct h2_sess *h2, struct h2_req *r2) /********************************************************************** * Incoming PRIORITY, possibly an ACK of one we sent. + * + * deprecated, rfc9113,l,1103,1104 */ static h2_error v_matchproto_(h2_rxframe_f) @@ -737,7 +739,7 @@ h2_rx_headers(struct worker *wrk, struct h2_sess *h2, struct h2_req *r2) if (r2 != NULL) { H2S_Lock_VSLb(h2, SLT_SessError, "H2: rx headers on non-idle stream"); - return (H2CE_PROTOCOL_ERROR); // XXX spec ? + return (H2CE_PROTOCOL_ERROR); // rfc9113,l,887,891 } if (h2->rxf_stream <= h2->highest_stream) { From nils.goroll at uplex.de Thu Feb 27 19:34:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 27 Feb 2025 19:34:06 +0000 (UTC) Subject: [master] 276be1aa1 cache_ban: Polish Message-ID: <20250227193406.3CBE7B2BEB@lists.varnish-cache.org> commit 276be1aa161cd0de9cb8be753d21bad494625196 Author: Nils Goroll Date: Thu Feb 27 20:33:24 2025 +0100 cache_ban: Polish diff --git a/bin/varnishd/cache/cache_ban_build.c b/bin/varnishd/cache/cache_ban_build.c index 8bb7c41f7..b370c563d 100644 --- a/bin/varnishd/cache/cache_ban_build.c +++ b/bin/varnishd/cache/cache_ban_build.c @@ -316,10 +316,10 @@ BAN_AddTest(struct ban_proto *bp, "expected conditional (%s) got \"%s\"", arg_operhelp[BAN_ARGIDX(pv->tag)], a2)); - if ((pv->flag & BANS_FLAG_DURATION) == 0) - return (ban_add_spec(bp, pv, op, a3)); - else + if (pv->flag & BANS_FLAG_DURATION) return (ban_add_duration(bp, pv, op, a3)); + else + return (ban_add_spec(bp, pv, op, a3)); } /*--------------------------------------------------------------------