From nils.goroll at uplex.de Fri Jan 3 09:16:07 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Fri, 3 Jan 2025 09:16:07 +0000 (UTC) Subject: [master] 414d1b266 doc: build breaks without sphinx and doctools on macos Message-ID: <20250103091607.5689D11E0D9@lists.varnish-cache.org> commit 414d1b2668b320336d9c1d9188dfc46e6f78fd5c Author: Per Buer Date: Thu Dec 12 08:49:27 2024 +0100 doc: build breaks without sphinx and doctools on macos diff --git a/doc/sphinx/installation/install_source.rst b/doc/sphinx/installation/install_source.rst index 1e10f9782..071600d0a 100644 --- a/doc/sphinx/installation/install_source.rst +++ b/doc/sphinx/installation/install_source.rst @@ -154,7 +154,7 @@ Build dependencies on macOS To compile varnish on macOS, these steps should install the required dependencies: -* Install ``xcode`` via the App Store +* Install xcode: `xcode-select --install` * Install dependencies via `brew`:: @@ -166,9 +166,11 @@ dependencies: docutils \ sphinx-doc -* Add sphinx to PATH as advised by the installer:: +* Add the sphinx tools and the docutils to PATH so `configure` will find them. - PATH="/usr/local/opt/sphinx-doc/bin:$PATH" + PATH="/opt/homebrew/opt/sphinx-doc/bin:/opt/homebrew/opt/docutils/bin:$PATH" + +It'll be a good idea to persist these changes so you can rebuild the source later. Then continue `Compiling Varnish`_ From nils.goroll at uplex.de Sat Jan 4 21:47:11 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Sat, 4 Jan 2025 21:47:11 +0000 (UTC) Subject: [master] 2133a93e5 [doc] Update vcl-backends.rst - fix typos Message-ID: <20250104214711.19A2812233A@lists.varnish-cache.org> commit 2133a93e5f606e69134d8d734bd25b4bd233c79f Author: Gilles Doge Date: Sat Jan 4 21:19:44 2025 +0100 [doc] Update vcl-backends.rst - fix typos diff --git a/doc/sphinx/users-guide/vcl-backends.rst b/doc/sphinx/users-guide/vcl-backends.rst index 99142cbca..423a688e5 100644 --- a/doc/sphinx/users-guide/vcl-backends.rst +++ b/doc/sphinx/users-guide/vcl-backends.rst @@ -21,7 +21,7 @@ Somewhere in the top there will be a section that looks a bit like this.:: # .port = "8080"; # } -We remove the comment markings in this text stanza making the it look like.:: +We remove the comment markings in this code block making it look like.:: backend default { .host = "127.0.0.1"; @@ -141,7 +141,7 @@ You can have something like this:: Note that the first regular expressions will match "foo.com", "www.foo.com", "zoop.foo.com" and any other host ending in "foo.com". In this example this is intentional but you might want it to be a bit -more tight, maybe relying on the ``==`` operator in stead, like this:: +more tight, maybe relying on the ``==`` operator instead, like this:: sub vcl_recv { if (req.http.host == "foo.com" || req.http.host == "www.foo.com") { From nils.goroll at uplex.de Mon Jan 13 14:40:10 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 13 Jan 2025 14:40:10 +0000 (UTC) Subject: [master] a104a2b4e Point to Jens Axboe's CONTRIBUTING.md for guidance Message-ID: <20250113144010.D1558115678@lists.varnish-cache.org> commit a104a2b4e469c35c7f60216be402ccb5dd7dbe3f Author: Nils Goroll Date: Mon Jan 13 15:31:00 2025 +0100 Point to Jens Axboe's CONTRIBUTING.md for guidance as agreed during bugwash Obsoletes #4251 diff --git a/CONTRIBUTING b/CONTRIBUTING index e5a889c6d..f15022d2d 100644 --- a/CONTRIBUTING +++ b/CONTRIBUTING @@ -27,3 +27,18 @@ Pull requests are handled like other tickets. Trivial pull requests (fix typos, etc) are welcomed, but they may be committed by a core team member and the author credited in the commit message. + +For anything non trivial, please take Jens Axboe's excellent contributing guide +as guidance: https://github.com/axboe/liburing/blob/master/CONTRIBUTING.md + +Notable differences for Varnish-Cache are: + +* For code style, we broadly follow bsd style(9) https://man.freebsd.org/cgi/man.cgi?query=style&sektion=9 + +* Regarding commit messages, we are usually less strict + +* The Varnish-Cache repository uses a linear history, so all changes are rebased + on top of the current upstream head always. When your PR can no longer be + merged, rebase it and force-push your changes. + +* For testing `make distcheck` should pass From phk at FreeBSD.org Tue Jan 14 08:38:09 2025 From: phk at FreeBSD.org (Poul-Henning Kamp) Date: Tue, 14 Jan 2025 08:38:09 +0000 (UTC) Subject: [master] df8ee3355 Fix two cases where sphinx detected URLs where it should not. Message-ID: <20250114083809.2E8E611632E@lists.varnish-cache.org> commit df8ee3355e519bdc9bdb33dc154769bcf7b3d13f Author: Poul-Henning Kamp Date: Tue Jan 14 08:36:36 2025 +0000 Fix two cases where sphinx detected URLs where it should not. diff --git a/doc/sphinx/phk/ipv6suckage.rst b/doc/sphinx/phk/ipv6suckage.rst index d57d9fbf7..d4dd18bf1 100644 --- a/doc/sphinx/phk/ipv6suckage.rst +++ b/doc/sphinx/phk/ipv6suckage.rst @@ -37,12 +37,16 @@ webserver on the same machine. No worries, says the power that controls what URLs look like, we will just stick the port number after the IP# with a colon: +.. code-block:: text + http://192.168.0.1:8080/... That obviously does not work with IPv6, so RFC3986 comes around and says "darn, we didn't think of that" and puts the IPV6 address in [...] giving us: +.. code-block:: text + http://[1080::8:800:200C:417A]:8080/ Remember that "harmless in shells" detail ? Yeah, sorry about that. @@ -55,11 +59,15 @@ even need to know if it is a IPv4 or IPv6 address in the first place. But it returns the IP# in one buffer and the port number in another, so if you want to format the sockaddr in the by RFC5952 recommended way (the same as RFC3986), you need to inspect the version field -in the sockaddr to see if you should do +in the sockaddr to see if you should do: + +.. code-block:: text "%s:%s", host, port -or +or: + +.. code-block:: text "[%s]:%s", host, port diff --git a/doc/sphinx/reference/varnishncsa.rst b/doc/sphinx/reference/varnishncsa.rst index de5ea8428..6375f7288 100644 --- a/doc/sphinx/reference/varnishncsa.rst +++ b/doc/sphinx/reference/varnishncsa.rst @@ -237,7 +237,7 @@ SIGNALS NOTES ===== -The %r formatter is equivalent to "%m http://%{Host}i%U%q %H". This +The %r formatter is equivalent to ``%m http://%{Host}i%U%q %H``. This differs from apache's %r behavior, equivalent to "%m %U%q %H". Furthermore, when using the %r formatter, if the Host header appears multiple times in a single transaction, the first occurrence is used. From nils.goroll at uplex.de Tue Jan 14 17:39:10 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Tue, 14 Jan 2025 17:39:10 +0000 (UTC) Subject: [master] c41241826 vmod_debug: some renames in preparation for more debug transports Message-ID: <20250114173910.7BC6C10210B@lists.varnish-cache.org> commit c412418261f2a64e48b36c93536358bc53f00ef7 Author: Nils Goroll Date: Tue Jan 14 18:34:19 2025 +0100 vmod_debug: some renames in preparation for more debug transports diff --git a/vmod/automake_boilerplate_debug.am b/vmod/automake_boilerplate_debug.am index 2a70c5e8f..0021d59e7 100644 --- a/vmod/automake_boilerplate_debug.am +++ b/vmod/automake_boilerplate_debug.am @@ -13,7 +13,7 @@ libvmod_debug_la_SOURCES = \ vmod_debug_dyn.c \ vmod_debug_filters.c \ vmod_debug_obj.c \ - vmod_debug_transports.c + vmod_debug_transport_reembarking_http1.c libvmod_debug_la_CFLAGS = diff --git a/vmod/vmod_debug.c b/vmod/vmod_debug.c index a5a097163..847477d0a 100644 --- a/vmod/vmod_debug.c +++ b/vmod/vmod_debug.c @@ -332,7 +332,7 @@ event_load(VRT_CTX, struct vmod_priv *priv) priv->methods = priv_vcl_methods; debug_add_filters(ctx); - debug_transport_init(); + debug_transport_reembarking_http1_init(); return (0); } @@ -1286,5 +1286,5 @@ xyzzy_resolve_range(VRT_CTX, struct VARGS(resolve_range) *args) VCL_VOID xyzzy_use_reembarking_http1(VRT_CTX) { - debug_transport_use_reembarking_http1(ctx); + debug_transport_reembarking_http1_use(ctx); } diff --git a/vmod/vmod_debug.h b/vmod/vmod_debug.h index ddff80dc8..be09c46cc 100644 --- a/vmod/vmod_debug.h +++ b/vmod/vmod_debug.h @@ -34,8 +34,8 @@ debug_add_filters(VRT_CTX); void debug_remove_filters(VRT_CTX); -/* vmod_debug_transports.c */ +/* vmod_debug_transport_reembarking_http1.c */ void -debug_transport_use_reembarking_http1(VRT_CTX); +debug_transport_reembarking_http1_use(VRT_CTX); void -debug_transport_init(void); +debug_transport_reembarking_http1_init(void); diff --git a/vmod/vmod_debug_transports.c b/vmod/vmod_debug_transport_reembarking_http1.c similarity index 98% rename from vmod/vmod_debug_transports.c rename to vmod/vmod_debug_transport_reembarking_http1.c index 8a60a5a62..32e176ab3 100644 --- a/vmod/vmod_debug_transports.c +++ b/vmod/vmod_debug_transport_reembarking_http1.c @@ -199,7 +199,7 @@ dbg_deliver_finish(struct req *req, struct v1l **v1lp, int err) static struct transport DBG_transport; void -debug_transport_init(void) +debug_transport_reembarking_http1_init(void) { DBG_transport = HTTP1_transport; DBG_transport.name = "DBG"; @@ -207,7 +207,7 @@ debug_transport_init(void) } void -debug_transport_use_reembarking_http1(VRT_CTX) +debug_transport_reembarking_http1_use(VRT_CTX) { struct req *req; From nils.goroll at uplex.de Mon Jan 20 14:41:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 20 Jan 2025 14:41:06 +0000 (UTC) Subject: [master] 818ca0991 vcp: Poll dying connection pools asynchronously Message-ID: <20250120144106.8F29112416C@lists.varnish-cache.org> commit 818ca099105e3ebe1d96fb8deaa91aed6ff1280e Author: Thibaut Artis Date: Mon Sep 23 17:45:47 2024 +0200 vcp: Poll dying connection pools asynchronously Before, we would wait in a blocking loop for connection pools to be freed before deleting them for good. This commit transforms this blocking loop into a CLI hook that will delete the freed connection pools at each trigger. Refs #4064 Better diff with the --ignore-all-space option. diff --git a/bin/varnishd/cache/cache_cli.c b/bin/varnishd/cache/cache_cli.c index 0a99bc3cf..3670a1bc4 100644 --- a/bin/varnishd/cache/cache_cli.c +++ b/bin/varnishd/cache/cache_cli.c @@ -78,6 +78,7 @@ cli_cb_before(const struct cli *cli) VSL(SLT_CLI, NO_VXID, "Rd %s", VSB_data(cli->cmd)); Lck_Lock(&cli_mtx); VCL_Poll(); + VCP_RelPoll(); } static void diff --git a/bin/varnishd/cache/cache_conn_pool.c b/bin/varnishd/cache/cache_conn_pool.c index 9e940d42d..301e658e3 100644 --- a/bin/varnishd/cache/cache_conn_pool.c +++ b/bin/varnishd/cache/cache_conn_pool.c @@ -105,14 +105,20 @@ struct conn_pool { }; static struct lock conn_pools_mtx; +static struct lock dead_pools_mtx; -static VRBT_HEAD(vrb, conn_pool) conn_pools = VRBT_INITIALIZER(&conn_pools); +VRBT_HEAD(vrb, conn_pool); VRBT_GENERATE_REMOVE_COLOR(vrb, conn_pool, entry, static) VRBT_GENERATE_REMOVE(vrb, conn_pool, entry, static) VRBT_GENERATE_FIND(vrb, conn_pool, entry, vcp_cmp, static) VRBT_GENERATE_INSERT_COLOR(vrb, conn_pool, entry, static) VRBT_GENERATE_INSERT_FINISH(vrb, conn_pool, entry, static) VRBT_GENERATE_INSERT(vrb, conn_pool, entry, vcp_cmp, static) +VRBT_GENERATE_NEXT(vrb, conn_pool, entry, static); +VRBT_GENERATE_MINMAX(vrb, conn_pool, entry, static); + +static struct vrb conn_pools = VRBT_INITIALIZER(&conn_pools); +static struct vrb dead_pools = VRBT_INITIALIZER(&dying_cps); /*-------------------------------------------------------------------- */ @@ -217,6 +223,22 @@ VCP_AddRef(struct conn_pool *cp) Lck_Unlock(&conn_pools_mtx); } +/*-------------------------------------------------------------------- + */ + +static void +vcp_destroy(struct conn_pool **cpp) +{ + struct conn_pool *cp; + + TAKE_OBJ_NOTNULL(cp, cpp, CONN_POOL_MAGIC); + AZ(cp->n_conn); + AZ(cp->n_kill); + Lck_Delete(&cp->mtx); + free(cp->endpoint); + FREE_OBJ(cp); +} + /*-------------------------------------------------------------------- * Release Conn pool, destroy if last reference. */ @@ -248,17 +270,57 @@ VCP_Rel(struct conn_pool **cpp) (void)shutdown(pfd->fd, SHUT_RDWR); cp->n_kill++; } - while (cp->n_kill) { - Lck_Unlock(&cp->mtx); - (void)usleep(20000); - Lck_Lock(&cp->mtx); - } Lck_Unlock(&cp->mtx); - Lck_Delete(&cp->mtx); - AZ(cp->n_conn); - AZ(cp->n_kill); - free(cp->endpoint); - FREE_OBJ(cp); + if (cp->n_kill == 0) { + vcp_destroy(&cp); + return; + } + Lck_Lock(&dead_pools_mtx); + /* + * Here we reuse cp's entry but it will probably not be correctly + * indexed because of the hack in VCP_RelPoll + */ + VRBT_INSERT(vrb, &dead_pools, cp); + Lck_Unlock(&dead_pools_mtx); +} + +void +VCP_RelPoll(void) +{ + struct vrb dead; + struct conn_pool *cp, *cp2; + + ASSERT_CLI(); + + Lck_Lock(&dead_pools_mtx); + if (VRBT_EMPTY(&dead_pools)) { + Lck_Unlock(&dead_pools_mtx); + return; + } + dead = dead_pools; + VRBT_INIT(&dead_pools); + Lck_Unlock(&dead_pools_mtx); + + VRBT_FOREACH_SAFE(cp, vrb, &dead, cp2) { + CHECK_OBJ_NOTNULL(cp, CONN_POOL_MAGIC); + if (cp->n_kill > 0) + continue; + VRBT_REMOVE(vrb, &dead, cp); + vcp_destroy(&cp); + } + + if (VRBT_EMPTY(&dead)) + return; + + Lck_Lock(&dead_pools_mtx); + /* + * The following insertion will most likely result in an + * unordered tree, but in this case it does not matter + * as we just want to iterate over all the elements + * in the tree in order to delete them. + */ + VRBT_INSERT(vrb, &dead_pools, dead.rbh_root); + Lck_Unlock(&dead_pools_mtx); } /*-------------------------------------------------------------------- @@ -582,6 +644,7 @@ void VCP_Init(void) { Lck_New(&conn_pools_mtx, lck_conn_pool); + Lck_New(&dead_pools_mtx, lck_dead_pool); } /**********************************************************************/ diff --git a/bin/varnishd/cache/cache_varnishd.h b/bin/varnishd/cache/cache_varnishd.h index a0e4987fa..34a616c8d 100644 --- a/bin/varnishd/cache/cache_varnishd.h +++ b/bin/varnishd/cache/cache_varnishd.h @@ -472,6 +472,7 @@ void VSL_Flush(struct vsl_log *, int overflow); struct conn_pool; void VCP_Init(void); void VCP_Panic(struct vsb *, struct conn_pool *); +void VCP_RelPoll(void); /* cache_backend_probe.c */ void VBP_Init(void); diff --git a/include/tbl/locks.h b/include/tbl/locks.h index 3de8e33e6..773c31ffa 100644 --- a/include/tbl/locks.h +++ b/include/tbl/locks.h @@ -45,6 +45,7 @@ LOCK(pipestat) LOCK(probe) LOCK(sess) LOCK(conn_pool) +LOCK(dead_pool) LOCK(vbe) LOCK(vcapace) LOCK(vcl) From nils.goroll at uplex.de Mon Jan 20 14:46:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 20 Jan 2025 14:46:05 +0000 (UTC) Subject: [master] c8c016d8a Add VMOD version information to Panic output, debug.vmod and ELF Message-ID: <20250120144605.8EE97124D99@lists.varnish-cache.org> commit c8c016d8abdc9144279fefa2b411e7ebda8af118 Author: Nils Goroll Date: Thu Mar 7 14:54:19 2024 +0100 Add VMOD version information to Panic output, debug.vmod and ELF This commit adds a facility which I had wanted for a very long time, and should have added much earlier: Analyzing bug reports with VMODs involved used to be complicated by the fact that they typically did not contain reliable version information. We add to vmodtool.py the generic code to create version information from generate.py with minor modifications. It adds the version set for autotools and, if possible, the git revision to the vmodtool-generated interface code. We copy that to struct vrt and output it for panics and the debug.vmod CLI command. For builds from distributions, save the git revision to a file. Being at it, we polish the panic output slightly for readability. There are two elements to the version information: - "version", which is intended to be user-friendly and - "vcs" (for version control system) which is intended to represent a commit id from the scm. The VCC file Stanza $Version overrides "version". As an example, the output from a panic now might look like this: vmods = { debug = {p=0x7f5b19c2c600, abi=\"Varnish trunk 172e7b43a49bdf43d82cc6b10ab08362939fc8a5\", vrt=19.1, vcs=\"172e7b43a49bdf43d82cc6b10ab08362939fc8a5\", version=\"Varnish trunk\"}, }, The debug.vmod CLI outout is, for example: 1 vtc (path="/home/slink/Devel/varnish-git/varnish-cache/vmod/.libs/libvmod_vtc.so", version="Varnish trunk", vcs="172e7b43a49bdf43d82cc6b10ab08362939fc8a5") And, finally, a readelf example to extract the vcs string: $ readelf -p.vmod_vcs ./vmod/.libs/libvmod_std.so String dump of section '.vmod_vcs': [ 0] 172e7b43a49bdf43d82cc6b10ab08362939fc8a5 diff --git a/.gitignore b/.gitignore index a6eb65683..fe61bb233 100644 --- a/.gitignore +++ b/.gitignore @@ -80,6 +80,7 @@ cscope.*out /vmod/vcc_*_if.c /vmod/vcc_*_if.h /vmod/vmod_*.rst +/vmod/vmod_vcs_version.txt # Man-files and binaries /man/*.1 diff --git a/bin/varnishd/cache/cache_vrt_vmod.c b/bin/varnishd/cache/cache_vrt_vmod.c index c1fa22070..3b5e7c564 100644 --- a/bin/varnishd/cache/cache_vrt_vmod.c +++ b/bin/varnishd/cache/cache_vrt_vmod.c @@ -65,6 +65,8 @@ struct vmod { const char *abi; unsigned vrt_major; unsigned vrt_minor; + const char *vcs; + const char *version; }; static VTAILQ_HEAD(,vmod) vmods = VTAILQ_HEAD_INITIALIZER(vmods); @@ -147,6 +149,8 @@ VPI_Vmod_Init(VRT_CTX, struct vmod **hdl, unsigned nbr, void *ptr, int len, v->abi = d->abi; v->vrt_major = d->vrt_major; v->vrt_minor = d->vrt_minor; + v->vcs = d->vcs; + v->version = d->version; REPLACE(v->nm, nm); REPLACE(v->path, path); @@ -201,9 +205,19 @@ VMOD_Panic(struct vsb *vsb) VSB_cat(vsb, "vmods = {\n"); VSB_indent(vsb, 2); - VTAILQ_FOREACH(v, &vmods, list) - VSB_printf(vsb, "%s = {%p, %s, %u.%u},\n", - v->nm, v, v->abi, v->vrt_major, v->vrt_minor); + VTAILQ_FOREACH(v, &vmods, list) { + VSB_printf(vsb, "%s = {", v->nm); + VSB_indent(vsb, 2); + VSB_printf(vsb, "p=%p, abi=\"%s\", vrt=%u.%u,\n", + v, v->abi, v->vrt_major, v->vrt_minor); + VSB_bcat(vsb, "vcs=", 4); + VSB_quote(vsb, v->vcs, -1, VSB_QUOTE_CSTR); + VSB_bcat(vsb, ", version=", 10); + VSB_quote(vsb, v->version, -1, VSB_QUOTE_CSTR); + VSB_indent(vsb, -2); + VSB_bcat(vsb, "},\n", 3); + } + VSB_indent(vsb, -2); VSB_cat(vsb, "},\n"); } @@ -218,8 +232,11 @@ ccf_debug_vmod(struct cli *cli, const char * const *av, void *priv) (void)av; (void)priv; ASSERT_CLI(); - VTAILQ_FOREACH(v, &vmods, list) - VCLI_Out(cli, "%5d %s (%s)\n", v->ref, v->nm, v->path); + VTAILQ_FOREACH(v, &vmods, list) { + VCLI_Out(cli, "%5d %s (path=\"%s\", version=\"%s\"," + " vcs=\"%s\")\n", v->ref, v->nm, v->path, v->version, + v->vcs); + } } static struct cli_proto vcl_cmds[] = { diff --git a/doc/changes.rst b/doc/changes.rst index e465cd0ed..a14b6ef66 100644 --- a/doc/changes.rst +++ b/doc/changes.rst @@ -41,6 +41,42 @@ Varnish Cache NEXT (2025-03-15) .. PLEASE keep this roughly in commit order as shown by git-log / tig (new to old) +* Two fields have been added to the VMOD data registered with varnish-cache: + + - ``vcs`` for Version Control System is intended as an identifier from the + source code management system, e.g. the git revision, to identify the exact + source code which was used to build a VMOD binary. + + - ``version`` is intended as a more user friendly identifier as to which + version of a vmod a binary represents. + + Panics and the ``debug.vmod`` CLI command output now contain these + identifiers. + + Where supported by the compiler and linker, the ``vcs`` identifier is also + reachable via the ``.vmod_vcs`` section of the vmod shared object ELF file and + can be extracted, for example, using ``readelf -p.vmod_vcs `` + +* ``vmodtool.py`` now creates a file ``vmod_vcs_version.txt`` in the current + working directory when called from a git tree. This file is intended to + transport version control system information to builds from distribution + bundles. + + vmod authors should add it to the distribution and otherwise ignore it for + SCM. + + Where git and automake are used, this can be accomplished by adding + ``vmod_vcs_version.txt`` to the ``.gitignore`` file and to the ``EXTRA_DIST`` + and ``DISTCLEANFILES`` variables in ``Makefile.am``. + + If neither git is used nor ``vmod_vcs_version.txt`` present, ``vmodtool.py`` + will add ``NOGIT`` to the vmod as the vcs identifier. + +* ``vmodtool.py`` now accepts a ``$Version`` stanza in vmod vcc files to set the + vmod version as registered with Varnish-Cache. If ``$Version`` is not present, + an attempt is made to extract ``PACKAGE_STRING`` from an automake + ``Makefile``, otherwise ``NOVERSION`` is used as the version identifier. + * The scope of VCL variables ``req.is_hitmiss`` and ``req.is_hitpass`` is now restricted to ``vcl_miss, vcl_deliver, vcl_pass, vcl_synth`` and ``vcl_pass, vcl_deliver, vcl_synth`` respectively. diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst index e8de061ec..cf28deac0 100644 --- a/doc/sphinx/reference/vmod.rst +++ b/doc/sphinx/reference/vmod.rst @@ -56,6 +56,7 @@ data structures that do all the hard work. The std VMODs vmod.vcc file looks somewhat like this:: $ABI strict + $Version my.version $Module std 3 "Varnish Standard Module" $Event event_function $Function STRING toupper(STRANDS s) @@ -73,6 +74,11 @@ version, use ``strict``. If it complies to the VRT and only needs to be rebuilt when breaking changes are introduced to the VRT API, use ``vrt``. +The ``$Version`` line is also optional. It specifies the version identifier +compiled into the VMOD binary for later identification. If omitted, +``PACKAGE_STRING`` from an automake ``Makefile`` will be used, or ``NOVERSION`` +otherwise. + The ``$Module`` line gives the name of the module, the manual section where the documentation will reside, and the description. diff --git a/include/vrt.h b/include/vrt.h index 1b4be285e..23cb77776 100644 --- a/include/vrt.h +++ b/include/vrt.h @@ -493,6 +493,8 @@ struct vmod_data { const char *proto; const char *json; const char *abi; + const char *vcs; + const char *version; }; /*********************************************************************** diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py index 21d91a7b4..f554ce8a8 100755 --- a/lib/libvcc/vmodtool.py +++ b/lib/libvcc/vmodtool.py @@ -45,6 +45,7 @@ import json import optparse import os import re +import subprocess import sys import time @@ -128,6 +129,7 @@ CTYPES.update(PRIVS) DEPRECATED = {} + ####################################################################### def deprecated(key, txt): @@ -930,6 +932,14 @@ class AliasStanza(Stanza): def json(self, jl): jl.append(["$ALIAS", self.sym_alias, self.sym_name]) +class VersionStanza(Stanza): + + ''' $Version version ... ''' + + def parse(self): + if len(self.toks) < 2: + self.syntax() + self.vcc.pkgstr = " ".join(self.toks[1:]) ####################################################################### @@ -944,6 +954,7 @@ DISPATCH = { "Synopsis": SynopsisStanza, "Alias": AliasStanza, "Restrict": RestrictStanza, + "Version": VersionStanza } @@ -966,6 +977,7 @@ class vcc(): self.auto_synopsis = True self.modname = None self.csn = None + self.pkgstr = None def openfile(self, fn): self.commit_files.append(fn) @@ -1180,10 +1192,60 @@ class vcc(): fo.write('\t\"\\n\\x03\"\n};\n') fo.write('#undef STRINGIFY\n') + # parts from varnish-cache include/generate.py + def version(self): + srcdir = os.path.dirname(self.inputfile) + + pkgstr = "NOVERSION" + + if self.pkgstr is not None: + pkgstr = self.pkgstr + else: + for d in [srcdir, "."]: + f = os.path.join(d, "Makefile") + if not os.path.exists(f): + continue + for pkgstr in open(f): + if pkgstr[:14] == "PACKAGE_STRING": + pkgstr = pkgstr.split("=")[1].strip() + break + break + return pkgstr + + # parts from varnish-cache include/generate.py + def vcs(self): + srcdir = os.path.dirname(self.inputfile) + + gitver = subprocess.check_output([ + "git -C %s rev-parse HEAD 2>/dev/null || echo NOGIT" % + srcdir], shell=True, universal_newlines=True).strip() + gitfile = "vmod_vcs_version.txt" + + if gitver == "NOGIT": + for d in [".", srcdir]: + f = os.path.join(d, gitfile) + if not os.path.exists(f): + continue + fh = open(f, "r") + if not fh: + continue + gitver = fh.read() + fh.close() + break; + else: + fh = open(gitfile, "w") + fh.write(gitver) + fh.close() + + return gitver def vmod_data(self, fo): + version = json.dumps(self.version()) + vcs = json.dumps(self.vcs()) vmd = "Vmod_%s_Data" % self.modname fo.write('\n') + fo.write('__attribute__((section(".vmod_vcs"), used))\n') + fo.write('const char vmod_vcs[] = %s;' % vcs) for i in (714, 759, 765): fo.write("/*lint -esym(%d, %s) */\n" % (i, vmd)) fo.write("\nextern const struct vmod_data %s;\n" % vmd) @@ -1197,6 +1259,8 @@ class vcc(): fo.write('\t.func_len =\tsizeof(%s),\n' % self.csn) fo.write('\t.json =\t\tVmod_Json,\n') fo.write('\t.abi =\t\tVMOD_ABI_Version,\n') + fo.write('\t.version =\t%s,\n' % version) + fo.write('\t.vcs =\tvmod_vcs,\n') fo.write("};\n") def mkcfile(self): diff --git a/vmod/Makefile.am b/vmod/Makefile.am index 1d2df6317..0449eaf4b 100644 --- a/vmod/Makefile.am +++ b/vmod/Makefile.am @@ -5,7 +5,9 @@ TESTS = @VMOD_TESTS@ include $(top_srcdir)/vsc.am include $(top_srcdir)/vtc.am -EXTRA_DIST = $(TESTS) +EXTRA_DIST = $(TESTS) vmod_vcs_version.txt + +DISTCLEANFILES = vmod_vcs_version.txt AM_LDFLAGS = $(AM_LT_LDFLAGS) From nils.goroll at uplex.de Mon Jan 20 18:58:08 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 20 Jan 2025 18:58:08 +0000 (UTC) Subject: [master] ab78f460d vcc_expr: gc (struct func_arg).cname Message-ID: <20250120185808.C7BAC105F9D@lists.varnish-cache.org> commit ab78f460dc0e4a3add6a1c6df8621d71ddf08583 Author: Nils Goroll Date: Mon Jan 20 19:34:30 2025 +0100 vcc_expr: gc (struct func_arg).cname as reported by Flexelint Ref c57f4405a450f89c905da3ef8327b64c1d2fcb8d diff --git a/lib/libvcc/vcc_expr.c b/lib/libvcc/vcc_expr.c index 1e3a6e054..003f1513c 100644 --- a/lib/libvcc/vcc_expr.c +++ b/lib/libvcc/vcc_expr.c @@ -444,7 +444,6 @@ vcc_priv_arg(struct vcc *tl, const char *p, struct symbol *sym) struct func_arg { vcc_type_t type; const struct vjsn_val *enums; - const char *cname; const char *name; const char *val; struct expr *result; From nils.goroll at uplex.de Mon Jan 20 18:58:08 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 20 Jan 2025 18:58:08 +0000 (UTC) Subject: [master] f01231959 vmodtool: make vcs_version static & move attribute to a better place Message-ID: <20250120185808.E5362105FA0@lists.varnish-cache.org> commit f01231959c15092a0b9c056f3b7c2a69365d511b Author: Nils Goroll Date: Mon Jan 20 19:37:52 2025 +0100 vmodtool: make vcs_version static & move attribute to a better place Reported by Flexelint. Ref c8c016d8abdc9144279fefa2b411e7ebda8af118 diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py index f554ce8a8..883fbe365 100755 --- a/lib/libvcc/vmodtool.py +++ b/lib/libvcc/vmodtool.py @@ -1244,8 +1244,8 @@ class vcc(): vcs = json.dumps(self.vcs()) vmd = "Vmod_%s_Data" % self.modname fo.write('\n') - fo.write('__attribute__((section(".vmod_vcs"), used))\n') - fo.write('const char vmod_vcs[] = %s;' % vcs) + fo.write('static const char vmod_vcs[] ') + fo.write('__attribute__((section(".vmod_vcs"), used)) = %s;\n' % vcs) for i in (714, 759, 765): fo.write("/*lint -esym(%d, %s) */\n" % (i, vmd)) fo.write("\nextern const struct vmod_data %s;\n" % vmd) From nils.goroll at uplex.de Mon Jan 20 18:58:09 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 20 Jan 2025 18:58:09 +0000 (UTC) Subject: [master] 52cd2f32f cache_conn_pool: cleaup VRBT_GENERATE macro invocations Message-ID: <20250120185809.15B11105FA3@lists.varnish-cache.org> commit 52cd2f32f8671c2728ed30994190e51b2eb2d87b Author: Nils Goroll Date: Mon Jan 20 19:42:42 2025 +0100 cache_conn_pool: cleaup VRBT_GENERATE macro invocations Reported by Flexelint Ref 818ca099105e3ebe1d96fb8deaa91aed6ff1280e diff --git a/bin/varnishd/cache/cache_conn_pool.c b/bin/varnishd/cache/cache_conn_pool.c index 301e658e3..f510e3b68 100644 --- a/bin/varnishd/cache/cache_conn_pool.c +++ b/bin/varnishd/cache/cache_conn_pool.c @@ -114,8 +114,8 @@ VRBT_GENERATE_FIND(vrb, conn_pool, entry, vcp_cmp, static) VRBT_GENERATE_INSERT_COLOR(vrb, conn_pool, entry, static) VRBT_GENERATE_INSERT_FINISH(vrb, conn_pool, entry, static) VRBT_GENERATE_INSERT(vrb, conn_pool, entry, vcp_cmp, static) -VRBT_GENERATE_NEXT(vrb, conn_pool, entry, static); -VRBT_GENERATE_MINMAX(vrb, conn_pool, entry, static); +VRBT_GENERATE_NEXT(vrb, conn_pool, entry, static) +VRBT_GENERATE_MINMAX(vrb, conn_pool, entry, static) static struct vrb conn_pools = VRBT_INITIALIZER(&conn_pools); static struct vrb dead_pools = VRBT_INITIALIZER(&dying_cps); From nils.goroll at uplex.de Wed Jan 22 14:20:07 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 22 Jan 2025 14:20:07 +0000 (UTC) Subject: [master] cf7451118 wrap __attribute__((x, used)) as v_used_(x) Message-ID: <20250122142007.C4E5C11B396@lists.varnish-cache.org> commit cf7451118279caf05b94201949746a7583a5d67c Author: Nils Goroll Date: Wed Jan 22 15:16:19 2025 +0100 wrap __attribute__((x, used)) as v_used_(x) SunCC does not support it. gcc apparently has it since at least 3.1, but I simply added it to the existing version check for v_cold_ under the assumption that 4.3 is already ancient enough for everyone to have it. diff --git a/include/vdef.h b/include/vdef.h index 1f3bbec43..381a4f799 100644 --- a/include/vdef.h +++ b/include/vdef.h @@ -243,8 +243,10 @@ int __llvm_gcov_flush(void); #if __GNUC_PREREQ__(4, 3) || defined(__clang__) # define v_cold_ __attribute__((cold)) +# define v_used_(x) __attribute__((x, used)) #else # define v_cold_ +# define v_used_(x) #endif #if defined __has_attribute diff --git a/lib/libvcc/vmodtool.py b/lib/libvcc/vmodtool.py index 883fbe365..308e53dae 100755 --- a/lib/libvcc/vmodtool.py +++ b/lib/libvcc/vmodtool.py @@ -1245,7 +1245,7 @@ class vcc(): vmd = "Vmod_%s_Data" % self.modname fo.write('\n') fo.write('static const char vmod_vcs[] ') - fo.write('__attribute__((section(".vmod_vcs"), used)) = %s;\n' % vcs) + fo.write('v_used_(section(".vmod_vcs")) = %s;\n' % vcs) for i in (714, 759, 765): fo.write("/*lint -esym(%d, %s) */\n" % (i, vmd)) fo.write("\nextern const struct vmod_data %s;\n" % vmd) From nils.goroll at uplex.de Wed Jan 22 18:53:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 22 Jan 2025 18:53:05 +0000 (UTC) Subject: [master] 586d0c6d2 cache_vcl: Add a facility for unlocked director iteration Message-ID: <20250122185305.BF29A7C6D@lists.varnish-cache.org> commit 586d0c6d23e1b35d44ede2a90b70e76c4df461ae Author: Nils Goroll Date: Fri Jul 26 20:02:59 2024 +0200 cache_vcl: Add a facility for unlocked director iteration Pondering #4140 made me realize that our central per-vcl director list constitutes a classic case of iterating a mutable list. That besides the already known fact that running potentially expensive iterator functions while holding the vcl_mtx is a very bad idea. So this patch is a suggestion on how to get out of this situation. It does not go all the way (it still leaves unsolved a similar problem of iterating over all backends of _all vcls_), but shows an attempt on how to tackle the "iterate over one VCL's backends". We add a fittingly named 'vdire' facility which manages the director list and, in particular, director retirement. The main change is that, while iterators are active on the director list, vcl_mtx is _not_ held and any request of a director to retire only ends up in a resignation, which manifests by this director being put on a second list. When all iterators have completed, the resignation list is worked and the actual retirement carried out. diff --git a/bin/varnishd/cache/cache_director.h b/bin/varnishd/cache/cache_director.h index 9a12c6b3d..43cc85a06 100644 --- a/bin/varnishd/cache/cache_director.h +++ b/bin/varnishd/cache/cache_director.h @@ -43,7 +43,8 @@ struct vcldir { struct director *dir; struct vcl *vcl; const struct vdi_methods *methods; - VTAILQ_ENTRY(vcldir) list; + VTAILQ_ENTRY(vcldir) directors_list; + VTAILQ_ENTRY(vcldir) resigning_list; const struct vdi_ahealth *admin_health; vtim_real health_changed; char *cli_name; diff --git a/bin/varnishd/cache/cache_vcl.c b/bin/varnishd/cache/cache_vcl.c index 262b48538..c8c31014d 100644 --- a/bin/varnishd/cache/cache_vcl.c +++ b/bin/varnishd/cache/cache_vcl.c @@ -368,8 +368,126 @@ VCL_Update(struct vcl **vcc, struct vcl *vcl) assert((*vcc)->temp->is_warm); } +/*-------------------------------------------------------------------- + * vdire: Vcl DIrector REsignation Management (born out of a dire situation) + * iterators over the director list need to register. + * while iterating, directors can not retire immediately, + * they get put on a list of resigning directors. The + * last iterator executes the retirement. + */ + +static struct vdire * +vdire_new(struct lock *mtx, const struct vcltemp **tempp) +{ + struct vdire *vdire; + + ALLOC_OBJ(vdire, VDIRE_MAGIC); + AN(vdire); + AN(mtx); + VTAILQ_INIT(&vdire->directors); + VTAILQ_INIT(&vdire->resigning); + vdire->mtx = mtx; + vdire->tempp = tempp; + return (vdire); +} + +/* starting an interation prevents removals from the directors list */ +void +vdire_start_iter(struct vdire *vdire) +{ + + CHECK_OBJ_NOTNULL(vdire, VDIRE_MAGIC); + + // https://github.com/varnishcache/varnish-cache/pull/4142#issuecomment-2593091097 + ASSERT_CLI(); + + Lck_Lock(vdire->mtx); + vdire->iterating++; + Lck_Unlock(vdire->mtx); +} + +void +vdire_end_iter(struct vdire *vdire) +{ + struct vcldir_head resigning = VTAILQ_HEAD_INITIALIZER(resigning); + const struct vcltemp *temp = NULL; + struct vcldir *vdir; + unsigned n; + + CHECK_OBJ_NOTNULL(vdire, VDIRE_MAGIC); + + Lck_Lock(vdire->mtx); + AN(vdire->iterating); + n = --vdire->iterating; + + if (n == 0) { + VTAILQ_SWAP(&vdire->resigning, &resigning, vcldir, resigning_list); + VTAILQ_FOREACH(vdir, &resigning, resigning_list) + VTAILQ_REMOVE(&vdire->directors, vdir, directors_list); + temp = *vdire->tempp; + } + Lck_Unlock(vdire->mtx); + + VTAILQ_FOREACH(vdir, &resigning, resigning_list) + vcldir_retire(vdir, temp); +} + +// if there are no iterators, remove from directors and retire +// otherwise put on resigning list to work when iterators end +void +vdire_resign(struct vdire *vdire, struct vcldir *vdir) +{ + const struct vcltemp *temp = NULL; + + CHECK_OBJ_NOTNULL(vdire, VDIRE_MAGIC); + AN(vdir); + + Lck_Lock(vdire->mtx); + if (vdire->iterating != 0) { + VTAILQ_INSERT_TAIL(&vdire->resigning, vdir, resigning_list); + vdir = NULL; + } else { + VTAILQ_REMOVE(&vdire->directors, vdir, directors_list); + temp = *vdire->tempp; + } + Lck_Unlock(vdire->mtx); + + if (vdir != NULL) + vcldir_retire(vdir, temp); +} + +// unlocked version of vcl_iterdir +// pat can also be NULL (to iterate all) +static int +vdire_iter(struct cli *cli, const char *pat, const struct vcl *vcl, + vcl_be_func *func, void *priv) +{ + int i, found = 0; + struct vcldir *vdir; + struct vdire *vdire = vcl->vdire; + + vdire_start_iter(vdire); + VTAILQ_FOREACH(vdir, &vdire->directors, directors_list) { + if (vdir->refcnt == 0) + continue; + if (pat != NULL && fnmatch(pat, vdir->cli_name, 0)) + continue; + found++; + i = func(cli, vdir->dir, priv); + if (i < 0) { + found = i; + break; + } + found += i; + } + vdire_end_iter(vdire); + return (found); +} + + /*--------------------------------------------------------------------*/ +// XXX locked case across VCLs - should do without static int vcl_iterdir(struct cli *cli, const char *pat, const struct vcl *vcl, vcl_be_func *func, void *priv) @@ -378,7 +496,7 @@ vcl_iterdir(struct cli *cli, const char *pat, const struct vcl *vcl, struct vcldir *vdir; Lck_AssertHeld(&vcl_mtx); - VTAILQ_FOREACH(vdir, &vcl->director_list, list) { + VTAILQ_FOREACH(vdir, &vcl->vdire->directors, directors_list) { if (fnmatch(pat, vdir->cli_name, 0)) continue; found++; @@ -416,10 +534,10 @@ VCL_IterDirector(struct cli *cli, const char *pat, vcl = NULL; } AZ(VSB_finish(vsb)); - Lck_Lock(&vcl_mtx); if (vcl != NULL) { - found = vcl_iterdir(cli, VSB_data(vsb), vcl, func, priv); + found = vdire_iter(cli, VSB_data(vsb), vcl, func, priv); } else { + Lck_Lock(&vcl_mtx); VTAILQ_FOREACH(vcl, &vcl_head, list) { i = vcl_iterdir(cli, VSB_data(vsb), vcl, func, priv); if (i < 0) { @@ -429,8 +547,8 @@ VCL_IterDirector(struct cli *cli, const char *pat, found += i; } } + Lck_Unlock(&vcl_mtx); } - Lck_Unlock(&vcl_mtx); VSB_destroy(&vsb); return (found); } @@ -439,31 +557,37 @@ static void vcl_BackendEvent(const struct vcl *vcl, enum vcl_event_e e) { struct vcldir *vdir; + struct vdire *vdire; ASSERT_CLI(); CHECK_OBJ_NOTNULL(vcl, VCL_MAGIC); AZ(vcl->busy); + vdire = vcl->vdire; - Lck_Lock(&vcl_mtx); - VTAILQ_FOREACH(vdir, &vcl->director_list, list) + vdire_start_iter(vdire); + VTAILQ_FOREACH(vdir, &vdire->directors, directors_list) VDI_Event(vdir->dir, e); - Lck_Unlock(&vcl_mtx); + vdire_end_iter(vdire); } static void vcl_KillBackends(const struct vcl *vcl) { struct vcldir *vdir, *vdir2; + struct vdire *vdire; CHECK_OBJ_NOTNULL(vcl, VCL_MAGIC); assert(vcl->temp == VCL_TEMP_COLD || vcl->temp == VCL_TEMP_INIT); + vdire = vcl->vdire; + CHECK_OBJ_NOTNULL(vdire, VDIRE_MAGIC); + /* - * Unlocked because no further directors can be added, and the + * Unlocked and sidelining vdire because no further directors can be added, and the * remaining ones need to be able to remove themselves. */ - VTAILQ_FOREACH_SAFE(vdir, &vcl->director_list, list, vdir2) + VTAILQ_FOREACH_SAFE(vdir, &vdire->directors, directors_list, vdir2) VDI_Event(vdir->dir, VCL_EVENT_DISCARD); - assert(VTAILQ_EMPTY(&vcl->director_list)); + assert(VTAILQ_EMPTY(&vdire->directors)); } /*--------------------------------------------------------------------*/ @@ -522,6 +646,7 @@ VCL_Open(const char *fn, struct vsb *msg) AN(vcl); vcl->dlh = dlh; vcl->conf = cnf; + vcl->vdire = vdire_new(&vcl_mtx, &vcl->temp); return (vcl); } @@ -533,6 +658,7 @@ VCL_Close(struct vcl **vclp) TAKE_OBJ_NOTNULL(vcl, vclp, VCL_MAGIC); assert(VTAILQ_EMPTY(&vcl->filters)); AZ(dlclose(vcl->dlh)); + FREE_OBJ(vcl->vdire); FREE_OBJ(vcl); } @@ -701,7 +827,6 @@ vcl_load(struct cli *cli, vcl->loaded_name = strdup(name); XXXAN(vcl->loaded_name); - VTAILQ_INIT(&vcl->director_list); VTAILQ_INIT(&vcl->ref_list); VTAILQ_INIT(&vcl->filters); @@ -912,6 +1037,7 @@ vcl_cli_discard(struct cli *cli, const char * const *av, void *priv) if (!strcmp(vcl->state, VCL_TEMP_LABEL->name)) { VTAILQ_REMOVE(&vcl_head, vcl, list); free(vcl->loaded_name); + AZ(vcl->vdire); FREE_OBJ(vcl); } else if (vcl->temp == VCL_TEMP_COLD) { VCL_Poll(); diff --git a/bin/varnishd/cache/cache_vcl.h b/bin/varnishd/cache/cache_vcl.h index d8827b667..7b7b28ce1 100644 --- a/bin/varnishd/cache/cache_vcl.h +++ b/bin/varnishd/cache/cache_vcl.h @@ -37,7 +37,18 @@ struct vfilter; struct vcltemp; VTAILQ_HEAD(vfilter_head, vfilter); +VTAILQ_HEAD(vcldir_head, vcldir); +struct vdire { + unsigned magic; +#define VDIRE_MAGIC 0x51748697 + unsigned iterating; + struct vcldir_head directors; + struct vcldir_head resigning; + // vcl_mtx for now - to be refactored into separate mtx? + struct lock *mtx; + const struct vcltemp **tempp; +}; struct vcl { unsigned magic; @@ -50,10 +61,10 @@ struct vcl { unsigned busy; unsigned discard; const struct vcltemp *temp; - VTAILQ_HEAD(,vcldir) director_list; VTAILQ_HEAD(,vclref) ref_list; - int nrefs; + struct vdire *vdire; struct vcl *label; + int nrefs; int nlabels; struct vfilter_head filters; }; @@ -92,3 +103,13 @@ extern const struct vcltemp VCL_TEMP_COOLING[1]; assert(vcl_active == NULL || \ vcl_active->temp->is_warm); \ } while (0) + +/* cache_vrt_vcl.c used in cache_vcl.c */ +struct vcldir; +void vcldir_retire(struct vcldir *vdir, const struct vcltemp *temp); + +/* cache_vcl.c */ +void vdire_resign(struct vdire *vdire, struct vcldir *vdir); + +void vdire_start_iter(struct vdire *vdire); +void vdire_end_iter(struct vdire *vdire); diff --git a/bin/varnishd/cache/cache_vrt_vcl.c b/bin/varnishd/cache/cache_vrt_vcl.c index 264ad3446..b021e2280 100644 --- a/bin/varnishd/cache/cache_vrt_vcl.c +++ b/bin/varnishd/cache/cache_vrt_vcl.c @@ -240,7 +240,7 @@ VRT_AddDirector(VRT_CTX, const struct vdi_methods *m, void *priv, Lck_AssertHeld(&vcl_mtx); temp = vcl->temp; if (temp != VCL_TEMP_COOLING) - VTAILQ_INSERT_TAIL(&vcl->director_list, vdir, list); + VTAILQ_INSERT_TAIL(&vcl->vdire->directors, vdir, directors_list); if (temp->is_warm) VDI_Event(vdir->dir, VCL_EVENT_WARM); Lck_Unlock(&vcl_mtx); @@ -267,19 +267,15 @@ VRT_StaticDirector(VCL_BACKEND b) vdir->flags |= VDIR_FLG_NOREFCNT; } -static void -vcldir_retire(struct vcldir *vdir) +// vcldir is already removed from the directors list +// to be called only from vdire_* +void +vcldir_retire(struct vcldir *vdir, const struct vcltemp *temp) { - const struct vcltemp *temp; CHECK_OBJ_NOTNULL(vdir, VCLDIR_MAGIC); assert(vdir->refcnt == 0); - CHECK_OBJ_NOTNULL(vdir->vcl, VCL_MAGIC); - - Lck_Lock(&vcl_mtx); - temp = vdir->vcl->temp; - VTAILQ_REMOVE(&vdir->vcl->director_list, vdir, list); - Lck_Unlock(&vcl_mtx); + AN(temp); if (temp->is_warm) VDI_Event(vdir->dir, VCL_EVENT_COLD); @@ -302,7 +298,7 @@ vcldir_deref(struct vcldir *vdir) Lck_Unlock(&vdir->dlck); if (!busy) - vcldir_retire(vdir); + vdire_resign(vdir->vcl->vdire, vdir); return (busy); } @@ -378,6 +374,7 @@ VRT_LookupDirector(VRT_CTX, VCL_STRING name) struct vcl *vcl; struct vcldir *vdir; VCL_BACKEND dd, d = NULL; + struct vdire *vdire; CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC); AN(name); @@ -388,15 +385,17 @@ VRT_LookupDirector(VRT_CTX, VCL_STRING name) vcl = ctx->vcl; CHECK_OBJ_NOTNULL(vcl, VCL_MAGIC); - Lck_Lock(&vcl_mtx); - VTAILQ_FOREACH(vdir, &vcl->director_list, list) { + vdire = vcl->vdire; + + vdire_start_iter(vdire); + VTAILQ_FOREACH(vdir, &vdire->directors, directors_list) { dd = vdir->dir; if (strcmp(dd->vcl_name, name)) continue; d = dd; break; } - Lck_Unlock(&vcl_mtx); + vdire_end_iter(vdire); return (d); } From nils.goroll at uplex.de Wed Jan 22 18:53:05 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 22 Jan 2025 18:53:05 +0000 (UTC) Subject: [master] af2f05d5a vdire: Wait for directors to retire before staring a new iteration Message-ID: <20250122185305.D456F7C6F@lists.varnish-cache.org> commit af2f05d5a9783bc8571eb0b0936ca2db6d3edb7e Author: Nils Goroll Date: Mon Oct 28 16:27:41 2024 +0100 vdire: Wait for directors to retire before staring a new iteration This is to prevent a potential pileup of resigned directors with a constant stream of iterators coming on diff --git a/bin/varnishd/cache/cache_vcl.c b/bin/varnishd/cache/cache_vcl.c index c8c31014d..2140bc664 100644 --- a/bin/varnishd/cache/cache_vcl.c +++ b/bin/varnishd/cache/cache_vcl.c @@ -387,6 +387,7 @@ vdire_new(struct lock *mtx, const struct vcltemp **tempp) VTAILQ_INIT(&vdire->directors); VTAILQ_INIT(&vdire->resigning); vdire->mtx = mtx; + PTOK(pthread_cond_init(&vdire->cond, NULL)); vdire->tempp = tempp; return (vdire); } @@ -402,6 +403,8 @@ vdire_start_iter(struct vdire *vdire) ASSERT_CLI(); Lck_Lock(vdire->mtx); + while (! VTAILQ_EMPTY(&vdire->resigning)) + (void)Lck_CondWait(&vdire->cond, vdire->mtx); vdire->iterating++; Lck_Unlock(vdire->mtx); } @@ -425,6 +428,7 @@ vdire_end_iter(struct vdire *vdire) VTAILQ_FOREACH(vdir, &resigning, resigning_list) VTAILQ_REMOVE(&vdire->directors, vdir, directors_list); temp = *vdire->tempp; + PTOK(pthread_cond_broadcast(&vdire->cond)); } Lck_Unlock(vdire->mtx); @@ -581,6 +585,14 @@ vcl_KillBackends(const struct vcl *vcl) vdire = vcl->vdire; CHECK_OBJ_NOTNULL(vdire, VDIRE_MAGIC); + /* + * ensure all retirement has finished (vdire_start_iter waits for it) + */ + vdire_start_iter(vdire); + vdire_end_iter(vdire); + AZ(vdire->iterating); + assert(VTAILQ_EMPTY(&vdire->resigning)); + /* * Unlocked and sidelining vdire because no further directors can be added, and the * remaining ones need to be able to remove themselves. @@ -658,6 +670,7 @@ VCL_Close(struct vcl **vclp) TAKE_OBJ_NOTNULL(vcl, vclp, VCL_MAGIC); assert(VTAILQ_EMPTY(&vcl->filters)); AZ(dlclose(vcl->dlh)); + PTOK(pthread_cond_destroy(&vcl->vdire->cond)); FREE_OBJ(vcl->vdire); FREE_OBJ(vcl); } diff --git a/bin/varnishd/cache/cache_vcl.h b/bin/varnishd/cache/cache_vcl.h index 7b7b28ce1..f7fc31b60 100644 --- a/bin/varnishd/cache/cache_vcl.h +++ b/bin/varnishd/cache/cache_vcl.h @@ -47,6 +47,8 @@ struct vdire { struct vcldir_head resigning; // vcl_mtx for now - to be refactored into separate mtx? struct lock *mtx; + // to signal when iterators can enter again + pthread_cond_t cond; const struct vcltemp **tempp; }; From nils.goroll at uplex.de Mon Jan 27 15:17:06 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Mon, 27 Jan 2025 15:17:06 +0000 (UTC) Subject: [master] b1d31057d doc: Explain that we do not have partial object caching Message-ID: <20250127151706.5D6A5110560@lists.varnish-cache.org> commit b1d31057dd3e00e5ebf1b9d70314d39b52ceb2c8 Author: Nils Goroll Date: Mon Jan 27 16:15:46 2025 +0100 doc: Explain that we do not have partial object caching closes #4256 diff --git a/doc/sphinx/users-guide/storage-backends.rst b/doc/sphinx/users-guide/storage-backends.rst index 043025c2c..36014ff81 100644 --- a/doc/sphinx/users-guide/storage-backends.rst +++ b/doc/sphinx/users-guide/storage-backends.rst @@ -18,6 +18,13 @@ configuration is to use the malloc backend with a limited size. For a serious Varnish deployment you probably would want to adjust the storage settings. +All built-in storage backends cache full objects only, so, for example, to +support *n* concurrent cache hits on 1GB sized objects, the storage backend +should be configured to provide at least *n*\ GB of storage. For uncacheable +objects, the rule of thumb is *n* x ``transit_buffer``. + +Storage backends are also called stevedores. + default ~~~~~~~ From nils.goroll at uplex.de Wed Jan 29 14:50:10 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Wed, 29 Jan 2025 14:50:10 +0000 (UTC) Subject: [master] e787ff486 storage_simple: mark buffers Message-ID: <20250129145010.A3B73122C8F@lists.varnish-cache.org> commit e787ff486e07ea32fa23f4226d7be137e91adf8e Author: Nils Goroll Date: Tue Nov 19 10:27:23 2024 +0100 storage_simple: mark buffers Make sure that we do not confuse buffers with object data. The flags field comes at no extra memory cost on 64bit. For 32bit, this change breaks binary compatibility with any out-of-tree storages using storage_simple. diff --git a/bin/varnishd/storage/storage_simple.c b/bin/varnishd/storage/storage_simple.c index 1dea4dc39..8c81c85ee 100644 --- a/bin/varnishd/storage/storage_simple.c +++ b/bin/varnishd/storage/storage_simple.c @@ -184,6 +184,7 @@ SML_AllocBuf(struct worker *wrk, const struct stevedore *stv, size_t size, if (st == NULL) return (NULL); assert(st->space >= size); + st->flags = STORAGE_F_BUFFER; st->len = size; *ppriv = (uintptr_t)st; return (st->ptr); @@ -198,6 +199,7 @@ SML_FreeBuf(struct worker *wrk, const struct stevedore *stv, uintptr_t priv) CHECK_OBJ_NOTNULL(stv, STEVEDORE_MAGIC); CAST_OBJ_NOTNULL(st, (void *)priv, STORAGE_MAGIC); + assert(st->flags == STORAGE_F_BUFFER); sml_stv_free(stv, st); } diff --git a/bin/varnishd/storage/storage_simple.h b/bin/varnishd/storage/storage_simple.h index 881e0b857..207a11505 100644 --- a/bin/varnishd/storage/storage_simple.h +++ b/bin/varnishd/storage/storage_simple.h @@ -38,7 +38,8 @@ struct storage { unsigned magic; #define STORAGE_MAGIC 0x1a4e51c0 - + unsigned flags; +#define STORAGE_F_BUFFER 1 VTAILQ_ENTRY(storage) list; void *priv; From nils.goroll at uplex.de Thu Jan 30 08:54:10 2025 From: nils.goroll at uplex.de (Nils Goroll) Date: Thu, 30 Jan 2025 08:54:10 +0000 (UTC) Subject: [master] f98201041 doc: Mention vsm related topics in the platform notes Message-ID: <20250130085410.AC607120405@lists.varnish-cache.org> commit f9820104174c9f532b9f76745dedd90a5d5e136f Author: Nils Goroll Date: Thu Jan 30 09:52:04 2025 +0100 doc: Mention vsm related topics in the platform notes We had them in release docs, but not in the obvious place for "TODOs after installation". diff --git a/doc/sphinx/installation/platformnotes.rst b/doc/sphinx/installation/platformnotes.rst index 494e4eb9d..d9a77ad08 100644 --- a/doc/sphinx/installation/platformnotes.rst +++ b/doc/sphinx/installation/platformnotes.rst @@ -10,6 +10,42 @@ Platform specific notes On some platforms it is necessary to adjust the operating system before running Varnish on it. The systems and steps known to us are described in this section. +On Linux, use tmpfs for the workdir +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Varnish uses mapped files for shared memory, for which performance depends on +writes not blocking. On Linux, however, write throttling implemented by some +file systems (which is generally useful in other scenarios) can interact badly +with the way Varnish works and cause lockups or performance impacts. To avoid +such problems, it is recommended to use a ``tmpfs`` "virtual memory file system" +as the *workdir*. + +To ensure ``tmpfs`` is used, check the following: + +Determine the *workdir*. If you use a specific ``-n`` option to ``varnishd`` or +set the ``VARNISH_DEFAULT_N`` variable, it is that value. Otherwise run +``varnishd -x options``, which outputs the *workdir* default. + +Run ``df *workdir*``. If it reports ``tmpfs`` as the file system in the first +column, no additional action is necessary. + +Otherwise, consider creating a ``tmpfs`` mountpoint at *workdir*, or configure +*workdir* on an existing ``tmpfs``. + +Note: Very valid reasons exist for *not* following this recommendation, if you +know what you are doing. + +Lift locked memory limits +~~~~~~~~~~~~~~~~~~~~~~~~~ + +For the same reason as explained above, varnish tries to lock shared memory in +RAM. Therefore, the locked memory limit should ideally be set to unlimited or +sufficiently high to accommodate all mapped files. The specific minimum required +value is dynamic, depending among other factors on the number of VCLs loaded and +backends configured. As a rule of thumb, it should be a generous multiple of the +size of *workdir* when varnish is running. + +See :ref:`ref-vsm` for details. Transparent hugepages on Redhat Enterprise Linux 6 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/sphinx/reference/vsm.rst b/doc/sphinx/reference/vsm.rst index ad6524519..6378efb69 100644 --- a/doc/sphinx/reference/vsm.rst +++ b/doc/sphinx/reference/vsm.rst @@ -3,6 +3,8 @@ SPDX-License-Identifier: BSD-2-Clause See LICENSE file for full text of license +.. _ref-vsm: + %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% VSM: Shared Memory Logging and Statistics %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%