[master] 53155cc7b Add release documentation

[Nils Goroll]

commit 53155cc7bf56fe3b3ff8de69328b5bc24f93ea75
Author: Nils Goroll <nils.goroll at uplex.de>
Date:   Thu Sep 12 10:37:43 2024 +0200

    Add release documentation

diff --git a/doc/sphinx/reference/vmod.rst b/doc/sphinx/reference/vmod.rst
index 367d7e8ac..e8de061ec 100644
--- a/doc/sphinx/reference/vmod.rst
+++ b/doc/sphinx/reference/vmod.rst
@@ -859,6 +859,8 @@ You should call ``VSC_*_New()`` when your VMOD is loaded and
 ``VSC_*.h`` file for the full details about the structure that contains
 your counters.
 
+.. _ref-vmod-tmpdir:
+
 Temporary Files
 ===============
 
diff --git a/doc/sphinx/whats-new/changes-trunk.rst b/doc/sphinx/whats-new/changes-trunk.rst
index 2070fadd5..45d099559 100644
--- a/doc/sphinx/whats-new/changes-trunk.rst
+++ b/doc/sphinx/whats-new/changes-trunk.rst
@@ -13,57 +13,152 @@ merged, may be found in the `change log`_.
 
 .. _change log: https://github.com/varnishcache/varnish-cache/blob/master/doc/changes.rst
 
+Changes applying to most varnish-cache programs
+===============================================
+
+The environment variable ``VARNISH_DEFAULT_N`` now provides the default "varnish
+name" / "workdir" as otherwise specified by he ``-n`` argument to ``varnishd``
+and ``varnish*`` utilities except ``varnishtest``.
+
 varnishd
 ========
 
-Parameters
-~~~~~~~~~~
+A new ``linux`` jail has been added (configured via the ``-j`` argument) which is
+now the default on Linux. For now, it is almost identical to the ``unix`` jail
+with one :ref:`whatsnew_upgrading_CURRENT_linux_jail` added.
 
-**XXX changes in -p parameters**
+The port of a *listen_endpoint* given with the ``-a`` argument to ``varnishd``
+can now also be a numerical port range like ``80-89``, besides the existing
+options of port number (e.g. ``80``) and service name (e.g. ``http``). With a
+port range, Varnish will accept connections on all ports within the range.
 
-Other changes in varnishd
-~~~~~~~~~~~~~~~~~~~~~~~~~
+.. _whatsnew_changes_CURRENT_connq:
 
-Changes to VCL
-==============
+Backend connection queuing
+~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-VCL variables
-~~~~~~~~~~~~~
+A feature has been added to instruct backend tasks to queue if the backend has
+reached its ``max_connections``. This allows tasks to wait for a connection to
+become available rather than immediately fail. This feature must be enabled
+through new global parameters or individual backend properties:
 
-**XXX new, deprecated or removed variables, or changed semantics**
+* ``backend_wait_timeout`` sets the amount of time a task will wait.
+* ``backend_wait_limit`` sets the maximum number of tasks that can wait.
 
-Other changes to VCL
-~~~~~~~~~~~~~~~~~~~~
+These parameters can also be set for individual backends using the
+``wait_timeout`` and ``wait_limit`` properties.
 
-VMODs
-=====
+Tasks waiting on a backend going sick (either explicitly via the
+``backend.set_health`` command or implicitly through the probe) fail
+immediately.
 
-**XXX changes in the bundled VMODs**
+Global VSC counters have been added under ``MAIN``:
 
-varnishlog
-==========
+* ``backend_wait`` counts tasks which waited in queue for a connection.
+* ``backend_wait_fail`` counts tasks which waited in queue but failed because
+  ``wait_timeout`` was reached or the backend went sick.
+
+Parameters
+~~~~~~~~~~
+
+The ``backend_wait_timeout`` and ``backend_wait_limit`` parameters have been
+added, see :ref:`whatsnew_changes_CURRENT_connq` above for details.
 
-**XXX changes concerning varnishlog(1) and/or vsl(7)**
+The size of the buffer to hold panic messages is now tunable through the new
+``panic_buffer`` parameter.
 
-varnishadm
+Changes to VCL
+==============
+
+The ``wait_timeout`` and ``wait_limit`` backend properties have been added, see
+:ref:`whatsnew_changes_CURRENT_connq` above for details.
+
+For backends using the ``.via`` attribute to connect through a *proxy*, the
+``connect_timeout``, ``first_byte_timeout`` and ``between_bytes_timeout``
+attributes are now inherited from *proxy* unless explicitly given.
+
+varnishlog
 ==========
 
-**XXX changes concerning varnishadm(1) and/or varnish-cli(7)**
+Additional ``SessError`` VSL events are now generated for various HTTP/2
+protocol errors. Some HTTP/2 log events have been changed from ``Debug`` and
+``Error`` to ``SessError``.
 
 varnishstat
 ===========
 
-**XXX changes concerning varnishstat(1) and/or varnish-counters(7)**
+VSC counters for waiters have been added:
+
+* ``conns`` to count waits on idle connections
+* ``remclose`` to count idle connections closed by the peer
+* ``timeout`` to count idle connections which timed out in the waiter
+* ``action`` to count idle connections which resulted in a read
+
+These can be found under ``WAITER.<poolname>.``.
+
+The ``MAIN.backend_wait`` and ``MAIN.backend_wait_fail`` counters have been
+added, see :ref:`whatsnew_changes_CURRENT_connq` above for details.
 
 varnishtest
 ===========
 
-**XXX changes concerning varnishtest(1) and/or vtc(7)**
+``varnishtest`` now supports the ``shutdown`` command corresponding to the
+``shutdown(2)`` standard C library call.
 
 Changes for developers and VMOD authors
 =======================================
 
-**XXX changes concerning VRT, the public APIs, source code organization,
-builds etc.**
+.. _whatsnew_changes_CURRENT_VDP:
+
+VDP filter API changes
+~~~~~~~~~~~~~~~~~~~~~~
+
+The Varnish Delivery Processor (VDP) filter API has been generalized to also
+accommodate future use for backend request bodies:
+
+``VDP_Init()`` gained a ``struct busyobj *`` argument for use of VDPs on the
+backend side, which is mutually exclusive with the existing ``struct req *``
+argument (one of the two needs to be ``NULL``). ``VDP_Init()`` also gained an
+``intmax_t *`` pointer, which needs to point to the known content length of the
+body data or ``-1`` for "unknown length". Filters can change this value.
+
+``struct vdp_ctx`` lost the ``req`` member, but gained ``struct objcore *oc``,
+``struct http *hp`` and ``intmax_t *clen`` members. The rationale here is that a
+VDP should be concerned mainly with transforming body data (for which ``clen``
+is relevant) and optionally changing (from the ``vdp_init_f``) the headers sent
+before the body data, for which ``hp`` is intended. Some VDPs also work directly
+on a ``struct objcore *``, so ``oc`` is provided to the first VDP in the chain
+only.
+
+Generic VDPs should specifically not access the request or be concerned with the
+object.
+
+Yet special purpose VDPs still can take from ``VRT_CTX`` whatever references
+they need in the ``vdp_init_f`` and store them in their private data.
+
+Consequent to what as been explained above, ``vdp_init_f`` lost its ``struct
+objcore *`` argument.
+
+VDPs with no ``vdp_bytes_f`` function are now supported if the ``vdp_init_f``
+returns a value greater than zero to signify that the filter is not to be added
+to the chain. This is useful to support VDPs which only need to work on headers.
+
+.. _whatsnew_changes_CURRENT_Obj:
+
+Object API changes
+~~~~~~~~~~~~~~~~~~
+
+The ``ObjWaitExtend()`` Object API function gained a ``statep`` argument to
+optionally return the busy object state consistent with the current extension.
+A ``NULL`` value may be passed if the caller does not require it.
+
+Other changes relevant for developers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``VSS_resolver_range()`` as been added to ``libvarnish`` to implement resolution
+of port ranges.
+
+The implementation of the ``transit_buffer`` has now been made the
+responsibility of storage engines.
 
 *eof*
diff --git a/doc/sphinx/whats-new/upgrading-trunk.rst b/doc/sphinx/whats-new/upgrading-trunk.rst
index 6143fde99..d286031ab 100644
--- a/doc/sphinx/whats-new/upgrading-trunk.rst
+++ b/doc/sphinx/whats-new/upgrading-trunk.rst
@@ -8,26 +8,58 @@ released versions of Varnish, see:** :ref:`whats-new-index`
 Upgrading to Varnish **$NEXT_RELEASE**
 %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
 
-**XXX: how to upgrade from previous deployments to this
-version. Limited to work that has to be done for an upgrade, new
-features are listed in "Changes". Explicitly mention what does *not*
-have to be changed, especially in VCL. May include, but is not limited
-to:**
+In general, upgrading from Varnish 7.5 to **$NEXT_RELEASE** should not require any changes
+besides the actual upgrade.
 
-* Elements of VCL that have been removed or are deprecated, or whose
-  semantics have changed.
+The changes mentioned below are considered noteworthy nevertheless:
 
-* -p parameters that have been removed or are deprecated, or whose
-  semantics have changed.
+Noteworthy changes when upgrading varnishd
+==========================================
 
-* Changes in the CLI.
+Warning about failed memory locking
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-* Changes in the output or interpretation of stats or the log, including
-  changes affecting varnishncsa/-hist/-top.
+The ``Warning: mlock() of VSM failed`` message is now emitted when locking of
+shared memory segments (via ``mlock(2)``) fails. As Varnish performance may be
+severely impacted if shared memory segments are not resident in RAM, users
+seeing this message are urged to review the ``RLIMIT_MEMLOCK`` resource control
+as set via ``ulimit -l`` or ``LimitMEMLOCK`` with ``systemd(1)``. This is not
+new at all, just now the warning has been added to make administrators more
+aware.
 
-* Changes that may be necessary in VTCs or in the use of varnishtest.
+.. _whatsnew_upgrading_CURRENT_linux_jail:
 
-* Changes in public APIs that may require changes in VMODs or VAPI/VUT
-  clients.
+Warning if tmpfs is not used
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+On Linux (when the new ``linux`` jail is used), the ``Working directory not
+mounted on tmpfs partition`` warning is now emitted if the working directory is
+found to reside on a file system other than ``tmpfs``. While other file systems
+are supported (and might be the right choice where administrators understand how
+to avoid blocking disk IO while ``varnishd`` is writing to shared memory),
+``tmpfs`` is the failsafe option to avoid performance issues.
+
+Upgrading VCL
+=============
+
+.. _RFC9110: https://www.rfc-editor.org/rfc/rfc9110.html#section-14.4
+
+An issue has been addressed in the ``builtin.vcl`` where backend responses
+would fail if they contained a ``Content-Range`` header when no range was
+requested. According to `RFC9110`_, this header should just be ignored, yet
+some Varnish users might prefer stricter checks. Thus, we decided to change
+the ``builtin.vcl`` only and users hitting this issue are advised to call
+``vcl_beresp_range`` from custom VCL.
+
+Changes for developers and VMOD authors
+=======================================
+
+The VDP filter API has changed. See :ref:`whatsnew_changes_CURRENT_VDP` for details.
+
+The signature of ``ObjWaitExtend()`` has changed. See
+:ref:`whatsnew_changes_CURRENT_Obj` for details.
+
+``varnishd`` now creates a ``worker_tmpdir`` which can be used by VMODs for
+temporary files. See :ref:`ref-vmod-event-functions` for details.
 
 *eof*