A plea for a more useful and discoverable built-in VCL

[Dridi Boukelmoune]

On Mon, Nov 6, 2023 at 6:36 AM Guillaume Quintard
<guillaume.quintard at gmail.com> wrote:
>
> Hi everybody!
>
> A bunch of questions I regularly get regarding Varnish behavior revolve around the built-in vcl, mainly, I get one of these three:
> - why is Varnish not caching?
> - how come something is happening in vcl_X even though I don't have it in my vcl?
> - what on earth is that built-in vcl you are talking about?
>
> As usual, I have a half-baked solution with a bunch of problems, which will hopefully inspire smarter people to fix the issue properly.
>
> What I came up with is here: https://github.com/varnish/toolbox/tree/verbose_builtin/vcls/verbose_builtin
> Essentially, use std.log() to explain what the built-in code is doing.
>
> At the moment, it's a purely opt-in solution, meaning that you need to know about builtin.vcl to find it, which doesn't really help with discoverability, but I intend on including that code in the docker image, which should raise awareness a bit.
> The absolute best in my mind would be to have something similar in core, but I can see how importing std would be a hurdle. Maybe as part of packaging, we could include that file in the provided default.vcl?
>
> I dismissed the performance penalty of printing a few more lines as negligible, but I could be wrong about that.

I think we should rather have new VSL tags for the call action in VCL
(calling and returning from a non-step subroutine) because this way if
they really consume too many resources (vsl_buffer, vsl_space, etc)
they could easily be masked.

For example:
- VCL_sub_call
- VCL_sub_return

I would probably not log subroutines called by VMODs and leave that as
an exercise to VMOD authors.

> There's also the question of phrasing, so we can have a message that is concise but also gives enough information to debug the behavior. But that's very minor, and the least of our worries here.
>
> Thoughts?

If you look at the VCL reference documentation between 6.0 and 7.4 you
may notice that we went from a monolithic manual to a bunch of them:

https://varnish-cache.org/docs/6.0/reference/
https://varnish-cache.org/docs/7.4/reference/

One thing I've wanted to do for a while is to add a vcl-builtin(7)
manual, and I think even a small synopsis and a dump of
bin/varnishd/builtin.vcl would improve discoverability. I would
however prefer to have something smarter than that, but haven't given
enough thoughts on how to proceed.

For example, the vcl-step(7) and vcl-var(7) manual are authoritative,
so updating them changes Varnish's behavior.

We could imagine having a vcl-builtin(7) manual where we document and
put individual subroutines authoritative snippets, and generate
bin/varnishd/builtin.vcl from that.

Cheers