vdd 2022-04-22 notes

Nils Goroll nils.goroll at uplex.de
Fri Apr 22 14:31:15 UTC 2022

VDD 2022q1 NOTES

Paris 2022-04-21 - 2022-04-22

#1 martin on varnishlog scalability

- logging is a killer feature which users' business cases depends upon
- Some users would rather see clients wait than losing a log record.
- log clients are single threaded
- for multiple queries, basic structure building done per log client
- ideas

    - binary log nice to have, not a solution

    - solve desync issue by giving the option to report lost entries

      - or throttling / blocking varnishd

    - merge logging into varnishd

    - drop session logging, copy session attributes into request scope

    - potential to optimize logging away (kills varnishlog -d, could keep
internal buffers longer instead)

    - varnishlogd muti-query multi-thread (read shm and fan out to pipes/udses)

    - structured shared memory

    - use utility threads inside varnishd to write logs to fs, which is then
read by the clients.

    - compiled queries

    - pluggable, custom log implementations in varnishd, renovate interface
first (phk)

- side topics

    - custom tags for vmods

    - log all counter diffs

#2 phk jumps agenda and talks about how he spent the covid period (rational
R1000 machine at datamuseum.dk)

#3 Varnish Association / Charity / Forening / Verein / Foundation

- martin talks about VS efforts in this direction
- VS owners have welcomed the idea
- slide set from law firm LUCENTUM

    - disputed: Transfer copyright to foundation?

    - maybe not even relevant?

    - Lots of commitees

    - trademark: VS AB owns, licenses it to Verein

    - cost: VS willing to take on running costs

- Should the entity spread money for development "centralized VML"
- nils presents ideas for bylaws
- most important questions

    - do the bylaws get the verein involved in money etc?

    - "grassroots" or "corporate" style?

#4 TLS

- arguments:
      - ToS / QoS by content
      - Adaptive Congestion Control by content
      - kernel TLS for mallanox hw offload
- use cases

    - dynamic loading of certificates based on SNI from VCL

    - not just by name, want to prefer specific cert

    - session key sharing

    - vcl_${proto} {}

    - client certificates for backend connections

- should use something along the lines of CF keyless

    - no private keys in varnishd worker process

    - cert server should be pluggable, should have some kind of CLI

    - could significantly complicate H3

- martin: have done poc for plugging "keyless" as openssl encryption engine

    - seems hard to do it ssl lib implemention independent - try to not make it
hard to switch impl

    - VS: linking against stock openssl

- stack transports?
- do the work

    - UPLEX has sponsor

    - VS might upstream as open source

    - limit the scope to OpenSSL >= 1.1 ?

    - strawman the vcertd (dridi?)

- varnishtest

    - dridi has done 2 rounds of polishing of https in varnishtest

    - dridi has plans for a 3 round before porting to trunk

    - to submodule SOONER (revive slinks PR?)

      - when varnish is branched off, the vtest code can be bundled (no submodule)

      - needs patch back-ports convenience

      - refs



#5 H3

- Lars is still working on the quant project
- should look at msquic
- quictls openssl fork for quic
- haproxy has own quic
- argument against openssl is that h3/quic moves fast
- probably needs a new survey of available implementations (asad)

#6 extensions, pluggable stevedores and protocols (xx)

- phk talks about changes which are going to happen
- martin brings up loading extensions in mgmt, seems not to be a killer argument
- built-in stevedores stay in varnishd
- built-in stevedores will register before extensions, so default=malloc|umem
will remain

- martins batch insertion

    - HSH_Insert() EXP_Insert() BAN_Insert() batched functions

    - martin+slink

#7 >1 level of return(vcl, prefix="example.com"")

- current implementation is just defensive
- parameter with absolute limit on switches, default should remain just 1 level
- structure label names? xyz/abc -> xyz can only return (vcl(xyz/abc))

    - is the prefix argument absolute or relative

    - write the docs, implement

#8 bundled VMODs

- discussion about varnish-modules and var vs. objvar vs. native variables in vcl

objvar today:

vcl_init {
    new myvar = taskvar.string();
set resp.http.foo = myvar.get()

objvar with designated setter/getter methods:

myvar = "foo";
set resp.http.foo = myvar;

native scoped variable suggestion:

{vcl,client,backend}.* symbols
set client.var.foo = <string expr>;
set backend.var.foo = <string expr>;
set vcl.var.foo = <string expr>;

native scoped variable alternate suggestion:

{be{re{q,sp}}}.var.* with req=>bereq and {beresp,obj}=>resp copies

another scoped variable suggestion:

[global] {[req|bereq|resp|beresp]} [type] var;

beresp variables would be persisted with the object

-> write a concrete proposal for user docs (guillaume)


    - vmod as is only works with non-persistent storage

    - should look at http-wg suggestion

    - important feature

suggestion for inclusion into varnish-cache

    accept: Filter accept-like headers

    bodyaccess: Client request body access

    header: Modify and change complex HTTP headers

    VS to upstream headerplus?

    str: String operations

    should turn into type methods, need arguments for type methods

    task for dridis apprentice

    tcp: TCP connections tweaking

    vsthrottle: Request and bandwidth throttling

    should be redone with modules and better methods

side discussion: issues with vmod cookie
- removes duplicate cookies
- need iterator over multiple cookies
-> simon please open cookie

#9 compiled VCL sideloading
- guillaume: still need vcl even if overriding cc_command for vmod import
- we should put vmods into .so
- complicated because of vmod random names

#10 slash stevedores (fixed size mem + persistent tiered)

#11 Homework assignment for friday, read: https://httpwg.org/http-extensions/
- asad recommends wg meeting videos, https://www.youtube.com/user/ietf/videos

#12 synth bodies and filters
- phk wants to remove the option to reset the synth body

- vsb is contigous region, good case for discontinous object creation
- dridi is of the opinion that vcl resetting the beresp.body should pay the cost
of resetting a vfp

#13 via backends


dridi syntax suggestion:

backend proxy {
    .type = connect; # registered by vext proxy
    .host = "...";
backend b{1..3} {
    .type = http1; # built-in, default value
    .tls = true;
    .client_cert = "/bla/c{1..3}";
    .via = proxy;
    .host = "...";
    .authority = "...";
backend default {
    .type = round_robin; # registered by vext directors
    .via = [b1, b2, b3]

#14 multiple headers with the same name

shall implement
- xtaskvar.foreach_header(HEADER, SUB)
- xtaskvar.foreach_cookie(HEADER, SUB)
- HEADER.count
- std.sanitize_headers()

    - collect all collectable headers

- C code fail for more duplicate headers

    - date?

Legacy headers which are SF compatible could become binary
- https://mnot.github.io/I-D/draft-nottingham-binary-structured-headers.html
- https://www.rfc-editor.org/internet-drafts/draft-ietf-httpbis-retrofit-00.html

-> how could VCL look if those became a reality?

#15 parallel/async CLI

Implement cli-like interface to vkeyservd with async/parallel syntax
Check how this could look like for varnishd

Only specific commands would become async (vcl.load and vcl.inline mostly)
bans could serve as a template (vcl.load -a name ... -> 205 async started /
vcl.list shows "pending"/"failed")
buffer output in vsb, cli command to query (vcl.show -o vcl_name, -o for
compiler output)
martin suggests a "job interface" jobs.list / jobs.wait name

#16 optional vmods / conditional code?

import cookie [from PATH] [or skip];

with (import cookieplus) {
    # cookieplus.do_stuff()
} else with (import cookie) {
    # cookie.do_stuff()
} else {
    # regex time


** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstraße 32
22301 Hamburg

tel +49 40 28805731
mob +49 170 2723133
fax +49 40 42949753



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20220422/d4d9afa8/attachment-0001.bin>

More information about the varnish-dev mailing list