VIP23 (VSL refactoring) design sketch

[Geoff Simmons]

On 5/3/19 10:51, Dridi Boukelmoune wrote:
> 
> First we lose the ability to treat the whole record as a single sting
> out of the box. The client will need to reconstruct the string from
> the various field types that may constitute a record. Think string and
> regex operators.

Yes, Martin and I recently had a conversation about this on #hacking.
For the string equality comparisons (eq and ne), we could conceivably
look into the comparison string and see if it has blanks that cross
field boundaries. But as you pointed out, some VSL payloads have blanks
that are not field separators (VCL_Log, FetchError, etc).

For regex comparisons, where the pattern could include things like "\s",
".*", "\W" and so forth, detecting whether the pattern might match a
field-separating blank has the look of a sophisticated computer
science-y problem.

Martin and I agreed that, at least in the first iteration, for VSLQ
queries with a string operator (equality or regex) we should render the
formatted payload into a temporary buffer and run the comparison against
that. Once we have it all working, maybe we could consider more clever
solutions, but I'd be surprised if we'd gain much from getting clever.

I think that an important part of this will be to emphasize in the docs
that authors of VSL queries should use field[n] indices and numeric
comparisons whenever it's appropriate. My (admittedly evidence-free)
impression is that not everyone uses them when they could.

Say you're scanning backend logs for fetches to backend foo, like this:

-q 'BackendOpen ~ "foo"'

Since the backend name is the second field, it would be better as:

-q 'BackendOpen[2] eq "foo"'

Or to look for backend fetches that take longer than 1 second, it
wouldn't surprise me if some people are doing something like:

-q 'Timestamp:Beresp ~ "[1-9]\.\d*$"'

When it should be:

-q 'Timestamp:Beresp[3] > 1.0'

We'll do much better if we can narrow the query to a field, and if we
don't have to convert numbers at all. And we should make sure that users
know about it.

> Finally, regarding the VSL file header that we'd hypothetically dump
> with varnishlog -w in the future (which I really hope we do), we could
> consider making this a VSL record itself that is not subject to
> vsl_reclen, querying or filtering, for the needs of dynamic VSLs, so
> that whenever something changes VSL clients see an update. It would
> then be varnishlog -w's job to build such a synthetic record in the
> VSL file before dumping actual logs.

I agree, and I'd say that when we do have meta-data entries in the log,
it should be possible for them to appear at any time, in both log files
and in-memory logs. That would put us on the path to dynamic SLT tags.

But we're not planning for that in the first iteration. phk has said
that he doesn't foresee dynamic VSL by the September release. And I
agree that we should get the rest working first.


Best,
Geoff
-- 
** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstraße 32
22301 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753

http://uplex.de

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20190503/445f71cb/attachment.bin>