[VDD] VSL query extensions

[Dridi Boukelmoune]

Hello everyone,

I'd like to not put this item on the agenda for next Monday so I'll
briefly present it here since nobody at Varnish Software thinks it
will be controversial. Attendees outside of Varnish Software, please
ack that this should have a de-facto "send patches" green light or we
can discuss this further next Monday.

It comes in 3 simple steps.

1) Multi-line queries

When someone wishes to capture logs for multiple reasons, they need
to OR them, possibly with brackets. Instead the proposal is to support
one query per line, ignore empty lines, and treat tokens starting with
'#' as comments.

Example:

    varnishlog -q '(my-first-query) or (my-second-query)'
    varnishlog -q '
        my-first-query
        my-second-query
        ...
    '

2) Read queries from a file

Similar to reading varnishncsa's format from a file:

    cat >bad-things.txt <<EOF
    # internal errors
    RespStatus >= 500

    # backend problems
    ...

    # internal incident 4253
    ...

    # varnish issue 1799
    ...
    EOF
    varnishlog -Q bad-things.txt

3) Query scoping

The current scope is always the transaction group, but sometimes we
may want conditions to apply to a subset of the `-g` option (for
example two fields of the same VSL record).

Proposed solution: inside brackets have the possibility to reduce the
scope with the very first token. The scope cannot outlive the parent
scope, and the query keeps its default @group transaction.

Example:

    # beresp cached more than 2m before v_b_r{}
    @tag TTL[1] eq RFC and TTL[2] >= 120

The groups would be:

- @group (default when omitted, matches -g option)
- @request (same as -g request)
- @vxid (same as -g vxid)
- @tag (single record)

Things that shouldn't work:

    varnishlog -q '@request ...'
    varnishlog -q '@tag ... and (@group ...)'
    varnishlog -g vxid -q '@request ...'
    varnishlog -q '@tag SomeRecord and DifferentRecord'

My opinion was that the current VSL query architecture should
accommodate such changes nicely, and Martin is also not too worried
about it.

4) Bonus topic

I also wanted to put a small performance topic on the agenda: mmap'ing
regular files when reading offline VSL. The current code treats files
(-r option) the same way live SHMLOG VSL is and allocates copies.

The idea is to mmap regular files and avoid needless copies, and keep
the current `-r file` handling for the stdin case. I hope to speed up
processing when it comes to dumps in the GBs.

Someone asked about binary VSL future plans. All of this should be
orthogonal to changes in the underlying VSL format. Nothing should
require an soname major bump in libvarnishapi (minor bump for the -Q
option).

Dridi