Fwd: Writing vmods in Rust

[Guillaume Quintard]

Hi all,

So, almost 2 months after the initial announcement and quite a lot of
dogfooding, varnish-rs has seen a few releases and grown a bit. Even though
I still consider it in the alpha stage, I wanted to present a couple of
vmods I built on top of it.

The first vmod is the smallest one: vmod_rers (
https://github.com/gquintard/vmod_rers) that deals with Regular Expressions
in RuSt.
A cursory look at the VCC file (
https://github.com/gquintard/vmod_rers/blob/main/vmod.vcc) will quickly
reveal that it does way less than `
https://code.uplex.de/uplex-varnish/libvmod-re2/` however, there's still a
couple of nice things about it:
- regex are cached in a LRU store, so you don't have to recompile the same
dynamic expression over and over again
- named patterns!
- there is both a VDP and a VFP, in other words: you can modify content as
it enters the cache AND as it leaves it. Note however that to do this, we
do cache the whole body, use this feature responsibly.

The second one is beefier: vmod_reqwest (
https://github.com/gquintard/vmod_reqwest) is a vmod_curl on steroids.
The initial idea here is also to allow the VCL to send and receive HTTP
requests, but things got a bit out of hand, and in the end, it has a few
extra features:
- handle multiple requests at the same time, within the same task
- fire-and-forget, non-blocking mode: you can send 15 requests at the same
time and continue regular VCL processing as they are in flight
- proper VCL backend, instead of the regular static backends, you can use
dynamic ones backed by reqwest, offering:
  - redirect following, if the response sports a 30X status, the vmod can
automatically follow it
  - connection pooling, of course
  - HTTPS backends
  - proxies for both HTTP and HTTPS
  - automatic brotli (and gzip and deflate) decompression
There are a few things missing at the moment, notably probes, but they are
coming.

So, things look pretty exciting BUT I haven't tested that in production,
and I've mostly been operating in a vacuum so far, so I would love some
feedback on APIs, performance, bugs, etc.

-- 
Guillaume Quintard


---------- Forwarded message ---------
From: Guillaume Quintard <guillaume.quintard at gmail.com>
Date: Sun, Dec 5, 2021 at 5:32 PM
Subject: Writing vmods in Rust
To: varnish-misc at varnish-cache.org <varnish-misc at varnish-cache.org>


Hi everyone,

I've been working for a while on a little project that ended up taking
quite a chunk of my time, and while it's nowhere near done, I feel like
it's time to show the current progress. *In other words: that stuff ain't
ready, don't put it in prod.*

Here we go: we can *build vmods using pure rust*.
For example, this guys:
https://github.com/gquintard/varnish-rs/tree/main/examples
And the crates `varnish` and `varnish-sys`, that make this possible have
been published on crates.io, and we have documentation!
https://docs.rs/varnish/latest/varnish/

Without trying to start a language war, there are a few benefits to using
Rust over C:
- the language is designed to prevent a few classes of annoying bugs, and
the compiler is very good at explaining where you went wrong
- a lot of the C practices (assert everywhere, miniobj, etc.) are basically
baked into Rust
- the barrier between C and Rust is razor-thin, with almost zero overhead
cost (curse you, null-terminated strings!)
- the tooling is absolutely stellar
- string manipulation is sane (yeah, easy stab at C, but I had to do it)

What you get at the moment:
- few dependencies: you only need cargo, python3 and the libvarnish dev
files to get going
- write the same vmod.vcc as you would for C, the boilerplate is generated
for you
- building is literally just one command: `cargo build`
- automatic type translation so you can write pure rust, and never need to
see the `VCL` objects under their C form (
https://docs.rs/varnish/latest/varnish/vcl/convert/index.html)
- support for PRIVobjects (
https://github.com/gquintard/varnish-rs/blob/main/examples/vmod_object/vmod.vcc
and
https://github.com/gquintard/varnish-rs/blob/main/examples/vmod_object/src/lib.rs
)
- you can iterate through and modify the HTTP objects easily (
https://docs.rs/varnish/latest/varnish/vcl/ctx/struct.Ctx.html)

What you don't get for now:
- a comprehensive API, some bits are still missing, like
WS_ReserveAll/WS_Release, but because of the type conversion, you can
possibly live without it
- a stable API. I'm heavily working on it, and things will change twenty
times before the next Varnish release (but it doesn't matter as the crates
are version-locked)
- something based solely on the public library (libvarnishapi), I
reimplemented some stuff using private bits to avoid copies
- all the types. I only implemented a subset of VCL types, but fret not,
the others will come.
- a sensible boilerplate code generation, the current approach involves
embedding a python script that imports `vmodtool.py` and running it from
rust. But there's a plan to fix that

In case you want to try it, I've also created a minimal example repo that
you can play with:
git clone https://github.com/gquintard/vmod-rs-template.git
cd vmod-rs-template
cargo build
cargo test

I will continue working on expanding the API, documentation and will start
filing issues to come up with a roadmap. If you are interested in helping,
have questions or feedback, please do ping me, I'd love to hear from you.

Cheers!

-- 
Guillaume Quintard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20220130/cf5cbeab/attachment-0001.html>