Varnish minus VCL?

[Luke Macpherson]

I'm still kicking around ideas here. I'm trying to think of a solution
which is workable in the presence of larger numbers of backends,
without sacrificing performance or configurability of smaller setups.
I imagine there are others running into the same problems.

There are, I think, two main problems to solve:
1) Managing large numbers of dynamically changing backends.
2) Improving performance of request to service mapping when the number of
    services is large. (Or is there a nice way to do it I'm not aware of?)

One option would be to introduce another shared-memory area for use
between a management daemon and varnishd. I imagine this area would
contain data-structures to be updated by the management daemon
periodically, and read by varnishd when processing requests.

A reader-writer locking scheme should give reasonable performance,
since writes to the structure by the management daemon would be very
infrequent.

Here is how I think it should work:
* shared hashtable of services
* each service has a list of backends (and assigned weights, strategy
params etc)
* default service or 4xx/5xx configurable for hashtable misses

Instead of deciding on a backend name, the VCL code should decide on a
service name. It should be able to either specify a literal name, or
generate a name string based on the Host/URL/other parameters.

For example: sub vcl_recv { req.service = req.http.host; } would give
good performance on large service maps, without precluding long lists
of regular expressions for those that really need them. (is this legal
VCL?)

Initial values for the shared data-structures could be set by the VCL
file to enable varnishd to operate as usual if an external management
daemon is not present.

I guess another (somewhat hacky) solution would be to continually
generate and update VCL on the fly.

Does anyone have further thoughts on this?

On 9/25/07, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
> In message <29f77e8a0709241712k2092c712u6fa3e7844f7ea52 at mail.gmail.com>, "Luke
> Macpherson" writes:
> >I am looking to deploy varnish in a configuration which has a large
> >number of backends, and integrate it with our existing management
> >infrastructure. I'd like to solicit your opinions on the best way to
> >do that.
> >
> >What I'm thinking of doing is using replacing the compiled VCL
> >configuration object with a custom-written object. This C library
> >could talk to our existing infrastructure to keep track of live
> >backends and choose the appropriate one at run-time based on
> >availability, load and other factors.
>
> VCL has a mechanism for including C language, so in theory you can
> do that from a VCL program.
>
> >Is this a reasonable approach? Is the VCL api stable and well-defined
> >enough? Is there a better place to plug-in this kind of functionality?
>
> VCL doesn't guarantee stable internal APIs, that's one of the reasons
> why we go to such great lengths to make it one single executable that
> contains everything.
>
> That said, I don't see anything very hard or impossible in what
> you are proposing, but it takes a good programmer to get it right.
>
> --
> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
> phk at FreeBSD.ORG         | TCP/IP since RFC 956
> FreeBSD committer       | BSD since 4.3-tahoe
> Never attribute to malice what can adequately be explained by incompetence.
>