Composable directors proposal

[Dridi Boukelmoune]

Hi,

TL;DR today you can't write a custom director and reuse an existing
selection strategy, and I would like to make it possible.

I have been pondering this for quite a while now, and I thought it was
time I'd share my idea on the mailing list.

As of today, it is not possible to compose directors. The closest
thing to composition is simply stacking them. It was great progress
from 3.0 to 4.0, but I think it can be pushed one step further.

When I talk about composition, I sort of think about the functional
programming-ish higher-order-functions way of doing composition.
I would like to separate director selection strategies from their
creations.

One could refactor all the existing built-in directors as a single
one: cluster. A vmod writer could then use an enumeration to pick
a strategy:

   new rr = directors.cluster(round_robin);

It also means that weight has to become a property of the (added)
director. It could be a property on a vcl backend definition, and a
method on vmod directors. Or it could also be set to any director with
the std (or directors?) module:

   directors.set_weight(rr.bacvkend(), 2);

It also means that a backend/director cannot belong to 2 different
directors with different weights. But is that a great loss?

So with this single "cluster" director, we could still compose it with
different strategies and achieve similar directors stacking to what
we can do today:

   backend www1 {
      [...]
   }

   [...]

   sub vcl_init {
      new active = directors.cluster(round_robin);
      active.add_backend(www1);
      [...]
      new passive = directors.cluster(round_robin);
      [...]
      new www = directos.cluster(fallback);
      www.add_backend(active);
      www.add_backend(passive);
   }

Now what if we name this "cluster" director "static" instead? After all,
once created, it doesn't change. What if we had dynamic backends
and directors? They could reuse the same strategies.

What if we have a DNS director? Or any other director creating
backends using whatever discovery/registry tool people use (etcd,
zookeeper, eureka, ...). They can go round robin, or even do sticky
sessions out of the box.

Also, if Varnish knows the relationship between directors, it can
maybe prevent cycles like this:

   import directors;

   sub vcl_init {
      new d1 = directors.round_robin();
      d1.add_backend(d1.backend());
   }

For this we'd need to:
- move the backend selection code to VRT
- rewrite the directors vmod

And if we have dynamic backends:
- support backend updates in the selection code
- write new built-in dns directors

Please note that it doesn't mean having to once again break everyone's
VCL making use of directors. We could keep the existing objects,
acting as aliases/wrappers to the new cluster director (for a while).

Comments? Ideas?

Best Regards,
Dridi