backend/director admin states

[Nils Goroll]

this is a follow up to a discussion between phk and myself on #varnish-hacking
today:

phk is in the process of restructuring director/backend APIs. One of the areas
currently reconsidered is the admin health status. As of now
(68a78f94b3a4e702edf73dea4e11e626c4c20301), the admin health has become private
to varnishd (off limits to vmods) and semantics have changed.

We discussed which admin health states should exist and their semantics. Notice
that for the "probe" admin health, the actual health state is determined by the
probe.

Here's my proposal:

* auto

  this state is the default and only exists as an argument to varnishadm
  backend.set_health. It resolves to either

  - "probe"	if the backend/director has a probe
  - "dependent"	if the director has no probe, but a healthy callback
  - "healthy"	otherwise (no probe and no healthy callback)

* healthy

  The backend/director is considered unconditionally healthy. Connections will
  be attempted.

  This state is used for backends without a probe and can be manually set
  using varnishadm backend.set_health and from vmods.

* sick

  The backend/director is considered unconditionally sick. Connections will not
  be attempted.

  Can be manually set using varnishadm backend.set_health and from vmods.

* disabled

  The health state is determined as for "probe" or "healthy" but a newly added
  VRT / vmod_std function (e.g. backend_disabled()) returns true. Directors may
  offer the choice to use disabled backends or not.

  Can be manually set using varnishadm backend.set_health and from vmods.

  The use case is to flag a backend for plannend maintenance such that it is
  used for existing sessions, but not for new ones.

* probe

  The health state is determined by a probe, yet connections can still be
  attempted even if the probe is unsuccessful and directors may offer the choice
  to still hand out unhealthy backends.

  This state can not be set explicitly, but is selected through
  varnishadm backend.set_health auto or from vmods.

* dependent

  The health state of the director is determined by the backends it refers to.

  This state can not be set explicitly, but is selected through
  varnishadm backend.set_health auto or from vmods.

* deleted

  The backend has been deleted (via a vmod) will be removed once all references
  have been cleared.



Illustrated example: varnishadm backend.list output

Backend name                  Admin     Probe      Last change
boot.b1                       probe     0/8 bad    Wed, 02 May 2018 13:22:10 GMT
boot.b2                       probe     8/8 good   Wed, 02 May 2018 13:22:10 GMT
boot.b3                       healthy   1/8 bad    Wed, 02 May 2018 13:22:10 GMT
boot.b4                       healthy   -          Wed, 02 May 2018 13:22:10 GMT
boot.b5                       sick      7/8 good   Wed, 02 May 2018 13:22:10 GMT
boot.b6                       sick      -          Wed, 02 May 2018 13:22:10 GMT
boot.b7                       disabled  7/8 good   Wed, 02 May 2018 13:22:10 GMT
boot.b8                       deleted   -          Wed, 02 May 2018 13:22:10 GMT
boot.d1                       dependent -	   Wed, 02 May 2018 13:22:10 GMT
boot.d2                       probe	3/4 good   Wed, 02 May 2018 13:22:10 GMT

in detail:

* boot.b1                       probe     0/8 bad

backend with a probe for which all recent health checks have failed

result from set_health=auto or default

* boot.b2                       probe     8/8 good

backend with a probe for which all recent health checks have succeeded

result from set_health=auto or default

* boot.b3                       healthy   1/8 bad

backend with a probe for which most recent health checks have failed

result from set_health=healthy

* boot.b4                       healthy   -

backend with no probe

result from set_health=healthy, set_health=auto or default

* boot.b5                       sick      7/8 good

backend with a probe for which most recent health checks have succeeded

result from set_health=sick

* boot.b6                       sick      -

backend with no probe

result from set_health=sick

* boot.b7                       disabled  7/8 good

backend with a probe for which most recent health checks have succeeded

result from set_health=disabled

custom vcl logic will not use this backend for new sessions

* boot.b8                       deleted   -

backend has been removed (by a vmod)

* boot.d1                       dependent -

director without a probe

result from set_health=auto or default

* boot.d2                       probe	3/4 good

director with a probe, which returns 3/4 success (may be the number of healthy
backends)

result from set_health=auto or default