[PATCH] Remove the waiter interface from cache.h, split, waited/waitfor, session: struct waited on ws

[Poul-Henning Kamp]

--------
In message <55675A15.5090504 at schokola.de>, Nils Goroll writes:

So before I reply in detail, let me do a quick sketch of what the heck
I have been hacking the last couple of weeks.

I was taken quite by surprise when various people started talking about
backends in multiple of 100k, not *backend connections* but *backends*.

In a number of places our code doesn't scale at that level, for instance
we have a thread per backend for healthchecks and just weeks before I had
added an entire waiter per backend-tcp-pool.

Multiplied by 100k that simply doesn't scale.

So instead I went the other way.

Waiters are now a general purpose event-driver for filedescriptors,
and we have a single waiter per thread-pool, which is used both for
client connections and backend connections.

Next step was to add a global "wait on this in any suitable thread-pool"
API, for things which do not have pool-affinity[1].

After that I want to change the backend health-polling to be event
driven, so that a single global thread kicks the polls off and the
waiters finish them using regular worker threads (still thinking
this through).  In particular I want the tcp connect(2) to be
asynchronous because the healthpolls will see into network problems
(in order that backend fetches should not).

With that background:

>* We should not include waiter.h from cache.h as vmods requiring cache.h
>  should not need the waiter interface (otherwise we'd need to install waiter.h)

I think I may have added waiter.h the wrong place in Makefile.am.

The waiter interface is/can/will be useful for VMODS, as it allows them
to do event-driven processing, so it should not be hidden from them.

>* For sessions, temporarily alloc struct waited on the ws - we don't need
>  the workspace while we are waiting, and we don't need struct waited unless
>  we are waiting

Good idea.

>* Add Waiter_Done to invalidate struct waited and change the signature of
>  Wait_Call / waiter_handle_f to struct waited ** to allow for invalidation
>  of the struct waited pointer.

Not so sure about this one, seems a tad strict to me.  (It's obviously
a good idea to clear it if/when it has no permanence (ie: on ws), but
I fail to see a particular reason to mandate clearing it ?

Poul-Henning


[1] This is a NUMA thing:  The client fd's we have accepted should and
in the future undoubtedly will have NUMA pool affinity.  Backend connections
and even less so backend health poll connections will not, so we need
to distribute those over the thread pools[2] in some fair fashion.

[2] I've been really tempted to rename to concept of thread pools
to "clans", because the NUMA aspect is about so much more than just
the threads.  Clans would have a (just a) pool of threads,
acceptor sockets, waiter, etc.  I may do this for 5.0.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.