HSH_Private & expire superseded

[Poul-Henning Kamp]

Nils and I discussed this briefly on IRC yesterday, and I promised to
ponder the issue some more.  This is the result of said ponderage:

I think HSH_Private() is here to stay, for a host of reasons, but
Nils point about being able to pull stuff from the hash-lists with
just one lock-operation is both valid and pertinent.

We already have some other troubles in that space, specifically that
the EXPire thread cannot keep up etc.

So bearing in mind that I have just come up with this in the shower
and not built a prototype yet, here is my current idea:

	We instantiate an expiry-thread per thread-pool.  This moves
	us a little bit closer to NUMA and gives us more expiry
	bandwidth.

	Each EXP instance will have a "you should look at this
	object" inbox-list, and whenever we add or yank something
	to a hash-list or when we change the obj.keep value, we
	append the objcore to the thread-pools EXP-inbox.

	The EXP thread will sort through its inbox, keeps the
	binheap updated and disposes of the objcores which died.

The major benefit of this scheme, is that we have someplace to dump
unwanted objects when we are in a hurry in HSH_Lookup(), and somebody
else will clean them up out of the critical path.

Yanking an object from a live hash-list will look something like:

	(we have the oh->mtx locked)
	...
	VTAILQ_REMOVE(...)
	oc->objhead = NULL;
	EXP_Feed(oc);
		lock(EXP-mtx)
		if (!(oc->flags & OC_F_EXP_QUEUED) {
			VTAILQ_INSERT_TAIL(EXP->queue, oc, exp_list)
			oc->flags |= OC_F_EXP_QUEUED;
		}
		unlock(EXP-mtx)
	...

It may also be beneficial to push LRU updates to the EXP thread,
since it could batch process all EXP updates in the inbox and move
it out of the critical path.

Comments, critique etc. welcome...


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.