per-workpool exp threads

[Nils Goroll]

First a side note on the private_oh: Couldn't the code handle a second magic 
private_oh which demands "don't queue here" and is always empty? This way we 
would only need to take on the additional sync overhead when needed (for 
debugging). (side note on the side note: why is this worrying me so much? 
Probably a benchmark would help putting this into perspective...)

On 09/13/13 09:43 AM, Poul-Henning Kamp wrote:
> Yanking an object from a live hash-list will look something like:
>
> 	(we have the oh->mtx locked)
> 	...
> 	VTAILQ_REMOVE(...)
> 	oc->objhead = NULL;
> 	EXP_Feed(oc);
> 		lock(EXP-mtx)
> 		if (!(oc->flags&  OC_F_EXP_QUEUED) {
> 			VTAILQ_INSERT_TAIL(EXP->queue, oc, exp_list)
> 			oc->flags |= OC_F_EXP_QUEUED;
> 		}
> 		unlock(EXP-mtx)
> 	...

Hm, reflecting on this verbosely:

The per-workerpool exp thread could minimize the critical section on EXP-mtx by 
just taking the whole queue onto its own private head for processing, so there 
really should be virtually no contention on this lock.

Also I find it quite appealing to take all of the expire / free handling out of 
the delivery code path with the exp thread mailbox concept.

Could we hand off additional functions to async processing in the exp thread, like

- EXP_Rearm
- EXP_Insert
- all of HSH_Deref* which is not related to the oh
   - BAN_DestroyObj
   - freeing stuff

?

On the other handl I am not so sure if keeping objs on the exp->q until their 
refcount becomes 0 is the best solution. Maybe one could make them private 
(optionally lockless private as proposed above) and let Deref hand them over to Exp.

Other than that, yes, I do understand that LRU has to rip objects out of the oh, 
and there is the special case that LRU needs to make room "now". Do I understand 
correctly that there currently is no "proactive LRU nuke" (trying keep free 
space above thresholt per stevedore)?

Thanks, Nils