push button lru nuking

[David Birdsong]

On Sat, Jan 16, 2010 at 4:27 PM, Michael Fischer <michael at dynamine.net> wrote:
> On Sat, Jan 16, 2010 at 4:16 PM, Michael Fischer <michael at dynamine.net>
> wrote:
>>
>> On Sat, Jan 16, 2010 at 4:10 PM, David Birdsong <david.birdsong at gmail.com>
>> wrote:
>>>
>>> On Sat, Jan 16, 2010 at 3:55 PM, Michael Fischer <michael at dynamine.net>
>>> wrote:
>>> > This scheme seems very baroque.  Why not just reduce the size of your
>>> > caches
>>> > so you don't page-thrash and let Varnish's builtin LRU algorithm handle
>>> > the
>>> > eviction?
>>>
>>> Then I wont be able to cache nearly as much.  I want to originate as
>>> much content as possible on the varnish servers ie. reduce backend
>>> fetches.  There is no way I could fit any useful amount of my working
>>> set into a storage that could handle the evictions without spending an
>>> unreasonable amount of money (basically fit it in RAM.)  -I'd love to
>>> be proven wrong though.  As far as random reads go, the SSD's are
>>> really good; it's just the writes that kill me.
>>>
>>> Right now a mostly filled cache server with ~80-160GB allocated can
>>> maintain between 90-92% cache hit ratio at 400-500Mb/sec.  When it
>>> fills up completely eviction cause the machine to keel over, parent
>>> can't ping the child, health checks fail -general badness.  I'd like
>>> to let the eviction run under supervision (automated supervision) and
>>> augment the eviction such that it buys back a few hours not minutes.
>>
>> What OS are you running?  This might be one of those rare cases where a
>> little more "swappiness" (i.e., aggressiveness of the pageout algorithm)
>> might buy you something.
>
> This page may be useful if you're running on Linux:
> http://www.westnet.com/~gsmith/content/linux-pdflush.htm
> --Michael

yes, this page was very helpful back when I had hopes of tuning my way
around this load problem.