multi-terabyte caching

[David Birdsong]

On Fri, Nov 20, 2009 at 2:19 PM, Eric Bowman <ebowman at boboco.ie> wrote:
> Hi,
>
> Apologies if this has been hashed out before.  I did some googling, and
> read the faq, but I could have been more thorough... ;)
>
> I'm considering using Varnish to handle caching for a mapping
> application.  After reading
> http://varnish.projects.linpro.no/wiki/ArchitectNotes, it seems like
> Varnish is maybe not a good choice for this.  In short I need to cache
> something like 500,000,000 files that take up about 2TB of storage.
>
> Using more 1975 technologies, one of the challenges has been how to
> distribute these across the file system without putting too many files
> per directory.  We have a solution we kind of like, and there are others
> out there.
>
> My impression is that we would start to put a big strain on Varnish and
> the OS using it in the standard way.  But maybe I'm wrong.  Or, is there
> a way to plugin a backend to manage this storage, without getting into
> the vm-thrash from which Squid suffers?
>
> Thanks for any advice -- Varnish gets such good press I'd really love if
> it were straightforward to use it in this case.
>
> -Eric
a straight forward way to store an unlimited amount of data is to find
the optimal cache storage capacity per varnish instance then:

optimal_size  / working_set = N

where N is the number of varnish instances you need to run.

then put a layer 7 switch in front of the pool of varnish instances,
hashing on the requests.

works like a charm.

finding optimal storage amount per varnish requires turning the knobs:
 - tuning VM
 - tuning kernel for high network traffic
 - balancing between big and fast storage medium
    random reads will skyrocket, minimize writing to storage while
serving if possible (pregenerate your working set, dont let anything
expire between generating )
 ..and test

> Eric Bowman
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>