Varnish use for purely binary files

[Michael S. Fischer]

On Jan 18, 2010, at 12:58 PM, pub crawler wrote:

> This is an inquiry for the Varnish community.
> 
> Wondering how many folks are using Varnish purely for binary storage
> and caching (graphic files, archives, audio files, video files, etc.)?
> 
> Interested specifically in large Varnish installations with either
> high number of files or where files are large in size.
> 
> Can anyone out there using Varnish for such care to say they are?

I guess it depends on your precise configuration.

Most kernels cache recently-accessed files in RAM, and so common web servers such as Apache can already serve up static objects very quickly if they are located in the buffer cache.  (Varnish's apparent speed is largely based on the same phenomenon.)  If the data is already cached in the origin server's buffer caches, then interposing an additional caching layer may actually be somewhat harmful because it will add some additional latency.

If you've evenly distributed your objects among a number of origin servers, assuming they do nothing but serve up these static objects, and the origin servers have a sum total of RAM larger than your caching servers, then you might be better off just serving directly from the origin servers.

On the other hand, there are some use cases, such as edge-caching, where interposing a caching layer can be quite helpful even if the origin servers are fast, because making the object available closer to the requestor can conserve network latency.  (In fact, overcommit may be OK in this situation if the I/O queue depth is reasonably shallow if you can guarantee that any additional I/O overhead is less than network latency incurred by having to go to the origin server.)

--Michael