Varnish use for purely binary files

[pub crawler]

> Most kernels cache recently-accessed files in RAM, and so common web servers such as Apache can ?>already serve up static objects very quickly if they are located in the buffer cache.  (Varnish's apparent >speed is largely based on the same phenomenon.)  If the data is already cached in the origin server's buffer >caches, then interposing an additional caching layer may actually be somewhat harmful because it will add >some additional latency.

So far Varnish is performing very well for us as a web server of these
cached objects.   The connection time for an item out of Varnish is
noticeably faster than with web servers we have used - even where the
items have been cached.  We are mostly using 3rd party tools like
webpagetest.org to look at the item times.

> If you've evenly distributed your objects among a number of origin servers, assuming they do nothing but >serve up these static objects, and the origin servers have a sum total of RAM larger than your caching >servers, then you might be better off just serving directly from the origin servers.

Varnish is good as a slice in a few different place in a cluster and a
few more when running distributed geographic clusters.   Aside from
Nginx or something highly optimized I am fairly certain Varnish
provides faster serving of cached objects as an out of the box default
experience.  I'll eventually find some time to test it in our
environment against web servers we use.

> On the other hand, there are some use cases, such as edge-caching, where interposing a caching layer >can be quite helpful even if the origin servers are fast, because making the object available closer to the

Edge caching and distributed cache front ends are exactly what's
needed.  It's a poor mans CDN but can be very effective if done well.

The question I posed is to see if this type of use (binary almost
purely) is being done and scaling well at large scale (50GB and
beyond).  Binary data usually poses more overhead as the data is
larger - less stored elements in RAM, often it can't be compressed
further, more FIFO type of purging due to this, etc.

-Paul