Caching a low quantity of huge files with Varnish

[Kristian Lyngstol]

Greetings,

On Thu, Oct 13, 2011 at 02:20:48PM -0300, Thiago Moraes wrote:
> I have one server that provides access to some hundreds of files. All
> of them are really big (some have more than 10GB). These files won't
> change and are read only, but I'll need to provide access to them via
> a WAN.
> 
> I want to make the access faster by using a reverse proxy server
> running near my users. For example:
> 
> -User X wants to access something on server A.
> -User X access a reverse proxy server on his LAN which causes a cache
> miss. The file is downloaded to this proxy server.
> -The next time user X wants the same file, he doesn't need to go get
> it in my main server.

My first thought is why you need a reverse proxy and not a regular one.

> I read something about Squid having problems caching files larger than
> 2GB. Does Varnish face the same kind of problem?

Short answer: No.

From Varnish 3, we do regular tests to see that files bigger than 4GB
are dealt with sensibly. 2/4GB being the barrier, and beyond that, the
logical limit is your memory manager/hardware/network/client, etc. 10GB
/shouldn't/ be a big problem.

But keep in mind:

- You need a 64bit system.
- Use the latest Varnish version. 3.0.1 or even 3.0.2 when that
  finishes. 2.1 is not recommended.
- If your users will access these files concurrently, you will really
  want to try out Martin's streaming branch, as that will ensure that
  multiple clients get the content as it arrives at the cache. Without
  Martin's branch, the first user that gets a cache miss will get the
  content "streamed" while Varnish downloads it, while other users have
  to wait for the download to finish before they get any data.
- If your data is compressible, make sure you use gzip. Even if your
  clients don't support it (e.g: scripts?), Varnish does. If it isn't
  compressible, consider disabling it.
- Make sure the cache is big enough, or that evictions are
  well-controlled. If you're mixing small and large files, you might end
  up evicting hundreds of small files to fit room for a single large
  one.

With all of that said, I'd like to emphasis that support for large
files is relatively new. I expect there's room for optimizations.

The less used features are also the less optimized features. As a
consequence of the lack of feedback.

I would greatly appreciate any feedback you have if/when you test this.

- Kristian

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: Digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20111014/c886b9cc/attachment-0003.pgp>