Caching a low quantity of huge files with Varnish

[Laurence Rowe]

On 13 October 2011 18:20, Thiago Moraes <thiago at cmoraes.com> wrote:
> I posted this on serverfault.com and haven't received any answers.
> Would appreciate any help here:
>
> I have one server that provides access to some hundreds of files. All
> of them are really big (some have more than 10GB). These files won't
> change and are read only, but I'll need to provide access to them via
> a WAN.
>
> I want to make the access faster by using a reverse proxy server
> running near my users. For example:
>
> -User X wants to access something on server A.
> -User X access a reverse proxy server on his LAN which causes a cache
> miss. The file is downloaded to this proxy server.
> -The next time user X wants the same file, he doesn't need to go get
> it in my main server.
>
> I know that this situation is trivial, and I won't have a lot of
> traffic, but my problem is more related to the file sizes themselves.
> I read something about Squid having problems caching files larger than
> 2GB. Does Varnish face the same kind of problem?
>
> Thank you for any thoughts on that.

I don't know of any specific size limits, but you will almost
certainly need the streaming support currently being developed at
https://github.com/mbgrydeland/varnish-cache-streaming. Without
streaming support the Varnish proxy will wait to download the entire
10GB file before returning any bytes downstream.

Laurence