Is Varnish useless with slow/remote back-ends & large files?

[Caunter, Stefan]

Sorry for the top post (outlook at work). Sounds like you need to prime the cache with a script so your clients don't get "first request" syndrome. 

 

Stefan Caunter :: Senior Systems Administrator :: TOPS

e: scaunter at topscms.com  ::  m: (416) 561-4871

www.thestar.com www.topscms.com

 

From: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] On Behalf Of Bjarni Rúnar Einarsson
Sent: November-09-10 2:23 PM
To: varnish-misc at varnish-cache.org
Subject: Is Varnish useless with slow/remote back-ends & large files?

 

Hi!

I am attempting to use Varnish in a somewhat unusual fashion, and before I give up and drop it for something else, I figured I'd describe my problem here and see if there's something I've missed.

What I am doing is implementing a service (http://pagekite.net/) to provide a "remote HTTP front-end" for people who want to run webservers off a personal computing device (possibly even a mobile device). I've got a nice proxy/tunnel solution up and running, and want to add some caching in the front-end to lighten the load on the back-end server and more importantly the tunnel to the back-end, which may be quite bandwidth constrained.

At the moment it seems Varnish is effectively useless for this, because it can't both cache and stream at the same time - it does one or the other, right?

This means when a client downloads a large file, things just hang while the whole thing gets transferred (over a slow link) to Varnish, likely even timing out. I can't automatically stream it anyway, as the only trick I've seen to switch to "pipe" mode for large content is to "restart" based on the content-length header, which means the object will be sent twice - doubling the load on exactly the link I'm trying to spare. The closest I can get is to automatically switch to "pipe" for certain paths based on URL regular expressions, which avoids the sloth and timeouts but provides no caching benefits and will still do the wrong thing for some large files.

Varnish is quite an awesome piece of software, but I'm getting the feeling that it just wasn't designed for my (admittedly unusual) environment. I'd love to be proven wrong!

Thanks... :-)

-- 
Bjarni R. Einarsson
The Beanstalks Project ehf.

Making personal web-pages fly: http://pagekite.net/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20101110/2b4c5d4f/attachment-0003.html>