Is Varnish useless with slow/remote back-ends & large files?

[Bjarni Rúnar Einarsson]

Hi!

I am attempting to use Varnish in a somewhat unusual fashion, and before I
give up and drop it for something else, I figured I'd describe my problem
here and see if there's something I've missed.

What I am doing is implementing a service (http://pagekite.net/) to provide
a "remote HTTP front-end" for people who want to run webservers off a
personal computing device (possibly even a mobile device). I've got a nice
proxy/tunnel solution up and running, and want to add some caching in the
front-end to lighten the load on the back-end server and more importantly
the tunnel to the back-end, which may be quite bandwidth constrained.

At the moment it seems Varnish is effectively useless for this, because it
can't both cache and stream at the same time - it does one or the other,
right?

This means when a client downloads a large file, things just hang while the
whole thing gets transferred (over a slow link) to Varnish, likely even
timing out. I can't automatically stream it anyway, as the only trick I've
seen to switch to "pipe" mode for large content is to "restart" based on the
content-length header, which means the object will be sent twice - doubling
the load on exactly the link I'm trying to spare. The closest I can get is
to automatically switch to "pipe" for certain paths based on URL regular
expressions, which avoids the sloth and timeouts but provides no caching
benefits and will still do the wrong thing for some large files.

Varnish is quite an awesome piece of software, but I'm getting the feeling
that it just wasn't designed for my (admittedly unusual) environment. I'd
love to be proven wrong!

Thanks... :-)

-- 
Bjarni R. Einarsson
The Beanstalks Project ehf.

Making personal web-pages fly: http://pagekite.net/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.varnish-cache.org/pipermail/varnish-misc/attachments/20101109/cb41b973/attachment.html>