Varnish

[Jeff]

Hi,

We would like to try Varnish with an eye towards putting it in our 
production environment; we've pushed Squid about as far as it can go 
performance-wise, and we'd have better luck waiting for Godot than 
proper HTTP 1.1 support.

We currently use squid to reverse-proxy for a large number of (mostly) 
small sites.  I've been working with Varnish 1.1.1 and it's passed my 
basic "does it work?" tests, but I've come up with a list of how-to 
questions that are between us and a full-scale trial deployment.

1) The "Guru Meditation" error messages, while very Amiga-nostalgic, 
aren't customer suitable, but appear to be hard-coded in 
cache_synthetic.c.  If we want nice, pretty error messages, are we 
basically on our own, or is there an imminent plan for this?

2) Since we have an extremely large hostname->backend map, we need to 
choose the right one efficiently and dynamically on a per connection 
basis; we cannot statically configure every possibility into a VCL file. 
  How can we tie some sort of external lookup (pretty much any sort will 
do) into VCL?

3) The log files look like they are great for debugging obscure 
complicated problems, but for day-to-day usage, we need something 
similar to the squid_access format (timestamp, client IP, URL, status 
code, fetch/cache status, bytes).  How would we approach this?

4) We would like to limit the number of simultaneous open connections 
from a single client IP to 10-16 or so to thwart certain types of 
malicious crawlers that open them by the dozens, and kick back a 403 
error to extra ones.  Is this possible with Varnish?

5) We need to If-Modified-Since: revalidate back to the origin server on 
every request, even if 99% of the time it gets a 304 response, in order 
to get log files on the back end that awstats can parse.  However, we 
want to preserve Expires: and max-age values to pass along to the 
client, so something as heavy-handed as setting the max TTL to 0 
probably would not work.  I think this can be done in VCL, I just can't 
seem to wrap my head around it.  What would be the best way for us to 
handle that?

I've been looking at the documentation and source, and will continue to 
do so, but if anyone can point me in the right direction on any of these 
issues, it would be very much appreciated.  Varnish is incredibly cool, 
and it's designed right; I would love to see it working on our network.

Thanks for any advice!

Jeff