Varnish serving content from wrong backend

[Michael Alger]

On Tue, Oct 11, 2011 at 11:54:45AM -0300, Paulo Paracatu wrote:
> > Can you provide your VCL, or at least a representative sample of
> > how you're mapping your sites to backends?
> 
> Sure, here is it: http://pastebin.com/uvFKH4ub
> 
> > If you're relying on host headers, are you properly handling all
> > possibilities? e.g. some browsers might send the port number so
> > you'll get something like "Host: www.some.site:80" which maybe
> > you're not handling (expecting only "www.some.site") and therefore
> > it's going through to the default backend.
> 
> Oh... Makes sense. As you can see in my VCL example, I'm not
> expecting the ":80".  Do you think this may be the problem, so?

It's a possibility... I don't know offhand which browsers include the
port in the host header for standard ports. It might be a good idea to
use a regsub to delete a trailing :\d+ from the host header on every
request just to be sure (assuming the port isn't important to you).

Also I had another thought that you should check your vcl_hash to make
sure it's including any information you or your backends use to decide
which content to serve. For example if it wasn't incorporating the
host header into the hash (only the request path) then the first
resource with a particular path would be cached by Varnish, and then
served for any request with the same path, regardless of host.

Do your backends make use of the X-Sub or X-Host header you add to the
request? If so, you'd want to make sure they're incorporated into the
object's hash. If you're only using them temporarily to build
redirects etc. then that's occurring before the cache lookup, so
shouldn't be a concern.

If you're not already, it might also be worthwhile checking the
request actually includes the host header and displaying an error
message or something if it's missing. Probably unlikely to be the case
with any graphical browser in use today, but it never hurts to be
thorough.

Lastly, if you can't find a way to reproduce this on demand, then
getting logs from it may be difficult. It might be worthwhile to set
up a "dummy" backend as the first one Varnish loads, which should
never actually be used by any of your sites. That way if it does
somehow get used, you'll know the request somehow failed to get
assigned to a backend, and you can log the request (on the dummy
backend, even) - maybe something about it will stick out as the
obvious culprit. Plus, it'd avoid accidentally serving certain images
to unsuspecting users.