Are ESI requests limited to originating backend?

[Lars Kellogg-Stedman]

> Your vcl_recv{} is responsible for setting the right backend based on
> the req.http.host header.

Is there any way to explicitly distinguish between an ESI request and
a request from an outside client?  Clients can provide arbitrary Host:
headers -- so that's not really an option --  and it looks like for an
ESI request that client.ip is still set to the address of the
originating client.  Or is the answer, "don't do that"?

Thanks,

-- Lars