Value of builtin.vcl - vcl_recv on modern websites

[Dridi Boukelmoune]

On Wed, Jul 12, 2017 at 9:14 AM, Matthew Johnson <matt at section.io> wrote:
> Hi There,
>
> This may be a biblical / philosophical question but when teaching people
> Varnish ive been struggling to justify leveraging the builtin.vcl - Mainly
> vcl_recv in builtin.vcl
>
> Since Varnish 1.0 the builtin (or default) vcl_recv has had this statement:
> if (req.http.Authorization || req.http.Cookie) {
>
> /* Not cacheable by default */
>
> return (pass);
>
> }

I can see a couple reasons:

- HTTP (in general) is poorly designed
- Cookie don't integrate well with other HTTP mechanisms
- Web applications are often poorly implemented

Not trying to be offensive, just factual. I could pinpoint the problems
in HTTP, but I already have a blog post [1] and an unfinished draft
for that. Not in the mood to duplicate that effort here, I don't really
enjoy digging in that area.

> My issue with this is the req.http.Cookie check, In any modern website
> cookies are always present.
>
> This means that out of the box Varnish doesnt cache any content, which may
> be an ok safe starting point but without the cookie check the default
> implementation of most webservers/applications would cache static content
> and not cache on dynamic content (HTML) - via cache-control / set-cookie
> response headers etc - A good outcome.

The problem is that we've often seen inconsistent cache directives,
untrustworthy backends. You could very well cache dynamic contents,
there's no point in using Varnish if you only cache static resources.
The problem is when a backend sends user-specific contents and
doesn't say so (Vary header) then you risk an information leak.

Varnish doesn't cache by default for this reason.

> Aside from the above poor caching start point (which could be ok from a low
> risk kickoff perspective), biggest issue with the cookie check is that it
> forces the user into a cookie management strategy.

If your backend speaks HTTP fluently and provides correct support for
cookies and caching, the aforementioned blog post [1] gives you a
simple solution to deal with that in pure VCL.

<snip>

> Do you still recommend configurations fall through to the underlying
> vcl_recv logic?
>
> Options i can think of:
> 1) Build lots of cookie whitelist/blacklist functionality above builtin.vcl
> so the underlying logic doesnt break things

This can be avoided with the blog post [1] trick if you are confident
your backend won't make Varnish leak information.

> 2) Remove cookies entirely for some request types (such as static objects)
> so the underlying logic always works for some content types - My experience
> is that this generates issues on some customers sites as they have static
> handlers that are looking for a cookie in the origin and then do
> redirects/change response content if no cookies are present.

So now you need to make assumptions about resources not managed by
Varnish. A common pattern is to remove cookies when the path
terminates with a "static resource" file extension. And then you run
into applications that generate images on the fly and need cookies.

> 3) Explicitly handle all scenarios and return(pass) or return(hash) to avoid
> vcl_recv in builtin.vcl  (and lift up the good bits of vcl_recv into the
> main config)

While I lean towards composing on top of the built-in, I don't mind
this approach.

> Interested in your views, As i work on internet facing websites - I would
> have thought this was the most common scenario but maybe there are more
> users doing other things with Varnish or i'm missing something simple in
> terms of handling cookies?

It can be simple only if your backend is reliable when it comes to
cookie handling and caching. Otherwise you're in for a VCL soup of
cookie filtering...

Dridi

[1] https://info.varnish-software.com/blog/yet-another-post-on-caching-vs-cookies