Value of builtin.vcl - vcl_recv on modern websites

[Geoff Simmons]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 07/12/2017 09:14 AM, Matthew Johnson wrote:
> 
> Since Varnish 1.0 the builtin (or default) vcl_recv has had this
> statement: if (req.http.Authorization || req.http.Cookie) {
> 
> /* Not cacheable by default */
> 
> return (pass);
> 
> }
> 
> My issue with this is the req.http.Cookie check, In any modern
> website cookies are always present.

While I'm sympathetic to all the things you said and don't mean to
disregard it but cutting things short, the answer here is simple:
there is no other choice for the default configuration of a caching
proxy, because those two headers mean that the response may be
personalized.

Many uses of cookies don't have that effect, of course, but Varnish
has no way of knowing that. As bad as the effect of the default config
may seem on sites that use cookies on every request -- Varnish doesn't
cache anything -- it would be much worse if someone sets up Varnish,
doesn't think of the consequences of not changing the default
configuration, and you end up seeing someone else's personal data in
cached responses.

This problem is not specific to Varnish, but to any server that tries
to do what Varnish tries to do. I know from experience that it's
generally futile to say so, but this situation really ought to lead to
some widespread re-thinking throughout the industry. Forgive me for
shouting, soapbox-style, but this gives me an opportunity to sound off
on a pet peeve:

==> Maybe modern web site SHOULD NOT use cookies on every request!
Because of the way cookies interfere with downstream caching.

I have come to the conviction that many uses of cookies are a result
of lazy thinking in app development. Many PHP devs, for example, are
in the habit of saying session_start(); at the beginning of every
script, without thinking twice about whether they really need it. I
have seen uses of cookies where "just toss that thing into a cookie"
was evidently the easy decision to make. I have seen cookies with
values that are 3KB long.

(Sometime over beer I'll tell you about that little database that
someone wanted to transport over a cookie, a base64-encoded CSV file
whose data was *also* base64-encoded, leading to a doubly
base64-encoded cookie value, in every request.)

This is an instance of an issue that you encounter a lot with the use
of Varnish in practice: app development that doesn't think outside of
its own box in terms of functionality and performance. Rather than
thinking about the benefits of handing off some of your work, by
letting someone else serve your cached responses for you.

HTTP was conceived from the beginning to enable caching as a means of
solving performance problems in slow networks. A well-configured
deployment of Varnish shows how beneficial that can be. But the
universal and unreflected use of cookies is one of the forces
presently at work that actively undermine that part of the equation.

> A classic example of this is someone adding javascript via Google
> Tag Manager which then sets a cookie.

One might have hoped that the Googlers, of all people, would have more
awareness of the trouble that they could cause by doing that.

> Do you still recommend configurations fall through to the
> underlying vcl_recv logic?
> 
> Options i can think of:

In a project where I am able to work with the app devs, I have had
good experience with working out a policy with them: if you MUST have
cookies in every request (although I WISH YOU WOULD RECONSIDER THAT),
then the caching proxy cannot make caching decisions on your behalf.
Only you can know if your response is cacheable, despite the presence
of cookie foo or bar, but is not cacheable if the cookie is baz.

So if you want your response to be cached, you MUST say so in a
Cache-Control header. The proxy will not cache any other responses.

Then we write VCL to bypass builtin's vcl_recv, and start Varnish with
- -t 0 (default TTL is 0s). Responses are then cached only if they
announce that they are cacheable.

Of course, this has the effect that you're lamenting -- Varnish
doesn't cache anything by default -- but in my experience, the result
is that devs have become very good at thinking about the cacheability
of their responses.

That boils down to answering your question by saying no, you can't use
builtin vcl_recv in a situation like that. When the cookies, like the
Evil, are always and everywhere (to paraphrase a saying in Germany),
and some cookies lead to cacheable responses while others don't, then
there's no other option for a caching proxy.


Best,
Geoff
- -- 
** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstraße 32
22301 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753

http://uplex.de
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJZZen0AAoJEOUwvh9pJNURnWEQAK9ucYSXEcrEwbmOrroBWoGK
iR9a8OFhst1rVKFQ2vNTUpw+OYM8vf8SJYToyq2VxG5/f/uGsT6nSVRWGKThgeV1
geyyUQfbbDte1at4aFy6HX6LeCt62Si0L9KUZMZMkI5C6m6FKgA/5HecUchhuXdP
Li17DXKvzrQyTBJpvbk2vBZGPVlErnVAUz75IeJMrD6t/WGO0PsZvkC/l8LZhqcD
2S2R9SHYMIyBrWSZm+YsI1DxMwvH6Gt84NRPpKcHQQ7TKEfvtOq0NwoqcNOB26EL
KIfOuVJdbiMvD5D+BZud/7a7UzSpJz5klLMdTcJMN60MrHJjGcok/5KiG7TNNowj
hMy5YUYpuIybsWzcvB5Ie/nteb0WyXt5+LYkxjP9dbN7AN3k+aU1PboSOyqYXO3u
KK1al00LMKHfzMHs1vF3QHRt2Q1Udud6dCdHuw6TyJ7eWCc9YGgU8NyboMLkXBhO
fBVNUQQjfNjaDWhKFvUIMEsGZhgvzzuMvjlZNhkc/lcDLmU8wVXyiMFSoBcuR1sX
7EM1wBUKcKix0wE4QPl9608ql/5LF3Ms+wqDpmS0ECgFIf1yKMWFZt9iHhMUbch2
7fhr77vVjVD1K6nKHqDGOuLp4Cq+lfBJkd7PX2huQUV/hc00C8+NEieD77wuAwk7
OPiGNt5YqmDNjtZmFUVH
=BlAn
-----END PGP SIGNATURE-----