varnish-misc Digest, Vol 136, Issue 12

[Matthew Johnson]

Apologies added this to a new thread instead of existing (as i have
digest mode set)


On Wed, Jul 12, 2017 at 9:12 PM, Matthew Johnson <matt at section.io> wrote:
>> Date: Wed, 12 Jul 2017 11:20:52 +0200
>> From: Geoff Simmons <geoff at uplex.de>
>> To: varnish-misc at varnish-cache.org
>> Subject: Re: Value of builtin.vcl - vcl_recv on modern websites
>> Message-ID: <bd1c8f9d-270f-90f0-afd5-226ffde3eeeb at uplex.de>
>> Content-Type: text/plain; charset=windows-1252
>>
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA256
>>
>> On 07/12/2017 09:14 AM, Matthew Johnson wrote:
>>>
>>> Since Varnish 1.0 the builtin (or default) vcl_recv has had this
>>> statement: if (req.http.Authorization || req.http.Cookie) {
>>>
>>> /* Not cacheable by default */
>>>
>>> return (pass);
>>>
>>> }
>>>
>>> My issue with this is the req.http.Cookie check, In any modern
>>> website cookies are always present.
>>
>> While I'm sympathetic to all the things you said and don't mean to
>> disregard it but cutting things short, the answer here is simple:
>> there is no other choice for the default configuration of a caching
>> proxy, because those two headers mean that the response may be
>> personalized.
>
> Agreed. I think the area im looking to explore is how you move
> forwards from the default configuration as there are a few paths that
> can be taken.
>
>>
>> Many uses of cookies don't have that effect, of course, but Varnish
>> has no way of knowing that. As bad as the effect of the default config
>> may seem on sites that use cookies on every request -- Varnish doesn't
>> cache anything -- it would be much worse if someone sets up Varnish,
>> doesn't think of the consequences of not changing the default
>> configuration, and you end up seeing someone else's personal data in
>> cached responses.
>>
>> This problem is not specific to Varnish, but to any server that tries
>> to do what Varnish tries to do. I know from experience that it's
>> generally futile to say so, but this situation really ought to lead to
>> some widespread re-thinking throughout the industry. Forgive me for
>> shouting, soapbox-style, but this gives me an opportunity to sound off
>> on a pet peeve:
>>
>> ==> Maybe modern web site SHOULD NOT use cookies on every request!
>> Because of the way cookies interfere with downstream caching.
>>
>> I have come to the conviction that many uses of cookies are a result
>> of lazy thinking in app development. Many PHP devs, for example, are
>> in the habit of saying session_start(); at the beginning of every
>> script, without thinking twice about whether they really need it. I
>> have seen uses of cookies where "just toss that thing into a cookie"
>> was evidently the easy decision to make. I have seen cookies with
>> values that are 3KB long.
>
> Most .Net sites are the same, Almost any application I come across
> takes a "session first" approach.
>
>
>>
>> (Sometime over beer I'll tell you about that little database that
>> someone wanted to transport over a cookie, a base64-encoded CSV file
>> whose data was *also* base64-encoded, leading to a doubly
>> base64-encoded cookie value, in every request.)
> Beer sounds good. There are definitely many war stories on my side of
> madness in the use of cookies. Database in cookie in a novel approach!
>
>>
>> This is an instance of an issue that you encounter a lot with the use
>> of Varnish in practice: app development that doesn't think outside of
>> its own box in terms of functionality and performance. Rather than
>> thinking about the benefits of handing off some of your work, by
>> letting someone else serve your cached responses for you.
>>
>> HTTP was conceived from the beginning to enable caching as a means of
>> solving performance problems in slow networks. A well-configured
>> deployment of Varnish shows how beneficial that can be. But the
>> universal and unreflected use of cookies is one of the forces
>> presently at work that actively undermine that part of the equation.
>>
>>> A classic example of this is someone adding javascript via Google
>>> Tag Manager which then sets a cookie.
>>
>> One might have hoped that the Googlers, of all people, would have more
>> awareness of the trouble that they could cause by doing that.
>>
> Whilst Google do contribute to the issue with their own scripts,
> Google Tag Manager allows non technical people to deploy additional
> 3rd party javascript on a website. In the web performance space this
> always leads to slow loading websites in the browser but the cookie
> problem then plays into caching rules aswell.
>
>>> Do you still recommend configurations fall through to the
>>> underlying vcl_recv logic?
>>>
>>> Options i can think of:
>>
>> In a project where I am able to work with the app devs, I have had
>> good experience with working out a policy with them: if you MUST have
>> cookies in every request (although I WISH YOU WOULD RECONSIDER THAT),
>> then the caching proxy cannot make caching decisions on your behalf.
>> Only you can know if your response is cacheable, despite the presence
>> of cookie foo or bar, but is not cacheable if the cookie is baz.
>>
>> So if you want your response to be cached, you MUST say so in a
>> Cache-Control header. The proxy will not cache any other responses.
>
> I consider this ideal if it's possible to take that approach. I am
> often caught with customers using off the shelf platforms and less
> than ideal control over their application.
>
>>
>> Then we write VCL to bypass builtin's vcl_recv, and start Varnish with
>> - -t 0 (default TTL is 0s). Responses are then cached only if they
>> announce that they are cacheable.
>>
>> Of course, this has the effect that you're lamenting -- Varnish
>> doesn't cache anything by default -- but in my experience, the result
>> is that devs have become very good at thinking about the cacheability
>> of their responses.
>>
>> That boils down to answering your question by saying no, you can't use
>> builtin vcl_recv in a situation like that. When the cookies, like the
>> Evil, are always and everywhere (to paraphrase a saying in Germany),
>> and some cookies lead to cacheable responses while others don't, then
>> there's no other option for a caching proxy.
>
> A great summary, I think that summarises Dridi views earlier around
> the way HTTP has been implemented in applications. No perfect wins!
>
>>
>>
>> Best,
>> Geoff
>> - --
>> ** * * UPLEX - Nils Goroll Systemoptimierung
>>
>> Scheffelstra?e 32
>> 22301 Hamburg
>>
>> Tel +49 40 2880 5731
>> Mob +49 176 636 90917
>> Fax +49 40 42949753
>>
>> http://uplex.de
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v2
>>
>> iQIcBAEBCAAGBQJZZen0AAoJEOUwvh9pJNURnWEQAK9ucYSXEcrEwbmOrroBWoGK
>> iR9a8OFhst1rVKFQ2vNTUpw+OYM8vf8SJYToyq2VxG5/f/uGsT6nSVRWGKThgeV1
>> geyyUQfbbDte1at4aFy6HX6LeCt62Si0L9KUZMZMkI5C6m6FKgA/5HecUchhuXdP
>> Li17DXKvzrQyTBJpvbk2vBZGPVlErnVAUz75IeJMrD6t/WGO0PsZvkC/l8LZhqcD
>> 2S2R9SHYMIyBrWSZm+YsI1DxMwvH6Gt84NRPpKcHQQ7TKEfvtOq0NwoqcNOB26EL
>> KIfOuVJdbiMvD5D+BZud/7a7UzSpJz5klLMdTcJMN60MrHJjGcok/5KiG7TNNowj
>> hMy5YUYpuIybsWzcvB5Ie/nteb0WyXt5+LYkxjP9dbN7AN3k+aU1PboSOyqYXO3u
>> KK1al00LMKHfzMHs1vF3QHRt2Q1Udud6dCdHuw6TyJ7eWCc9YGgU8NyboMLkXBhO
>> fBVNUQQjfNjaDWhKFvUIMEsGZhgvzzuMvjlZNhkc/lcDLmU8wVXyiMFSoBcuR1sX
>> 7EM1wBUKcKix0wE4QPl9608ql/5LF3Ms+wqDpmS0ECgFIf1yKMWFZt9iHhMUbch2
>> 7fhr77vVjVD1K6nKHqDGOuLp4Cq+lfBJkd7PX2huQUV/hc00C8+NEieD77wuAwk7
>> OPiGNt5YqmDNjtZmFUVH
>> =BlAn
>> -----END PGP SIGNATURE-----
>>
>>
>>
>> *********************************************