.. Copyright (c) 2012-2021 Varnish Software AS SPDX-License-Identifier: BSD-2-Clause See LICENSE file for full text of license .. _tutorial-intro: Varnish: The beef in the sandwich --------------------------------- You may have heard the term "web-delivery-sandwich" used in relation to Varnish, and it is a pretty apt metafor:: ┌─────────┐ │ browser │ └─────────┘ ┌─────────┐ \ ┌─────────┐│ ┌─────┐ ╔═════════╗ ┌─────┐ ┌─────────┐ ┌─────────┐│┘ │ app │ --- ║ Network ║ -- │ TLS │ -- │ Varnish │ -- │ Backend │┘ └─────┘ ╚═════════╝ └─────┘ └─────────┘ └─────────┘ / ┌────────────┐ │ API-client │ └────────────┘ The top layer of the sandwich, 'TLS' is responsible for handling the TLS ("https") encryption, which means it must have access to the cryptographic certificate which authenticates your website. The bottom layer of the sandwich are your webservers, CDNs, API-servers, business backend systems and all the other sources for your web-content. Varnish goes in the middle, where it provides caching, policy, analytics, visibility and mitigation for your webtraffic. How Varnish works ----------------- For each and every request, Varnish runs through the 'VCL' program to decide what should happen: Which backend has this content, how long time can we cache it, is it accessible for this request, should it be redirected elsewhere and so on. If that particular backend is down, varnish can find another or substitute different content until it comes back up. Your first VCL program will probably be trivial, for instance just splitting the traffic between two different backend servers:: sub vcl_recv { if (req.url ~ "^/wiki") { set req.backend_hint = wiki_server; } else { set req.backend_hint = wordpress_server; } } When you load the VCL program into Varnish, it is compiled into a C-program which is compiled into a shared library, which varnish then loads and calls into, therefore VCL code is *fast*. Everything Varnish does is recorded in 'VSL' log records which can be examined and monitored in real time or recorded for later use in native or NCSA format, and when we say 'everything' we mean *everything*:: * << Request >> 318737 - Begin req 318736 rxreq - Timestamp Start: 1612787907.221931 0.000000 0.000000 - Timestamp Req: 1612787907.221931 0.000000 0.000000 - VCL_use boot - ReqStart 192.0.2.24 39698 a1 - ReqMethod GET - ReqURL /vmods/ - ReqProtocol HTTP/1.1 - ReqHeader Host: varnish-cache.org - ReqHeader Accept: text/html, application/rss+xml, […] - ReqHeader Accept-Encoding: gzip,deflate - ReqHeader Connection: close - ReqHeader User-Agent: Mozilla/5.0 […] - ReqHeader X-Forwarded-For: 192.0.2.24 - VCL_call RECV - VCL_acl NO_MATCH bad_guys - VCL_return hash […] These `VSL` log records are written to a circular buffer in shared memory, from where other programs can subscribe to them via a supported API. One such program is `varnishncsa` which produces NCSA-style log records:: 192.0.2.24 - - [08/Feb/2021:12:42:35 +0000] "GET http://vmods/ HTTP/1.1" 200 0 […] Varnish is also engineered for uptime, it is not necessary to restart varnish to change the VCL program, in fact, multiple VCL programs can be loaded at the same time and you can switch between them instantly. Caching with Varnish -------------------- When Varnish receives a request, VCL can decide to look for a reusable answer in the cache, if there is one, that becomes one less request to put load on your backend applications database. Cache-hits take less than a millisecond, often mere microseconds, to deliver. If there is nothing usable in the cache, the answer from the backend can, again under VCL control, be put in the cache for some amount of time, so future requests for the same object can find it there. Varnish understands the `Cache-Control` HTTP header if your backend server sends one, but ultimately the VCL program makes the decision to cache and how long, and if you want to send a different `Cache-Control` header to the clients, VCL can do that too. Content Composition with Varnish -------------------------------- Varnish supports `ESI - Edge Side Includes` which makes it possible to send responses to clients which are composed of different bits from different backends, with the very important footnote that the different bits can have very different caching policies. With ESI a backend can tell varnish to edit the content of another object into a HTML page::