From guillaume.quintard at gmail.com Sat Sep 4 22:06:17 2021 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Sat, 4 Sep 2021 15:06:17 -0700 Subject: PSA: varnish-modules packages for arch and alpine Message-ID: Hi everyone, Here's a short email to let you know that the varnish-modules[1] collection is now available in the AUR[2] and on alpine testing[3]. Those two join the existing varnish-module packages for Fedora[4] (if someone wants to package for Debian, let me know!). Those packages are fresh out of the oven, so if there is any issue, please report it either on the AUR page, the Alpine gitlab or directly on the varnish-module github page. *# Alpine instructions* Activate the testing repository in the configuration file, or just do it as a one-shot in the command line: apk add --no-cache -X http://dl-cdn.alpinelinux.org/alpine/edge/testing varnish-modules *# Arch instructions* If you already have an AUR-capable package manager[5] such as yay or yaourt, you can install it directly: yay -Sy varnish-modules Otherwise, build it yourself: sudo pacman -Sy git base-devel git clone https://aur.archlinux.org/varnish-modules.git cd varnish-modules/ makepkg -frisc Have a great week-end! [1]: https://github.com/varnish/varnish-modules [2]: https://aur.archlinux.org/packages/varnish-modules/ [3]: https://pkgs.alpinelinux.org/package/edge/testing/x86_64/varnish-modules [4]: https://src.fedoraproject.org/rpms/varnish-modules [5]: https://wiki.archlinux.org/title/AUR_helpers#Comparison_tables -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrhosseini at hotmail.com Mon Sep 6 14:21:11 2021 From: hrhosseini at hotmail.com (Hamidreza Hosseini) Date: Mon, 6 Sep 2021 14:21:11 +0000 Subject: Difference between adjusting set req backend in vcl_recv and vcl_backend_fetch Message-ID: Hi, This is part of my varnish configuration file for sharding: ``` cat /etc/varnish/default.vcl vcl 4.1; import directors; probe myprobe { .request = "HEAD /healthcheck.php HTTP/1.1" "Connection: close" "User-Agent: Varnish Health Probe"; .timeout = 1s; .interval = 5s; .window = 5; .threshold = 3; } backend b-01 { .host = "b-01"; .port = "80"; .probe = myprobe; } backend b-02 { .host = "b-02"; .port = "80"; .probe = myprobe; } backend b-03 { .host = "b-03"; .port = "80"; .probe = myprobe; } sub vcl_init { new hls_cluster = directors.shard(); hls_cluster.add_backend(b-01); hls_cluster.add_backend(b-02); hls_cluster.add_backend(b-03); new p = directors.shard_param(); hls_cluster.set_rampup(30s); # hls_cluster.set_warmup(0.5); hls_cluster.reconfigure(); hls_cluster.associate(p.use()); } acl purge { "localhost"; } sub vcl_recv { set req.backend_hint = hls_cluster.backend(); } sub vcl_backend_fetch { p.set(by=KEY, key=hls_cluster.key(bereq.url)); set bereq.backend = hls_cluster.backend(resolve=LAZY, healthy=CHOSEN); } ``` 1. there are two set backend in this config, one is on vcl_recv: "set req.backend_hint = hls_cluster.backend();" and one in vcl_backend_fetch: "set bereq.backend = hls_cluster.backend(resolve=LAZY, healthy=CHOSEN);" should I remove set in vcl_recv cause I think if I adjust it , all requset will go through this backend list and configs like healthy=CHOSEN in vcl_backend_fetch wouldnt be applied! Am I true? 2.Actually what is difference between vcl_backend_fetch and vcl_recv? 3.should I remove "set req.backend_hint" from vcl_recv? Best regards,Hamidreza -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpkorma at yahoo.com Tue Sep 7 15:12:43 2021 From: jpkorma at yahoo.com (John Kormanec) Date: Tue, 7 Sep 2021 11:12:43 -0400 Subject: Varnish Cache Runaway RAM Usage! References: <00080229-cee0-1c59-9860-889479287aeb.ref@yahoo.com> Message-ID: <00080229-cee0-1c59-9860-889479287aeb@yahoo.com> An HTML attachment was scrubbed... URL: -------------- next part -------------- [Unit] Description=Varnish Cache, a high-performance HTTP accelerator After=network-online.target [Service] Type=forking KillMode=process # Maximum number of open files (for ulimit -n) LimitNOFILE=131072 # Locked shared memory - should suffice to lock the shared memory log # (varnishd -l argument) # Default log size is 80MB vsl + 1M vsm + header -> 82MB # unit is bytes LimitMEMLOCK=85983232 # Enable this to avoid "fork failed" on reload. TasksMax=infinity # Maximum size of the corefile. LimitCORE=infinity # Configure Varnish listening port, default "vcl" file location, memory allocation, etc: ExecStart=/usr/sbin/varnishd -a :80 -f /etc/varnish/default.vcl -s malloc,14G -s Transient=malloc,512M -T 0.0.0.0:2000 ExecReload=/usr/sbin/varnishreload [Install] WantedBy=multi-user.target From dridi at varni.sh Tue Sep 7 15:43:17 2021 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 7 Sep 2021 15:43:17 +0000 Subject: Varnish Cache Runaway RAM Usage! In-Reply-To: <00080229-cee0-1c59-9860-889479287aeb@yahoo.com> References: <00080229-cee0-1c59-9860-889479287aeb.ref@yahoo.com> <00080229-cee0-1c59-9860-889479287aeb@yahoo.com> Message-ID: On Tue, Sep 7, 2021 at 3:14 PM John Kormanec wrote: > > Hello, > > We have been using the open source (free) version of Varnish Cache for my company's website for several months but are repeatedly running into an issue where Varnish's memory usage increases until all of the server's available memory is consumed, rendering the server unresponsive.??? This continues to happen despite having Varnish's malloc and transient malloc storage settings dialed way down (currently set at 50% combined of available RAM).??? Here's an excerpt from our storage backend configuration showing these settings.??? I've attached our full backend storage settings to this message for review. > > # Configure Varnish listening port, default "vcl" file location, memory allocation, etc: > ExecStart=/usr/sbin/varnishd -a :80 -f /etc/varnish/default.vcl -s malloc,14G -s Transient=malloc,512M -T 0.0.0.0:2000 > ExecReload=/usr/sbin/varnishreload > > We're running version 6.0.2 of Varnish Cache on a CentOS 8 virtual server.?????? The server has 30 GB of RAM, 29.3 GB of which are available for use. You should upgrade to 6.0.8 to get the latest bug fixes for the 6.0 branch. > I've read a number of technical posts from other Varnish users complaining about the same issue, including this one:??????https://stackoverflow.com/questions/65450044/varnish-6-lts-w-centos-8-not-respecting-memory-limits.??? Unfortunately, I've not seen any published resolution for this problem, but several tuning articles I've read talk about adjusting the server's memory allocation manager to more aggressively cleanup fragmentation, expired objects, etc.?????? Here's one such post that talks about adjusting the "jemalloc" memory allocation manager settings as a possible fix:??? https://info.varnish-software.com/blog/understanding-varnish-cache-memory-usage.??? I searched our CentOS 8 server to see what packages are installed but jemalloc is not in the list. I think you would have better results with jemalloc 3.6 that better fits Varnish workloads, but EPEL8 ships 5.2.1 currently. One thing to consider besides the jemalloc overhead is that you are only limiting cache storage to 14.5GB and there will be a memory footprint for everything else. > I'm still relatively new to Varnish Cache and am not sure what the next steps should be for troubleshooting & identifying the issue.??? FWIW, I reached out to the folks at Varnish Cache asking if they could offer any suggestions, but they said we'd have to upgrade to their Enterprise version, which uses something called a "massive storage engine" that would eliminate this problem.??? Not sure what the differences are between the paid / free versions, but I'm hoping to find a solution to this problem here before having to upgrade. > > Thanks in advance for any assistance the community can provide. Full disclosure, I work for Varnish Software, but what you were told was correct. The Massive Storage Engine brings a feature called Memory Governor that will allow your Varnish instance to pull all the levers available to decrease its footprint back to 15GB whenever it crosses the threshold. You could probably also expect less jemalloc overhead out of the box and have a decent memory target for 15GB of storage, for example 20GB. With Varnish Cache alone it is more difficult to plan for memory usage. Dridi From jpkorma at yahoo.com Tue Sep 7 17:37:27 2021 From: jpkorma at yahoo.com (John Kormanec) Date: Tue, 7 Sep 2021 13:37:27 -0400 Subject: Varnish Cache Runaway RAM Usage! In-Reply-To: References: <00080229-cee0-1c59-9860-889479287aeb.ref@yahoo.com> <00080229-cee0-1c59-9860-889479287aeb@yahoo.com> Message-ID: An HTML attachment was scrubbed... URL: From luca.gervasi at gmail.com Tue Sep 21 13:32:05 2021 From: luca.gervasi at gmail.com (Luca Gervasi) Date: Tue, 21 Sep 2021 15:32:05 +0200 Subject: Serve stale content if backend is healthy but "not too healthy" Message-ID: Hello everyone, We have a backend that actually proxies different services (mangling the original response). Sometimes one of those backends are not available and the general response goes from 200 to a 50x. Is there a way to serve a stale (valid) content (if present) for a request that comes from a backend in a healthy state? I was thinking about something like this: sub backend_fetch { if (beresp.status >= 500) { return_a_stale; } } >From the state machine (https://varnish-cache.org/docs/6.0/reference/states.html) it seems that I'm not allowed to return(hash) nor switch to an unhealthy backend (that i keep configured) to reach what I want. Please forgive me if do exists a facility to reach my goal and feel free to direct me to the right document. Ah. Varnish 6.x. Thanks Luca From dridi at varni.sh Tue Sep 21 14:36:53 2021 From: dridi at varni.sh (Dridi Boukelmoune) Date: Tue, 21 Sep 2021 14:36:53 +0000 Subject: Serve stale content if backend is healthy but "not too healthy" In-Reply-To: References: Message-ID: On Tue, Sep 21, 2021 at 1:33 PM Luca Gervasi wrote: > > Hello everyone, > > We have a backend that actually proxies different services (mangling > the original response). Sometimes one of those backends are not > available and the general response goes from 200 to a 50x. > Is there a way to serve a stale (valid) content (if present) for a > request that comes from a backend in a healthy state? > > I was thinking about something like this: > sub backend_fetch { > if (beresp.status >= 500) { > return_a_stale; > } > } > > From the state machine > (https://varnish-cache.org/docs/6.0/reference/states.html) it seems > that I'm not allowed to return(hash) nor switch to an unhealthy > backend (that i keep configured) to reach what I want. > > Please forgive me if do exists a facility to reach my goal and feel > free to direct me to the right document. > > Ah. Varnish 6.x. Hi Luca, Varnish Cache does not have this feature, you should be able to do that with Varnish Enterprise instead. What you are looking for is stale-if-error and you may find some implementations using VCL but I can't vouch for any, not having experience with them. https://docs.varnish-software.com/varnish-cache-plus/vmods/stale/#description Cheers, Dridi From guillaume.quintard at gmail.com Tue Sep 21 15:35:57 2021 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Tue, 21 Sep 2021 08:35:57 -0700 Subject: Serve stale content if backend is healthy but "not too healthy" In-Reply-To: References: Message-ID: Hi, As Dridi said, what you are looking for is exactly vmod_stale, but I wanted to point out that part: > We have a backend that actually proxies different services In that case, it might be good to actually have a Varnish backend for each type of backend behind the proxies. The backend definition would be exactly the same, but the probe definitions would be different, with a specific URL/host. this way, Varnish would be aware of who is actually unhealthy and you don't have to deal with the stale thing. If you need an open-source approach, I reckon the best you can do is restart with a zero TTL if you detect a bad response. It does have a couple of race conditions baked-in that vmod_stale sidesteps, but that's usually good enough: sub vcl_recv { # be hopeful that the backend will send us something good, ignore grace if (req.restarts == 0) { set req.grace = 0s; } } sub vcl_deliver { # welp, that didn't go well, try again without limiting the grace if (req.restarts == 0 && resp.status >= 500) { set req.ttl = 10y; return (restart); } } Main issue is that you restart, so you are going to spend a lil bit more time/resources processing the request, and the object in cache may have expired by the time you realize you need it. Hope that helps, -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlos.abalde at gmail.com Tue Sep 21 16:19:22 2021 From: carlos.abalde at gmail.com (Carlos Abalde) Date: Tue, 21 Sep 2021 18:19:22 +0200 Subject: Serve stale content if backend is healthy but "not too healthy" In-Reply-To: References: Message-ID: <7323D919-3424-4BD6-8ADB-825BB3245C21@gmail.com> Hi Luca, As Dridi and Guillaume said, what you're looking for is something like the stale VMOD. I think the OSS approach suggested by Guillaume could be extended a little bit in order to improve performance and avoid request serialization. Problem is the final VCL is *ugly* and hard to understand. In any case, next I'm sharing a VTC showing how it works: varnishtest "Full vs. limited grace" server s_backend { # First request when the cache is empty (see 1). rxreq txresp # Background fetch done while inside the limited grace period (see 2). rxreq txresp # Another request to refresh the content after the limited grace period has # finished (see 3). rxreq txresp # Another request to refresh the content after the limited grace period has # finished one more time (see 4). This time, we return an error so a cached # version will have to be returned to the client. rxreq txresp -status 500 } -start varnish v_backend -vcl { import std; backend default { .host = "${s_backend_addr}"; .port = "${s_backend_port}"; } sub vcl_recv { # Clean up internal headers. if (req.restarts == 0) { unset req.http.X-Varnish-Restarted-5xx; } unset req.http.X-Varnish-Use-Limited-Grace; # Set a limited grace unless a restart has been done to use full grace. if (!(req.restarts > 0 && req.http.X-Varnish-Restarted-5xx)) { set req.http.X-Varnish-Use-Limited-Grace = "1"; set req.grace = 2s; } else { set req.grace = 100y; } } sub vcl_backend_response { set beresp.ttl = 1s; # Set full grace value. This could be done by returning a proper value for # the stale-while-revalidate property in the Cache-Control header, which # Varnish understands (that's not the case with the stale-if-error # property). set beresp.grace = 24h; # Send requests with a broken backend response to vcl_backend_error so they # can be restarted. if (beresp.status >= 500 && beresp.status < 600) { return (error); } } sub vcl_backend_error { if (bereq.http.X-Varnish-Use-Limited-Grace && !bereq.uncacheable) { # Trigger restart in the client side in order to enable full grace and # try to deliver a staled object. Also, cache error response but under # a variant to avoid overwritting the staled object that may already be # in cache. Grace and keep are explicitly disabled to overwrite current # default behaviour (https://github.com/varnishcache/varnish-cache/issues/3024). set beresp.http.X-Varnish-Restart-5xx = "1"; set beresp.ttl = 1s; set beresp.grace = 0s; set beresp.keep = 0s; set beresp.http.Vary = "X-Varnish-Use-Limited-Grace"; return (deliver); } else { # Jump to 'vcl_synth' with a 503 status code. return (abandon); } } sub vcl_deliver { # Execute restart if the backend side requested so (see 'vcl_backend_error'). if (resp.http.X-Varnish-Restart-5xx && !req.http.X-Varnish-Restarted-5xx) { set req.http.X-Varnish-Restarted-5xx = "1"; return (restart); } # Clean up Vary header. if (resp.http.Vary == "X-Varnish-Use-Limited-Grace") { unset resp.http.Vary; } # Debug. set resp.http.X-Cache-Hits = obj.hits; } sub vcl_backend_fetch { # Clean up internal headers. if (bereq.retries == 0) { unset bereq.http.X-Varnish-Restart-5xx; } # Do not retry requests restarted due to 5xx backend responses, no # matter if a staled object has not been found or if a bgfetch has been # spawned after serving staled content. if (bereq.retries == 0 && bereq.http.X-Varnish-Restarted-5xx) { # Jump to 'vcl_synth' with a 503 status code. return (abandon); } } } -start client c1 -connect ${v_backend_sock} { # 1: Ask for a content. This will hit the backend as the cache is empty. txreq rxresp expect resp.status == 200 expect resp.http.X-Cache-Hits == 0 # Wait until the TTL is over. delay 1.5 # 2: Further requests to the same content inside the limited grace period # (set to 2 seconds) will be resolved by the cache. A bgfetch to the # backend is silently made. txreq rxresp expect resp.status == 200 expect resp.http.X-Cache-Hits == 1 # Wait until the new TTL and new limited grace period are over. delay 5.0 # 3: Even if the content is in the cache (it's being stored for the full # cache period: 24 hours), limited grace makes sure fresh content is # recovered from the backend. A request to the content, therefore, # produces a new hit in the backend. txreq rxresp expect resp.status == 200 expect resp.http.X-Cache-Hits == 0 # Wait again until the new TTL and new limited grace period are over. delay 3.5 # 4: A new request will try to get fresh content, but the backend now returns # an error response, so Varnish restarts the request and serves stalled # content. No background fetch is done as Varnish abort the attempt to # disturb the failing backend. txreq rxresp expect resp.status == 200 expect resp.http.X-Cache-Hits == 1 } -run varnish v_backend -expect client_req == 4 Best, -- Carlos Abalde -------------- next part -------------- An HTML attachment was scrubbed... URL: