From hrhosseini at hotmail.com Sun Aug 1 11:17:34 2021 From: hrhosseini at hotmail.com (Hamidreza Hosseini) Date: Sun, 1 Aug 2021 11:17:34 +0000 Subject: Best practice for caching scenario with different backend servers but same content Message-ID: Hi, I want to use varnish in my scenario as cache service, I have about 10 http servers that serve Hls fragments as the backend servers and about 5 varnish servers for caching purpose, the problem comes in when I use round-robin director for backend servers in varnish, if a varnish for specific file requests to one backend server and for the same file but to another backend server it would cache that file again because of different Host headers ! so my solution is using fallback director instead of round-robin as follow: ``` In varnish-1: new hls_cluster = directors.fallback(); hls_cluster.add_backend(b1()); hls_cluster.add_backend(b2()); hls_cluster.add_backend(b3()); hls_cluster.add_backend(b4()); hls_cluster.add_backend(b5()); hls_cluster.add_backend(b6()); hls_cluster.add_backend(b7()); hls_cluster.add_backend(b8()); hls_cluster.add_backend(b9()); hls_cluster.add_backend(b10()); In varnish-2: new hls_cluster = directors.fallback(); hls_cluster.add_backend(b10()); hls_cluster.add_backend(b1()); hls_cluster.add_backend(b2()); hls_cluster.add_backend(b3()); hls_cluster.add_backend(b4()); hls_cluster.add_backend(b5()); hls_cluster.add_backend(b6()); hls_cluster.add_backend(b7()); hls_cluster.add_backend(b8()); hls_cluster.add_backend(b9()); In varnish-3: new hls_cluster = directors.fallback(); hls_cluster.add_backend(b9()); hls_cluster.add_backend(b1()); hls_cluster.add_backend(b2()); hls_cluster.add_backend(b3()); hls_cluster.add_backend(b4()); hls_cluster.add_backend(b5()); hls_cluster.add_backend(b6()); hls_cluster.add_backend(b7()); hls_cluster.add_backend(b8()); hls_cluster.add_backend(b10()); ``` But I think this is not the best solution, because there is no load balancing despite, I used different backend for the first argument of fallback directive, What is varnish recommendation for this scenario? -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Sun Aug 1 16:30:23 2021 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Sun, 1 Aug 2021 09:30:23 -0700 Subject: Best practice for caching scenario with different backend servers but same content In-Reply-To: References: Message-ID: Hi, There are a lot of things to unpack here. > if a varnish for specific file requests to one backend server and for the same file but to another backend server it would cache that file again because of different Host headers ! so my solution is using fallback director instead of round-robin The two aren't related, if you have a hashing problem causing you to cache the same object twice, changing the directors isn't going to save you. Ideally, the requests will get normalized (host header and path) in vcl_recv{} so that they will be properly hashed in vcl_hash{}. The backend resolution only happens after you have exited vcl_backend_fetch{}, long after you have (not) found the object in the cache, and the best solution for video is usually to use consistent-hashing. In open-source this means vmod_shard ( https://varnish-cache.org/docs/trunk/reference/vmod_directors.html#directors-shard), in Enterprise, it'll be udo ( https://docs.varnish-software.com/varnish-cache-plus/vmods/udo/#set-hash), they are going to handle about the same except udo makes it easier to set the hash, which may be important for live (more info below). With consistent hashing, you can configure all Varnish servers the same, and they will determine which backend to use based on the request. Typically, the same request will always go to the same backend. This provides pretty good load-balancing over time, and additionally it leverages the internally caching that most video origins have. If you are serving VOD, that is all you need, but if you are serving Live, you need to care about one other thing: you want consistent hashing not per request, but per stream. Because the origins may be slightly out of sync, you may get a manifest on origin A which will advertise a chunk that isn't available anywhere yet, and if you don't fetch the new chunk from origin A, you'll get a 404 or a 412. So, for live, you will need to use shard's key() ( https://varnish-cache.org/docs/trunk/reference/vmod_directors.html#int-xshard-key-string) or udo's set_hash() ( https://docs.varnish-software.com/varnish-cache-plus/vmods/udo/#set-hash) to create a hash based on the stream path. For example, consider these paths: - /live/australia/Channel5/480p/manifest.m3u8 and /live/australia/Channel5/480p/chunk_43212123.ts: the stream path is /live/australia/Channel5/480p/ - /video/live/52342645323/manifest.dash and /video/live/52342645323/manifest.dash?time=4216432432&bitrate=80000000&audio=523453: the stream path is /video/live/52342645323/manifest.dash On top of all this, if you start having more than 5 Varnish servers, you might want to consider adding an extra layer of caching between the client-facing Varnish nodes and the origins (origin shields) to reduce the load on the origins. In that case, the shields would be the one handling the consistent hashing. Hope this helps -- Guillaume Quintard On Sun, Aug 1, 2021 at 4:18 AM Hamidreza Hosseini wrote: > Hi, > I want to use varnish in my scenario as cache service, I have about 10 > http servers that serve Hls fragments as the backend servers and about 5 > varnish servers for caching purpose, the problem comes in when I use > round-robin director for backend servers in varnish, > if a varnish for specific file requests to one backend server and for the > same file but to another backend server it would cache that file again > because of different Host headers ! so my solution is using fallback > director instead of round-robin as follow: > > ``` > In varnish-1: > new hls_cluster = directors.fallback(); > hls_cluster.add_backend(b1()); > hls_cluster.add_backend(b2()); > hls_cluster.add_backend(b3()); > hls_cluster.add_backend(b4()); > hls_cluster.add_backend(b5()); > hls_cluster.add_backend(b6()); > hls_cluster.add_backend(b7()); > hls_cluster.add_backend(b8()); > hls_cluster.add_backend(b9()); > hls_cluster.add_backend(b10()); > > > > In varnish-2: > new hls_cluster = directors.fallback(); > hls_cluster.add_backend(b10()); > hls_cluster.add_backend(b1()); > hls_cluster.add_backend(b2()); > hls_cluster.add_backend(b3()); > hls_cluster.add_backend(b4()); > hls_cluster.add_backend(b5()); > hls_cluster.add_backend(b6()); > hls_cluster.add_backend(b7()); > hls_cluster.add_backend(b8()); > hls_cluster.add_backend(b9()); > > > In varnish-3: > new hls_cluster = directors.fallback(); > hls_cluster.add_backend(b9()); > hls_cluster.add_backend(b1()); > hls_cluster.add_backend(b2()); > hls_cluster.add_backend(b3()); > hls_cluster.add_backend(b4()); > hls_cluster.add_backend(b5()); > hls_cluster.add_backend(b6()); > hls_cluster.add_backend(b7()); > hls_cluster.add_backend(b8()); > hls_cluster.add_backend(b10()); > > ``` > But I think this is not the best solution, because there is no load > balancing despite, I used different backend for the first argument of > fallback directive, > What is varnish recommendation for this scenario? > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Mon Aug 2 06:36:10 2021 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 2 Aug 2021 06:36:10 +0000 Subject: Creating ACLs in Varnish In-Reply-To: References: Message-ID: > Is there a better approach to this in Varnish Cache? We?re also going to be evaluating Varnish Enterprise, so if there?s something in VE, that would also be good to know. Hello, There are better ways to do this, but not out of the box with Varnish Cache. You would need something like https://code.uplex.de/uplex-varnish/varnish-objvar/ to index ACLs by host names. With Varnish Enterprise, you can manage this out of the box with the combination of built-in modules vmod_aclplus and vmod_kvstore. https://docs.varnish-software.com/varnish-cache-plus/vmods/aclplus/ https://docs.varnish-software.com/varnish-cache-plus/vmods/kvstore/ Feel free to reach out to me directly to discuss the Varnish Enterprise solution with you. Best Regards, Dridi From dridi at varni.sh Mon Aug 2 07:07:43 2021 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 2 Aug 2021 07:07:43 +0000 Subject: Upgrading to 6.6 can't reload In-Reply-To: References: Message-ID: >> Can you try removing the -S option from varnishd? >> >> Since you only listen to the CLI on localhost, there's likely no >> remote access, so leaving the secret out will make varnishd generate a >> random one. Basically, if you want to use varnishadm you need local >> root privileges, same as your current setup. > > > I tried this and it makes no difference, I think the fundamental issue is that calling varnishadm without args seems (regardless the args I pass to varnishd) to end in the message "No -T in shared memory" if run from root. This is very strange, I would probably need to somehow peek at the working directory to figure out what's happening. The only way to see no -T in shared memory is to explicitly ask for it with `-T none`, which clearly you didn't do. > If I run from another user, I do get the message "could not get hold of varnishd, is it running?" This on the other hand sounds like basic unix permissions coming short, as one would expect. > I guess I could update the reload script to pass the -T and -S args, but this seems wrong, just concerned there is a general issue on focal, Is anyone else running 6.6 on focal? The varnishreload script is meant to focus on the system service integration use case: you have a local Varnish instance that can also be operated locally. So we shouldn't need to add options to specify -T or -S, we should find them on the running instance. You could use -T none if you have an alternative mode of operations. For example varnishd -d puts you in debug mode and stdin/stdout is used for the CLI. The alternative would be the -M option that lets varnishd "reverse" connect to its operator, but then you would leave the varnishreload use case. Not sure about focal users, but I will try to spin up a VM and see if I can reproduce it. > Looking at the source code in 6 and 6.6 I can't see anywhere that the -T would default from and yet on 6 under bionic varnishadm as a root user just works without any -T or -S flags. > > https://github.com/varnishcache/varnish-cache/blob/6.0/bin/varnishadm/varnishadm.c Correct, the default behavior is to inspect the running Varnish instance to find them. >> A surefire way to see whether varnishlog connects to a running varnish >> instance is to try: >> >> varnishlog -d -g raw > > > I am running as root. If I execute this it connects but I get no output, I know it is connected because when I restart the varnish process I get the message, "Log abandoned (vsm)" which you always see when a new varbnishd process starts. I am definitely hitting the varnish server, as I am executing curl requests to localhost:80, but there is no output from varnishlog. That seems to indicate that Varnish may have been started on a different hostname and what you are "connecting" to is the remnant of a dead instance. What is the output of `ls /var/lib/varnish` ? > I am about to spin up some more boxes, so will check to see wheter this is just specific to this box or not, I did initially install 6.2 on this server and varnishlog was working as expected with that. I would recommend sticking to the 6.0 LTS series, unless you absolutely need a feature released after 6.0 that hasn't been back-ported to the stable branch. https://packagecloud.io/varnishcache/varnish60lts/install If you don't use an LTS series, I recommend always sticking to the latest release. The 6.2 series is EOL and no longer maintained, which may include security vulnerabilities such as VSV7. https://varnish-cache.org/security/VSV00007.html Dridi From hrhosseini at hotmail.com Thu Aug 5 10:31:33 2021 From: hrhosseini at hotmail.com (Hamidreza Hosseini) Date: Thu, 5 Aug 2021 10:31:33 +0000 Subject: Best practice for caching scenario with different backend servers but same content In-Reply-To: References: Message-ID: Hi, 1. Is there any way to normalize host headers and other things to say to varnish not to cache the same content for different backend? I want to use round robin director but after fetching the content I want to normalize the header and cache the content, I would appreciate if you give me an example about this and how I can do it. 2. I couldn't find any good example for directors-shard and xshard-key-string, I would appreciate if you could give example about this too. Many Thanks ________________________________ From: varnish-misc on behalf of Hamidreza Hosseini Sent: Sunday, August 1, 2021 4:17 AM To: varnish-misc at varnish-cache.org Subject: Best practice for caching scenario with different backend servers but same content Hi, I want to use varnish in my scenario as cache service, I have about 10 http servers that serve Hls fragments as the backend servers and about 5 varnish servers for caching purpose, the problem comes in when I use round-robin director for backend servers in varnish, if a varnish for specific file requests to one backend server and for the same file but to another backend server it would cache that file again because of different Host headers ! so my solution is using fallback director instead of round-robin as follow: ``` In varnish-1: new hls_cluster = directors.fallback(); hls_cluster.add_backend(b1()); hls_cluster.add_backend(b2()); hls_cluster.add_backend(b3()); hls_cluster.add_backend(b4()); hls_cluster.add_backend(b5()); hls_cluster.add_backend(b6()); hls_cluster.add_backend(b7()); hls_cluster.add_backend(b8()); hls_cluster.add_backend(b9()); hls_cluster.add_backend(b10()); In varnish-2: new hls_cluster = directors.fallback(); hls_cluster.add_backend(b10()); hls_cluster.add_backend(b1()); hls_cluster.add_backend(b2()); hls_cluster.add_backend(b3()); hls_cluster.add_backend(b4()); hls_cluster.add_backend(b5()); hls_cluster.add_backend(b6()); hls_cluster.add_backend(b7()); hls_cluster.add_backend(b8()); hls_cluster.add_backend(b9()); In varnish-3: new hls_cluster = directors.fallback(); hls_cluster.add_backend(b9()); hls_cluster.add_backend(b1()); hls_cluster.add_backend(b2()); hls_cluster.add_backend(b3()); hls_cluster.add_backend(b4()); hls_cluster.add_backend(b5()); hls_cluster.add_backend(b6()); hls_cluster.add_backend(b7()); hls_cluster.add_backend(b8()); hls_cluster.add_backend(b10()); ``` But I think this is not the best solution, because there is no load balancing despite, I used different backend for the first argument of fallback directive, What is varnish recommendation for this scenario? -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Thu Aug 5 16:49:21 2021 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Thu, 5 Aug 2021 09:49:21 -0700 Subject: Best practice for caching scenario with different backend servers but same content In-Reply-To: References: Message-ID: Hi, I'm pretty sure there's a confusion with the sequence of actions here. Normalization happen *before* you look into the cache, so that way before you fetch anything from the backend. By the time you cache the data (vcl_backend_response), the hash key has already been set (vcl_hash), it's way too late to normalize the request. As to normalization, it's usually done in vcl_recv, and it can range from just setting the host header to a static string to using std.tolower() and removing the host port. for the sake of the example: sub vcl_vcl { set req.http.host = "myvideoservice.com"; } For shard example, look at the VCTs, for example: https://github.com/varnishcache/varnish-cache/blob/6.6/bin/varnishtest/tests/d00029.vtc#L66 import directors; sub vcl_init { new shard_dir = directors.shard(); shard_dir.add_backend(be1); shard_dir.add_backend(be2); shard_dir.add_backend(be3; new p = directors.shard_param(); vd.associate(p.use()); vd.reconfigure(replicas=25);} sub vcl_backend_fetch { p.set(by=KEY, key=bereq.url); set bereq.backend_hint = shard_dir.backend(resolve=LAZY);} For udo: import crypto; import udo; sub vcl_init { new udo_dir = udo.director(); udo_dir.set_type(random); udo_dir.add_backend(be1); udo_dir.add_backend(be2); udo_dir.add_backend(be3); udo_dir.set_type(hash);} sub vcl_backend_fetch { set bereq.backend_hint = udo_dir.backend(); udo_dir.set_hash(crypto.hash(sha256, bereq.url));} These have been written without testing, so don't put them straight into production. -- Guillaume Quintard On Thu, Aug 5, 2021 at 3:33 AM Hamidreza Hosseini wrote: > Hi, > 1. > > Is there any way to normalize host headers and other things to say to > varnish not to cache the same content for different backend? > I want to use round robin director but after fetching the content I want > to normalize the header and cache the content, > I would appreciate if you give me an example about this and how I can do > it. > > 2. > I couldn't find any good example for directors-shard and > xshard-key-string, I would appreciate if you could give example about this > too. > > Many Thanks > ------------------------------ > *From:* varnish-misc hotmail.com at varnish-cache.org> on behalf of Hamidreza Hosseini < > hrhosseini at hotmail.com> > *Sent:* Sunday, August 1, 2021 4:17 AM > *To:* varnish-misc at varnish-cache.org > *Subject:* Best practice for caching scenario with different backend > servers but same content > > Hi, > I want to use varnish in my scenario as cache service, I have about 10 > http servers that serve Hls fragments as the backend servers and about 5 > varnish servers for caching purpose, the problem comes in when I use > round-robin director for backend servers in varnish, > if a varnish for specific file requests to one backend server and for the > same file but to another backend server it would cache that file again > because of different Host headers ! so my solution is using fallback > director instead of round-robin as follow: > > ``` > In varnish-1: > new hls_cluster = directors.fallback(); > hls_cluster.add_backend(b1()); > hls_cluster.add_backend(b2()); > hls_cluster.add_backend(b3()); > hls_cluster.add_backend(b4()); > hls_cluster.add_backend(b5()); > hls_cluster.add_backend(b6()); > hls_cluster.add_backend(b7()); > hls_cluster.add_backend(b8()); > hls_cluster.add_backend(b9()); > hls_cluster.add_backend(b10()); > > > > In varnish-2: > new hls_cluster = directors.fallback(); > hls_cluster.add_backend(b10()); > hls_cluster.add_backend(b1()); > hls_cluster.add_backend(b2()); > hls_cluster.add_backend(b3()); > hls_cluster.add_backend(b4()); > hls_cluster.add_backend(b5()); > hls_cluster.add_backend(b6()); > hls_cluster.add_backend(b7()); > hls_cluster.add_backend(b8()); > hls_cluster.add_backend(b9()); > > > In varnish-3: > new hls_cluster = directors.fallback(); > hls_cluster.add_backend(b9()); > hls_cluster.add_backend(b1()); > hls_cluster.add_backend(b2()); > hls_cluster.add_backend(b3()); > hls_cluster.add_backend(b4()); > hls_cluster.add_backend(b5()); > hls_cluster.add_backend(b6()); > hls_cluster.add_backend(b7()); > hls_cluster.add_backend(b8()); > hls_cluster.add_backend(b10()); > > ``` > But I think this is not the best solution, because there is no load > balancing despite, I used different backend for the first argument of > fallback directive, > What is varnish recommendation for this scenario? > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Fri Aug 6 00:59:57 2021 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Thu, 5 Aug 2021 17:59:57 -0700 Subject: Docker images: Alpine and new architecture support Message-ID: Hi everyone, I just wanted to everyone that we have now closed two important tickets: - https://github.com/varnish/docker-varnish/issues/2 (alpine support) - https://github.com/varnish/docker-varnish/issues/12 (arm support) In short, all images are now supported on amd64, arm32v7, arm64v8, i386, ppc64le, and s390x. And on top of this, the "fresh" tags are also accessible with an "-alpine" suffix (fresh-alpine, 6.6.1-alpine, etc.). What you don't get, for now, is an alpine variant of the stable image as we still need a couple of backports before it's viable. The way images are built have changed quite a lot (they don't rely on packagecloud anymore), and there could be a few quirks lying around, so please give it a go and report anything odd on the usual bug tracker: https://github.com/varnish/docker-varnish/issues And I need to thank again @tianon and @yosifkit over at https://github.com/docker-library/official-images for their help and unending patience. Cheers, -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrhosseini at hotmail.com Sat Aug 7 09:19:58 2021 From: hrhosseini at hotmail.com (Hamidreza Hosseini) Date: Sat, 7 Aug 2021 09:19:58 +0000 Subject: Best practice for caching scenario with different backend servers but same content In-Reply-To: References: Message-ID: Hi, I read the sample config that you sent but Can I use "bereq.url" in this way: for example I want to shard my requests for live streams based on the url's that clients enter, for example if the following url's are different live streams (stream1 and stream2 are the name of different streams ): ``` mydomain.com/live/australia/stream1/chunk_43212123.ts mydomain.com/live/australia/stream2/chunk_43212123.ts mydomain.com/live/australia/stream3/chunk_43212123.ts ``` Now think I want just the url excluded with chunk file becomes hashed and sharded: Just this part: "/live/australia/stream{1,2,3}/" Not :"/live/australia/stream{1,2,3}/chunk_43212123.ts" So by adjusting " p.set(by=KEY, key=bereq.url) " it would shard "bereq.url", it means "/live/australia/stream{1,2,3}/chunk_43212123.ts" ________________________________ From: varnish-misc on behalf of Hamidreza Hosseini Sent: Sunday, August 1, 2021 4:17 AM To: varnish-misc at varnish-cache.org Subject: Best practice for caching scenario with different backend servers but same content Hi, I want to use varnish in my scenario as cache service, I have about 10 http servers that serve Hls fragments as the backend servers and about 5 varnish servers for caching purpose, the problem comes in when I use round-robin director for backend servers in varnish, if a varnish for specific file requests to one backend server and for the same file but to another backend server it would cache that file again because of different Host headers ! so my solution is using fallback director instead of round-robin as follow: ``` In varnish-1: new hls_cluster = directors.fallback(); hls_cluster.add_backend(b1()); hls_cluster.add_backend(b2()); hls_cluster.add_backend(b3()); hls_cluster.add_backend(b4()); hls_cluster.add_backend(b5()); hls_cluster.add_backend(b6()); hls_cluster.add_backend(b7()); hls_cluster.add_backend(b8()); hls_cluster.add_backend(b9()); hls_cluster.add_backend(b10()); In varnish-2: new hls_cluster = directors.fallback(); hls_cluster.add_backend(b10()); hls_cluster.add_backend(b1()); hls_cluster.add_backend(b2()); hls_cluster.add_backend(b3()); hls_cluster.add_backend(b4()); hls_cluster.add_backend(b5()); hls_cluster.add_backend(b6()); hls_cluster.add_backend(b7()); hls_cluster.add_backend(b8()); hls_cluster.add_backend(b9()); In varnish-3: new hls_cluster = directors.fallback(); hls_cluster.add_backend(b9()); hls_cluster.add_backend(b1()); hls_cluster.add_backend(b2()); hls_cluster.add_backend(b3()); hls_cluster.add_backend(b4()); hls_cluster.add_backend(b5()); hls_cluster.add_backend(b6()); hls_cluster.add_backend(b7()); hls_cluster.add_backend(b8()); hls_cluster.add_backend(b10()); ``` But I think this is not the best solution, because there is no load balancing despite, I used different backend for the first argument of fallback directive, What is varnish recommendation for this scenario? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrhosseini at hotmail.com Mon Aug 9 08:50:13 2021 From: hrhosseini at hotmail.com (Hamidreza Hosseini) Date: Mon, 9 Aug 2021 08:50:13 +0000 Subject: Best practice for caching scenario with different backend servers but same content In-Reply-To: References: Message-ID: Hi, This is my configuration based on Doc and sample configuration, I would appreciate if you answer my questions: ``` probe myprobe { .request = "HEAD / HTTP/1.1" "Connection: close" "User-Agent: Varnish Health Probe"; .timeout = 1s; .interval = 5s; .window = 5; .threshold = 3; } backend B1 { .host = "B1.mydomain.local"; .port = "80"; .probe = myprobe; } backend B2 { .host = "B2.mydomain.local"; .port = "80"; .probe = myprobe; } backend B3 { .host = "B3.mydomain.local"; .port = "80"; .probe = myprobe; } backend B4 { .host = "B4.mydomain.local"; .port = "80"; .probe = myprobe; } backend B5 { .host = "B5.mydomain.local"; .port = "80"; .probe = myprobe; } sub vcl_init { new hls_cluster = directors.shard(); hls_cluster.add_backend(B1); hls_cluster.add_backend(B2); hls_cluster.add_backend(B3); hls_cluster.add_backend(B4); hls_cluster.add_backend(B5); new p = directors.shard_param(); hls_cluster.reconfigure(p, replicas=25); hls_cluster.associate(p.use()); } sub vcl_backend_fetch { p.set(by=KEY, key=bereq.url); set bereq.backend_hint = hls_cluster.backend(resolve=LAZY, healthy=ALL);} } ``` 1. First of all I think following section is not true, Do we need define both shard parameter(p) and replicas in the reconfigure directive? "hls_cluster.reconfigure(p, replicas=25);" or just "hls_cluster.reconfigure(replicas=25);" 2. What does "replicas=25" means in the sample configuration? Doc says: (default ident being the backend name) for each backend and for a running number n from 1 to replicas this is what I found out: varnish will choose one number for a backend randomly from 1 to "replicas" number and then will combine it with name of backend and then will hash all for circular ring, But when we have backends with different names this hash would be different for each backend because they have different name! why is this neccessary? `https://varnish-cache.org/docs/6.2/reference/vmod_directors.html#bool-xshard-reconfigure-int-replicas-67` 3. In the shard.backend(...) section and About "resolve=LAZY": I couldn't understand what does LAZY resolve mean? DOC: ``` LAZY: return an instance of this director for later backend resolution. LAZY mode is required for referencing shard director instances, for example as backends for other directors (director layering). ``` https://varnish-cache.org/docs/6.1/reference/vmod_directors.generated.html#shard-backend 4. For returning healthy backend besides defining probes as I adjust, should I configure healthy=ALL as follow? ``` set bereq.backend_hint = hls_cluster.backend(resolve=LAZY, healthy=ALL);} ``` DOC: ``` ALL: Check health state also for alternative backend selection ``` 5. About rampup and warmup : rampup: I understand that if a backend goes down and become healthy again if we defined a rampup period for it, it would wait till this period passed and then varnish will send the request to that backend for this fraction of time it will return alternative backend warmup: for a choosen backend for specific key it will spread request between two backend (the original backend and its alternative if we define 0.5 for warmup) Please correct me if I said anything wrong. I would apprecate if you could explain about the functionality of this two parameter. 6. conceptual question: 1.What's the exact difference between hash and shard directive and when should we use which one? the Doc says when the backend changes shard is more consistent than hash but how? 2.What will it happen when I'm using shard director based on "key=bereq.url" if I add/delete one backend from backend lists? will it change the consistent hashing ring for requests? Thanks for your answers in advance Best regards, Hamidreza ________________________________ From: Hamidreza Hosseini Sent: Sunday, August 1, 2021 4:17 AM To: varnish-misc at varnish-cache.org Subject: Best practice for caching scenario with different backend servers but same content Hi, I want to use varnish in my scenario as cache service, I have about 10 http servers that serve Hls fragments as the backend servers and about 5 varnish servers for caching purpose, the problem comes in when I use round-robin director for backend servers in varnish, if a varnish for specific file requests to one backend server and for the same file but to another backend server it would cache that file again because of different Host headers ! so my solution is using fallback director instead of round-robin as follow: ``` In varnish-1: new hls_cluster = directors.fallback(); hls_cluster.add_backend(b1()); hls_cluster.add_backend(b2()); hls_cluster.add_backend(b3()); hls_cluster.add_backend(b4()); hls_cluster.add_backend(b5()); hls_cluster.add_backend(b6()); hls_cluster.add_backend(b7()); hls_cluster.add_backend(b8()); hls_cluster.add_backend(b9()); hls_cluster.add_backend(b10()); In varnish-2: new hls_cluster = directors.fallback(); hls_cluster.add_backend(b10()); hls_cluster.add_backend(b1()); hls_cluster.add_backend(b2()); hls_cluster.add_backend(b3()); hls_cluster.add_backend(b4()); hls_cluster.add_backend(b5()); hls_cluster.add_backend(b6()); hls_cluster.add_backend(b7()); hls_cluster.add_backend(b8()); hls_cluster.add_backend(b9()); In varnish-3: new hls_cluster = directors.fallback(); hls_cluster.add_backend(b9()); hls_cluster.add_backend(b1()); hls_cluster.add_backend(b2()); hls_cluster.add_backend(b3()); hls_cluster.add_backend(b4()); hls_cluster.add_backend(b5()); hls_cluster.add_backend(b6()); hls_cluster.add_backend(b7()); hls_cluster.add_backend(b8()); hls_cluster.add_backend(b10()); ``` But I think this is not the best solution, because there is no load balancing despite, I used different backend for the first argument of fallback directive, What is varnish recommendation for this scenario? -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Mon Aug 9 12:49:25 2021 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 9 Aug 2021 14:49:25 +0200 Subject: Best practice for caching scenario with different backend servers but same content In-Reply-To: References: Message-ID: Hello, The best way to answer these questions is to start with the last one: On 8/9/21 10:50, Hamidreza Hosseini wrote: > > 6. conceptual question: > > 1.What's the exact difference between hash and shard directive and when > should we use which one? > the Doc says when the backend changes shard is more consistent than hash > but how? It's not "more consistent", it's "consistent hashing", which is the name of a hashing algorithm intended for load balancing: https://en.wikipedia.org/wiki/Consistent_hashing The hash director uses a more traditional kind of hashing algorithm to map requests to backends. Consistent hashing, implemented by the shard director, is intended to mitigate problems that can arise when backends become unhealthy and then healthy again. The focus is mainly on backends that do something expensive for new requests, such as their own caching, but can work faster for requests that they've seen before. A hash algorithm computes a number h(b) for a backend b, whose value is in a large range, say 32 bit. Then if you have N backends, the traditional algorithm indexes them from 0 to N-1, and picks h(b) mod N. Varnish's hash director is something like that. Say you have N=10 backends, so the traditional algorithm picks h(b) mod 10. Then a backend goes unhealthy, so now N=9. Since x mod 10 is unequal to x mod 9 for all x, the mapping of requests to backends shifts completely. This can be painful for backends that benefit from getting mostly the same requests, for example due to caching. After a while, the unhealthy backend becomes healthy again, so we go from N=9 back to N=10. If in the meantime the backends had "gotten used to" the changed distribution of requests, say by filling their local caches, then they get the pain all over again. Consistent hashing attempts to lessen the pain. If a backend drops out, then the mapping of requests to that backend must change, but the mapping stays the same for all other backends. So the distribution to backends changes only as much as it has to. Disclaimer: the original version of the shard director was developed at my company. Some lines of code that I contributed are still in there. > 2.What will it happen when I'm using shard director based on > "key=bereq.url" if I add/delete one backend from backend lists? will it > change the consistent hashing ring for requests? That's what consistent hashing is about. It only changes for the backends that were added or deleted, not for the rest. > 1. First of all I think following section is not true, Do we need define > both shard parameter(p) and replicas in the reconfigure directive? > "hls_cluster.reconfigure(p, replicas=25);" or just > "hls_cluster.reconfigure(replicas=25);" There is no 2-argument form of reconfigure(). It has one optional argument (replicas) with a default value. > 2. What does "replicas=25" means in the sample configuration? > > why is this neccessary? The short answer is that if you have to ask, then use the default. Since the argument is optional, and set to the default if you leave it out, just leave it out: reconfigure() You should NOT set it to 25. replicas is an internal parameter of the algorithm, and there aren't many guidelines as how it should be set, so we wanted to be able to experiment with it. It's really there so that developers of Varnish and the director can test it. Most users of Varnish don't need to worry about it. It turns out that the value doesn't matter all that much, as long as it isn't too low. 25 is too low. I advise against setting it lower than the default (67). What replicas does and why it's necessary gets into details of the algorithm. We can say more about that if there's interest, but this email is getting pretty long as it is. (The Wikipedia article gets into this in the section "Practical Extensions".) > 3. In the shard.backend(...) section and About "resolve=LAZY": > I couldn't understand what does LAZY resolve mean? When you set bereq.backend to a director, most of the Varnish directors do not immediately execute the algorithm to choose a backend, not until it's time to actually send the backend request. This has some advantages, for example if the VCL logic after setting bereq.backend results in not going to that backend after all. resolve=LAZY works this way. The alternative resolve=NOW is for contexts where you return the backend (read the return value of shard.backend()) and need to know which backend it's going to be. Then the backend is chosen right away, and stays that way when the backend request is sent. > 4. For returning healthy backend besides defining probes as I adjust, > should I configure healthy=ALL as follow? The parameters alt, healthy, rampup and warmup give you some control over what happens when one or more backends drop out. Say your request ordinarily maps to b1, but b1 is unhealthy; then there is a specific backend b2 that is chosen next. If b2 is also unhealthy, then there is a specific alternative b3, and so on. These are always the same. In other words, the order of alternatives chosen for unhealthy backends is constant for a given configuration. If you set alt=N with N > 0, then the Nth backend in that order is chosen. This is mainly for retries -- by setting alt=bereq.retries, you try a different backend on each retry, in the order established by b1, b2, b3 ... etc. The healthy parameter controls what happens when the director searches down the list due to unhealthy backends. healthy=ALL means that it continues searching, starting at alt=N (from the start when alt=0), until a healthy backend is found (or fail if they're all unhealthy). healthy=CHOSEN means don't skip ahead due to alt, just search for a healthy backend starting from the beginning of the list. healthy=IGNORE means don't consider the health status, and just choose the backend at alt=N no matter what. > 5. About rampup and warmup : > rampup: I understand that if a backend goes down and become healthy > again if we defined a rampup period for it, it would wait till this > period passed and then varnish will send the request to that backend for > this fraction of time it will return alternative backend Not quite. When a backend is added and rampup is set, then the probability of choosing that backend increases during the time set by rampup. Say for rampup=60s, the probability is very low just after the backend goes healthy, 25% after 15 seconds, 50% after 30 seconds, and so on. The idea is for backends that do expensive operations such as caching for "new" requests. Such a backend could be overwhelmed if Varnish sends all of the new requests at once when the backend becomes healthy, so rampup increases the load slowly. > warmup: for a choosen backend for specific key it will spread request > between two backend (the original backend and its alternative if we > define 0.5 for warmup) Yes. warmup is the probability that the next backend down the line is chosen, even if the first backend that would have been chosen is healthy. This is also for backends that would suffer under heavy load if another backend goes unhealthy, due to new requests that would have gone to the unhealthy backend. With warmup, the backend gets a portion of those requests even when the other backend is healthy, so that it's partially prepared for the additional load if the other backend drops out. HTH, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 840 bytes Desc: OpenPGP digital signature URL: From geoff at uplex.de Mon Aug 9 13:04:02 2021 From: geoff at uplex.de (Geoff Simmons) Date: Mon, 9 Aug 2021 15:04:02 +0200 Subject: Best practice for caching scenario with different backend servers but same content In-Reply-To: References: Message-ID: <5432a6e8-533e-3c33-bcb2-ea9d8098d96b@uplex.de> On 8/9/21 14:49, Geoff Simmons wrote: > > A hash algorithm computes a number h(b) for a backend b, ... Sorry, this should have been more like h(bereq), meaning that the number is computed from features of the request. From that you get to the choice of a backend. The Varnish hash director uses the same hash value that was computed for caching. The shard director also does that by default, but some other choices can be set with the parameter by in shard.backend(): by=HASH (default), by=URL, by=KEY, by=BLOB Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 840 bytes Desc: OpenPGP digital signature URL: From dridi at varni.sh Thu Aug 12 09:26:06 2021 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 12 Aug 2021 09:26:06 +0000 Subject: Best practice for caching scenario with different backend servers but same content In-Reply-To: References: Message-ID: On Sat, Aug 7, 2021 at 9:21 AM Hamidreza Hosseini wrote: > > Hi, > I read the sample config that you sent but Can I use "bereq.url" in this way: > for example I want to shard my requests for live streams based on the url's that clients enter, for example if the following url's are different live streams (stream1 and stream2 are the name of different streams ): > ``` > mydomain.com/live/australia/stream1/chunk_43212123.ts > mydomain.com/live/australia/stream2/chunk_43212123.ts > mydomain.com/live/australia/stream3/chunk_43212123.ts > ``` > Now think I want just the url excluded with chunk file becomes hashed and sharded: > Just this part: "/live/australia/stream{1,2,3}/" > Not :"/live/australia/stream{1,2,3}/chunk_43212123.ts" > > So by adjusting " p.set(by=KEY, key=bereq.url) " it would shard "bereq.url", it means "/live/australia/stream{1,2,3}/chunk_43212123.ts" If you want to do that, you need to extract the relevant part and compute a key from it: > set bereq.http.my-key = bereq.http.host + regsub(bereq.url, "/chunk_.*", ""); > set bereq.backend = dir.backend(by=KEY, key=dir.key(bereq.http.my-key)); Below is a test case where both requests go through the same backend: --- varnishtest "shard by subset of url" server s1 { loop 2 { rxreq txresp } } -start server s2 { loop 2 { rxreq txresp } } -start varnish v1 -vcl+backend { import directors; sub vcl_init { new dir = directors.shard(); dir.add_backend(s1); dir.add_backend(s2); } sub vcl_backend_fetch { set bereq.http.shard-key = bereq.http.host + regsub(bereq.url, "/chunk_.*", ""); set bereq.backend = dir.backend(by=KEY, key=dir.key(bereq.http.shard-key)); } sub vcl_backend_response { set beresp.http.backend = beresp.backend; } } -start client c1 { txreq -url "/live/australia/stream1/chunk_123.ts" rxresp expect resp.http.backend == s1 txreq -url "/live/australia/stream1/chunk_456.ts" rxresp expect resp.http.backend == s1 } -run --- Dridi From geoff at uplex.de Fri Aug 13 10:27:55 2021 From: geoff at uplex.de (Geoff Simmons) Date: Fri, 13 Aug 2021 12:27:55 +0200 Subject: Docker images: Alpine and new architecture support In-Reply-To: References: Message-ID: <6af5fbe2-9a5b-b2b9-bd33-562d261db06c@uplex.de> On 8/6/21 02:59, Guillaume Quintard wrote: > > In short, all images are now supported on amd64, arm32v7, arm64v8, i386, > ppc64le, and s390x. And on top of this, the "fresh" tags are also > accessible with an "-alpine" suffix (fresh-alpine, 6.6.1-alpine, etc.). Guillaume this is good news, thanks for this effort. Can you (or anyone) share some info about how well Varnish performs with musl libc? I like the goals of the musl project, and I sure would like to get Varnish to run on smaller images, but I've been reluctant to seriously try it on alpine. Because I'm not sure how well optimized musl is compared to the old standby glibc, and Varnish needs to perform well. glibc's reputation is that it's been around for a long time, and has had a lot of opportunities for performance improvement. On the other hand, glibc bugs me sometimes by taking liberties with non-standard features. musl aims for stronger standards compliance, but is much newer, which has made me wonder if it could support the performance we like to get from Varnish. But it's been a few years now, so maybe I shouldn't worry about it. Also, I recall that for a while the alpine build was unable to pass make check. Is that working now? Thanks again, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 840 bytes Desc: OpenPGP digital signature URL: From hrhosseini at hotmail.com Sat Aug 14 10:50:39 2021 From: hrhosseini at hotmail.com (Hamidreza Hosseini) Date: Sat, 14 Aug 2021 10:50:39 +0000 Subject: Best practice for caching scenario with different backend servers but same content In-Reply-To: References: Message-ID: Hi, Thanks to you and all varnish team for such answers that helped me alot, I read the default varnish cache configuration again: https://github.com/varnishcache/varnish-cache/blob/6.0/bin/varnishd/builtin.vcl and find out vcl_hash as follow: ``` sub vcl_hash { hash_data(req.url); if (req.http.host) { hash_data(req.http.host); } else { hash_data(server.ip); } return (lookup); } ``` So, if I change vcl_hash like following , would it be enough for my purpose?(I mean caching the same object from different backends just once with roundrobin directive !:) ``` sub vcl_hash { hash_data(req.url); return (lookup); } ``` By this config I told varnish just cache the content based on the 'req.url' not 'req.http.host' therefore with the same content but different backend varnish would cache once(If I want to use round robin directive instead of shard directive ), Is this true? what bad consequences may it cause in the future by this configuration? ________________________________ From: varnish-misc on behalf of Hamidreza Hosseini Sent: Sunday, August 1, 2021 4:17 AM To: varnish-misc at varnish-cache.org Subject: Best practice for caching scenario with different backend servers but same content Hi, I want to use varnish in my scenario as cache service, I have about 10 http servers that serve Hls fragments as the backend servers and about 5 varnish servers for caching purpose, the problem comes in when I use round-robin director for backend servers in varnish, if a varnish for specific file requests to one backend server and for the same file but to another backend server it would cache that file again because of different Host headers ! so my solution is using fallback director instead of round-robin as follow: ``` In varnish-1: new hls_cluster = directors.fallback(); hls_cluster.add_backend(b1()); hls_cluster.add_backend(b2()); hls_cluster.add_backend(b3()); hls_cluster.add_backend(b4()); hls_cluster.add_backend(b5()); hls_cluster.add_backend(b6()); hls_cluster.add_backend(b7()); hls_cluster.add_backend(b8()); hls_cluster.add_backend(b9()); hls_cluster.add_backend(b10()); In varnish-2: new hls_cluster = directors.fallback(); hls_cluster.add_backend(b10()); hls_cluster.add_backend(b1()); hls_cluster.add_backend(b2()); hls_cluster.add_backend(b3()); hls_cluster.add_backend(b4()); hls_cluster.add_backend(b5()); hls_cluster.add_backend(b6()); hls_cluster.add_backend(b7()); hls_cluster.add_backend(b8()); hls_cluster.add_backend(b9()); In varnish-3: new hls_cluster = directors.fallback(); hls_cluster.add_backend(b9()); hls_cluster.add_backend(b1()); hls_cluster.add_backend(b2()); hls_cluster.add_backend(b3()); hls_cluster.add_backend(b4()); hls_cluster.add_backend(b5()); hls_cluster.add_backend(b6()); hls_cluster.add_backend(b7()); hls_cluster.add_backend(b8()); hls_cluster.add_backend(b10()); ``` But I think this is not the best solution, because there is no load balancing despite, I used different backend for the first argument of fallback directive, What is varnish recommendation for this scenario? -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Sun Aug 15 14:55:37 2021 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Sun, 15 Aug 2021 07:55:37 -0700 Subject: Docker images: Alpine and new architecture support In-Reply-To: References: <6af5fbe2-9a5b-b2b9-bd33-562d261db06c@uplex.de> Message-ID: Replying to all this time On Fri, Aug 13, 2021, 08:48 Guillaume Quintard wrote: > > On Fri, Aug 13, 2021 at 3:31 AM Geoff Simmons wrote: > >> Can you (or anyone) share some info about how well Varnish performs with >> musl libc? >> > > Good question, and unfortunately, I don't have a good answer in return as > I haven't benchmarked it, so feedback is more than welcome. > > What I can comment on is that musl is quite adamant about being standard > and pure, so it will hopefully be more portable and will let compilers do > more work. As you mentioned, we've had issues in the past compiling and > testing with it, but it should be all behind us now: > - there were some header issues that prevented us from compiling, but we > fixed that a couple of years ago > - libbacktrace isn't available on Alpine, which prompted the move to > libunwind (can we make it the default now?) > - it has help fix a few compiler warning issues lately > > Cheers, > > -- > Guillaume Quintard > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Mon Aug 16 05:30:37 2021 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 16 Aug 2021 05:30:37 +0000 Subject: Best practice for caching scenario with different backend servers but same content In-Reply-To: References: Message-ID: On Sat, Aug 14, 2021 at 10:54 AM Hamidreza Hosseini wrote: > > Hi, > Thanks to you and all varnish team for such answers that helped me alot, > I read the default varnish cache configuration again: > https://github.com/varnishcache/varnish-cache/blob/6.0/bin/varnishd/builtin.vcl > and find out vcl_hash as follow: > > ``` > sub vcl_hash { > hash_data(req.url); > if (req.http.host) { > hash_data(req.http.host); > } else { > hash_data(server.ip); > } > return (lookup); > } > > ``` > So, if I change vcl_hash like following , would it be enough for my purpose?(I mean caching the same object from different backends just once with roundrobin directive !:) > > ``` > > sub vcl_hash { > hash_data(req.url); > return (lookup); > } > > ``` > > By this config I told varnish just cache the content based on the 'req.url' not 'req.http.host' therefore with the same content but different backend varnish would cache once(If I want to use round robin directive instead of shard directive ), Is this true? what bad consequences may it cause in the future by this configuration? In this case req.http.host usually refers to the the domain end users resolve to find your varnish server (or other hops in front of it). It is usually the same for every client, let's take www.myapp.com as an example. If your varnish server is in front of multiple services, you should be handling the different host headers explicitly. For exampe if you have exactly two domains you should normalize them to some canonical form. Using the same example domain that could be www.myapp.com and static.myapp.com for instance. In that case hashing the URL only would prevent you from adding new domains through your Varnish server. It won't hurt if you know you will only ever have one domain to deal with, but hashing the host will also not hurt as long as you normalize it to a unique value. You are correct that by default hashing the request appropriately will help the shard director do the right thing out of the box. I remember however that you only wanted to hash a subset of the URL for video segments, so hashing the URL as-is won't provide the behavior you are looking for. Dridi From hrhosseini at hotmail.com Mon Aug 16 13:34:26 2021 From: hrhosseini at hotmail.com (Hamidreza Hosseini) Date: Mon, 16 Aug 2021 13:34:26 +0000 Subject: Best practice for caching scenario with different backend servers but same content In-Reply-To: References: Message-ID: > In that case, hashing the URL only would prevent you from adding new domains through your Varnish server. It won't hurt if you know you will only ever have one domain to deal with, but hashing the host will also not hurt as long as you normalize it to a unique value. Hi, Let me elaborate my architecture more: I have some backend servers to serve hls fragments for video live stream,e.g: ``` hls_backend_01 hls_backend_02 hls_backend_03 hls_backend_04 hls_backend_05 hls_backend_06 hls_backend_07 hls_backend_08 hls_backend_09 hls_backend_10 ``` There is same content on all hls backend servers, there are 5 varnish in front of them for caching Now If I use round-robin director on Varnishes, because varnish would cache " req.http.host + req.url ", so for the same content but from different backends it would cache double! for example: if varnish for the first request and "test.ts" file goes to "hls_backend_01" backend server, would cache it and for the next request from other clients because it is using round-robin director it goes to "hls_backend_02" and would cache the same file again due to different "req.http.host" So now I have a solution to use Shard director based on "key=req.url" instead of round robin another way is to use round robin but adjusting the hash vcl to something like bellow: ``` sub vcl_hash { hash_data(req.url); return (lookup); } ``` In this way varnish just hash the "req.url" not "req.http.host" So, Varnish would cache the content based on the content uniqueness not based on the difference between backends. 1. At first, I asked how I can normalize it, Is it possible at all according to what I said!? Would you please explain it more with an example? 2. You give an example about other domains, In this case I do not understand what it has to do with the domain? 3.Maybe I'm thinking in wrong way because if varnish hash the data based on req.url : 'hash_data(req.url)' It shouldn't cache the same content but different backends again! for example my request is : http://varnish-01:/hls/test.ts for first request it goes to "hls_backend_01" backend and cache it and for next request it goes to "hls_backend_02" backend, so for each request it caches it again because backends are different? Many Thanks, Hamidreza ________________________________ From: varnish-misc on behalf of Dridi Boukelmoune Sent: Sunday, August 15, 2021 10:30 PM To: varnish-misc at varnish-cache.org Subject: Re: Best practice for caching scenario with different backend servers but same content On Sat, Aug 14, 2021 at 10:54 AM Hamidreza Hosseini wrote: > > Hi, > Thanks to you and all varnish team for such answers that helped me alot, > I read the default varnish cache configuration again: > https://github.com/varnishcache/varnish-cache/blob/6.0/bin/varnishd/builtin.vcl > and find out vcl_hash as follow: > > ``` > sub vcl_hash { > hash_data(req.url); > if (req.http.host) { > hash_data(req.http.host); > } else { > hash_data(server.ip); > } > return (lookup); > } > > ``` > So, if I change vcl_hash like following , would it be enough for my purpose?(I mean caching the same object from different backends just once with roundrobin directive !:) > > ``` > > sub vcl_hash { > hash_data(req.url); > return (lookup); > } > > ``` > > By this config I told varnish just cache the content based on the 'req.url' not 'req.http.host' therefore with the same content but different backend varnish would cache once(If I want to use round robin directive instead of shard directive ), Is this true? what bad consequences may it cause in the future by this configuration? In this case req.http.host usually refers to the the domain end users resolve to find your varnish server (or other hops in front of it). It is usually the same for every client, let's take www.myapp.com as an example. If your varnish server is in front of multiple services, you should be handling the different host headers explicitly. For exampe if you have exactly two domains you should normalize them to some canonical form. Using the same example domain that could be www.myapp.com and static.myapp.com for instance. In that case hashing the URL only would prevent you from adding new domains through your Varnish server. It won't hurt if you know you will only ever have one domain to deal with, but hashing the host will also not hurt as long as you normalize it to a unique value. You are correct that by default hashing the request appropriately will help the shard director do the right thing out of the box. I remember however that you only wanted to hash a subset of the URL for video segments, so hashing the URL as-is won't provide the behavior you are looking for. Dridi _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From marco.dickert at evolver.de Wed Aug 18 08:02:08 2021 From: marco.dickert at evolver.de (Marco Dickert - evolver group) Date: Wed, 18 Aug 2021 10:02:08 +0200 Subject: Varnish returns 503 error, because it "Could not get storage" Message-ID: <20210818080208.GA9730@marco.evolver.de> Hi all, I'm still investigating issues with one of our varnish instances. We use varnish as a cache and loadbalancer behind nginx and in front of a docker platform. We experienced an outage for about 20 minutes as clients received 503 errors being produced by varnish while the docker containers responded correct (according to the containers' logs). Setup is: [ nginx ==> varnish ] ==> [ docker swarm (4 hosts, lots of containers) ] Sites are distinguished by the exposed ports of the respective swarm services. Mapping site to service is done with a director containing the 4 hosts and the respective service port as backends. By comparing nginx logs with container logs we could confirm varnish being the culprit. It seemed like the backend request succeeds, but varnish returns a 503 error anyway. To investigate further, I activated some logging, which revealed some concerning information. Apparently varnish sometimes has problems with the storage, as the "FetchError" says "Could not get storage". ``` * << BeReq >> 70780723 - Begin bereq 70780722 pass [...] - Storage malloc Transient - Fetch_Body 2 chunked - - FetchError Could not get storage ``` I have attached two complete log examples to this mail. I did some extensive searching including the varnish book and stuff but so far did not come up with an explanation. Can anyone help understand why this happens and how to avoid it? Here are some additional information about our varnish instance: - Debian buster - system: HP DL360p G8, 32G RAM, Intel Xeon E5-2630 - varnish 6.6.0-1~buster (using the varnish repos) - varnish start options: ``` ExecStart=/usr/sbin/varnishd -a :6081 \ -T :6082 \ -f /etc/varnish/default.vcl \ -p ping_interval=6 -p cli_timeout=10 -p pipe_timeout=600 \ -p listen_depth=4096 -p thread_pool_min=200 -p thread_pool_max=500 -p workspace_client=128k -p nuke_limit=1000 -S /etc/varnish/secret \ -s malloc,12G \ -s Transient=malloc,3500M ``` Thanks in advance! -- Marco Dickert -------------- next part -------------- * << BeReq >> 1013636267 - Begin bereq 1013636266 pass - VCL_use xxx - Timestamp Start: 1629187432.080789 0.000000 0.000000 - BereqMethod GET - BereqURL /fonts/vendor/@fortawesome/fontawesome-pro/webfa-solid-900.woff2?978b27ec5d8b81d2b15aa28aaaae1fcb - BereqProtocol HTTP/1.0 - BereqHeader Host: xxx - BereqHeader X-Real-IP: 46.114.x.x - BereqHeader X-Forwarded-Proto: https - BereqHeader Front-End-Https: on - BereqHeader user-agent: Mozilla/5.0 (Android 10; Mobile; rv:91.0) Gecko/91.0 Firefox/91.0 - BereqHeader accept: application/font-woff2;q=1.0,application/font-woff;q=0.9,*/*;q=0.8 - BereqHeader accept-language: de-DE - BereqHeader accept-encoding: identity - BereqHeader referer: xxx - BereqHeader sec-fetch-dest: font - BereqHeader sec-fetch-mode: cors - BereqHeader sec-fetch-site: same-origin - BereqHeader dnt: 1 - BereqHeader sec-gpc: 1 - BereqHeader X-Forwarded-For: 46.114.x.x - BereqHeader cookie: __cmpcc=1; XSRF-TOKEN=xxxxxxxxxxxxxxxxxxxx - BereqHeader X-CMS-Environment: zeitungsportal - BereqHeader X-SEGMENT: GRP0 - BereqHeader X-iat: 1628771228 - BereqHeader x-token-status: expired - BereqHeader X-Return-URL: xxx - BereqProtocol HTTP/1.1 - BereqHeader X-Varnish: 1013636267 - VCL_call BACKEND_FETCH - VCL_return fetch - Timestamp Fetch: 1629187432.080825 0.000036 0.000036 - Timestamp Connected: 1629187432.080832 0.000043 0.000007 - BackendOpen 419 docker_1_8080 10.132.100.1 8080 192.168.11.32 40538 reuse - Timestamp Bereq: 1629187432.080880 0.000091 0.000047 - Timestamp Beresp: 1629187432.082310 0.001521 0.001429 - BerespProtocol HTTP/1.1 - BerespStatus 200 - BerespReason OK - BerespHeader Date: Tue, 17 Aug 2021 08:03:51 GMT - BerespHeader Server: Apache - BerespHeader Last-Modified: Tue, 03 Aug 2021 13:38:39 GMT - BerespHeader Accept-Ranges: bytes - BerespHeader Content-Length: 136824 - BerespHeader Cache-Control: max-age=31536000 - BerespHeader Expires: Wed, 17 Aug 2022 08:03:51 GMT - BerespHeader Access-Control-Allow-Origin: * - BerespHeader Content-Type: font/woff2 - VCL_call BACKEND_RESPONSE - BerespHeader x-host: xxx - BerespHeader x-url: /fonts/vendor/@fortawesome/fontawesome-pro/webfa-solid-900.woff2?978b27ec5d8b81d2b15aa28aaaae1fcb - BereqUnset cookie: __cmpcc=1; XSRF-TOKEN=xxxxxxxxxxxxxxxxxx - VCL_return deliver - Timestamp Process: 1629187432.082324 0.001535 0.000014 - Filters esi - Storage malloc Transient - Fetch_Body 3 length - - ESI_xmlerror No ESI processing, first char not '<'. (See feature esi_disable_xml_check) - FetchError Could not get storage - BackendClose 419 docker_1_8080 close - Timestamp Error: 1629187432.086258 0.005469 0.003933 - BerespProtocol HTTP/1.1 - BerespStatus 503 - BerespReason Backend fetch failed - BerespHeader Date: Tue, 17 Aug 2021 08:03:52 GMT - BerespHeader Server: Varnish - VCL_call BACKEND_ERROR - BerespHeader Content-Type: text/html; charset=iso8859-1 - VCL_return deliver - Storage malloc Transient - Length 17892 - BereqAcct 2859 0 2859 295 91928 92223 - End -------------- next part -------------- * << BeReq >> 73306576 - Begin bereq 73306575 fetch - VCL_use xxx - Timestamp Start: 1629187693.112352 0.000000 0.000000 - BereqMethod GET - BereqURL /media/Lesermarkt/Aboservice-Mailings/Coronavvirus_320x300.jpg - BereqProtocol HTTP/1.0 - BereqHeader Host: xxx - BereqHeader X-Real-IP: 92.117.x.x - BereqHeader X-Forwarded-Proto: https - BereqHeader Front-End-Https: on - BereqHeader user-agent: Mozilla/5.0 (Linux; Android 7.0; BLN-L21 Build/HONORBLN-L21; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/92.0.4515.131 Mobile Safari/537.36 - BereqHeader accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8 - BereqHeader x-requested-with: com.android.email - BereqHeader sec-fetch-site: cross-site - BereqHeader sec-fetch-mode: no-cors - BereqHeader sec-fetch-dest: image - BereqHeader accept-language: de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7 - BereqHeader X-Forwarded-For: 92.117.x.x - BereqHeader Accept-Encoding: gzip - BereqProtocol HTTP/1.1 - BereqHeader X-Varnish: 73306576 - VCL_call BACKEND_FETCH - VCL_return fetch - Timestamp Fetch: 1629187693.112391 0.000038 0.000038 - Timestamp Connected: 1629187693.112398 0.000045 0.000006 - BackendOpen 364 docker_1_8128 10.132.100.1 8128 192.168.11.32 56088 reuse - Timestamp Bereq: 1629187693.112428 0.000075 0.000030 - Timestamp Beresp: 1629187693.115675 0.003322 0.003246 - BerespProtocol HTTP/1.1 - BerespStatus 200 - BerespReason OK - BerespHeader Date: Tue, 17 Aug 2021 08:08:12 GMT - BerespHeader Server: Apache/2.4.38 (Debian) - BerespHeader Last-Modified: Tue, 22 Sep 2020 09:18:06 GMT - BerespHeader ETag: "155687-5afe371e05d32" - BerespHeader Accept-Ranges: bytes - BerespHeader Content-Length: 1398407 - BerespHeader Content-Type: image/jpeg - TTL RFC 120 10 0 1629187693 1629187693 1629187692 0 0 cacheable - VCL_call BACKEND_RESPONSE - BerespHeader x-host: xxx - BerespHeader x-url: /media/Lesermarkt/Aboservice-Mailings/Coronavvirus_320x300.jpg - TTL VCL 120 10 0 1629187693 uncacheable - VCL_return deliver - Timestamp Process: 1629187693.115703 0.003350 0.000028 - Filters esi - Storage malloc Transient - Fetch_Body 3 length - - ESI_xmlerror No ESI processing, first char not '<'. (See feature esi_disable_xml_check) - FetchError Could not get storage - BackendClose 364 docker_1_8128 close - Timestamp Error: 1629187693.120795 0.008442 0.005092 - BerespProtocol HTTP/1.1 - BerespStatus 503 - BerespReason Backend fetch failed - BerespHeader Date: Tue, 17 Aug 2021 08:08:13 GMT - BerespHeader Server: Varnish - VCL_call BACKEND_ERROR - BerespHeader Content-Type: text/html; charset=iso8859-1 - VCL_return deliver - Storage malloc Transient - Length 17892 - BereqAcct 662 0 662 237 137815 138052 - End From justinl at arena.net Thu Aug 19 20:11:30 2021 From: justinl at arena.net (Justin Lloyd) Date: Thu, 19 Aug 2021 20:11:30 +0000 Subject: Varnish and AWS ALBs Message-ID: Hi all, Is anyone else running Varnish behind AWS ALBs? I just encountered an issue today with how I have been using X-Forwarded-For to check against a Varnish ACL in that is more restrictive than the ALB's security group, but I realized the hard way that since X-Forwarded-For can be arbitrarily set, a malicious actor can set it to an address that is permitted by the Varnish ACL, whether through guessing or other knowledge. Since Varnish gets XFF from the ALB, which in turn trusts existing XFF headers, you can't then really trust client.ip since it's just taken from XFF. Unless I'm missing something... I've opened a support case with AWS to see if there's a way to configure an ALB to not trust XFF and use the IP from the original TCP connection, but I'm not hopeful. I'll likely have to go back to using two ALBs rather than one relatively open one and one with a Varnish ACL for tigher controls to a certain subset of the web sites behind the single ALB. Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlos.abalde at gmail.com Thu Aug 19 20:35:17 2021 From: carlos.abalde at gmail.com (Carlos Abalde) Date: Thu, 19 Aug 2021 22:35:17 +0200 Subject: Varnish and AWS ALBs In-Reply-To: References: Message-ID: <6B7E67FB-3886-4176-A2F7-08F8B1209418@gmail.com> Hi Justin, You cannot rely on the first IP in XFF (I guess you're doing that at the moment), but you can rely on the next-to-last. The last one is added to XFF by Varnish before entering 'vcl_recv', and the next-to-last is added by the ALB. That's the client IP as seen by the ALB and cannot be forged by clients. Best, -- Carlos Abalde > On 19 Aug 2021, at 22:11, Justin Lloyd wrote: > > Hi all, > > Is anyone else running Varnish behind AWS ALBs? I just encountered an issue today with how I have been using X-Forwarded-For to check against a Varnish ACL in that is more restrictive than the ALB?s security group, but I realized the hard way that since X-Forwarded-For can be arbitrarily set, a malicious actor can set it to an address that is permitted by the Varnish ACL, whether through guessing or other knowledge. Since Varnish gets XFF from the ALB, which in turn trusts existing XFF headers, you can?t then really trust client.ip since it?s just taken from XFF. Unless I?m missing something... > > I?ve opened a support case with AWS to see if there?s a way to configure an ALB to not trust XFF and use the IP from the original TCP connection, but I?m not hopeful. I?ll likely have to go back to using two ALBs rather than one relatively open one and one with a Varnish ACL for tigher controls to a certain subset of the web sites behind the single ALB. > > Justin > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From justinl at arena.net Thu Aug 19 20:39:16 2021 From: justinl at arena.net (Justin Lloyd) Date: Thu, 19 Aug 2021 20:39:16 +0000 Subject: Varnish and AWS ALBs In-Reply-To: <6B7E67FB-3886-4176-A2F7-08F8B1209418@gmail.com> References: <6B7E67FB-3886-4176-A2F7-08F8B1209418@gmail.com> Message-ID: Hi Carlos, Correct, but that?s my problem. There are only two IPs in XFF: the original sender (and thus the one that can be spoofed) and the ALB?s IP, which is basically useless. I realize I can?t trust the first IP, but the client IP being the ALB doesn?t help because I need to restrict by the actual source IP, which apparently (unless AWS support tells me differently) I can?t actually know for sure once the request is behind the ALB. If that is the case, then I?ll definitely have to go back to the two ALB solution since then the special Varnish ACL can be handled once again by the ALB?s SG. I don?t like having two ALBs to manage, but it seems it?s probably the only way to handle this situation. Justin From: Carlos Abalde Sent: Thursday, August 19, 2021 1:35 PM To: Justin Lloyd Cc: varnish-misc at varnish-cache.org Subject: Re: Varnish and AWS ALBs Hi Justin, You cannot rely on the first IP in XFF (I guess you're doing that at the moment), but you can rely on the next-to-last. The last one is added to XFF by Varnish before entering 'vcl_recv', and the next-to-last is added by the ALB. That's the client IP as seen by the ALB and cannot be forged by clients. Best, -- Carlos Abalde On 19 Aug 2021, at 22:11, Justin Lloyd > wrote: Hi all, Is anyone else running Varnish behind AWS ALBs? I just encountered an issue today with how I have been using X-Forwarded-For to check against a Varnish ACL in that is more restrictive than the ALB?s security group, but I realized the hard way that since X-Forwarded-For can be arbitrarily set, a malicious actor can set it to an address that is permitted by the Varnish ACL, whether through guessing or other knowledge. Since Varnish gets XFF from the ALB, which in turn trusts existing XFF headers, you can?t then really trust client.ip since it?s just taken from XFF. Unless I?m missing something... I?ve opened a support case with AWS to see if there?s a way to configure an ALB to not trust XFF and use the IP from the original TCP connection, but I?m not hopeful. I?ll likely have to go back to using two ALBs rather than one relatively open one and one with a Varnish ACL for tigher controls to a certain subset of the web sites behind the single ALB. Justin _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlos.abalde at gmail.com Thu Aug 19 20:48:47 2021 From: carlos.abalde at gmail.com (Carlos Abalde) Date: Thu, 19 Aug 2021 22:48:47 +0200 Subject: Varnish and AWS ALBs In-Reply-To: References: <6B7E67FB-3886-4176-A2F7-08F8B1209418@gmail.com> Message-ID: Hi, No so sure about that. Let's assume the client address is 1.1.1.1. Two possible scenarios: - The client request reaches the ALB without XFF. The ALB will inject XFF with value 1.1.1.1. Then Varnish will modify XFF adding the ALB's address (i.e., 1.1.1.1,). Using the next-to-last IP you're using the right client address. - The client request reaches the ALB with a forged XFF (e.g. 127.0.0.1). The ALB will will modify XFF (i.e. 127.0.0.1,1.1.1.1). The Varnish will do the same (i.e. 127.0.0.1,1.1.1.1,). Using the next-to-last IP you're still using the right client address. I've not checked using a ALB, but that should be the expected behaviour for me. Best, -- Carlos Abalde -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Thu Aug 19 20:59:56 2021 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Thu, 19 Aug 2021 13:59:56 -0700 Subject: Varnish and AWS ALBs In-Reply-To: References: <6B7E67FB-3886-4176-A2F7-08F8B1209418@gmail.com> Message-ID: Hi, If I read this correctly: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/x-forwarded-headers.html , you can trust the before-last IP, because it was added by the ALB, always. (and using vmod_str makes it easy to retrieve https://github.com/varnish/varnish-modules/blob/master/src/vmod_str.vcc#L42) Side question: would an NLB work? They support proxy-protocol, that would also solve your problem. Cheers, -- Guillaume Quintard On Thu, Aug 19, 2021 at 1:52 PM Carlos Abalde wrote: > Hi, > > No so sure about that. Let's assume the client address is 1.1.1.1. Two > possible scenarios: > > - The client request reaches the ALB without XFF. The ALB will inject XFF > with value 1.1.1.1. Then Varnish will modify XFF adding the ALB's address > (i.e., 1.1.1.1,). Using the next-to-last IP you're using the right > client address. > > - The client request reaches the ALB with a forged XFF (e.g. 127.0.0.1). > The ALB will will modify XFF (i.e. 127.0.0.1,1.1.1.1). The Varnish will do > the same (i.e. 127.0.0.1,1.1.1.1,). Using the next-to-last IP > you're still using the right client address. > > > I've not checked using a ALB, but that should be the expected behaviour > for me. > > Best, > > -- > Carlos Abalde > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From justinl at arena.net Thu Aug 19 21:39:03 2021 From: justinl at arena.net (Justin Lloyd) Date: Thu, 19 Aug 2021 21:39:03 +0000 Subject: Varnish and AWS ALBs In-Reply-To: References: <6B7E67FB-3886-4176-A2F7-08F8B1209418@gmail.com> Message-ID: Hi Guillaume! It looks like you and Carlos are both correct. For some reason, before I was not seeing the Varnish XFF values from faked XFFs, not sure why, but now I?m seeing the fakes I?m using against one of my dev sites and I?m seeing the three values where it?s FAKED_IP, REAL_IP, ALB_IP. So with a little bit more VCL code (or probably easier once I move to Varnish Enterprise next year), I should be able to handle this. I?ll give it a whirl and see how it goes. Thanks! Justin From: Guillaume Quintard Sent: Thursday, August 19, 2021 2:00 PM To: Carlos Abalde Cc: Justin Lloyd ; varnish-misc at varnish-cache.org Subject: Re: Varnish and AWS ALBs Hi, If I read this correctly: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/x-forwarded-headers.html , you can trust the before-last IP, because it was added by the ALB, always. (and using vmod_str makes it easy to retrieve https://github.com/varnish/varnish-modules/blob/master/src/vmod_str.vcc#L42) Side question: would an NLB work? They support proxy-protocol, that would also solve your problem. Cheers, -- Guillaume Quintard On Thu, Aug 19, 2021 at 1:52 PM Carlos Abalde > wrote: Hi, No so sure about that. Let's assume the client address is 1.1.1.1. Two possible scenarios: - The client request reaches the ALB without XFF. The ALB will inject XFF with value 1.1.1.1. Then Varnish will modify XFF adding the ALB's address (i.e., 1.1.1.1,). Using the next-to-last IP you're using the right client address. - The client request reaches the ALB with a forged XFF (e.g. 127.0.0.1). The ALB will will modify XFF (i.e. 127.0.0.1,1.1.1.1). The Varnish will do the same (i.e. 127.0.0.1,1.1.1.1,). Using the next-to-last IP you're still using the right client address. I've not checked using a ALB, but that should be the expected behaviour for me. Best, -- Carlos Abalde _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From hermunn at varnish-software.com Fri Aug 20 10:08:03 2021 From: hermunn at varnish-software.com (=?UTF-8?Q?P=C3=A5l_Hermunn_Johansen?=) Date: Fri, 20 Aug 2021 12:08:03 +0200 Subject: Varnish returns 503 error, because it "Could not get storage" In-Reply-To: <20210818080208.GA9730@marco.evolver.de> References: <20210818080208.GA9730@marco.evolver.de> Message-ID: Hello. I have not looked at the attachments, but you have limited Transient to 3500 MB. Getting "Could not get storage" should not be unexpected if a large enough amount of your transactions use Transient. You can figure out which transactions are transient by filtering on the Storage tag. Both varnishlog and varnishncsa (with a good formatting string and both -b and -c enabled) can be used for this. If no other alternative presents itself, maybe you need to switch to return (pipe) for some of your non-cacheable traffic just to save memory, but this disqualifies H2 and will give you a low connection reuse, so it is not optimal. Best, P?l ons. 18. aug. 2021 kl. 10:04 skrev Marco Dickert - evolver group < marco.dickert at evolver.de>: > Hi all, > > I'm still investigating issues with one of our varnish instances. We > use varnish as a cache and loadbalancer behind nginx and in front of a > docker platform. We experienced an outage for about 20 minutes as > clients received 503 errors being produced by varnish while the docker > containers responded correct (according to the containers' logs). > > Setup is: > > [ nginx ==> varnish ] ==> [ docker swarm (4 hosts, lots of containers) ] > > > Sites are distinguished by the exposed ports of the respective swarm > services. Mapping site to service is done with a director containing > the 4 hosts and the respective service port as backends. > > By comparing nginx logs with container logs we could confirm varnish > being the culprit. It seemed like the backend request succeeds, but > varnish returns a 503 error anyway. > > To investigate further, I activated some logging, which revealed some > concerning information. Apparently varnish sometimes has problems with > the storage, as the "FetchError" says "Could not get storage". > > ``` > * << BeReq >> 70780723 > - Begin bereq 70780722 pass > [...] > - Storage malloc Transient > - Fetch_Body 2 chunked - > - FetchError Could not get storage > ``` > > I have attached two complete log examples to this mail. > > I did some extensive searching including the varnish book and stuff but > so far did not come up with an explanation. Can anyone help understand > why this happens and how to avoid it? > > Here are some additional information about our varnish instance: > - Debian buster > - system: HP DL360p G8, 32G RAM, Intel Xeon E5-2630 > - varnish 6.6.0-1~buster (using the varnish repos) > - varnish start options: > > ``` > ExecStart=/usr/sbin/varnishd -a :6081 \ > -T :6082 \ > -f /etc/varnish/default.vcl \ > -p ping_interval=6 -p cli_timeout=10 -p pipe_timeout=600 \ > -p listen_depth=4096 -p thread_pool_min=200 > -p thread_pool_max=500 -p workspace_client=128k > -p nuke_limit=1000 -S /etc/varnish/secret \ > -s malloc,12G \ > -s Transient=malloc,3500M > ``` > > Thanks in advance! > > -- > Marco Dickert > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From justinl at arena.net Fri Aug 20 11:08:24 2021 From: justinl at arena.net (Justin Lloyd) Date: Fri, 20 Aug 2021 11:08:24 +0000 Subject: Varnish and AWS ALBs In-Reply-To: References: <6B7E67FB-3886-4176-A2F7-08F8B1209418@gmail.com> Message-ID: I was just trying to get varnish-modules to build (having to install varnish and build tools on my dev manager server) and hit the limitation that vmod_str isn?t available until Varnish 6.6. I?m on Varnish 6.5 so I?d need to test the 6.6 upgrade in dev and then roll that out to live, which will take some time (higher priority and urgency issues and projects on my plate). I?ll play with regsub() some more to see if I can figure out a temporary approach. Thanks, Justin From: varnish-misc On Behalf Of Justin Lloyd Sent: Thursday, August 19, 2021 2:39 PM To: Guillaume Quintard ; Carlos Abalde Cc: varnish-misc at varnish-cache.org Subject: RE: Varnish and AWS ALBs Hi Guillaume! It looks like you and Carlos are both correct. For some reason, before I was not seeing the Varnish XFF values from faked XFFs, not sure why, but now I?m seeing the fakes I?m using against one of my dev sites and I?m seeing the three values where it?s FAKED_IP, REAL_IP, ALB_IP. So with a little bit more VCL code (or probably easier once I move to Varnish Enterprise next year), I should be able to handle this. I?ll give it a whirl and see how it goes. Thanks! Justin From: Guillaume Quintard > Sent: Thursday, August 19, 2021 2:00 PM To: Carlos Abalde > Cc: Justin Lloyd >; varnish-misc at varnish-cache.org Subject: Re: Varnish and AWS ALBs Hi, If I read this correctly: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/x-forwarded-headers.html , you can trust the before-last IP, because it was added by the ALB, always. (and using vmod_str makes it easy to retrieve https://github.com/varnish/varnish-modules/blob/master/src/vmod_str.vcc#L42) Side question: would an NLB work? They support proxy-protocol, that would also solve your problem. Cheers, -- Guillaume Quintard On Thu, Aug 19, 2021 at 1:52 PM Carlos Abalde > wrote: Hi, No so sure about that. Let's assume the client address is 1.1.1.1. Two possible scenarios: - The client request reaches the ALB without XFF. The ALB will inject XFF with value 1.1.1.1. Then Varnish will modify XFF adding the ALB's address (i.e., 1.1.1.1,). Using the next-to-last IP you're using the right client address. - The client request reaches the ALB with a forged XFF (e.g. 127.0.0.1). The ALB will will modify XFF (i.e. 127.0.0.1,1.1.1.1). The Varnish will do the same (i.e. 127.0.0.1,1.1.1.1,). Using the next-to-last IP you're still using the right client address. I've not checked using a ALB, but that should be the expected behaviour for me. Best, -- Carlos Abalde _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlos.abalde at gmail.com Fri Aug 20 11:11:46 2021 From: carlos.abalde at gmail.com (Carlos Abalde) Date: Fri, 20 Aug 2021 13:11:46 +0200 Subject: Varnish and AWS ALBs In-Reply-To: References: <6B7E67FB-3886-4176-A2F7-08F8B1209418@gmail.com> Message-ID: <772041F6-DD47-49C6-90E6-CA6E74B92F4F@gmail.com> This is a possible regsub() to extract the next-to-last IP address (it assumes at lest two are available): set req.http.X-Client-Ip = regsub( req.http.X-Forwarded-For, "^.*(?:^|,)\s*([^,\s]+)\s*,[^,]+$", "\1"); Best, -- Carlos Abalde > On 20 Aug 2021, at 13:08, Justin Lloyd wrote: > > I was just trying to get varnish-modules to build (having to install varnish and build tools on my dev manager server) and hit the limitation that vmod_str isn?t available until Varnish 6.6. I?m on Varnish 6.5 so I?d need to test the 6.6 upgrade in dev and then roll that out to live, which will take some time (higher priority and urgency issues and projects on my plate). I?ll play with regsub() some more to see if I can figure out a temporary approach. > > Thanks, > Justin > > > From: varnish-misc On Behalf Of Justin Lloyd > Sent: Thursday, August 19, 2021 2:39 PM > To: Guillaume Quintard ; Carlos Abalde > Cc: varnish-misc at varnish-cache.org > Subject: RE: Varnish and AWS ALBs > > Hi Guillaume! > > It looks like you and Carlos are both correct. For some reason, before I was not seeing the Varnish XFF values from faked XFFs, not sure why, but now I?m seeing the fakes I?m using against one of my dev sites and I?m seeing the three values where it?s FAKED_IP, REAL_IP, ALB_IP. So with a little bit more VCL code (or probably easier once I move to Varnish Enterprise next year), I should be able to handle this. I?ll give it a whirl and see how it goes. > > Thanks! > > Justin > > > From: Guillaume Quintard > > Sent: Thursday, August 19, 2021 2:00 PM > To: Carlos Abalde > > Cc: Justin Lloyd >; varnish-misc at varnish-cache.org > Subject: Re: Varnish and AWS ALBs > > Hi, > > If I read this correctly: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/x-forwarded-headers.html , you can trust the before-last IP, because it was added by the ALB, always. (and using vmod_str makes it easy to retrieve https://github.com/varnish/varnish-modules/blob/master/src/vmod_str.vcc#L42 ) > > Side question: would an NLB work? They support proxy-protocol, that would also solve your problem. > > Cheers, > > -- > Guillaume Quintard > > > On Thu, Aug 19, 2021 at 1:52 PM Carlos Abalde > wrote: > Hi, > > No so sure about that. Let's assume the client address is 1.1.1.1. Two possible scenarios: > > - The client request reaches the ALB without XFF. The ALB will inject XFF with value 1.1.1.1. Then Varnish will modify XFF adding the ALB's address (i.e., 1.1.1.1,). Using the next-to-last IP you're using the right client address. > > - The client request reaches the ALB with a forged XFF (e.g. 127.0.0.1). The ALB will will modify XFF (i.e. 127.0.0.1,1.1.1.1). The Varnish will do the same (i.e. 127.0.0.1,1.1.1.1,). Using the next-to-last IP you're still using the right client address. > > I've not checked using a ALB, but that should be the expected behaviour for me. > > Best, > > -- > Carlos Abalde > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From justinl at arena.net Fri Aug 20 11:37:51 2021 From: justinl at arena.net (Justin Lloyd) Date: Fri, 20 Aug 2021 11:37:51 +0000 Subject: Varnish and AWS ALBs In-Reply-To: <772041F6-DD47-49C6-90E6-CA6E74B92F4F@gmail.com> References: <6B7E67FB-3886-4176-A2F7-08F8B1209418@gmail.com> <772041F6-DD47-49C6-90E6-CA6E74B92F4F@gmail.com> Message-ID: Hey Carlos, That seems to do the trick, thanks! I?ve always thought I was pretty good with PCRE (used Perl heavily for like 15 years) but for some reason this one was eluding me. I appreciate the help! Justin From: Carlos Abalde Sent: Friday, August 20, 2021 4:12 AM To: Justin Lloyd Cc: Guillaume Quintard ; varnish-misc at varnish-cache.org Subject: Re: Varnish and AWS ALBs This is a possible regsub() to extract the next-to-last IP address (it assumes at lest two are available): set req.http.X-Client-Ip = regsub( req.http.X-Forwarded-For, "^.*(?:^|,)\s*([^,\s]+)\s*,[^,]+$", "\1"); Best, -- Carlos Abalde On 20 Aug 2021, at 13:08, Justin Lloyd > wrote: I was just trying to get varnish-modules to build (having to install varnish and build tools on my dev manager server) and hit the limitation that vmod_str isn?t available until Varnish 6.6. I?m on Varnish 6.5 so I?d need to test the 6.6 upgrade in dev and then roll that out to live, which will take some time (higher priority and urgency issues and projects on my plate). I?ll play with regsub() some more to see if I can figure out a temporary approach. Thanks, Justin From: varnish-misc > On Behalf Of Justin Lloyd Sent: Thursday, August 19, 2021 2:39 PM To: Guillaume Quintard >; Carlos Abalde > Cc: varnish-misc at varnish-cache.org Subject: RE: Varnish and AWS ALBs Hi Guillaume! It looks like you and Carlos are both correct. For some reason, before I was not seeing the Varnish XFF values from faked XFFs, not sure why, but now I?m seeing the fakes I?m using against one of my dev sites and I?m seeing the three values where it?s FAKED_IP, REAL_IP, ALB_IP. So with a little bit more VCL code (or probably easier once I move to Varnish Enterprise next year), I should be able to handle this. I?ll give it a whirl and see how it goes. Thanks! Justin From: Guillaume Quintard > Sent: Thursday, August 19, 2021 2:00 PM To: Carlos Abalde > Cc: Justin Lloyd >; varnish-misc at varnish-cache.org Subject: Re: Varnish and AWS ALBs Hi, If I read this correctly: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/x-forwarded-headers.html , you can trust the before-last IP, because it was added by the ALB, always. (and using vmod_str makes it easy to retrieve https://github.com/varnish/varnish-modules/blob/master/src/vmod_str.vcc#L42) Side question: would an NLB work? They support proxy-protocol, that would also solve your problem. Cheers, -- Guillaume Quintard On Thu, Aug 19, 2021 at 1:52 PM Carlos Abalde > wrote: Hi, No so sure about that. Let's assume the client address is 1.1.1.1. Two possible scenarios: - The client request reaches the ALB without XFF. The ALB will inject XFF with value 1.1.1.1. Then Varnish will modify XFF adding the ALB's address (i.e., 1.1.1.1,). Using the next-to-last IP you're using the right client address. - The client request reaches the ALB with a forged XFF (e.g. 127.0.0.1). The ALB will will modify XFF (i.e. 127.0.0.1,1.1.1.1). The Varnish will do the same (i.e. 127.0.0.1,1.1.1.1,). Using the next-to-last IP you're still using the right client address. I've not checked using a ALB, but that should be the expected behaviour for me. Best, -- Carlos Abalde _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From marco.dickert at evolver.de Fri Aug 20 15:58:20 2021 From: marco.dickert at evolver.de (Marco Dickert - evolver group) Date: Fri, 20 Aug 2021 17:58:20 +0200 Subject: Varnish returns 503 error, because it "Could not get storage" In-Reply-To: References: <20210818080208.GA9730@marco.evolver.de> Message-ID: <20210820155819.GA7222@marco.evolver.de> Hi P?l, On 2021-08-20 12:08:03, P?l Hermunn Johansen wrote: > I have not looked at the attachments, but you have limited Transient to > 3500 MB. Getting "Could not get storage" should not be unexpected if a > large enough amount of your transactions use Transient. the problem is, that we cannot afford to not limit the transient storage, because of this jemalloc-Problem (see [1] and the answer from Reza, pointing to [2]). We currently limited s0 to 6GB and Transient to 3GB, however varnish uses 25GB in total. Using different jemalloc versions didn't help. Anyway, we will try to swap the limits and check if the problem persists or not, or how it possibly affects it. [1] https://varnish-cache.org/lists/pipermail/varnish-misc/2021-April/027022.html [2] https://github.com/varnishcache/varnish-cache/issues/3511 > You can figure out which transactions are transient by filtering on the > Storage tag. Both varnishlog and varnishncsa (with a good formatting string > and both -b and -c enabled) can be used for this. Apparently our ratio between transient and s0 is about 50:50. > If no other alternative presents itself, maybe you need to switch to return > (pipe) for some of your non-cacheable traffic just to save memory, but this > disqualifies H2 and will give you a low connection reuse, so it is not > optimal. We will consider this. Maybe we can use pipe for at least a subset of requests. So thank you very much for the hints! -- Marco Dickert