From justinl at arena.net Mon Jul 26 10:29:09 2021 From: justinl at arena.net (Justin Lloyd) Date: Mon, 26 Jul 2021 10:29:09 +0000 Subject: Creating ACLs in Varnish Message-ID: Hi all, I'm wondering if there's a better way to handle defining ACLs in Varnish. Currently I have an AWS ALB fronting a set of web servers hosting several related sites. However, one of the sites (previously two, hence the two examples here) has a more restrictive ACL requirement than the others, but since an ALB can only use a single Security Group, I've had to implement the tighter ACL in Varnish. Basically it looks like this, assuming our main sites are in example.com and the special ones are under example.net: acl specialsite1_acl { "127.0.0.1/32"; "10.0.0.0/8"; "60.70.80.0/24"; # this is made up for example purposes } acl specialsite2_acl { "127.0.0.1/32"; "10.0.0.0/8"; "12.34.56.0/24"; # this is made up for example purposes } # I know the two if-statements could be joined, but this is programmatically generated from a Jinja template in Salt, # so it was cleaner to do it this way. sub check_acls { if (req.http.host ~ "^specialsite1.example.net$" && !std.ip(regsub(req.http.X-Forwarded-For, ",.*$", "")) ~ specialsite1_acl) { return (synth(403, "Access Forbidden")); } if (req.http.host ~ "^specialsite2.example.net$" && !std.ip(regsub(req.http.X-Forwarded-For, ",.*$", "")) ~ specialsite2_acl) { return (synth(403, "Access Forbidden")); } } sub vcl_recv { ... # Check the site-specific ACLs. if (req.http.host ~ "\.example\.net$") { call check_acls; } ... } Is there a better approach to this in Varnish Cache? We're also going to be evaluating Varnish Enterprise, so if there's something in VE, that would also be good to know. Thanks, Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailme.s at yandex.com Tue Jul 27 10:16:22 2021 From: mailme.s at yandex.com (s s) Date: Tue, 27 Jul 2021 14:16:22 +0400 Subject: Varnish Dynamic Page Caching & Cache Purging vs Nginx+Redis Message-ID: <298681627380770@mail.yandex.com> An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Tue Jul 27 14:42:33 2021 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Tue, 27 Jul 2021 07:42:33 -0700 Subject: Varnish Dynamic Page Caching & Cache Purging vs Nginx+Redis In-Reply-To: <298681627380770@mail.yandex.com> References: <298681627380770@mail.yandex.com> Message-ID: Hi, That's a very broad question, and so I'll keep the answer pretty high-level. All in all, Varnish has a lot fewer internal rules than nginx and really only cares about requests at an HTTP level. This means "dynamic" content doesn't matter to Varnish, it's just requests/objects with specific headers, querystrings, etc. As a result, I feel that Varnish is way better equipped to functionally handle any kind of traffic. Of course, because Varnish operates at a lower level, with fewer rules, it needs an excellent configuration scheme, and that's probably what trips people: the configuration language is actually a programming language that allows you to dictate very precisely how each request is handled. Here an article I wrote some weeks ago about this: https://info.varnish-software.com/blog/finally-understanding-built-in-vcl For purging, I won't mince my words: nginx is bad and you should stay away from it, it's limited and impractical. Varnish on the other side is once again very low-level and will force you to implement your own logic, but the primitives are much more powerful. And, lucky you, here's a ready-made VCL framework you can use: https://github.com/varnish/toolbox/tree/master/vcls/invalidate Hope this helps Cheers, -- Guillaume Quintard On Tue, Jul 27, 2021 at 3:17 AM s s wrote: > Hello all, > I am quite new to Varnish. I have been reading about both Varnish and > Nginx+Redis for page caching, and I am particularly interested in dynamic > caching and cache purging. I have read in a number of posts that Varnish > is "more flexible" in this regard, but without many additional details on > this. Can you please explain what features Varnish provides for dynamic > page caching and cache purging, especially which are not available (or are > more limited) in Nginx+Redis? Please forgive me if my question is very > basic/ignorant. As I said, I am new to Varnish. > > Thanks and Best Regards, > Sal > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailme.s at yandex.com Wed Jul 28 09:12:27 2021 From: mailme.s at yandex.com (s s) Date: Wed, 28 Jul 2021 13:12:27 +0400 Subject: Varnish Dynamic Page Caching & Cache Purging vs Nginx+Redis In-Reply-To: References: <298681627380770@mail.yandex.com> Message-ID: <100621627463329@mail.yandex.com> An HTML attachment was scrubbed... URL: From tom.anheyer at berlinonline.de Wed Jul 28 09:47:57 2021 From: tom.anheyer at berlinonline.de (Tom Anheyer | BerlinOnline) Date: Wed, 28 Jul 2021 09:47:57 +0000 Subject: Varnish Dynamic Page Caching & Cache Purging vs Nginx+Redis In-Reply-To: <298681627380770@mail.yandex.com> References: <298681627380770@mail.yandex.com> Message-ID: Hello, Varnish provides ESI (Edge Side Includes). By using ESI you can split your pages in static parts (cachable) and dynamic parts (not cacheable). Each ESI can be cached with different rules. In my opinion, it makes no sense to use redis as an HTTP cache. I see the task of redis mainly as an object cache in the backend. tom Am Dienstag, den 27.07.2021, 14:16 +0400 schrieb s s: Hello all, I am quite new to Varnish. I have been reading about both Varnish and Nginx+Redis for page caching, and I am particularly interested in dynamic caching and cache purging. I have read in a number of posts that Varnish is "more flexible" in this regard, but without many additional details on this. Can you please explain what features Varnish provides for dynamic page caching and cache purging, especially which are not available (or are more limited) in Nginx+Redis? Please forgive me if my question is very basic/ignorant. As I said, I am new to Varnish. Thanks and Best Regards, Sal _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From mailme.s at yandex.com Wed Jul 28 12:15:48 2021 From: mailme.s at yandex.com (s s) Date: Wed, 28 Jul 2021 16:15:48 +0400 Subject: Varnish Dynamic Page Caching & Cache Purging vs Nginx+Redis In-Reply-To: References: <298681627380770@mail.yandex.com> Message-ID: <169641627474184@mail.yandex.com> An HTML attachment was scrubbed... URL: From tom.anheyer at berlinonline.de Wed Jul 28 14:46:20 2021 From: tom.anheyer at berlinonline.de (Tom Anheyer | BerlinOnline) Date: Wed, 28 Jul 2021 14:46:20 +0000 Subject: Varnish Dynamic Page Caching & Cache Purging vs Nginx+Redis In-Reply-To: <169641627474184@mail.yandex.com> References: <298681627380770@mail.yandex.com> <169641627474184@mail.yandex.com> Message-ID: <9c524c09495a0cc8524eb3facf82465ff77b3b60.camel@BerlinOnline.de> Hi, in our case (berlin.de) we use mosltly the standard ttl of 2mins. So we have no need for a persistent cache storage. There a configuration tricks to combine two or more varnishd instances to act as a cluster with a "shared" cache. Maybe this helps: https://info.varnish-software.com/blog/creating-self-routing-varnish-cluster tom Am Mittwoch, den 28.07.2021, 16:15 +0400 schrieb s s: Hi Tom, Thanks for the tip on ESI. Indeed, Redis is more commonly used as an object cache. There are, however 3 features provided by Redis that I hope are available with Varnish, namely: 1) A single shared cache storage backend that can be used by multiple Nginx/etc. servers in a cluster. 2) Scalability (of storage capacity and of write+read throughput) and high availability via clustering with auto sharding & replication. 3) Both in-memory and persistent on-disk storage. This is useful, for example, in the event of a failure, after which the data persisted on-disk can be used to restart. Does Varnish provide the above capabilities (either on its own or in combination with other tools)? If not, is it on the roadmap? Thanks again for your help, Sal 28.07.2021, 13:49, "Tom Anheyer | BerlinOnline" : Hello, Varnish provides ESI (Edge Side Includes). By using ESI you can split your pages in static parts (cachable) and dynamic parts (not cacheable). Each ESI can be cached with different rules. In my opinion, it makes no sense to use redis as an HTTP cache. I see the task of redis mainly as an object cache in the backend. tom Am Dienstag, den 27.07.2021, 14:16 +0400 schrieb s s: Hello all, I am quite new to Varnish. I have been reading about both Varnish and Nginx+Redis for page caching, and I am particularly interested in dynamic caching and cache purging. I have read in a number of posts that Varnish is "more flexible" in this regard, but without many additional details on this. Can you please explain what features Varnish provides for dynamic page caching and cache purging, especially which are not available (or are more limited) in Nginx+Redis? Please forgive me if my question is very basic/ignorant. As I said, I am new to Varnish. Thanks and Best Regards, Sal _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc , _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -- Tom Anheyer Senior Developer BerlinOnline Stadtportal GmbH & Co. KG Alte Jakobstra?e 105 10969 Berlin Germany Tel.: +49 30 2327-5210 Fax: +49 30 2327-5596 E-Mail: tom.anheyer at berlinonline.de berlin.de | berlinonline.net Amtsgericht Berlin-Charlottenburg, HRA 31951 Sitz der Gesellschaft: Berlin, Deutschland USt-IdNr.: DE219483549 Pers?nlich haftender Gesellschafter: BerlinOnline Stadtportalbeteiligungsgesellschaft mbH Amtsgericht Berlin-Charlottenburg, HRB 79077 Sitz der Gesellschaft: Berlin, Deutschland Gesch?ftsf?hrung: Olf Dziadek -------------- next part -------------- An HTML attachment was scrubbed... URL: