From justinl at arena.net Tue Jan 10 22:20:10 2023 From: justinl at arena.net (Justin Lloyd) Date: Tue, 10 Jan 2023 22:20:10 +0000 Subject: Centralizing varnish logs Message-ID: Hi all, I need to centralize logs of multiple Varnish servers in my web server environments, generally just 4 or 6 servers depending on the environment. I'd like to be able to do this either with Splunk or an Amazon OpenSearch cluster, i.e., a managed ELK stack. However, not having worked with either tool for such a purpose, I'm not clear on how I could then review, replay, etc. the centralized logs similar to the output from tools like varnishlog and varnishtop. Are there existing tools for handling Varnish logs in these kinds of centralized log management systems, or would I be somewhat constrained on what I could do with the stored logs? Aside from the benefit of unifying the logs across all of my web servers, I am trying to reduce how much I need to log in to the individual log servers to monitor ongoing issues, etc. FWIW, I haven't checked how much log data our production web servers generate in a day, but when I checked several years ago (before moving into AWS and when the sites were much smaller), it was on the order of like 1 GB per day per server. Thanks, Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Wed Jan 11 07:51:49 2023 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Tue, 10 Jan 2023 23:51:49 -0800 Subject: Centralizing varnish logs In-Reply-To: References: Message-ID: Hi Justin, happy new year! Without getting too much in the details, it should look like a basic shell command with a few pipes. Splunk for example has the universal forwarder that is going to push logs to the server where you can then review and search for the ingested logs. The main issue is to push something meaningful to the log collector, and this is where things are a bit lacking, mainly because it's better to push structured info, and varnish isn't great at it yet. For example, for logs, you have about three choices: - varnishncsa, treat each line as a string and be done with it. It's not amazing as you'll be forced to use regex to filter requests, since you just logged a string - varnishncsa -j, it's better, you can carefully craft a format line to look like an LDJSON object, and now the log analyzer (I know splunk does it, at least) will allow you to look for "resp.status == 200 && req.url ~ /foo//". The annoyance is that you need to explicitly decide which headers you want to log, and the format line/file is going to be disgustingly verbose and painful to maintain. - enterprise varnishlog has support for LDJSON output, which is great and is as comprehensive as you can get. It could be too verbose (i.e. storage heavy), it's only in Varnish Enterprise, and it'll log everything, including the headers that got deleted/modified. I believe that what we need is a JSON logger that just log a synthetic view of the transaction, something like this for example { "req": { "method": "GET", "url": "/foo.html", "headers": [ {"name": "host", "value": "example.com" }, {"name": "accept-encoding", "value": "gzip"} ], "start_time": 123456789, "end_time": 123456790, "bytes": { "headers": 67, "body": 500, "total": 567 }, "processing": "miss", }, "resp": {...}, "bereq": {...}, } we have all the information in varnishlog, it's just a matter of formatting it correctly. With that, you have something that's easily filtered and is more natural and comprehensive than what we currently have. It turns out it's been on my mind for a while, and I intend to get on it, but for now I'm having way too much fun with rust, vmods and backends to promise any commitment. HOWEVER, if somebody wants to code some C/rust to scratch that itch, I'll be happy to lend a hand! Does this make sense? -- Guillaume Quintard On Tue, Jan 10, 2023 at 2:22 PM Justin Lloyd wrote: > Hi all, > > > > I need to centralize logs of multiple Varnish servers in my web server > environments, generally just 4 or 6 servers depending on the environment. > I?d like to be able to do this either with Splunk or an Amazon OpenSearch > cluster, i.e., a managed ELK stack. However, not having worked with either > tool for such a purpose, I?m not clear on how I could then review, replay, > etc. the centralized logs similar to the output from tools like > *varnishlog* and *varnishtop*. Are there existing tools for handling > Varnish logs in these kinds of centralized log management systems, or would > I be somewhat constrained on what I could do with the stored logs? Aside > from the benefit of unifying the logs across all of my web servers, I am > trying to reduce how much I need to log in to the individual log servers to > monitor ongoing issues, etc. > > > > FWIW, I haven?t checked how much log data our production web servers > generate in a day, but when I checked several years ago (before moving into > AWS and when the sites were much smaller), it was on the order of like 1 GB > per day per server. > > > > Thanks, > > Justin > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martynas at atomgraph.com Thu Jan 26 17:30:00 2023 From: martynas at atomgraph.com (=?UTF-8?Q?Martynas_Jusevi=C4=8Dius?=) Date: Thu, 26 Jan 2023 18:30:00 +0100 Subject: varnish:6.0.11 Docker image crashing on Apple M1 processor Message-ID: Hi, We have a Docker image based on varnish:6.0.11. A user on Apple M1 processor is reporting a crash: https://github.com/AtomGraph/LinkedDataHub/issues/149 Error: Message from VCC-compiler: Assert error in vju_subproc(), mgt/mgt_jail_unix.c line 212: Condition((initgroups(vju_user, vju_gid)) == 0) not true. errno = 1 (Operation not permitted) qemu: uncaught target signal 6 (Aborted) - core dumped Running VCC-compiler failed, signal 6 VCL compilation failed Do you provide images with linux/arm64/v8 support as well? Or what is the course of action here? Thanks. Martynas From guillaume.quintard at gmail.com Thu Jan 26 17:49:48 2023 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Thu, 26 Jan 2023 09:49:48 -0800 Subject: varnish:6.0.11 Docker image crashing on Apple M1 processor In-Reply-To: References: Message-ID: Hi, I'm not sure what is going in on here as we do have arm64v8 official images: https://github.com/docker-library/official-images/blob/master/library/varnish Could it just be a permissions issue? -- Guillaume Quintard On Thu, Jan 26, 2023 at 9:32 AM Martynas Jusevi?ius wrote: > Hi, > > We have a Docker image based on varnish:6.0.11. > > A user on Apple M1 processor is reporting a crash: > https://github.com/AtomGraph/LinkedDataHub/issues/149 > > Error: > Message from VCC-compiler: > Assert error in vju_subproc(), mgt/mgt_jail_unix.c line 212: > Condition((initgroups(vju_user, vju_gid)) == 0) not true. > errno = 1 (Operation not permitted) > qemu: uncaught target signal 6 (Aborted) - core dumped > Running VCC-compiler failed, signal 6 > VCL compilation failed > > Do you provide images with linux/arm64/v8 support as well? Or what is > the course of action here? > > Thanks. > > Martynas > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martynas at atomgraph.com Thu Jan 26 20:58:57 2023 From: martynas at atomgraph.com (=?UTF-8?Q?Martynas_Jusevi=C4=8Dius?=) Date: Thu, 26 Jan 2023 21:58:57 +0100 Subject: varnish:6.0.11 Docker image crashing on Apple M1 processor In-Reply-To: References: Message-ID: Hi Guillaume, I reproduced the same error as well, running Terminal in Rosetta on MacOS. Can it be a permissions issue if the same exact docker-compose setup runs fine on Windows? Here's our Dockerfile: https://github.com/AtomGraph/varnish/blob/master/Dockerfile Martynas On Thu, Jan 26, 2023 at 6:50 PM Guillaume Quintard wrote: > > Hi, > > I'm not sure what is going in on here as we do have arm64v8 official images: https://github.com/docker-library/official-images/blob/master/library/varnish > > Could it just be a permissions issue? > > -- > Guillaume Quintard > > > On Thu, Jan 26, 2023 at 9:32 AM Martynas Jusevi?ius wrote: >> >> Hi, >> >> We have a Docker image based on varnish:6.0.11. >> >> A user on Apple M1 processor is reporting a crash: >> https://github.com/AtomGraph/LinkedDataHub/issues/149 >> >> Error: >> Message from VCC-compiler: >> Assert error in vju_subproc(), mgt/mgt_jail_unix.c line 212: >> Condition((initgroups(vju_user, vju_gid)) == 0) not true. >> errno = 1 (Operation not permitted) >> qemu: uncaught target signal 6 (Aborted) - core dumped >> Running VCC-compiler failed, signal 6 >> VCL compilation failed >> >> Do you provide images with linux/arm64/v8 support as well? Or what is >> the course of action here? >> >> Thanks. >> >> Martynas >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From dridi at varni.sh Thu Jan 26 21:44:55 2023 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 26 Jan 2023 21:44:55 +0000 Subject: varnish:6.0.11 Docker image crashing on Apple M1 processor In-Reply-To: References: Message-ID: Hi Martynas, On Thu, Jan 26, 2023 at 9:00 PM Martynas Jusevi?ius wrote: > > Hi Guillaume, > > I reproduced the same error as well, running Terminal in Rosetta on MacOS. > > Can it be a permissions issue if the same exact docker-compose setup > runs fine on Windows? The problem with containers is the uncertainty regarding the host operating system running them, so something working in a container on a specific platform does not guarantee the container will run fine everywhere. > Here's our Dockerfile: > https://github.com/AtomGraph/varnish/blob/master/Dockerfile Try this: ENTRYPOINT ["/usr/local/bin/docker-varnish-entrypoint", "-jnone"] Arguably, the container system already applies the principle of least privileges, so we don't know Varnish's jail feature. Who knows? Not me. Cheers, Dridi