<div dir="ltr">Hi Justin, happy new year!<div><br></div><div>Without getting too much in the details, it should look like a basic shell command with a few pipes. Splunk for example has the universal forwarder that is going to push logs to the server where you can then review and search for the ingested logs.</div><div><br></div><div>The main issue is to push something meaningful to the log collector, and this is where things are a bit lacking, mainly because it's better to push structured info, and varnish isn't great at it yet.</div><div><br></div><div>For example, for logs, you have about three choices:</div><div>- varnishncsa, treat each line as a string and be done with it. It's not amazing as you'll be forced to use regex to filter requests, since you just logged a string</div><div>- varnishncsa -j, it's better, you can carefully craft a format line to look like an LDJSON object, and now the log analyzer (I know splunk does it, at least) will allow you to look for "resp.status == 200 && req.url ~ /foo//". The annoyance is that you need to explicitly decide which headers you want to log, and the format line/file is going to be disgustingly verbose and painful to maintain.<br></div><div>- enterprise varnishlog has support for LDJSON output, which is great and is as comprehensive as you can get. It could be too verbose (i.e. storage heavy), it's only in Varnish Enterprise, and it'll log everything, including the headers that got deleted/modified.<br></div><div><br></div><div>I believe that what we need is a JSON logger that just log a synthetic view of the transaction, something like this for example<br><br>{</div><div> "req": {</div><div> "method": "GET",</div><div> "url": "/foo.html",</div><div> "headers": [ {"name": "host", "value": "<a href="http://example.com">example.com</a>" }, {"name": "accept-encoding", "value": "gzip"} ],</div><div> "start_time": 123456789,</div><div> "end_time": 123456790,</div><div> "bytes": { "headers": 67, "body": 500, "total": 567 },<br></div><div> "processing": "miss",<br></div><div> },</div><div> "resp": {...},</div><div> "bereq": {...},<br></div><div>}</div><div><br></div><div>we have all the information in varnishlog, it's just a matter of formatting it correctly. With that, you have something that's easily filtered and is more natural and comprehensive than what we currently have.<br></div><div><br></div><div>It turns out it's been on my mind for a while, and I intend to get on it, but for now I'm having way too much fun with rust, vmods and backends to promise any commitment.</div><div>HOWEVER, if somebody wants to code some C/rust to scratch that itch, I'll be happy to lend a hand!</div><div><br></div><div>Does this make sense?<br></div><div><br></div><div><div><div dir="ltr" data-smartmail="gmail_signature"><div dir="ltr"><div>-- <br></div><div>Guillaume Quintard<br></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jan 10, 2023 at 2:22 PM Justin Lloyd <<a href="mailto:justinl@arena.net" target="_blank">justinl@arena.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>
<div lang="EN-US">
<div>
<p class="MsoNormal">Hi all,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">I need to centralize logs of multiple Varnish servers in my web server environments, generally just 4 or 6 servers depending on the environment. I’d like to be able to do this either with Splunk or an Amazon OpenSearch cluster, i.e., a
managed ELK stack. However, not having worked with either tool for such a purpose, I’m not clear on how I could then review, replay, etc. the centralized logs similar to the output from tools like
<i>varnishlog</i> and <i>varnishtop</i>. Are there existing tools for handling Varnish logs in these kinds of centralized log management systems, or would I be somewhat constrained on what I could do with the stored logs? Aside from the benefit of unifying
the logs across all of my web servers, I am trying to reduce how much I need to log in to the individual log servers to monitor ongoing issues, etc.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">FWIW, I haven’t checked how much log data our production web servers generate in a day, but when I checked several years ago (before moving into AWS and when the sites were much smaller), it was on the order of like 1 GB per day per server.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
<p class="MsoNormal">Justin<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org" target="_blank">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
</div></blockquote></div>