Centralizing varnish logs

[Justin Lloyd]

Hi all,

I need to centralize logs of multiple Varnish servers in my web server environments, generally just 4 or 6 servers depending on the environment. I'd like to be able to do this either with Splunk or an Amazon OpenSearch cluster, i.e., a managed ELK stack. However, not having worked with either tool for such a purpose, I'm not clear on how I could then review, replay, etc. the centralized logs similar to the output from tools like varnishlog and varnishtop. Are there existing tools for handling Varnish logs in these kinds of centralized log management systems, or would I be somewhat constrained on what I could do with the stored logs? Aside from the benefit of unifying the logs across all of my web servers, I am trying to reduce how much I need to log in to the individual log servers to monitor ongoing issues, etc.

FWIW, I haven't checked how much log data our production web servers generate in a day, but when I checked several years ago (before moving into AWS and when the sites were much smaller), it was on the order of like 1 GB per day per server.

Thanks,
Justin

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230110/18096ca2/attachment.html>