From batanun at hotmail.com Thu Jan 9 14:55:51 2025 From: batanun at hotmail.com (Batanun B) Date: Thu, 9 Jan 2025 14:55:51 +0000 Subject: Varnish suddenly started using much more memory In-Reply-To: References:

Message-ID: Hi Guillaume, This time it is I who needs to apologize for a late reply (7 months later). We are strictly on the LTS diet, I'm afraid. So we can't upgrade the major version until there is a new LTS series of Varnish. So it's 6.0.x for a while still I guess. We are on Ubuntu 20 though, and I guess we could update to the latest LTS version there. ________________________________ From: Guillaume Quintard Sent: Thursday, June 13, 2024 10:07 PM To: Batanun B Cc: varnish-misc at varnish-cache.org Subject: Re: Varnish suddenly started using much more memory Sorry Batanun, this thread got lost in my inbox. Would you be able to upgrade to 7.5 and see if you get the same results? I'm pretty sure it's a jemalloc issue, but upgrading should make it clear. You are on Ubuntu, right? Which version? -- Guillaume Quintard On Mon, May 20, 2024 at 1:50?AM Batanun B > wrote: > Sorry, I should have been clearer, I meant: where are the varnish packages coming from? Are they from the official repositories, from https://packagecloud.io/varnishcache/ or built from source maybe? Ah, I see. They come from varnishcache packagecloud. More specifically, we use: https://packagecloud.io/install/repositories/varnishcache/varnish60lts/script.deb.sh > you should really invest some time in something like prometheus, it would probably have made the issue obvious Yes, in hindsight we definitely should have done that. I will discuss this with my coworkers going forward. > Is there any chance you can run the old version on the server to explore the differences? Possibly, for a limited time. If so, what types of tests would I do? And how long time would I need to run the old version? Note that with our setup, we wouldn't be able to run two different images at the same time, in the same environment, with both recieving traffic. So all traffic would be routed to this version (multiple servers, but all running the same image). An alternative approach that I'm considering, is to switch to the old image, but manually update the VCL to the new version. If the problem remains, then the issue is almost certainly with the VLC. But if the problem disapears, then it's more likely something else. > what's the output of: varnishstat -1 -f '*g_bytes' SMA.default.g_bytes 10951750929 . Bytes outstanding SMA.large.g_bytes 8587329728 . Bytes outstanding SMA.Transient.g_bytes 3177920 . Bytes outstanding So, the default storage usage has gone up with 2GB since my first message here, while the others have remained the same. Meanwhile, the total memory usage of Varnish has gone up to 26 GB, an increase of 3 GB. So now the overhead has gone up with 1GB to a total of 6 GB. Going forward, it will be interesting to see how the memory consumption changes after the default storage has reached its max (2 GB from where it is now). If we're lucky, it will stabilize, and then I'm not sure if it's worth it to troubleshoot any further. Otherwise, the free memory would get a bit too close to zero for our comfort, with no indication of stopping. Does Varnish keep track of total available OS memory, and start releasing memory by throwing out objects from the cache? Or will it continue to eat memory until something fails? > have you tweaked any workspaces/thread parameters? Nope. As I said, we haven't changed any OS or Varnish configuration. _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From batanun at hotmail.com Thu Jan 9 14:58:27 2025 From: batanun at hotmail.com (Batanun B) Date: Thu, 9 Jan 2025 14:58:27 +0000 Subject: Use Varnish to serve static files directly from disk? Message-ID: Hi, We use Varnish 6.0 (LTS), and we have the need to serve static files directly from disk. I would prefer not having to add another service (like nginx) and add it as a backend to varnish. If possible we would like to make Varnish serve the static files directly. We already do that for the html page for 500-errors, using this in vcl_backend_error: synthetic(std.fileread("/etc/varnish/500.html")); return (deliver); Can we do something similar for regular content? Say that any path that starts with "/static-files/" gets "routed" to a file on disk under /some-files/. So a request for "/static-files/abc/123/x.txt" means it will serve the file "/some-files/abc/123/x.txt". And if the file doesn't exist we would like to serve a static html-file for the 404 error. Is this is doable, and sensible? If so, is this documented somewhere? For the time being, we don't really need to update the files, so maybe std.fileread is enough (if I understand it correctly it will only read one file once, then cache it forever)? But eventually people likely will want to update existing files, and without us having to restart Varnish. Would it make sense to start with std.fileread, or should we start with something else from the start? If the latter, is the "file" vmod the recommended way? The documentation mentions requiring the master branch of Varnish, which I'm not sure what that means. We use 6.0 (LTS) and we are not willing to move away from LTS. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Wed Jan 22 18:45:08 2025 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Wed, 22 Jan 2025 10:45:08 -0800 Subject: [RFC] using environment variables for configuration Message-ID: Hello everyone! One of my goals for 2025 is to make Varnish easier to operate Varnish in dynamic environments, so I'm ready to open a big can of worms: configuration through environment variables. Yes, env variables are terrible, but they are the primary configuration mode for containers and they also allow users to redefine part of the configuration without having to rewrite the whole command line. Notably, docker images and helm charts use them to configure varnishd but they have to reimplement everything all the time (variable names, ordering, etc.), and having the feature in core would simplify downstream integration and bring some standardization, lowering the barrier of entry. In practice, I have relatively modest initial goals: - have varnishd read a handful of variables such as VARNISH_LISTEN_HTTP, VARNISH_STORAGE_MALLOC, VARNISH_VCL_PATH, VARNISH_HTTP2, and VARNISH_PARAMETER_* for all the -p arguments - those would be processed before regular command line arguments - possibly scan the environment looking for unsupported VARNISH_* variables to detect user error This is a pretty vague plan right now, but I'd rather telegraph it than implement it and be blocked during the PR review :-) So, how heretical am I being here? -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: