Varnish memory usage

[Luca Gervasi]

Hello,
I'm following an issue that caused some OOMs during a huge spike.
Currently I have the following properties set:

 -p thread_pool_max=2000 \
-p thread_pool_min=35 \
-p feature=+esi_disable_xml_check \
-p feature=+esi_ignore_other_elements \
-p feature=+esi_remove_bom \
-p feature=+http2 \
-p pcre_match_limit_recursion=64 \
-p syslog_cli_traffic=off \
-p sigsegv_handler=on \
-p workspace_client=2m \
-p workspace_backend=2m \
-p http_max_hdr=128 \
-p http_req_hdr_len=32k \
-p http_resp_hdr_len=32k \
-p http_req_size=2m \
-p http_resp_size=2m \
-p thread_pool_stack=256k \
-s malloc,20G \
-S /config/varnish/secret \
-f /config/varnish/default.vcl \
-p vsl_reclen=700

And I'm using this formula to somehow calculate how much memory is varnish
going to use:
thread_pool_max*thread_pool_stack+thread_pool_max*(workspace_client+http_req_hdr_len+http_req_size)
+
malloc

And it is somewhere close to where I get OOM. I suspect that what did screw
up was the high esi-to-request ration that I normally have:

MAIN.client_req        469.22
MAIN.esi_req          5489.25

Hit rate on that esi requests is pretty high as we use high ttls and
invalidations, but I think that those ESIs have to be taken into account
too (as well as the backend requests too...).

I wasn't able to find an official memory sizing guide, sorry. How do you
size correctly memory usage?
I know I should use MSE, at the moment is not on my list.

Thanks
Luca
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20240701/282d37ea/attachment.html>