Varnish returns 503 error, because it "Could not get storage"

[Marco Dickert - evolver group]

Hi Pål,

On 2021-08-20 12:08:03, Pål Hermunn Johansen wrote:
> I have not looked at the attachments, but you have limited Transient to
> 3500 MB. Getting "Could not get storage" should not be unexpected if a
> large enough amount of your transactions use Transient.

the problem is, that we cannot afford to not limit the transient storage,
because of this jemalloc-Problem (see [1] and the answer from Reza, pointing to
[2]). We currently limited s0 to 6GB and Transient to 3GB, however
varnish uses 25GB in total. Using different jemalloc versions didn't help.
Anyway, we will try to swap the limits and check if the problem persists or not,
or how it possibly affects it.

[1] https://varnish-cache.org/lists/pipermail/varnish-misc/2021-April/027022.html
[2] https://github.com/varnishcache/varnish-cache/issues/3511

> You can figure out which transactions are transient by filtering on the
> Storage tag. Both varnishlog and varnishncsa (with a good formatting string
> and both -b and -c enabled) can be used for this.

Apparently our ratio between transient and s0 is about 50:50.

> If no other alternative presents itself, maybe you need to switch to return
> (pipe) for some of your non-cacheable traffic just to save memory, but this
> disqualifies H2 and will give you a low connection reuse, so it is not
> optimal.

We will consider this. Maybe we can use pipe for at least a subset of requests.

So thank you very much for the hints!

-- 
Marco Dickert