Varnish over memory allocation

[Alex Wakefield]

Hey Dridi,

Thanks for the reply. I originally thought that it might've been transient storage as well however checking that shows that only 287MB of storage has been allocated from it. Its allocated and released 1.06TB though, is that perhaps the cause of the issue?

SMA.s0.g_bytes shows the correct size of 23.99GB allocated which is what I expect however a pmap of the varnish process shows 29977920K used, 6GB over what was set.

Cheers,
Alex

On Mon, 25 May 2020, at 8:02 PM, Dridi Boukelmoune wrote:
> On Mon, May 25, 2020 at 8:07 AM Alex Wakefield
> <AlexWakefield at fastmail.com.au> wrote:
> >
> > Whoops, knew I forgot to specify something!
> >
> > We're using malloc. Command line switch is specifically `-s malloc,24GB`
> 
> The -s option only specifies the storage size (HTTP responses with
> some metadata). The rest of Varnish's memory footprint goes on top,
> things like loaded VCLs, ongoing VCL transactions, all kinds of data
> structures. VMODs like XKey may add their own footprint on top, the
> list goes on.
> 
> Even on the storage side, if you only declare a malloc storage like
> you did, you will get an unlimited Transient storage by default for
> short-lived or uncacheable responses.
> 
> The only way today to tell a Varnish instance to limit itself to 24GB
> (and still on a best-effort basis) is with Varnish Enterprise's memory
> governor.
> 
> Dridi
>