Varnish 2.0.3 consuming excessive memory

Harald Friessnegger harald at webmeisterei.com
Wed Apr 8 13:42:59 CEST 2009


hi darryl

i had a simlar problem (varnish child process consuming lots of memory) a week 
ago (see the thread "make varnish don't start a subprocess")

the solution for my problem seems to be to use a high-enough cache-size:

starting varnish with

  -s file,"/tmp/storage",300M

will make the child process use *about* 800M RAM (the RES column in top) after 
some hours. after that i needed to restart varnish to free some ram for my 
webserver.


starting with a bigger cache

  -s file,"/tmp/storage",600M

makes it using about 500M RAM (the RES column in top after 5 days of uptime) 
and happily coexist with zope ;-)



maybe this helps you too.

regards, fRiSi




Am Mittwoch, 8. April 2009 05:05:36 schrieb Darryl Dixon - Winterhouse 
Consulting:
> > Not that I have an answer, but I'd be curious to see the differences
> > in 'pmap -x <pid>' output for the different children.
> >
> > --Michael
>
> That's a good thought - I'd almost forgotten about pmap. Anyhow, the
> 'normal' process is set up with a 2GB storage cache and pmap shows pretty
> much what you would expect - the largest anonymous segment is 41MB, with
> the vast majority of allocated memory booked against storage.cache (2GB).
> The pmap for the 'runaway' process shows 256MB booked against
> storage.cache, and then one of the an anonymous allocations is 2.3GB. For
> example:
>
>
> Normal process:
> ------------------------------------------------------
> 3896:   /u01/app/varnish/sbin/varnishd -a web-cms1.prd.internal:8080
> web-cms.prd.internal:8080 -T web-cms1.prd.internal:18080 -t 108000 -f
> /u01/data/prd/varnish/vcl.conf -s
> file,/u01/data/prd/varnish/storage.cache,2048M -n
> /u01/data/prd/varnish/state -P /u01/data/prd/varnish/varnishd.pid
> Address           Kbytes     RSS    Anon  Locked Mode   Mapping
> 0000000000400000     312       -       -       - r-x--  varnishd
> 000000000064d000       8       -       -       - rw---  varnishd
> 000000000064f000      68       -       -       - rw---    [ anon ]
> 0000000040000000       4       -       -       - -----    [ anon ]
> 0000000040001000   10240       -       -       - rw---    [ anon ]
> [...snip...]
> 00002aaab0900000 2097152       -       -       - rw-s-  storage.cache
> 00002aab30900000      28       -       -       - r-x--  vcl.1P9zoqAU.so
> 00002aab30907000    2044       -       -       - -----  vcl.1P9zoqAU.so
> 00002aab30b06000       8       -       -       - rw---  vcl.1P9zoqAU.so
> 00002aab30c00000    2056       -       -       - rw---    [ anon ]
> 00002aab30f00000    2056       -       -       - rw---    [ anon ]
> 00002aab31200000   41984       -       -       - rw---    [ anon ]
> 00007fffae8f2000      88       -       -       - rw---    [ stack ]
> ffffffffff600000    8192       -       -       - -----    [ anon ]
> ----------------  ------  ------  ------  ------
> total kB         2468512       -       -       -
> ------------------------------------------------------
>
>
>
> Runaway process:
> ------------------------------------------------------
> 3940:   /u01/app/varnish/sbin/varnishd -a web-cms-edit1.prd.internal:8080
> web-cms-edit.prd.internal:8080 -T web-cms-edit1.prd.internal:18080 -t 3600
> -f /u01/data/prd-edit/varnish/vcl.conf -s
> file,/u01/data/prd-edit/varnish/storage.cache,256M -n
> /u01/data/prd-edit/varnish/state -P
> /u01/data/prd-edit/varnish/varnishd.pid
> Address           Kbytes     RSS    Anon  Locked Mode   Mapping
> 0000000000400000     312       -       -       - r-x--  varnishd
> 000000000064d000       8       -       -       - rw---  varnishd
> 000000000064f000      68       -       -       - rw---    [ anon ]
> 0000000040000000       4       -       -       - -----    [ anon ]
> 0000000040001000   10240       -       -       - rw---    [ anon ]
> [...snip...]
> 00002aaab0900000  262144       -       -       - rw-s-  storage.cache
> 00002aaac0900000      28       -       -       - r-x--  vcl.1P9zoqAU.so
> 00002aaac0907000    2048       -       -       - -----  vcl.1P9zoqAU.so
> 00002aaac0b07000       8       -       -       - rw---  vcl.1P9zoqAU.so
> 00002aaac0c00000    2056       -       -       - rw---    [ anon ]
> 00002aaac0f00000    2052       -       -       - rw---    [ anon ]
> 00002aaac1200000  233476       -       -       - rw---    [ anon ]
> 00002aaacf700000 2337792       -       -       - rw---    [ anon ]
> 00007fff4b619000      84       -       -       - rw---    [ stack ]
> ffffffffff600000    8192       -       -       - -----    [ anon ]
> ----------------  ------  ------  ------  ------
> total kB         3121808       -       -       -
> ------------------------------------------------------
>
>
> Both processes have been running for the same amount of time - 1day
> 17hours.
>
> regards,
> Darryl Dixon
> Winterhouse Consulting Ltd
> http://www.winterhouseconsulting.com
>
> > On Apr 7, 2009, at 6:27 PM, Darryl Dixon - Winterhouse Consulting wrote:
> >>> Hi All,
> >>>
> >>> I have an odd problem that I have only noticed happening since
> >>> moving from
> >>> 1.1.2 to 2.0.3 - excessive memory consumption of the varnish child
> >>> process. For example, I have a varnish instance with a 256MB cache
> >>> allocated, that is currently consuming 4.9GB of resident memory
> >>> (6.5GB
> >>> virtual). The instance has only been running for 4 days and has
> >>> only got
> >>> 25MB of objects in the cache.
> >>>
> >>> This is clearly excessive and is causing us some serious problems
> >>> in terms
> >>> of memory pressure on the machine. Our VCL is largely unchanged
> >>> from our
> >>> 1.1.2 setup to the 2.0.3 except for the obvious vcl.syntax changes,
> >>> and
> >>> the introduction of request restarts under certain limited
> >>> scenarios. Can
> >>> anyone shed some light?
> >>
> >> One further footnote to this. I have a second varnish instance
> >> running on
> >> the same machine which talks to different backend servers (still
> >> primarily
> >> the same sort of content though), with the VCL only fractionally
> >> different
> >> - it does not seem to suffer from the runaway memory consumption of
> >> the
> >> first instance. The only difference in the VCL between the two is
> >> that in
> >> the one with runaway memory this is present in vcl_recv():
> >>
> >> +    if (req.http.Pragma ~ ".*no-cache.*" || req.http.Cache-Control ~
> >> ".*no-cache.*") {
> >> +        purge_url(regsub(req.url, "[?].*$", ".*$"));
> >> +    }
> >> +
> >>
> >> Is there possibly something in the regsub engine being triggered
> >> that is
> >> very expensive and would cause it to consume and hold on to large
> >> amounts
> >> of memory?
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc



-- 
Webmeisterei GmbH - Büro für Netzfragen
Tel: +43 5572 908877,  Fax: +43 5572 908877-66
Steinebach 18, A-6850 Dornbirn

http://www.webmeisterei.com



More information about the varnish-misc mailing list