Varnish 2.06 running away with memory

[Scott Wilson]

Hi Sam,

On 18 May 2010 10:47, Sam Crawford <samcrawford at gmail.com> wrote:
> Morning all,
>
> A bit of a newbie question I fear... We've been successfully running
> Varnish 2.04 for over a year now on our intranet, and have found in
> the past two days that our instances in one particular region are
> eating up a lot of system memory. We've upgraded to 2.06 a couple of
> days ago, and are still experiencing the same issue.
>

> n_purge                246879          .   N total active purges
> n_purge_add            252285         5.61 N new purges added
> n_purge_retire           5406         0.12 N old purges deleted
> n_purge_obj_test       245731         5.46 N objects tested
> n_purge_re_test     214813964      4773.11 N regexps tested against
> n_purge_dups             3227         0.07 N duplicate purges removed
> hcb_nolock                  0         0.00 HCB Lookups without lock
> hcb_lock                    0         0.00 HCB Lookups with lock
> hcb_insert                  0         0.00 HCB Inserts
> esi_parse                   0         0.00 Objects ESI parsed (unlock)
> esi_errors                  0         0.00 ESI parse errors (unlock)
>

It looks to me like you're generating an awful lot of purge list
entries.  We struggled for a long time with this problem before
abandoning purge in favour of setting object TTLs to 0.  See this FAQ
entry:

http://varnish-cache.org/wiki/VCLExamplePurging

The problem we were having was that we were using purge to implement
force-refresh functionality, but there are so many badly behaved
crawlers and other clients out there that send always send the NoCache
header that the purge list would swamp all available resources.

Hope that helps!  cheers,
scott