Varnish virtual memory usage

[Darryl Dixon - Winterhouse Consulting]

Can you paste your VCL please? Do you use the purge_url function in it?

regards,
Darryl Dixon
Winterhouse Consulting Ltd
http://www.winterhouseconsulting.com

> I don't remember how long it took exactly, but it filled up rather
> quickly. Probably a couple of hours. I also noticed the LRU-moved stat was
> pretty constant all along, no large spikes towards the end of the swap.
>
> Anyways, it was WAY over 1GB, we managed to fill more around 6-7GB.
>
> Just for kicks, I tried changing the storage to malloc, and limit it to
> 1GB, to see the difference. We have live traffic now, so I don't wanna run
> the load testing scripts, but we should hit the 1GB limit within a day or
> two.
>
> -----Original Message-----
> From: Kristian Lyngstol [mailto:kristian at redpill-linpro.com]
> Sent: Wednesday, October 21, 2009 3:28 PM
> To: Roi Avinoam
> Cc: varnish-misc at projects.linpro.no
> Subject: Re: Varnish virtual memory usage
>
> On Wed, Oct 21, 2009 at 03:07:00PM +0200, Roi Avinoam wrote:
>> Thanks for your reply.
>>
>> 1. What I did was create 100 simultaneous processes, and each process
>> requested the same page (with 'curl'):
>> 	a. Once with the exact same URL - which resulted in a 99.9% hit-ratio
>> and VERY high performance on varnish.
>> 	b. And once with a random key that changes the URL (something like
>> 'index.php?rand=193837364'), thus forcing Varnish to hit the backend,
>> and store the multiple objects in memory.
>> 	c. A combination of the two - in an attempt to maintain a 60-70%
>> hit-ratio.
>>
>> What we saw is that the kernel simply filled all of the RAM *and* the
>> swap until it crashed.
>
> When did it fill up and how fast?
>
>> 2. I'm sorry, but I'm still confused about the mmaped file. If it's
>> limited to 1G, Varnish shouldn't use more than 1G of virtual memory,
>> correct? Or in our setup - 1G of RAM?
>
> The -s parameter only applies to storage, which is to say that it's an
> approximiation and there will be some additional overhead which is not
> stored there, ie datastructures for threads, sessions, backends etc.
>
> So with -s ..., 1G, you should expect a little bit over 1GB, but not
> nearly as much as you describe.
>
> --
> Kristian Lyngstøl
> Redpill Linpro AS
> Tlf: +47 21544179
> Mob: +47 99014497
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc
>