varnish 2.1.4 swap and increase in objects:heads.

[Per Buer]

Hi.

The issues are actually a bit different. One uses "-s file" which
really makes the kernel handle the memory management implicitly in
comparison to you Rodrigo, which uses -s malloc, where you clearly
state that Varnish should allocate this much memory.

Regarding -s malloc we recommend that on Linux one should not use more
then 75% of the physical memory for -s malloc. In addition to the
actual object store Varnish uses quite some memory for it's internal
data structures. If you go above 75% chances are that Linux will start
swapping a bit which might have a negative effect on overall
performance. Tollef is getting back to you outside the list regarding
the issue you've raised with us.

Per.

On Fri, Oct 29, 2010 at 1:59 PM, Rodrigo Benzaquen
<rodrigo at mercadolibre.com> wrote:
> Hi Taylan, exactly the same is happening to us, we already open a ticket
> with Varnish software.
>
> Please let me know if you find a solution.
>
> Thanks
> Rodrigo
>
>
>
>
> ------------------
>
> Rodrigo Benzaquen
> @rbenzaquen
>
>
>
> ________________________________
> From: "Taylan Develioglu" <tdevelioglu at ebuddy.com>
> To: varnish-misc at varnish-cache.org
> Sent: Viernes, 29 de Octubre 2010 8:37:21
> Subject: varnish 2.1.4 swap and increase in objects:heads.
>
> Debian Lenny, Linux varnish0 2.6.33.7 #1 SMP Thu Sep 2 15:27:53 CEST
> 2010 x86_64 GNU/Linux
>
> I have a strange situation on one of our varnish servers.
>
> We recently raised the expiry on previously uncached content a
> (Cache-Control: max-age=0, to Cache-Control: max-age=5).
>
> The number of objects vs. heads increased and the system started
> swapping (vm.swappiness=0).
>
> http://lcontent.ebuddy.com/varnish/memory-week.png
> http://lcontent.ebuddy.com/varnish/varnish_objects-week.png
> http://lcontent.ebuddy.com/varnish/varnish_expunge-week.png
>
>
> I don't understand why this is happening, bug maybe ?
>
> DAEMON_OPTS="-a :27535 \
>              -T localhost:6082 \
>              -f /etc/varnish/default.vcl \
>              -s file,/var/cache/varnish/sda/varnish_storage.bin,135G \
>              -s file,/var/cache/varnish/sdb/varnish_storage.bin,135G \
>              -s file,/var/cache/varnish/sdc/varnish_storage.bin,135G \
>              -s file,/var/cache/varnish/sdd/varnish_storage.bin,135G \
>              -s file,/var/cache/varnish/sde/varnish_storage.bin,135G \
>              -s file,/var/cache/varnish/sdf/varnish_storage.bin,135G \
>              -p thread_pools=4 \
>              -p thread_pool_min=200 \
>              -p thread_pool_max=2400 \
>              -p thread_pool_add_delay=2 \
>              -p lru_interval=20"
>
>
>
> ---vcl---
> sub vcl_recv {
>     unset req.http.cookie;
>     unset req.http.Authorization;
>
>     if (req.http.host ~ "^.*host.domain.com.*") {
>         set req.http.host = "host.domain.com";
>     }
>
>     if (req.request == "HEAD") {
>         return(pass);
>     }
>
>     if (req.request == "PURGE") {
>         if (!client.ip ~ purge) {
>             error 405 "Not allowed.";
>         }
>         return(lookup);
>     }
>
>     if (req.request == "GET" && req.url == "/monitor") {
>         error 200 "OK.";
>     }
>
>     if (req.backend.healthy) {
>         set req.grace = 30s;
>   } else {
>         set req.grace = 48h;
>     }
>
> }
>
>  sub vcl_fetch {
>     if (beresp.status == 404 || beresp.status >= 300) {
>          return (pass);
>     }
>     set beresp.grace = 48h;
>
>  }
>
> sub vcl_hit {
>     if (req.request == "PURGE") {
>         set obj.ttl = 0s;
>         error 200 "Purged.";
>     }
> }
>
> sub vcl_miss {
>     if (req.request == "PURGE") {
>         error 404 "Not in cache.";
>     }
> }
> ---EOF---
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://lists.varnish-cache.org/mailman/listinfo/varnish-misc
>



-- 
Per Buer,  Varnish Software
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer