varnish 2.1.5 memory and swap

[Hettwer, Marian]

Hi Per and Stig,



On 25.05.11 09:13, "Per Buer" <perbu at varnish-software.com> wrote:

>On Tue, May 24, 2011 at 11:39 AM, Hettwer, Marian
><mhettwer at team.mobile.de> wrote:
>
>
>Reply to myself
>
>Although varnishd runs with malloc 6G, it seems that it's using much more
>memory:
> PID USER      PR  NI  VIRT  RES  SHR S %CPU COMMAND
>
>
>
>21693 nobody    20   0 12.5g 7.2g  80m S    0 varnishd
>
>
>
>
>
>Hu? Resistent at 7,2GB and Virtual at 12,5GB. Why?
>
>
>
>Virtual memory is, uhm, virtual. There is nothing tangible about it, so
>you shouldn't really pay any attention to it.  Just spawning a thread
>will take up a lot of virtual memory without hardly using any physical
>memory. Stig has already answered your other question, I see.
>

Okay, I just ignore Virtual and have a look at Resistent.
If I understood Stig correct, the behaviour of varnish is expected. Using
7,2GB RES mem, although malloc 6G was configured.
I can live with that :)

Still it's kinda odd, that Linux starts swapping out stuff, although RAM
would have been sufficient. The machine has 8 gig ram and varnishd is
using 7,2 gig.
Linux decided to use nearly 1GB of swap.
I hope that I can stop the kernel doing this by adding vm.swappiness=10...
Let's see. If that doesn't do the trick, it seems that I have to lower the
malloc 6G.
Which I actually could happily do. I don't see any LRU_nuked objects.

My vcl is configured to just cache specific urls. Every other url gets
PASSed. Come to think of it. Looks like pipe would be better here?
sub vcl_recv {

        # 
        # always use this backend
        set req.backend = febayk46;

        # normalize accept-encoding header
    if (req.http.Accept-Encoding) {
        if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
   # No point in compressing these
            remove req.http.Accept-Encoding;
        } elsif (req.http.Accept-Encoding ~ "gzip") {
            set req.http.Accept-Encoding = "gzip";        } elsif
(req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
            set req.http.Accept-Encoding = "deflate";
        } else {            # unkown algorithm
            remove req.http.Accept-Encoding;
        }    }

        if (    req.url ~ "^/anzeigen/s-beliebte-angebote.html.*$" ||
                req.url ~ "^/anzeigen/s-suchbegriff-empfehlungen.html.*$"
||
                req.url ~ "^/anzeigen/sitemap_.*$" ) {
                # always cache urls above
                unset req.http.Accept-Encoding;
                return(lookup);
        }

        # static content caching
        if (    req.url ~ "^/static/.*$" ) {
                return(lookup);
        }
        if (    req.url ~ "^/REL-.*\.[\d]+/" ) {
                return(lookup);
        }



                # don't cache the rest
                return(pass);
    

}

sub vcl_fetch {
        # backend behaves stupid and set-cookie on sitemap*.xml
        # we remove this here
        if (    req.url ~ "^/anzeigen/s-beliebte-angebote.html.*$" ||
                req.url ~ "^/anzeigen/s-suchbegriff-empfehlungen.html.*$"
||
                req.url ~ "^/anzeigen/sitemap_.*$" ) {
                unset beresp.http.Set-Cookie;
        }

}



Thanks for your answers, Stig and Per :)
./Marian