varnish 2.1.5 memory and swap

[David Birdsong]

On Wed, May 25, 2011 at 2:05 AM, Hettwer, Marian
<mhettwer at team.mobile.de> wrote:
> Hi Per and Stig,
>
>
>
> On 25.05.11 09:13, "Per Buer" <perbu at varnish-software.com> wrote:
>
>>On Tue, May 24, 2011 at 11:39 AM, Hettwer, Marian
>><mhettwer at team.mobile.de> wrote:
>>
>>
>>Reply to myself
>>
>>Although varnishd runs with malloc 6G, it seems that it's using much more
>>memory:
>> PID USER      PR  NI  VIRT  RES  SHR S %CPU COMMAND
>>
>>
>>
>>21693 nobody    20   0 12.5g 7.2g  80m S    0 varnishd
>>
>>
>>
>>
>>
>>Hu? Resistent at 7,2GB and Virtual at 12,5GB. Why?
>>
>>
>>
>>Virtual memory is, uhm, virtual. There is nothing tangible about it, so
>>you shouldn't really pay any attention to it.  Just spawning a thread
>>will take up a lot of virtual memory without hardly using any physical
>>memory. Stig has already answered your other question, I see.
>>
>
> Okay, I just ignore Virtual and have a look at Resistent.
> If I understood Stig correct, the behaviour of varnish is expected. Using
> 7,2GB RES mem, although malloc 6G was configured.
> I can live with that :)
>
> Still it's kinda odd, that Linux starts swapping out stuff, although RAM
> would have been sufficient. The machine has 8 gig ram and varnishd is
> using 7,2 gig.
> Linux decided to use nearly 1GB of swap.
> I hope that I can stop the kernel doing this by adding vm.swappiness=10...
> Let's see. If that doesn't do the trick, it seems that I have to lower the
> malloc 6G.

Check out cat /proc/sys/vm/min_free_kbytes; it's probably trying to
maintain that level.

Also, kswapd will run through malloc'd address spaces looking for
inactive pages to flush to the swap partition/file. So you may have
loaded a bunch of objects into the address space by pulling them
through varnish, but kswapd may decide that the object storage portion
of the address space is swap'able since they're not actively written
to (or read from?).

> Which I actually could happily do. I don't see any LRU_nuked objects.
>
> My vcl is configured to just cache specific urls. Every other url gets
> PASSed. Come to think of it. Looks like pipe would be better here?
> sub vcl_recv {
>
>        #
>        # always use this backend
>        set req.backend = febayk46;
>
>        # normalize accept-encoding header
>    if (req.http.Accept-Encoding) {
>        if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
>   # No point in compressing these
>            remove req.http.Accept-Encoding;
>        } elsif (req.http.Accept-Encoding ~ "gzip") {
>            set req.http.Accept-Encoding = "gzip";        } elsif
> (req.http.Accept-Encoding ~ "deflate" && req.http.user-agent !~ "MSIE") {
>            set req.http.Accept-Encoding = "deflate";
>        } else {            # unkown algorithm
>            remove req.http.Accept-Encoding;
>        }    }
>
>        if (    req.url ~ "^/anzeigen/s-beliebte-angebote.html.*$" ||
>                req.url ~ "^/anzeigen/s-suchbegriff-empfehlungen.html.*$"
> ||
>                req.url ~ "^/anzeigen/sitemap_.*$" ) {
>                # always cache urls above
>                unset req.http.Accept-Encoding;
>                return(lookup);
>        }
>
>        # static content caching
>        if (    req.url ~ "^/static/.*$" ) {
>                return(lookup);
>        }
>        if (    req.url ~ "^/REL-.*\.[\d]+/" ) {
>                return(lookup);
>        }
>
>
>
>                # don't cache the rest
>                return(pass);
>
>
> }
>
> sub vcl_fetch {
>        # backend behaves stupid and set-cookie on sitemap*.xml
>        # we remove this here
>        if (    req.url ~ "^/anzeigen/s-beliebte-angebote.html.*$" ||
>                req.url ~ "^/anzeigen/s-suchbegriff-empfehlungen.html.*$"
> ||
>                req.url ~ "^/anzeigen/sitemap_.*$" ) {
>                unset beresp.http.Set-Cookie;
>        }
>
> }
>
>
>
> Thanks for your answers, Stig and Per :)
> ./Marian
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> http://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>