bitrate and memory allocated

[Miguel González]

Dear all,

  I have Varnish 4.0.3 in front of a WHM/Cpanel hosting around 40
Wordpress sites. Only 4-5 sites are quite visited but the rest are blogs
that don´t have many visits.

  Varnish seems to work fine, I have increased the RAM of the VPS from 7
to 21 Gb of RAM because from time to time the RAM of the system got
exhausted and I had to restart the OS.

  I have signed up in datadoghq to get a handle of how well Varnish is
doing and try to squeeze the performance of it. Mostly because I have
realized that when I test any site with a tool like pingdom, the first
test gets a worse performance than consecutive tests (when I assume
Varnish starts caching).

  I have assigned in malloc 6 Gb of RAM but I have been checking the
numbers provided by varnishstat and after 12 hours I don´t even get
close to 1 Gb of RAM (I assume c_bytes is what stands for) used by varnish:

 varnishstat -1 |grep SMA
SMA.s0.c_req                             33969         0.76 Allocator
requests
SMA.s0.c_fail                                0         0.00 Allocator
failures
SMA.s0.c_bytes                       984052835     21896.06 Bytes allocated
SMA.s0.c_freed                       395844358      8807.89 Bytes freed
SMA.s0.g_alloc                           19833          .   Allocations
outstanding
SMA.s0.g_bytes                       588208477          .   Bytes
outstanding
SMA.s0.g_space                      5854242467          .   Bytes available

 I had a ttl of 2 hours and now I have changed it to 24 hours, maybe I
will increase it if I see the RAM allocated to Varnish is not entirely
used. The bitrate is erratic, reaching almost 1 and then dropping to
less than 0.5.

Are my assumptions right?

Another question, I know that running varnishtop I get the current most
visited site, but how do I aggregate that info over the hours? I would
like to get a picture of which are the most used sites (I´m considering
separating storages and dedicate only RAM for the most used and have
disk storage for the rest).

Thanks in advance

Miguel

ps: Here is my default.vcl

#
# This is an example VCL file for Varnish.
#
# It does not do anything by default, delegating control to the
# builtin VCL. The builtin VCL is called when there is no explicit
# return statement.
#
# See the VCL chapters in the Users Guide at
https://www.varnish-cache.org/docs/
# and http://varnish-cache.org/trac/wiki/VCLExamples for more examples.

# Marker to tell the VCL compiler that this VCL has been adapted to the
# new 4.0 format.
vcl 4.0;

import std;

# Default backend definition. Set this to point to your content server.
backend default {
    .host = "MYIP";
    .port = "82";
    .connect_timeout = 600s;
    .first_byte_timeout = 600s;
    .between_bytes_timeout = 600s;


}

acl purge {
        "localhost";
        "127.0.0.1";
        "MYIP";
}

# This function is used when a request is send by a HTTP client (Browser)
sub vcl_recv {

        # remove ?ver=xxxxx strings from urls so css and js files are
cached.
        # Watch out when upgrading WordPress, need to restart Varnish or
flush cache.
        set req.url = regsub(req.url, "\?ver=.*$", "");

        # Remove "replytocom" from requests to make caching better.
        set req.url = regsub(req.url, "\?replytocom=.*$", "");

        #We pass real IP to the backend

        if (req.restarts == 0) {
            if (req.http.X-Forwarded-For) {
               set req.http.X-Forwarded-For = req.http.X-Forwarded-For +
", " + client.ip;
           } else {
            set req.http.X-Forwarded-For = client.ip;
           }
        }


        # Normalize the header, remove the port (in case you're testing
this on various TCP ports)

        set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");

        # Remove has_js and CloudFlare/Google Analytics __* cookies.
        set req.http.Cookie = regsuball(req.http.Cookie,
"(^|;\s*)(_[_a-z]+|has_js)=[^;]*", "");
        # Remove a ";" prefix, if present.
        set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", "");


        # Allow purging from ACL
        if (req.method == "PURGE") {
                # If not allowed then a error 405 is returned
                if (!client.ip ~ purge) {
                        return(synth(405, "This IP is not allowed to
send PURGE requests."));
                }
                # If allowed, do a cache_lookup -> vlc_hit() or vlc_miss()
                return (purge);
        }

        # Post requests will not be cached
        #if (req.http.Authorization || req.method == "POST") {
        #       return (pass);
        #}

        # Pass anything other than GET and HEAD directly.
        if (req.method != "GET" && req.method != "HEAD") {
                return( pass );
        }      /* We only deal with GET and HEAD by default */

        # --- WordPress specific configuration

        # Did not cache the admin and login pages
        if (req.url ~
"nocache|cart|my-account|checkout|addons|tienda|carro|wp-admin|wp-(comments-post|login|signup|activate|mail|cron)\.php|preview\=true|admin-ajax\.php|xmlrpc\.php|bb-admin|whm-server-status|server-status|control\.php|bb-login\.php|bb-reset-password\.php|register\.php")
{
            return (pass);
        }

        if (req.url ~ "(ajax|dynamic|custom)") {
             return(pass);
        }

        # Remove the "has_js" cookie
        set req.http.Cookie = regsuball(req.http.Cookie, "has_js=[^;]+(;
)?", "");

        # Remove any Google Analytics based cookies
        set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(;
)?", "");

        # Remove the Quant Capital cookies (added by some plugin, all __qca)
        set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(;
)?", "");

        # Remove the wp-settings-1 cookie
        set req.http.Cookie = regsuball(req.http.Cookie,
"wp-settings-1=[^;]+(; )?", "");

        # Remove the wp-settings-time-1 cookie
        set req.http.Cookie = regsuball(req.http.Cookie,
"wp-settings-time-1=[^;]+(; )?", "");

        # Remove the wp test cookie
        set req.http.Cookie = regsuball(req.http.Cookie,
"wordpress_test_cookie=[^;]+(; )?", "");

        # Are there cookies left with only spaces or that are empty?
        if (req.http.cookie ~ "^ *$") {
                    unset req.http.cookie;
        }

        # Cache the following files extensions
        if (req.url ~ "\.(txt|css|js|png|gif|jp(e)?g|swf|ico)") {
                unset req.http.cookie;
        }

        # Normalize Accept-Encoding header and compression
        # https://www.varnish-cache.org/docs/3.0/tutorial/vary.html
        if (req.http.Accept-Encoding) {
                # Do no compress compressed files...
                if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
                                unset req.http.Accept-Encoding;
                } elsif (req.http.Accept-Encoding ~ "gzip") {
                        set req.http.Accept-Encoding = "gzip";
                } elsif (req.http.Accept-Encoding ~ "deflate") {
                        set req.http.Accept-Encoding = "deflate";
                } else {
                        unset req.http.Accept-Encoding;
                }
        }

        # Check the cookies for wordpress-specific items
        if (req.http.Cookie ~ "wordpress_" || req.http.Cookie ~
"comment_") {
                return (pass);
        }
        if (!req.http.cookie) {
                unset req.http.cookie;
        }

        # --- End of WordPress specific configuration

        # Did not cache HTTP authentication and HTTP Cookie
        if (req.http.Authorization || req.http.Cookie) {
                # Not cacheable by default
                return (pass);
        }

        # Cache all others requests
        return (hash);
}

sub vcl_pipe {
        return (pipe);
}

sub vcl_pass {
        return (fetch);
}

# The data on which the hashing will take place
sub vcl_hash {
        hash_data(req.url);
        if (req.http.host) {
        hash_data(req.http.host);
        } else {
        hash_data(server.ip);
        }

        # If the client supports compression, keep that in a different cache
        if (req.http.Accept-Encoding) {
                hash_data(req.http.Accept-Encoding);
        }

        return (lookup);
}

# This function is used when a request is sent by our backend (Nginx server)
sub vcl_backend_response {
        # Remove some headers we never want to see
        unset beresp.http.Server;
        unset beresp.http.X-Powered-By;

        # For static content strip all backend cookies
        if (bereq.url ~ "\.(css|js|png|gif|jp(e?)g)|swf|ico") {
                unset beresp.http.cookie;
        }
        # Don't store backend
        if (bereq.url ~ "wp-(login|admin)" || bereq.url ~ "preview=true") {
                set beresp.uncacheable = true;
                set beresp.ttl = 30s;
                return (deliver);
        }

        # Only allow cookies to be set if we're in admin area
                if (!(bereq.url ~
"(wp-login|cart|my-account|checkout|addons|tienda|carro|wp-admin|preview=true)"))
{
                unset beresp.http.set-cookie;
        }

        # don't cache response to posted requests or those with basic auth
        if ( bereq.method == "POST" || bereq.http.Authorization ) {
                set beresp.uncacheable = true;
                set beresp.ttl = 120s;
                return (deliver);
        }

        # don't cache search results
        if ( bereq.url ~ "\?s=" ){
                set beresp.uncacheable = true;
                set beresp.ttl = 120s;
                return (deliver);
        }

        # only cache status ok
        if ( beresp.status != 200 ) {
                set beresp.uncacheable = true;
                set beresp.ttl = 120s;
                return (deliver);
        }

        # A TTL of 24h
        set beresp.ttl = 24h;
        # Define the default grace period to serve cached content
        #set beresp.grace = 30s;
        set beresp.grace = 1h;

        return (deliver);
}

# The routine when we deliver the HTTP request to the user
# Last chance to modify headers that are sent to the client
sub vcl_deliver {
        if (obj.hits > 0) {
                set resp.http.X-Cache = "cached";
        } else {
                set resp.http.x-Cache = "uncached";
        }

        # Remove some headers: PHP version
        unset resp.http.X-Powered-By;

        # Remove some headers: Apache version & OS
        unset resp.http.Server;

        # Remove some heanders: Varnish
        unset resp.http.Via;
        unset resp.http.X-Varnish;

        return (deliver);
}

sub vcl_init {
        return (ok);
}

sub vcl_fini {
        return (ok);
}