Monitor per site in Varnish

[Paul A. Procacci]

On Wed, Oct 10, 2012 at 06:09:44PM +0200, Johnny Halfmoon wrote:
>    On 09/28/2012 12:02 AM, Johan Olsson wrote:
>
>    Hi
>
>    I've been looking on how to monitor varnish. I've found that there
>    exists a snmp for varnish which gives some info that is good to have.
>    I've found it and looked at it
>    ([1]http://sourceforge.net/projects/varnishsnmp/), but it dosen't give
>    all that I need (I think). What I'm missing is to be able to monitor
>    how much traffic one site is using. So if I have two sites like
>    [2]www.example1.com and [3]www.example2.com, I would like to be able to
>    get how many connections each one gets and how much Mbps each one is
>    using.
>
>    Is this possible to do?

<snip>

>    Hi Johan,
>    Maybe I can help. I've got about 35 sites running on a 4 node Varnish
>    cluster here and monitor throughput, request rate and http status codes
>    per-site using Cacti and Nagios via SNMP . The way it works is like
>    this:
>    Each server is running the exact same varnish config. In this config
>    there's a VCL chunk that defines a bunch of macro's for gathering site
>    info:

<snip>

The snmp features are very useful no doubt.  I understand the OP opened his
question in relation to snmp, but later asked two very simple questions.
The first was regarding how many connections and the second was how much
Mbps _per site_.

For a simple task like this, I'd suggest awstats.  In fact, I do this
currrently for client of mine whom has currently 104 sites sitting behind
his varnish instance.  Here is how we tackled it:

- I've got varnishncsa running with the following:
varnishncsa -D -P /var/run/varnishncsa.pid -a -c -w /var/log/varnishncsa.log -F '%{Host}i %h %l %u %t \"%r\" %s %b \"%{Referer}i\" \"%{User-agent}i\"'

- newsyslog running to rotate at a specific time:
/var/log/varnishncsa.log       644 10       * $D3  BR /usr/local/scripts/run_awstats.sh

- The following shell script that newsyslog invokes as per the above:
########################################
LOGFILE=/var/log/varnishncsa.log.0

kill -HUP `cat /var/run/varnishncsa.pid`

count=0
for i in `ls /usr/local/www/awstats/cgi-bin/*www\.*`; do
  i=${i##*/awstats\.};
  i=${i%%\.conf};

  /usr/bin/nice -n 5 su -m www -c "/usr/local/www/awstats/cgi-bin/awstats.pl -update -config=${i} -configdir=/usr/local/www/awstats/cgi-bin/ -LogFile=${LOGFILE}" &
  count=`expr ${count} + 1`;
  [ $((count%4)) -eq 0 ] && wait
done;

wait

bzip2 ${LOGFILE}
########################################

- With the above, the awstats configuration simply has to be updated with
  the correct log format:
LogFormat="%virtualname %host %other %logname %time1 %methodurl %code %bytesd %refererquot %uaquot"

-----------------
Again, just one way to tackle this.  You can probably use any other log
analyzer to grab the same information.  Hopefully this helps the OP.

~Paul

________________________________

This message may contain confidential or privileged information. If you are not the intended recipient, please advise us immediately and delete this message. See http://www.datapipe.com/legal/email_disclaimer/ for further information on confidentiality and the risks of non-secure electronic communication. If you cannot access these links, please notify us by reply message and we will send the contents to you.