Varnish graphs

[Mark Moseley]

On Wed, Jan 27, 2010 at 2:37 PM, pablort <pablort+varnish at gmail.com> wrote:
> Hello there,
>
> I've sucessfully created graphics based on varnishstat -1 output using cacti
> and snmp and I'd really like to do the same thing using varnishtop to graph
> TxStatus responses, but it didn't work as I expected.
>
> $ varnishtop -1 -i TxStatus
>  29481.00 TxStatus
>   3280.00 TxStatus
>   1196.00 TxStatus
>    376.00 TxStatus
>     23.00 TxStatus
>      3.00 TxStatus
>      3.00 TxStatus
>
> The numbers do reflect the TxStatus'es that I see in interactive varnishtop,
> but when I try to run it with -1, it doesn't show which entry corresponds to
> which status. LOL.
>
> So, is this supposed to be like this or should I file a bug for that ?
>
> Also, what do you guys use for performance analysis ?


I use varnishstat + collectd with a little python module (which means
collectd 4.9.0 or higher, with the python module included -- I can
post the Perl version that the python module replaced, which will work
in pre-4.9.x). I'm only grabbing a few things and aggregating some
(like all LRU events), but it'd be easy to modify to grab other
things.

If you'd like to take a look, I'll paste the module here (but be
forewarned I'm not python wiz, so this "good enough" for me). In
collectd.conf, along with the regular config, you'll need:

<LoadPlugin python>
        Globals true
</LoadPlugin>

<Plugin python>
        ModulePath "/path/to/your/collectd/python/modules"
        Import "varnish.stats"
        # Optional if different from path in module
        # <Module "varnish.stats">
        #        varnishstat_binary "/some/other/path/to/varnishstat"
        # </Module>
</Plugin>


Here's varnish/stats.py

import collectd, sys, time, subprocess, os.path
from pprint import pformat

varnishstat_binary = "/usr/bin/varnishstat"

# Map fields
field_map = {
	"client_req": "count_reqs",
	"cache_hit": "count_hits",
	"n_wrk_failed": "count_workerr",
	"n_wrk_max": "count_workerr",
	"n_wrk_queue": "count_workerr",
	"n_wrk_overflow": "count_workerr",
	"n_wrk_drop": "count_workerr",
	"n_lru_nuked": "count_lru",
	"n_lru_saved": "count_lru",
	"n_lru_moved": "count_lru",
}

fields_to_query = ",".join( field_map.keys() )


def varnish_stats_config ( Cfg ):
	global varnishstat_binary
	for child in Cfg.children:
		if child.key == "varnishstat_binary":
			collectd.debug( "[varnish_stats_config] config arg set key %s: %s"
% ( child.key, child.values[0] ) )
			varnishstat_binary = child.values[0]


def varnish_stats_init ( ):
	if not os.path.exists( varnishstat_binary ):
		collectd.error( "Can't find varnishstat binary at %s, disabling
plugin" % ( varnishstat_binary ) )
		collectd.unregister_read( varnish_stats_read )
		

def varnish_stats_read ( Data=None ):
	count_lru, count_hits, count_reqs, count_workerr = 0, 0, 0, 0
	stats = {}

	fh = subprocess.Popen( [ varnishstat_binary, "-1", "-f",
fields_to_query ], stdout=subprocess.PIPE )
	lines = fh.stdout.readlines()
	fh.stdout.close()

	for line in lines:
		field, val, dummy = line.strip().split( None, 2 )

		if field_map.has_key( field ):
			if stats.has_key( field_map[ field ] ):
				stats[ field_map[ field ] ] += int( val )
			else:
				stats[ field_map[ field ] ] = int( val )

	for field in ( stats.keys() ):
		if stats.has_key( field ):
			collectd.debug( "[varnish_stats_read] Dispatch %s: %d" % ( field,
stats[ field ] ) )
			stats_data = collectd.Values( type="operations" )
			stats_data.plugin = "varnish-stats"
			stats_data.dispatch( values=[ stats[ field ] ], type_instance=field )
		else:
			continue
			collectd.error( "[varnish_stats_read] Unable to dispatch key: %s" %
( field ) )
		


collectd.register_config( varnish_stats_config )
collectd.register_read( varnish_stats_read )
collectd.register_init( varnish_stats_init )