getting near real time request counts

[James Pearson]

Excerpts from Matthew M. Boedicker's message of 2013-03-26 01:08:08 -0700:
> What is the best way to get real time or near real time request counts per
> url? I want to maintain this data in a database, redis or something else
> that my app can use.
> 
> I have considered:
> 
> - writing something to parse the log generated by varnishncsa
> 
> - writing a replacement for varnishncsa that reads the shared memory logs
> and does something else instead of writing a file
> 
> - using a varnish module to talk to redis or memcache
> 
> - some VCL hack to pass data through to the backend

While there is at least a memcached module, I tend to be wary of doing
reporting directly within varnish, because a slow connection to memcached could
slow down request responses.  OTOH, a separate log-parsing utility could get
behind and never catch up, if you're not careful.

How close to realtime do you need?  What sorts of grouping are you wanting
(e.g. the number of requests in the last minute)?  How long do you need to
store this data?  Are you ok with losing precision as time passes, as happens
in rrdtool and friends?  Are you concerned with all urls, or only certain ones,
or only the top x%?  How important is it to you to not lose any data (is it ok
to send stats around via UDP)?

 - P