High CPU usage when using ban with HTTP PURGE

[Lasse Karstensen]

Connor Walls:
> So, I've been having an issue with CPU usage recently. We run a cluster of
> 3 varnish boxes, running 3.0.2, serving content on a particular port, and
> accepting HTTP PURGE request on another (:82). This is used so whenever a
> user on the site changes something on their profile page, one of the
> backend servers will send a HTTP PURGE request to each box in the cluster,
> containing a regex in the header, which is then used to ban based on a
> particular URL - the relevant VCL is as follows:
[..]
> Now what we've been seeing is very high CPU usage with this functionality,
> which has been slowly growing over time with increased traffic on the
> website, and is now averaging above 70%, peaking at 100% at times and
> causing some user requests to fail. The CPU usage correlates very closely
> with timeout exceptions we get when making the HTTP PURGE requests. This
> became an issue yesterday to the point where we disabled the HTTP PURGE -
> CPU usage has now fallen from averaging > 70% to averaging around 8%, so
> clearly the purges are the issue. What I'm wondering is whether or not
> banning like this is going to be inherently heavy and I need to find a way
> to throttle the purge requests, or if I could rework the VCL so it will be
> less CPU heavy to process these requests? Will the CPU usage depend on what
> the actual regex is?

There are two methods for cache invalidation in Varnish 3.0: Purges and bans.
Bans are flexible and cool, and purges are less flexible but very fast. Use
purges if you can.

If you are using a high amount of bans, you should use what Kristian calls
"Smart bans" in the Varnish Book: 

    https://www.varnish-software.com/static/book/Cache_invalidation.html#smart-bans


-- 
Lasse Karstensen
Varnish Software AS
http://www.varnish-software.com/