Benchmarking and stress testing Varnish

[Ken Brownfield]

Those all seem very useful to me, but I think the lowest-hanging performance fruit right now is simultaneous connections and the threading model (including the discussions about stacksize and memory usage, etc).

Modeling Varnish's behavior with certain ranges of simultaneous worker and backend threads would be useful, IMHO -- both from slow backends, slow clients, and lots of keepalive threads.
-- 
kb

On Nov 13, 2009, at 5:38 AM, Kristian Lyngstol wrote:

> As some of you might already know, I regular stress tests of Varnish, most
> of the time it's a matter of testing the same thing over and over to ensure
> that there aren't any huge surprises during the development of Varnish (we
> run nightly builds and stress tests of trunk - they need to be as
> predictable as possible to have any value).
> 
> However, I also do occasional tests which involve trying to find the
> various breaking points of Varnish. Last time I did this, I used roughly 42
> cpu cores spread over about 30 computers to generate traffic against a
> single Varnish server on our quad xenon core machine. The result thus far
> was that all of the clients ran far too close to 100% cpu usage, and
> Varnish was serving between 140 and 150 thousand requests per second.
> 
> The reason I'm telling you this is because I'm looking for input on aspects
> that should be tested next time I do this, which will most likely be during
> Christmas (far easier to borrow machine power). So far on my list, I've
> got:
> 
> 1. Test performance over some time period when pages are evicted more
> frequently. (ie: X thousand pages requested repeatedly, but random expiry
> time).
> 
> 2. Test with fewer requests per session (this is somewhat hard because the
> clients tend to turn into the bottleneck).
> 
> 3. Test raw hot-hit cache rate (more of what I did before - get a high
> number).
> 
> 4. Test raw performance with a huge data set that's bound to be swapped
> out.
> 
> 5. Various tests of -sfile versus -smalloc and large datasets, combined
> with critbit/classic tests.
> 
> 6. Find some reasonably optimal settings, then fidget around trying to find
> a worst-case scenario.
> 
> 7. Test the effect of session lingering with really slow clients.
> 
> ....
> 
> One thing that should be noted is that the test server is limited to 1gbit,
> which means that for "raw req/s", we're basically forced to use tiny pages,
> or we just end up starving the network.
> 
> The goal is to test theories regarding performance, stability and
> predictability. Basically find the breaking points, what's good and what's
> not, and what we have to care about and what we can silently ignore.
> 
> As you can see, the list is getting long and this is off the top of my
> head, but any suggestions are welcome.
> 
> -- 
> Kristian Lyngstøl
> Redpill Linpro AS
> Tlf: +47 21544179
> Mob: +47 99014497
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc