Benchmarking and stress testing Varnish

[Kristian Lyngstol]

As some of you might already know, I regular stress tests of Varnish, most
of the time it's a matter of testing the same thing over and over to ensure
that there aren't any huge surprises during the development of Varnish (we
run nightly builds and stress tests of trunk - they need to be as
predictable as possible to have any value).

However, I also do occasional tests which involve trying to find the
various breaking points of Varnish. Last time I did this, I used roughly 42
cpu cores spread over about 30 computers to generate traffic against a
single Varnish server on our quad xenon core machine. The result thus far
was that all of the clients ran far too close to 100% cpu usage, and
Varnish was serving between 140 and 150 thousand requests per second.

The reason I'm telling you this is because I'm looking for input on aspects
that should be tested next time I do this, which will most likely be during
Christmas (far easier to borrow machine power). So far on my list, I've
got:

1. Test performance over some time period when pages are evicted more
frequently. (ie: X thousand pages requested repeatedly, but random expiry
time).

2. Test with fewer requests per session (this is somewhat hard because the
clients tend to turn into the bottleneck).

3. Test raw hot-hit cache rate (more of what I did before - get a high
number).

4. Test raw performance with a huge data set that's bound to be swapped
out.

5. Various tests of -sfile versus -smalloc and large datasets, combined
with critbit/classic tests.

6. Find some reasonably optimal settings, then fidget around trying to find
a worst-case scenario.

7. Test the effect of session lingering with really slow clients.

....

One thing that should be noted is that the test server is limited to 1gbit,
which means that for "raw req/s", we're basically forced to use tiny pages,
or we just end up starving the network.

The goal is to test theories regarding performance, stability and
predictability. Basically find the breaking points, what's good and what's
not, and what we have to care about and what we can silently ignore.

As you can see, the list is getting long and this is off the top of my
head, but any suggestions are welcome.

-- 
Kristian Lyngstøl
Redpill Linpro AS
Tlf: +47 21544179
Mob: +47 99014497
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20091113/38c1f3ee/attachment-0003.pgp>