Automated stress testing with Fryer

[Kristian Lyngstol]

Greetings,

I've just now put fryer in "production". Which means you can throw your
branch in the fryer and see if it comes out well-done or over-cooked.

I did some rudimentary documentation of usage at:
https://www.varnish-cache.org/trac/wiki/AutomatedStressTesting

The gist of it is:

 - Fryer checks out varnish from a clean git repo of our choice, builds
   and installs on a dedicated server
 - Fryer then executes a number of different tests, using 3 other
   machines to generate traffic.
 - The result is evaluated by looking for assert errors and certain
   values in varnishstat. Typical things to test for is that the
   requested number of requests matches the actual number. The size.
   Number of expired objects. Hitpass. Etc.
 - Results are sent to varnish-test at varnish-cache.org, see
   https://www.varnish-cache.org/lists/pipermail/varnish-test/2011-September/thread.html
   for a few of the test-runs I ran this weekend.
 - If you have git commit access, you can add or remove what branches
   are tested. If you don't, just ask :)
 - New tests are written on-demand. They mainly use httperf, with siege
   as an option.
 - 24 tests currently exist.

The source is currently not available, but will be, when I get around to
it. (It's cleared with The Boss (if he remembers)).

I can also, of course, run fryer manually on request. Each complete run
takes several hours (2-ish, see the report mails), but I can run
individual tests.

For those who haven't been paying attention, we've already either
discovered or verified several bugs in the last few weeks using fryer.
I'm quite happy with that.

All comments are welcomed.

- Kristian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: Digital signature
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20110905/a64bb755/attachment-0003.pgp>