varnishtest documentation

[Poul-Henning Kamp]

In message <20131221140238.61593pibn6wz20w0 at art.yamanotecorporation.com>, varni
sh at syrota.com writes:

Hi Sergey,

Yes, varnishtest is pretty undocumented.

I trust it you have spotted the 347 test-descriptions in the
source tree ?   They're probably the main documentation for it.

The main reason varnishtest has not been documented, is that it
we weren't sure if anybody would ever bother read the documentation
if we wrote it, and given that we have more ideas for things to
do than time to do them it ended up somewhat low in the pile.

Also, truth to be told, it is more than a little bit obscure,
because it has been written to get out into obscure corners of
varnishd behaviour, so it has all sorts of really weird magic
code, which is not useful in more than one or maybe two of
the official test-cases.

That said, I often find myself rumaging around the test-cases
and source code, to try to remember how to do X with varnishtest
so documentation would absolutely be a good idea.

So yes, We'd love it if you write a "rst2man" file for varnishtest :-)

>script commands here:  
>http://blog.syrota.com/2013/12/varnishtest-script-file-commands.html

A quick top-level architectural overview:

The varnishtest program can run test-cases, each one written in a
*.vtc file. 

The "main" part of the program will make a list of tests to be
run (-n# + filename arguments) and schedule -j# of them in
parallel.

Unless -k is given, it stops on the first test-failure.

Normally, it will report a single "success" line per test-case
or dump the full log if the test-case failed.

-q is quiet mode, only test failuers are reported and only with
a single summary line.

-v is verbose mode, it always dumps the full log.

-i tells it that it's running in the source tree (special
trickery is involved in finding the varnishd program and the vmods.)



Each test-case consists of a number of independent "instances" of
various kinds, clients, servers, varnishd's and so on.  These are
run in separate threads or even processes, to gain concurrency.

Most of the instances (client/server/varnish at least) take 
	-start	= Start now
	-wait	= Wait for completion
	-run	= -start -wait
arguments.

Once the test-case is over, all started instances gets a "we're done"
signal and an implict -wait behind the scenes.

The most typical use case is:
		server ... -start
		varnish ... -start
		client ... -run

This will start a server, a varnishd and a client in parallel
and wait for the client to complete.

Semaphores mostly used to synchronize clients and servers "around"
varnish, so that the server will not send something particular
until the client tells it to, but it can also be used synchronize
multiple clients or servers running in parallel.



-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.