On tests

[Poul-Henning Kamp]

--------
In message <CAOXZevBF0Y-VgfqS2E-1LfxqMyw759LJpVbfwLT+r01iu58KvA at mail.gmail.com>
, Per Buer writes:

>> But the one thing I would *really* like to get is some kind of statistics
>> of *which* test-cases we have this problem with, and I wish Jenkins could
>> somehow be persuaded to collect such info.
>
>That was the main objective.
>
>Having the testrunner have an optional interface to something like a
>database would be trivial.  Having something that is able to collect a bit
>of context to a testrun in a database (load, IO response, etc) would be
>helpful in order to figure out _why_ certain tests are failing on certain
>platforms.

At one point I considered having varnishtest do a DNS lookup of
	$testname.arghhhhhh.varnish-cache.org
everytime a test failed.

>Because scheduling the tests and running the tests are different tasks.
>Scheduling and interfacing with a database is pretty trivial in Python and
>not something we should do directly in the varnishtest itself.

I disagree, varnishtest is probably a lot *better* at scheduling and
running the tests than anything python can ever do.

But it utterly lacks any support for collecting the stats, so that's where
the focus should be IMO.

>It would be really trivial to make varnishtest collect the failing
>> tests on a list and then run that list after the main-run in
>> single-threaded mode (enabled by some argv)
>>
>
>Absolutely. And that is what the proposed solution does. It allows us to
>keep varnishtest a relativly simple program that deal with running a single
>test [...]

Uhm, varnishtest *already* has scheduling and parallelism built in...

But I wouldn't be surprised if autocrap ignores all that.

In that case, it should be obvious where I think the problem lies :-)


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.