On tests

Poul-Henning Kamp phk at phk.freebsd.dk
Mon Mar 9 11:49:21 CET 2015


--------
In message <CAOXZevCbv95KkamMUb=LPezgP0jO+Njn+NbP5-VBTBwpTuif3g at mail.gmail.com>
, Per Buer writes:

>Intermittently failing tests seems to be an issue.

Yes, indeed.

But the one thing I would *really* like to get is some kind of statistics
of *which* test-cases we have this problem with, and I wish Jenkins could
somehow be persuaded to collect such info.

>Another option would be to start giving hints on how the test should be
>executed within the test itself. By having a header on the test that the
>testrunner could read one can set up certain parameters.
>
># parallel=0,load<1,timeout=15s,platforms=!netbsd,retries=3

I'm not very keen on this, it is an open invitation to not take
the battle and fix the test-cases properly.

(With respect to "platforms=!netbsd" we already have a "feature"
keyword, that looks for specific details in the run-environment.)

>Let me know if I should commit this and start replacement of the current
>way tests are run.

So my first question is:  Why don't we just teach varnishtest to do this ?

It would be really trivial to make varnishtest collect the failing
tests on a list and then run that list after the main-run in
single-threaded mode (enabled by some argv)

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.



More information about the varnish-dev mailing list