On tests

[Tollef Fog Heen]

2015-03-09 12:25 GMT+01:00 Per Buer <perbu at varnish-software.com>:

> On Mon, Mar 9, 2015 at 11:49 AM, Poul-Henning Kamp <phk at phk.freebsd.dk> wrote:
>>
>> >Let me know if I should commit this and start replacement of the current
>>
>> >way tests are run.
>>
>> So my first question is:  Why don't we just teach varnishtest to do this ?
>
> Because scheduling the tests and running the tests are different tasks. Scheduling and interfacing with a database is pretty trivial in Python and not something we should do directly in the varnishtest itself.
>
> The ultimate goal here is to be able to produce reports on how the various tests are doing. What tests are timing sensitive and what platforms they seem to be struggling.

I'd argue that scheduling and collecting the results are different
again.  Having varnishtest able to output TAP or similar and then
having something that consumes that and feeds it into a database (and
then a third tool to display/distribute this) sounds like a better
design. (varnishtest already knows a bit about scheduling with -j and
all that.)

- Tollef