sporadic unknown result for backend probes (random director)

[Szerb Roger]

Hi!

We are investigating apache2 threads spiking issue, but bumped into a
backend probe problem(?).

Apache2 backends configured for keep-alive connections, varnish poll
probes just check if static.txt file exists on the backend host, but
varnishlog says sporadic 4--X--- .


Can anyone give a hint?



Thanks and Merry Christmas! :)




VCL:

probe healthcheck { .url = "/static.txt"; .timeout = 300ms; .interval
= 2s; .window = 10; .threshold = 2; }

backend hnode  { .host = "10.0.80.2"; .port = "8081"; .probe = healthcheck; }
backend h2node { .host = "10.0.80.2"; .port = "8082"; .probe = healthcheck; }
backend hFBnode   { .host = "10.0.80.2"; .port = "8083"; .probe = healthcheck; }

...

varnishlog | grep health

    0 Backend_health - hnode Still healthy 4--X-RH 10 2 10 0.000484
0.000537 HTTP/1.1 200 OK
    0 Backend_health - h2node Still healthy 4--X-RH 10 2 10 0.000512
0.000602 HTTP/1.1 200 OK
    0 Backend_health - hFBnode Still healthy 4--X-RH 10 2 10 0.000495
0.000579 HTTP/1.1 200 OK
    0 Backend_health - hnode Still healthy 4--X-RH 10 2 10 0.000568
0.000544 HTTP/1.1 200 OK
    0 Backend_health - h2node Still healthy 4--X-RH 10 2 10 0.000452
0.000564 HTTP/1.1 200 OK
    0 Backend_health - hFBnode Still healthy 4--X-RH 10 2 10 0.000648
0.000597 HTTP/1.1 200 OK
???????
    0 Backend_health - hnode Still healthy 4--X--- 9 2 10 0.000000 0.000544
    0 Backend_health - h2node Still healthy 4--X--- 9 2 10 0.000000 0.000564
    0 Backend_health - hFBnode Still healthy 4--X--- 9 2 10 0.000000 0.000597
???????
    0 Backend_health - hnode Still healthy 4--X-RH 9 2 10 0.000914
0.000637 HTTP/1.1 200 OK
    0 Backend_health - h2node Still healthy 4--X-RH 9 2 10 0.000873
0.000641 HTTP/1.1 200 OK
    0 Backend_health - hFBnode Still healthy 4--X-RH 9 2 10 0.000802
0.000648 HTTP/1.1 200 OK
    0 Backend_health - hnode Still healthy 4--X-RH 9 2 10 0.000672
0.000646 HTTP/1.1 200 OK
    0 Backend_health - h2node Still healthy 4--X-RH 9 2 10 0.000551
0.000619 HTTP/1.1 200 OK
    0 Backend_health - hFBnode Still healthy 4--X-RH 9 2 10 0.000578
0.000631 HTTP/1.1 200 OK
    0 Backend_health - hnode Still healthy 4--X-RH 9 2 10 0.000647
0.000646 HTTP/1.1 200 OK
    0 Backend_health - h2node Still healthy 4--X-RH 9 2 10 0.000585
0.000610 HTTP/1.1 200 OK
    0 Backend_health - hFBnode Still healthy 4--X-RH 9 2 10 0.000545
0.000609 HTTP/1.1 200 OK