Converting 3-to-5 - strange "No backend" errors

[Mark Hanford]

Hi folks. I'm in the process of migrating to from v3 to v5, and have
finally got the config compiling at least. Now I'm having some problems
with the backends that I can't seem to work out. Apologies, this is a bit
of a wordy one...

Let's say I have two backends, web01 and web02, and these are put into a
director called admin_director for load-balancing. This is all in
"backends.vcl":

import directors;

probe healthcheck {
.request =
"GET / HTTP/1.1"
"Host: www.mydomain.co.uk"
"Connection: close";
.timeout = 30s;
.interval = 15s;
.window = 10;
.threshold = 8;
.expected_response = 302;
}


backend web01 { .host = "192.168.1.11"; .port = "80"; .first_byte_timeout =
600s; .probe = healthcheck; }
backend web02 { .host = "192.168.1.12"; .port = "80"; .first_byte_timeout =
600s; .probe = healthcheck; }

sub vcl_init {
new admin_director = directors.shard();
admin_director.add_backend(web01);
admin_director.add_backend(web02);
}


I've now removed all my main VCL for clarification, so my vcl_recv.vcl file
contains just:


import std;

sub vcl_recv {
set req.backend_hint = admin_director.backend();
std.log("Backend hint: " + req.backend_hint);
}



I then have my main "default.vcl":


vcl 4.0;
include "backends.vcl";
include "vcl_recv.vcl";


Now to the problem. If I try to access a URL, I always get a "FetchError:
No backend"

*   << BeReq    >> 3
-   Begin          bereq 2 fetch
-   Timestamp      Start: 1490802880.486355 0.000000 0.000000
-   BereqMethod    GET
-   BereqURL       /Common/Images/NewLoginImages/training.png
-   BereqProtocol  HTTP/1.1
-   BereqHeader    Host: www.mydomain.com
-   BereqHeader    User-Agent: curl/7.35.0
-   BereqHeader    Accept: */*
-   BereqHeader    X-Forwarded-Proto: https
-   BereqHeader    X-Real-Ip: 1.2.3.4
-   BereqHeader    Accept-Encoding: gzip
-   BereqHeader    X-Forwarded-For: 1.2.3.4, 192.168.1.23
-   BereqHeader    X-Varnish: 3
-   VCL_call       BACKEND_FETCH
-   VCL_return     fetch
-   FetchError     No backend
-   Timestamp      Beresp: 1490802880.486447 0.000092 0.000092
-   Timestamp      Error: 1490802880.486452 0.000098 0.000006
-   BerespProtocol HTTP/1.1
-   BerespStatus   503
-   BerespReason   Service Unavailable
-   BerespReason   Backend fetch failed
-   BerespHeader   Date: Wed, 29 Mar 2017 15:54:40 GMT
-   BerespHeader   Server: Varnish
-   VCL_call       BACKEND_ERROR
-   BerespHeader   Content-Type: text/html; charset=utf-8
-   BerespHeader   Retry-After: 5
-   VCL_return     deliver
-   Storage        malloc Transient
-   ObjProtocol    HTTP/1.1
-   ObjStatus      503
-   ObjReason      Backend fetch failed
-   ObjHeader      Date: Wed, 29 Mar 2017 15:54:40 GMT
-   ObjHeader      Server: Varnish
-   ObjHeader      Content-Type: text/html; charset=utf-8
-   ObjHeader      Retry-After: 5
-   Length         278
-   BereqAcct      0 0 0 0 0 0
-   End

*   << Request  >> 32770
-   Begin          req 32769 rxreq
-   Timestamp      Start: 1490803780.494500 0.000000 0.000000
-   Timestamp      Req: 1490803780.494500 0.000000 0.000000
-   ReqStart       192.168.1.23 36896
-   ReqMethod      GET
-   ReqURL         /Common/Images/NewLoginImages/training.png
-   ReqProtocol    HTTP/1.1
-   ReqHeader      Host: www.mydomain.com
-   ReqHeader      User-Agent: curl/7.35.0
-   ReqHeader      Accept: */*
-   ReqHeader      X-Forwarded-For: 1.2.3.4
-   ReqHeader      X-Forwarded-Proto: https
-   ReqHeader      X-Real-Ip: 1.2.3.4
-   ReqHeader      Accept-Encoding: gzip
-   ReqUnset       X-Forwarded-For: 1.2.3.4
-   ReqHeader      X-Forwarded-For: 1.2.3.4, 192.168.1.23
-   VCL_call       RECV
-   Error          shard admin_director: no backends
-   VCL_Log        Backend hint:
-   VCL_return     hash
-   VCL_call       HASH
-   VCL_return     lookup
-   VCL_call       MISS
-   VCL_return     fetch
-   Link           bereq 32771 fetch
-   Timestamp      Fetch: 1490803780.495705 0.001204 0.001204
-   RespProtocol   HTTP/1.1
-   RespStatus     503
-   RespReason     Backend fetch failed
-   RespHeader     Date: Wed, 29 Mar 2017 16:09:40 GMT
-   RespHeader     Server: Varnish
-   RespHeader     Content-Type: text/html; charset=utf-8
-   RespHeader     Retry-After: 5
-   RespHeader     X-Varnish: 32770
-   RespHeader     Age: 0
-   RespHeader     Via: 1.1 varnish (Varnish/5.1)
-   VCL_call       DELIVER
-   VCL_return     deliver
-   Timestamp      Process: 1490803780.495723 0.001223 0.000019
-   RespHeader     Content-Length: 282
-   Debug          "RES_MODE 2"
-   RespHeader     Connection: keep-alive
-   Timestamp      Resp: 1490803780.495806 0.001305 0.000082
-   ReqAcct        238 0 238 250 282 532
-   End

But my backend.list seems to always be healthy:

    boot.web01                     probe      Healthy             10/10
Wed, 29 Mar 2017 15:54:18 GMT
    boot.web02                     probe      Healthy             10/10
Wed, 29 Mar 2017 15:54:18 GMT

>From the varnish server, I can reach the backends successfully manually:

    curl --resolve www.mydomain.com:192.168.1.11
http://www.mydomain.com/Common/Images/NewLoginImages/training.png

Not sure what it all means.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20170329/9000809e/attachment.html>