ESI from remote host not working?

[Andreas Loewen Internetservices]

Dear all,

i am using current varnish varnish 4. I have the following config:

backend esi{
  .host = "x.x.x.x";
  .port = "8081";
}

...

sub vcl_recv {
     set req.http.host = "xxx.yyy.de";
...
     if (req.url ~ "^/typo3/.*" || req.url ~ "^/secure/.*") {
             set req.backend_hint = typo3be;
             return (synth(403, "Forbidden"));
     } else if (req.url ~ "\.mesi\?" || req.url ~ "\.htm(\?.*)?$" || 
req.url ~ "^/javax\.faces\..*" || req.url ~ "^/templates/.*" || req.url 
~"^/f/.*" || req.url ~"^/i/.*") {
         set req.backend_hint = esi;
         set req.hash_always_miss = true;
     } else {
             set req.backend_hint = typo3;
         unset req.http.cookie;
     }

...
}

sub vcl_backend_response {
                 set beresp.do_esi = true;
...

}

The esi backend is on a remote host, but it is reachable from this server.

The funny thing is, that I keep getting 404's for my esi requests for 
/fe_terminliste.mesi?pakid=1 . But they are comming from the default 
backend (local Apache on port 81). varnishlog reads like:

*   << BeReq    >> 5
-   Begin          bereq 4 pass
-   Timestamp      Start: 1461670355.552619 0.000000 0.000000
-   BereqMethod    GET
-   BereqURL       /fe_terminliste.mesi?pakid=1
-   BereqProtocol  HTTP/1.1
-   BereqHeader    Cache-Control: max-age=0
-   BereqHeader    Accept: 
text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
-   BereqHeader    Upgrade-Insecure-Requests: 1
-   BereqHeader    User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) 
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.112 Safari/537.36
-   BereqHeader    DNT: 1
-   BereqHeader    Accept-Encoding: gzip, deflate
-   BereqHeader    Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4
-   BereqHeader    Cookie: PHPSESSID=h7hk5n8pa95q4sd2m46ebbju14; 
Typo3InstallTool=tgikum56sujjsv1cu6ra236kb0; 
be_typo_user=5ad26762d12ad4b9fe2d321979fb75e4
-   BereqHeader    Host: xxx:8081
-   BereqHeader    X-Forwarded-For: 80.171.39.71
-   BereqHeader    X-Varnish: 5
-   VCL_call       BACKEND_FETCH
-   VCL_return     fetch
-   BackendOpen    20 default(127.0.0.1,,81) 127.0.0.1 33017
-   Backend        20 default default(127.0.0.1,,81)
-   Timestamp      Bereq: 1461670355.552877 0.000257 0.000257
-   Timestamp      Beresp: 1461670355.553530 0.000911 0.000654
-   BerespProtocol HTTP/1.1
-   BerespStatus   404
-   BerespReason   Not Found
-   BerespHeader   Date: Tue, 26 Apr 2016 11:32:35 GMT
-   BerespHeader   Server: Apache/2.2.15 (CentOS)
-   BerespHeader   Content-Length: 310
-   BerespHeader   Connection: close
-   BerespHeader   Content-Type: text/html; charset=iso-8859-1
-   TTL            RFC 120 -1 -1 1461670356 1461670356 1461670355 0 0
-   VCL_call       BACKEND_RESPONSE
-   TTL            VCL 120 10 0 1461670356
-   VCL_return     deliver
-   BerespHeader   Content-Encoding: gzip
-   BerespUnset    Content-Length: 310
-   Storage        malloc Transient
-   ObjProtocol    HTTP/1.1
-   ObjStatus      404
-   ObjReason      Not Found
-   ObjHeader      Date: Tue, 26 Apr 2016 11:32:35 GMT
-   ObjHeader      Server: Apache/2.2.15 (CentOS)
-   ObjHeader      Content-Type: text/html; charset=iso-8859-1
-   ObjHeader      Content-Encoding: gzip
-   Fetch_Body     3 length -
-   Gzip           G F E 310 268 80 2064 2074
-   Timestamp      BerespBody: 1461670355.554030 0.001411 0.000500
-   BackendClose   20 default(127.0.0.1,,81)
-   Length         268
-   BereqAcct      623 0 623 180 310 490
-   End


So, does varnish restrict esi to localhost for some reason? Or what is 
wrong?