Connection resets / timout with Varnish 6.0 and HTTP/2

[Winkelmann, Thomas (RADIO TELE FFH - Online)]

Hello everbody,

finally we got Varnish 6.0 + Vmods + Hitch TLS running on Ubuntu. So far everything works fine, also HTTP/2 Support.
But as soon as we are receiving some more requests (approx. > 500req/s) varnish does not deliver all requests anymore. The syslog is flooded with messages like:

Jul  4 08:04:05 cache1-vm hitch[5480]: 20180704T080405.289397 [ 5502] {backend-connect}: Connection refused
Jul  4 08:04:05 cache1-vm hitch[5480]: 20180704T080405.290213 [ 5502] {backend-connect}: Connection refused
...

Jul  4 07:57:29 cache1-vm hitch[5480]: 20180704T075729.837457 [ 5504] xxx.xxx.xxx.xxx:5835 :0 1469:1470 backend connect timeout
Jul  4 07:57:29 cache1-vm hitch[5480]: 20180704T075729.851809 [ 5510] xxx.xxx.xxx.xxx:54396 :0 7190:7191 backend connect timeout
...

Jul  4 07:57:31 cache1-vm hitch[5480]: 20180704T075731.488096 [ 5510] {backend} Socket error: Connection reset by peer

We had some similar problems in the past with HTTPS. We could solve them by adding:

net.ipv4.ip_local_port_range = 4096 64999
net.ipv4.tcp_tw_reuse = 1

to /etc/sysctl.conf But this seems to be not the problem here.

As soon as we remove alpn-protos = "h2,http/1.1" from hitch.conf everything is working normally.

Are there any limitations regarding HTTP/2 within varnish?

Our config:

[Service]
Type=simple
LimitNOFILE=131072
LimitMEMLOCK=82000
ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :80 -a '[::1]:6086,PROXY' -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -p thread_pools=2 -p thread_pool_min=200 -p thread_pool_max=5000 -p timeout_idle=100 -p send_timeout=3600 -p feature=+http2 -s malloc,2g
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
PrivateDevices=true

Also DefaultLimitNOFILE is set to 250000 in /etc/systemd/system.conf.

Hitch config:

# Listening
frontend = "[*]:443"
ciphers  = "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH"
# TLS 1.0 wird gebraucht, damit alte Server (intranet-srv) eine Verbindung aufbauen können
tls-protos = TLSv1.1 TLSv1.2 TLSv1.0

# Send traffic to the Varnish backend using the PROXY protocol
backend        = "[::1]:6086"
write-proxy-v2 = on
alpn-protos = "h2,http/1.1"

# Number of processes
workers = 8

We already searched on the varnish github account for similar problem, but did not found anything...

Thanks,
Thomas

________________________________
RADIO / TELE FFH GmbH & Co. Betriebs-KG
FFH-Platz 1, 61111 Bad Vilbel
HRA - Nr. 26092 Frankfurt/Main
USt.IdNr. DE 112152620
Geschäftsführer / Programmdirektor: Hans-Dieter Hillmoth
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20180704/22f68e16/attachment.html>