Incomplete responses for some requests
Carl Alexander
carlalexander at globalvoices.org
Thu Sep 15 02:45:22 CEST 2016
Hi everyone,
I'm having a weird issue that I have no idea how to solve. I have a setup
where I have varnish sitting between two vhosts. An SSL termination proxy
which forwards requests to varnish who forwards them to a backend.
The setup works perfectly for 99% of pages. But, for certain pages, varnish
seems to choke and only returns part of the html response. This can happen
consistently or at random for the same page. It also varies between
browsers sometimes.
When digging into varnishlog, I can find this FetchError for the backend:
- Fetch_Body 4 eof stream
- FetchError Resource temporarily unavailable
- FetchError eof socket fail
- BackendClose 24 boot.webserver
- BereqAcct 1274 0 1274 333 32554 32887
- End
So I figured that maybe it's a nginx issue. When I turn on debug logs,
nginx tells me that it's varnish that closed the connection. It was never
able to send the complete html response back (that's probably why it's
truncated). Here's the nginx logs:
2016/09/14 23:49:08 [info] 24670#24670: *566 epoll_wait() reported that
client prematurely closed connection, so upstream connection is closed too
while reading upstream
So on one side, I have varnish saying that nginx became unavailable. And on
the other, I have nginx telling me that varnish closed the connection. I
don't know which one is right.
But if I remove varnish and send the proxy request to the backend directly,
everything works. So the signs seem to point to varnish, but I can't seem
to find what's causing this. It started on Varnish 4.1.1, but upgrading to
4.1.3 didn't change anything.
I'm also not sure how to troubleshoot this further. I tried to look at the
varnish source code. The FetchError is just a generic C error so I don't
have much else to go on.
I've also attached the output of "param.show" below:
accept_filter off [bool] (default)
acceptor_sleep_decay 0.9 (default)
acceptor_sleep_incr 0.000 [seconds] (default)
acceptor_sleep_max 0.050 [seconds] (default)
auto_restart on [bool] (default)
backend_idle_timeout 60.000 [seconds] (default)
ban_dups on [bool] (default)
ban_lurker_age 60.000 [seconds] (default)
ban_lurker_batch 1000 (default)
ban_lurker_sleep 0.010 [seconds] (default)
between_bytes_timeout 60.000 [seconds] (default)
cc_command "exec gcc -std=gnu99 -g -O2 -fstack-protector
--param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall -Werror
-Wno-error=unused-result -pthread -fpic -shared -Wl,-x -o %o %s" (default)
cli_buffer 8k [bytes] (default)
cli_limit 48k [bytes] (default)
cli_timeout 60.000 [seconds] (default)
clock_skew 10 [seconds] (default)
connect_timeout 3.500 [seconds] (default)
critbit_cooloff 180.000 [seconds] (default)
debug none (default)
default_grace 10.000 [seconds] (default)
default_keep 0.000 [seconds] (default)
default_ttl 3600.000 [seconds]
feature none (default)
fetch_chunksize 16k [bytes] (default)
fetch_maxchunksize 0.25G [bytes] (default)
first_byte_timeout 60.000 [seconds] (default)
gzip_buffer 32k [bytes] (default)
gzip_level 6 (default)
gzip_memlevel 8 (default)
http_gzip_support on [bool] (default)
http_max_hdr 64 [header lines] (default)
http_range_support on [bool] (default)
http_req_hdr_len 8k [bytes] (default)
http_req_size 32k [bytes] (default)
http_resp_hdr_len 8k [bytes] (default)
http_resp_size 32k [bytes] (default)
idle_send_timeout 60.000 [seconds] (default)
listen_depth 1024 [connections] (default)
lru_interval 2.000 [seconds] (default)
max_esi_depth 5 [levels] (default)
max_restarts 4 [restarts] (default)
max_retries 4 [retries] (default)
nuke_limit 50 [allocations] (default)
pcre_match_limit 10000 (default)
pcre_match_limit_recursion 20 (default)
ping_interval 3 [seconds] (default)
pipe_timeout 60.000 [seconds] (default)
pool_req 10,100,10 (default)
pool_sess 10,100,10 (default)
pool_vbo 10,100,10 (default)
prefer_ipv6 off [bool] (default)
rush_exponent 3 [requests per request] (default)
send_timeout 600.000 [seconds] (default)
session_max 100000 [sessions] (default)
shm_reclen 255b [bytes] (default)
shortlived 10.000 [seconds] (default)
sigsegv_handler on [bool] (default)
syslog_cli_traffic on [bool] (default)
tcp_fastopen off [bool] (default)
tcp_keepalive_intvl 75.000 [seconds] (default)
tcp_keepalive_probes 9 [probes] (default)
tcp_keepalive_time 7200.000 [seconds] (default)
thread_pool_add_delay 0.000 [seconds] (default)
thread_pool_destroy_delay 1.000 [seconds] (default)
thread_pool_fail_delay 0.200 [seconds] (default)
thread_pool_max 5000 [threads] (default)
thread_pool_min 100 [threads] (default)
thread_pool_stack 48k [bytes] (default)
thread_pool_timeout 300.000 [seconds] (default)
thread_pools 2 [pools] (default)
thread_queue_limit 20 (default)
thread_stats_rate 10 [requests] (default)
timeout_idle 5.000 [seconds] (default)
timeout_linger 0.050 [seconds] (default)
vcc_allow_inline_c off [bool] (default)
vcc_err_unref on [bool] (default)
vcc_unsafe_path on [bool] (default)
vcl_cooldown 600.000 [seconds] (default)
vcl_dir /etc/varnish (default)
vmod_dir /usr/lib/varnish/vmods (default)
vsl_buffer 4k [bytes] (default)
vsl_mask -VCL_trace,-WorkThread,-Hash,-VfpAcct (default)
vsl_reclen 255b [bytes] (default)
vsl_space 80M [bytes] (default)
vsm_free_cooldown 60.000 [seconds] (default)
vsm_space 1M [bytes] (default)
workspace_backend 64k [bytes] (default)
workspace_client 64k [bytes] (default)
workspace_session 0.50k [bytes] (default)
workspace_thread 2k [bytes] (default)
Thanks in advance for any help,
Carl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20160914/990987a5/attachment.html>
More information about the varnish-misc
mailing list