Incomplete responses for some requests
Carl Alexander
carlalexander at globalvoices.org
Fri Sep 16 19:54:38 CEST 2016
I've managed to find a workaround to the issue. If I use "return(pipe)"
instead of "return(pass)" in my vcl, the problem disappears. That said,
this isn't ideal at all. I shouldn't have to use pipe for HTML content.
So this gives credence that something is going on inside Varnish. Should I
create a ticket for this? I've hit the limit of my ability troubleshooting
this at the moment.
On Wed, Sep 14, 2016 at 5:45 PM, Carl Alexander <
carlalexander at globalvoices.org> wrote:
> Hi everyone,
>
> I'm having a weird issue that I have no idea how to solve. I have a setup
> where I have varnish sitting between two vhosts. An SSL termination proxy
> which forwards requests to varnish who forwards them to a backend.
>
> The setup works perfectly for 99% of pages. But, for certain pages,
> varnish seems to choke and only returns part of the html response. This can
> happen consistently or at random for the same page. It also varies between
> browsers sometimes.
>
> When digging into varnishlog, I can find this FetchError for the backend:
>
> - Fetch_Body 4 eof stream
> - FetchError Resource temporarily unavailable
> - FetchError eof socket fail
> - BackendClose 24 boot.webserver
> - BereqAcct 1274 0 1274 333 32554 32887
> - End
>
>
> So I figured that maybe it's a nginx issue. When I turn on debug logs,
> nginx tells me that it's varnish that closed the connection. It was never
> able to send the complete html response back (that's probably why it's
> truncated). Here's the nginx logs:
>
>
> 2016/09/14 23:49:08 [info] 24670#24670: *566 epoll_wait() reported that
> client prematurely closed connection, so upstream connection is closed too
> while reading upstream
>
>
> So on one side, I have varnish saying that nginx became unavailable. And
> on the other, I have nginx telling me that varnish closed the connection. I
> don't know which one is right.
>
> But if I remove varnish and send the proxy request to the backend
> directly, everything works. So the signs seem to point to varnish, but I
> can't seem to find what's causing this. It started on Varnish 4.1.1, but
> upgrading to 4.1.3 didn't change anything.
>
> I'm also not sure how to troubleshoot this further. I tried to look at the
> varnish source code. The FetchError is just a generic C error so I don't
> have much else to go on.
>
> I've also attached the output of "param.show" below:
>
> accept_filter off [bool] (default)
> acceptor_sleep_decay 0.9 (default)
> acceptor_sleep_incr 0.000 [seconds] (default)
> acceptor_sleep_max 0.050 [seconds] (default)
> auto_restart on [bool] (default)
> backend_idle_timeout 60.000 [seconds] (default)
> ban_dups on [bool] (default)
> ban_lurker_age 60.000 [seconds] (default)
> ban_lurker_batch 1000 (default)
> ban_lurker_sleep 0.010 [seconds] (default)
> between_bytes_timeout 60.000 [seconds] (default)
> cc_command "exec gcc -std=gnu99 -g -O2 -fstack-protector
> --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall -Werror
> -Wno-error=unused-result -pthread -fpic -shared -Wl,-x -o %o %s" (default)
> cli_buffer 8k [bytes] (default)
> cli_limit 48k [bytes] (default)
> cli_timeout 60.000 [seconds] (default)
> clock_skew 10 [seconds] (default)
> connect_timeout 3.500 [seconds] (default)
> critbit_cooloff 180.000 [seconds] (default)
> debug none (default)
> default_grace 10.000 [seconds] (default)
> default_keep 0.000 [seconds] (default)
> default_ttl 3600.000 [seconds]
> feature none (default)
> fetch_chunksize 16k [bytes] (default)
> fetch_maxchunksize 0.25G [bytes] (default)
> first_byte_timeout 60.000 [seconds] (default)
> gzip_buffer 32k [bytes] (default)
> gzip_level 6 (default)
> gzip_memlevel 8 (default)
> http_gzip_support on [bool] (default)
> http_max_hdr 64 [header lines] (default)
> http_range_support on [bool] (default)
> http_req_hdr_len 8k [bytes] (default)
> http_req_size 32k [bytes] (default)
> http_resp_hdr_len 8k [bytes] (default)
> http_resp_size 32k [bytes] (default)
> idle_send_timeout 60.000 [seconds] (default)
> listen_depth 1024 [connections] (default)
> lru_interval 2.000 [seconds] (default)
> max_esi_depth 5 [levels] (default)
> max_restarts 4 [restarts] (default)
> max_retries 4 [retries] (default)
> nuke_limit 50 [allocations] (default)
> pcre_match_limit 10000 (default)
> pcre_match_limit_recursion 20 (default)
> ping_interval 3 [seconds] (default)
> pipe_timeout 60.000 [seconds] (default)
> pool_req 10,100,10 (default)
> pool_sess 10,100,10 (default)
> pool_vbo 10,100,10 (default)
> prefer_ipv6 off [bool] (default)
> rush_exponent 3 [requests per request] (default)
> send_timeout 600.000 [seconds] (default)
> session_max 100000 [sessions] (default)
> shm_reclen 255b [bytes] (default)
> shortlived 10.000 [seconds] (default)
> sigsegv_handler on [bool] (default)
> syslog_cli_traffic on [bool] (default)
> tcp_fastopen off [bool] (default)
> tcp_keepalive_intvl 75.000 [seconds] (default)
> tcp_keepalive_probes 9 [probes] (default)
> tcp_keepalive_time 7200.000 [seconds] (default)
> thread_pool_add_delay 0.000 [seconds] (default)
> thread_pool_destroy_delay 1.000 [seconds] (default)
> thread_pool_fail_delay 0.200 [seconds] (default)
> thread_pool_max 5000 [threads] (default)
> thread_pool_min 100 [threads] (default)
> thread_pool_stack 48k [bytes] (default)
> thread_pool_timeout 300.000 [seconds] (default)
> thread_pools 2 [pools] (default)
> thread_queue_limit 20 (default)
> thread_stats_rate 10 [requests] (default)
> timeout_idle 5.000 [seconds] (default)
> timeout_linger 0.050 [seconds] (default)
> vcc_allow_inline_c off [bool] (default)
> vcc_err_unref on [bool] (default)
> vcc_unsafe_path on [bool] (default)
> vcl_cooldown 600.000 [seconds] (default)
> vcl_dir /etc/varnish (default)
> vmod_dir /usr/lib/varnish/vmods (default)
> vsl_buffer 4k [bytes] (default)
> vsl_mask -VCL_trace,-WorkThread,-Hash,-VfpAcct (default)
> vsl_reclen 255b [bytes] (default)
> vsl_space 80M [bytes] (default)
> vsm_free_cooldown 60.000 [seconds] (default)
> vsm_space 1M [bytes] (default)
> workspace_backend 64k [bytes] (default)
> workspace_client 64k [bytes] (default)
> workspace_session 0.50k [bytes] (default)
> workspace_thread 2k [bytes] (default)
>
>
>
> Thanks in advance for any help,
> Carl
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20160916/9e192569/attachment.html>
More information about the varnish-misc
mailing list