<div dir="ltr"><div>I've managed to find a workaround to the issue. If I use "return(pipe)" instead of "return(pass)" in my vcl, the problem disappears. That said, this isn't ideal at all. I shouldn't have to use pipe for HTML content.</div><div><br></div><div>So this gives credence that something is going on inside Varnish. Should I create a ticket for this? I've hit the limit of my ability troubleshooting this at the moment.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 14, 2016 at 5:45 PM, Carl Alexander <span dir="ltr"><<a href="mailto:carlalexander@globalvoices.org" target="_blank">carlalexander@globalvoices.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi everyone,</div><div><br></div><div>I'm having a weird issue that I have no idea how to solve. I have a setup where I have varnish sitting between two vhosts. An SSL termination proxy which forwards requests to varnish who forwards them to a backend. </div><div><br></div><div>The setup works perfectly for 99% of pages. But, for certain pages, varnish seems to choke and only returns part of the html response. This can happen consistently or at random for the same page. It also varies between browsers sometimes.</div><div><br></div><div>When digging into varnishlog, I can find this FetchError for the backend:</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>- Fetch_Body 4 eof stream</div><div>- FetchError Resource temporarily unavailable</div><div>- FetchError eof socket fail</div><div>- BackendClose 24 boot.webserver</div><div>- BereqAcct 1274 0 1274 333 32554 32887</div><div>- End</div></blockquote><div><br></div><div>So I figured that maybe it's a nginx issue. When I turn on debug logs, nginx tells me that it's varnish that closed the connection. It was never able to send the complete html response back (that's probably why it's truncated). Here's the nginx logs: </div><div><br></div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><br></div><div>2016/09/14 23:49:08 [info] 24670#24670: *566 epoll_wait() reported that client prematurely closed connection, so upstream connection is closed too while reading upstream</div></blockquote></div><div><br></div><div>So on one side, I have varnish saying that nginx became unavailable. And on the other, I have nginx telling me that varnish closed the connection. I don't know which one is right. </div><div><br></div><div>But if I remove varnish and send the proxy request to the backend directly, everything works. So the signs seem to point to varnish, but I can't seem to find what's causing this. It started on Varnish 4.1.1, but upgrading to 4.1.3 didn't change anything.</div><div><br></div><div>I'm also not sure how to troubleshoot this further. I tried to look at the varnish source code. The FetchError is just a generic C error so I don't have much else to go on.</div><div><br></div><div>I've also attached the output of "param.show" below:</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div>accept_filter off [bool] (default)</div><div>acceptor_sleep_decay 0.9 (default)</div><div>acceptor_sleep_incr 0.000 [seconds] (default)</div><div>acceptor_sleep_max 0.050 [seconds] (default)</div><div>auto_restart on [bool] (default)</div><div>backend_idle_timeout 60.000 [seconds] (default)</div><div>ban_dups on [bool] (default)</div><div>ban_lurker_age 60.000 [seconds] (default)</div><div>ban_lurker_batch 1000 (default)</div><div>ban_lurker_sleep 0.010 [seconds] (default)</div><div>between_bytes_timeout 60.000 [seconds] (default)</div><div>cc_command "exec gcc -std=gnu99 -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -Wall -Werror -Wno-error=unused-result -pthread -fpic -shared -Wl,-x -o %o %s" (default)</div><div>cli_buffer 8k [bytes] (default)</div><div>cli_limit 48k [bytes] (default)</div><div>cli_timeout 60.000 [seconds] (default)</div><div>clock_skew 10 [seconds] (default)</div><div>connect_timeout 3.500 [seconds] (default)</div><div>critbit_cooloff 180.000 [seconds] (default)</div><div>debug none (default)</div><div>default_grace 10.000 [seconds] (default)</div><div>default_keep 0.000 [seconds] (default)</div><div>default_ttl 3600.000 [seconds]</div><div>feature none (default)</div><div>fetch_chunksize 16k [bytes] (default)</div><div>fetch_maxchunksize 0.25G [bytes] (default)</div><div>first_byte_timeout 60.000 [seconds] (default)</div><div>gzip_buffer 32k [bytes] (default)</div><div>gzip_level 6 (default)</div><div>gzip_memlevel 8 (default)</div><div>http_gzip_support on [bool] (default)</div><div>http_max_hdr 64 [header lines] (default)</div><div>http_range_support on [bool] (default)</div><div>http_req_hdr_len 8k [bytes] (default)</div><div>http_req_size 32k [bytes] (default)</div><div>http_resp_hdr_len 8k [bytes] (default)</div><div>http_resp_size 32k [bytes] (default)</div><div>idle_send_timeout 60.000 [seconds] (default)</div><div>listen_depth 1024 [connections] (default)</div><div>lru_interval 2.000 [seconds] (default)</div><div>max_esi_depth 5 [levels] (default)</div><div>max_restarts 4 [restarts] (default)</div><div>max_retries 4 [retries] (default)</div><div>nuke_limit 50 [allocations] (default)</div><div>pcre_match_limit 10000 (default)</div><div>pcre_match_limit_recursion 20 (default)</div><div>ping_interval 3 [seconds] (default)</div><div>pipe_timeout 60.000 [seconds] (default)</div><div>pool_req 10,100,10 (default)</div><div>pool_sess 10,100,10 (default)</div><div>pool_vbo 10,100,10 (default)</div><div>prefer_ipv6 off [bool] (default)</div><div>rush_exponent 3 [requests per request] (default)</div><div>send_timeout 600.000 [seconds] (default)</div><div>session_max 100000 [sessions] (default)</div><div>shm_reclen 255b [bytes] (default)</div><div>shortlived 10.000 [seconds] (default)</div><div>sigsegv_handler on [bool] (default)</div><div>syslog_cli_traffic on [bool] (default)</div><div>tcp_fastopen off [bool] (default)</div><div>tcp_keepalive_intvl 75.000 [seconds] (default)</div><div>tcp_keepalive_probes 9 [probes] (default)</div><div>tcp_keepalive_time 7200.000 [seconds] (default)</div><div>thread_pool_add_delay 0.000 [seconds] (default)</div><div>thread_pool_destroy_delay 1.000 [seconds] (default)</div><div>thread_pool_fail_delay 0.200 [seconds] (default)</div><div>thread_pool_max 5000 [threads] (default)</div><div>thread_pool_min 100 [threads] (default)</div><div>thread_pool_stack 48k [bytes] (default)</div><div>thread_pool_timeout 300.000 [seconds] (default)</div><div>thread_pools 2 [pools] (default)</div><div>thread_queue_limit 20 (default)</div><div>thread_stats_rate 10 [requests] (default)</div><div>timeout_idle 5.000 [seconds] (default)</div><div>timeout_linger 0.050 [seconds] (default)</div><div>vcc_allow_inline_c off [bool] (default)</div><div>vcc_err_unref on [bool] (default)</div><div>vcc_unsafe_path on [bool] (default)</div><div>vcl_cooldown 600.000 [seconds] (default)</div><div>vcl_dir /etc/varnish (default)</div><div>vmod_dir /usr/lib/varnish/vmods (default)</div><div>vsl_buffer 4k [bytes] (default)</div><div>vsl_mask -VCL_trace,-WorkThread,-Hash,-<wbr>VfpAcct (default)</div><div>vsl_reclen 255b [bytes] (default)</div><div>vsl_space 80M [bytes] (default)</div><div>vsm_free_cooldown 60.000 [seconds] (default)</div><div>vsm_space 1M [bytes] (default)</div><div>workspace_backend 64k [bytes] (default)</div><div>workspace_client 64k [bytes] (default)</div><div>workspace_session 0.50k [bytes] (default)</div><div>workspace_thread 2k [bytes] (default)</div></blockquote><div><br></div><div><br></div><div>Thanks in advance for any help,</div><div>Carl</div></div>
</blockquote></div><br></div>