<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Well, sure there are some objects that are rather big (for a regular web site, up to maybe 50 MB), but most objects are maybe 10-100 kB. The last image I tried, that had intermittent problems, was about 700 kB.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Some numbers from varnishstat:</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
MAIN.uptime: 23+01:28:11<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
MAIN.n_lru_nuked: 887748<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
MAIN.n_lru_limited: 459<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
SMA.s0.c_bytes: 17.59G<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
SMA.s0.c_freed: 17.49G<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
SMA.s0.g_bytes: 99.34M<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
SMA.s0.g_space: 607.18K<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-family: Calibri, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important">"n_lru_nuked" seems high. Would you recommend a bigger cache in this case?</span><br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Below is the output from <span style="color: rgb(50, 49, 48); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; background-color: rgb(255, 255, 255); display: inline !important">"varnishadm
param.show". I'm suspecting that when we did the initial tweaking (actually only focusing on the vcl logic, not cache sizes) we glanced at this output when the server was recently started, and didn't have much traffic. Now the server has been running for a
while, and the traffic has <span style="font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; background-color: rgb(255, 255, 255); display: inline !important">
increased<span> </span></span>(still testing environment only though).</span></div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">------</span><br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span>accept_filter -<br>
</span>
<div>acceptor_sleep_decay 0.9 (default)<br>
</div>
<div>acceptor_sleep_incr 0.000 [seconds] (default)<br>
</div>
<div>acceptor_sleep_max 0.050 [seconds] (default)<br>
</div>
<div>auto_restart on [bool] (default)<br>
</div>
<div>backend_idle_timeout 60.000 [seconds] (default)<br>
</div>
<div>backend_local_error_holddown 10.000 [seconds] (default)<br>
</div>
<div>backend_remote_error_holddown 0.250 [seconds] (default)<br>
</div>
<div>ban_cutoff 0 [bans] (default)<br>
</div>
<div>ban_dups on [bool] (default)<br>
</div>
<div>ban_lurker_age 60.000 [seconds] (default)<br>
</div>
<div>ban_lurker_batch 1000 (default)<br>
</div>
<div>ban_lurker_holdoff 0.010 [seconds] (default)<br>
</div>
<div>ban_lurker_sleep 0.010 [seconds] (default)<br>
</div>
<div>between_bytes_timeout 60.000 [seconds] (default)<br>
</div>
<div>cc_command exec gcc -g -O2 -fdebug-prefix-map=/build/varnish-ZKkrdt/varnish-6.0.6=. -fstack-protector-strong -Wformat -Werror=format-security -Wall -Werror -Wno-error=unused-result -pthread -fpic -shared -Wl,-x -o %o %s (default)<br>
</div>
<div>cli_limit 48k [bytes] (default)<br>
</div>
<div>cli_timeout 60.000 [seconds] (default)<br>
</div>
<div>clock_skew 10 [seconds] (default)<br>
</div>
<div>clock_step 1.000 [seconds] (default)<br>
</div>
<div>connect_timeout 3.500 [seconds] (default)<br>
</div>
<div>critbit_cooloff 180.000 [seconds] (default)<br>
</div>
<div>debug none (default)<br>
</div>
<div>default_grace 10.000 [seconds] (default)<br>
</div>
<div>default_keep 0.000 [seconds] (default)<br>
</div>
<div>default_ttl 120.000 [seconds] (default)<br>
</div>
<div>esi_iovs 10 [struct iovec] (default)<br>
</div>
<div>feature none (default)<br>
</div>
<div>fetch_chunksize 16k [bytes] (default)<br>
</div>
<div>fetch_maxchunksize 0.25G [bytes] (default)<br>
</div>
<div>first_byte_timeout 60.000 [seconds] (default)<br>
</div>
<div>gzip_buffer 32k [bytes] (default)<br>
</div>
<div>gzip_level 6 (default)<br>
</div>
<div>gzip_memlevel 8 (default)<br>
</div>
<div>h2_header_table_size 4k [bytes] (default)<br>
</div>
<div>h2_initial_window_size 65535b [bytes] (default)<br>
</div>
<div>h2_max_concurrent_streams 100 [streams] (default)<br>
</div>
<div>h2_max_frame_size 16k [bytes] (default)<br>
</div>
<div>h2_max_header_list_size 2147483647b [bytes] (default)<br>
</div>
<div>h2_rx_window_increment 1M [bytes] (default)<br>
</div>
<div>h2_rx_window_low_water 10M [bytes] (default)<br>
</div>
<div>http_gzip_support on [bool] (default)<br>
</div>
<div>http_max_hdr 64 [header lines] (default)<br>
</div>
<div>http_range_support on [bool] (default)<br>
</div>
<div>http_req_hdr_len 8k [bytes] (default)<br>
</div>
<div>http_req_size 32k [bytes] (default)<br>
</div>
<div>http_resp_hdr_len 8k [bytes] (default)<br>
</div>
<div>http_resp_size 32k [bytes] (default)<br>
</div>
<div>idle_send_timeout 60.000 [seconds] (default)<br>
</div>
<div>listen_depth 1024 [connections] (default)<br>
</div>
<div>lru_interval 2.000 [seconds] (default)<br>
</div>
<div>max_esi_depth 5 [levels] (default)<br>
</div>
<div>max_restarts 4 [restarts] (default)<br>
</div>
<div>max_retries 4 [retries] (default)<br>
</div>
<div>nuke_limit 50 [allocations] (default)<br>
</div>
<div>pcre_match_limit 10000 (default)<br>
</div>
<div>pcre_match_limit_recursion 20 (default)<br>
</div>
<div>ping_interval 3 [seconds] (default)<br>
</div>
<div>pipe_timeout 60.000 [seconds] (default)<br>
</div>
<div>pool_req 10,100,10 (default)<br>
</div>
<div>pool_sess 10,100,10 (default)<br>
</div>
<div>pool_vbo 10,100,10 (default)<br>
</div>
<div>prefer_ipv6 off [bool] (default)<br>
</div>
<div>rush_exponent 3 [requests per request] (default)<br>
</div>
<div>send_timeout 600.000 [seconds] (default)<br>
</div>
<div>shm_reclen 255b [bytes] (default)<br>
</div>
<div>shortlived 10.000 [seconds] (default)<br>
</div>
<div>sigsegv_handler on [bool] (default)<br>
</div>
<div>syslog_cli_traffic on [bool] (default)<br>
</div>
<div>tcp_fastopen off [bool] (default)<br>
</div>
<div>tcp_keepalive_intvl 75.000 [seconds] (default)<br>
</div>
<div>tcp_keepalive_probes 9 [probes] (default)<br>
</div>
<div>tcp_keepalive_time 7200.000 [seconds] (default)<br>
</div>
<div>thread_pool_add_delay 0.000 [seconds] (default)<br>
</div>
<div>thread_pool_destroy_delay 1.000 [seconds] (default)<br>
</div>
<div>thread_pool_fail_delay 0.200 [seconds] (default)<br>
</div>
<div>thread_pool_max 5000 [threads] (default)<br>
</div>
<div>thread_pool_min 100 [threads] (default)<br>
</div>
<div>thread_pool_reserve 0 [threads] (default)<br>
</div>
<div>thread_pool_stack 48k [bytes] (default)<br>
</div>
<div>thread_pool_timeout 300.000 [seconds] (default)<br>
</div>
<div>thread_pool_watchdog 60.000 [seconds] (default)<br>
</div>
<div>thread_pools 2 [pools] (default)<br>
</div>
<div>thread_queue_limit 20 (default)<br>
</div>
<div>thread_stats_rate 10 [requests] (default)<br>
</div>
<div>timeout_idle 5.000 [seconds] (default)<br>
</div>
<div>timeout_linger 0.050 [seconds] (default)<br>
</div>
<div>vcc_allow_inline_c off [bool] (default)<br>
</div>
<div>vcc_err_unref on [bool] (default)<br>
</div>
<div>vcc_unsafe_path on [bool] (default)<br>
</div>
<div>vcl_cooldown 600.000 [seconds] (default)<br>
</div>
<div>vcl_dir /etc/varnish:/usr/share/varnish/vcl (default)<br>
</div>
<div>vcl_path /etc/varnish:/usr/share/varnish/vcl (default)<br>
</div>
<div>vmod_dir /usr/lib/varnish/vmods (default)<br>
</div>
<div>vmod_path /usr/lib/varnish/vmods (default)<br>
</div>
<div>vsl_buffer 4k [bytes] (default)<br>
</div>
<div>vsl_mask -ObjProtocol,-ObjStatus,-ObjReason,-ObjHeader,-VCL_trace,-WorkThread,-Hash,-VfpAcct,-H2RxHdr,-H2RxBody,-H2TxHdr,-H2TxBody (default)<br>
</div>
<div>vsl_reclen 255b [bytes] (default)<br>
</div>
<div>vsl_space 80M [bytes] (default)<br>
</div>
<div>vsm_free_cooldown 60.000 [seconds] (default)<br>
</div>
<div>vsm_space 1M [bytes] (default)<br>
</div>
<div>workspace_backend 64k [bytes] (default)<br>
</div>
<div>workspace_client 64k [bytes] (default)<br>
</div>
<div>workspace_session 0.50k [bytes] (default)<br>
</div>
<span>workspace_thread 2k [bytes] (default)</span><br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
------</div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Guillaume Quintard <guillaume@varnish-software.com><br>
<b>Sent:</b> Friday, May 8, 2020 7:34 PM<br>
<b>To:</b> Batanun B <batanun@hotmail.com><br>
<b>Cc:</b> varnish-misc@varnish-cache.org <varnish-misc@varnish-cache.org><br>
<b>Subject:</b> Re: Varnish intermittently returns incomplete images</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi,
<div><br>
</div>
<div>Do you have objects that are sensibly smaller that your images in your cache?</div>
<div><br>
</div>
<div>What you are describing sounds like LRU failure (check nuke_limit in "varnishadm param.show"), basically, on a miss, varnish couldn't evict enough objects and make room for the new object, so it had to truncate it and throw it away.</div>
<div><br>
</div>
<div>If that's the issue, you can increase nuke_limit, or get a bigger cache, or segregate small and large objects into different storages.</div>
<div><br clear="all">
<div>
<div dir="ltr" class="x_gmail_signature">
<div dir="ltr">
<div>-- <br>
</div>
Guillaume Quintard<br>
</div>
</div>
</div>
<br>
</div>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Fri, May 8, 2020 at 10:14 AM Batanun B <<a href="mailto:batanun@hotmail.com">batanun@hotmail.com</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Our Varnish (test environment) intermittently returns incomplete images. So the binary content is not complete. When requesting the image from the backend directly (using curl), the complete image is returned every time (I tested 1000 times using a script).</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
This happens intermittently. Sometimes Varnish returns the complete image, sometimes half of it, sometimes 20% etc... The incomplete image is returned quickly, so I don't think there is a timeout involved (we have not configured any specific timeout in varnish).</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I see nothing special in varnishlog when this happens. But I don't know how to troubleshoot this in a good way. Any suggestions?</div>
</div>
_______________________________________________<br>
varnish-misc mailing list<br>
<a href="mailto:varnish-misc@varnish-cache.org" target="_blank">varnish-misc@varnish-cache.org</a><br>
<a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" rel="noreferrer" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
</blockquote>
</div>
</div>
</body>
</html>