From marc.fournier at camptocamp.com Fri Nov 9 16:23:49 2018 From: marc.fournier at camptocamp.com (Marc Fournier) Date: Fri, 09 Nov 2018 17:23:49 +0100 Subject: 5.1.3: unexplained lockup during several minutes Message-ID: <871s7ujk2y.fsf@lonquimay.wrk.lsn.camptocamp.com> Hello, I'm struggling figuring out a problem that occurs rarely, and random times, on different machines with the same setup (so excluding hardware problems). This leads to varnish not answering any requests anymore for several minutes. Running on a 8 CPU amazon EC2 instance with the following parameters: varnishd -F -f /etc/varnish/config.vcl -I /etc/varnish/settings.cli -n varnishd -s malloc,8G -a 127.0.0.1:8080,PROXY When the lockup occurs, the loadavg increases from 0 to about 20 and 2 out of 8 CPUs are 100% in iowait mode (maybe because I use 2 threadpools ?) There is plenty of free memory, and no disk IO activity, nothing in dmesg. Apart from varnish, there's haproxy and a couple of varnishlog and other monitoring tools running on these machines. The 2 non-default settings I set are: thread_pool_min 1000 timeout_idle 45 On specificity: the backends set quite short TTLs on the objects they produce (less than 10s), but grace time is set to 300s, so I was wondering is there might be a side-effect with the Transient store. Any hint or clue would be much welcome ! Below are the log messages and the output of varnishstat -1: 2018-10-24T02:40:27.455363000Z Error: Child (14) not responding to CLI, killed it. 2018-10-24T02:40:27.455899000Z Error: Unexpected reply from ping: 400 CLI communication error (hdr) 2018-10-24T02:41:27.512111000Z Error: Child (14) not responding to CLI, killed it. 2018-10-24T02:41:27.512321000Z Error: Unexpected reply from ping: 400 CLI communication error (hdr) 2018-10-24T02:42:27.571685000Z Error: Child (14) not responding to CLI, killed it. 2018-10-24T02:42:27.571883000Z Error: Unexpected reply from ping: 400 CLI communication error (hdr) 2018-10-24T02:43:27.598954000Z Error: Child (14) not responding to CLI, killed it. 2018-10-24T02:43:27.599156000Z Error: Unexpected reply from ping: 400 CLI communication error (hdr) 2018-10-24T02:44:27.645895000Z Error: Child (14) not responding to CLI, killed it. 2018-10-24T02:44:27.646108000Z Error: Unexpected reply from ping: 400 CLI communication error (hdr) 2018-10-24T02:45:03.252791000Z Error: Child (14) not responding to CLI, killed it. 2018-10-24T02:45:03.253069000Z Error: Unexpected reply from ping: 400 CLI communication error (hdr) 2018-10-24T02:45:03.253274000Z Error: Child (14) not responding to CLI, killed it. 2018-10-24T02:45:03.253465000Z Error: Unexpected reply from ping: 400 CLI communication error 2018-10-24T02:45:03.253769000Z Error: Child (14) died signal=3 (core dumped) 2018-10-24T02:45:03.254007000Z Debug: Child cleanup complete 2018-10-24T02:45:03.255242000Z Debug: Child (2202) Started 2018-10-24T02:45:03.339992000Z Info: Child (2202) said Child starts 2018-11-06T20:03:29.675883000Z Error: Child (2202) not responding to CLI, killed it. 2018-11-06T20:03:29.676394000Z Error: Unexpected reply from ping: 400 CLI communication error (hdr) 2018-11-06T20:04:29.701217000Z Error: Child (2202) not responding to CLI, killed it. 2018-11-06T20:04:29.701422000Z Error: Unexpected reply from ping: 400 CLI communication error (hdr) 2018-11-06T20:05:29.758068000Z Error: Child (2202) not responding to CLI, killed it. 2018-11-06T20:05:29.758269000Z Error: Unexpected reply from ping: 400 CLI communication error (hdr) 2018-11-06T20:06:29.781659000Z Error: Child (2202) not responding to CLI, killed it. 2018-11-06T20:06:29.781867000Z Error: Unexpected reply from ping: 400 CLI communication error (hdr) 2018-11-06T20:07:29.804125000Z Error: Child (2202) not responding to CLI, killed it. 2018-11-06T20:07:29.804336000Z Error: Unexpected reply from ping: 400 CLI communication error (hdr) 2018-11-06T20:08:29.864111000Z Error: Child (2202) not responding to CLI, killed it. 2018-11-06T20:08:29.864318000Z Error: Unexpected reply from ping: 400 CLI communication error (hdr) 2018-11-06T20:09:29.873115000Z Error: Child (2202) not responding to CLI, killed it. 2018-11-06T20:09:29.873321000Z Error: Unexpected reply from ping: 400 CLI communication error (hdr) 2018-11-06T20:10:29.932102000Z Error: Child (2202) not responding to CLI, killed it. 2018-11-06T20:10:29.932307000Z Error: Unexpected reply from ping: 400 CLI communication error (hdr) 2018-11-06T20:11:29.992180000Z Error: Child (2202) not responding to CLI, killed it. 2018-11-06T20:11:29.992387000Z Error: Unexpected reply from ping: 400 CLI communication error (hdr) 2018-11-06T20:12:30.052123000Z Error: Child (2202) not responding to CLI, killed it. 2018-11-06T20:12:30.052328000Z Error: Unexpected reply from ping: 400 CLI communication error (hdr) 2018-11-06T20:13:30.112074000Z Error: Child (2202) not responding to CLI, killed it. 2018-11-06T20:13:30.112275000Z Error: Unexpected reply from ping: 400 CLI communication error (hdr) 2018-11-06T20:14:30.172123000Z Error: Child (2202) not responding to CLI, killed it. 2018-11-06T20:14:30.172326000Z Error: Unexpected reply from ping: 400 CLI communication error (hdr) 2018-11-06T20:15:03.116961000Z Error: Child (2202) not responding to CLI, killed it. 2018-11-06T20:15:03.117291000Z Error: Unexpected reply from ping: 400 CLI communication error (hdr) 2018-11-06T20:15:03.117517000Z Error: Child (2202) not responding to CLI, killed it. 2018-11-06T20:15:03.117747000Z Error: Unexpected reply from ping: 400 CLI communication error 2018-11-06T20:15:03.119003000Z Error: Child (2202) died signal=3 (core dumped) 2018-11-06T20:15:03.119377000Z Debug: Child cleanup complete 2018-11-06T20:15:03.120560000Z Debug: Child (4327) Started 2018-11-06T20:15:03.207214000Z Info: Child (4327) said Child starts MAIN.uptime 244472 1.00 Child process uptime MAIN.sess_conn 10813893 44.23 Sessions accepted MAIN.sess_drop 0 0.00 Sessions dropped MAIN.sess_fail 0 0.00 Session accept failures MAIN.client_req_400 5 0.00 Client requests received, subject to 400 errors MAIN.client_req_417 0 0.00 Client requests received, subject to 417 errors MAIN.client_req 11329721 46.34 Good client requests received MAIN.cache_hit 9785347 40.03 Cache hits MAIN.cache_hitpass 0 0.00 Cache hits for pass. MAIN.cache_hitmiss 87463 0.36 Cache hits for miss. MAIN.cache_miss 439975 1.80 Cache misses MAIN.backend_conn 1233017 5.04 Backend conn. success MAIN.backend_unhealthy 0 0.00 Backend conn. not attempted MAIN.backend_busy 0 0.00 Backend conn. too many MAIN.backend_fail 0 0.00 Backend conn. failures MAIN.backend_reuse 1641618 6.71 Backend conn. reuses MAIN.backend_recycle 1647099 6.74 Backend conn. recycles MAIN.backend_retry 4 0.00 Backend conn. retry MAIN.fetch_head 6 0.00 Fetch no body (HEAD) MAIN.fetch_length 264385 1.08 Fetch with Length MAIN.fetch_chunked 15594 0.06 Fetch chunked MAIN.fetch_eof 1226485 5.02 Fetch EOF MAIN.fetch_bad 0 0.00 Fetch bad T-E MAIN.fetch_none 1 0.00 Fetch no body MAIN.fetch_1xx 0 0.00 Fetch no body (1xx) MAIN.fetch_204 8 0.00 Fetch no body (204) MAIN.fetch_304 1367111 5.59 Fetch no body (304) MAIN.fetch_failed 2 0.00 Fetch failed (all causes) MAIN.fetch_no_thread 0 0.00 Fetch failed (no thread) MAIN.pools 2 . Number of thread pools MAIN.threads 2000 . Total number of threads MAIN.threads_limited 0 0.00 Threads hit max MAIN.threads_created 2012 0.01 Threads created MAIN.threads_destroyed 12 0.00 Threads destroyed MAIN.threads_failed 0 0.00 Thread creation failed MAIN.thread_queue_len 0 . Length of session queue MAIN.busy_sleep 3558 0.01 Number of requests sent to sleep on busy objhdr MAIN.busy_wakeup 3558 0.01 Number of requests woken after sleep on busy objhdr MAIN.busy_killed 0 0.00 Number of requests killed after sleep on busy objhdr MAIN.sess_queued 1288 0.01 Sessions queued for thread MAIN.sess_dropped 0 0.00 Sessions dropped for thread MAIN.n_object 1645 . object structs made MAIN.n_vampireobject 0 . unresurrected objects MAIN.n_objectcore 3407 . objectcore structs made MAIN.n_objecthead 3649 . objecthead structs made MAIN.n_backend 2 . Number of backends MAIN.n_expired 438348 . Number of expired objects MAIN.n_lru_nuked 0 . Number of LRU nuked objects MAIN.n_lru_moved 5255908 . Number of LRU moved objects MAIN.losthdr 0 0.00 HTTP header overflows MAIN.s_sess 10813893 44.23 Total sessions seen MAIN.s_req 11329721 46.34 Total requests seen MAIN.s_pipe 1724 0.01 Total pipe sessions seen MAIN.s_pass 1017203 4.16 Total pass-ed requests seen MAIN.s_fetch 1457178 5.96 Total backend fetches initiated MAIN.s_synth 85381 0.35 Total synthethic responses made MAIN.s_req_hdrbytes 6238979422 25520.22 Request header bytes MAIN.s_req_bodybytes 197010427 805.86 Request body bytes MAIN.s_resp_hdrbytes 6906180092 28249.37 Response header bytes MAIN.s_resp_bodybytes 1332774227295 5451643.65 Response body bytes MAIN.s_pipe_hdrbytes 2686309 10.99 Pipe request header bytes MAIN.s_pipe_in 1409866 5.77 Piped bytes from client MAIN.s_pipe_out 17882349 73.15 Piped bytes to client MAIN.sess_closed 33230 0.14 Session Closed MAIN.sess_closed_err 5187 0.02 Session Closed with error MAIN.sess_readahead 0 0.00 Session Read Ahead MAIN.sess_herd 4943201 20.22 Session herd MAIN.sc_rem_close 10784144 44.11 Session OK REM_CLOSE MAIN.sc_req_close 21829 0.09 Session OK REQ_CLOSE MAIN.sc_req_http10 932 0.00 Session Err REQ_HTTP10 MAIN.sc_rx_bad 0 0.00 Session Err RX_BAD MAIN.sc_rx_body 0 0.00 Session Err RX_BODY MAIN.sc_rx_junk 5 0.00 Session Err RX_JUNK MAIN.sc_rx_overflow 0 0.00 Session Err RX_OVERFLOW MAIN.sc_rx_timeout 4250 0.02 Session Err RX_TIMEOUT MAIN.sc_tx_pipe 1724 0.01 Session OK TX_PIPE MAIN.sc_tx_error 0 0.00 Session Err TX_ERROR MAIN.sc_tx_eof 543 0.00 Session OK TX_EOF MAIN.sc_resp_close 0 0.00 Session OK RESP_CLOSE MAIN.sc_overload 0 0.00 Session Err OVERLOAD MAIN.sc_pipe_overflow 0 0.00 Session Err PIPE_OVERFLOW MAIN.sc_range_short 0 0.00 Session Err RANGE_SHORT MAIN.sc_req_http20 0 0.00 Session Err REQ_HTTP20 MAIN.sc_vcl_failure 0 0.00 Session Err VCL_FAILURE MAIN.shm_records 921799607 3770.57 SHM records MAIN.shm_writes 82947584 339.29 SHM writes MAIN.shm_flushes 1821 0.01 SHM flushes due to overflow MAIN.shm_cont 961287 3.93 SHM MTX contention MAIN.shm_cycles 436 0.00 SHM cycles through buffer MAIN.backend_req 2862111 11.71 Backend requests made MAIN.n_vcl 2 0.00 Number of loaded VCLs in total MAIN.n_vcl_avail 2 0.00 Number of VCLs available MAIN.n_vcl_discard 0 0.00 Number of discarded VCLs MAIN.vcl_fail 0 0.00 VCL failures MAIN.bans 1 . Count of bans MAIN.bans_completed 1 . Number of bans marked 'completed' MAIN.bans_obj 0 . Number of bans using obj.* MAIN.bans_req 0 . Number of bans using req.* MAIN.bans_added 1 0.00 Bans added MAIN.bans_deleted 0 0.00 Bans deleted MAIN.bans_tested 0 0.00 Bans tested against objects (lookup) MAIN.bans_obj_killed 0 0.00 Objects killed by bans (lookup) MAIN.bans_lurker_tested 0 0.00 Bans tested against objects (lurker) MAIN.bans_tests_tested 0 0.00 Ban tests tested against objects (lookup) MAIN.bans_lurker_tests_tested 0 0.00 Ban tests tested against objects (lurker) MAIN.bans_lurker_obj_killed 0 0.00 Objects killed by bans (lurker) MAIN.bans_lurker_obj_killed_cutoff 0 0.00 Objects killed by bans for cutoff (lurker) MAIN.bans_dups 0 0.00 Bans superseded by other bans MAIN.bans_lurker_contention 0 0.00 Lurker gave way for lookup MAIN.bans_persisted_bytes 16 . Bytes used by the persisted ban lists MAIN.bans_persisted_fragmentation 0 . Extra bytes in persisted ban lists due to fragmentation MAIN.n_purges 0 . Number of purge operations executed MAIN.n_obj_purged 0 . Number of purged objects MAIN.exp_mailed 3272785 13.39 Number of objects mailed to expiry thread MAIN.exp_received 3272785 13.39 Number of objects received by expiry thread MAIN.hcb_nolock 10225413 41.83 HCB Lookups without lock MAIN.hcb_lock 348309 1.42 HCB Lookups with lock MAIN.hcb_insert 348094 1.42 HCB Inserts MAIN.esi_errors 0 0.00 ESI parse errors (unlock) MAIN.esi_warnings 0 0.00 ESI parse warnings (unlock) MAIN.vmods 1 . Loaded VMODs MAIN.n_gzip 1262083 5.16 Gzip operations MAIN.n_gunzip 41886 0.17 Gunzip operations MAIN.n_test_gunzip 0 0.00 Test gunzip operations MAIN.vsm_free 973840 . Free VSM space MAIN.vsm_used 83960768 . Used VSM space MAIN.vsm_cooling 0 . Cooling VSM space MAIN.vsm_overflow 0 . Overflow VSM space MAIN.vsm_overflowed 0 0.00 Overflowed VSM space MGT.uptime 3145114 12.86 Management process uptime MGT.child_start 3 0.00 Child process started MGT.child_exit 0 0.00 Child process normal exit MGT.child_stop 0 0.00 Child process unexpected exit MGT.child_died 2 0.00 Child process died (signal) MGT.child_dump 2 0.00 Child process core dumped MGT.child_panic 0 0.00 Child process panic MEMPOOL.busyobj.live 8 . In use MEMPOOL.busyobj.pool 11 . In Pool MEMPOOL.busyobj.sz_wanted 65536 . Size requested MEMPOOL.busyobj.sz_actual 65504 . Size allocated MEMPOOL.busyobj.allocs 2875324 11.76 Allocations MEMPOOL.busyobj.frees 2875316 11.76 Frees MEMPOOL.busyobj.recycle 2862130 11.71 Recycled from pool MEMPOOL.busyobj.timeout 108162 0.44 Timed out from pool MEMPOOL.busyobj.toosmall 0 0.00 Too small to recycle MEMPOOL.busyobj.surplus 0 0.00 Too many for pool MEMPOOL.busyobj.randry 13194 0.05 Pool ran dry MEMPOOL.req0.live 3 . In use MEMPOOL.req0.pool 14 . In Pool MEMPOOL.req0.sz_wanted 65536 . Size requested MEMPOOL.req0.sz_actual 65504 . Size allocated MEMPOOL.req0.allocs 7871304 32.20 Allocations MEMPOOL.req0.frees 7871301 32.20 Frees MEMPOOL.req0.recycle 7850975 32.11 Recycled from pool MEMPOOL.req0.timeout 82821 0.34 Timed out from pool MEMPOOL.req0.toosmall 0 0.00 Too small to recycle MEMPOOL.req0.surplus 2158 0.01 Too many for pool MEMPOOL.req0.randry 20329 0.08 Pool ran dry MEMPOOL.sess0.live 14 . In use MEMPOOL.sess0.pool 13 . In Pool MEMPOOL.sess0.sz_wanted 512 . Size requested MEMPOOL.sess0.sz_actual 480 . Size allocated MEMPOOL.sess0.allocs 5407625 22.12 Allocations MEMPOOL.sess0.frees 5407611 22.12 Frees MEMPOOL.sess0.recycle 5342462 21.85 Recycled from pool MEMPOOL.sess0.timeout 184908 0.76 Timed out from pool MEMPOOL.sess0.toosmall 0 0.00 Too small to recycle MEMPOOL.sess0.surplus 30850 0.13 Too many for pool MEMPOOL.sess0.randry 65163 0.27 Pool ran dry MEMPOOL.req1.live 4 . In use MEMPOOL.req1.pool 18 . In Pool MEMPOOL.req1.sz_wanted 65536 . Size requested MEMPOOL.req1.sz_actual 65504 . Size allocated MEMPOOL.req1.allocs 7870901 32.20 Allocations MEMPOOL.req1.frees 7870897 32.20 Frees MEMPOOL.req1.recycle 7850560 32.11 Recycled from pool MEMPOOL.req1.timeout 82989 0.34 Timed out from pool MEMPOOL.req1.toosmall 0 0.00 Too small to recycle MEMPOOL.req1.surplus 1794 0.01 Too many for pool MEMPOOL.req1.randry 20341 0.08 Pool ran dry MEMPOOL.sess1.live 17 . In use MEMPOOL.sess1.pool 18 . In Pool MEMPOOL.sess1.sz_wanted 512 . Size requested MEMPOOL.sess1.sz_actual 480 . Size allocated MEMPOOL.sess1.allocs 5406276 22.11 Allocations MEMPOOL.sess1.frees 5406259 22.11 Frees MEMPOOL.sess1.recycle 5340931 21.85 Recycled from pool MEMPOOL.sess1.timeout 184777 0.76 Timed out from pool MEMPOOL.sess1.toosmall 0 0.00 Too small to recycle MEMPOOL.sess1.surplus 30947 0.13 Too many for pool MEMPOOL.sess1.randry 65345 0.27 Pool ran dry SMA.s0.c_req 4309927 17.63 Allocator requests SMA.s0.c_fail 0 0.00 Allocator failures SMA.s0.c_bytes 127888920949 523122.98 Bytes allocated SMA.s0.c_freed 127714379518 522409.03 Bytes freed SMA.s0.g_alloc 3072 . Allocations outstanding SMA.s0.g_bytes 174541431 . Bytes outstanding SMA.s0.g_space 8415393161 . Bytes available SMA.Transient.c_req 3511659 14.36 Allocator requests SMA.Transient.c_fail 0 0.00 Allocator failures SMA.Transient.c_bytes 17678767500 72314.08 Bytes allocated SMA.Transient.c_freed 17678606604 72313.42 Bytes freed SMA.Transient.g_alloc 196 . Allocations outstanding SMA.Transient.g_bytes 160896 . Bytes outstanding SMA.Transient.g_space 0 . Bytes available VBE.conf-2018-11-06_142400.default.happy 0 . Happy health probes VBE.conf-2018-11-06_142400.default.bereq_hdrbytes 1601813566 6552.14 Request header bytes VBE.conf-2018-11-06_142400.default.bereq_bodybytes 197010427 805.86 Request body bytes VBE.conf-2018-11-06_142400.default.beresp_hdrbytes 1481249422 6058.97 Response header bytes VBE.conf-2018-11-06_142400.default.beresp_bodybytes 47877555284 195840.65 Response body bytes VBE.conf-2018-11-06_142400.default.pipe_hdrbytes 2718879 11.12 Pipe request header bytes VBE.conf-2018-11-06_142400.default.pipe_out 1409866 5.77 Piped bytes to backend VBE.conf-2018-11-06_142400.default.pipe_in 17882349 73.15 Piped bytes from backend VBE.conf-2018-11-06_142400.default.conn 8 . Concurrent connections to backend VBE.conf-2018-11-06_142400.default.req 2875327 11.76 Backend requests sent LCK.backend.creat 3 0.00 Created locks LCK.backend.destroy 0 0.00 Destroyed locks LCK.backend.locks 5750647 23.52 Lock Operations LCK.backend_tcp.creat 1 0.00 Created locks LCK.backend_tcp.destroy 0 0.00 Destroyed locks LCK.backend_tcp.locks 9038321 36.97 Lock Operations LCK.ban.creat 1 0.00 Created locks LCK.ban.destroy 0 0.00 Destroyed locks LCK.ban.locks 5084447 20.80 Lock Operations LCK.busyobj.creat 3213172 13.14 Created locks LCK.busyobj.destroy 3214904 13.15 Destroyed locks LCK.busyobj.locks 30427598 124.46 Lock Operations LCK.cli.creat 1 0.00 Created locks LCK.cli.destroy 0 0.00 Destroyed locks LCK.cli.locks 81496 0.33 Lock Operations LCK.exp.creat 1 0.00 Created locks LCK.exp.destroy 0 0.00 Destroyed locks LCK.exp.locks 13382080 54.74 Lock Operations LCK.hcb.creat 1 0.00 Created locks LCK.hcb.destroy 0 0.00 Destroyed locks LCK.hcb.locks 696280 2.85 Lock Operations LCK.lru.creat 2 0.00 Created locks LCK.lru.destroy 0 0.00 Destroyed locks LCK.lru.locks 8967041 36.68 Lock Operations LCK.mempool.creat 5 0.00 Created locks LCK.mempool.destroy 0 0.00 Destroyed locks LCK.mempool.locks 60929423 249.23 Lock Operations LCK.objhdr.creat 349642 1.43 Created locks LCK.objhdr.destroy 346251 1.42 Destroyed locks LCK.objhdr.locks 75491395 308.79 Lock Operations LCK.pipestat.creat 1 0.00 Created locks LCK.pipestat.destroy 0 0.00 Destroyed locks LCK.pipestat.locks 1724 0.01 Lock Operations LCK.sess.creat 10813321 44.23 Created locks LCK.sess.destroy 10813836 44.23 Destroyed locks LCK.sess.locks 16814084 68.78 Lock Operations LCK.vbe.creat 1 0.00 Created locks LCK.vbe.destroy 0 0.00 Destroyed locks LCK.vbe.locks 81489 0.33 Lock Operations LCK.vcapace.creat 1 0.00 Created locks LCK.vcapace.destroy 0 0.00 Destroyed locks LCK.vcapace.locks 0 0.00 Lock Operations LCK.vcl.creat 1 0.00 Created locks LCK.vcl.destroy 0 0.00 Destroyed locks LCK.vcl.locks 5815522 23.79 Lock Operations LCK.vxid.creat 1 0.00 Created locks LCK.vxid.destroy 0 0.00 Destroyed locks LCK.vxid.locks 2346 0.01 Lock Operations LCK.waiter.creat 2 0.00 Created locks LCK.waiter.destroy 0 0.00 Destroyed locks LCK.waiter.locks 19733708 80.72 Lock Operations LCK.wq.creat 3 0.00 Created locks LCK.wq.destroy 0 0.00 Destroyed locks LCK.wq.locks 56100309 229.48 Lock Operations LCK.wstat.creat 1 0.00 Created locks LCK.wstat.destroy 0 0.00 Destroyed locks LCK.wstat.locks 21802570 89.18 Lock Operations LCK.sma.creat 2 0.00 Created locks LCK.sma.destroy 0 0.00 Destroyed locks LCK.sma.locks 15639885 63.97 Lock Operations Thanks ! Marc Fournier From junaid.mukhtar at gmail.com Wed Nov 28 16:11:50 2018 From: junaid.mukhtar at gmail.com (Junaid Mukhtar) Date: Wed, 28 Nov 2018 16:11:50 +0000 Subject: Varnish Closing Backend Connection Message-ID: hi Guys I am facing a rather tricky issue, where it appears that the varnish is closing the backend connection without waiting for a respones from the backend. We are using Nginx to serve static content Below is teh sequence of messages Varnish sends POST request to App App sends back 500 Internal Server Error Varnish interprets the 500 internal Server Error Varnish sends GET request to Nginx server on the same box to serve static content Varnish shows error message (even though Nginx sends the response successfully within milliseconds) - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 38 boot.staticpages 127.0.0.1 82 127.0.0.1 35064 - BackendStart 127.0.0.1 82 - FetchError backend write error: 0 (Success) - Timestamp Bereq: 1543420795.016075 5.106813 0.000099 - BackendClose 38 boot.staticpages - Timestamp Beresp: 1543420795.016497 5.107235 0.000422 - Timestamp Error: 1543420795.016503 5.107241 0.000005 - BerespProtocol HTTP/1.1 - BerespStatus 503 - BerespReason Service Unavailable - BerespReason Backend fetch failed - BerespHeader Date: Wed, 28 Nov 2018 15:59:55 GMT - BerespHeader Server: Varnish - VCL_call BACKEND_ERROR Varnish then again goes the same Nginx server to display default content. Nginx sends response and varnish accepts it and sends it back to the customer I am a bit stumped, any help with this is highly appreciated -------- Regards, Junaid -------------- next part -------------- An HTML attachment was scrubbed... URL: From miguel_3_gonzalez at yahoo.es Thu Nov 29 19:19:45 2018 From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=) Date: Thu, 29 Nov 2018 20:19:45 +0100 Subject: banning content from a multiuser web server Message-ID: <070a399e-2c3e-e396-0533-c654855ff99a@yahoo.es> Hi, ?? I do have a Cpanel server with Wordpress sites. Varnish plugin is working fine but some users do have non-Wordpress websites and sometimes they want to purge the whole website and currently I have to do it as root. ? How can I ban a domain within the SSH user account and make sure it only purges the domains that are hosted in his account? ? Regards, ? Miguel --- This email has been checked for viruses by AVG. https://www.avg.com From guillaume at varnish-software.com Thu Nov 29 22:33:52 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 29 Nov 2018 14:33:52 -0800 Subject: Varnish Closing Backend Connection In-Reply-To: References: Message-ID: Hi, Weird, are you able to show the two full backend request logs? (the one getting you the 500, then the 503)? Kind regards, -- Guillaume Quintard On Wed, Nov 28, 2018 at 8:14 AM Junaid Mukhtar wrote: > hi Guys > > I am facing a rather tricky issue, where it appears that the varnish is > closing the backend connection without waiting for a respones from the > backend. > > We are using Nginx to serve static content Below is teh sequence of > messages > > Varnish sends POST request to App > App sends back 500 Internal Server Error > Varnish interprets the 500 internal Server Error > Varnish sends GET request to Nginx server on the same box to serve static > content > Varnish shows error message (even though Nginx sends the response > successfully within milliseconds) > - VCL_call BACKEND_FETCH > - VCL_return fetch > - BackendOpen 38 boot.staticpages 127.0.0.1 82 127.0.0.1 35064 > - BackendStart 127.0.0.1 82 > - FetchError backend write error: 0 (Success) > - Timestamp Bereq: 1543420795.016075 5.106813 0.000099 > - BackendClose 38 boot.staticpages > - Timestamp Beresp: 1543420795.016497 5.107235 0.000422 > - Timestamp Error: 1543420795.016503 5.107241 0.000005 > - BerespProtocol HTTP/1.1 > - BerespStatus 503 > - BerespReason Service Unavailable > - BerespReason Backend fetch failed > - BerespHeader Date: Wed, 28 Nov 2018 15:59:55 GMT > - BerespHeader Server: Varnish > - VCL_call BACKEND_ERROR > > Varnish then again goes the same Nginx server to display default content. > Nginx sends response and varnish accepts it and sends it back to the > customer > > I am a bit stumped, any help with this is highly appreciated > > -------- > Regards, > Junaid > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kokoniimasu at gmail.com Fri Nov 30 08:12:56 2018 From: kokoniimasu at gmail.com (kokoniimasu) Date: Fri, 30 Nov 2018 17:12:56 +0900 Subject: Custom counter module(vmod_xcounter) Message-ID: Hi, All. I created a vmod_xcounter that can easily add counters. I think it can use in various situations. I'm using vsthrottle together. - vcl vcl 4.0; import xcounter; import vsthrottle; sub vcl_init { new atk_login = xcounter.vsc(hidevclname=1, groupname="attack."); ... } sub vcl_recv { ... if(req.method == "POST" && req.url ~ ****){ if(vsthrottle.is_denied("LOGIN:" + req.http.****, **, 60s)) { atk_login.incr(1); return (synth(429, "Too Many Requests")); } ... } ... } - output https://raw.githubusercontent.com/xcir/libvmod-xcounter/master/res/varnishstat.png https://github.com/xcir/libvmod-xcounter I hope that this vmod is of help to you. -- Shohei Tanaka(@xcir) http://blog.xcir.net/ (JP) From guillaume at varnish-software.com Fri Nov 30 18:12:02 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Fri, 30 Nov 2018 10:12:02 -0800 Subject: Custom counter module(vmod_xcounter) In-Reply-To: References: Message-ID: Nice! -- Guillaume Quintard On Fri, Nov 30, 2018 at 12:14 AM kokoniimasu wrote: > Hi, All. > > I created a vmod_xcounter that can easily add counters. > I think it can use in various situations. > > I'm using vsthrottle together. > > - vcl > > vcl 4.0; > import xcounter; > import vsthrottle; > > sub vcl_init { > new atk_login = xcounter.vsc(hidevclname=1, groupname="attack."); > ... > } > > sub vcl_recv { > ... > if(req.method == "POST" && req.url ~ ****){ > if(vsthrottle.is_denied("LOGIN:" + req.http.****, **, 60s)) { > atk_login.incr(1); > return (synth(429, "Too Many Requests")); > } > ... > } > ... > } > > - output > > https://raw.githubusercontent.com/xcir/libvmod-xcounter/master/res/varnishstat.png > > > https://github.com/xcir/libvmod-xcounter > > I hope that this vmod is of help to you. > > -- > Shohei Tanaka(@xcir) > http://blog.xcir.net/ (JP) > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: