Swap exceed & Child not responding to CLI, killing it
Hans Schou
hans at moc.net
Wed Jul 3 09:09:29 CEST 2013
Den 06/20/2013 04:51 PM, Reinis Rozitis skrev:
>> Any hint or help much appreciated.
>
> Try setting sysctl vm.swappiness=0
Well, after trying that a couple of days, and upgrading to 3.0.4, I
think I have hit a memory leak. Varnish keeps up eating mem and when
there is no more mem I got a restart. Last one was:
Child (6477) died signal=6
Child (6477) Panic message: Assert error in VGZ_NewGzip(), cache_gzip.c
line 209:#012 Condition(Z_OK == i) not true.#012errno = 12 (Cannot
allocate memory)...
Is there anything in 'varnishstat -1' which can give a clue?
client_conn 626603 9.22 Client connections accepted
client_drop 0 0.00 Connection dropped, no sess/wrk
client_req 2501923 36.83 Client requests received
cache_hit 2135392 31.43 Cache hits
cache_hitpass 2486 0.04 Cache hits for pass
cache_miss 347931 5.12 Cache misses
backend_conn 53645 0.79 Backend conn. success
backend_unhealthy 0 0.00 Backend conn. not attempted
backend_busy 0 0.00 Backend conn. too many
backend_fail 1 0.00 Backend conn. failures
backend_reuse 319326 4.70 Backend conn. reuses
backend_toolate 17374 0.26 Backend conn. was closed
backend_recycle 336723 4.96 Backend conn. recycles
backend_retry 5 0.00 Backend conn. retry
fetch_head 535 0.01 Fetch head
fetch_length 357385 5.26 Fetch with Length
fetch_chunked 11490 0.17 Fetch chunked
fetch_eof 0 0.00 Fetch EOF
fetch_bad 0 0.00 Fetch had bad headers
fetch_close 702 0.01 Fetch wanted close
fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed
fetch_zero 0 0.00 Fetch zero len
fetch_failed 0 0.00 Fetch failed
fetch_1xx 0 0.00 Fetch no body (1xx)
fetch_204 0 0.00 Fetch no body (204)
fetch_304 1243 0.02 Fetch no body (304)
n_sess_mem 207 . N struct sess_mem
n_sess 70 . N struct sess
n_object 252984 . N struct object
n_vampireobject 0 . N unresurrected objects
n_objectcore 253053 . N struct objectcore
n_objecthead 58090 . N struct objecthead
n_waitinglist 94 . N struct waitinglist
n_vbc 19 . N struct vbc
n_wrk 400 . N worker threads
n_wrk_create 400 0.01 N worker threads created
n_wrk_failed 0 0.00 N worker threads not created
n_wrk_max 0 0.00 N worker threads limited
n_wrk_lqueue 0 0.00 work request queue length
n_wrk_queued 6 0.00 N queued work requests
n_wrk_drop 0 0.00 N dropped work requests
n_backend 17 . N backends
n_expired 94945 . N expired objects
n_lru_nuked 0 . N LRU nuked objects
n_lru_moved 1415668 . N LRU moved objects
losthdr 1 0.00 HTTP header overflows
n_objsendfile 0 0.00 Objects sent with sendfile
n_objwrite 2436792 35.87 Objects sent with write
n_objoverflow 0 0.00 Objects overflowing workspace
s_sess 626604 9.22 Total Sessions
s_req 2501923 36.83 Total Requests
s_pipe 1610 0.02 Total pipe
s_pass 23428 0.34 Total pass
s_fetch 371355 5.47 Total fetch
s_hdrbytes 907219967 13354.04 Total header bytes
s_bodybytes 21866125041 321863.59 Total body bytes
sess_closed 107908 1.59 Session Closed
sess_pipeline 4939 0.07 Session Pipeline
sess_readahead 1677 0.02 Session Read Ahead
sess_linger 2426224 35.71 Session Linger
sess_herd 1424318 20.97 Session herd
shm_records 118481149 1744.01 SHM records
shm_writes 6661958 98.06 SHM writes
shm_flushes 6 0.00 SHM flushes due to overflow
shm_cont 6534 0.10 SHM MTX contention
shm_cycles 53 0.00 SHM cycles through buffer
sms_nreq 302 0.00 SMS allocator requests
sms_nobj 0 . SMS outstanding allocations
sms_nbytes 0 . SMS outstanding bytes
sms_balloc 401612 . SMS bytes allocated
sms_bfree 401612 . SMS bytes freed
backend_req 371356 5.47 Backend requests made
n_vcl 1 0.00 N vcl total
n_vcl_avail 1 0.00 N vcl available
n_vcl_discard 0 0.00 N vcl discarded
n_ban 298 . N total active bans
n_ban_gone 277 . N total gone bans
n_ban_add 298 0.00 N new bans added
n_ban_retire 0 0.00 N old bans deleted
n_ban_obj_test 32976 0.49 N objects tested
n_ban_re_test 290209 4.27 N regexps tested against
n_ban_dups 276 0.00 N duplicate bans removed
hcb_nolock 2485461 36.59 HCB Lookups without lock
hcb_lock 139934 2.06 HCB Lookups with lock
hcb_insert 139934 2.06 HCB Inserts
esi_errors 0 0.00 ESI parse errors (unlock)
esi_warnings 0 0.00 ESI parse warnings (unlock)
accept_fail 0 0.00 Accept failures
client_drop_late 0 0.00 Connection dropped late
uptime 67936 1.00 Client uptime
dir_dns_lookups 0 0.00 DNS director lookups
dir_dns_failed 0 0.00 DNS director failed lookups
dir_dns_hit 0 0.00 DNS director cached lookups hit
dir_dns_cache_full 0 0.00 DNS director full dnscache
vmods 0 . Loaded VMODs
n_gzip 131411 1.93 Gzip operations
n_gunzip 295578 4.35 Gunzip operations
sess_pipe_overflow 0 . Dropped sessions due to
session pipe overflow
LCK.sms.creat 1 0.00 Created locks
LCK.sms.destroy 0 0.00 Destroyed locks
LCK.sms.locks 906 0.01 Lock Operations
LCK.sms.colls 0 0.00 Collisions
LCK.smp.creat 0 0.00 Created locks
LCK.smp.destroy 0 0.00 Destroyed locks
LCK.smp.locks 0 0.00 Lock Operations
LCK.smp.colls 0 0.00 Collisions
LCK.sma.creat 1 0.00 Created locks
LCK.sma.destroy 0 0.00 Destroyed locks
LCK.sma.locks 994791 14.64 Lock Operations
LCK.sma.colls 0 0.00 Collisions
LCK.smf.creat 1 0.00 Created locks
LCK.smf.destroy 0 0.00 Destroyed locks
LCK.smf.locks 341171 5.02 Lock Operations
LCK.smf.colls 0 0.00 Collisions
LCK.hsl.creat 0 0.00 Created locks
LCK.hsl.destroy 0 0.00 Destroyed locks
LCK.hsl.locks 0 0.00 Lock Operations
LCK.hsl.colls 0 0.00 Collisions
LCK.hcb.creat 1 0.00 Created locks
LCK.hcb.destroy 0 0.00 Destroyed locks
LCK.hcb.locks 222504 3.28 Lock Operations
LCK.hcb.colls 0 0.00 Collisions
LCK.hcl.creat 0 0.00 Created locks
LCK.hcl.destroy 0 0.00 Destroyed locks
LCK.hcl.locks 0 0.00 Lock Operations
LCK.hcl.colls 0 0.00 Collisions
LCK.vcl.creat 1 0.00 Created locks
LCK.vcl.destroy 0 0.00 Destroyed locks
LCK.vcl.locks 3172 0.05 Lock Operations
LCK.vcl.colls 0 0.00 Collisions
LCK.stat.creat 1 0.00 Created locks
LCK.stat.destroy 0 0.00 Destroyed locks
LCK.stat.locks 626744 9.23 Lock Operations
LCK.stat.colls 0 0.00 Collisions
LCK.sessmem.creat 1 0.00 Created locks
LCK.sessmem.destroy 0 0.00 Destroyed locks
LCK.sessmem.locks 631760 9.30 Lock Operations
LCK.sessmem.colls 0 0.00 Collisions
LCK.wstat.creat 1 0.00 Created locks
LCK.wstat.destroy 0 0.00 Destroyed locks
LCK.wstat.locks 168421 2.48 Lock Operations
LCK.wstat.colls 0 0.00 Collisions
LCK.herder.creat 1 0.00 Created locks
LCK.herder.destroy 0 0.00 Destroyed locks
LCK.herder.locks 1 0.00 Lock Operations
LCK.herder.colls 0 0.00 Collisions
LCK.wq.creat 2 0.00 Created locks
LCK.wq.destroy 0 0.00 Destroyed locks
LCK.wq.locks 3206044 47.19 Lock Operations
LCK.wq.colls 0 0.00 Collisions
LCK.objhdr.creat 140007 2.06 Created locks
LCK.objhdr.destroy 81917 1.21 Destroyed locks
LCK.objhdr.locks 10343620 152.26 Lock Operations
LCK.objhdr.colls 0 0.00 Collisions
LCK.exp.creat 1 0.00 Created locks
LCK.exp.destroy 0 0.00 Destroyed locks
LCK.exp.locks 517536 7.62 Lock Operations
LCK.exp.colls 0 0.00 Collisions
LCK.lru.creat 2 0.00 Created locks
LCK.lru.destroy 0 0.00 Destroyed locks
LCK.lru.locks 354670 5.22 Lock Operations
LCK.lru.colls 0 0.00 Collisions
LCK.cli.creat 1 0.00 Created locks
LCK.cli.destroy 0 0.00 Destroyed locks
LCK.cli.locks 22642 0.33 Lock Operations
LCK.cli.colls 0 0.00 Collisions
LCK.ban.creat 1 0.00 Created locks
LCK.ban.destroy 0 0.00 Destroyed locks
LCK.ban.locks 544373 8.01 Lock Operations
LCK.ban.colls 0 0.00 Collisions
LCK.vbp.creat 1 0.00 Created locks
LCK.vbp.destroy 0 0.00 Destroyed locks
LCK.vbp.locks 13583 0.20 Lock Operations
LCK.vbp.colls 0 0.00 Collisions
LCK.vbe.creat 1 0.00 Created locks
LCK.vbe.destroy 0 0.00 Destroyed locks
LCK.vbe.locks 107279 1.58 Lock Operations
LCK.vbe.colls 0 0.00 Collisions
LCK.backend.creat 17 0.00 Created locks
LCK.backend.destroy 0 0.00 Destroyed locks
LCK.backend.locks 834359 12.28 Lock Operations
LCK.backend.colls 0 0.00 Collisions
SMF.s0.c_req 194123 2.86 Allocator requests
SMF.s0.c_fail 0 0.00 Allocator failures
SMF.s0.c_bytes 4399304704 64756.60 Bytes allocated
SMF.s0.c_freed 3393925120 49957.68 Bytes freed
SMF.s0.g_alloc 58453 . Allocations outstanding
SMF.s0.g_bytes 1005379584 . Bytes outstanding
SMF.s0.g_space 63419129856 . Bytes available
SMF.s0.g_smf 58533 . N struct smf
SMF.s0.g_smf_frag 3 . N small free smf
SMF.s0.g_smf_large 77 . N large free smf
SMA.Transient.c_req 550339 8.10 Allocator requests
SMA.Transient.c_fail 0 0.00 Allocator failures
SMA.Transient.c_bytes 40394887294 594602.09 Bytes allocated
SMA.Transient.c_freed 30467874668 448479.08 Bytes freed
SMA.Transient.g_alloc 340120 . Allocations outstanding
SMA.Transient.g_bytes 9927012626 . Bytes outstanding
SMA.Transient.g_space 0 . Bytes available
VBE.web15(192.168.4.15,,80).vcls 1 . VCL references
VBE.web15(192.168.4.15,,80).happy 0 . Happy health
probes
VBE.web25(192.168.4.25,,80).vcls 1 . VCL references
VBE.web25(192.168.4.25,,80).happy 0 . Happy
health probes
VBE.web78(192.168.4.78,,80).vcls 1 . VCL references
VBE.web78(192.168.4.78,,80).happy 0 . Happy health
probes
VBE.web78(192.168.4.78,,8081).vcls 1 . VCL references
VBE.web78(192.168.4.78,,8081).happy 0 . Happy
health probes
VBE.web32(192.168.4.32,,80).vcls 1 . VCL
references
VBE.web32(192.168.4.32,,80).happy 0 . Happy
health probes
VBE.web53(192.168.4.53,,80).vcls 1 . VCL
references
VBE.web53(192.168.4.53,,80).happy 0 . Happy
health probes
VBE.web48(192.168.4.48,,80).vcls 1 . VCL
references
VBE.web48(192.168.4.48,,80).happy 0 . Happy
health probes
VBE.web42(192.168.4.42,,80).vcls 1 . VCL
references
VBE.web42(192.168.4.42,,80).happy 0 . Happy
health probes
VBE.web43(192.168.4.43,,80).vcls 1 . VCL
references
VBE.web43(192.168.4.43,,80).happy 0 . Happy
health probes
VBE.web45(192.168.4.45,,80).vcls 1 . VCL
references
VBE.web45(192.168.4.45,,80).happy 0 . Happy
health probes
VBE.web21(192.168.4.21,,80).vcls 1 . VCL
references
VBE.web21(192.168.4.21,,80).happy 0 . Happy
health probes
VBE.web62(192.168.4.62,,80).vcls 1 . VCL
references
VBE.web62(192.168.4.62,,80).happy 18446744073709551615 .
Happy health probes
VBE.web56(192.168.4.56,,80).vcls 1 . VCL
references
VBE.web56(192.168.4.56,,80).happy 0 . Happy
health probes
VBE.web30(192.168.4.30,,80).vcls 1 . VCL
references
VBE.web30(192.168.4.30,,80).happy 0 . Happy
health probes
VBE.web55(192.168.4.55,,80).vcls 1 . VCL
references
VBE.web55(192.168.4.55,,80).happy 0 . Happy
health probes
VBE.web75(192.168.4.75,,80).vcls 1 . VCL
references
VBE.web75(192.168.4.75,,80).happy 0 . Happy
health probes
VBE.web88(192.168.4.88,,80).vcls 1 . VCL
references
VBE.web88(192.168.4.88,,80).happy 0 . Happy
health probes
--
Venlig hilsen
Hans Schou
tel:46923438
More information about the varnish-misc
mailing list