Swap exceed & Child not responding to CLI, killing it

Hans Schou hans at moc.net
Wed Jul 3 09:09:29 CEST 2013


Den 06/20/2013 04:51 PM, Reinis Rozitis skrev:
>> Any hint or help much appreciated.
>
> Try setting sysctl vm.swappiness=0
Well, after trying that a couple of days, and upgrading to 3.0.4, I 
think I have hit a memory leak. Varnish keeps up eating mem and when 
there is no more mem I got a restart. Last one was:

Child (6477) died signal=6
Child (6477) Panic message: Assert error in VGZ_NewGzip(), cache_gzip.c 
line 209:#012  Condition(Z_OK == i) not true.#012errno = 12 (Cannot 
allocate memory)...

Is there anything in 'varnishstat -1' which can give a clue?

client_conn             626603         9.22 Client connections accepted
client_drop                  0         0.00 Connection dropped, no sess/wrk
client_req             2501923        36.83 Client requests received
cache_hit              2135392        31.43 Cache hits
cache_hitpass             2486         0.04 Cache hits for pass
cache_miss              347931         5.12 Cache misses
backend_conn             53645         0.79 Backend conn. success
backend_unhealthy            0         0.00 Backend conn. not attempted
backend_busy                 0         0.00 Backend conn. too many
backend_fail                 1         0.00 Backend conn. failures
backend_reuse           319326         4.70 Backend conn. reuses
backend_toolate          17374         0.26 Backend conn. was closed
backend_recycle         336723         4.96 Backend conn. recycles
backend_retry                5         0.00 Backend conn. retry
fetch_head                 535         0.01 Fetch head
fetch_length            357385         5.26 Fetch with Length
fetch_chunked            11490         0.17 Fetch chunked
fetch_eof                    0         0.00 Fetch EOF
fetch_bad                    0         0.00 Fetch had bad headers
fetch_close                702         0.01 Fetch wanted close
fetch_oldhttp                0         0.00 Fetch pre HTTP/1.1 closed
fetch_zero                   0         0.00 Fetch zero len
fetch_failed                 0         0.00 Fetch failed
fetch_1xx                    0         0.00 Fetch no body (1xx)
fetch_204                    0         0.00 Fetch no body (204)
fetch_304                 1243         0.02 Fetch no body (304)
n_sess_mem                 207          .   N struct sess_mem
n_sess                      70          .   N struct sess
n_object                252984          .   N struct object
n_vampireobject              0          .   N unresurrected objects
n_objectcore            253053          .   N struct objectcore
n_objecthead             58090          .   N struct objecthead
n_waitinglist               94          .   N struct waitinglist
n_vbc                       19          .   N struct vbc
n_wrk                      400          .   N worker threads
n_wrk_create               400         0.01 N worker threads created
n_wrk_failed                 0         0.00 N worker threads not created
n_wrk_max                    0         0.00 N worker threads limited
n_wrk_lqueue                 0         0.00 work request queue length
n_wrk_queued                 6         0.00 N queued work requests
n_wrk_drop                   0         0.00 N dropped work requests
n_backend                   17          .   N backends
n_expired                94945          .   N expired objects
n_lru_nuked                  0          .   N LRU nuked objects
n_lru_moved            1415668          .   N LRU moved objects
losthdr                      1         0.00 HTTP header overflows
n_objsendfile                0         0.00 Objects sent with sendfile
n_objwrite             2436792        35.87 Objects sent with write
n_objoverflow                0         0.00 Objects overflowing workspace
s_sess                  626604         9.22 Total Sessions
s_req                  2501923        36.83 Total Requests
s_pipe                    1610         0.02 Total pipe
s_pass                   23428         0.34 Total pass
s_fetch                 371355         5.47 Total fetch
s_hdrbytes           907219967     13354.04 Total header bytes
s_bodybytes        21866125041    321863.59 Total body bytes
sess_closed             107908         1.59 Session Closed
sess_pipeline             4939         0.07 Session Pipeline
sess_readahead            1677         0.02 Session Read Ahead
sess_linger            2426224        35.71 Session Linger
sess_herd              1424318        20.97 Session herd
shm_records          118481149      1744.01 SHM records
shm_writes             6661958        98.06 SHM writes
shm_flushes                  6         0.00 SHM flushes due to overflow
shm_cont                  6534         0.10 SHM MTX contention
shm_cycles                  53         0.00 SHM cycles through buffer
sms_nreq                   302         0.00 SMS allocator requests
sms_nobj                     0          .   SMS outstanding allocations
sms_nbytes                   0          .   SMS outstanding bytes
sms_balloc              401612          .   SMS bytes allocated
sms_bfree               401612          .   SMS bytes freed
backend_req             371356         5.47 Backend requests made
n_vcl                        1         0.00 N vcl total
n_vcl_avail                  1         0.00 N vcl available
n_vcl_discard                0         0.00 N vcl discarded
n_ban                      298          .   N total active bans
n_ban_gone                 277          .   N total gone bans
n_ban_add                  298         0.00 N new bans added
n_ban_retire                 0         0.00 N old bans deleted
n_ban_obj_test           32976         0.49 N objects tested
n_ban_re_test           290209         4.27 N regexps tested against
n_ban_dups                 276         0.00 N duplicate bans removed
hcb_nolock             2485461        36.59 HCB Lookups without lock
hcb_lock                139934         2.06 HCB Lookups with lock
hcb_insert              139934         2.06 HCB Inserts
esi_errors                   0         0.00 ESI parse errors (unlock)
esi_warnings                 0         0.00 ESI parse warnings (unlock)
accept_fail                  0         0.00 Accept failures
client_drop_late             0         0.00 Connection dropped late
uptime                   67936         1.00 Client uptime
dir_dns_lookups              0         0.00 DNS director lookups
dir_dns_failed               0         0.00 DNS director failed lookups
dir_dns_hit                  0         0.00 DNS director cached lookups hit
dir_dns_cache_full           0         0.00 DNS director full dnscache
vmods                        0          .   Loaded VMODs
n_gzip                  131411         1.93 Gzip operations
n_gunzip                295578         4.35 Gunzip operations
sess_pipe_overflow           0          .   Dropped sessions due to 
session pipe overflow
LCK.sms.creat                1         0.00 Created locks
LCK.sms.destroy              0         0.00 Destroyed locks
LCK.sms.locks              906         0.01 Lock Operations
LCK.sms.colls                0         0.00 Collisions
LCK.smp.creat                0         0.00 Created locks
LCK.smp.destroy              0         0.00 Destroyed locks
LCK.smp.locks                0         0.00 Lock Operations
LCK.smp.colls                0         0.00 Collisions
LCK.sma.creat                1         0.00 Created locks
LCK.sma.destroy              0         0.00 Destroyed locks
LCK.sma.locks           994791        14.64 Lock Operations
LCK.sma.colls                0         0.00 Collisions
LCK.smf.creat                1         0.00 Created locks
LCK.smf.destroy              0         0.00 Destroyed locks
LCK.smf.locks           341171         5.02 Lock Operations
LCK.smf.colls                0         0.00 Collisions
LCK.hsl.creat                0         0.00 Created locks
LCK.hsl.destroy              0         0.00 Destroyed locks
LCK.hsl.locks                0         0.00 Lock Operations
LCK.hsl.colls                0         0.00 Collisions
LCK.hcb.creat                1         0.00 Created locks
LCK.hcb.destroy              0         0.00 Destroyed locks
LCK.hcb.locks           222504         3.28 Lock Operations
LCK.hcb.colls                0         0.00 Collisions
LCK.hcl.creat                0         0.00 Created locks
LCK.hcl.destroy              0         0.00 Destroyed locks
LCK.hcl.locks                0         0.00 Lock Operations
LCK.hcl.colls                0         0.00 Collisions
LCK.vcl.creat                1         0.00 Created locks
LCK.vcl.destroy              0         0.00 Destroyed locks
LCK.vcl.locks             3172         0.05 Lock Operations
LCK.vcl.colls                0         0.00 Collisions
LCK.stat.creat               1         0.00 Created locks
LCK.stat.destroy             0         0.00 Destroyed locks
LCK.stat.locks          626744         9.23 Lock Operations
LCK.stat.colls               0         0.00 Collisions
LCK.sessmem.creat            1         0.00 Created locks
LCK.sessmem.destroy            0         0.00 Destroyed locks
LCK.sessmem.locks         631760         9.30 Lock Operations
LCK.sessmem.colls              0         0.00 Collisions
LCK.wstat.creat                1         0.00 Created locks
LCK.wstat.destroy              0         0.00 Destroyed locks
LCK.wstat.locks           168421         2.48 Lock Operations
LCK.wstat.colls                0         0.00 Collisions
LCK.herder.creat               1         0.00 Created locks
LCK.herder.destroy             0         0.00 Destroyed locks
LCK.herder.locks               1         0.00 Lock Operations
LCK.herder.colls               0         0.00 Collisions
LCK.wq.creat                   2         0.00 Created locks
LCK.wq.destroy                 0         0.00 Destroyed locks
LCK.wq.locks             3206044        47.19 Lock Operations
LCK.wq.colls                   0         0.00 Collisions
LCK.objhdr.creat          140007         2.06 Created locks
LCK.objhdr.destroy         81917         1.21 Destroyed locks
LCK.objhdr.locks        10343620       152.26 Lock Operations
LCK.objhdr.colls               0         0.00 Collisions
LCK.exp.creat                  1         0.00 Created locks
LCK.exp.destroy                0         0.00 Destroyed locks
LCK.exp.locks             517536         7.62 Lock Operations
LCK.exp.colls                  0         0.00 Collisions
LCK.lru.creat                  2         0.00 Created locks
LCK.lru.destroy                0         0.00 Destroyed locks
LCK.lru.locks             354670         5.22 Lock Operations
LCK.lru.colls                  0         0.00 Collisions
LCK.cli.creat                  1         0.00 Created locks
LCK.cli.destroy                0         0.00 Destroyed locks
LCK.cli.locks              22642         0.33 Lock Operations
LCK.cli.colls                  0         0.00 Collisions
LCK.ban.creat                  1         0.00 Created locks
LCK.ban.destroy                0         0.00 Destroyed locks
LCK.ban.locks             544373         8.01 Lock Operations
LCK.ban.colls                  0         0.00 Collisions
LCK.vbp.creat                  1         0.00 Created locks
LCK.vbp.destroy                0         0.00 Destroyed locks
LCK.vbp.locks              13583         0.20 Lock Operations
LCK.vbp.colls                  0         0.00 Collisions
LCK.vbe.creat                  1         0.00 Created locks
LCK.vbe.destroy                0         0.00 Destroyed locks
LCK.vbe.locks             107279         1.58 Lock Operations
LCK.vbe.colls                  0         0.00 Collisions
LCK.backend.creat             17         0.00 Created locks
LCK.backend.destroy            0         0.00 Destroyed locks
LCK.backend.locks         834359        12.28 Lock Operations
LCK.backend.colls              0         0.00 Collisions
SMF.s0.c_req              194123         2.86 Allocator requests
SMF.s0.c_fail                  0         0.00 Allocator failures
SMF.s0.c_bytes        4399304704     64756.60 Bytes allocated
SMF.s0.c_freed        3393925120     49957.68 Bytes freed
SMF.s0.g_alloc             58453          .   Allocations outstanding
SMF.s0.g_bytes        1005379584          .   Bytes outstanding
SMF.s0.g_space       63419129856          .   Bytes available
SMF.s0.g_smf               58533          .   N struct smf
SMF.s0.g_smf_frag              3          .   N small free smf
SMF.s0.g_smf_large            77          .   N large free smf
SMA.Transient.c_req       550339         8.10 Allocator requests
SMA.Transient.c_fail           0         0.00 Allocator failures
SMA.Transient.c_bytes  40394887294    594602.09 Bytes allocated
SMA.Transient.c_freed  30467874668    448479.08 Bytes freed
SMA.Transient.g_alloc       340120          .   Allocations outstanding
SMA.Transient.g_bytes   9927012626          .   Bytes outstanding
SMA.Transient.g_space            0          .   Bytes available
VBE.web15(192.168.4.15,,80).vcls            1          .   VCL references
VBE.web15(192.168.4.15,,80).happy           0          .   Happy health 
probes
VBE.web25(192.168.4.25,,80).vcls              1          .   VCL references
VBE.web25(192.168.4.25,,80).happy             0          .   Happy 
health probes
VBE.web78(192.168.4.78,,80).vcls            1          .   VCL references
VBE.web78(192.168.4.78,,80).happy           0          .   Happy health 
probes
VBE.web78(192.168.4.78,,8081).vcls            1          .   VCL references
VBE.web78(192.168.4.78,,8081).happy           0          .   Happy 
health probes
VBE.web32(192.168.4.32,,80).vcls                    1          . VCL 
references
VBE.web32(192.168.4.32,,80).happy                   0          . Happy 
health probes
VBE.web53(192.168.4.53,,80).vcls                      1          . VCL 
references
VBE.web53(192.168.4.53,,80).happy                     0          . Happy 
health probes
VBE.web48(192.168.4.48,,80).vcls                    1          . VCL 
references
VBE.web48(192.168.4.48,,80).happy                   0          . Happy 
health probes
VBE.web42(192.168.4.42,,80).vcls                      1          . VCL 
references
VBE.web42(192.168.4.42,,80).happy                     0          . Happy 
health probes
VBE.web43(192.168.4.43,,80).vcls                      1          . VCL 
references
VBE.web43(192.168.4.43,,80).happy                     0          . Happy 
health probes
VBE.web45(192.168.4.45,,80).vcls                     1          . VCL 
references
VBE.web45(192.168.4.45,,80).happy                    0          . Happy 
health probes
VBE.web21(192.168.4.21,,80).vcls                     1          . VCL 
references
VBE.web21(192.168.4.21,,80).happy                    0          . Happy 
health probes
VBE.web62(192.168.4.62,,80).vcls                   1          . VCL 
references
VBE.web62(192.168.4.62,,80).happy 18446744073709551615          .   
Happy health probes
VBE.web56(192.168.4.56,,80).vcls                     1          . VCL 
references
VBE.web56(192.168.4.56,,80).happy                    0          . Happy 
health probes
VBE.web30(192.168.4.30,,80).vcls                    1          . VCL 
references
VBE.web30(192.168.4.30,,80).happy                   0          . Happy 
health probes
VBE.web55(192.168.4.55,,80).vcls                     1          . VCL 
references
VBE.web55(192.168.4.55,,80).happy                    0          . Happy 
health probes
VBE.web75(192.168.4.75,,80).vcls                    1          . VCL 
references
VBE.web75(192.168.4.75,,80).happy                   0          . Happy 
health probes
VBE.web88(192.168.4.88,,80).vcls                   1          . VCL 
references
VBE.web88(192.168.4.88,,80).happy                  0          . Happy 
health probes


-- 
Venlig hilsen
Hans Schou
tel:46923438




More information about the varnish-misc mailing list