can not load more VCL with error "cannot allocate memory", but with plenty memory left
Zheng Liu
huozhe at gmail.com
Sat Sep 8 02:43:14 CEST 2012
Hi,
I have seen people report exactly the same error, but there is no solution
to it. I hope some one here can shed some light on the issue. I saw a very
high SMA.s0.c_fail "allocator failures". I think it is related. Is there a
way to free some memory to just fit in the VCL config? I hate to restart
the instance, since filling it up (160G) takes 1 day.
thanks,
Zheng
vcl.load newconfig /etc/varnish/default.vcl
106
VCL compiled.dlopen(./vcl.lA3o8iw_.so): ./vcl.lA3o8iw_.so: cannot apply
additional memory protection after relocation: Cannot allocate memory
# /usr/sbin/varnishd -V
varnishd (varnish-3.0.3 revision 9e6a70f)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2011 Varnish Software AS
# free -g
total used free shared buffers cached
Mem: 189 176 12 0 0 0
-/+ buffers/cache: 175 13
Swap: 3 0 3
# uname -a
Linux hcache6.houzz.com 3.0.0-15-server #26-Ubuntu SMP Fri Jan 20 19:07:39
UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
# ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 1550517
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1550517
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
# varnishstat -1
client_conn 520291099 520291099.00 Client connections accepted
client_drop 52 52.00 Connection dropped, no sess/wrk
client_req 520311099 520311099.00 Client requests received
cache_hit 479753215 479753215.00 Cache hits
cache_hitpass 5396 5396.00 Cache hits for pass
cache_miss 40571613 40571613.00 Cache misses
backend_conn 18647472 18647472.00 Backend conn. success
backend_unhealthy 0 0.00 Backend conn. not attempted
backend_busy 0 0.00 Backend conn. too many
backend_fail 84912 84912.00 Backend conn. failures
backend_reuse 21937169 21937169.00 Backend conn. reuses
backend_toolate 35368 35368.00 Backend conn. was closed
backend_recycle 21973014 21973014.00 Backend conn. recycles
backend_retry 1529 1529.00 Backend conn. retry
fetch_head 0 0.00 Fetch head
fetch_length 40567091 40567091.00 Fetch with Length
fetch_chunked 0 0.00 Fetch chunked
fetch_eof 0 0.00 Fetch EOF
fetch_bad 0 0.00 Fetch had bad headers
fetch_close 0 0.00 Fetch wanted close
fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed
fetch_zero 0 0.00 Fetch zero len
fetch_failed 1924 1924.00 Fetch failed
fetch_1xx 0 0.00 Fetch no body (1xx)
fetch_204 0 0.00 Fetch no body (204)
fetch_304 0 0.00 Fetch no body (304)
n_sess_mem 1119 . N struct sess_mem
n_sess 18 . N struct sess
n_object 4173835 . N struct object
n_vampireobject 0 . N unresurrected objects
n_objectcore 4174219 . N struct objectcore
n_objecthead 4190726 . N struct objecthead
n_waitinglist 12437 . N struct waitinglist
n_vbc 7 . N struct vbc
n_wrk 400 . N worker threads
n_wrk_create 465 465.00 N worker threads created
n_wrk_failed 256 256.00 N worker threads not created
n_wrk_max 0 0.00 N worker threads limited
n_wrk_lqueue 0 0.00 work request queue length
n_wrk_queued 16153 16153.00 N queued work requests
n_wrk_drop 53 53.00 N dropped work requests
n_backend 2 . N backends
n_expired 191407 . N expired objects
n_lru_nuked 36119853 . N LRU nuked objects
n_lru_moved 424530660 . N LRU moved objects
losthdr 0 0.00 HTTP header overflows
n_objsendfile 0 0.00 Objects sent with sendfile
n_objwrite 529243867 529243867.00 Objects sent with write
n_objoverflow 0 0.00 Objects overflowing workspace
s_sess 520314370 520314370.00 Total Sessions
s_req 520311099 520311099.00 Total Requests
s_pipe 2677 2677.00 Total pipe
s_pass 37582 37582.00 Total pass
s_fetch 40511177 40511177.00 Total fetch
s_hdrbytes 203065278649 203065278649.00 Total header bytes
s_bodybytes 23783551235305 23783551235305.00 Total body bytes
sess_closed 520303774 520303774.00 Session Closed
sess_pipeline 0 0.00 Session Pipeline
sess_readahead 0 0.00 Session Read Ahead
sess_linger 72892 72892.00 Session Linger
sess_herd 16001 16001.00 Session herd
shm_records 24046733247 24046733247.00 SHM records
shm_writes 2162622737 2162622737.00 SHM writes
shm_flushes 0 0.00 SHM flushes due to overflow
shm_cont 13971844 13971844.00 SHM MTX contention
shm_cycles 10341 10341.00 SHM cycles through buffer
sms_nreq 44028 44028.00 SMS allocator requests
sms_nobj 0 . SMS outstanding allocations
sms_nbytes 0 . SMS outstanding bytes
sms_balloc 18403704 . SMS bytes allocated
sms_bfree 18403704 . SMS bytes freed
backend_req 40582821 40582821.00 Backend requests made
n_vcl 1 1.00 N vcl total
n_vcl_avail 1 1.00 N vcl available
n_vcl_discard 0 0.00 N vcl discarded
n_ban 1 . N total active bans
n_ban_gone 1 . N total gone bans
n_ban_add 1 1.00 N new bans added
n_ban_retire 0 0.00 N old bans deleted
n_ban_obj_test 0 0.00 N objects tested
n_ban_re_test 0 0.00 N regexps tested against
n_ban_dups 0 0.00 N duplicate bans removed
hcb_nolock 519442684 519442684.00 HCB Lookups without lock
hcb_lock 40524771 40524771.00 HCB Lookups with lock
hcb_insert 40524740 40524740.00 HCB Inserts
esi_errors 0 0.00 ESI parse errors (unlock)
esi_warnings 0 0.00 ESI parse warnings (unlock)
accept_fail 0 0.00 Accept failures
client_drop_late 1 1.00 Connection dropped late
uptime 755101 755101.00 Client uptime
dir_dns_lookups 0 0.00 DNS director lookups
dir_dns_failed 0 0.00 DNS director failed lookups
dir_dns_hit 0 0.00 DNS director cached lookups hit
dir_dns_cache_full 0 0.00 DNS director full dnscache
vmods 0 . Loaded VMODs
n_gzip 0 0.00 Gzip operations
n_gunzip 77505 77505.00 Gunzip operations
LCK.sms.creat 1 1.00 Created locks
LCK.sms.destroy 0 0.00 Destroyed locks
LCK.sms.locks 132084 132084.00 Lock Operations
LCK.sms.colls 0 0.00 Collisions
LCK.smp.creat 0 0.00 Created locks
LCK.smp.destroy 0 0.00 Destroyed locks
LCK.smp.locks 0 0.00 Lock Operations
LCK.smp.colls 0 0.00 Collisions
LCK.sma.creat 2 2.00 Created locks
LCK.sma.destroy 0 0.00 Destroyed locks
LCK.sma.locks 200409429 200409429.00 Lock Operations
LCK.sma.colls 0 0.00 Collisions
LCK.smf.creat 0 0.00 Created locks
LCK.smf.destroy 0 0.00 Destroyed locks
LCK.smf.locks 0 0.00 Lock Operations
LCK.smf.colls 0 0.00 Collisions
LCK.hsl.creat 0 0.00 Created locks
LCK.hsl.destroy 0 0.00 Destroyed locks
LCK.hsl.locks 0 0.00 Lock Operations
LCK.hsl.colls 0 0.00 Collisions
LCK.hcb.creat 1 1.00 Created locks
LCK.hcb.destroy 0 0.00 Destroyed locks
LCK.hcb.locks 76879907 76879907.00 Lock Operations
LCK.hcb.colls 0 0.00 Collisions
LCK.hcl.creat 0 0.00 Created locks
LCK.hcl.destroy 0 0.00 Destroyed locks
LCK.hcl.locks 0 0.00 Lock Operations
LCK.hcl.colls 0 0.00 Collisions
LCK.vcl.creat 1 1.00 Created locks
LCK.vcl.destroy 0 0.00 Destroyed locks
LCK.vcl.locks 26598 26598.00 Lock Operations
LCK.vcl.colls 0 0.00 Collisions
LCK.stat.creat 1 1.00 Created locks
LCK.stat.destroy 0 0.00 Destroyed locks
LCK.stat.locks 520315541 520315541.00 Lock Operations
LCK.stat.colls 0 0.00 Collisions
LCK.sessmem.creat 1 1.00 Created locks
LCK.sessmem.destroy 0 0.00 Destroyed locks
LCK.sessmem.locks 521596676 521596676.00 Lock Operations
LCK.sessmem.colls 0 0.00 Collisions
LCK.wstat.creat 1 1.00 Created locks
LCK.wstat.destroy 0 0.00 Destroyed locks
LCK.wstat.locks 1611720 1611720.00 Lock Operations
LCK.wstat.colls 0 0.00 Collisions
LCK.herder.creat 1 1.00 Created locks
LCK.herder.destroy 0 0.00 Destroyed locks
LCK.herder.locks 2884 2884.00 Lock Operations
LCK.herder.colls 0 0.00 Collisions
LCK.wq.creat 8 8.00 Created locks
LCK.wq.destroy 0 0.00 Destroyed locks
LCK.wq.locks 1046638906 1046638906.00 Lock Operations
LCK.wq.colls 0 0.00 Collisions
LCK.objhdr.creat 40523726 40523726.00 Created locks
LCK.objhdr.destroy 36334480 36334480.00 Destroyed locks
LCK.objhdr.locks 2153405517 2153405517.00 Lock Operations
LCK.objhdr.colls 0 0.00 Collisions
LCK.exp.creat 1 1.00 Created locks
LCK.exp.destroy 0 0.00 Destroyed locks
LCK.exp.locks 77551306 77551306.00 Lock Operations
LCK.exp.colls 0 0.00 Collisions
LCK.lru.creat 2 2.00 Created locks
LCK.lru.destroy 0 0.00 Destroyed locks
LCK.lru.locks 76604938 76604938.00 Lock Operations
LCK.lru.colls 0 0.00 Collisions
LCK.cli.creat 1 1.00 Created locks
LCK.cli.destroy 0 0.00 Destroyed locks
LCK.cli.locks 65 65.00 Lock Operations
LCK.cli.colls 0 0.00 Collisions
LCK.ban.creat 1 1.00 Created locks
LCK.ban.destroy 0 0.00 Destroyed locks
LCK.ban.locks 77555176 77555176.00 Lock Operations
LCK.ban.colls 0 0.00 Collisions
LCK.vbp.creat 1 1.00 Created locks
LCK.vbp.destroy 0 0.00 Destroyed locks
LCK.vbp.locks 0 0.00 Lock Operations
LCK.vbp.colls 0 0.00 Collisions
LCK.vbe.creat 1 1.00 Created locks
LCK.vbe.destroy 0 0.00 Destroyed locks
LCK.vbe.locks 37466927 37466927.00 Lock Operations
LCK.vbe.colls 0 0.00 Collisions
LCK.backend.creat 3 3.00 Created locks
LCK.backend.destroy 0 0.00 Destroyed locks
LCK.backend.locks 100145698 100145698.00 Lock Operations
LCK.backend.colls 0 0.00 Collisions
SMA.s0.c_req 123055964 123055964.00 Allocator requests
SMA.s0.c_fail 3551561563273 3551561563273.00 Allocator failures
SMA.s0.c_bytes 1615116324709 1615116324709.00 Bytes allocated
SMA.s0.c_freed 1447612608403 1447612608403.00 Bytes freed
SMA.s0.g_alloc 8654138 . Allocations outstanding
SMA.s0.g_bytes 167503716306 . Bytes outstanding
SMA.s0.g_space 8238 . Bytes available
SMA.Transient.c_req 43934 43934.00 Allocator requests
SMA.Transient.c_fail 0 0.00 Allocator failures
SMA.Transient.c_bytes 646003217 646003217.00 Bytes allocated
SMA.Transient.c_freed 646003217 646003217.00 Bytes freed
SMA.Transient.g_alloc 0 . Allocations outstanding
SMA.Transient.g_bytes 0 . Bytes outstanding
SMA.Transient.g_space 0 . Bytes available
VBE.node_master(192.168.0.21,,80).vcls 1 . VCL
references
VBE.node_master(192.168.0.21,,80).happy 0 . Happy
health probes
VBE.node_slave(192.168.0.39,,80).vcls 1 . VCL
references
VBE.node_slave(192.168.0.39,,80).happy 0 . Happy
health probes
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20120907/3b98cb48/attachment-0001.html>
More information about the varnish-misc
mailing list