[Varnish] #669: taking to long on startup when RLIMIT_NOFILE is very high
Varnish
varnish-bugs at varnish-cache.org
Wed Mar 24 20:19:35 CET 2010
#669: taking to long on startup when RLIMIT_NOFILE is very high
----------------------+-----------------------------------------------------
Reporter: slink | Owner: phk
Type: defect | Status: new
Priority: normal | Milestone:
Component: varnishd | Version: trunk
Severity: normal | Keywords:
----------------------+-----------------------------------------------------
When RLIMIT_NOFILE is set really high, the varnishd worker child spends a
long time closing all possible fds - so long that the master process will
give up on it.
Workaround: set RLIMIT_NOFILE to a reasonable value (e.g. ulimit -n
128000)
{{{
root at haggis:~# ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
open files (-n) 1048576
pipe size (512 bytes, -p) 10
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 29995
virtual memory (kbytes, -v) unlimited
}}}
excepts from a truss:
{{{
19795: sigaction(SIGCLD, 0xFFFFFD7FFFDFF120, 0x00000000) = 0
19795: sigaction(SIGPIPE, 0xFFFFFD7FFFDFF160, 0x00000000) = 0
19795: sigaction(SIGHUP, 0xFFFFFD7FFFDFF160, 0x00000000) = 0
19795: so_socket(PF_INET, SOCK_STREAM, IPPROTO_IP, 0x00000000,
SOV_DEFAULT) = 7
19795: setsockopt(7, SOL_SOCKET, SO_REUSEADDR, 0xFFFFFD7FFFDFF110, 4,
SOV_DEFAULT) = 0
19795: bind(7, 0x0048D160, 16, SOV_SOCKBSD) = 0
19795: ioctl(7, FIONBIO, 0xFFFFFD7FFFDFF118) = 0
19795: pipe() = 8 [9]
19795: pipe() = 10 [11]
19795: pipe() = 12 [13]
19795: forkx(0) = 19797
19797: forkx() (returning as child ...) = 19795
19797: getpid() = 19797 [19795]
19797: lwp_self() = 1
19797: lwp_sigmask(SIG_SETMASK, 0x00000000, 0x00000000) = 0xFFBFFEFF
[0x0000FFFF]
19797: getuid() = 0 [0]
19797: setgid(60001) = 0
19797: setuid(60001) = 0
19797: close(0) = 0
19797: open("/dev/null", O_RDONLY) = 0
19797: fcntl(13, F_DUP2FD, 0x00000001) = 1
19797: fcntl(13, F_DUP2FD, 0x00000002) = 2
19797: fstat(4294967295, 0xFFFFFD7FFFDFF0C0) Err#9 EBADF
19797: write(1, " C l o s e d f d s :", 11) = 11
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(3) = 0
19797: write(1, " 3", 2) = 2
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(4) = 0
19797: write(1, " 4", 2) = 2
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(5) = 0
19797: write(1, " 5", 2) = 2
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(6) = 0
19797: write(1, " 6", 2) = 2
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(9) = 0
19797: write(1, " 9", 2) = 2
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(10) = 0
19797: write(1, " 1 0", 3) = 3
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(12) = 0
19797: write(1, " 1 2", 3) = 3
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(13) = 0
19797: write(1, " 1 3", 3) = 3
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(14) Err#9 EBADF
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(15) Err#9 EBADF
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(16) Err#9 EBADF
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(17) Err#9 EBADF
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(18) Err#9 EBADF
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(19) Err#9 EBADF
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(20) Err#9 EBADF
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(21) Err#9 EBADF
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(22) Err#9 EBADF
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(23) Err#9 EBADF
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(24) Err#9 EBADF
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(25) Err#9 EBADF
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(26) Err#9 EBADF
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(27) Err#9 EBADF
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(28) Err#9 EBADF
...
19797: close(261) Err#9 EBADF
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19795: write(9, 0x004824E0, 32) = 32
19795: v c l . l o a d b o o t . / v c l . O R k 8 t 3 R P . s o\n
19797: close(262) Err#9 EBADF
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
...
9797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19795: pollsys(0xFFFFFD7FFFDFEF30, 1, 0xFFFFFD7FFFDFEF00, 0x00000000)
(sleeping...)
19797: close(33129) Err#9 EBADF
...
19797: close(977638) Err#9 EBADF
19795: pollsys(0xFFFFFD7FFFDFEF30, 1, 0xFFFFFD7FFFDFEF00, 0x00000000) = 0
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(977639) Err#9 EBADF
19795: write(2, 0x00452392, 21) = 21
19795: P u s h i n g v c l s f a i l e d :
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(977640) Err#9 EBADF
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19795: write(2, 0x0048AF30, 23) = 23
19795: C L I c o m m u n i c a t i o n e r r o r
19795: write(2, "\n", 1) = 1
19795: fstat(14, 0xFFFFFD7FFFDFE640) = 0
19795: time() = 1269457119
19797: close(977641) Err#9 EBADF
19795: getpid() = 19795 [1]
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19795: putmsg(14, 0xFFFFFD7FFFDFF010, 0xFFFFFD7FFFDFF020, 0) = 0
19795: open("/var/run/syslog_door", O_RDONLY) = 7
19795: door_info(7, 0xFFFFFD7FFFDFE3E0) = 0
19795: getpid() = 19795 [1]
19797: close(977642) Err#9 EBADF
19795: door_call(7, 0xFFFFFD7FFFDFE410) = 0
19795: close(7) = 0
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(977643) Err#9 EBADF
19795: read(12, 0x0050E570, 1023) = 30
19795: C l o s e d f d s : 3 4 5 6 9 1 0 1 2 1 3
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19795: getpid() = 19795 [1]
19797: close(977644) Err#9 EBADF
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19795: unlink("./vcl.ORk8t3RP.so") = 0
19797: close(977645) Err#9 EBADF
...
19797: close(977655) Err#9 EBADF
19795: _exit(2)
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
...
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: close(1048575) Err#9 EBADF
19797: getrlimit(RLIMIT_NOFILE, 0xFFFFFD7FFFDFF140) = 0
19797: write(1, "\n", 1) Err#32 EPIPE
19797: Received signal #13, SIGPIPE [ignored]
...
19797/1: brk(0x00770FA0) = 0
19797/1: brk(0x00770FA0) = 0
19797/1: brk(0x00774FA0) = 0
19797/1: write(1, " R e a d y\n", 6) Err#32
EPIPE
19797/1: Received signal #13, SIGPIPE [ignored]
19797/1: pollsys(0xFFFFFD7FFFDFF110, 1, 0x00000000, 0x00000000) = 1
19797/1: write(2, 0x0044CC28, 31) Err#32
EPIPE
19797/1: E O F o n C L I c o n n e c t i o n , e x i t i
n g\n
19797/1: Received signal #13, SIGPIPE [ignored]
19797/1: getpid() = 19797
[1]
19797/1: _exit(0)
}}}
--
Ticket URL: <http://www.varnish-cache.org/ticket/669>
Varnish <http://varnish-cache.org/>
The Varnish HTTP Accelerator
More information about the varnish-bugs
mailing list