[Varnish] #1328: Varnish stuck in a vcl.load frenzy
Varnish
varnish-bugs at varnish-cache.org
Fri Jul 12 14:31:46 CEST 2013
#1328: Varnish stuck in a vcl.load frenzy
-------------------+----------------------
Reporter: nuba | Type: defect
Status: new | Priority: normal
Milestone: | Component: varnishd
Version: 3.0.3 | Severity: normal
Keywords: |
-------------------+----------------------
Hello there,
This has happened a few times already, and it usually happens at about the
same time in all three instances I am running this setup: varnish being
passed a new vcl to use via varnish-agent every two minutes or so.
When it happens, varnish becomes unresponsive, and unable to enter
varnishadm.
Varnishlog shows a neverending flood of:
{{{
0 CLI - Rd vcl.load "ip-10-232-70-74_1373202357"
./vcl.Hv_KYu88.so
0 CLI - Wr 200 58 Loaded "./vcl.Hv_KYu88.so" as
"ip-10-232-70-74_1373202357"
0 CLI - Rd vcl.load "ip-10-232-70-74_1373202457"
./vcl.jdny_Rpn.so
0 CLI - Wr 200 58 Loaded "./vcl.jdny_Rpn.so" as
"ip-10-232-70-74_1373202457"
0 CLI - Rd vcl.load "ip-10-232-70-74_1373202562"
./vcl.la0GFhua.so
0 CLI - Wr 200 58 Loaded "./vcl.la0GFhua.so" as
"ip-10-232-70-74_1373202562"
0 CLI - Rd vcl.load "ip-10-232-70-74_1373202666"
./vcl.4Duon614.so
0 CLI - Wr 200 58 Loaded "./vcl.4Duon614.so" as
"ip-10-232-70-74_1373202666"
0 CLI - Rd vcl.load "ip-10-232-70-74_1373202774"
./vcl.1zMXtVdb.so
0 CLI - Wr 200 58 Loaded "./vcl.1zMXtVdb.so" as
"ip-10-232-70-74_1373202774"
0 CLI - Rd vcl.load "ip-10-232-70-74_1373202875"
./vcl.5PZ5oLqZ.so
0 CLI - Wr 200 58 Loaded "./vcl.5PZ5oLqZ.so" as
"ip-10-232-70-74_1373202875"
0 CLI - Rd vcl.load "ip-10-232-70-74_1373202976"
./vcl.vrp9XgjZ.so
0 CLI - Wr 200 58 Loaded "./vcl.vrp9XgjZ.so" as
"ip-10-232-70-74_1373202976"
0 CLI - Rd vcl.load "ip-10-232-70-74_1373203076"
./vcl.uaMggMif.so
0 CLI - Wr 200 58 Loaded "./vcl.uaMggMif.so" as
"ip-10-232-70-74_1373203076"
0 CLI - Rd vcl.load "ip-10-232-70-74_1373203176"
./vcl.K29CXnmc.so
0 CLI - Wr 200 58 Loaded "./vcl.K29CXnmc.so" as
"ip-10-232-70-74_1373203176"
0 CLI - Rd vcl.load "ip-10-232-70-74_1373203282"
./vcl.9KfydVWX.so
0 CLI - Wr 200 58 Loaded "./vcl.9KfydVWX.so" as
"ip-10-232-70-74_1373203282"
0 CLI - Rd vcl.load "ip-10-232-70-74_1373203388"
./vcl.LIeqzMOq.so
0 CLI - Wr 200 58 Loaded "./vcl.LIeqzMOq.so" as
"ip-10-232-70-74_1373203388"
0 CLI - Rd vcl.load "ip-10-232-70-74_1373203494"
./vcl.ujTZ4BCO.so
0 CLI - Wr 200 58 Loaded "./vcl.ujTZ4BCO.so" as
"ip-10-232-70-74_1373203494"
0 CLI - Rd vcl.load "ip-10-232-70-74_1373203597"
./vcl.DJZi5KHF.so
0 CLI - Wr 200 58 Loaded "./vcl.DJZi5KHF.so" as
"ip-10-232-70-74_1373203597"
0 CLI - Rd vcl.load "ip-10-232-70-74_1373203696"
./vcl.exCEPgli.so
}}}
this is on a machine that has this many compiled vcls:
{{{
root at ip-10-232-70-74:~# ls -l /var/lib/varnish/ip-10-232-70-74/ | wc -l
20615
}}}
If needed, I can "prepare" this setup in a box you could ssh to and debug
it up & broken. Or let me know if there's anything else you'd like me to
collect.
Thanks,
Nuba
Some extra info, in case it might be useful
{{{
Linux ip-10-232-70-74 3.2.0-25-virtual #40-Ubuntu SMP Wed May 23 22:20:17
UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
ubuntu at ip-10-232-70-74:~$ ldd /usr/sbin/varnishd
linux-vdso.so.1 => (0x00007fff8d3ff000)
libvarnish.so => /usr/lib/varnish/libvarnish.so
(0x00007fc8c9955000)
libvarnishcompat.so => /usr/lib/varnish/libvarnishcompat.so
(0x00007fc8c9753000)
libvcl.so => /usr/lib/varnish/libvcl.so (0x00007fc8c9531000)
libvgz.so => /usr/lib/varnish/libvgz.so (0x00007fc8c931d000)
libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3
(0x00007fc8c90d8000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2
(0x00007fc8c8ed3000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc8c8bd7000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fc8c89ba000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc8c85fa000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
(0x00007fc8c83f2000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc8c9b6b000)
ubuntu at ip-10-232-70-74:~$ ldd /usr/lib/varnish/*
/usr/lib/varnish/libvarnishcompat.so:
linux-vdso.so.1 => (0x00007fff01fff000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fa765ca0000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa7658e1000)
/lib64/ld-linux-x86-64.so.2 (0x00007fa7660c9000)
/usr/lib/varnish/libvarnish.so:
linux-vdso.so.1 => (0x00007fff85736000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1
(0x00007fdb911c4000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fdb90ec8000)
libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3
(0x00007fdb90c8a000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fdb90a6d000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdb906ae000)
/lib64/ld-linux-x86-64.so.2 (0x00007fdb915ea000)
/usr/lib/varnish/libvcl.so:
linux-vdso.so.1 => (0x00007fff9b1ff000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f73942ed000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7393f2e000)
/lib64/ld-linux-x86-64.so.2 (0x00007f7394735000)
/usr/lib/varnish/libvgz.so:
linux-vdso.so.1 => (0x00007fff609ff000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fb29b2c8000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb29af09000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb29b703000)
/usr/lib/varnish/vmods:
ldd: /usr/lib/varnish/vmods: not regular file
ubuntu at ip-10-232-70-74:~$ ldd /usr/lib/varnish/*/*
/usr/lib/varnish/vmods/libvmod_header.a:
not a dynamic executable
/usr/lib/varnish/vmods/libvmod_header.la:
not a dynamic executable
/usr/lib/varnish/vmods/libvmod_header.so:
linux-vdso.so.1 => (0x00007fff28bff000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007effe782f000)
/lib64/ld-linux-x86-64.so.2 (0x00007effe7dfc000)
/usr/lib/varnish/vmods/libvmod_std.so:
linux-vdso.so.1 => (0x00007fff3b9ff000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007f97a198b000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f97a15cc000)
/lib64/ld-linux-x86-64.so.2 (0x00007f97a1db7000)
ubuntu at ip-10-232-70-74:~$
my varnish came from
ubuntu at ip-10-232-70-74:~/varnish-3.0.3$ grep varnish /etc/apt/sources.list
deb http://repo.varnish-cache.org/ubuntu/ precise varnish-3.0
ubuntu at ip-10-232-70-74:~/varnish-3.0.3$ dpkg -l '*varnish*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-
pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version
Description
+++-==============================-==============================-============================================================================
un libvarnish-dev <none> (no
description available)
un libvarnish1 <none> (no
description available)
ii libvarnishapi-dev 3.0.3-1~precise
development files for Varnish
ii libvarnishapi1 3.0.3-1~precise shared
libraries for Varnish
ii varnish 3.0.3-1~precise state of
the art, high-performance web accelerator
un varnish-doc <none> (no
description available)
ubuntu at ip-10-232-70-74:~/varnish-3.0.3$ dpkg -p varnish
Package: varnish
Priority: optional
Section: web
Installed-Size: 1276
Maintainer: Varnish Package Maintainers <pkg-varnish-
devel at lists.alioth.debian.org>
Architecture: amd64
Version: 3.0.3-1~precise
Replaces: libvarnishapi1 (<< 3.0.0-5)
Depends: libc6 (>= 2.14), libedit2 (>= 2.11-20080614-1), libncurses5 (>=
5.5-5~), libpcre3 (>= 8.10), libtinfo5, libvarnishapi1, gcc (>= 3.3),
libc6-dev | libc6.1-dev | libc-dev, adduser
Suggests: varnish-doc
Size: 534824
Description: state of the art, high-performance web accelerator
Varnish Cache is a state of the art web accelerator written with
performance and flexibility in mind.
.
Varnish Cache stores web pages in memory so web servers don't have to
create the same web page over and over again. Varnish serves pages
much faster than any application server; giving the website a
significant speed up.
.
Some of the features include:
* A modern design
* VCL - a very flexible configuration language
* Load balancing with health checking of backends
* Partial support for ESI - Edge Side Includes
* URL rewriting
* Graceful handling of "dead" backends
Homepage: http://varnish-cache.org/
The headers vmod was compiled from source.
ubuntu at ip-10-232-70-74:~/varnish-3.0.3$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
stepping : 7
microcode : 0x70d
cpu MHz : 1795.673
cache size : 20480 KB
physical id : 1
siblings : 2
core id : 0
cpu cores : 1
apicid : 32
initial apicid : 32
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu de tsc msr pae cx8 sep cmov pat clflush mmx fxsr sse
sse2 ss ht syscall nx lm constant_tsc rep_good nopl nonstop_tsc aperfmperf
pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt
tsc_deadline_timer aes hypervisor lahf_lm arat epb xsaveopt pln pts dts
bogomips : 3591.34
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
stepping : 7
microcode : 0x70d
cpu MHz : 1795.673
cache size : 20480 KB
physical id : 1
siblings : 2
core id : 0
cpu cores : 1
apicid : 32
initial apicid : 32
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu de tsc msr pae cx8 sep cmov pat clflush mmx fxsr sse
sse2 ss ht syscall nx lm constant_tsc rep_good nopl nonstop_tsc aperfmperf
pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt
tsc_deadline_timer aes hypervisor lahf_lm arat epb xsaveopt pln pts dts
bogomips : 3591.34
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
ubuntu at ip-10-232-70-74:~/varnish-3.0.3$ free
total used free shared buffers cached
Mem: 7629480 2077108 5552372 0 80224 1036820
-/+ buffers/cache: 960064 6669416
Swap: 0 0 0
}}}
--
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1328>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator
More information about the varnish-bugs
mailing list