Varnish crash (SIGABRT) about every 10 mins
Gea-Suan Lin
gslin at gslin.org
Thu Nov 8 22:52:56 CET 2007
Okay it dies, last varnishstat screenshot:
0+00:36:40
Hitrate ratio: 10 100 1000
Hitrate avg: 0.7220 0.2649 0.2517
31612 0.00 14.37 Client connections accepted
34339 0.00 15.61 Client requests received
5134 0.00 2.33 Cache hits
0 0.00 0.00 Cache hits for pass
29165 0.00 13.26 Cache misses
29202 0.00 13.27 Backend connections success
0 0.00 0.00 Backend connections failures
0 0.00 0.00 Backend connections reuses
0 0.00 0.00 Backend connections recycles
6 0.00 0.00 Backend connections unused
25 . . N struct srcaddr
7 . . N active struct srcaddr
23 . . N struct sess_mem
10 . . N struct sess
29176 . . N struct object
29176 . . N struct objecthead
58338 . . N struct smf
0 . . N small free smf
1 . . N large free smf
0 . . N struct vbe_conn
17 . . N worker threads
17 0.00 0.01 N worker threads created
0 0.00 0.00 N worker threads not created
0 0.00 0.00 N worker threads limited
0 0.00 0.00 N queued work requests
17 0.00 0.01 N overflowed work requests
0 0.00 0.00 N dropped work requests
0 . . N expired objects
0 . . N LRU nuked objects
0 . . N LRU saved objects
0 . . N objects on deathrow
0 0.00 0.00 HTTP header overflows
0 0.00 0.00 Objects sent with sendfile
32398 0.00 14.73 Objects sent with write
31606 0.00 14.37 Total Sessions
34335 0.00 15.61 Total Requests
37 0.00 0.02 Total pipe
0 0.00 0.00 Total pass
29164 0.00 13.26 Total fetch
8150499 0.00 3704.77 Total header bytes
5990402809 0.00 2722910.37 Total body bytes
31100 0.00 14.14 Session Closed
0 0.00 0.00 Session Pipeline
0 0.00 0.00 Session Read Ahead
3249 0.00 1.48 Session herd
2113909 0.00 960.87 SHM records
130287 0.00 59.22 SHM writes
13 0.00 0.01 SHM MTX contention
58337 0.00 26.52 allocator requests
58337 . . outstanding allocations
5724512256 . . bytes allocated
28635226112 . . bytes free
29164 0.00 13.26 Backend requests made
and gdb console:
gslin at testphp [~] (5:15) sudo gdb /home/service/varnish/sbin/varnishd 58794
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Attaching to program: /home/service/varnish/sbin/varnishd, process 58794
Error while mapping shared library sections:
./bin.Z0UDwDnk: No such file or directory.
Reading symbols from /home/service/varnish/lib/libvarnish.so.0...done.
Loaded symbols for /home/service/varnish/lib/libvarnish.so.0
Reading symbols from /home/service/varnish/lib/libvarnishcompat.so.0...done.
Loaded symbols for /home/service/varnish/lib/libvarnishcompat.so.0
Reading symbols from /home/service/varnish/lib/libvcl.so.0...done.
Loaded symbols for /home/service/varnish/lib/libvcl.so.0
Reading symbols from /usr/lib/libthr.so.2...done.
[New Thread 0xa67d600 (LWP 100358)]
[New Thread 0xa67d400 (LWP 100334)]
[New Thread 0xa67d200 (LWP 100202)]
[New Thread 0xa67d000 (LWP 100186)]
[New Thread 0x53de00 (LWP 100184)]
[New Thread 0x53da00 (LWP 100092)]
[New Thread 0x53d800 (LWP 100067)]
[New Thread 0x53d600 (LWP 100056)]
[New Thread 0x53d000 (LWP 100235)]
Loaded symbols for /usr/lib/libthr.so.2
Reading symbols from /lib/libm.so.4...done.
Loaded symbols for /lib/libm.so.4
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Error while reading shared library symbols:
./bin.Z0UDwDnk: No such file or directory.
Reading symbols from /libexec/ld-elf.so.1...done.
Loaded symbols for /libexec/ld-elf.so.1
[Switching to Thread 0xa67d600 (LWP 100358)]
0x0000000800c7bf5a in read () from /lib/libc.so.6
(gdb) c
Continuing.
[New Thread 0xa67d800 (LWP 100407)]
[New Thread 0xa67da00 (LWP 100580)]
[New Thread 0xa67dc00 (LWP 100660)]
[New Thread 0xa67de00 (LWP 100706)]
[New Thread 0xad49000 (LWP 100770)]
[New Thread 0xad49200 (LWP 100817)]
[New Thread 0xad49400 (LWP 100879)]
[New Thread 0xad49600 (LWP 100891)]
[New Thread 0xad49800 (LWP 101241)]
[New Thread 0xad49a00 (LWP 101286)]
[New Thread 0xad49c00 (LWP 101292)]
[New Thread 0xad49e00 (LWP 101296)]
[New Thread 0xbb4e000 (LWP 101298)]
[New Thread 0xbb4e200 (LWP 101321)]
Program received signal SIGKILL, Killed.
[Switching to Thread 0xbb4e200 (LWP 101321)]
0x0000000800c1716f in brk () from /lib/libc.so.6
(gdb) bt
#0 0x0000000800c1716f in brk () from /lib/libc.so.6
#1 0x000000000c2f1000 in ?? ()
#2 0x0000000800c165ba in _UTF8_init () from /lib/libc.so.6
#3 0x0000000800c167e8 in _UTF8_init () from /lib/libc.so.6
#4 0x0000000800c170e6 in _UTF8_init () from /lib/libc.so.6
#5 0x0000000800c7966b in calloc () from /lib/libc.so.6
#6 0x0000000000411262 in HSH_Prealloc (sp=0xa6dd008) at cache_hash.c:80
#7 0x0000000000411875 in HSH_Lookup (sp=0xa6dd008) at cache_hash.c:185
#8 0x000000000040e890 in CNT_Session (sp=0xa6dd008) at cache_center.c:534
#9 0x0000000000416209 in wrk_thread (priv=0x53e5e0) at cache_pool.c:193
#10 0x000000080098729e in pthread_create () from /usr/lib/libthr.so.2
#11 0x0000000000000000 in ?? ()
Error accessing memory address 0x7ffffd1ea000: Bad address.
(gdb) up
#1 0x000000000c2f1000 in ?? ()
(gdb) up
#2 0x0000000800c165ba in _UTF8_init () from /lib/libc.so.6
(gdb) up
#3 0x0000000800c167e8 in _UTF8_init () from /lib/libc.so.6
(gdb) up
#4 0x0000000800c170e6 in _UTF8_init () from /lib/libc.so.6
(gdb) up
#5 0x0000000800c7966b in calloc () from /lib/libc.so.6
(gdb) up
#6 0x0000000000411262 in HSH_Prealloc (sp=0xa6dd008) at cache_hash.c:80
80 w->nobjhead = calloc(sizeof *w->nobjhead, 1);
(gdb) l
75 CHECK_OBJ_NOTNULL(sp, SESS_MAGIC);
76 CHECK_OBJ_NOTNULL(sp->wrk, WORKER_MAGIC);
77 w = sp->wrk;
78
79 if (w->nobjhead == NULL) {
80 w->nobjhead = calloc(sizeof *w->nobjhead, 1);
81 XXXAN(w->nobjhead);
82 w->nobjhead->magic = OBJHEAD_MAGIC;
83 VTAILQ_INIT(&w->nobjhead->objects);
84 MTX_INIT(&w->nobjhead->mtx);
(gdb) up
#7 0x0000000000411875 in HSH_Lookup (sp=0xa6dd008) at cache_hash.c:185
185 HSH_Prealloc(sp);
(gdb) l
180 CHECK_OBJ_NOTNULL(sp->http, HTTP_MAGIC);
181 AN(hash);
182 w = sp->wrk;
183 h = sp->http;
184
185 HSH_Prealloc(sp);
186 if (sp->obj != NULL) {
187 CHECK_OBJ_NOTNULL(sp->obj, OBJECT_MAGIC);
188 o = sp->obj;
189 oh = o->objhead;
(gdb) up
#8 0x000000000040e890 in CNT_Session (sp=0xa6dd008) at cache_center.c:534
534 o = HSH_Lookup(sp);
(gdb) l
529
530 VCL_hash_method(sp);
531 /* XXX check error */
532 }
533
534 o = HSH_Lookup(sp);
535
536 if (o == NULL) {
537 /*
538 * We hit a busy object, disembark worker thread and expect
(gdb) up
#9 0x0000000000416209 in wrk_thread (priv=0x53e5e0) at cache_pool.c:193
193 CNT_Session(wrq->sess);
(gdb) l
188 CHECK_OBJ_NOTNULL(wrq->sess, SESS_MAGIC);
189 wrq->sess->wrk = w;
190 CHECK_OBJ_ORNULL(w->nobj, OBJECT_MAGIC);
191 CHECK_OBJ_ORNULL(w->nobjhead, OBJHEAD_MAGIC);
192 w->used = NAN;
193 CNT_Session(wrq->sess);
194 assert(!isnan(w->used));
195 CHECK_OBJ_ORNULL(w->nobj, OBJECT_MAGIC);
196 CHECK_OBJ_ORNULL(w->nobjhead, OBJHEAD_MAGIC);
197 w->wrq = NULL;
(gdb) up
#10 0x000000080098729e in pthread_create () from /usr/lib/libthr.so.2
(gdb) up
#11 0x0000000000000000 in ?? ()
(gdb) up
Error accessing memory address 0x7ffffd1ea000: Bad address.
(gdb)
gslin at testphp [~] (5:14) sudo /usr/bin/env -i /home/service/varnish/sbin/varnishd -a 60.199.247.118:80 -f /usr/local/etc/varn
ish/image.vcl -h classic,1048583 -P /var/run/varnishd.pid -s file,/home/service/varnish-cache.mmap,32G -T 127.0.0.1:11957 -t
604800 -w 32,4096 -d -d
storage_file: filename: /home/service/varnish-cache.mmap size 32768 MegaBytes.
Classic hash: 1048583 buckets
Using old SHMFILE
rolling(1)...
rolling(2)...
start
start child pid 58794
200 0
Child said (2, 58794): <<Child starts
sizeof(struct ws) = 48
sizeof(struct http) = 584
sizeof(struct http_conn) = 48
sizeof(struct acct) = 64
sizeof(struct worker) = 1232
sizeof(struct workreq) = 24
sizeof(struct bereq) = 656
sizeof(struct storage) = 72
sizeof(struct object) = 824
sizeof(struct objhead) = 56
sizeof(struct sess) = 448
sizeof(struct vbe_conn) = 48
sizeof(struct backend) = 88
managed to mmap 34359738368 bytes of 34359738368
Ready
CLI ready
>>
200 0
200 0
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
(lots of "Child not ...")
--
* Gea-Suan Lin (public key: Using https://keyserver.pgp.com/ to search)
* If you cannot convince them, confuse them. -- Harry S Truman
More information about the varnish-misc
mailing list