Varnish shutting down under heavy load

[Anup Shukla]

Poul-Henning Kamp wrote:
> In message <468E3D6C.9080000 at iamcool.net>, Anup Shukla writes:
>   
>> Anup Shukla wrote:
>>     
>
>   
>> Did not take it long to go down again.
>> But i am not able to find any "core" dumps.
>> Some help as to how to get one will be appreciated.
>>     
>
> Ok, next step:
>
> Run varnishd in the forground by giving it two -d options:
>
> 	varnishd -d -d [your other options]
>   

This is the output from the debug console (a lot of "Child not 
responding" messages scrolled beyond the screen buffer)
=============================
.....
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Cache child died pid=29313 status=0x86
Clean child
Child cleaned
start child pid 2961
Child said (2, 2961): <<Child starts
managed to mmap 8589934592 bytes of 8589934592
Ready
CLI ready
 >>
Child said (2, 2961): <<socktest: linger=0 sndtimeo=0 rcvtimeo=0
 >>
Child not responding to ping
Child not responding to ping
Child not responding to ping
Child not responding to ping
Cache child died pid=2961 status=0x9
Clean child
Child cleaned
start child pid 5942
Pushing vcls failed:
CLI communication error
Child said (1, 5942): <<Child starts
 >>
unlink /tmp/vcl.XXxtW5RB


varnish.log
=============================
....
  199 TxHeader     c ETag: "56373-7414-43120356ea200"
  199 TxHeader     c Content-Type: application/x-shockwave-flash
  199 TxHeader     c Content-Length: 29716
  199 TxHeader     c X-Varnish: 1888641652
  199 TxHeader     c Age: 0
  199 TxHeader     c Via: 1.1 varnish
  199 ReqEnd       c 1888641652 1183753467.604381000 
1183753469.254338000 0.000404000 0.764325000 0.885632000
    0 StatAddr       218.164.51.156 0 1534 136 274 2 0 188 62079 46278586
    0 CLI            Rd vcl.load boot /tmp/vcl.XXxtW5RB
    0 Error          CLI write failed (errno=32)


The core dump is 11gb in size !!

gdb [varnishd] [corefile]
=============================
Core was generated by `varnishd -d -d -a 192.168.100.109:80 -f 
/etc/varnish/default.vcl -s file,/cache'.
Program terminated with signal 6, Aborted.
Error while mapping shared library sections:
/tmp/vcl.XXxtW5RB: No such file or directory.
Reading symbols from /usr/lib64/libvarnish.so.0...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib64/libvarnish.so.0
Reading symbols from /usr/lib64/libvcl.so.0...(no debugging symbols 
found)...done.
Loaded symbols for /usr/lib64/libvcl.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/tls/librt.so.1...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/tls/librt.so.1
Reading symbols from /lib64/tls/libpthread.so.0...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/tls/libpthread.so.0
Reading symbols from /lib64/tls/libc.so.6...
(no debugging symbols found)...done.
Loaded symbols for /lib64/tls/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols 
found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Error while reading shared library symbols:
/tmp/vcl.XXxtW5RB: No such file or directory.
#0  0x000000386742e21d in raise () from /lib64/tls/libc.so.6
(gdb) bt
#0  0x000000386742e21d in raise () from /lib64/tls/libc.so.6
#1  0x000000386742fa1e in abort () from /lib64/tls/libc.so.6
#2  0x000000321d502205 in lbv_assert () from /usr/lib64/libvarnish.so.0
#3  0x000000000041b9fc in VSL_MgtInit ()
#4  0x000000000040a611 in Fetch ()
#5  0x00000000004086f3 in CNT_Session ()
#6  0x000000000040ed0b in WRK_Sendfile ()
#7  0x000000000040ef12 in WRK_Sendfile ()
#8  0x0000003868106137 in start_thread () from /lib64/tls/libpthread.so.0
#9  0x00000038674c7113 in clone () from /lib64/tls/libc.so.6

Anything else that i should do?

Regards
A.S