varnishd child panic

[Dridi Boukelmoune]

Hi,

According to the log you provided, I understand that you have a failed
allocation in your worker thread (WS_Alloc returns NULL). Check the
n_objoverflow counter in varnishstat. You can control the size of your
workspace with the sess_workspace. The more you do in your VCL, the
more memory you'll likely need for workers.

Best Regards,
Dridi

On Thu, Apr 11, 2013 at 3:25 AM, Gao Yongwei <itxx00 at gmail.com> wrote:
> Hello,lists.
> I an using a varnish for testing on a CentOS6.3 box,  some basic info:
>     [root at cdn001 ~]# uname -r
>     2.6.32-358.2.1.el6.x86_64
>     [root at cdn001 ~]# rpm -q varnish
>     varnish-3.0.3-3.el6.art.x86_64
>     [root at cdn001 ~]# free -m
>              total       used       free     shared    buffers     cached
> Mem:         24023       1054      22968          0        137        240
> -/+ buffers/cache:        676      23347
> Swap:        12079          0      12079
> uptime:
> 09:05:24 up 6 days, 11:29,  1 user,  load average: 0.00, 0.00, 0.00
>
> and my /etc/sysconfig/varnish looks like below:
>
> WORKER_STACK_SIZE=512
> ... ...
> VARNISH_STORAGE_SIZE=16G
> VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"
> VARNISH_TTL=900
> thread_pools=2
> thread_pool_min=500
> thread_pool_max=4000
> thread_pool_timeout=120
> thread_pool_add_delay=2
> thread_pool_fail_delay=100
> sess_workspace=32768
> session_max=500000
> thread_pool_stack=16384
> connect_timeout=10
> first_byte_timeout=60
> between_bytes_timeout=60
> DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
>              -f ${VARNISH_VCL_CONF} \
>              -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT}
> \
>              -t ${VARNISH_TTL} \
>              -u varnish -g varnish \
>              -S ${VARNISH_SECRET_FILE} \
>              -p thread_pools=${thread_pools} \
>              -p thread_pool_min=${thread_pool_min} \
>              -p thread_pool_max=${thread_pool_max} \
>              -p thread_pool_timeout=${thread_pool_timeout} \
>              -p thread_pool_add_delay=${thread_pool_add_delay} \
>              -p thread_pool_fail_delay=${thread_pool_fail_delay} \
>              -p sess_workspace=${sess_workspace} \
>              -p session_max=${session_max} \
>              -p connect_timeout=${connect_timeout} \
>              -p first_byte_timeout=${first_byte_timeout} \
>              -p between_bytes_timeout=${between_bytes_timeout} \
>              -s ${VARNISH_STORAGE}"
>
>  as my backend web servers runs in many vms,so I use ' dns director':
>
> director dnsdomain dns {
>     .list = {
>         .port = "80";
>         "10.0.0.0"/24;
>     }
>     .ttl = 12h;
> }
>
> and  the connect_timeout in /etc/sysconfig/varnish has been set to a large
> number(10 s).
> everything works good since last night, I got a panic message in system log.
> By default I have 1000 workers on startup, but when this panic occur,I can
> just say 500+ workers
> using varnishstat command.  bellow is the panic message in syslog:
>
> Apr 10 22:12:58 cdn001 varnishd[25730]: Child (25731) Panic message: Assert
> error in VRT_IP_string(), cache_vrt.c line 312:#012  Condition((p =
> WS_Alloc(sp->http->ws, len)) != 0) not true.#012thread =
> (cache-worker)#012ident =
> Linux,2.6.32-358.2.1.el6.x86_64,x86_64,-smalloc,-smalloc,-hcritbit,epoll#012Backtrace:#012
> 0x42ee88: /usr/sbin/varnishd() [0x42ee88]#012  0x436dc5:
> /usr/sbin/varnishd(VRT_IP_string+0x135) [0x436dc5]#012  0x7f56bc4b922f:
> ./vcl.xlWKkvTA.so(+0xb622f) [0x7f56bc4b922f]#012  0x436203:
> /usr/sbin/varnishd(VCL_recv_method+0x43) [0x436203]#012  0x418eaf:
> /usr/sbin/varnishd(CNT_Session+0xb7f) [0x418eaf]#012  0x430bd1:
> /usr/sbin/varnishd() [0x430bd1]#012  0x7f56c388f851:
> /lib64/libpthread.so.0(+0x7851) [0x7f56c388f851]#012  0x7f56c35dd90d:
> /lib64/libc.so.6(clone+0x6d) [0x7f56c35dd90d]#012sp = 0x7f569b089008 {#012
> fd = 320, id = 320, xid = 1298044854,#012  client = 10.0.0.170 33991,#012
> step = STP_RECV,#012  handling = deliver,#012  err_code = 404, err_reason =
> (null),#012  restarts = 0, esi_level = 0#012  flags = #012  bodystatus =
> 3#012  ws = 0x7f569b089080 { overflow#012    id = "sess",#012    {s,f,r,e} =
> {0x7f569b089c78,+32768,(nil),+32768},#012  },#012  http[req] = {#012    ws =
> 0x7f569b089080[sess]#012      "GET",#012      "/images/ico_hots2.gif",#012
> "HTTP/1.1",#012      "User-Agent: Opera/9.80 (Android; Opera
> Mini/6.7.30171/29.3222; U; zh) Presto/2.8.119 Version/11.10",#012
> "Host: www.example.com",#012      "Accept: text/html, application/xml;q=0.9,
> application/xhtml+xml, image/png, image/webp, image/jpeg, image/gif,
> image/x-xbitmap, */*;q=0.1",#012      "Accept-Language: zh-cn,en;q=0.9",#012
> "Accept-Encoding: gzip, deflate",#012      "Referer:
> http://www.example.com/",#012      "Connection: Keep-Alive",#012
> "clientip: 117.149.35.78",#012      "X-OperaMini-Features: advanced,
> file_system, camera, touch, folding, viewport",#012      "Device-Stock-UA:
> Mozilla/5.0 (Linux; U; Android 2.3.5; zh-cn; BOWAY I5 Build/MocorDroid2.3.5)
> AppleWebKit/533.1
> Apr 10 22:12:58 cdn001 varnishd[25730]: child (7270) Started
> Apr 10 22:12:59 cdn001 kernel: varnishd[7270]: segfault at 0 ip
> 000000000041067d sp 00007fff4f027a40 error 6 in varnishd[400000+70000]
> Apr 10 22:12:59 cdn001 varnishd[25730]: Pushing vcls failed:#012CLI
> communication error (hdr)
> Apr 10 22:12:59 cdn001 varnishd[25730]: Child (7270) died signal=11
> Apr 10 22:12:59 cdn001 varnishd[25730]: Child (-1) said Child starts
>
> I have restarted varnish daemon  and things looks fine now, but I am
> thinking that how could this error happen? is there something wrong in my
> varnish configuration ? or something else ?
>
>     thanks .
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc