[Varnish] #644: Deadlock in 2.0.x under high load.

[Varnish]

#644: Deadlock in 2.0.x under high load.
----------------------+-----------------------------------------------------
 Reporter:  kmcfate   |       Owner:  phk     
     Type:  defect    |      Status:  new     
 Priority:  normal    |   Milestone:          
Component:  varnishd  |     Version:  2.0     
 Severity:  normal    |    Keywords:  deadlock
----------------------+-----------------------------------------------------
 Varnish will dead lock with all threads in queued status dropping all
 subsequent work requests when under very high load.

 Backtrace on all worker threads show blocked in write call to vca_pipes.

 vca_main thread backtrace shows it in epoll_wait.

 It is my understanding that epoll_wait in vca_main should return where
 there is a write to vca_pipes, however all writes are being blocked and no
 requests are being cleared from request queue.

 The child ping will eventually timeout and restart the child process.

 Here are the appropriate back traces:
 {{{
 #0  0x00002ad7dd327d09 in pthread_cond_wait@@GLIBC_2.3.2 () from
 /lib64/libpthread.so.0
 #1  0x000000000041afac in Lck_CondWait (cond=0x653000, lck=<value
 optimized out>) at cache_lck.c:150
 #2  0x000000000041cd49 in wrk_herder_thread (priv=<value optimized out>)
 at cache_pool.c:624
 #3  0x00002ad7dd323617 in start_thread () from /lib64/libpthread.so.0
 #4  0x00002ad7ddaa7c2d in clone () from /lib64/libc.so.6
 ------
 #0  0x00002ad7dd32afe1 in nanosleep () from /lib64/libpthread.so.0
 #1  0x00000032b08056a1 in TIM_sleep (t=<value optimized out>) at
 time.c:161
 #2  0x000000000041d54e in wrk_herdtimer_thread (priv=<value optimized
 out>) at cache_pool.c:545
 #3  0x00002ad7dd323617 in start_thread () from /lib64/libpthread.so.0
 #4  0x00002ad7ddaa7c2d in clone () from /lib64/libc.so.6
 -----
 #0  0x00002ad7dda6e0b1 in nanosleep () from /lib64/libc.so.6
 #1  0x00002ad7dda6ded4 in sleep () from /lib64/libc.so.6
 #2  0x0000000000413e19 in exp_timer (sp=0x2aaaab302008, priv=<value
 optimized out>) at cache_expire.c:294
 #3  0x000000000041d829 in wrk_bgthread (arg=0x2ad7dd004120) at
 cache_pool.c:661
 #4  0x00002ad7dd323617 in start_thread () from /lib64/libpthread.so.0
 #5  0x00002ad7ddaa7c2d in clone () from /lib64/libc.so.6
 ------
 #0  0x00002ad7dd32afe1 in nanosleep () from /lib64/libpthread.so.0
 #1  0x00000032b08056a1 in TIM_sleep (t=<value optimized out>) at
 time.c:161
 #2  0x000000000040a808 in vca_sess_timeout_ticker (arg=<value optimized
 out>) at cache_acceptor_epoll.c:216
 #3  0x00002ad7dd323617 in start_thread () from /lib64/libpthread.so.0
 #4  0x00002ad7ddaa7c2d in clone () from /lib64/libc.so.6
 ------
 #0  0x00002ad7ddaa8018 in epoll_wait () from /lib64/libc.so.6
 #1  0x000000000040a1a8 in vca_main (arg=<value optimized out>) at
 cache_acceptor_epoll.c:176
 #2  0x00002ad7dd323617 in start_thread () from /lib64/libpthread.so.0
 #3  0x00002ad7ddaa7c2d in clone () from /lib64/libc.so.6
 ------
 #0  0x00002ad7dda9ee46 in poll () from /lib64/libc.so.6
 #1  0x0000000000409523 in vca_acct (arg=<value optimized out>) at
 cache_acceptor.c:221
 #2  0x00002ad7dd323617 in start_thread () from /lib64/libpthread.so.0
 #3  0x00002ad7ddaa7c2d in clone () from /lib64/libc.so.6
 ---
 ALL 4000 of the worker threads show:

 #0  0x00002ad7dd32a6bb in write () from /lib64/libpthread.so.0
 #1  0x00000000004097a9 in vca_return_session (sp=0x2aacefff2008) at
 cache_acceptor.c:325
 #2  0x0000000000411b97 in CNT_Session (sp=0x2aacefff2008) at
 cache_center.c:110
 #3  0x000000000041d9e2 in wrk_do_cnt_sess (w=0x44441bd0, priv=<value
 optimized out>) at cache_pool.c:398
 #4  0x000000000041cf4f in wrk_thread (priv=0x2ad7dd00a0b0) at
 cache_pool.c:310
 #5  0x00002ad7dd323617 in start_thread () from /lib64/libpthread.so.0
 #6  0x00002ad7ddaa7c2d in clone () from /lib64/libc.so.6
 }}}

-- 
Ticket URL: <http://varnish-cache.org/ticket/644>
Varnish <http://varnish.projects.linpro.no/>
The Varnish HTTP Accelerator