race condition issue?

[Jammy]

We're using varnish 3.0.3 in production, yesterday, we're riding on a heavy DDoS attack. And two of our varnish nodes crashed.

it's very similar to https://www.varnish-cache.org/trac/ticket/516.  Following is our stack traces.

Any ideas? Thanks


(gdb) info threads
  999 Thread 0x7ff5eb7ff700 (LWP 3100)  0x00000031d000e54d in read () from /lib64/libpthread.so.0
  998 Thread 0x7ff5ed9fd700 (LWP 3097)  0x00000031d000e54d in read () from /lib64/libpthread.so.0
... ...
  5 Thread 0x7ff88a8ff700 (LWP 30905)  0x00000031d000e54d in read () from /lib64/libpthread.so.0
  4 Thread 0x7ff88cbfd700 (LWP 30903)  0x00000031d000ed2d in nanosleep () from /lib64/libpthread.so.0
  3 Thread 0x7ff88dfff700 (LWP 30901)  0x00000031d000ed2d in nanosleep () from /lib64/libpthread.so.0
  2 Thread 0x7ff7c81f2700 (LWP 31211)  0x00000031d000e054 in __lll_lock_wait () from /lib64/libpthread.so.0
* 1 Thread 0x7ff894cca780 (LWP 30898)  0x00000031d000e054 in __lll_lock_wait () from /lib64/libpthread.so.0


Program terminated with signal 3, Quit. 
#0 0x00000031d000e054 in __lll_lock_wait () from /lib64/libpthread.so.0 
Missing separate debuginfos, use: debuginfo-install varnish-3.0.3-1.201301210212.production.x86_64 
(gdb) bt 
#0 0x00000031d000e054 in __lll_lock_wait () from /lib64/libpthread.so.0 
#1 0x00000031d0009388 in _L_lock_854 () from /lib64/libpthread.so.0 
#2 0x00000031d0009257 in pthread_mutex_lock () from /lib64/libpthread.so.0 
#3 0x0000000000433b90 in vsl_get () 
#4 0x0000000000433d48 in VSLR () 
#5 0x0000000000433f12 in VSL () 
#6 0x000000000041a18e in cli_cb_before () 
#7 0x00007ff89559be05 in cls_vlu2 (priv=0x7ff894b41540, av=0x7ff76d5b4080) at cli_serve.c:260 
#8 0x00007ff89559c48b in cls_vlu (priv=0x7ff894b41540, p=0x2 <Address 0x2 out of bounds>) at cli_serve.c:339 
#9 0x00007ff89559fe19 in LineUpProcess (l=0x7ff894b33580) at vlu.c:154 
#10 0x00007ff89559ce8d in VCLS_Poll (cs=0x7ff894b03240, timeout=<value optimized out>) at cli_serve.c:528 
#11 0x000000000041a291 in CLI_Run () 
#12 0x000000000042e24c in child_main () 
#13 0x0000000000440d2c in start_child () 
#14 0x00000000004416b8 in MGT_Run () 
#15 0x000000000044fb4f in main ()



----------------------------------
Best wishes,
Jammy






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20130123/e9089d34/attachment-0001.html>