High CPU issue with streaming on

[Jammy]

#0. We got varnish 3.0.2 running on box 2.6.32-220.4.1.el6.x86_64, with streaming on.

#1. the process
top - 22:19:21 up 14 days, 22:28,  2 users,  load average: 10.86, 10.87, 19.51
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
Cpu(s): 64.9%us,  0.5%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.2%si, 34.5%st
Mem:   7646264k total,  6948476k used,   697788k free,   109384k buffers
Swap:  4194296k total,     7636k used,  4186660k free,  3502984k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                            
24551 nobody    20   0 4241m  40m  18m S 143.2  0.5  50:07.07 varnishd 


#2. the threads

top - 22:19:35 up 14 days, 22:29,  2 users,  load average: 11.50, 11.02, 19.42
Tasks: 637 total,  11 running, 626 sleeping,   0 stopped,   0 zombie
Cpu(s):  4.5%us,  0.8%sy,  0.0%ni, 92.3%id,  0.3%wa,  0.0%hi,  0.3%si,  1.9%st
Mem:   7646264k total,  6948856k used,   697408k free,   109396k buffers
Swap:  4194296k total,     7636k used,  4186660k free,  3503000k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                            
24757 nobody    20   0 4241m  40m  18m R 22.9  0.5   8:10.68 varnishd                                                                                                            
24759 nobody    20   0 4241m  40m  18m R 21.5  0.5   7:59.45 varnishd                                                                                                            
24958 nobody    20   0 4241m  40m  18m R 21.5  0.5   8:59.62 varnishd                                                                                                            
24761 nobody    20   0 4241m  40m  18m R 16.2  0.5   5:26.20 varnishd                                                                                                            
24954 nobody    20   0 4241m  40m  18m R 16.2  0.5   7:46.43 varnishd                                                                                                            
24952 nobody    20   0 4241m  40m  18m R 14.8  0.5   4:20.41 varnishd                                                                                                            
24960 nobody    20   0 4241m  40m  18m R 14.8  0.5   9:16.54 varnishd  

#3. the stacktraces

Thread 206 (Thread 0x7f5047012700 (LWP 24757)):
#0  0x000000360df3e606 in __memcpy_ssse3_back () from /lib64/libc.so.6
#1  0x0000003610206ac7 in ?? () from /usr/lib64/varnish/libvgz.so
#2  0x00000036102072c6 in inflate () from /usr/lib64/varnish/libvgz.so
#3  0x00000000004233aa in VGZ_Gunzip ()
#4  0x0000000000423ce1 in VGZ_WrwGunzip ()
#5  0x000000000042f702 in RES_StreamPoll ()
#6  0x0000000000423607 in ?? ()
#7  0x0000000000421e1a in FetchBody ()
#8  0x0000000000416ffd in ?? ()
#9  0x000000000041785c in CNT_Session ()
#10 0x000000000042f508 in ?? ()
#11 0x000000000042e6eb in ?? ()
#12 0x000000360e2077f1 in start_thread () from /lib64/libpthread.so.0
#13 0x000000360dee592d in clone () from /lib64/libc.so.6

Thread 204 (Thread 0x7f5045c10700 (LWP 24759)):
#0  0x0000003610206cb4 in inflate () from /usr/lib64/varnish/libvgz.so
#1  0x00000000004233aa in VGZ_Gunzip ()
#2  0x0000000000423ce1 in VGZ_WrwGunzip ()
#3  0x000000000042f702 in RES_StreamPoll ()
#4  0x0000000000423607 in ?? ()
#5  0x0000000000421e1a in FetchBody ()
#6  0x0000000000416ffd in ?? ()
#7  0x000000000041785c in CNT_Session ()
#8  0x000000000042f508 in ?? ()
#9  0x000000000042e6eb in ?? ()
#10 0x000000360e2077f1 in start_thread () from /lib64/libpthread.so.0
#11 0x000000360dee592d in clone () from /lib64/libc.so.6

Most of the threads eating lots of CPUs were all with the similar stacktraces, and never finished...

I doubt there must be a infinite-loop or something similar.  

@list, any suggestions?

Thanks

----------------------------------
Best wishes,
Jammy






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20120314/dcfb6fb3/attachment.html>