[Varnish] #808: Assert crash on load testing

[Varnish]

#808: Assert crash on load testing
----------------------+-----------------------------------------------------
 Reporter:  diego     |        Owner:  phk     
     Type:  defect    |       Status:  reopened
 Priority:  normal    |    Milestone:          
Component:  varnishd  |      Version:  trunk   
 Severity:  normal    |   Resolution:          
 Keywords:            |  
----------------------+-----------------------------------------------------

Comment(by diego):

 Conclusion: the 5491 commit does resolve the above bug report.

 I have isolated my end of the problem and it is ridiculous, almost
 ludicrous.

 To make sure it wasn't this one machine acting up I tested the new SVN
 builds on 8 almost identical servers. Out of the 8, 2 are using a 3.0ghz
 old Intel XEON chip in hyperthreading mode.The other 6 are using 3.2ghz
 same generation Intel Xeon Chip in hyperthreading mode. Everything else is
 the same: bios, os, ram, hd, etc.

 The 6 servers are running perfectly under load testing. For whatever god
 knows what, the 2 servers running the 3.0ghz chips are asserting at the
 same point using httperf regardless of amount of times I remove all
 instances of libvar*, varnish*, and re-compile/re-install. This is much
 more than coincidence and and I don't know what to make of it. Hardware
 bug? Whatever the case, I'm taking these 2 duds offline.

 Sorry for wasting your time on this witch hunt. I still can't believe what
 I found.

 Here is the debug output with -d option from those 2 screw-ups just for
 fun.


 {{{
 Child (13102) died signal=6
 Child (13102) Panic message: Assert error in http_Write(), cache_http.c
 line 918:
   Condition((hp->hd[HTTP_HDR_STATUS].b) != 0) not true.
 thread = (cache-worker)
 ident = Linux,2.6.18-194.17.4.el5,x86_64,-smalloc,-hcritbit,epoll
 Backtrace:
   0x424ad6: pan_ic+b6
   0x4204c3: http_Write+293
   0x42703b: RES_WriteObj+12b
   0x413c0a: cnt_deliver+19a
   0x414979: CNT_Session+369
   0x426e98: wrk_do_cnt_sess+b8
   0x426187: wrk_thread_real+337
   0x2ac420a5173d: _end+2ac4203eb565
   0x2ac4211d5f6d: _end+2ac420b6fd95
 sp = 0x2aaaee480008 {
   fd = 20, id = 20, xid = 477739414,
   client = 111.151.227.72 27870,
   step = STP_DELIVER,
   handling = deliver,
   err_code = 200, err_reason = (null),
   restarts = 0, esis = 0
   ws = 0x2aaaee480080 {
     id = "sess",
     {s,f,r,e} = {0x2aaaee480cd8,+216,(nil),+512000},
   },
   http[req] = {
     ws = 0x2aaaee480080[sess]
       "GET",
       "/",
       "HTTP/1.1",
       "User-Agent: httperf/0.9.0",
       "X-Forwarded-For: 111.151.227.72",
       "host: www.myserver.net",
       "Accept-Encoding: gzip",
   },
   worker = 0x2aaaaef04e60 {
     ws = 0x2aaaaef04fd0 { overflow
       id = "wrk",
       {s,f,r,e} = {0x2aaaaee85e10,+512000,(nil),+512000},
     },
     },
     vcl = {
       srcname = {
         "input",
         "Default",
       },
     },
   obj = 0x2aaaab11c800 {
     xid = 477735378,
     ws = 0x2aaaab11c820 {
       id = "obj",
       {s,f,r,e} = {0x2aaaab11ca38,+352,(nil),+360},
     },
     http[obj] = {
       ws = 0x2aaaab11c820[obj]
         "HTTP/1.1",
         "OK",
         "Content-Encoding: gzip",
         "Vary: Accept-Encoding",
         "Date: Wed, 03 Nov 2010 15:42:30 GMT",
         "Server: LiteSpeed",
         "Keep-Alive: timeout=5, max=100",
         "Cache-Control: public,max-age=300",
         "Last-Modified: Wed, 03 Nov 2010 15:42:30 GMT",
         "Content-Type: text/html; charset=UTF-8",
         "Content-Length: 5793",
     },
     len = 5793,
     store = {
       5793 {
         1f 8b 08 00 00 00 00 00 02 03 d5 3c 6b 73 db b6 |...........<ks..|
         96 9f a3 99 fb 1f 50 a5 0d ed 49 24 4a b2 63 27 |......P...I$J.c'|
         b6 a4 5e c7 71 da 74 6a 3b 1b 2b ed de 99 9d d5 |..^.q.tj;.+.....|
         40 24 24 31 e6 ab 04 28 59 ed e6 bf ef 39 07 00 |@$$1...(Y....9..|
         [5729 more]
       },
     },
   },
 },


 Child cleanup complete

 }}}

-- 
Ticket URL: <http://varnish-cache.org/trac/ticket/808#comment:7>
Varnish <http://varnish-cache.org/>
The Varnish HTTP Accelerator