Too many 503 backend errors

[Anand Shah]

Hi,


I am running varnish on 64 bit Centos. Getting too many 503 errors
recently and i tried debugging it with some help from Forums on
mailinglist.

All requests and responses given below. I am not able to reach any
conclusion; please help me as what could be done to understand the
exact issue.


Attempts:

1. Varnish in debug mode
2. gdb
3. strace (linux system and lib calls)

All outputs given below for reference:


VARNISHLOGS:

  180 ReqStart     c 203.223.188.130 4147 583956431
  180 RxRequest    c GET
  180 RxURL        c
/0/OasDefault/LO_TataNano_Kerala_01/nano_640x480.swf?click=http://ads.mydomain.com/media/ads/click_lx.ads/www.mydomainmail.com/logout.htm/L-29/562622447/Middle/OasDefault/LO_TataNano_Kerala_01/LO_TataNano_UP_0101.html/79392b38676b7a2b4548304141525538?URL
  180 RxProtocol   c HTTP/1.1
  180 RxHeader     c Host: imads.mydomain.com
  180 RxHeader     c User-Agent: Mozilla/5.0 (Windows; U; Windows NT
5.1; en-US; rv:1.9.2.11) Gecko/20101012 Firefox/3.6.11
  180 RxHeader     c Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  180 RxHeader     c Accept-Language: en-us,en;q=0.5
  180 RxHeader     c Accept-Encoding: gzip,deflate
  180 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
  180 RxHeader     c Keep-Alive: 115
  180 RxHeader     c Connection: keep-alive
  180 RxHeader     c Referer:
http://login.mydomain.com/bn/logout.cgi?formname=general&function_name=logout&login=gopal.pgv&session_id=6L25PK1KWKlno6s8Ul4BgQBopj70kV10
  180 RxHeader     c Cookie: OAX=y9+8gkz+EH0AARU8;
RLOC=%5F%5F4i1NtmpD2pc%5F%5FU9WPPAYrSWc%5F%5FtHonjGX8AnI%5F%5Find%5F%5F;
RMFL=011PVICcU3018r0; RuW=1292401506681957; accounttype=77;
Rp=g%3D2%26a%3D20%26c%3D10%26s%3D30%26cn%3D099%26z%3D000000%26p%3D031%26e%3D00%26d%3D_9_%26i
  180 VCL_call     c recv lookup
  180 VCL_call     c hash hash
  180 VCL_call     c miss fetch
  180 FetchError   c no backend connection
  180 VCL_call     c error deliver
  180 Length       c 487
  180 VCL_call     c deliver deliver
  180 TxProtocol   c HTTP/1.1
  180 TxStatus     c 503
  180 TxResponse   c Service Unavailable
  180 TxHeader     c Content-Type: text/html; charset=utf-8
  180 TxHeader     c Content-Length: 487
  180 TxHeader     c Date: Wed, 22 Dec 2010 06:39:13 GMT
  180 TxHeader     c Connection: close
  180 TxHeader     c Server: mydomain/2.0.6
  180 TxHeader     c X-Cache: TCP_MISS
  180 ReqEnd       c 583956431 1292999948.983546972
1292999953.983323097 0.000041008 4.999706984 0.000069141


  MANUALLY SENT THE SAME REQUEST USING CURL:

  [root at mymachine ~]# curl --head --header host:imads.mydomain.com
"http://localhost//0/OasDefault/IIPM_CPL_Nov10_B3_MW/iipm_winter02_300x600_10dec_03.gif"
  HTTP/1.1 200 OK
  Last-Modified: Thu, 02 Dec 2010 12:19:14 GMT
  P3P: CP="NON NID PSAa PSDa OUR IND UNI COM NAV STA",policyref="/w3c/p3p.xml"
  Content-Type: image/gif
  cache-control: max-age=604800
  Content-Length: 4150
  Date: Wed, 22 Dec 2010 06:39:26 GMT
  Connection: keep-alive
  H-Served-By: mymachine
  Server: mydomain/2.0.6
  X-Cache: TCP_MISS

  [root at mymachine ~]# curl --head --header host:imads.mydomain.com
"http://localhost//0/OasDefault/IIPM_CPL_Nov10_B3_MW/iipm_winter02_300x600_10dec_03.gif"
  HTTP/1.1 200 OK
  Last-Modified: Thu, 02 Dec 2010 12:19:14 GMT
  P3P: CP="NON NID PSAa PSDa OUR IND UNI COM NAV STA",policyref="/w3c/p3p.xml"
  Content-Type: image/gif
  cache-control: max-age=604800
  Content-Length: 4150
  Date: Wed, 22 Dec 2010 06:39:33 GMT
  Connection: keep-alive
  H-Served-By: mymachine
  Server: mydomain/2.0.6
  X-Cache: TCP_HIT




GDB Trace:

[New Thread 0x40ccd940 (LWP 11258)]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libnsl.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging
symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/librt.so.1...
(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libpcre.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcre.so.0
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Symbol file not found for ./vcl.FANefPfn.so
0x0000003c31acae46 in poll () from /lib64/libc.so.6
(gdb) n
Cannot find bounds of current function
(gdb) bt
#0  0x0000003c31acae46 in poll () from /lib64/libc.so.6
#1  0x00000035d58056aa in CLS_Poll () from /usr/lib64/libvarnish.so.1
#2  0x0000000000414b11 in CLI_Run ()
#3  0x00000000004212a3 in child_main ()
#4  0x000000000042f6d9 in ?? ()
#5  0x000000000042ff09 in ?? ()
#6  0x00000035d5807c47 in ?? () from /usr/lib64/libvarnish.so.1
#7  0x00000035d58082d8 in vev_schedule () from /usr/lib64/libvarnish.so.1
#8  0x000000000042f943 in MGT_Run ()
#9  0x000000000043deca in main ()


VARNISG DEBUG OUTPUT:

Child (11257) not responding to ping, killing it.
Child (11257) not responding to ping, killing it.
Child (11257) not responding to ping, killing it.
Child (11257) not responding to ping, killing it.
Child (11257) not responding to ping, killing it.
Child (11257) not responding to ping, killing it.
Child (11257) not responding to ping, killing it.
Child (11257) not responding to ping, killing it.
Child (11257) not responding to ping, killing it.
Child (11257) not responding to ping, killing it.
Child (11257) not responding to ping, killing it.
Child (11257) not responding to ping, killing it.
Child (11257) not responding to ping, killing it.



Regards,
    Anand