varnish 2.0.4 backend errors

[Kristian Lyngstol]

On Sat, Jul 11, 2009 at 12:21:38AM +0200, Lazy wrote:
> We are having hard time figuring out what's cosing varnish 503 error,
> our backend is apache is debian 5 default, os is linux x86_64 2.6.26,
> everything is running on a single machine
> 
> /usr/local/sbin/varnishd -a 0.0.0.0:80 -f
> /usr/local/etc/varnish/default.vcl -s malloc -T localhost:9999 -w
> 10,6000,300 -u nobody

6000 threads is too much. Since it's per pool, it'll cause up to 12 000
threads to start. That's not likely to go over all that well. If you have
that sort of traffic, you need to scale out. Also, 10 thread minimum is
pretty low.

I typically recommend setting the minimum thread count to what you expect
your normal traffic to be at peak hours. It's probably a dedicated
machines, and idle threads have barely any overhead, while creating new
threads can take some time.

> running with a single backend
> .connect_timeout = 1s; added to the backend definition

Any particular reason for adding that?

> I added
> 
> sub vcl_error {
>     if (req.restarts < 10) {
>         restart;
>     }
> }
> 
> (is it possible to add a pause before doing restart ?)

No. This is also a dirty workaround for a fundamental problem.

> In about 0.1% of request we get
> 
>    10 TxRequest    b POST
>    10 TxURL        b /php
>    10 TxProtocol   b HTTP/1.1
>    10 TxHeader     b x-requested-with: XMLHttpRequest
>    10 TxHeader     b Accept-Language: pl
>    10 TxHeader     b Referer: http://www.xxxxx/php
>    10 TxHeader     b Accept: text/html, */*
>    10 TxHeader     b Content-Type: application/x-www-form-urlencoded
>    10 TxHeader     b UA-CPU: x86
>    10 TxHeader     b Accept-Encoding: gzip, deflate
>    10 TxHeader     b User-Agent: Mozilla/4.0 (compatible; MSIE 7.0;
> Windows NT 5.1)
>    10 TxHeader     b Content-Length: 8
>    10 TxHeader     b Cookie: _.1
>    10 TxHeader     b X-NovINet: v1.2
>    10 TxHeader     b Host: www.kinograj.cinema-city.pl
>    10 TxHeader     b X-Varnish: 603437812
>    10 TxHeader     b X-Forwarded-For: 79.162.xxx
>    10 BackendClose b default
>    31 VCL_call     c error
>    31 VCL_return   c deliver
>    31 Length       c 465
>    31 VCL_call     c deliver
>    31 VCL_return   c deliver
>    31 TxProtocol   c HTTP/1.1
>    31 TxStatus     c 503
> 
> machine is not overloaded, there are 150 apache running 80% of them is idle
> 
> what does
> 31 VCL_call     c error mean , a connection error, apache returned
> invalid response ?

No, it just means that vcl_error is called. BackendClose notes that the
connection to the backend was closed.

> can I get some more information about this error using some syslog in
> vcl_error or mayby in some other way ?

Possibly, but using syslog in vcl is the last thing I'd recommend.

Does your syslog say anything meaningful? Like assert-errors...

(...)
>        60064  Backend connections failures
> this is old and it's not changing now

Did the error-rate go down once you solved this? What was causing these
problems?
 
>           20  N worker threads
>         4152  N worker threads created
>            0  N worker threads not created
>            0  N worker threads limited
>            0  N queued work requests
>       226847  N overflowed work requests

This is what I mean with -w 10,6000 being wrong. After the initial startup,
overflowed work requests shouldn't grow much, and you're currently running
at only 20 threads (the minimum), which will cause overflows very fast
(consider how many connections a single client will use to fetch a front
page... You can easily imagine overflowing with just 3-4 concurrent
clients.)

But that's not really causing any 503s. Just delays while threads are
created (and removed).

-- 
Kristian Lyngstøl
Redpill Linpro AS
Tlf: +47 21544179
Mob: +47 99014497
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20090714/619a4118/attachment-0003.pgp>