"Large" Binary Files triggering 503 Response
Douglas R Taber
drtaber at northcarolina.edu
Thu Jan 12 23:48:11 CET 2012
Hi all-
I'm relatively new to varnish, but i haven't seen/stumbled upon a similar situation in the last few months of mailing list archives. I'm having a weird issue where below approximately 200k, varnish returns binary files just fine, but above that it returns a 503 page. According to the log, apache is returning a 200 status code from the backend. I'm not really sure what to make of it, but it looks like the exact error being thrown back is:
11 FetchError c straight read_error: 0 0 (Remote closed connection)
I've included my configs and the associated log segment. Hopefully someone can shed some light on it. The server version information is:
Varnish Server:
RHEL6.2
Varnish 3.0.2 (from the EL5 RPM)
Apache Server:
RHEL6.2
Apache 2.2.x
The varnish command is(from /etc/sysconfig/varnish):
DAEMON_OPTS="-a 80\
-f /etc/varnish/default.vcl \
-T 127.0.0.1:6082 \
-t 120 \
-w 10,5000,120
-u varnish -g varnish \
-S /etc/varnish/secret \
-p thread_pool_add_delay=2 \
-p thread_pools=4 \
-p thread_pool_min=200 \
-p thread_pool_max=4000 \
-p cli_timeout=25 \
-p session_linger=60 \
-p sess_timeout=60 \
-s malloc,10G \
-s $file,/var/lib/varnish/varnish_storage.bin,15G"
The default config is:
backend default {
.host = "10.11.12.1";
.port = "80";
}
include "/etc/varnish/federation.vcl";
include "/etc/varnish/mailwatch.vcl";
include "/etc/varnish/devcollab.vcl";
include "/etc/varnish/www.vcl";
include "/etc/varnish/collab.vcl";
include "/etc/varnish/uncdm.vcl";
include "/etc/varnish/services.vcl";
include "/etc/varnish/devfederation.vcl";
include "/etc/varnish/devservices.vcl";
include "/etc/varnish/devuncdm.vcl";
include "/etc/varnish/devwww.vcl";
sub vcl_recv {
# if (req.url ~ "(?i)\.(png|gif|jpeg|jpg|ico|css|js|gz|tgz|bz2|tbz|mp3|ogg|mp4|flv|f4v|pdf|swf)(\?[a-z0-9]+)?$") {
# unset req.http.Cookie;
# return(lookup);
# }
if (req.restarts == 0) {
if (req.http.x-forwarded-for) {
set req.http.X-Forwarded-For = req.http.X-Forwarded-For;
} else {
set req.http.X-Forwarded-For = client.ip;
}
}
if (req.request != "GET" &&
req.request != "HEAD" &&
req.request != "PUT" &&
req.request != "POST" &&
req.request != "TRACE" &&
req.request != "OPTIONS" &&
req.request != "DELETE") {
/* Non-RFC2616 or CONNECT which is weird. */
return (pipe);
}
if (req.request != "GET" && req.request != "HEAD") {
/* We only deal with GET and HEAD by default */
return (pass);
}
if (req.http.Authorization || req.http.Cookie) {
/* Not cacheable by default */
return (pass);
}
return (lookup);
}
sub vcl_error {
set obj.http.Content-Type = "text/html; charset=utf-8";
synthetic {"
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<style type='text/css'>
#msg
{
text-align: center;
}
{
text-align: center;
}
</style>
</head>
<body>
<div id='msg'>
<h3>The University of North Carolina General Administration</h3>
<h3>Our websites are currently down for maintenance.<br />
We apologize for the inconvenience.</h3>
</div>
</body>
</html>
"};
return (deliver);
}
The error log is:
11 SessionOpen c 71.77.0.73 49448 :80
11 ReqStart c 71.77.0.73 49448 570029255
11 RxRequest c GET
11 RxURL c /QA/exams/standards_and_guidelines.pdf
11 RxProtocol c HTTP/1.1
11 RxHeader c Host: 152.4.21.240
11 RxHeader c User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/534.52.7 (KHTML, like Gecko) Version/5.1.2 Safari/534.52.7
11 RxHeader c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
11 RxHeader c Accept-Language: en-us
11 RxHeader c Accept-Encoding: gzip, deflate
11 RxHeader c Cookie: __utma=67627729.194036926.1325818231.1325818231.1326406657.2; __utmb=67627729.5.10.1326406657; __utmc=67627729; __utmz=67627729.1325818231.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); PHPSESSID=4mvdfvuv1gd9h59m8nqg5gsgs1
11 RxHeader c Connection: keep-alive
11 VCL_call c recv lookup
11 VCL_call c hash
11 Hash c /QA/exams/standards_and_guidelines.pdf
11 Hash c 152.4.21.240
11 VCL_return c hash
11 VCL_call c miss fetch
11 Backend c 4 devservices devservices2
11 TTL c 570029255 RFC 0 -1 -1 1326407133 0 1326407118 375007920 0
11 VCL_call c fetch
11 TTL c 570029255 VCL 302 -1 -1 1326407131 -2
11 VCL_return c deliver
11 ObjProtocol c HTTP/1.1
11 ObjResponse c OK
11 ObjHeader c Date: Thu, 12 Jan 2012 22:25:18 GMT
11 ObjHeader c Server: Apache/2.2.15 (Red Hat)
11 ObjHeader c X-Powered-By: PHP/5.3.3
11 ObjHeader c Set-Cookie: PHPSESSID=af0ivec0rh864s6rifiirs0gu4; path=/
11 ObjHeader c Expires: Thu, 19 Nov 1981 08:52:00 GMT
11 ObjHeader c Cache-Control: public, must-revalidate
11 ObjHeader c Pragma: hack
11 ObjHeader c X-Pad: avoid browser bug
11 ObjHeader c Content-Encoding: gzip
11 ObjHeader c Vary: Accept-Encoding
11 ObjHeader c Content-Type: application/pdf
11 FetchError c straight read_error: 0 0 (Remote closed connection)
11 Gzip c u F - 90795 96895 80 726280 726290
11 VCL_call c error deliver
11 VCL_call c deliver deliver
11 TxProtocol c HTTP/1.1
11 TxStatus c 503
11 TxResponse c Service Unavailable
11 TxHeader c Server: Varnish
11 TxHeader c Content-Type: text/html; charset=utf-8
11 TxHeader c Content-Length: 690
11 TxHeader c Accept-Ranges: bytes
11 TxHeader c Date: Thu, 12 Jan 2012 22:25:33 GMT
11 TxHeader c X-Varnish: 570029255
11 TxHeader c Age: 2
11 TxHeader c Via: 1.1 varnish
11 TxHeader c Connection: close
11 Length c 690
11 ReqEnd c 570029255 1326407130.792037964 1326407133.193643808 0.000042439 2.401573896 0.000031948
11 SessionClose c error
11 StatSess c 71.77.0.73 49448 2 1 1 0 0 0 240 690
Thanks in advance if anyone has any insight into this.
More information about the varnish-misc
mailing list