v2.1.3 w/ http_range_support strips Content-Range?
James A. Robinson
jim.robinson at stanford.edu
Fri Sep 3 22:49:37 CEST 2010
Hi folks,
We've got a problem dealing with Range requests via Varnish. We're
using varnish-2.1.3 SVN 5049:5055, and have activated
http_range_support.
It looks as though the base logic is that Varnish will (a) strip Range
from the request back to the backend, and (b) will surpress
Content-Range from the response.
HTTPH("Content-Range", H_Content_Range, 2, 3, HTTPH_R_PASS | HTTPH_A_PASS | HTTPH_R_FETCH | HTTPH_A_INS, 0, 0) /* RFC2616 14.16 */
HTTPH("Range", H_Range, 1, 0, HTTPH_R_FETCH | HTTPH_A_INS, 0, 0) /* RFC2616 14.35 */
When http_range_support is on, Varnish will deal with the incoming Range
itself if it has the item in cache. Otherwise it appears to try and
simply let the backend deal with it. Unfortunately it looks to me as
though Varnish accidently strips out the Content-Range header in that
situation.
First, I want to show that our back end client handles the range
requests. On the first request we ask for bytes 0-10:
$ curl -i -H'Range: bytes=0-10' -H'Host: host.mydomain.org' -s http://apache/podcast/test.mp3
HTTP/1.1 206 Partial Content
Date: Fri, 03 Sep 2010 18:34:48 GMT
Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14 OpenSSL/0.9.7a
Content-Range: bytes 0-10/15041999
Accept-Ranges: bytes
Last-Modified: Thu, 01 Jul 2010 15:14:34 GMT
ETag: "1a8d50-e585cf-4c2cb0da"
Content-Length: 11
Set-Cookie: JSESSIONID=79642FE5D7FEABBCFEC11F37B819A1F5.tomcat.mydomain.org; Path=/
Content-Type: audio/mpeg;charset=UTF-8
On the second we request bytes 10-20:
$ curl -i -H'Range: bytes=10-20' -H'Host: host.mydomain.org' -s http://apache/podcast/test.mp3
HTTP/1.1 206 Partial Content
Date: Fri, 03 Sep 2010 18:34:56 GMT
Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14 OpenSSL/0.9.7a
Content-Range: bytes 10-20/15041999
Accept-Ranges: bytes
Last-Modified: Thu, 01 Jul 2010 15:14:34 GMT
ETag: "1a8d50-e585cf-4c2cb0da"
Content-Length: 11
Set-Cookie: JSESSIONID=F915B3C040B37844E8DAA451377C0F8A.backend.mydomain.org; Path=/
Content-Type: audio/mpeg;charset=UTF-8
You can see that the apache server returns a Content-Range in both
cases.
Next, we try fetching via the front end varnish. Our first request
succeeds, an an examination of varnishlog shows that varnish is
stripping the incoming Range header when it makes the request to the
backend, and is generating its own Content-Range header for the
outgoing response:
$ curl -i -H'Range: bytes=0-10' -s http://host.mydomain.org/podcast/test.mp3
HTTP/1.1 206 Partial Content
Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14 OpenSSL/0.9.7a
Date: Fri, 03 Sep 2010 18:18:10 GMT
Content-Type: audio/mpeg;charset=UTF-8
Connection: keep-alive
Last-Modified: Thu, 01 Jul 2010 15:14:34 GMT
ETag: "1a8d50-e585cf-4c2cb0da"
Set-Cookie: JSESSIONID=EFC4B1B3B2B2774BE2B8FD7D26D179C6.tomcat.mydomain.org; Path=/
Accept-Ranges: bytes
X-Varnish: 1884335308
Age: 1
Via: 1.1 varnish
X-Varnish-Hostname: varnish1.mydomain.org
X-Varnish-Cache: miss
Content-Range: bytes 0-10/15041999
Content-Length: 11
So far so good!
But at this point Varnish has added the resource to its hit-for-pass
list (it has determined the item is not cachable). That ought to mean
varnish will simply route the request back to the apache server and
return the response:
$ curl -i -H'Range: bytes=10-20' -s http://host.mydomain.org/podcast/test.mp3
HTTP/1.1 206 Partial Content
Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14 OpenSSL/0.9.7a
Date: Fri, 03 Sep 2010 18:18:11 GMT
Content-Type: audio/mpeg;charset=UTF-8
Connection: keep-alive
Last-Modified: Thu, 01 Jul 2010 15:14:34 GMT
ETag: "1a8d50-e585cf-4c2cb0da"
Set-Cookie: JSESSIONID=E3144DE996CFA3B0BA4F2DE41A197510.tomcat.mydomain.org; Path=/
Content-Length: 11
Accept-Ranges: bytes
X-Varnish: 1884335658
Age: 0
Via: 1.1 varnish
X-Varnish-Hostname: varnish1.mydomain.org
X-Varnish-Cache: miss
Notice the Content-Range header has been stripped out for some reason when
varnish transmits the response. This is not, as far as I can determine,
to spec, since
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.2.7
indicates that a 206 response must include
Either a Content-Range header field (section 14.16) indicating the
range included with this response, or a multipart/byteranges
Content-Type including Content-Range fields for each part. If a
Content-Length header field is present in the response, its value
MUST match the actual number of OCTETs transmitted in the
message-body.
Looking at varnishlog, the TxHeader made to the backend did contain
the Range header, and Apache returned the requested portion and
included a Content-Range response, but that header was never sent
on to the client.
If I clear the resource from Varnish, so that the resource is no longer
in the hit-for-pass list, then re-request the 2nd range:
$ curl http://host.mydomain.org/varnish/purge/podcast/test.mp3
... success page returned indicating the item was added to the ban list ...
$ curl -i -H'Range: bytes=10-20' -s http://host.mydomain.org/podcast/test.mp3
HTTP/1.1 206 Partial Content
Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14 OpenSSL/0.9.7a
Date: Fri, 03 Sep 2010 18:32:54 GMT
Content-Type: audio/mpeg;charset=UTF-8
Connection: keep-alive
Last-Modified: Thu, 01 Jul 2010 15:14:34 GMT
ETag: "1a8d50-e585cf-4c2cb0da"
Set-Cookie: JSESSIONID=D496D1EEC4BB7E050D1776ED8C7209F7.tomcat.mydomain.org; Path=/
Accept-Ranges: bytes
X-Varnish: 1884521183
Age: 1
Via: 1.1 varnish
X-Varnish-Hostname: varnish1.mydomain.org
X-Varnish-Cache: miss
Content-Range: bytes 10-20/15041999
Content-Length: 11
The new Content-Range comes through. A 3rd request will, once again, return the
result stripped of its Content-Range until I purge it.
Here's what I see in varnishlog. On the first request the incoming Header specifies
the range request:
...
12 RxHeader c Range: bytes=10-20
12 VCL_call c recv
12 VCL_acl c MATCH highwire_acl 127.0.0.1
12 VCL_return c lookup
12 VCL_call c hash
12 VCL_return c hash
12 VCL_call c miss
12 VCL_return c fetch
At this point the TxHeader list indicates *no* Range request passed
on, so apache returns the entire request:
15 RxProtocol b HTTP/1.1
15 RxStatus b 200
15 RxResponse b OK
15 RxHeader b Date: Fri, 03 Sep 2010 20:08:02 GMT
15 RxHeader b Server: Apache/2.2.14 (Unix) DAV/2 mod_jk/1.2.28 mod_ssl/2.2.14 OpenSSL/0.9.7a
15 RxHeader b Accept-Ranges: bytes
15 RxHeader b Last-Modified: Thu, 01 Jul 2010 15:14:34 GMT
15 RxHeader b ETag: "1a8d50-e585cf-4c2cb0da"
15 RxHeader b Content-Length: 15041999
15 RxHeader b Set-Cookie: JSESSIONID=35F2019D47A900C13424120CE340EBED.tomcat.mydomain.org; Path=/
15 RxHeader b Content-Type: audio/mpeg;charset=UTF-8
Varnish then appears to deal with the Range request itself:
12 TTL c 1791943837 RFC 900 1283544482 0 0 0 0
12 VCL_call c fetch
12 VCL_return c pass
...
12 TxHeader c Content-Range: bytes 10-20/15041999
12 TxHeader c Content-Length: 11
...
15 Length b 15041999
...
12 VCL_call c deliver
12 VCL_acl c MATCH highwire_acl 127.0.0.1
12 VCL_return c deliver
12 TxProtocol c HTTP/1.1
12 TxStatus c 206
12 TxResponse c Partial Content
...
12 TxHeader c Content-Type: audio/mpeg;charset=UTF-8
12 TxHeader c Accept-Ranges: bytes
12 TxHeader c Date: Fri, 03 Sep 2010 20:08:03 GMT
12 TxHeader c X-Varnish: 1791943837
12 TxHeader c Age: 1
12 TxHeader c Via: 1.1 varnish
12 TxHeader c Connection: close
12 TxHeader c X-Varnish-Hostname: varnish-dev.mydomain.org
12 TxHeader c X-Varnish-Cache: miss
12 Length c 11
12 ReqEnd c 1791943837 1283544482.199923038 1283544483.532929897 0.000079155 1.332911968 0.000094891
12 SessionClose c Connection: close
12 StatSess c 127.0.0.1 37588 1 1 1 0 0 1 1239 11
Now on the 2nd request we see something different occuring. In this
situation Varnish is transmitting the Range request to the backend,
the backend is responding with status 206 and is providing a
Content-Range header in the response. Varnish is allowing most of
that response to come back, but is stripping out Content-Range:
12 RxHeader c Range: bytes=10-20
12 VCL_call c recv
12 VCL_acl c MATCH highwire_acl 127.0.0.1
12 VCL_return c lookup
12 VCL_call c hash
12 VCL_return c hash
12 HitPass c 1791943837
12 VCL_call c pass
12 VCL_return c pass
15 TxHeader b Range: bytes=10-20
15 RxProtocol b HTTP/1.1
15 RxStatus b 206
15 RxResponse b Partial Content
...
15 RxHeader b Content-Range: bytes 10-20/15041999
15 RxHeader b Accept-Ranges: bytes
15 RxHeader b Last-Modified: Thu, 01 Jul 2010 15:14:34 GMT
15 RxHeader b ETag: "1a8d50-e585cf-4c2cb0da"
15 RxHeader b Content-Length: 11
...
12 TTL c 1791943838 RFC 900 1283544541 0 0 0 0
12 VCL_call c fetch
12 VCL_return c pass
12 ObjProtocol c HTTP/1.1
12 ObjStatus c 206
12 ObjResponse c Partial Content
...
12 ObjHeader c Content-Type: audio/mpeg;charset=UTF-8
...
15 Length b 11
12 VCL_call c deliver
12 VCL_acl c MATCH highwire_acl 127.0.0.1
12 VCL_return c deliver
12 TxProtocol c HTTP/1.1
12 TxStatus c 206
12 TxResponse c Partial Content
...
12 TxHeader c Content-Type: audio/mpeg;charset=UTF-8
...
12 TxHeader c Content-Length: 11
12 TxHeader c Accept-Ranges: bytes
12 TxHeader c Date: Fri, 03 Sep 2010 20:09:01 GMT
12 TxHeader c X-Varnish: 1791943838
12 TxHeader c Age: 0
12 TxHeader c Via: 1.1 varnish
12 TxHeader c Connection: close
12 TxHeader c X-Varnish-Hostname: varnish-dev.mydomain.org
12 TxHeader c X-Varnish-Cache: miss
12 Length c 11
12 ReqEnd c 1791943838 1283544541.880913019 1283544541.910290003 0.000096083 0.029315948 0.000061035
12 SessionClose c Connection: close
12 StatSess c 127.0.0.1 51087 0 1 1 0 1 1 1202 11
Jim Robinson
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
James A. Robinson jim.robinson at stanford.edu
Stanford University HighWire Press http://highwire.stanford.edu/
+1 650 7237294 (Work) +1 650 7259335 (Fax)
More information about the varnish-misc
mailing list