Range handling in Varnish

[Martin Blix Grydeland]

I've been dealing with ranges issues in Varnish lately, mostly trying to
fix issues with the 4.0 behaviour. As part of that I've also looked
carefully at the new implementation that went into Varnish 4.1. As this is
a complex subject, I ended up trying to document how it is supposed to
work. The following is a draft for a Varnish range handling page.Please
read and give feedback if my descriptions are accurate, and if this is the
functionality as we want to support it.


BEGIN DRAFT DOCS


Range handling
==============

Range support allows client requests to retrieve only parts of an object by
selecting the bytes needed in the Range request header. On a successful
range request, the HTTP response code will be changed from 200 to 206, and
the Content-Range response header describes the range delivered. Note that
Varnish only supports specifying a single range in each request.

The following applies only to cache hits. On passes, Varnish does not
intervene with the range handling, but the handling is passed on to the
backend server.

Advertisement of range capability
---------------------------------

Varnish will advertise range support for any cached object with a 200
status code, except if that object includes ESI instructions or when it's
used as part of an ESI subdelivery (or range support is turned off by the
http_range_support runtime parameter). Advertisement is done through a
"Accept-Ranges: bytes" response header.

Final object size knowledge
---------------------------

The range handling in Varnish behaves differently based on whether Varnish
is aware of the final size of the object being delivered. Varnish knows the
final object size when either the backend provided a Content-Length header
or the fetch operation has completed (non-streaming deliveries).

If the delivery is for a client that does not support gzip, and the object
is cached compressed, Varnish will uncompress the object as part of the
client delivery. In this case Varnish will only know the final uncompressed
object size after the fetch operation has completed, regardless if the
backend provided a Content-Length header.

Parsing of request range headers
--------------------------------

A malformed client Range header results in a 416 "Request Range Not
Satisfiable" response. The Varnish log will contain an error message
describing the error.

Range delivery
-----------
Varnish only handles ranges when it has knowledge of final object size.
Both closed (have both low and high byte) and open-ended (missing either
low or high) are supported. The range is truncated if it falls outside the
object space. Varnish responds with status 206, an appropriate
Content-Range header and the corresponding range of bytes in the body. It
is also always delivered with a Content-Length header matching the
Content-Range.

If Varnish does not have the final object size knowledge, it will not
honour range requests. They are instead delivered as normal 200 responses
with the complete object, as if the Range header was not present on the
request in the first place.


END DRAFT DOCS

Some info on what it used to be:

In our previous version (4.0), things behaved differently. Varnish would
attempt to do range handling also when we did not know the final object
size, using a snapshot of the current size instead, on the premise that the
client could know something that Varnish doesn't.

Open-ended ranges failed with this approach as the transient size was used
to find the end of the object leading to completely wrong responses for
e.g. "give me the last 50 bytes".

Also closed ranges failed at least for some clients. The Content-Range
response header has a total byte field, and we would report our transient
object size in this field. Some clients would use this information and get
a wrong impression of the final object size.

RFC7233 states that it is legal to report an asterisk instead of the total
bytes if the total bytes in unknown, and seems to offer a nice way to
indicate that we are uncertain. Master used to behave this way until just
before the 4.1 release. It was changed as part of #1777 as search crawler
bots seemed to always ask a set range, and because we would then also
provide a Content-Length the connection failed hard if the object happened
to be smaller than the given range.

I wonder if the resolution of #1777 is wrong, and that it would be viable
to still do opportunistic ranges successfully with the uncertainty of the
asterisk total length, but with chunked delivery applied instead of forcing
a Content-Length.

The resolution of #1777 also left some amount of dead code in
cache_range.c, along with a kind of convoluted set of if-tests. The
attached patch removes this if we are certain that we don't want that
behaviour.

Martin

-- 
<http://varnish-software.com>*Martin Blix Grydeland*
Senior Developer | Varnish Software AS
Mobile: +47 992 74 756
We Make Websites Fly!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20151006/a49a5a31/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-Remove-dead-code.patch
Type: text/x-patch
Size: 2060 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20151006/a49a5a31/attachment.bin>