Request for code review of streaming patches and merging into master

[Martin Blix Grydeland]

The streaming implementation has reached a state now where I believe it
should be considered for merging into master. I will post all of the
patches to varnish-dev.

There are some outstanding points left, outlined below. And also there will
be some cache_center.c restructuring coming up, but it should be possible
to do that without needing major rewriting of the streaming stuff, and
having the code in master when starting on that should be an advantage.

Any comments and inputs much appreciated.

Outstanding streaming work items:

* Streaming and client 304

  If we can deliver a 304, we should for the pass-case just drop the
  backend connection (alternately do the bg fetch and throw away the
  result?). For the non-pass-case we should perform the bg fetch while
  still delivering a 304 to the client. The decision making will have
  to be influenced by wether we manage to grab a thread or not, and
  what to do when we don't have a 2nd thread has to be decided.

  This is best implemented after the vcl changes and cache_center.c
  changes coming from that.

* Streaming and range

  Currently ranged delivery isn't performed when doing streaming, full
  object is always delivered when do_stream is true. This should be
  fixed. Some patches from the 3.0 streaming by Thomas Souvignet has
  been mailed to varnish-dev. Should be possible to use this as input
  when adding it to the new streaming implementation.

* Break backend connection when streaming, pass-mode and client goes away

  If we loose the client while streaming in pass-mode we should
  probably just drop the backend connection as well.

* Adaptive streaming

  I don't think it makes sense to do streaming for very small
  objects. For things less than a kilobyte, by the time we have the
  2nd thread ready the full object is probably already on the wire and
  it would be faster to just read this in full before delivering the
  object.

  This is easy if there is a content-length header, we could just test
  this against a configurable minimum size. This decision policy
  probably makes the much sense in VCL?

  For chunked/eof transfer from backend, we can't know the size at
  decision making time. We could perhaps implement that streaming is
  first started after having received a minimum number of bytes?

-- 
Martin Blix Grydeland
Varnish Software AS
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20120122/aef1b9be/attachment.html>