[Varnish] #1506: Make better use of Content-Length information: Avoid chunked responses, more control over Range handling

[Varnish]

#1506: Make better use of Content-Length information: Avoid chunked responses,
more control over Range handling
--------------------------+----------------------------------
 Reporter:  DonMacAskill  |       Owner:  phk
     Type:  defect        |      Status:  new
 Priority:  normal        |   Milestone:  Varnish 4.0 release
Component:  varnishd      |     Version:  4.0.0
 Severity:  critical      |  Resolution:
 Keywords:                |
--------------------------+----------------------------------

Comment (by r):

 The b2b6e3 change fixes one problem, but it seems there is another
 problem.
 We observe Varnish occasionally trying to (unsuccessfully) send a chunked
 response even if beresp.do_stream = false.  When this happens, Varnish
 abruptly closes the connection with no payload being sent.

 From a rather quick code inspection, it seems the bug is a race condition
 between threads.  Here is a more detailed description of what is going on:

 CNT_Request() in cache_req_fsm.c processes the request by going through
 the
 different states of the FSM.  In a case of cache miss, cnt_miss() is
 entered
 which calls VBF_Fetch().  This routine starts fetching the object
 concurrently
 in a separate thread:

 https://www.varnish-
 cache.org/trac/browser/bin/varnishd/cache/cache_fetch.c?rev=bfe7cd198d07ab2197f7d178b08df0caa5e6b7a3#L917

 The caller waits only for the partial fetch i.e. it waits for the
 BOS_STREAM
 state which is set and signalled on vbf_stp_fetch().  Then in continues,
 while
 the fetch is still in progress.  The FSM  eventually reaches cnt_deliver()
 routine which acquires a reference to req->obj->objcore->busyobj and calls
 V1D_Deliver():

 https://www.varnish-
 cache.org/trac/browser/bin/varnishd/cache/cache_req_fsm.c?rev=bfe7cd198d07ab2197f7d178b08df0caa5e6b7a3#L169

 The busyobj is cleared when vbf_fetch_thread() calls HSH_Complete(),
 therefore
 it is volatile when the reference is being acquired; i.e. the fetch thread
 may
 or may not have completed its fetch.  When it is not, the following check
 in
 V1D_Deliver() fails:

 https://www.varnish-
 cache.org/trac/browser/bin/varnishd/cache/cache_http1_deliver.c?rev=bfe7cd198d07ab2197f7d178b08df0caa5e6b7a3#L251

 This results in RES_CHUNKED being set and eventual failure as described in
 the beginning.  Note: the code inspection was done on 4.0.2 source code.

-- 
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1506#comment:8>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator