[Varnish] #1617: Varnish 4 weird memory consumption / calculation

[Varnish]

#1617: Varnish 4 weird memory consumption / calculation
----------------------+-------------------------
 Reporter:  whocares  |       Owner:
     Type:  defect    |      Status:  closed
 Priority:  normal    |   Milestone:
Component:  varnishd  |     Version:  4.0.2
 Severity:  normal    |  Resolution:  worksforme
 Keywords:            |
----------------------+-------------------------
Changes (by daghf):

 * status:  new => closed
 * resolution:   => worksforme


Comment:

 Hi

 When a backend transmits a chunked transfer encoded response, Varnish
 can't know up front how much storage to allocate for that object. It will
 allocate 'fetch_chunksize' sized chunks, one at a time as the object is
 being fetched.

 In the case that streaming is disabled, a trim operation is issued on the
 last storage chunk, which will effectively resize it to the correct size,
 thus avoiding the overhead of always consuming a full sized chunk.
 However, with streaming enabled (as is the default in Varnish 4.0), we
 currently can't reliably trim a storage chunk.

 There are plans and some work in current master to redesign the
 storage/stevedore api which will allow us to handle this differently, but
 currently it's mostly on the drawing board.

 For now the best bet is to tune fetch_chunksize. The default has since
 been decreased from 128k to 16k
 (c84d1f886671fd98317890b5a41bd60f0837206f), which should very much lessen
 the overhead some users are seeing.

 Alternatively, you can set beresp.do_stream = false in vcl_backend_fetch,
 to disable streaming. This will enable 3.0-style trimming of storage
 chunks.

 Closing this as a configuration issue.

-- 
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1617#comment:6>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator