beresp.do_stream and memory usage

[Daniel]

Hi,

I have a question about beresp.do_stream and memory usage in Varnish 4.0.2.

Example:

The response (JSON) from backend is 73 kB in size uncompressed.

Looking at the SMA.s0.g_bytes after one request I can see that Varnish 
allocated 132 kB memory for this request. When compression is enabled 
Varnish still allocates 132 kB of memory. I have also tested requests 
with different sizes and as long as the request is smaller than 132 kB 
Varnish seems to always allocate 132 kB of memory, even if the request 
is only 2 kB in size.

If I then add beresp.do_stream = false in vcl_backend_reponse I can see 
that SMA.s0.g_bytes after one request is 74 kB and 22 kB with 
compression enabled. The backend is responding with "Transfer-Encoding: 
chunked", if that matters. I would like some help to understand why I am 
seeing this behaviour. Is it expected behaviour?


Let me know if you need further information.

Thanks!

Regards,
Daniel