beresp.do_stream and memory usage

[Daniel]

Hi again,

On 2014-10-15 18:32, Daniel wrote:
> I have a question about beresp.do_stream and memory usage in Varnish 4.0.2.
>
> Example:
>
> The response (JSON) from backend is 73 kB in size uncompressed.
>
> Looking at the SMA.s0.g_bytes after one request I can see that Varnish
> allocated 132 kB memory for this request. When compression is enabled
> Varnish still allocates 132 kB of memory. I have also tested requests
> with different sizes and as long as the request is smaller than 132 kB
> Varnish seems to always allocate 132 kB of memory, even if the request
> is only 2 kB in size.
>
> If I then add beresp.do_stream = false in vcl_backend_reponse I can see
> that SMA.s0.g_bytes after one request is 74 kB and 22 kB with
> compression enabled. The backend is responding with "Transfer-Encoding:
> chunked", if that matters. I would like some help to understand why I am
> seeing this behaviour. Is it expected behaviour?
>
>
> Let me know if you need further information.

After some more testing I decided to add beresp.do_stream = false to our 
production Varnish.

Our Varnish in production is configured with 9G malloc, 3m TTL and 60m 
grace. With this configuration around 70 000 objects was stored in the 
cache, all memory was used (according to SMA.s0.g_bytes) and the 
lru_nuked counter started to increase. After adding beresp.do_stream = 
false the number of objects in cache has slightly increased, 
SMA.s0.g_bytes has dropped to 330 MB and the hitrate is the same as before.

Any ideas why I see such a dramatic decrease in memory usage when adding 
beresp.do_stream = false?

Thanks!


Regards,
Daniel