Can't get "streaming" or pipe to work, Varnish still waits for the full response

[Dridi Boukelmoune]

On Sat, Sep 12, 2020 at 10:08 PM Batanun B <batanun at hotmail.com> wrote:
>
> Hi,
>
> We have some old (legacy) internal admin pages that do some classic old school processing while the page is loading, and outputting the current status as it is working. When requesting these pages directly (through Tomcat), I can see the results in the browser at the same time as the results are written on the other end. But when I go through Varnish, no matter what I try, I only see a blank page that is loading/waiting, and then when the backend is done writing, then I get the entire result in one go.
>
> How can I configure Varnish to bring any content to the client the moment it gets it from the backend, and not wait until the entire response is done?
>
> In vcl_backend_response I do this:
>   set beresp.do_stream = true;
>   set beresp.uncacheable = true;
>   return (deliver);

Streaming is on by default, you don't need to do anything.

> I have also tried returning (pipe) in vcl_recv (with and without do_stream and uncacheable). And gzip is turn off. But nothing helps. What can I do more? And how can I debug this? Varnishlog shows nothing that is telling me that it is buffering, or waiting for the response, or anything like that.

It is indeed hard to get that information just from Varnish, you could
try to capture TCP packets to check how long it takes for backend
traffic to be forwarded to clients. It's not obvious why a response
would be delayed, but it could happen to be related to the
fetch_chunksize parameter.

However I have never come across a setup where we needed to
tune that knob...

Dridi