Varnish 3.0 slow backend fetches

[Christian Hartlage]

Hi guys,

we're troubleshooting with a problem with our varnish setup - it would be great if you could help us.

Our server setup is as follows:

NGINX (as ssl-terminator) > Varnish  > NGINX/php-fpm (multiple web-server instances) 

Cacheable URLs work fine, even with around 5 ESI per site,  we can deliver 1000 req/s in around 100ms.

When it comes to non-cacheable URLs our setup fails completely. We can only serve about 5 req/s through varnish. If we leave out varnish for these urls and ask the nginx backends directly, we’re able to serve about 25 req/s. Without varnish, nginx directs these 25 req/s to the php-fpm processes and they get some nice cpu load.
With Varnish the php-fpm processes have nearyl zero cpu load, because varnish gets only 5req/s to the nginx backend.

Testing these requests with curl showed that it takes nginx/php-fpm about 0.2sec for a single response. For us it seems like varnish can do only 1 fetch at once, so that the backend request are blocking. Is that possible?
During load testing the client_conn counter jumps to 200 almost instantly and then increases with about 5 units per second. the backend_conn counter does not have the initial jump to 200 but increases from zero with 5 units per second.
After the load test is completed the client_conn counter does not increase anymore (obviously), but the backend_conn counter will continue increasing slowly until it reaches the amount of client_conn. This also made us think that varnish processes the backend requests sequentially, even when the clients are not actually waiting for the content anymore.

Best Regards, 

Christian Hartlage