Intermittent latency issues on Solaris10

[Igor Minar]

Hi Folks!

I'm trying set up a Varnish instance on Solaris 10 and I'm facing some  
weird issue there. I was wondering if anyone could give me some  
insight into what's going on.

After applying a few patches (I submited all of them to Trac), I was  
able compile Varnish 2.0.2 and 2.0.3 with Sun Studio compiler. Varnish  
starts and is able to handle requests, but from time to time, I see  
multi-second latency when executing requests. As if varnish "hung" for  
a few seconds. This latency occurs for cache hits and is not caused by  
the backed app.

Interesting thing is that I don't see this issue on OpenSolaris  
2008.11 (varnish was compiled on both OSes with the latest Sun Studio  
compiler).

Here is to visualize what I see:
correct behavior on opensolaris: http://skitch.com/me3x/bf7j2/varnish-on-opensolaris
incorrect behavior on solaris10: http://skitch.com/me3x/bf7kr/varnish-on-solaris10

As you can see, on Solaris 10 there are at 3 requests that took more  
than 4 seconds to execute. I checked varnish logs and all of them were  
cache hits.

In the screenshots above, there was no other traffic hitting the  
server except for my single browser.

The problem is quite easily reproducible.

thanks,
Igor