Hit ratio dropped significantly after recent upgrades

[Justin Lloyd]

I have been doing a lot of digging with varnishtop and varnishlog, and our VCL really didn’t change from this upgrade except as needed to migrate from Varnish 3 to 4. As I mentioned, our web app is MediaWiki so we don't control its caching requirements and recommendations, so what I'm trying to understand is whether the drop in the hit rate is due to some change(s) in MediaWiki's cookie and/or cache handling (e.g. via Cache-Control and Set-Cookie headers) or if something in Varnish changed that affects how it determines  things. For example, a while back I had been using the Varnish hit and miss metrics in Collectd to calculate the ratio but apparently how those values are calculated with respect to purges changed so the hit ratio dropped, causing me to change the ratio calculation to use incoming connections and backend requests instead.

That said, based on my varnishlog and varnishtop testing, I have a strong feeling that the biggest part of the problem is thumbnail images. If you look again at my VCL code (https://gist.github.com/Calygos/105957a997ea3bde6b8257a1f34bbd20), you can see I strip cookies from thumbnails so they should get cached, but I seem to get a lot more misses than hits when watching for thumbnail URL requests through varnishtop. I give 8 GB to Varnish and its process is typically only around 1 to 2 GB when previous it would be at 8 GB with frequent nukes and the occasional spike of expires that would temporarily eliminate nukes while memory filled up again. For what it's worth, I added the thumbnail stripping a couple of years ago due to a performance issue and it helped tremendously, so I don't know why it would become problematic with these latest upgrades.

Justin

-----Original Message-----
From: Dridi Boukelmoune [mailto:dridi at varni.sh] 
Sent: Thursday, December 8, 2016 6:49 AM
To: Jason Price <japrice at gmail.com>
Cc: Justin Lloyd <justinl at arena.net>; varnish-misc at varnish-cache.org
Subject: Re: Hit ratio dropped significantly after recent upgrades

On Thu, Dec 8, 2016 at 2:35 PM, Jason Price <japrice at gmail.com> wrote:
> I think we're going to need something a little more specific to go on.  
> That is a mile of changes all at once.

Yes: varnishlog, coffee, and a lot of patience.

> Finding a single request that should be cached, but isn't and 
> producing the varnish log for that request will probably help illuminate what's going on.

There's currently no way to query the transaction log of a specific request:
https://github.com/varnishcache/varnish-cache/issues/2154

I'm just saying...

Dridi