Hit ratio dropped significantly after recent upgrades

[Justin Lloyd]

I really am looking at what's happening as well. I have been looking at both varnishlog and varnishtop and I see a lot of thumbnail image requests being sent to the backend when there is still plenty of room for them in the cache, so even though there are a lot of thumbnail images, I shouldn't see so many backend requests for them. As I previously mentioned, I give Varnish 8 GB and it used to stay full (based on RSS usage and looking at nukes vs. expires) but now it hovers around only about 2 GB used. A related statistics is that there used to be 600-700k objects in Varnish (based on our graphs of MAIN.n_object via Collectd's varnish-default-struct.objects-object metric) but now there are only roughly 40-70k objects in Varnish at any given time. So it's definitely caching a lot fewer things than it was before the upgrade, and most of the requested URLs for requests that have cookies are for a lot of images and thumbnails. Images shouldn't be cached due to size and overall volume but thumbnails should, which is why I strip cookies from the thumbnails. These varnishtop commands break out /images and /images/thumb client requests, showing IMHO too many regular images being cached and nowhere near enough thumbnails:

# varnishtop -c -i VCL_call -q 'ReqURL ~ "/images/" and not ReqURL ~ "/images/thumb"'

   349.47 VCL_call       HASH
   349.47 VCL_call       RECV
   349.47 VCL_call       DELIVER
   207.22 VCL_call       HIT
   116.40 VCL_call       MISS
   116.30 VCL_call       PASS

# varnishtop -c -i VCL_call -q 'ReqURL ~ "/images/thumb"'                            

  1859.60 VCL_call       HASH
  1859.60 VCL_call       RECV
  1859.60 VCL_call       DELIVER
  1424.83 VCL_call       MISS
   422.84 VCL_call       HIT
   218.82 VCL_call       PASS

I'm still poking around trying to correlate caching of other types of URLs based on whether or not the requests have cookies, if Cache-Control gets returned, etc. but I just wanted to reply with this info. I do appreciate the responses I'm getting! :)


-----Original Message-----
From: Dridi Boukelmoune [mailto:dridi at varni.sh] 
Sent: Friday, December 9, 2016 10:11 AM
To: Justin Lloyd <justinl at arena.net>
Cc: Dag Haavi Finstad <daghf at varnish-software.com>; varnish-misc at varnish-cache.org
Subject: Re: Hit ratio dropped significantly after recent upgrades

> To reiterate on a point in another of my responses in this thread, I think it may be something about MediaWiki thumbnail images not being cached properly despite our current VCL in that regard not having changed from how it worked prior to the upgrade during which time we were seeing a very high (86%-ish) hit ratio from the same formula.

To reiterate on a point I made on a couple occasions, it's time to give varnishlog a spin. Too much focus on VCL, and not enough on what's happening.

Dridi