Best approach for expiring objects

[Tyler Love]

I know this is a commonly misunderstood aspect of Varnish, but I think I
have the hang of it.

I am using varnish on a very high traffic website and have yet to find a
satisfactory approach to expiring objects from the cache.

My goal is to increase ttls (1 day, maybe even a week), to maximize hit
rates, and expire objects when content is changed. This has proven to be
more difficult than I anticipated.



The first approach was to use the "purge" function that varnish gives you.
The caveat that defated this approach was that purge is basically a memory
leak, that also slows down requests.

Please correct me if I have a misunderstanding of purge but, when you call
purge you add either a url or a regex to a list stored within varnish, and
then it checks all incoming requests against this url and makes sure to not
serve them from the cache.

This make sense with an understanding of how objects are stored in varnish,
but the name "purge" is less than ideal and even misleading (I read this is
being changed in trunk?).

Second, banned items are not served in grace mode, which is behavior I do
not want.



The alternative is to set the ttl on an object to 0 (zero). The first caveat
was being fully aware of every single resource/url that needs to be called
for a given purge/expiration. One big problem area is caching on paginated
urls.

Second was the sheer volume of purges I need to send. At the scale I am at,
I have to purge approximately 400-800 urls per second (I have multiple
instances of varnish running by the way).

Lastly, purging the objects with different compression (Accept-Encoding:
gzip, deflate, etc) in the cache. I now have to multiply the amount of urls
for every possible encoding I may or may not use.



So there you have it, removing objects from varnish is tedious, and full of
pitfalls. I understand the architecture of varnish just well enough to know
why it works this way, but I still wish it was better.

If I am approaching this problem the wrong way, I am more than happy to hear
some thoughts.

Tyler
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110118/e44eb536/attachment-0003.html>