HTCP CLR operation for purging clustered reverse proxy cache

[Ben Truitt]

Last comment on this, and then I'll leave it to you all to discuss as you see fit:

In investigating Varnish more today, I was very surprised to learn that it does not respect section 13.10 of the HTTP/1.1 RFC (http://www.w3.org/Protocols/rfc2616/rfc2616-sec13.html#sec13), which states "Some HTTP methods MUST cause a cache to invalidate an entity. These methods are: PUT, DELETE, POST."

I found this thread (http://www.gossamer-threads.com/lists/varnish/misc/19516), where Per Buer comments: "We won't do this because it would totally break if you have more then one Varnish servers. It's the backend job to notify the caches when the content actually changes."

I disagree. According to the spec, this is the job of the cache. The spec doesn't address a clustered cache, but I would expect a clustered cache to perform clustered invalidation. HTCP CLR is designed for this purpose. Therefore, to me it makes total sense to incorporate this into Varnish proper.

Thanks again for your time and attention!

-Benjamin

On Jul 6, 2011, at 11:37 AM,  wrote:

Good question!

In my opinion, invalidating cache entries is an essential operation of a cache. Varnish already supports HTTP PURGE without a separate process, so invalidating entries is supported in Varnish directly.  What is not supported directly is invalidating entries across a cluster. This requires listening on a separate port, but that doesn't seem like a very strong reason for breaking the functionality into a separate program.

It also seems as though keeping the code for HTCP alongside the rest of Varnish will promote maintainability.

Adding this to Varnish proper should lead to a good start for supporting the other operations in HTCP, specifically the ability for a server to query its peers to determine whether they have a cached representation available.

Furthermore, Squid supports HTCP out of the box, so feature parity in this area will make it easier to gain market share.

Thoughts?

-Benjamin

On Jul 6, 2011, at 10:44 AM, Per Buer wrote:

Hi,

Why should this be in Varnish directly? Whats wrong with having it in a separate program?

Per.

On Wed, Jul 6, 2011 at 4:38 PM, Ben Truitt <btruitt at rackspace.com<mailto:btruitt at rackspace.com>> wrote:
I was able to find the location of the source code for the HTCP plugin I referred to in the below email (thanks, Artur Bergman):

https://svn.wikia-code.com/utils/varnishhtcpd/

I'm still curious to know whether there are any plans to incorporate this or similar functionality for purging a clustered cache into Varnish going forward.

Thanks,
Benjamin

On Jul 5, 2011, at 4:56 PM,  wrote:

> Hi -
> Thanks for the great work on Varnish!
>
> My team at Rackspace is considering adopting Varnish for some RESTful services we're creating. We're planning to have a cluster of reverse proxy cache servers in front of our origin servers. One of the issues that is important to us is the ability to keep the cache consistent with the origin server's data store as much as possible.
>
> I realize that Varnish supports HTTP PURGE operation, but we'd prefer to use something like HTCP's CLR operation for a multicast purge.  I came across a fairly old varnish-dev mailing list item related to this:
>
> http://www.mail-archive.com/varnish-dev@projects.linpro.no/msg00315.html
>
> That post indicates that HTCP CLR would be incorporated into Varnish, but I don't see that it has been.
>
> I'd love to know:
> 1) Are there plans to include HTCP CLR in Varnish any time soon?
> 2) Is there a plugin (such as the one Artur Bergman created) available that I can use today?
>
> Thanks in advance for your time!
>
> Best,
> Benjamin Truitt

This email may include confidential information. If you received it in error, please delete it.


_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org<mailto:varnish-misc at varnish-cache.org>
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc



--
[http://www.varnish-software.com/sites/default/files/varnishsoft_white_190x47.png]
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
Varnish makes websites fly!
Whitepapers<http://www.varnish-software.com/whitepapers> | Video<http://www.youtube.com/watch?v=x7t2Sp174eI> | Twitter<https://twitter.com/varnishsoftware>




This email may include confidential information. If you received it in error, please delete it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110706/c89f0557/attachment-0003.html>