Stale-While-Revalidate or Similar

Tue Mar 20 11:21:12 CET 2012

Hey graham, thanks for your reply,

> Yeah, you want grace mode:
> https://www.varnish-cache.org/docs/trunk/tutorial/handling_misbehaving_servers.html#grace-mode

As far as I tried, grace mode does serve stale content to 9999 / 10000 users, but one still has to refresh the cache and wait for the backend servers. 
Doing this async, does not seem to be supported by grace-mode, as far as I understood. Correct me, if I'm wrong.

Greetings

Michael

Von: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] Im Auftrag von Graham Lyons
Gesendet: Dienstag, 20. März 2012 11:17
An: varnish-misc at varnish-cache.org
Betreff: Re: Stale-While-Revalidate or Similar

Yeah, you want grace mode:
https://www.varnish-cache.org/docs/trunk/tutorial/handling_misbehaving_servers.html#grace-mode

As it says on that page, "it serves stale content to avoid request pile-up."

On 20 Mar 2012, at 10:06, varnish-misc-request at varnish-cache.org wrote:

Send varnish-misc mailing list submissions to
	varnish-misc at varnish-cache.org

To subscribe or unsubscribe via the World Wide Web, visit
	https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
or, via email, send a message with subject or body 'help' to
	varnish-misc-request at varnish-cache.org

You can reach the person managing the list at
	varnish-misc-owner at varnish-cache.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of varnish-misc digest..."

Today's Topics:

  1. Re: Stale-While-Revalidate or Similar (Graham Lyons)
  2. Re: Stale-While-Revalidate or Similar (Michael Borejdo)

----------------------------------------------------------------------

Message: 1
Date: Tue, 20 Mar 2012 09:50:54 +0000
From: Graham Lyons <graham.lyons at bbc.co.uk>
To: varnish-misc at varnish-cache.org
Subject: Re: Stale-While-Revalidate or Similar
Message-ID: <9303601D-DC59-45DA-BCCB-6B0D3621557E at bbc.co.uk>
Content-Type: text/plain; charset="us-ascii"

Hi Michael,

There is a similar feature to this available in Varnish:
https://www.varnish-cache.org/docs/trunk/tutorial/handling_misbehaving_servers.html

Grace and saint modes correspond roughly, respectively to stale-while-revalidate and stale-if-error proposed in RFC5861.

Graham.

On 20 Mar 2012, at 09:24, varnish-misc-request at varnish-cache.org wrote:

Send varnish-misc mailing list submissions to
	varnish-misc at varnish-cache.org

To subscribe or unsubscribe via the World Wide Web, visit
	https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
or, via email, send a message with subject or body 'help' to
	varnish-misc-request at varnish-cache.org

You can reach the person managing the list at
	varnish-misc-owner at varnish-cache.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of varnish-misc digest..."

Today's Topics:

 1. Re: Varnish client IP address not display (MassiveScale.net)
 2. VUG5 Paris BOF (Jonathan Matthews)
 3. Re: Communication between Varnish and the Backend (Hugues Alary)
 4. using the max-age request header (Jeroen Ooms)
 5. Re: Communication between Varnish and the Backend (Per Buer)
 6. Stale-While-Revalidate or Similar (Michael Borejdo)

----------------------------------------------------------------------

Message: 1
Date: Mon, 19 Mar 2012 12:39:36 +0100
From: "MassiveScale.net" <info at massivescale.net>
To: varnish-misc at varnish-cache.org
Subject: Re: Varnish client IP address not display
Message-ID: <4F671AF8.3060201 at massivescale.net>
Content-Type: text/plain; charset=ISO-8859-1

On 03/19/2012 09:02 AM, Stewart Robinson wrote:
Jewel,
What header are you checking on the back end?
You should be reading the x-forwarded-for header rather than some form
of remote_addr.

Alternatively, if your backend server is Apache, you could use mod_rpaf
http://stderr.net/apache/rpaf/

-- 
Andrzej Godziuk
http://MassiveScale.net/

------------------------------

Message: 2
Date: Mon, 19 Mar 2012 13:25:22 +0000
From: Jonathan Matthews <contact at jpluscplusm.com>
To: varnish-misc at varnish-cache.org
Subject: VUG5 Paris BOF
Message-ID:
	<CAKsTx7Dv1rf62dJ6XYMo0n+vcB02nP7fyxX9CcPaPw2KbHUD-Q at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

Hi all -

I'm heading along to VUG5 this week - really looking forward to
meeting a lot of you and seeing how you're using and abusing our
favourite tool :-)

When I signed up, I don't recall ticking the "BOF" checkbox - could
anyone let me know if it's too late to get in on that action?

Either way, are people (like me) who are attending *just* the user day
(Thursday) perhaps looking for more Varnish chat on the Friday or the
weekend? I'm there 'til Sunday, and would love to get the most out of
a high concentration of Varnish-literate folk!

All the best,
Jonathan
-- 
Jonathan Matthews
London, Oxford, UK
http://www.jpluscplusm.com/contact.html

------------------------------

Message: 3
Date: Mon, 19 Mar 2012 12:50:23 -0700
From: Hugues Alary <hugues.alary at gmail.com>
To: varnish-misc at varnish-cache.org
Subject: Re: Communication between Varnish and the Backend
Message-ID:
	<CAN-YAk8S=dCxUXR5GbyC3+s3oSgis0+doetcw0N8QUGPJ1Mteg at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Happy Monday everybody,

Nobody, got some good advices for me? :)

Thanks,
-Hugues

On Fri, Mar 16, 2012 at 3:42 PM, Hugues Alary <hugues.alary at gmail.com>wrote:

Hi everybody,

I'm configuring varnish to be used with my web application.

I need my backend to be able to give instructions to Varnish depending on
the page requested. For example, some of my pages contain ESI includes,
some don't.

I want my application to tell varnish wether or not do the esi
processing. This approach allows me to keep my VCL configuration file
abstracted from the application. I can then redistribute the VCL to other
users and let their application control the cache.

So far my solution is to instruct varnish via HTTP headers.

In my current example I set a header X-Application-DoEsi to 1 or 0. When
varnish receives the response from the backend, it looks for the value
of X-Application-DoEsi and does or not the ESI processing.

 - I was wondering if communicating with varnish via HTTP header is a
 good solution?
 - How bad can it be performance wise?
 - Should I be worried about security (interception/modification of the
 communication between Varnish and the backend), even if no credential will
 never be sent through HTTP headers? (Also, currently Varnish and the
 backend are on the same machine, but chances are that they will in the
 future not live on the same host).

In the future, I plan on instructing varnish not to cache certain pages
containing user defined query strings. I want the user to be able to
specify these un-cachable urls query strings directly in the application.
The application will then send the un-cachable query strings in a header
X-Application-QueryStringNoCache: "querystring1,querystring2,...".

Is that a bad idea?

Thank you for your help!

--
Hugues ALARY

-- 
Hugues ALARY
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20120319/a5ee34d6/attachment-0001.html>

------------------------------

Message: 4
Date: Mon, 19 Mar 2012 17:11:57 -0700
From: Jeroen Ooms <jeroen.ooms at stat.ucla.edu>
To: varnish-misc at varnish-cache.org
Subject: using the max-age request header
Message-ID:
	<CABFfbXtVc3vMVB-7_vEUZW6fXup5zte6rVhwL7avsQdqHX1ZHQ at mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

How do I configure my VLC to let Varnish respect the 'Cache-Control
max-age' request header? Someone on SO suggested the following:

if (req.http.Cache-Control ~ '\bmax-age=\b`) {
  if (std.duration(regsub(req.http.Cache-Control,
".*\bmax-age=(\d+)\b.*", "\1s"), 0s) < obj.age) {
      return (pass);
  }
}

However, what I don't like about this is that vcl_pass does not
actually update the cache with the new entry. How can I configure it
so that when max age < obj.age, a fresh copy will be fetched and
inserted into cache?

------------------------------

Message: 5
Date: Tue, 20 Mar 2012 07:25:05 +0100
From: Per Buer <perbu at varnish-software.com>
To: Hugues Alary <hugues.alary at gmail.com>
Cc: varnish-misc at varnish-cache.org
Subject: Re: Communication between Varnish and the Backend
Message-ID:
	<CAOXZevDAcWRxDFbUA_mSh4_Hh2S1ukF9PGbmN6X2UThC-Md+5Q at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

On Fri, Mar 16, 2012 at 11:42 PM, Hugues Alary <hugues.alary at gmail.com>wrote:
(..)

 - I was wondering if communicating with varnish via HTTP header is a
 good solution?

Yes.

 - How bad can it be performance wise?

Negligible.

 - Should I be worried about security (interception/modification of the
 communication between Varnish and the backend), even if no credential will
 never be sent through HTTP headers? (Also, currently Varnish and the
 backend are on the same machine, but chances are that they will in the
 future not live on the same host).

It's more or less impossible to turn Varnish into an open proxy so I
wouldn't worry about that.

In the future, I plan on instructing varnish not to cache certain pages
containing user defined query strings. I want the user to be able to
specify these un-cachable urls query strings directly in the application.
The application will then send the un-cachable query strings in a header
X-Application-QueryStringNoCache: "querystring1,querystring2,...".

Is that a bad idea?

No. If it works for you then go for it.

-- 
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20120320/59e9f0ef/attachment-0001.html>

------------------------------

Message: 6
Date: Tue, 20 Mar 2012 09:24:02 +0000
From: Michael Borejdo <mib at electronic-minds.de>
To: "varnish-misc at varnish-cache.org" <varnish-misc at varnish-cache.org>
Subject: Stale-While-Revalidate or Similar
Message-ID:
	<BE4EDDBE2C55604EB9F3EBA421CB227A23971439 at EM-MBX-1.electronic-minds.local>

Content-Type: text/plain; charset="us-ascii"

Hello List,

Is there support for something like "stale-while-revalidate" coming to Varnish?
I need to make the request to my backend-servers async and serve slightly stale content (defined by grace) and I do not want a user to wait for the backend at any time.
Or is there any other way with using vcl trickery to accomplish this?
(Unfortunately I cannot seem to find any info on this topic and the thread I found on the list has not been answered)  [1]

[1] https://www.varnish-cache.org/lists/pipermail/varnish-misc/2012-February/021644.html

Thanks

Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20120320/c76920a3/attachment.html>

------------------------------

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

End of varnish-misc Digest, Vol 72, Issue 21
********************************************

http://www.bbc.co.uk/
This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20120320/20e8c5c4/attachment-0001.html>

------------------------------

Message: 2
Date: Tue, 20 Mar 2012 10:06:04 +0000
From: Michael Borejdo <mib at electronic-minds.de>
To: "varnish-misc at varnish-cache.org" <varnish-misc at varnish-cache.org>
Subject: Re: Stale-While-Revalidate or Similar
Message-ID:
	<BE4EDDBE2C55604EB9F3EBA421CB227A2397145E at EM-MBX-1.electronic-minds.local>

Content-Type: text/plain; charset="iso-8859-1"

Hi there,

There is a similar feature to this available in Varnish:
https://www.varnish-cache.org/docs/trunk/tutorial/handling_misbehaving_servers.html

Unfortunately my servers are healthy (they do not send 5XX or the like), but are just slow as hell sometimes.
Or how is "health" "measured" by varnish?

I think this solution will not work for me, since my backend seems to be healthy. 
What I basically want to accomplish is to prevent "the poor sod" from waiting for a request. Stale content is acceptable, waiting isn't.

Thanks

Michael

---------------------------------------------------------------
Von: varnish-misc-bounces at varnish-cache.org [mailto:varnish-misc-bounces at varnish-cache.org] Im Auftrag von Graham Lyons
Gesendet: Dienstag, 20. M?rz 2012 10:51
An: varnish-misc at varnish-cache.org
Betreff: Re: Stale-While-Revalidate or Similar

Hi Michael,

There is a similar feature to this available in Varnish:
https://www.varnish-cache.org/docs/trunk/tutorial/handling_misbehaving_servers.html

Grace and saint modes correspond roughly, respectively to stale-while-revalidate and stale-if-error proposed in RFC5861.

Graham.

On 20 Mar 2012, at 09:24, varnish-misc-request at varnish-cache.org wrote:

Send varnish-misc mailing list submissions to
	varnish-misc at varnish-cache.org

To subscribe or unsubscribe via the World Wide Web, visit
	https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
or, via email, send a message with subject or body 'help' to
	varnish-misc-request at varnish-cache.org

You can reach the person managing the list at
	varnish-misc-owner at varnish-cache.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of varnish-misc digest..."

Today's Topics:

??1. Re: Varnish client IP address not display (MassiveScale.net)
??2. VUG5 Paris BOF (Jonathan Matthews)
??3. Re: Communication between Varnish and the Backend (Hugues Alary)
??4. using the max-age request header (Jeroen Ooms)
??5. Re: Communication between Varnish and the Backend (Per Buer)
??6. Stale-While-Revalidate or Similar (Michael Borejdo)

----------------------------------------------------------------------

Message: 1
Date: Mon, 19 Mar 2012 12:39:36 +0100
From: "MassiveScale.net" <info at massivescale.net>
To: varnish-misc at varnish-cache.org
Subject: Re: Varnish client IP address not display
Message-ID: <4F671AF8.3060201 at massivescale.net>
Content-Type: text/plain; charset=ISO-8859-1

On 03/19/2012 09:02 AM, Stewart Robinson wrote:

Jewel,
What header are you checking on the back end?
You should be reading the x-forwarded-for header rather than some form
of remote_addr.

Alternatively, if your backend server is Apache, you could use mod_rpaf
http://stderr.net/apache/rpaf/

-- 
Andrzej Godziuk
http://MassiveScale.net/

------------------------------

Message: 2
Date: Mon, 19 Mar 2012 13:25:22 +0000
From: Jonathan Matthews <contact at jpluscplusm.com>
To: varnish-misc at varnish-cache.org
Subject: VUG5 Paris BOF
Message-ID:
	<CAKsTx7Dv1rf62dJ6XYMo0n+vcB02nP7fyxX9CcPaPw2KbHUD-Q at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

Hi all -

I'm heading along to VUG5 this week - really looking forward to
meeting a lot of you and seeing how you're using and abusing our
favourite tool :-)

When I signed up, I don't recall ticking the "BOF" checkbox - could
anyone let me know if it's too late to get in on that action?

Either way, are people (like me) who are attending *just* the user day
(Thursday) perhaps looking for more Varnish chat on the Friday or the
weekend? I'm there 'til Sunday, and would love to get the most out of
a high concentration of Varnish-literate folk!

All the best,
Jonathan
-- 
Jonathan Matthews
London, Oxford, UK
http://www.jpluscplusm.com/contact.html

------------------------------

Message: 3
Date: Mon, 19 Mar 2012 12:50:23 -0700
From: Hugues Alary <hugues.alary at gmail.com>
To: varnish-misc at varnish-cache.org
Subject: Re: Communication between Varnish and the Backend
Message-ID:
	<CAN-YAk8S=dCxUXR5GbyC3+s3oSgis0+doetcw0N8QUGPJ1Mteg at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Happy Monday everybody,

Nobody, got some good advices for me? :)

Thanks,
-Hugues

On Fri, Mar 16, 2012 at 3:42 PM, Hugues Alary <hugues.alary at gmail.com>wrote:

Hi everybody,

I'm configuring varnish to be used with my web application.

I need my backend to be able to give instructions to Varnish depending on
the page requested. For example, some of my pages contain ESI includes,
some don't.

I want my application to tell varnish wether or not do the esi
processing. This approach allows me to keep my VCL configuration file
abstracted from the application. I can then redistribute the VCL to other
users and let their application control the cache.

So far my solution is to instruct varnish via HTTP headers.

In my current example I set a header X-Application-DoEsi to 1 or 0. When
varnish receives the response from the backend, it looks for the value
of X-Application-DoEsi and does or not the ESI processing.

??- I was wondering if communicating with varnish via HTTP header is a
??good solution?
??- How bad can it be performance wise?
??- Should I be worried about security (interception/modification of the
??communication between Varnish and the backend), even if no credential will
??never be sent through HTTP headers? (Also, currently Varnish and the
??backend are on the same machine, but chances are that they will in the
??future not live on the same host).

In the future, I plan on instructing varnish not to cache certain pages
containing user defined query strings. I want the user to be able to
specify these un-cachable urls query strings directly in the application.
The application will then send the un-cachable query strings in a header
X-Application-QueryStringNoCache: "querystring1,querystring2,...".

Is that a bad idea?

Thank you for your help!

--
Hugues ALARY

-- 
Hugues ALARY
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20120319/a5ee34d6/attachment-0001.html>

------------------------------

Message: 4
Date: Mon, 19 Mar 2012 17:11:57 -0700
From: Jeroen Ooms <jeroen.ooms at stat.ucla.edu>
To: varnish-misc at varnish-cache.org
Subject: using the max-age request header
Message-ID:
	<CABFfbXtVc3vMVB-7_vEUZW6fXup5zte6rVhwL7avsQdqHX1ZHQ at mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

How do I configure my VLC to let Varnish respect the 'Cache-Control
max-age' request header? Someone on SO suggested the following:

if (req.http.Cache-Control ~ '\bmax-age=\b`) {
???if (std.duration(regsub(req.http.Cache-Control,
".*\bmax-age=(\d+)\b.*", "\1s"), 0s) < obj.age) {
???????return (pass);
???}
}

However, what I don't like about this is that vcl_pass does not
actually update the cache with the new entry. How can I configure it
so that when max age < obj.age, a fresh copy will be fetched and
inserted into cache?

------------------------------

Message: 5
Date: Tue, 20 Mar 2012 07:25:05 +0100
From: Per Buer <perbu at varnish-software.com>
To: Hugues Alary <hugues.alary at gmail.com>
Cc: varnish-misc at varnish-cache.org
Subject: Re: Communication between Varnish and the Backend
Message-ID:
	<CAOXZevDAcWRxDFbUA_mSh4_Hh2S1ukF9PGbmN6X2UThC-Md+5Q at mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

On Fri, Mar 16, 2012 at 11:42 PM, Hugues Alary <hugues.alary at gmail.com>wrote:
(..)

??- I was wondering if communicating with varnish via HTTP header is a
??good solution?

Yes.

??- How bad can it be performance wise?

Negligible.

??- Should I be worried about security (interception/modification of the
??communication between Varnish and the backend), even if no credential will
??never be sent through HTTP headers? (Also, currently Varnish and the
??backend are on the same machine, but chances are that they will in the
??future not live on the same host).

It's more or less impossible to turn Varnish into an open proxy so I
wouldn't worry about that.

In the future, I plan on instructing varnish not to cache certain pages
containing user defined query strings. I want the user to be able to
specify these un-cachable urls query strings directly in the application.
The application will then send the un-cachable query strings in a header
X-Application-QueryStringNoCache: "querystring1,querystring2,...".

Is that a bad idea?

No. If it works for you then go for it.

-- 
Per Buer, CEO
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / Skype: per.buer
*Varnish makes websites fly!*
Whitepapers <http://www.varnish-software.com/whitepapers> |
Video<http://www.youtube.com/watch?v=x7t2Sp174eI> |
Twitter <https://twitter.com/varnishsoftware>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20120320/59e9f0ef/attachment-0001.html>

------------------------------

Message: 6
Date: Tue, 20 Mar 2012 09:24:02 +0000
From: Michael Borejdo <mib at electronic-minds.de>
To: "varnish-misc at varnish-cache.org" <varnish-misc at varnish-cache.org>
Subject: Stale-While-Revalidate or Similar
Message-ID:
	<BE4EDDBE2C55604EB9F3EBA421CB227A23971439 at EM-MBX-1.electronic-minds.local>

Content-Type: text/plain; charset="us-ascii"

Hello List,

Is there support for something like "stale-while-revalidate" coming to Varnish?
I need to make the request to my backend-servers async and serve slightly stale content (defined by grace) and I do not want a user to wait for the backend at any time.
Or is there any other way with using vcl trickery to accomplish this?
(Unfortunately I cannot seem to find any info on this topic and the thread I found on the list has not been answered) ?[1]

[1] https://www.varnish-cache.org/lists/pipermail/varnish-misc/2012-February/021644.html

Thanks

Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20120320/c76920a3/attachment.html>

------------------------------

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

End of varnish-misc Digest, Vol 72, Issue 21
********************************************

?

http://www.bbc.co.uk
This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this. 

------------------------------

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

End of varnish-misc Digest, Vol 72, Issue 22
********************************************

http://www.bbc.co.uk
This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated.
If you have received it in error, please delete it from your system.
Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately.
Please note that the BBC monitors e-mails sent or received.
Further communication will signify your consent to this. 