<div dir="ltr"><div>Thank you for help.</div><div><br></div><div>I would like to show the detail of my thinking:</div><div><br></div><div>1. Client requests /foo.jpg, And the request header has an additional field which called x-rewrite-level (x-rewrite-level have three value: 1,2,3, indicate different rewrite level). For example some request header may contain the "x-rewrite-level: 1".</div>
<div>2. Varnish looks up foo.jpg based on "url + req.x-rewrite-level" hash value, at the first time it will miss.</div><div>3. Varnish goes to backend and asks for /foo.jpg and backend responds.</div><div>4. Varnish notifies that this should be rewritten, and rewrites it according to x-rewrite-level value by the rewrite module. </div>
<div>5. The rewritten image replaces the original one(fetch from backend) in the varnish storage.</div><div>6. Subsequent request /foo.jpg with req.x-rewrite-level = 1 will hit the rewritten object in varnish.</div><div>7. Subsequent request /foo.jpg with req.x-rewrite-level = 2 or 3 will miss based on "url + req.x-rewrite-level" hash value. It goes to step 3.</div>
<div><br></div><div>The request with the same url and different x-rewrite-level value will receive different image from varnish.</div><div>Varnish should store 3 different objects according to x-rewrite-level value(1,2,3)</div>
<div><br></div><div>Regards,</div><div>--</div><div>Shawn Wang</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/2/2 Per Buer <span dir="ltr"><<a href="mailto:perbu@varnish-software.com" target="_blank">perbu@varnish-software.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">That is an interesting use case. So, if I understand it correctly what would happen would be something like this:<div>
<br></div><div>1. Client requests /foo</div><div>2. Varnish goes to backend and asks for /foo</div>
<div>3. Backend responds and notifies that this would be accessed in the future as /bar</div><div>4. Varnish changes the object in memory or maybe copies it.</div><div><br></div><div>I'm guessing that would not be possible as the hashing happens quite early and you would have to alter the hash before looking it up. It might be easier to maintain a map of various rewrites in memory, memcache or redis and lookup the rewrite in vcl_recv.</div>
<div><br></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Sat, Feb 2, 2013 at 3:35 AM, Xianzhe Wang <span dir="ltr"><<a href="mailto:wxz19861013@gmail.com" target="_blank">wxz19861013@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi,</div><div>I'm sorry, I wasn't being clear. Thank you for your patience.</div><div>I'm trying to estimate the response header from backend, if the "content-type:" is " image/*", we know that the response body is image. Then I'll rewrite</div>
<div> the image and insert it into varnish cache. Subsequent request will hit the rewritten image.</div><div>That's I wanna to do. Fetch backend response, rewrite image and insert into varnish cache.</div><div><br></div>
<div>Regards,</div><div>--</div><div>Shawn Wang</div><div><br></div><div><font color="#000000" face="Consolas, Lucida Console, monospace"><span style="font-size:12px;white-space:pre-wrap"><br></span></font></div>
<div><font color="#000000" face="Consolas, Lucida Console, monospace"><span style="font-size:12px;white-space:pre-wrap"><br></span></font></div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">
2013/2/1 Per Buer <span dir="ltr"><<a href="mailto:perbu@varnish-software.com" target="_blank">perbu@varnish-software.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>I don't quite understand what you're trying to do. Varnish will store the jpg together with the response headers in memory. When you request the object Varnish will deliver it verbatim along with the HTTP headers. What exactly are you trying to do?</div>
<div><br></div><div><br></div><div>PS: I see we haven't built packages of 3.0.3-plus, yet. This should pop up in the repo next week. Until then 3.0.2s might suffice.</div></div><div><div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Fri, Feb 1, 2013 at 8:01 AM, Xianzhe Wang <span dir="ltr"><<a href="mailto:wxz19861013@gmail.com" target="_blank">wxz19861013@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div><div> Hi,</div><div>Thanks for clarification. What you say is very clear.</div><div>I am sorry to show my poor English, but I have tried my best to communicate.</div><div><br></div><div>There is another question.For example, if we request a .jpg file(cacheable), varnish will encapsulation it as an object and insert into memory. How can we get the .jpg file from the object?</div>
<div><br></div><div>Thank you for help again.</div><div><br></div><div>-Shawn Wang</div></div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/1/30 Per Buer <span dir="ltr"><<a href="mailto:perbu@varnish-software.com" target="_blank">perbu@varnish-software.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi,<br></div><div><br></div><div>I was a bit quick and I didn't read the whole email the first time. Sorry about that. You're actually using the streaming branch, already I see. What you're writing is really, really odd. There is a slight lock while the "first" object is being fetched where other requests will be put on the waiting list. However, when the hit-for-pass object is created these should be released and pass'ed to the clients.<div>
<div><div><br></div><div>If the backend takes forever coming back with the response headers then the situation would be something like what you describe. However, that would be odd and doesn't make much sense.</div>
<div><br></div><div>PS: The streaming branch was renamed "plus" when it got other experimental features. You'll find source <a href="https://github.com/mbgrydeland/varnish-cache" target="_blank">https://github.com/mbgrydeland/varnish-cache</a> and packages at <a href="http://repo.varnish-cache.org/test" target="_blank">repo.varnish-cache.org/test</a> if I recall correctly.</div>
<div><br></div><div><br></div></div></div></div></div><div class="gmail_extra"><div><div><br><br><div class="gmail_quote">On Wed, Jan 30, 2013 at 3:32 AM, Xianzhe Wang <span dir="ltr"><<a href="mailto:wxz19861013@gmail.com" target="_blank">wxz19861013@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><font face="arial, sans-serif"><div><span style="font-size:14px">Hi,</span></div><div><span style="font-size:14px">Thanks a lot.</span></div>
<div><span style="font-size:14px"><br></span></div><div><span style="font-size:14px">I tried option </span></div><div>
<div><span style="font-size:14px">"set req.hash_ignore_busy = true;"</span></div><div><span style="font-size:14px">in vlc_recv. </span></div></div><div><span style="font-size:14px">I think it works. But there are side effects: it would increase backend load.</span></div>
<div><span style="font-size:14px"><br></span></div><div><span style="font-size:14px">I have an idea about it in my previous email. what do you think about it?</span></div><div><span style="font-size:14px"><br></span></div>
<div><span style="font-size:14px">Another question is that where can I find the "plus" branch of Varnish which matches this issue.</span></div><div><div><span style="font-size:14px"><br></span></div>
<div><span style="font-size:14px"> Any suggestions will be appreciate. </span></div>
</div><div><span style="font-size:14px">Thanks again for help.</span></div><div><span style="font-size:14px"><br></span></div><div><span style="font-size:14px">Regards,</span></div><div><span style="font-size:14px">--</span></div>
<div><span style="font-size:14px">Shawn Wang</span></div><div><span style="font-size:14px"><br></span></div></font><div><div><div><div><br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Xianzhe Wang</b> <span dir="ltr"><<a href="mailto:wxz19861013@gmail.com" target="_blank">wxz19861013@gmail.com</a>></span><br>
Date: 2013/1/30<br>Subject: Re: How to make multiple clients can get the response at the same time by stream.<br>To: Jakub Słociński <<a href="mailto:kuba@ovh.net" target="_blank">kuba@ovh.net</a>><br><br><br><div dir="ltr">
<div>Hi Jakub S.</div>
<div>Thank you very much.</div><div>I tried, and take a simple test, two client request the big file at the same time, they get the response stream immediately, so it works.</div><div>
In that case, multiple requests will go directly to "pass", they do not need to wait, but it would increase backend load. </div><div>We need to balance the benefits and drawbacks.</div><div><br></div><div>I wanna is that:</div>
<div> Client 1 requests url /foo</div><div> Client 2..N request url /foo</div><div> Varnish tasks a worker to fetch /foo for Client 1</div><div> Client 2..N are now queued pending response from the worker</div>
<div> Worker fetch response header(just header not include body) from backend, and find it non-cacheable, then make the remaining requests(Client 2..N) go directly to "pass". And creat the hit_for_pass object synchronously in the first request(Client 1).</div>
<div> Subsequent requests are now given the hit_for_pass object instructing them to go to the backend as long as the hit_for_pass object exists.</div><div><br></div><div>As I mentioned below, is it feasible? Or do you have any Suggestions? </div>
<div><br></div><div>Thanks again for help.</div><div><br></div><div>Regards,</div><div>--</div><div>Shawn Wang</div><div><div style="text-align:justify">
</div><div style="text-align:justify"><font color="#444444" face="sans-serif"><span style="font-size:17px;line-height:21px"><br></span></font></div></div></div><div><div><div><div><div class="gmail_extra"><br>
<br>
<div class="gmail_quote">2013/1/29 Jakub Słociński <span dir="ltr"><<a href="mailto:kuba@ovh.net" target="_blank">kuba@ovh.net</a>></span><br>
<blockquote class="gmail_quote" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi Xianzhe Wang,<br>
you should try option<br>
"set req.hash_ignore_busy = true;"<br>
in vlc_recv.<br>
<br>
Regards,<br>
--<br>
Jakub S.<br>
<br>
<br>
Xianzhe Wang napisał(a):<br>
<div><div>> Hello everyone,<br>
> My varnish version is 3.0.2-streaming release.And I set<br>
> "beresp.do_stream = true" in vcl_fetch in order to "Deliver the object to<br>
> the client directly without fetching the whole object into varnish";<br>
><br>
> This is a part of my *.vcl file:<br>
><br>
> sub vcl_fetch {<br>
> set beresp.grace = 30m;<br>
><br>
> set beresp.do_stream = true;<br>
><br>
> if (beresp.http.Content-Length && beresp.http.Content-Length ~<br>
> "[0-9]{8,}") {<br>
> return (hit_for_pass);<br>
> }<br>
><br>
> if (beresp.http.Pragma ~ "no-cache" || beresp.http.Cache-Control ~<br>
> "no-cache" || beresp.http.Cache-Control ~ "private") {<br>
> return (hit_for_pass);<br>
> }<br>
><br>
> if (beresp.ttl <= 0s ||<br>
> beresp.http.Set-Cookie ||<br>
> beresp.http.Vary == "*") {<br>
><br>
> set beresp.ttl = 120 s;<br>
> return (hit_for_pass);<br>
> }<br>
><br>
> return (deliver);<br>
> }<br>
><br>
> Then I request a big file(about 100M+) like "xxx.zip" from clients.There is<br>
> only one client can access the object.because "the object will marked as<br>
> busy as it is delivered."<br>
><br>
> But if the request goes directly to “pass” ,multiple clients can get the<br>
> response at the same time.<br>
><br>
> Also if I remove<br>
> if (beresp.http.Content-Length && beresp.http.Content-Length ~<br>
> "[0-9]{8,}") {<br>
> return (hit_for_pass);<br>
> }<br>
> to make the file cacheable,multiple clients can get the response at the<br>
> same time.<br>
><br>
> Now I want "multiple clients can get the response at the same time." in all<br>
> situations("pass","hit","hit_for_pass").<br>
><br>
> What can I do for it?<br>
> Any suggestions will be appreciate.<br>
> Thank you.<br>
><br>
> -Shawn Wang<br>
<br>
</div></div>> _______________________________________________<br>
> varnish-misc mailing list<br>
> <a href="mailto:varnish-misc@varnish-cache.org" target="_blank">varnish-misc@varnish-cache.org</a><br>
> <a href="https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc" target="_blank">https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc</a><br>
<br>
</blockquote></div><br></div>
</div></div></div></div></div><br></div></div></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br></font></span><div style="color:rgb(34,34,34);font-size:12.727272033691406px;font-family:arial,sans-serif;margin:5px 15px 0px 0px;padding-bottom:5px">
<span><font color="#888888">
</font></span><span><font color="#888888">
</font></span><span><font color="#888888"></font></span><table border="0" cellpadding="0" cellspacing="0" style="text-align:start;font-size:12px;line-height:1.5em;font-family:'Helvetica Neue',Arial,sans-serif;color:rgb(102,102,102);width:550px;border-top-width:1px;border-top-style:solid;border-top-color:rgb(238,238,238);border-bottom-width:1px;border-bottom-style:solid;border-bottom-color:rgb(238,238,238);margin-top:20px;padding-top:5px;padding-bottom:5px">
<tbody><tr><td width="100" style="font-family:arial,sans-serif;margin:0px"><a href="http://www.varnish-software.com/" style="color:rgb(17,85,204)" target="_blank"><img style="float:left;margin-left:10px"></a></td>
<td style="font-family:arial,sans-serif;margin:0px"><span><font color="#888888"><b style="font-size:14px;color:rgb(34,34,34)">Per Buer</b></font></span><div><br>CEO | Varnish Software AS<br>Phone: <a href="tel:%2B47%20958%2039%20117" value="+4795839117" target="_blank">+47 958 39 117</a> | Skype: per.buer<br>
<span style="font-weight:bold">We Make Websites Fly!<br>
<br></span></div></td></tr></tbody></table><div></div></div><div style="clear:both;color:rgb(34,34,34);font-size:12.727272033691406px;font-family:arial,sans-serif;margin:15px 0px"></div>
</div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div style="color:rgb(34,34,34);font-size:12.727272033691406px;font-family:arial,sans-serif;margin:5px 15px 0px 0px;padding-bottom:5px">
<table border="0" cellpadding="0" cellspacing="0" style="text-align:start;font-size:12px;line-height:1.5em;font-family:'Helvetica Neue',Arial,sans-serif;color:rgb(102,102,102);width:550px;border-top-width:1px;border-top-style:solid;border-top-color:rgb(238,238,238);border-bottom-width:1px;border-bottom-style:solid;border-bottom-color:rgb(238,238,238);margin-top:20px;padding-top:5px;padding-bottom:5px">
<tbody><tr><td width="100" style="font-family:arial,sans-serif;margin:0px"><a href="http://www.varnish-software.com/" style="color:rgb(17,85,204)" target="_blank"><img style="float:left;margin-left:10px"></a></td>
<td style="font-family:arial,sans-serif;margin:0px"><b style="font-size:14px;color:rgb(34,34,34)">Per Buer</b><br>CEO | Varnish Software AS<br>Phone: <a href="tel:%2B47%20958%2039%20117" value="+4795839117" target="_blank">+47 958 39 117</a> | Skype: per.buer<br>
<span style="font-weight:bold">We Make Websites Fly!<br>
<br></span></td></tr></tbody></table><div></div></div><div style="clear:both;color:rgb(34,34,34);font-size:12.727272033691406px;font-family:arial,sans-serif;margin:15px 0px"></div>
</div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div style="color:rgb(34,34,34);font-size:12.727272033691406px;font-family:arial,sans-serif;margin:5px 15px 0px 0px;padding-bottom:5px">
<table border="0" cellpadding="0" cellspacing="0" style="text-align:start;font-size:12px;line-height:1.5em;font-family:'Helvetica Neue',Arial,sans-serif;color:rgb(102,102,102);width:550px;border-top-width:1px;border-top-style:solid;border-top-color:rgb(238,238,238);border-bottom-width:1px;border-bottom-style:solid;border-bottom-color:rgb(238,238,238);margin-top:20px;padding-top:5px;padding-bottom:5px">
<tbody><tr><td width="100" style="font-family:arial,sans-serif;margin:0px"><a href="http://www.varnish-software.com/" style="color:rgb(17,85,204)" target="_blank"><img style="float:left;margin-left:10px"></a></td>
<td style="font-family:arial,sans-serif;margin:0px"><b style="font-size:14px;color:rgb(34,34,34)">Per Buer</b><br>CEO | Varnish Software AS<br>Phone: +47 958 39 117 | Skype: per.buer<br><span style="font-weight:bold">We Make Websites Fly!<br>
<br></span></td></tr></tbody></table><div></div></div><div style="clear:both;color:rgb(34,34,34);font-size:12.727272033691406px;font-family:arial,sans-serif;margin:15px 0px"></div>
</div>
</div></div></blockquote></div><br></div>