From aprabhu at squarespace.com Tue Sep 1 13:52:49 2020 From: aprabhu at squarespace.com (Ameya Prabhu) Date: Tue, 1 Sep 2020 09:52:49 -0400 Subject: Performance degradation on LRU nuking Message-ID: Hi, We are seeing performance issues with Varnish?s file storage when the storage reaches its capacity and Varnish starts LRU nuking. As soon as varnish begins nuking objects, the number of worker threads(MAIN.threads) starts increasing until it hits the limit(thread_pool_max) and remains pinned on the thread_pool_max. This then leads to incoming requests getting queued. We are running Varnish with following configuration: Varnish version: 6.1 Storage type: file storage Storage size: 150 Gb Granularity: 4096 Thread_pools: 2 Thread_pool_max: 5000 Nuke_limit: 5000 OS: Centos Linux release 7.5.1804 Can we tune any varnishadm params to optimize the nuking ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Tue Sep 1 14:01:09 2020 From: slink at schokola.de (Nils Goroll) Date: Tue, 1 Sep 2020 16:01:09 +0200 Subject: Performance degradation on LRU nuking In-Reply-To: References: Message-ID: <290963d9-b822-ca77-135d-ef49dcac5a9b@schokola.de> On 01/09/2020 15:52, Ameya Prabhu wrote: > > We are seeing performance issues with Varnish?s file storage when the storage > reaches its capacity and Varnish starts LRU nuking. As soon as varnish begins > nuking objects, the number of worker threads(MAIN.threads) starts increasing > until it hits the limit(thread_pool_max) and remains pinned on the > thread_pool_max. This then leads to incoming requests getting queued.? > some 12-18 months back, I made an effort to prototype improvements to the LRU nuking, but never reached an anything mature enough for a pull request. That said, in my case I used malloc storage and the issue was high lock contention. As you mention you are using file storage, I would suspect that IO performance could be the bottleneck. There is no magic knob I would be aware of, other than using more RAM and faster IO. Nils -------------- next part -------------- An HTML attachment was scrubbed... URL: From aprabhu at squarespace.com Tue Sep 1 20:29:12 2020 From: aprabhu at squarespace.com (Ameya Prabhu) Date: Tue, 1 Sep 2020 16:29:12 -0400 Subject: Performance degradation on LRU nuking In-Reply-To: <290963d9-b822-ca77-135d-ef49dcac5a9b@schokola.de> References: <290963d9-b822-ca77-135d-ef49dcac5a9b@schokola.de> Message-ID: Thanks Nils. What were the shortcomings that you were trying to improve ? Did any of the releases post v6.1 show any improvement ? On Tue, Sep 1, 2020 at 10:01 AM Nils Goroll wrote: > On 01/09/2020 15:52, Ameya Prabhu wrote: > > > We are seeing performance issues with Varnish?s file storage when the > storage reaches its capacity and Varnish starts LRU nuking. As soon as > varnish begins nuking objects, the number of worker threads(MAIN.threads) > starts increasing until it hits the limit(thread_pool_max) and remains > pinned on the thread_pool_max. This then leads to incoming requests getting > queued. > > some 12-18 months back, I made an effort to prototype improvements to the > LRU nuking, but never reached an anything mature enough for a pull request. > > That said, in my case I used malloc storage and the issue was high lock > contention. As you mention you are using file storage, I would suspect that > IO performance could be the bottleneck. > > There is no magic knob I would be aware of, other than using more RAM and > faster IO. > > Nils > -------------- next part -------------- An HTML attachment was scrubbed... URL: From slink at schokola.de Wed Sep 2 05:52:12 2020 From: slink at schokola.de (Nils Goroll) Date: Wed, 2 Sep 2020 07:52:12 +0200 Subject: Performance degradation on LRU nuking In-Reply-To: References: <290963d9-b822-ca77-135d-ef49dcac5a9b@schokola.de> Message-ID: On 01/09/2020 22:29, Ameya Prabhu wrote: > Thanks Nils. What were the shortcomings that you were trying to improve ? As I wrote before: ??? in my case I used malloc storage and the issue was high lock contention. Also I should add that the exp_thread was basically taking a full CPU. > Did any of the releases post v6.1 show any improvement ?? I am not aware of any work in that area. From batanun at hotmail.com Thu Sep 3 10:56:13 2020 From: batanun at hotmail.com (Batanun B) Date: Thu, 3 Sep 2020 10:56:13 +0000 Subject: Possible to detect a previous xkey softpurge? Message-ID: Hi, We sometimes have a problem with the backend using its internal cache for a few seconds too long after something has been updated. We trigger a softpurge (xkey vmod) in varnish, but if someone requests the page again very soon after that, the data that Varnish gets from the backend might be old. In this case, we would like to be able to tell the backend, maybe using an extra header, that it should skip its internal caches and give us the updated content. But, I'm not sure how to archive this in Varnish. Is it possible to detect that the page requested has been softpurged earlier? If yes, is it also possible to see when that softpurge took place? Because we would only ever need to do this if the softpurge happened less than let's say 30 seconds ago. And the reason that the backend data might be old after an update is that what we call "the backend" (from a Varnish perspective) is actually a complex setup of services. And there, the update happens in one place, and our "backend" is actually a frontend server that sometimes don't get the information about the update quick enough. I think that the implementation of this system is a bit faulty, but it is what it is, and I would like to use the power of Varnish to handle this, if possible. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Sep 3 14:27:08 2020 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 3 Sep 2020 07:27:08 -0700 Subject: Possible to detect a previous xkey softpurge? In-Reply-To: References: Message-ID: Hi, You can't detect a softpurge, but you can tell Varnish to ignore grace: ``` sub vcl_recv { if (req.http.no-grace) { set req.grace = 0s; } } ``` the softpurge kill the ttl at the object level, and this kills the grace at the request level, so Varnish will reach out to the backend. But note that it will also do the same even without a prior softpurge, it just needs an expired ttl. -- Guillaume Quintard On Thu, Sep 3, 2020 at 3:57 AM Batanun B wrote: > Hi, > > We sometimes have a problem with the backend using its internal cache for > a few seconds too long after something has been updated. We trigger a > softpurge (xkey vmod) in varnish, but if someone requests the page again > very soon after that, the data that Varnish gets from the backend might be > old. In this case, we would like to be able to tell the backend, maybe > using an extra header, that it should skip its internal caches and give us > the updated content. > > But, I'm not sure how to archive this in Varnish. Is it possible to detect > that the page requested has been softpurged earlier? If yes, is it also > possible to see when that softpurge took place? Because we would only ever > need to do this if the softpurge happened less than let's say 30 seconds > ago. > > And the reason that the backend data might be old after an update is that > what we call "the backend" (from a Varnish perspective) is actually a > complex setup of services. And there, the update happens in one place, and > our "backend" is actually a frontend server that sometimes don't get the > information about the update quick enough. I think that the implementation > of this system is a bit faulty, but it is what it is, and I would like to > use the power of Varnish to handle this, if possible. > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From batanun at hotmail.com Thu Sep 3 16:55:05 2020 From: batanun at hotmail.com (Batanun B) Date: Thu, 3 Sep 2020 16:55:05 +0000 Subject: Possible to detect a previous xkey softpurge? In-Reply-To: References: , Message-ID: Hi, I'm not sure how that "no-grace" header would be set. The softpurge could theoretically impact hundred of URLs, and what we would like is that any requests for these URLs after the softpurge should include a special header when talking with the backend. Skipping grace in general, and sending that special header to all requests to the backend, is not what we want. But now I am thinking of an alternative, that might give us somewhat what we want while being much simpler and not needing to know if a softpurge has happened or not. Since we really only need to do this in a short time period after a softpurge, and the softpurge sets ttl to zero, then we can skip the "after a softpurge" requirement and simply check if the ttl recently expired. As far as I understand it, when the ttl has expired, the obj.ttl is a negative value indicating how many seconds since the ttl expired. So 15 seconds after the ttl, it would be -15s. Then we can have something like this in in vcl_hit: ``` if (obj.ttl > -15s) { set req.http.X-my-backend-skip-cache = "true"; return (miss); } ``` I can't check this right now, from the computer I am at. But it should work, right? Then the only "false positives" we will end up with are the requests that happen to come in within 15 seconds of the regular ttl expiring. But if we get the cache invalidation to work fully (including in the backend), then we should be able to increase the regular ttl higher than the current 5s, and then this false positive should happen much more rarely. Guillaume Quintard wrote: > > Hi, > > You can't detect a softpurge, but you can tell Varnish to ignore grace: > > ``` > sub vcl_recv { > ? ? if (req.http.no-grace) { > ? ? ? ? set req.grace = 0s; > ? ? } > } > ``` > > the softpurge kill the ttl at the object level, and this kills the grace at the request level, so Varnish will reach out to the backend. > > But note that it will also do the same even without a prior softpurge, it just needs an expired ttl. > From guillaume at varnish-software.com Thu Sep 3 17:02:56 2020 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 3 Sep 2020 10:02:56 -0700 Subject: Possible to detect a previous xkey softpurge? In-Reply-To: References: Message-ID: Wait a minute, I only offered to reduce the grace depending on a header because I thought you only wanted to do that for some privilege clients. >From what I understand now, you want to make sure that after a softpurge, you never get the stale object. So, if I may: why use a softpurge at all? Just remove the objects completely and be done with it. -- Guillaume Quintard On Thu, Sep 3, 2020 at 9:55 AM Batanun B wrote: > Hi, > > I'm not sure how that "no-grace" header would be set. The softpurge could > theoretically impact hundred of URLs, and what we would like is that any > requests for these URLs after the softpurge should include a special header > when talking with the backend. > > Skipping grace in general, and sending that special header to all requests > to the backend, is not what we want. > > But now I am thinking of an alternative, that might give us somewhat what > we want while being much simpler and not needing to know if a softpurge has > happened or not. Since we really only need to do this in a short time > period after a softpurge, and the softpurge sets ttl to zero, then we can > skip the "after a softpurge" requirement and simply check if the ttl > recently expired. As far as I understand it, when the ttl has expired, the > obj.ttl is a negative value indicating how many seconds since the ttl > expired. So 15 seconds after the ttl, it would be -15s. Then we can have > something like this in in vcl_hit: > > ``` > if (obj.ttl > -15s) { > set req.http.X-my-backend-skip-cache = "true"; > return (miss); > } > ``` > > I can't check this right now, from the computer I am at. But it should > work, right? Then the only "false positives" we will end up with are the > requests that happen to come in within 15 seconds of the regular ttl > expiring. But if we get the cache invalidation to work fully (including in > the backend), then we should be able to increase the regular ttl higher > than the current 5s, and then this false positive should happen much more > rarely. > > > Guillaume Quintard wrote: > > > > Hi, > > > > You can't detect a softpurge, but you can tell Varnish to ignore grace: > > > > ``` > > sub vcl_recv { > > if (req.http.no-grace) { > > set req.grace = 0s; > > } > > } > > ``` > > > > the softpurge kill the ttl at the object level, and this kills the grace > at the request level, so Varnish will reach out to the backend. > > > > But note that it will also do the same even without a prior softpurge, > it just needs an expired ttl. > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Thu Sep 3 18:59:14 2020 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 3 Sep 2020 18:59:14 +0000 Subject: Possible to detect a previous xkey softpurge? In-Reply-To: References: Message-ID: On Thu, Sep 3, 2020 at 10:57 AM Batanun B wrote: > > Hi, > > We sometimes have a problem with the backend using its internal cache for a few seconds too long after something has been updated. We trigger a softpurge (xkey vmod) in varnish, but if someone requests the page again very soon after that, the data that Varnish gets from the backend might be old. In this case, we would like to be able to tell the backend, maybe using an extra header, that it should skip its internal caches and give us the updated content. You can't really do that, but the closest I can think of would look like this: sub vcl_backend_fetch { if (bereq.is_bgfetch) { # tell the backend to somehow not serve from cache } } Since a soft purge artificially moves you inside the grace period, any hit on those objects would trigger a background fetch. Arguably, if you use Varnish to cache responses, you might as well always tell your backend not to serve from cache. Because if a soft purge moves you inside the grace period, there's no guarantee that the next hit will happen before the object leaves the grace period. At this point this will no longer trigger a background fetch... > But, I'm not sure how to archive this in Varnish. Is it possible to detect that the page requested has been softpurged earlier? If yes, is it also possible to see when that softpurge took place? Because we would only ever need to do this if the softpurge happened less than let's say 30 seconds ago. > > And the reason that the backend data might be old after an update is that what we call "the backend" (from a Varnish perspective) is actually a complex setup of services. And there, the update happens in one place, and our "backend" is actually a frontend server that sometimes don't get the information about the update quick enough. I think that the implementation of this system is a bit faulty, but it is what it is, and I would like to use the power of Varnish to handle this, if possible. Objects are immutable (modulus TTL) once they enter the cache so there's nothing actionable to record that they were soft-purged. Dridi From batanun at hotmail.com Sat Sep 12 21:44:39 2020 From: batanun at hotmail.com (Batanun B) Date: Sat, 12 Sep 2020 21:44:39 +0000 Subject: Possible to detect a previous xkey softpurge? In-Reply-To: References: , Message-ID: > From what I understand now, you want to make sure that after a softpurge, > you never get the stale object. So, if I may: why use a softpurge at all? > Just remove the objects completely and be done with it. I think I was not explaining my use case properly. It is not the Varnish cache I was talking about. The backend has it's own cache, and it can sometimes serve stale data (if requested seconds after an update). So completely removing the object from the _Varnish_ cache does not help here. And in fact it would just do harm, since if then the backend goes down we don't have any stale data from Varnish cache to serve. With softpurge and a long grace, we always are ready for a backend failure (assuming the data is in the cache). -------------- next part -------------- An HTML attachment was scrubbed... URL: From batanun at hotmail.com Sat Sep 12 21:54:52 2020 From: batanun at hotmail.com (Batanun B) Date: Sat, 12 Sep 2020 21:54:52 +0000 Subject: Possible to detect a previous xkey softpurge? In-Reply-To: References: , Message-ID: > Arguably, if you use Varnish to cache responses, you might as well > always tell your backend not to serve from cache. Because if a soft > purge moves you inside the grace period, there's no guarantee that the > next hit will happen before the object leaves the grace period. At > this point this will no longer trigger a background fetch... Well, the caching in the backend is not on the same level as the Varnish cache. In Varnish, a single request results in a single object to cache. In the backend, a single request can result in hundreds or thousands of separate lookups (some involving separate http calls to other services), each cachable with their own unique key. And most of these objects are reused from the cache for other requests as well. And skipping that internal cache completely, letting the backend do all these lookups and sub-requests every single time a request comes in to the backend, that would be terrible for performance. So we really only want to skip this internal cache in special circumstances. -------------- next part -------------- An HTML attachment was scrubbed... URL: From batanun at hotmail.com Sat Sep 12 22:06:55 2020 From: batanun at hotmail.com (Batanun B) Date: Sat, 12 Sep 2020 22:06:55 +0000 Subject: Can't get "streaming" or pipe to work, Varnish still waits for the full response Message-ID: Hi, We have some old (legacy) internal admin pages that do some classic old school processing while the page is loading, and outputting the current status as it is working. When requesting these pages directly (through Tomcat), I can see the results in the browser at the same time as the results are written on the other end. But when I go through Varnish, no matter what I try, I only see a blank page that is loading/waiting, and then when the backend is done writing, then I get the entire result in one go. How can I configure Varnish to bring any content to the client the moment it gets it from the backend, and not wait until the entire response is done? In vcl_backend_response I do this: set beresp.do_stream = true; set beresp.uncacheable = true; return (deliver); I have also tried returning (pipe) in vcl_recv (with and without do_stream and uncacheable). And gzip is turn off. But nothing helps. What can I do more? And how can I debug this? Varnishlog shows nothing that is telling me that it is buffering, or waiting for the response, or anything like that. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Mon Sep 14 06:26:27 2020 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 14 Sep 2020 06:26:27 +0000 Subject: Possible to detect a previous xkey softpurge? In-Reply-To: References: Message-ID: On Sat, Sep 12, 2020 at 9:56 PM Batanun B wrote: > > > Arguably, if you use Varnish to cache responses, you might as well > > always tell your backend not to serve from cache. Because if a soft > > purge moves you inside the grace period, there's no guarantee that the > > next hit will happen before the object leaves the grace period. At > > this point this will no longer trigger a background fetch... > > Well, the caching in the backend is not on the same level as the Varnish cache. In Varnish, a single request results in a single object to cache. In the backend, a single request can result in hundreds or thousands of separate lookups (some involving separate http calls to other services), each cachable with their own unique key. And most of these objects are reused from the cache for other requests as well. And skipping that internal cache completely, letting the backend do all these lookups and sub-requests every single time a request comes in to the backend, that would be terrible for performance. So we really only want to skip this internal cache in special circumstances. In this case you might want to coordinate your tiers with Last-Modified headers. Cached objects are immutable with Varnish, except for the timing attributes so my previous answer is still applicable: you can't record that an object was soft-purged by a VMOD. The bereq.is_bgfetch variable will tell you whether you are in the grace period but that's about it. Dridi From dridi at varni.sh Mon Sep 14 06:35:38 2020 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 14 Sep 2020 06:35:38 +0000 Subject: Can't get "streaming" or pipe to work, Varnish still waits for the full response In-Reply-To: References: Message-ID: On Sat, Sep 12, 2020 at 10:08 PM Batanun B wrote: > > Hi, > > We have some old (legacy) internal admin pages that do some classic old school processing while the page is loading, and outputting the current status as it is working. When requesting these pages directly (through Tomcat), I can see the results in the browser at the same time as the results are written on the other end. But when I go through Varnish, no matter what I try, I only see a blank page that is loading/waiting, and then when the backend is done writing, then I get the entire result in one go. > > How can I configure Varnish to bring any content to the client the moment it gets it from the backend, and not wait until the entire response is done? > > In vcl_backend_response I do this: > set beresp.do_stream = true; > set beresp.uncacheable = true; > return (deliver); Streaming is on by default, you don't need to do anything. > I have also tried returning (pipe) in vcl_recv (with and without do_stream and uncacheable). And gzip is turn off. But nothing helps. What can I do more? And how can I debug this? Varnishlog shows nothing that is telling me that it is buffering, or waiting for the response, or anything like that. It is indeed hard to get that information just from Varnish, you could try to capture TCP packets to check how long it takes for backend traffic to be forwarded to clients. It's not obvious why a response would be delayed, but it could happen to be related to the fetch_chunksize parameter. However I have never come across a setup where we needed to tune that knob... Dridi From batanun at hotmail.com Mon Sep 14 16:52:40 2020 From: batanun at hotmail.com (Batanun B) Date: Mon, 14 Sep 2020 16:52:40 +0000 Subject: Can't get "streaming" or pipe to work, Varnish still waits for the full response In-Reply-To: References: , Message-ID: > It's not obvious why a response would be delayed, but it could > happen to be related to the fetch_chunksize parameter. Hmm... fetch_chunksize is 16k by default, and 4k minmum. Does this mean that even if streaming is working fine, it will still buffer at least 4kb of data before sending that to the client? That would be way to much for this use case. The output is basically plain text, and it is flushed after a few lines of text has been written. We are talking maybe 50-100 bytes per flush. > However I have never come across a setup where we needed to Well, I actually bit the bullet and decided to refactor this particular admin page that I needed this for, so that the job is done in the background and the current status can be fetched whenever one feels like. That way the response is always quick. Hopefully we won't need this for too many of these old legacy admin pages. ________________________________ From: Dridi Boukelmoune Sent: Monday, September 14, 2020 8:35 AM To: Batanun B Cc: varnish-misc at varnish-cache.org Subject: Re: Can't get "streaming" or pipe to work, Varnish still waits for the full response On Sat, Sep 12, 2020 at 10:08 PM Batanun B wrote: > > Hi, > > We have some old (legacy) internal admin pages that do some classic old school processing while the page is loading, and outputting the current status as it is working. When requesting these pages directly (through Tomcat), I can see the results in the browser at the same time as the results are written on the other end. But when I go through Varnish, no matter what I try, I only see a blank page that is loading/waiting, and then when the backend is done writing, then I get the entire result in one go. > > How can I configure Varnish to bring any content to the client the moment it gets it from the backend, and not wait until the entire response is done? > > In vcl_backend_response I do this: > set beresp.do_stream = true; > set beresp.uncacheable = true; > return (deliver); Streaming is on by default, you don't need to do anything. > I have also tried returning (pipe) in vcl_recv (with and without do_stream and uncacheable). And gzip is turn off. But nothing helps. What can I do more? And how can I debug this? Varnishlog shows nothing that is telling me that it is buffering, or waiting for the response, or anything like that. It is indeed hard to get that information just from Varnish, you could try to capture TCP packets to check how long it takes for backend traffic to be forwarded to clients. It's not obvious why a response would be delayed, but it could happen to be related to the fetch_chunksize parameter. However I have never come across a setup where we needed to tune that knob... Dridi -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrhosseini at hotmail.com Sat Sep 26 10:39:23 2020 From: hrhosseini at hotmail.com (Hamidreza Hosseini) Date: Sat, 26 Sep 2020 10:39:23 +0000 Subject: varnish crush every 30 minutes in high pressure Message-ID: Hi Im using varnish , before i hadn't any problem in varnish but today after crushing my dns (I fixed it now) and in high pressure , it restart whole machine almost every 30 minutes even Netdata that i'm using for monitoring and it shows in service status: Sep 26 13:49:48 varnish-1 varnishd[945]: Child (1548) not responding to CLI, killed it. Sep 26 13:49:48 varnish-1 varnishd[945]: Unexpected reply from ping: 400 CLI communication error (hdr) Sep 26 13:49:48 varnish-1 varnishd[945]: Child (1548) died signal=9 What should i do for this problem? My varnish version: varnishd (varnish-6.0.5 revision 3065ccaacc4bb537fb976a524bd808db42c5fe40) Copyright (c) 2006 Verdens Gang AS Copyright (c) 2006-2019 Varnish Software AS -------------- next part -------------- An HTML attachment was scrubbed... URL: From neuteboom.log at gmail.com Sat Sep 26 16:39:30 2020 From: neuteboom.log at gmail.com (Johan Hendriks) Date: Sat, 26 Sep 2020 18:39:30 +0200 Subject: Question regarding vcl config Message-ID: Hello all, I have inherited a varnish server. And I am a little confused about a few lines in that config. It has the following in vcl_recv # Don't cache if there are request cookies if (req.http.Cookie) { set req.http.X-Debug-Varnish-Nocache-Recv = "Got request cookie (" + req.http.Cookie + ")"; return (pass); } if (req.url ~ "\.(png|gif|jpg|jpeg|ico|swf|css|js|pdf|ico|js|ogg|mp4)$") { unset req.http.cookie; } if (req.url ~ "^/(includes|images|templates)") { unset req.http.cookie; } So the first if block tells request with req.http.Cookie to pass to the backend. The second tells if the requested url ends with .png|gif|jpg|jpeg|ico|swf|css|js|pdf|ico|js|ogg|mp4 and so on to unset the cookie. But will these rules be executed or is the request already passed to the backend? Thanks for your time regards Neuteboom -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Sat Sep 26 16:55:25 2020 From: dridi at varni.sh (Dridi Boukelmoune) Date: Sat, 26 Sep 2020 16:55:25 +0000 Subject: Question regarding vcl config In-Reply-To: References: Message-ID: On Sat, Sep 26, 2020 at 4:41 PM Johan Hendriks wrote: > > Hello all, I have inherited a varnish server. And I am a little confused about a few lines in that config. > > It has the following in vcl_recv > > # Don't cache if there are request cookies > if (req.http.Cookie) { > set req.http.X-Debug-Varnish-Nocache-Recv = "Got request cookie (" + req.http.Cookie + ")"; > > return (pass); > } > > if (req.url ~ "\.(png|gif|jpg|jpeg|ico|swf|css|js|pdf|ico|js|ogg|mp4)$") { > unset req.http.cookie; > } > if (req.url ~ "^/(includes|images|templates)") { > unset req.http.cookie; > } > > So the first if block tells request with req.http.Cookie to pass to the backend. > The second tells if the requested url ends with .png|gif|jpg|jpeg|ico|swf|css|js|pdf|ico|js|ogg|mp4 and so on to unset the cookie. > But will these rules be executed or is the request already passed to the backend? The return(pass) statement will definitely prevent the following code to be executed. You might want to move the first if block after the two others, but it's hard to tell without prior knowledge of how the backend behaves... Dridi From dridi at varni.sh Sun Sep 27 06:39:14 2020 From: dridi at varni.sh (Dridi Boukelmoune) Date: Sun, 27 Sep 2020 06:39:14 +0000 Subject: Question regarding vcl config In-Reply-To: References: Message-ID: On Sat, Sep 26, 2020 at 8:31 PM Johan Hendriks wrote: > > Thank you very much for your quick responce, so the order of the rules matter when you write vcl? > I didn't know that, but it makes sense. Yes, VCL is an imperative programming language and statements are executed in order. > I start some tests. > Thanks again. Please keep the mailing list CC'd if you have more questions. Dridi From hrhosseini at hotmail.com Mon Sep 28 08:39:26 2020 From: hrhosseini at hotmail.com (Hamidreza Hosseini) Date: Mon, 28 Sep 2020 08:39:26 +0000 Subject: Varnish crashes periodically with high disk read and write Message-ID: Hi I'm using varnish-6.0.5, 1. 3nodes of varnish crashes every day and I should restart it else the whole of my system doesnt work anymore I don't know the reason but my disks at that time shows that It has 50 MB/s (400Mb/s) Read and 48 MB/s write but my outbound network shows 120Mb/s I put the link of Netdata picture at that time, http://uupload.ir/files/c60r_varnish.png How can I solve this problem? 2. My next question is that I have another varnish that I set malloc to 68Gb (the server has 86 Gb ram) but it will pass 68g till 86 and server crash, Why does it exceed from 68g through I have adjusted the accurate and permitted number? But really I'm confused and annoyed with first problem because it destroys my cluster ``` Sep 27 13:46:19 varnish-16 varnishd[121910]: Error: Manager got SIGTERM Sep 27 13:46:19 varnish-16 varnishd[121910]: Manager got SIGTERM Sep 27 13:46:19 varnish-16 varnishd[121910]: Debug: Stopping Child Sep 27 13:46:19 varnish-16 varnishd[121910]: Stopping Child Sep 27 13:46:30 varnish-16 varnishd[121910]: Error: Child (121932) died signal=15 Sep 27 13:46:30 varnish-16 varnishd[121910]: Child (121932) died signal=15 Sep 27 13:46:30 varnish-16 varnishd[121910]: Debug: Child cleanup complete Sep 27 13:46:30 varnish-16 varnishd[121910]: Child cleanup complete Sep 27 13:46:30 varnish-16 varnishd[121910]: Info: manager stopping child Sep 27 13:46:30 varnish-16 varnishd[121910]: Info: manager dies Sep 27 13:46:30 varnish-16 varnishd[121910]: manager stopping child Sep 27 13:46:30 varnish-16 varnishd[121910]: manager dies Sep 27 13:44:46 varnish-16 kernel: lowmem_reserve[]: 0 2934 15972 15972 15972 Sep 27 13:44:46 varnish-16 kernel: Node 0 DMA32 free:55712kB min:2968kB low:3708kB high:4452kB active_anon:0kB inactive_anon:2674740kB active_file:104kB inactive_file:0kB unevictable:15528kB isolated(ano Sep 27 13:44:47 varnish-16 kernel: lowmem_reserve[]: 0 0 13037 13037 13037 Sep 27 13:44:47 varnish-16 kernel: Node 0 Normal free:21044kB min:13196kB low:16492kB high:19792kB active_anon:492kB inactive_anon:12208096kB active_file:548kB inactive_file:1300kB unevictable:70304kB is Sep 27 13:44:47 varnish-16 kernel: lowmem_reserve[]: 0 0 0 0 0 Sep 27 13:44:47 varnish-16 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (E) 3*64kB (UE) 2*128kB (UE) 2*256kB (UE) 0*512kB 1*1024kB (U) 2*2048kB (ME) 2*4096kB (M) = 14332kB Sep 27 13:44:47 varnish-16 kernel: Node 0 DMA32: 7633*4kB (UME) 2869*8kB (UMEH) 39*16kB (UM) 14*32kB (UM) 12*64kB (U) 2*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 55580kB Sep 27 13:44:47 varnish-16 kernel: Node 0 Normal: 5132*4kB (UMEH) 69*8kB (UMH) 1*16kB (H) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 21096kB Sep 27 13:44:47 varnish-16 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Sep 27 13:44:47 varnish-16 kernel: 3742574 total pagecache pages Sep 27 13:44:47 varnish-16 kernel: 3720854 pages in swap cache Sep 27 13:44:47 varnish-16 kernel: Swap cache stats: add 473619640, delete 469898786, find 188596514/233894822 Sep 27 13:44:47 varnish-16 kernel: Free swap = 216675600kB Sep 27 13:44:47 varnish-16 kernel: Total swap = 524287996kB Sep 27 13:44:47 varnish-16 kernel: 4194157 pages RAM Sep 27 13:44:47 varnish-16 kernel: 3720854 pages in swap cache Sep 27 13:44:47 varnish-16 kernel: Swap cache stats: add 473619640, delete 469898786, find 188596514/233894822 Sep 27 13:44:47 varnish-16 kernel: Free swap = 216675600kB Sep 27 13:44:47 varnish-16 kernel: Total swap = 524287996kB Sep 27 13:44:47 varnish-16 kernel: 4194157 pages RAM Sep 27 13:44:47 varnish-16 kernel: 0 pages HighMem/MovableOnly Sep 27 13:44:47 varnish-16 kernel: 90468 pages reserved Sep 27 13:44:47 varnish-16 kernel: 0 pages cma reserved Sep 27 13:44:47 varnish-16 kernel: 0 pages hwpoisoned Sep 27 13:44:47 varnish-16 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name Sep 27 13:44:47 varnish-16 kernel: [ 509] 0 509 10968 358 22 3 607 0 systemd-journal Sep 27 13:44:47 varnish-16 kernel: [ 546] 0 546 23692 130 16 3 51 0 lvmetad Sep 27 13:44:47 varnish-16 kernel: [ 557] 0 557 11194 307 22 3 327 -1000 systemd-udevd Sep 27 13:44:47 varnish-16 kernel: [ 639] 0 639 28742 364 49 3 320 0 vmtoolsd Sep 27 13:44:47 varnish-16 kernel: [ 703] 100 703 25080 235 19 3 85 0 systemd-timesyn Sep 27 13:44:47 varnish-16 kernel: [ 969] 0 969 6510 350 18 3 59 0 atd Sep 27 13:44:47 varnish-16 kernel: [ 983] 107 983 10721 208 28 3 119 -900 dbus-daemon Sep 27 13:44:47 varnish-16 kernel: [ 1010] 0 1010 6931 401 19 3 94 0 cron Sep 27 13:44:47 varnish-16 kernel: [ 1012] 104 1012 64097 156 28 3 324 0 rsyslogd Sep 27 13:44:47 varnish-16 kernel: [ 1013] 0 1013 21359 372 33 3 351 0 VGAuthService Sep 27 13:44:47 varnish-16 kernel: [ 1019] 0 1019 68649 76 37 3 211 0 accounts-daemon Sep 27 13:44:47 varnish-16 kernel: [ 1025] 0 1025 1098 288 8 3 35 0 acpid Sep 27 13:44:47 varnish-16 kernel: [ 1028] 0 1028 7164 366 18 3 86 0 systemd-logind Sep 27 13:44:47 varnish-16 kernel: [ 1038] 0 1038 40225 209 15 3 114 0 lxcfs Sep 27 13:44:47 varnish-16 kernel: [ 1100] 0 1100 3342 255 11 3 43 0 mdadm Sep 27 13:44:47 varnish-16 kernel: [ 1114] 0 1114 69271 223 40 3 180 0 polkitd Sep 27 13:44:47 varnish-16 kernel: [ 1224] 0 1224 1304 359 8 3 31 0 iscsid Sep 27 13:44:47 varnish-16 kernel: [ 1225] 0 1225 1429 879 8 3 0 -17 iscsid Sep 27 13:44:47 varnish-16 kernel: [ 1251] 0 1251 16377 348 36 3 190 -1000 sshd Sep 27 13:44:47 varnish-16 kernel: [ 1303] 0 1303 3663 312 13 3 39 0 agetty Sep 27 13:44:47 varnish-16 kernel: [ 1314] 0 1314 4903 314 15 3 125 0 irqbalance Sep 27 13:44:47 varnish-16 kernel: [ 1358] 999 1358 13541 0 29 2 11886 1000 netdata Sep 27 13:44:47 varnish-16 kernel: [121910] 0 121910 9067 422 19 3 113 0 varnishd Sep 27 13:44:47 varnish-16 kernel: [121932] 112 121932 86615131 7577 159392 334 76899838 0 cache-main Sep 27 13:44:47 varnish-16 kernel: [25236] 999 25236 1253 0 6 2 857 1000 apps.plugin Sep 27 13:44:47 varnish-16 kernel: [35166] 999 35166 406 0 5 2 91 1000 bash Sep 27 13:44:47 varnish-16 kernel: [44898] 999 44898 24372 337 35 3 3600 1000 python Sep 27 13:44:47 varnish-16 kernel: Out of memory: Kill process 1358 (netdata) score 1000 or sacrifice child Sep 27 13:44:47 varnish-16 kernel: Killed process 25236 (apps.plugin) total-vm:5012kB, anon-rss:0kB, file-rss:0kB ``` ``` varnishd (varnish-6.0.5 revision 3065ccaacc4bb537fb976a524bd808db42c5fe40) Copyright (c) 2006 Verdens Gang AS Copyright (c) 2006-2019 Varnish Software AS ``` Best regards. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Mon Sep 28 18:45:17 2020 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 28 Sep 2020 11:45:17 -0700 Subject: Varnish crashes periodically with high disk read and write In-Reply-To: References: Message-ID: Hi, This post should help understand a couple of points: https://info.varnish-software.com/blog/understanding-varnish-cache-memory-usage On top of this, you are probably using Transient: https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage -- Guillaume Quintard On Mon, Sep 28, 2020 at 1:40 AM Hamidreza Hosseini wrote: > > Hi > I'm using varnish-6.0.5, > 1. > 3nodes of varnish crashes every day and I should restart it else the whole > of my system doesnt work anymore > I don't know the reason but my disks at that time shows that It has 50 > MB/s (400Mb/s) Read and 48 MB/s write but my outbound network shows 120Mb/s > I put the link of Netdata picture at that time, > http://uupload.ir/files/c60r_varnish.png > How can I solve this problem? > > 2. > My next question is that I have another varnish that I set malloc to 68Gb > (the server has 86 Gb ram) but it will pass 68g till 86 and server crash, > Why does it exceed from 68g through I have adjusted the accurate and > permitted number? > > But really I'm confused and annoyed with first problem because it destroys > my cluster > > ``` > > Sep 27 13:46:19 varnish-16 varnishd[121910]: Error: Manager got SIGTERM > Sep 27 13:46:19 varnish-16 varnishd[121910]: Manager got SIGTERM > Sep 27 13:46:19 varnish-16 varnishd[121910]: Debug: Stopping Child > Sep 27 13:46:19 varnish-16 varnishd[121910]: Stopping Child > Sep 27 13:46:30 varnish-16 varnishd[121910]: Error: Child (121932) died > signal=15 > Sep 27 13:46:30 varnish-16 varnishd[121910]: Child (121932) died signal=15 > Sep 27 13:46:30 varnish-16 varnishd[121910]: Debug: Child cleanup complete > Sep 27 13:46:30 varnish-16 varnishd[121910]: Child cleanup complete > Sep 27 13:46:30 varnish-16 varnishd[121910]: Info: manager stopping child > Sep 27 13:46:30 varnish-16 varnishd[121910]: Info: manager dies > Sep 27 13:46:30 varnish-16 varnishd[121910]: manager stopping child > Sep 27 13:46:30 varnish-16 varnishd[121910]: manager dies > Sep 27 13:44:46 varnish-16 kernel: lowmem_reserve[]: 0 2934 15972 15972 > 15972 > Sep 27 13:44:46 varnish-16 kernel: Node 0 DMA32 free:55712kB min:2968kB > low:3708kB high:4452kB active_anon:0kB inactive_anon:2674740kB > active_file:104kB inactive_file:0kB unevictable:15528kB isolated(ano > Sep 27 13:44:47 varnish-16 kernel: lowmem_reserve[]: 0 0 13037 13037 13037 > Sep 27 13:44:47 varnish-16 kernel: Node 0 Normal free:21044kB min:13196kB > low:16492kB high:19792kB active_anon:492kB inactive_anon:12208096kB > active_file:548kB inactive_file:1300kB unevictable:70304kB is > Sep 27 13:44:47 varnish-16 kernel: lowmem_reserve[]: 0 0 0 0 0 > Sep 27 13:44:47 varnish-16 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB > (U) 1*32kB (E) 3*64kB (UE) 2*128kB (UE) 2*256kB (UE) 0*512kB 1*1024kB (U) > 2*2048kB (ME) 2*4096kB (M) = 14332kB > Sep 27 13:44:47 varnish-16 kernel: Node 0 DMA32: 7633*4kB (UME) 2869*8kB > (UMEH) 39*16kB (UM) 14*32kB (UM) 12*64kB (U) 2*128kB (U) 0*256kB 0*512kB > 0*1024kB 0*2048kB 0*4096kB = 55580kB > Sep 27 13:44:47 varnish-16 kernel: Node 0 Normal: 5132*4kB (UMEH) 69*8kB > (UMH) 1*16kB (H) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB > 0*4096kB = 21096kB > Sep 27 13:44:47 varnish-16 kernel: Node 0 hugepages_total=0 > hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB > Sep 27 13:44:47 varnish-16 kernel: 3742574 total pagecache pages > Sep 27 13:44:47 varnish-16 kernel: 3720854 pages in swap cache > Sep 27 13:44:47 varnish-16 kernel: Swap cache stats: add 473619640, delete > 469898786, find 188596514/233894822 > Sep 27 13:44:47 varnish-16 kernel: Free swap = 216675600kB > Sep 27 13:44:47 varnish-16 kernel: Total swap = 524287996kB > Sep 27 13:44:47 varnish-16 kernel: 4194157 pages RAM > Sep 27 13:44:47 varnish-16 kernel: 3720854 pages in swap cache > Sep 27 13:44:47 varnish-16 kernel: Swap cache stats: add 473619640, delete > 469898786, find 188596514/233894822 > Sep 27 13:44:47 varnish-16 kernel: Free swap = 216675600kB > Sep 27 13:44:47 varnish-16 kernel: Total swap = 524287996kB > Sep 27 13:44:47 varnish-16 kernel: 4194157 pages RAM > Sep 27 13:44:47 varnish-16 kernel: 0 pages HighMem/MovableOnly > Sep 27 13:44:47 varnish-16 kernel: 90468 pages reserved > Sep 27 13:44:47 varnish-16 kernel: 0 pages cma reserved > Sep 27 13:44:47 varnish-16 kernel: 0 pages hwpoisoned > Sep 27 13:44:47 varnish-16 kernel: [ pid ] uid tgid total_vm rss > nr_ptes nr_pmds swapents oom_score_adj name > Sep 27 13:44:47 varnish-16 kernel: [ 509] 0 509 10968 358 > 22 3 607 0 systemd-journal > Sep 27 13:44:47 varnish-16 kernel: [ 546] 0 546 23692 130 > 16 3 51 0 lvmetad > Sep 27 13:44:47 varnish-16 kernel: [ 557] 0 557 11194 307 > 22 3 327 -1000 systemd-udevd > Sep 27 13:44:47 varnish-16 kernel: [ 639] 0 639 28742 364 > 49 3 320 0 vmtoolsd > Sep 27 13:44:47 varnish-16 kernel: [ 703] 100 703 25080 235 > 19 3 85 0 systemd-timesyn > Sep 27 13:44:47 varnish-16 kernel: [ 969] 0 969 6510 350 > 18 3 59 0 atd > Sep 27 13:44:47 varnish-16 kernel: [ 983] 107 983 10721 208 > 28 3 119 -900 dbus-daemon > Sep 27 13:44:47 varnish-16 kernel: [ 1010] 0 1010 6931 401 > 19 3 94 0 cron > Sep 27 13:44:47 varnish-16 kernel: [ 1012] 104 1012 64097 156 > 28 3 324 0 rsyslogd > Sep 27 13:44:47 varnish-16 kernel: [ 1013] 0 1013 21359 372 > 33 3 351 0 VGAuthService > Sep 27 13:44:47 varnish-16 kernel: [ 1019] 0 1019 68649 76 > 37 3 211 0 accounts-daemon > Sep 27 13:44:47 varnish-16 kernel: [ 1025] 0 1025 1098 288 > 8 3 35 0 acpid > Sep 27 13:44:47 varnish-16 kernel: [ 1028] 0 1028 7164 366 > 18 3 86 0 systemd-logind > Sep 27 13:44:47 varnish-16 kernel: [ 1038] 0 1038 40225 209 > 15 3 114 0 lxcfs > Sep 27 13:44:47 varnish-16 kernel: [ 1100] 0 1100 3342 255 > 11 3 43 0 mdadm > Sep 27 13:44:47 varnish-16 kernel: [ 1114] 0 1114 69271 223 > 40 3 180 0 polkitd > Sep 27 13:44:47 varnish-16 kernel: [ 1224] 0 1224 1304 359 > 8 3 31 0 iscsid > Sep 27 13:44:47 varnish-16 kernel: [ 1225] 0 1225 1429 879 > 8 3 0 -17 iscsid > Sep 27 13:44:47 varnish-16 kernel: [ 1251] 0 1251 16377 348 > 36 3 190 -1000 sshd > Sep 27 13:44:47 varnish-16 kernel: [ 1303] 0 1303 3663 312 > 13 3 39 0 agetty > Sep 27 13:44:47 varnish-16 kernel: [ 1314] 0 1314 4903 314 > 15 3 125 0 irqbalance > Sep 27 13:44:47 varnish-16 kernel: [ 1358] 999 1358 13541 0 > 29 2 11886 1000 netdata > Sep 27 13:44:47 varnish-16 kernel: [121910] 0 121910 9067 422 > 19 3 113 0 varnishd > Sep 27 13:44:47 varnish-16 kernel: [121932] 112 121932 86615131 7577 > 159392 334 76899838 0 cache-main > Sep 27 13:44:47 varnish-16 kernel: [25236] 999 25236 1253 0 > 6 2 857 1000 apps.plugin > Sep 27 13:44:47 varnish-16 kernel: [35166] 999 35166 406 0 > 5 2 91 1000 bash > Sep 27 13:44:47 varnish-16 kernel: [44898] 999 44898 24372 337 > 35 3 3600 1000 python > Sep 27 13:44:47 varnish-16 kernel: Out of memory: Kill process 1358 > (netdata) score 1000 or sacrifice child > Sep 27 13:44:47 varnish-16 kernel: Killed process 25236 (apps.plugin) > total-vm:5012kB, anon-rss:0kB, file-rss:0kB > > > ``` > > ``` > > varnishd (varnish-6.0.5 revision 3065ccaacc4bb537fb976a524bd808db42c5fe40) > Copyright (c) 2006 Verdens Gang AS > Copyright (c) 2006-2019 Varnish Software AS > > ``` > > Best regards. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Tue Sep 29 14:44:12 2020 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Tue, 29 Sep 2020 07:44:12 -0700 Subject: Varnish 6.5 Docker image Message-ID: Hello everyone, As you surely know, Varnish 6.5 was released a couple of weeks ago, and we now have a docker image for it: https://hub.docker.com/_/varnish Note that it embeds the 6.5.1 version directly (we skipped 6.5.0). Please have a look at it, and let us know if you find any bugs, either in how we build the images (https://github.com/varnish/docker-varnish) or in the documentation ( https://github.com/docker-library/docs/tree/master/varnish). Feedback and pull requests are most welcome. Cheers! -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: From marianmvelez at gmail.com Tue Sep 29 20:08:08 2020 From: marianmvelez at gmail.com (Marian Velez) Date: Tue, 29 Sep 2020 17:08:08 -0300 Subject: Modifying req.url and restart() produces ncsa to log wrong URL resource Message-ID: Hiya all! I've been trying to implement an abtest engine on my varnish. The problem I've come across is related to the manipulation of the original req.url property. What I'm experiencing is that when setting a new "req.url" value and after forcing a restart(), the URL that varnishncsa uses to log is just the one it has set when it re-enters recv flow, but not the one it uses from the backend or the last req.URL object either (which will even have much more sense since). I've made a sample snippet of what I observe just for the sake of reproducing it and to take away all the abtest logic that may be just confusing. vcl 4.1; backend default { .host = "127.0.0.1"; .port = "8000"; } sub vcl_recv { if (req.restarts == 0) { set req.http.X-Original-URL = req.url; set req.url = "/this_url_does_not_exist"; return(restart); } else { #restore original URL set req.url = req.http.X-Original-URL; } } That vcl just sets an alternative req.url and triggers a flow restart. varnishncsa logs this entry ::1 - - [29/Sep/2020:16:49:08 -0300] "GET http://localhost:6081/this_url_does_not_exist HTTP/1.1" 200 12 "-" "curl/7.72.0" Of course that URL does not exist, but since its using the req.URL object from which varnish recv flow started, it just keeps that for logging, and not any later req.URL set. Here attached is the varnishlog output from a fresh start: * << Request >> 2 - Begin req 1 rxreq - Timestamp Start: 1601409664.461391 0.000000 0.000000 - Timestamp Req: 1601409664.461391 0.000000 0.000000 - VCL_use boot - ReqStart ::1 55180 a0 - ReqMethod GET - ReqURL /this_object_exists - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost:6081 - ReqHeader User-Agent: curl/7.72.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - ReqHeader X-Original-URL: /this_object_exists - ReqURL /this_url_does_not_exist - VCL_return restart - VCL_call HASH - VCL_return lookup - Timestamp Restart: 1601409664.461429 0.000037 0.000037 - Link req 3 restart - End * << BeReq >> 4 - Begin bereq 3 fetch - VCL_use boot - Timestamp Start: 1601409664.461553 0.000000 0.000000 - BereqMethod GET - BereqURL /this_object_exists - BereqProtocol HTTP/1.1 - BereqHeader User-Agent: curl/7.72.0 - BereqHeader Accept: */* - BereqHeader X-Forwarded-For: ::1 - BereqHeader host: localhost:6081 - BereqHeader Accept-Encoding: gzip - BereqHeader X-Varnish: 4 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 26 default 127.0.0.1 8000 127.0.0.1 59480 connect - Timestamp Bereq: 1601409664.462196 0.000642 0.000642 - Timestamp Beresp: 1601409664.463115 0.001561 0.000918 - BerespProtocol HTTP/1.0 - BerespStatus 200 - BerespReason OK - BerespHeader Server: SimpleHTTP/0.6 Python/3.8.6 - BerespHeader Date: Tue, 29 Sep 2020 20:01:04 GMT - BerespHeader Content-type: application/octet-stream - BerespHeader Content-Length: 12 - BerespHeader Last-Modified: Tue, 29 Sep 2020 19:36:31 GMT - TTL RFC 120 10 0 1601409664 1601409664 1601409664 0 0 cacheable - VCL_call BACKEND_RESPONSE - VCL_return deliver - Filters - Storage malloc s0 - Fetch_Body 3 length stream - BackendClose 26 default close - Timestamp BerespBody: 1601409664.463236 0.001682 0.000120 - Length 12 - BereqAcct 155 0 155 199 12 211 - End * << Request >> 3 - Begin req 2 restart - Timestamp Start: 1601409664.461429 0.000037 0.000000 - ReqStart ::1 55180 a0 - ReqMethod GET - ReqURL /this_url_does_not_exist - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost:6081 - ReqHeader User-Agent: curl/7.72.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - ReqHeader X-Original-URL: /this_object_exists - VCL_call RECV - ReqURL /this_object_exists - ReqUnset X-Original-URL: /this_object_exists - ReqUnset Host: localhost:6081 - ReqHeader host: localhost:6081 - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 4 fetch - Timestamp Fetch: 1601409664.463212 0.001821 0.001783 - RespProtocol HTTP/1.0 - RespStatus 200 - RespReason OK - RespHeader Server: SimpleHTTP/0.6 Python/3.8.6 - RespHeader Date: Tue, 29 Sep 2020 20:01:04 GMT - RespHeader Content-type: application/octet-stream - RespHeader Content-Length: 12 - RespHeader Last-Modified: Tue, 29 Sep 2020 19:36:31 GMT - RespProtocol HTTP/1.1 - RespHeader X-Varnish: 3 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish (Varnish/6.5) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1601409664.463225 0.001833 0.000012 - Filters - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1601409664.463286 0.001894 0.000061 - ReqAcct 96 0 96 299 12 311 - End * << Session >> 1 - Begin sess 0 HTTP/1 - SessOpen ::1 55180 a0 ::1 6081 1601409664.461310 18 - Link req 2 rxreq - SessClose REM_CLOSE 0.002 - End As you can see from the FETCH block, what it effectively fetches from backend is not what varnishncsa logs. I think it has something to do with the order in which the restart affects the flow, since its not replacing the req.url object, just appending a new one. BACKEND FETCH uses the right URL object since it uses the last one, but varnishncsa logs the first it picked up from that restart. I've tried setting a blank req.url or unsetting in an attempt to avoid having several req.URL objects introducing problems, but it seems that property is readonly since I get errors like "Variable cannot be unset" or "Setting req.url to empty sting - FAILED". Unfortunately I have no other way to implement my abtest engine than by using the restart() order from varnish, but this logging issue is extremely affecting what I can see to debug my stuff. Any clue of what trick I could use to workaround this? Thanks in advance! -------------- next part -------------- An HTML attachment was scrubbed... URL: From lampe at hauke-lampe.de Wed Sep 30 20:52:48 2020 From: lampe at hauke-lampe.de (Hauke Lampe) Date: Wed, 30 Sep 2020 22:52:48 +0200 Subject: varnish crush every 30 minutes in high pressure In-Reply-To: References: Message-ID: <1f93b016-ca41-7bc9-8882-3f2a578c5fc3@hauke-lampe.de> On 2020-09-26 12:39, Hamidreza Hosseini wrote: > Im using varnish , before i hadn't any problem in varnish but today after crushing my dns (I fixed it now) and in high pressure , it > restart whole machine almost every 30 minutes It could be this: when the filesystem that contains the shared memory files comes under I/O load, communication through the shared memory can stall. I'd recommend putting them on a tmpfs/RAM-disk (and/or modify the "dirty-background-bytes" system setting so the OS doesn't accumulate large amounts of dirty cached data).