From arunabha.saha at gmail.com Thu Mar 4 04:22:10 2021 From: arunabha.saha at gmail.com (Arunabha Saha) Date: Wed, 3 Mar 2021 20:22:10 -0800 Subject: Accessing original object in varnish background Fetch Message-ID: Hello, Something I am trying to do is update an existing value in a cached object after every background fetch. I can't seem to figure out how to access the existing object parameters during the background fetch. Example vcl_backend_response { if (!bereq.is_bgfetch) { set beresp.http.myval = beresp.http.val_from_backend; } else { # #Here beresp.http.myval (on RHS of assignment expression). # should be the original hdr value stored with the object but i can't seem to access it this way # or any other way. # set beresp.http.myval = beresp.http.myval + beresp.http.val_from_backend; } } -- regards, Arun From guillaume at varnish-software.com Thu Mar 4 04:42:33 2021 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 3 Mar 2021 20:42:33 -0800 Subject: Accessing original object in varnish background Fetch In-Reply-To: References: Message-ID: Hi, This expectation is wrong: # Here beresp.http.myval (on RHS of assignment expression). # should be the original hdr value stored with the object beresp is the new backend response, and VCL doesn't make the old one available. There are two ways to deal with this. The first one is vmod_stale ( https://docs.varnish-software.com/varnish-cache-plus/vmods/stale/#get-header) in Varnish Enterprise that will allow you to get direct access to that header. The second one, if you want to stick to open source, uses vcl_hit to stash the old headers in req so that they are available in bereq. That would look something like this (untested code, use with caution): sub vcl_recv { # avoid injections from the clients unset req.http.myval; } sub vcl_hit { # store the header set req.http.myval = obj.http.myval; } sub vcl_backend_response { # bereq is basically a req copy, so myval crossed over # note that I do that unconditionally here, but you can indeed check bereq.is_bg_fetch set beresp.http.myval = bereq.http.myval + " extra string"; } that should do the trick, however, be careful, the code above will add more text to the header at every background fetch, so you are more than likely to hit the header length limit if you are not careful. Does that make sense? -- Guillaume Quintard On Wed, Mar 3, 2021 at 8:23 PM Arunabha Saha wrote: > Hello, > Something I am trying to do is update an existing value in a > cached object after every background fetch. I can't seem to figure > out how to access the existing object parameters during the background > fetch. > > Example > > vcl_backend_response { > if (!bereq.is_bgfetch) { > set beresp.http.myval = beresp.http.val_from_backend; > } else { > # > #Here beresp.http.myval (on RHS of assignment expression). > # should be the original hdr value stored with the object > but i can't seem to access it this way > # or any other way. > # > set beresp.http.myval = beresp.http.myval + > beresp.http.val_from_backend; > } > } > > -- > regards, > Arun > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arunabha.saha at gmail.com Thu Mar 4 06:08:22 2021 From: arunabha.saha at gmail.com (Arunabha Saha) Date: Wed, 3 Mar 2021 22:08:22 -0800 Subject: Accessing original object in varnish background Fetch In-Reply-To: References: Message-ID: Thank you. This(open source version) worked very well for me. The part in vcl_hit was what i wasn't aware of. To clarify the header value returned is a number and the value stored back into the object is an average of the existing header value and the number number obtained from upstream so the length of the header value should not be a real concern. On Wed, Mar 3, 2021 at 8:42 PM Guillaume Quintard wrote: > > Hi, > > This expectation is wrong: > # Here beresp.http.myval (on RHS of assignment expression). > # should be the original hdr value stored with the object > > beresp is the new backend response, and VCL doesn't make the old one available. > > There are two ways to deal with this. > > The first one is vmod_stale (https://docs.varnish-software.com/varnish-cache-plus/vmods/stale/#get-header) in Varnish Enterprise that will allow you to get direct access to that header. > > The second one, if you want to stick to open source, uses vcl_hit to stash the old headers in req so that they are available in bereq. That would look something like this (untested code, use with caution): > > sub vcl_recv { > # avoid injections from the clients > unset req.http.myval; > } > > sub vcl_hit { > # store the header > set req.http.myval = obj.http.myval; > } > > sub vcl_backend_response { > # bereq is basically a req copy, so myval crossed over > > # note that I do that unconditionally here, but you can indeed check bereq.is_bg_fetch > > set beresp.http.myval = bereq.http.myval + " extra string"; > } > > > that should do the trick, however, be careful, the code above will add more text to the header at every background fetch, so you are more than likely to hit the header length limit if you are not careful. > > Does that make sense? > > -- > Guillaume Quintard > > > On Wed, Mar 3, 2021 at 8:23 PM Arunabha Saha wrote: >> >> Hello, >> Something I am trying to do is update an existing value in a >> cached object after every background fetch. I can't seem to figure >> out how to access the existing object parameters during the background >> fetch. >> >> Example >> >> vcl_backend_response { >> if (!bereq.is_bgfetch) { >> set beresp.http.myval = beresp.http.val_from_backend; >> } else { >> # >> #Here beresp.http.myval (on RHS of assignment expression). >> # should be the original hdr value stored with the object >> but i can't seem to access it this way >> # or any other way. >> # >> set beresp.http.myval = beresp.http.myval + >> beresp.http.val_from_backend; >> } >> } >> >> -- >> regards, >> Arun >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -- regards, Arun From martynas at atomgraph.com Wed Mar 10 20:30:57 2021 From: martynas at atomgraph.com (=?UTF-8?Q?Martynas_Jusevi=C4=8Dius?=) Date: Wed, 10 Mar 2021 21:30:57 +0100 Subject: Varnish cache MISS Message-ID: Hi, one again I'm mystified as to why responses from the backend are not cached -- or rather, why only some of them are. At the bottom you can find a log with 2 requests, one after another -- 1st one miss, 2nd one hit. The only difference I can see is that the 1st one contains an Authorization header. I know that by default Varnish doesn't cache authenticated interactions, but I changed my VCL to specifically allow that (I think): sub vcl_recv { ... if (req.http.Cookie) { // allow req.http.Authorization /* Not cacheable by default */ return (pass); } return (hash); } And this configuration works on a different host -- I can see requests even with Authorization being hit and delivered. But not on this one... What am I missing? I'm at a loss. === varnishadmi log === * << Request >> 470 - Begin req 385 rxreq - Timestamp Start: 1615407572.284968 0.000000 0.000000 - Timestamp Req: 1615407572.284968 0.000000 0.000000 - ReqStart 172.26.0.6 33016 - ReqMethod GET - ReqURL /smth/root-admin-prod/sparql?%24Ontology=%3Chttps%3A%2F%2Fsmth.com%2Fadmin%2Fns%23%3E&%24Mode=%3Chttp%3A%2F%2Fwww.w3.org%2Fns%2Fauth%2Facl%23Read%3E&%24AuthenticatedAgentClass=%3Chttp%3A%2F%2Fwww.w3.org%2F2000%2F01%2Frdf-schema%23Resour - ReqProtocol HTTP/1.1 - ReqHeader Authorization: Basic **************************************** - ReqHeader Accept: application/rdf+xml;q=0.8,application/n-triples;q=0.9,application/trix;q=0.8,application/ld+json;q=0.8,application/rdf+thrift,text/turtle;q=0.8,text/rdf+n3;q=0.8,application/rdf+json;q=0.8 - ReqHeader User-Agent: Jersey/2.30.1 (Apache HttpClient 4.5.10) - ReqHeader Host: atomgraph.smth.varnish - ReqHeader Connection: Keep-Alive - ReqHeader Accept-Encoding: gzip,deflate - ReqHeader X-Forwarded-For: 172.26.0.6 - VCL_call RECV - VCL_return hash - ReqUnset Accept-Encoding: gzip,deflate - ReqHeader Accept-Encoding: gzip - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 471 fetch - Timestamp Fetch: 1615407572.305271 0.020303 0.020303 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx/1.10.3 (Ubuntu) - RespHeader Date: Wed, 10 Mar 2021 20:19:32 GMT - RespHeader Content-Type: application/n-triples; charset=UTF-8 - RespHeader Last-Modified: Wed, 10 Mar 2021 20:19:32 GMT - RespHeader Etag: 7f4f7c54-96d7-114d-bbe6-6a7cac14f19a - RespHeader Vary: Accept, Accept-Datetime, Accept-Encoding, Origin, Revision - RespHeader Cache-Control: private - RespHeader Request-Id: ECA4A374-81DD-11EB-87DD-901B0E95D742 - RespHeader Strict-Transport-Security: max-age=31536000 - RespHeader Client-Request-Id: 69f411cc-3616-4dea-a42a-423dae6d79f9 - RespHeader Access-Control-Max-Age: 86400 - RespHeader Access-Control-Allow-Credentials: true - RespHeader Access-Control-Allow-Origin: atomgraph.smth.com - RespHeader Access-Control-Allow-Headers: Accept, Accept-Asynchronous, Accept-Datetime, Asynchronous-Content-Type, Asynchronous-Location, Asynchronous-Method, Authorization, Content-Encoding, Content-Type, Graph, Introspection-Content-Type, Link, Location, Revision, - RespHeader Access-Control-Expose-Headers: * - RespHeader X-Varnish: 470 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish (Varnish/5.2) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1615407572.305315 0.020348 0.000044 - RespHeader Accept-Ranges: bytes - RespHeader Content-Length: 0 - RespHeader Connection: keep-alive - Timestamp Resp: 1615407572.305420 0.020452 0.000104 - ReqAcct 4868 0 4868 1020 0 1020 - End * << Request >> 33273 - Begin req 33229 rxreq - Timestamp Start: 1615407574.651147 0.000000 0.000000 - Timestamp Req: 1615407574.651147 0.000000 0.000000 - ReqStart 172.26.0.6 33014 - ReqMethod GET - ReqURL /smth/contexts-prod/sparql?auth_token=*************************&%24this=%3Chttps%3A%2F%2Fsmth.com%2Fadmin%2Facl%2Fagents%2F678e3f45-c9b0-4673-9cc7-217d7804e7e4%2F%3E&query=BASE%20%20%20%20%3Chttps%3A%2F%2Fsmth.com%2F%3E%0APREFIX%20% - ReqProtocol HTTP/1.1 - ReqHeader Accept: application/rdf+xml;q=0.8,application/n-triples;q=0.9,application/trix;q=0.8,application/ld+json;q=0.8,application/rdf+thrift,text/turtle;q=0.8,text/rdf+n3;q=0.8,application/rdf+json;q=0.8 - ReqHeader User-Agent: Jersey/2.30.1 (Apache HttpClient 4.5.10) - ReqHeader Host: atomgraph.smth.varnish - ReqHeader Connection: Keep-Alive - ReqHeader Accept-Encoding: gzip,deflate - ReqHeader X-Forwarded-For: 172.26.0.6 - VCL_call RECV - VCL_return hash - ReqUnset Accept-Encoding: gzip,deflate - ReqHeader Accept-Encoding: gzip - VCL_call HASH - VCL_return lookup - Hit 98307 85213.183949 10.000000 0.000000 - VCL_call HIT - VCL_return deliver - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx/1.10.3 (Ubuntu) - RespHeader Date: Wed, 10 Mar 2021 19:59:47 GMT - RespHeader Content-Type: application/n-triples; charset=UTF-8 - RespHeader Last-Modified: Wed, 10 Mar 2021 19:59:47 GMT - RespHeader Etag: a69bec4d-2a04-804c-8b79-16a8e598e9b8 - RespHeader Vary: Accept, Accept-Datetime, Accept-Encoding, Origin, Revision - RespHeader Cache-Control: private - RespHeader Request-Id: 2A365E1A-81DB-11EB-87DD-901B0E95D742 - RespHeader Strict-Transport-Security: max-age=31536000 - RespHeader Access-Control-Max-Age: 86400 - RespHeader Access-Control-Allow-Credentials: true - RespHeader Access-Control-Allow-Origin: * - RespHeader Access-Control-Allow-Headers: Accept, Accept-Asynchronous, Accept-Datetime, Asynchronous-Content-Type, Asynchronous-Location, Asynchronous-Method, Authorization, Content-Encoding, Content-Type, Graph, Introspection-Content-Type, Link, Location, Revision, - RespHeader Access-Control-Expose-Headers: * - RespHeader X-Varnish: 33273 98307 - RespHeader Age: 1186 - RespHeader Via: 1.1 varnish (Varnish/5.2) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1615407574.651280 0.000133 0.000133 - RespHeader Accept-Ranges: bytes - RespHeader Content-Length: 4454 - RespHeader Connection: keep-alive - Timestamp Resp: 1615407574.651352 0.000205 0.000072 - ReqAcct 1961 0 1961 959 4454 5413 - End From martynas at atomgraph.com Wed Mar 10 21:07:15 2021 From: martynas at atomgraph.com (=?UTF-8?Q?Martynas_Jusevi=C4=8Dius?=) Date: Wed, 10 Mar 2021 22:07:15 +0100 Subject: Varnish cache MISS In-Reply-To: References: Message-ID: Nevermind :) I realized the URLs in this log are truncated and the first one contains a unique ID in it... On Wed, Mar 10, 2021 at 9:30 PM Martynas Jusevi?ius wrote: > > Hi, > > one again I'm mystified as to why responses from the backend are not > cached -- or rather, why only some of them are. > > At the bottom you can find a log with 2 requests, one after another -- > 1st one miss, 2nd one hit. > The only difference I can see is that the 1st one contains an > Authorization header. I know that by default Varnish doesn't cache > authenticated interactions, but I changed my VCL to specifically allow > that (I think): > > sub vcl_recv { > ... > > if (req.http.Cookie) { // allow req.http.Authorization > /* Not cacheable by default */ > return (pass); > } > return (hash); > } > > And this configuration works on a different host -- I can see requests > even with Authorization being hit and delivered. > > But not on this one... What am I missing? I'm at a loss. > > > === varnishadmi log === > > * << Request >> 470 > - Begin req 385 rxreq > - Timestamp Start: 1615407572.284968 0.000000 0.000000 > - Timestamp Req: 1615407572.284968 0.000000 0.000000 > - ReqStart 172.26.0.6 33016 > - ReqMethod GET > - ReqURL > /smth/root-admin-prod/sparql?%24Ontology=%3Chttps%3A%2F%2Fsmth.com%2Fadmin%2Fns%23%3E&%24Mode=%3Chttp%3A%2F%2Fwww.w3.org%2Fns%2Fauth%2Facl%23Read%3E&%24AuthenticatedAgentClass=%3Chttp%3A%2F%2Fwww.w3.org%2F2000%2F01%2Frdf-schema%23Resour > - ReqProtocol HTTP/1.1 > - ReqHeader Authorization: Basic **************************************** > - ReqHeader Accept: > application/rdf+xml;q=0.8,application/n-triples;q=0.9,application/trix;q=0.8,application/ld+json;q=0.8,application/rdf+thrift,text/turtle;q=0.8,text/rdf+n3;q=0.8,application/rdf+json;q=0.8 > - ReqHeader User-Agent: Jersey/2.30.1 (Apache HttpClient 4.5.10) > - ReqHeader Host: atomgraph.smth.varnish > - ReqHeader Connection: Keep-Alive > - ReqHeader Accept-Encoding: gzip,deflate > - ReqHeader X-Forwarded-For: 172.26.0.6 > - VCL_call RECV > - VCL_return hash > - ReqUnset Accept-Encoding: gzip,deflate > - ReqHeader Accept-Encoding: gzip > - VCL_call HASH > - VCL_return lookup > - VCL_call MISS > - VCL_return fetch > - Link bereq 471 fetch > - Timestamp Fetch: 1615407572.305271 0.020303 0.020303 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespHeader Server: nginx/1.10.3 (Ubuntu) > - RespHeader Date: Wed, 10 Mar 2021 20:19:32 GMT > - RespHeader Content-Type: application/n-triples; charset=UTF-8 > - RespHeader Last-Modified: Wed, 10 Mar 2021 20:19:32 GMT > - RespHeader Etag: 7f4f7c54-96d7-114d-bbe6-6a7cac14f19a > - RespHeader Vary: Accept, Accept-Datetime, Accept-Encoding, > Origin, Revision > - RespHeader Cache-Control: private > - RespHeader Request-Id: ECA4A374-81DD-11EB-87DD-901B0E95D742 > - RespHeader Strict-Transport-Security: max-age=31536000 > - RespHeader Client-Request-Id: 69f411cc-3616-4dea-a42a-423dae6d79f9 > - RespHeader Access-Control-Max-Age: 86400 > - RespHeader Access-Control-Allow-Credentials: true > - RespHeader Access-Control-Allow-Origin: atomgraph.smth.com > - RespHeader Access-Control-Allow-Headers: Accept, > Accept-Asynchronous, Accept-Datetime, Asynchronous-Content-Type, > Asynchronous-Location, Asynchronous-Method, Authorization, > Content-Encoding, Content-Type, Graph, Introspection-Content-Type, > Link, Location, Revision, > - RespHeader Access-Control-Expose-Headers: * > - RespHeader X-Varnish: 470 > - RespHeader Age: 0 > - RespHeader Via: 1.1 varnish (Varnish/5.2) > - VCL_call DELIVER > - VCL_return deliver > - Timestamp Process: 1615407572.305315 0.020348 0.000044 > - RespHeader Accept-Ranges: bytes > - RespHeader Content-Length: 0 > - RespHeader Connection: keep-alive > - Timestamp Resp: 1615407572.305420 0.020452 0.000104 > - ReqAcct 4868 0 4868 1020 0 1020 > - End > > * << Request >> 33273 > - Begin req 33229 rxreq > - Timestamp Start: 1615407574.651147 0.000000 0.000000 > - Timestamp Req: 1615407574.651147 0.000000 0.000000 > - ReqStart 172.26.0.6 33014 > - ReqMethod GET > - ReqURL > /smth/contexts-prod/sparql?auth_token=*************************&%24this=%3Chttps%3A%2F%2Fsmth.com%2Fadmin%2Facl%2Fagents%2F678e3f45-c9b0-4673-9cc7-217d7804e7e4%2F%3E&query=BASE%20%20%20%20%3Chttps%3A%2F%2Fsmth.com%2F%3E%0APREFIX%20% > - ReqProtocol HTTP/1.1 > - ReqHeader Accept: > application/rdf+xml;q=0.8,application/n-triples;q=0.9,application/trix;q=0.8,application/ld+json;q=0.8,application/rdf+thrift,text/turtle;q=0.8,text/rdf+n3;q=0.8,application/rdf+json;q=0.8 > - ReqHeader User-Agent: Jersey/2.30.1 (Apache HttpClient 4.5.10) > - ReqHeader Host: atomgraph.smth.varnish > - ReqHeader Connection: Keep-Alive > - ReqHeader Accept-Encoding: gzip,deflate > - ReqHeader X-Forwarded-For: 172.26.0.6 > - VCL_call RECV > - VCL_return hash > - ReqUnset Accept-Encoding: gzip,deflate > - ReqHeader Accept-Encoding: gzip > - VCL_call HASH > - VCL_return lookup > - Hit 98307 85213.183949 10.000000 0.000000 > - VCL_call HIT > - VCL_return deliver > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespHeader Server: nginx/1.10.3 (Ubuntu) > - RespHeader Date: Wed, 10 Mar 2021 19:59:47 GMT > - RespHeader Content-Type: application/n-triples; charset=UTF-8 > - RespHeader Last-Modified: Wed, 10 Mar 2021 19:59:47 GMT > - RespHeader Etag: a69bec4d-2a04-804c-8b79-16a8e598e9b8 > - RespHeader Vary: Accept, Accept-Datetime, Accept-Encoding, > Origin, Revision > - RespHeader Cache-Control: private > - RespHeader Request-Id: 2A365E1A-81DB-11EB-87DD-901B0E95D742 > - RespHeader Strict-Transport-Security: max-age=31536000 > - RespHeader Access-Control-Max-Age: 86400 > - RespHeader Access-Control-Allow-Credentials: true > - RespHeader Access-Control-Allow-Origin: * > - RespHeader Access-Control-Allow-Headers: Accept, > Accept-Asynchronous, Accept-Datetime, Asynchronous-Content-Type, > Asynchronous-Location, Asynchronous-Method, Authorization, > Content-Encoding, Content-Type, Graph, Introspection-Content-Type, > Link, Location, Revision, > - RespHeader Access-Control-Expose-Headers: * > - RespHeader X-Varnish: 33273 98307 > - RespHeader Age: 1186 > - RespHeader Via: 1.1 varnish (Varnish/5.2) > - VCL_call DELIVER > - VCL_return deliver > - Timestamp Process: 1615407574.651280 0.000133 0.000133 > - RespHeader Accept-Ranges: bytes > - RespHeader Content-Length: 4454 > - RespHeader Connection: keep-alive > - Timestamp Resp: 1615407574.651352 0.000205 0.000072 > - ReqAcct 1961 0 1961 959 4454 5413 > - End From dridi at varni.sh Wed Mar 10 21:17:26 2021 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 10 Mar 2021 21:17:26 +0000 Subject: Varnish cache MISS In-Reply-To: References: Message-ID: On Wed, Mar 10, 2021 at 9:09 PM Martynas Jusevi?ius wrote: > > Nevermind :) I realized the URLs in this log are truncated and the > first one contains a unique ID in it... You might also want to pay attention to backend response headers: - RespHeader Cache-Control: private If this cache-control came from the backend then there's a fair chance that it made the response uncacheable. Dridi From martynas at atomgraph.com Wed Mar 10 21:20:15 2021 From: martynas at atomgraph.com (=?UTF-8?Q?Martynas_Jusevi=C4=8Dius?=) Date: Wed, 10 Mar 2021 22:20:15 +0100 Subject: Varnish cache MISS In-Reply-To: References: Message-ID: Yes, that could be a problem, but I think I sorted this by using On Wed, Mar 10, 2021 at 10:18 PM Dridi Boukelmoune wrote: > > On Wed, Mar 10, 2021 at 9:09 PM Martynas Jusevi?ius > wrote: > > > > Nevermind :) I realized the URLs in this log are truncated and the > > first one contains a unique ID in it... > > You might also want to pay attention to backend response headers: > > - RespHeader Cache-Control: private > > If this cache-control came from the backend then there's a fair chance > that it made the response uncacheable. > > Dridi From martynas at atomgraph.com Wed Mar 10 21:20:35 2021 From: martynas at atomgraph.com (=?UTF-8?Q?Martynas_Jusevi=C4=8Dius?=) Date: Wed, 10 Mar 2021 22:20:35 +0100 Subject: Varnish cache MISS In-Reply-To: References: Message-ID: ...by using sub vcl_backend_response { return (deliver); } On Wed, Mar 10, 2021 at 10:20 PM Martynas Jusevi?ius wrote: > > Yes, that could be a problem, but I think I sorted this by using > > On Wed, Mar 10, 2021 at 10:18 PM Dridi Boukelmoune wrote: > > > > On Wed, Mar 10, 2021 at 9:09 PM Martynas Jusevi?ius > > wrote: > > > > > > Nevermind :) I realized the URLs in this log are truncated and the > > > first one contains a unique ID in it... > > > > You might also want to pay attention to backend response headers: > > > > - RespHeader Cache-Control: private > > > > If this cache-control came from the backend then there's a fair chance > > that it made the response uncacheable. > > > > Dridi From G.Bertels at circit.de Fri Mar 19 09:03:03 2021 From: G.Bertels at circit.de (G.Bertels at circit.de) Date: Fri, 19 Mar 2021 09:03:03 +0000 Subject: Varnish memory consumption grows infinitely Message-ID: <129A7B7F-6797-4D38-8A9A-06A8B363B650@contoso.com> Hello, we are using Varnish 6.0.9 with Red Hat Enterprise Linux 8.3 and I notice that the Varnish consumes more memory than allocated to it. The Varnish is started with the following parameters: ExecStart=/usr/sbin/varnish\ -P /var/run/varnish.pid \ -f /etc/varnish/default.vcl \ -a :6081 -T 127.0.0.1:6082 \ -S /etc/varnish/secret \ -s Transient=malloc,8G \ -s malloc,48G \ -p http_req_hdr_len=16k \ -p timeout_idle=10 \ -p workspace_client=256k \ -p workspace_backend=256k The virtual memory and RAM consumption keeps growing and I have to restart the process every few days. My search on the internet shows some results that a few also have the problem, but no solution. It looks like RedHat does not use jemalloc (https://bugzilla.redhat.com/show_bug.cgi?id=1656034) ldd /usr/sbin/varnishd linux-vdso.so.1 (0x00007ffe365ea000) libpcre.so.1 => /lib64/libpcre.so.1 (0x00007ff6869f2000) libdl.so.2 => /lib64/libdl.so.2 (0x00007ff6867ee000) librt.so.1 => /lib64/librt.so.1 (0x00007ff6865e6000) libm.so.6 => /lib64/libm.so.6 (0x00007ff686264000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff686044000) libc.so.6 => /lib64/libc.so.6 (0x00007ff685c81000) /lib64/ld-linux-x86-64.so.2 (0x00007ff686f99000) I am at a loss as to how to proceed. Because restarting the Varnish every few days can't be the solution and I hope there is a way to solve the problem. Regards G?nter ________________________________ circ IT GmbH & Co KG ? 40549 D?sseldorf Sitz D?sseldorf ? Amtsgericht D?sseldorf HRA 19974 Pers?nlich haftender Gesellschafter: circ IT Verwaltungs-GmbH (AG D?sseldorf HRB 58609) Gesch?ftsf?hrer: Michael Staade -------------- next part -------------- An HTML attachment was scrubbed... URL: From G.Bertels at circit.de Fri Mar 19 11:33:24 2021 From: G.Bertels at circit.de (G.Bertels at circit.de) Date: Fri, 19 Mar 2021 11:33:24 +0000 Subject: Varnish memory consumption grows infinitely In-Reply-To: <129A7B7F-6797-4D38-8A9A-06A8B363B650@contoso.com> References: <129A7B7F-6797-4D38-8A9A-06A8B363B650@contoso.com> Message-ID: Sorry, i mean Varnish 6.0.6 Von: varnish-misc im Auftrag von "Bertels, G?nter" Datum: Freitag, 19. M?rz 2021 um 10:05 An: "varnish-misc at varnish-cache.org" Betreff: Varnish memory consumption grows infinitely Hello, we are using Varnish 6.0.9 with Red Hat Enterprise Linux 8.3 and I notice that the Varnish consumes more memory than allocated to it. The Varnish is started with the following parameters: ExecStart=/usr/sbin/varnish\ -P /var/run/varnish.pid \ -f /etc/varnish/default.vcl \ -a :6081 -T 127.0.0.1:6082 \ -S /etc/varnish/secret \ -s Transient=malloc,8G \ -s malloc,48G \ -p http_req_hdr_len=16k \ -p timeout_idle=10 \ -p workspace_client=256k \ -p workspace_backend=256k The virtual memory and RAM consumption keeps growing and I have to restart the process every few days. My search on the internet shows some results that a few also have the problem, but no solution. It looks like RedHat does not use jemalloc (https://bugzilla.redhat.com/show_bug.cgi?id=1656034) ldd /usr/sbin/varnishd linux-vdso.so.1 (0x00007ffe365ea000) libpcre.so.1 => /lib64/libpcre.so.1 (0x00007ff6869f2000) libdl.so.2 => /lib64/libdl.so.2 (0x00007ff6867ee000) librt.so.1 => /lib64/librt.so.1 (0x00007ff6865e6000) libm.so.6 => /lib64/libm.so.6 (0x00007ff686264000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ff686044000) libc.so.6 => /lib64/libc.so.6 (0x00007ff685c81000) /lib64/ld-linux-x86-64.so.2 (0x00007ff686f99000) I am at a loss as to how to proceed. Because restarting the Varnish every few days can't be the solution and I hope there is a way to solve the problem. Regards G?nter ________________________________ circ IT GmbH & Co KG ? 40549 D?sseldorf Sitz D?sseldorf ? Amtsgericht D?sseldorf HRA 19974 Pers?nlich haftender Gesellschafter: circ IT Verwaltungs-GmbH (AG D?sseldorf HRB 58609) Gesch?ftsf?hrer: Michael Staade ________________________________ circ IT GmbH & Co KG ? 40549 D?sseldorf Sitz D?sseldorf ? Amtsgericht D?sseldorf HRA 19974 Pers?nlich haftender Gesellschafter: circ IT Verwaltungs-GmbH (AG D?sseldorf HRB 58609) Gesch?ftsf?hrer: Michael Staade -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Tue Mar 23 04:16:20 2021 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 22 Mar 2021 21:16:20 -0700 Subject: Varnish 6.6 packages and images Message-ID: Hi everyone, As I'm sure most of you already know, Varnish 6.6 was released last week ( https://varnish-cache.org/releases/index.html), and this week we have packages (https://packagecloud.io/varnishcache) and docker images ( https://hub.docker.com/_/varnish). We are aware that the docker image was mistagged (6.5 in addition of 6.6) and this is being worked on. Let us know if you have questions! -- Guillaume Quintard -------------- next part -------------- An HTML attachment was scrubbed... URL: