From guillaume at varnish-software.com Thu May 3 05:03:15 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 03 May 2018 05:03:15 +0000 Subject: Query regarding Varnish modules for parsing XML body In-Reply-To: References: Message-ID: Hi Deepak, Currently, no, no way to parse the response body. But for the request body, you have the bodyaccess vmod -- Guillaume Quintard On Mon, Apr 30, 2018, 22:39 Deepak Verma wrote: > Hi, > > > > I am trying to see if Varnish as a reverse proxy cache is suitable for my > use case. Are there any Varnish modules available which can help me parse > XML data present in POST response body. > > > > Regards, > > Deepak > NOTICE TO RECIPIENT: This email, including attachments, may contain > information which is confidential, proprietary, attorney-client privileged > and / or controlled under U.S. export laws and regulations and may be > restricted from disclosure by applicable State and Federal law. Nothing in > this email shall create any legal binding agreement between the parties > unless expressly stated herein and provided by an authorized representative > of Comtech Telecommunications Corp. or its subsidiaries. If you are not the > intended recipient of this message, be advised that any dissemination, > distribution, or use of the contents of this message is strictly > prohibited. If you received this message in error, please notify us > immediately by return email and permanently delete all copies of the > original email and any attached documentation from any computer or other > media. > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From hetardik.p at gmail.com Thu May 3 06:16:55 2018 From: hetardik.p at gmail.com (Hardik Prajapati) Date: Thu, 3 May 2018 11:46:55 +0530 Subject: High swap usage in varnish 5.2.1 Message-ID: Hi Team, Please find specs for systems on which varnish 5.2.1 is running. RAM - 96GB Disk - 30TB Param setting, -------------------- thread_pool_max 15000 [threads] thread_pool_min 5000 [threads] Problem is high (almost 100%) swap usage. Can you please suggest correct thread value ? #]grep Vm /proc/20676/status VmPeak: 13334288076 kB VmSize: 13334152500 kB VmLck: 876 kB VmPin: 0 kB VmHWM: 77463128 kB VmRSS: 67573800 kB VmData: 65535344 kB VmStk: 132 kB VmExe: 1076 kB VmLib: 146172 kB VmPTE: 714076 kB VmSwap: 2024936 kB Thank you Hardik From guillaume at varnish-software.com Thu May 3 06:21:59 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 03 May 2018 06:21:59 +0000 Subject: High swap usage in varnish 5.2.1 In-Reply-To: References: Message-ID: Hi, Are you using the file storage? -- Guillaume Quintard On Thu, May 3, 2018, 08:17 Hardik Prajapati wrote: > Hi Team, > > Please find specs for systems on which varnish 5.2.1 is running. > > RAM - 96GB > Disk - 30TB > > Param setting, > -------------------- > thread_pool_max 15000 [threads] > > thread_pool_min 5000 [threads] > > Problem is high (almost 100%) swap usage. Can you please suggest > correct thread value ? > > > #]grep Vm /proc/20676/status > > VmPeak: 13334288076 kB > > VmSize: 13334152500 kB > > VmLck: 876 kB > > VmPin: 0 kB > > VmHWM: 77463128 kB > > VmRSS: 67573800 kB > > VmData: 65535344 kB > > VmStk: 132 kB > > VmExe: 1076 kB > > VmLib: 146172 kB > > VmPTE: 714076 kB > > VmSwap: 2024936 kB > > > Thank you > Hardik > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ajay.kamble.ak at gmail.com Thu May 3 10:18:18 2018 From: ajay.kamble.ak at gmail.com (Ajay Kamble) Date: Thu, 3 May 2018 11:18:18 +0100 Subject: varnish regular expression for url Message-ID: Hello All, Need a help in creating a vcl rule to cache specific pages. Ask is if url contain /order/{x}/{y} cache it (now x and y is random string which may contain alphanumeric and special char) do not cache anything below /order/{x}/order/{x}/{y}/{z}/order/{x}/{y}/{z}/.../{n} thank you for taking look. Aj -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergio at sergiorus.com Fri May 11 13:58:46 2018 From: sergio at sergiorus.com (Sergio Rus) Date: Fri, 11 May 2018 14:58:46 +0100 Subject: 503 (Service Unavailable) from Varnish on POST request using Digest Authentication Message-ID: I would like to validate my theory over a weird issue I had with Varnish 3. So we have an endpoint on the server using Digest Authentication and I send a POST request to that endpoint without including the auth header: curl --data-urlencode foobar@/home/foo/bar.txt https://www.example.com/foobar/ Varnish is set up to pass all the requests going to that endpoint. By default POST requests are passed too, and I didn't change that behaviour. So we can assume Varnish is not trying to cache anything here. I would expect to get the response 401 UNAUTHORIZED from the server, but instead I get a 503 coming from Varnish. This is the log in Varnish for that request: SessionOpen c 127.0.0.1 34005 localhost:6081 ReqStart c 127.0.0.1 34005 1873165756 RxRequest c POST RxURL c /foobar/ RxProtocol c HTTP/1.0 RxHeader c Host: www.example.com RxHeader c X-Real-IP: 123.123.123.123 RxHeader c X-Forwarded-For: 123.123.123.123 RxHeader c X-Forwarded-Protocol: https RxHeader c X-SSL-Protocol: TLSv1.2 RxHeader c X-Forwarded-Proto: https RxHeader c X-Url-Scheme: https RxHeader c Connection: close RxHeader c Content-Length: 7811 RxHeader c User-Agent: foobar RxHeader c Accept: */* RxHeader c Content-Type: application/x-www-form-urlencoded VCL_call c recv pass VCL_call c hash Hash c /foobar/ Hash c www.example.com VCL_return c hash VCL_call c pass pass Backend c 17 default default TTL c 1873165756 RFC -1 -1 -1 1526037072 0 1526037072 0 0 VCL_call c fetch TTL c 1873165756 VCL -1 3600 -1 1526037072 -0 TTL c 1873165756 VCL 120 3600 -1 1526037072 -0 VCL_return c hit_for_pass ObjProtocol c HTTP/1.1 ObjResponse c UNAUTHORIZED ObjHeader c Server: foobar ObjHeader c Date: Fri, 11 May 2018 11:11:12 GMT ObjHeader c Content-Type: text/html; charset=utf-8 ObjHeader c WWW-Authenticate: Digest nonce="1526677072.03:1636:cb98ghghgh49495f1174226f25a2f02c79", realm="foobar", algorithm="MD5", opaque="9cbc87eb65454ff85bff73b2b38c06c34343e8", qop="auth", stale="false" ObjHeader c Content-Language: en-us ObjHeader c Vary: Accept-Encoding, Cookie, User-Agent VCL_call c error deliver VCL_call c deliver deliver TxProtocol c HTTP/1.1 TxStatus c 503 TxResponse c Service Unavailable TxHeader c Server: Varnish TxHeader c Content-Type: text/html; charset=utf-8 TxHeader c Retry-After: 5 TxHeader c Content-Length: 419 TxHeader c Accept-Ranges: bytes TxHeader c Date: Fri, 11 May 2018 11:11:12 GMT TxHeader c X-Varnish: 1873165756 TxHeader c Age: 0 TxHeader c Via: 1.1 varnish TxHeader c Connection: close Length c 419 ReqEnd c 1873165756 1526037072.059784174 1526037072.061955214 0.000038624 0.002078533 0.000092506 But reducing the size of the payload a little bit, it works: SessionOpen c 127.0.0.1 36019 localhost:6081 ReqStart c 127.0.0.1 36019 1873166315 RxRequest c POST RxURL c /foobar/ RxProtocol c HTTP/1.0 RxHeader c Host: www.example.com RxHeader c X-Real-IP: 123.123.123.123 RxHeader c X-Forwarded-For: 123.123.123.123 RxHeader c X-Forwarded-Protocol: https RxHeader c X-SSL-Protocol: TLSv1.2 RxHeader c X-Forwarded-Proto: https RxHeader c X-Url-Scheme: https RxHeader c Connection: close RxHeader c Content-Length: 7261 RxHeader c User-Agent: foobar RxHeader c Accept: */* RxHeader c Content-Type: application/x-www-form-urlencoded VCL_call c recv pass VCL_call c hash Hash c /foobar/ Hash c www.example.com VCL_return c hash VCL_call c pass pass Backend c 17 default default TTL c 1873166315 RFC -1 -1 -1 1526037112 0 1526037112 0 0 VCL_call c fetch TTL c 1873166315 VCL -1 3600 -1 1526037112 -0 TTL c 1873166315 VCL 120 3600 -1 1526037112 -0 VCL_return c hit_for_pass ObjProtocol c HTTP/1.1 ObjResponse c UNAUTHORIZED ObjHeader c Server: gunicorn/19.0.0 ObjHeader c Date: Fri, 11 May 2018 11:11:52 GMT ObjHeader c Content-Type: text/html; charset=utf-8 ObjHeader c WWW-Authenticate: Digest nonce="15260334112.14:3917:d3d558ed34342e8e723d8219bf3b20fc38", realm="foobar", algorithm="MD5", opaque="b32cd3f7b82213212397ded88c00ea06f111a29c", qop="auth", stale="false" ObjHeader c Content-Language: en-us ObjHeader c Vary: Accept-Encoding, Cookie, User-Agent VCL_call c deliver deliver TxProtocol c HTTP/1.1 TxStatus c 401 TxResponse c UNAUTHORIZED TxHeader c Server: gunicorn/19.0.0 TxHeader c Content-Type: text/html; charset=utf-8 TxHeader c WWW-Authenticate: Digest nonce="15260334112.14:3917:d3d558ed34342e8e723d8219bf3b20fc38", realm="foobar", algorithm="MD5", opaque="b32cd3f7b82213212397ded88c00ea06f111a29c", qop="auth", stale="false" TxHeader c Content-Language: en-us TxHeader c Vary: Accept-Encoding, Cookie, User-Agent TxHeader c Content-Length: 0 TxHeader c Accept-Ranges: bytes TxHeader c Date: Fri, 11 May 2018 11:11:52 GMT TxHeader c X-Varnish: 1873166315 TxHeader c Age: 0 TxHeader c Via: 1.1 varnish TxHeader c Connection: close Length c 0 ReqEnd c 1873166315 1526037112.192578554 1526037112.194519758 0.000039816 0.001864910 0.000076294 Why is that happening? Well, I have my own theory and this is what I would like to validate here. I think it's because there is some sort of limit set in Varnish for the size of the objects to keep in memory, so when the request size (headers + payload) is bigger than a threshold, and the request can not be delivered to the backend, it fails with 503. But if the request is smaller, Varnish keeps the object in memory and just sends to the client the response from the backend. Anyone here that could help me understand this weird behaviour? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Fri May 11 14:15:10 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 11 May 2018 16:15:10 +0200 Subject: 503 (Service Unavailable) from Varnish on POST request using Digest Authentication In-Reply-To: References: Message-ID: On Fri, May 11, 2018 at 3:58 PM, Sergio Rus wrote: > I would like to validate my theory over a weird issue I had with Varnish 3. It's reached EOL for a long while now, we aren't supporting Varnish 3... > So we have an endpoint on the server using Digest Authentication and I send > a POST request to that endpoint without including the auth header: > > curl --data-urlencode foobar@/home/foo/bar.txt > https://www.example.com/foobar/ > > Varnish is set up to pass all the requests going to that endpoint. By > default POST requests are passed too, and I didn't change that behaviour. So > we can assume Varnish is not trying to cache anything here. > > I would expect to get the response 401 UNAUTHORIZED from the server, but > instead I get a 503 coming from Varnish. This is the log in Varnish for that > request: > > SessionOpen c 127.0.0.1 34005 localhost:6081 > ReqStart c 127.0.0.1 34005 1873165756 > RxRequest c POST > RxURL c /foobar/ > RxProtocol c HTTP/1.0 > RxHeader c Host: www.example.com > RxHeader c X-Real-IP: 123.123.123.123 > RxHeader c X-Forwarded-For: 123.123.123.123 > RxHeader c X-Forwarded-Protocol: https > RxHeader c X-SSL-Protocol: TLSv1.2 > RxHeader c X-Forwarded-Proto: https > RxHeader c X-Url-Scheme: https > RxHeader c Connection: close > RxHeader c Content-Length: 7811 > RxHeader c User-Agent: foobar > RxHeader c Accept: */* > RxHeader c Content-Type: application/x-www-form-urlencoded > VCL_call c recv pass > VCL_call c hash > Hash c /foobar/ > Hash c www.example.com > VCL_return c hash > VCL_call c pass pass > Backend c 17 default default > TTL c 1873165756 RFC -1 -1 -1 1526037072 0 1526037072 0 0 > VCL_call c fetch > TTL c 1873165756 VCL -1 3600 -1 1526037072 -0 > TTL c 1873165756 VCL 120 3600 -1 1526037072 -0 > VCL_return c hit_for_pass > ObjProtocol c HTTP/1.1 > ObjResponse c UNAUTHORIZED > ObjHeader c Server: foobar > ObjHeader c Date: Fri, 11 May 2018 11:11:12 GMT > ObjHeader c Content-Type: text/html; charset=utf-8 > ObjHeader c WWW-Authenticate: Digest > nonce="1526677072.03:1636:cb98ghghgh49495f1174226f25a2f02c79", > realm="foobar", algorithm="MD5", > opaque="9cbc87eb65454ff85bff73b2b38c06c34343e8", qop="auth", stale="false" > ObjHeader c Content-Language: en-us > ObjHeader c Vary: Accept-Encoding, Cookie, User-Agent > VCL_call c error deliver According to this log record you could have a return(error) in your VCL, could it be that your larger object triggers some condition in your VCL that leads to a return(error) statement? > VCL_call c deliver deliver > TxProtocol c HTTP/1.1 > TxStatus c 503 > TxResponse c Service Unavailable > TxHeader c Server: Varnish > TxHeader c Content-Type: text/html; charset=utf-8 > TxHeader c Retry-After: 5 > TxHeader c Content-Length: 419 > TxHeader c Accept-Ranges: bytes > TxHeader c Date: Fri, 11 May 2018 11:11:12 GMT > TxHeader c X-Varnish: 1873165756 > TxHeader c Age: 0 > TxHeader c Via: 1.1 varnish > TxHeader c Connection: close > Length c 419 > ReqEnd c 1873165756 1526037072.059784174 1526037072.061955214 > 0.000038624 0.002078533 0.000092506 > > But reducing the size of the payload a little bit, it works: > > SessionOpen c 127.0.0.1 36019 localhost:6081 > ReqStart c 127.0.0.1 36019 1873166315 > RxRequest c POST > RxURL c /foobar/ > RxProtocol c HTTP/1.0 > RxHeader c Host: www.example.com > RxHeader c X-Real-IP: 123.123.123.123 > RxHeader c X-Forwarded-For: 123.123.123.123 > RxHeader c X-Forwarded-Protocol: https > RxHeader c X-SSL-Protocol: TLSv1.2 > RxHeader c X-Forwarded-Proto: https > RxHeader c X-Url-Scheme: https > RxHeader c Connection: close > RxHeader c Content-Length: 7261 > RxHeader c User-Agent: foobar > RxHeader c Accept: */* > RxHeader c Content-Type: application/x-www-form-urlencoded > VCL_call c recv pass > VCL_call c hash > Hash c /foobar/ > Hash c www.example.com > VCL_return c hash > VCL_call c pass pass > Backend c 17 default default > TTL c 1873166315 RFC -1 -1 -1 1526037112 0 1526037112 0 0 > VCL_call c fetch > TTL c 1873166315 VCL -1 3600 -1 1526037112 -0 > TTL c 1873166315 VCL 120 3600 -1 1526037112 -0 > VCL_return c hit_for_pass > ObjProtocol c HTTP/1.1 > ObjResponse c UNAUTHORIZED > ObjHeader c Server: gunicorn/19.0.0 > ObjHeader c Date: Fri, 11 May 2018 11:11:52 GMT > ObjHeader c Content-Type: text/html; charset=utf-8 > ObjHeader c WWW-Authenticate: Digest > nonce="15260334112.14:3917:d3d558ed34342e8e723d8219bf3b20fc38", > realm="foobar", algorithm="MD5", > opaque="b32cd3f7b82213212397ded88c00ea06f111a29c", qop="auth", stale="false" > ObjHeader c Content-Language: en-us > ObjHeader c Vary: Accept-Encoding, Cookie, User-Agent > VCL_call c deliver deliver > TxProtocol c HTTP/1.1 > TxStatus c 401 > TxResponse c UNAUTHORIZED > TxHeader c Server: gunicorn/19.0.0 > TxHeader c Content-Type: text/html; charset=utf-8 > TxHeader c WWW-Authenticate: Digest > nonce="15260334112.14:3917:d3d558ed34342e8e723d8219bf3b20fc38", > realm="foobar", algorithm="MD5", > opaque="b32cd3f7b82213212397ded88c00ea06f111a29c", qop="auth", stale="false" > TxHeader c Content-Language: en-us > TxHeader c Vary: Accept-Encoding, Cookie, User-Agent > TxHeader c Content-Length: 0 > TxHeader c Accept-Ranges: bytes > TxHeader c Date: Fri, 11 May 2018 11:11:52 GMT > TxHeader c X-Varnish: 1873166315 > TxHeader c Age: 0 > TxHeader c Via: 1.1 varnish > TxHeader c Connection: close > Length c 0 > ReqEnd c 1873166315 1526037112.192578554 1526037112.194519758 > 0.000039816 0.001864910 0.000076294 > > Why is that happening? Well, I have my own theory and this is what I would > like to validate here. I think it's because there is some sort of limit set > in Varnish for the size of the objects to keep in memory, so when the > request size (headers + payload) is bigger than a threshold, and the request > can not be delivered to the backend, it fails with 503. But if the request > is smaller, Varnish keeps the object in memory and just sends to the client > the response from the backend. > > Anyone here that could help me understand this weird behaviour? If it's the size of the response, only headers count, not the body. In which case you do have knobs to allow larger HTTP responses: https://varnish-cache.org/docs/3.0/reference/varnishd.html#run-time-parameters See http_resp_size and http_resp_hdr_len for respectively the size of the whole set of response headers and the size of individual headers. Dridi From sergio at sergiorus.com Fri May 11 14:38:48 2018 From: sergio at sergiorus.com (Sergio Rus) Date: Fri, 11 May 2018 15:38:48 +0100 Subject: 503 (Service Unavailable) from Varnish on POST request using Digest Authentication In-Reply-To: References: Message-ID: Varnish 3 is quite old, I know. But it's still supported by Ubuntu 14 LTS. That's why I'm still using it. I will move to a recent version soon. In regards to the issue, I don't have any return(error) in the VCL I wrote. So maybe it's coming from somewhere in the default VCL? I read about those parameters you mentioned already, and even increased them just for testing, but didn't work at that time, testing bigger files. So in any case that would be a partial solution, because it would fail at some point with larger payloads. I'm currently using the default values for those parameters, 8K, which is very close to the threshold I manually found in the logs I posted. If you see the payload size, it's nearly 8K. Only adding or removing a few bytes in the payload changes the behaviour in Varnish. But the headers size in any case is the same, so that's why I don't understand what's happening. The payload size seems to be affecting here. Cheers -------------- next part -------------- An HTML attachment was scrubbed... URL: From ilinuxer.85 at gmail.com Mon May 14 07:40:41 2018 From: ilinuxer.85 at gmail.com (jack Linux) Date: Mon, 14 May 2018 09:40:41 +0200 Subject: Problem with Varnish X-Cache: MISS all the time Message-ID: I Have website running on cloudflare + nginx + varnish on centos 6.9 arch 64bit Distribution ============ Header status = ============ HTTP/1.1 200 OK Date: Mon, 14 May 2018 07:25:34 GMT Content-Type: text/html Connection: keep-alive Set-Cookie: __cfduid=d0d5a5c3e3423f286ca476d1fecbeeb7a1526282734; expires=Tue, 14-May-19 07:25:34 GMT; path=/; domain=.site.com; HttpOnly X-Powered-By: PHP/5.4.45 Set-Cookie: PHPSESSID=4a3847f117e1a757e8b3f92784987bfd; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: private, must-revalidate Pragma: no-cache Set-Cookie: detect_mobiles=20; expires=Tue, 15-May-2018 00:08:54 GMT; path=/ Vary: Accept-Encoding X-Varnish: 1615241683 Age: 0 Via: 1.1 varnish X-Cache: MISS Set-Cookie: site.com=web01; path=/ Server: cloudflare CF-RAY: 41aba1b134769c35-AMS ============= = varnish config = ============= /* SET THE HOST AND PORT OF SITE * *********************************************************/ backend default { .host = "10.0.0.1"; .port = "8080"; } # SET THE ALLOWED IP OF PURGE REQUESTS # ########################################################## acl purge { "localhost"; "10.0.0.1"; } #THE RECV FUNCTION # ########################################################## sub vcl_recv { #remove HTTPOXY CGI vulnerability unset req.http.proxy; #remove extraneous host ports set req.http.host = regsub(req.http.Host, ":[0-9]+", ""); # set realIP by trimming CloudFlare IP which will be used for various checks set req.http.X-Actual-IP = regsub(req.http.X-Forwarded-For, "[, ].*$", ""); # Enable smart refreshing if (req.http.Cache-Control ~ "no-cache" && client.ip ~ purge) { set req.hash_always_miss = true; } # Unset cloudflare cookies # Remove has_js and CloudFlare/Google Analytics __* cookies. set req.http.Cookie = regsuball(req.http.Cookie, "(^|;\s*)(_[_a-z]+|has_js)=[^;]*", ""); # Remove a ";" prefix, if present. set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", ""); # For Testing: If you want to test with Varnish passing (not caching) uncomment # return( pass ); # FORWARD THE IP OF THE REQUEST if (req.restarts == 0) { if (req.http.x-forwarded-for) { set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ", " + client.ip; } else { set req.http.X-Forwarded-For = client.ip; } } # DO NOT CACHE RSS FEED if (req.url ~ "/feed/") { return ( pass ); } ## Do not cache search results, comment these 3 lines if you do want to cache them if (req.url ~ "/\?s\=") { return ( pass ); } # CLEAN UP THE ENCODING HEADER. # SET TO GZIP, DEFLATE, OR REMOVE ENTIRELY. WITH VARY ACCEPT-ENCODING # VARNISH WILL CREATE SEPARATE CACHES FOR EACH # DO NOT ACCEPT-ENCODING IMAGES, ZIPPED FILES, AUDIO, ETC. # ########################################################## if (req.http.Accept-Encoding) { if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") { # No point in compressing these remove req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { # unknown algorithm remove req.http.Accept-Encoding; } } # IF THIS IS A PURGE REQUEST, THEN CHECK THE IPS SET ABOVE # BLOCK IF NOT ONE OF THOSE IPS # ########################################################## if (req.request == "PURGE") { if ( !client.ip ~ purge ) { error 405 "Not allowed."; } return (lookup); } # PIPE ALL NON-STANDARD REQUESTS # ########################################################## if (req.request != "GET" && req.request != "HEAD" && req.request != "PUT" && req.request != "POST" && req.request != "TRACE" && req.request != "OPTIONS" && req.request != "DELETE") { return (pipe); } # ONLY CACHE GET AND HEAD REQUESTS # ########################################################## if (req.request != "GET" && req.request != "HEAD") { return (pass); } # IF YOU GET HERE THEN THIS REQUEST SHOULD BE CACHED # ########################################################## return (lookup); } sub vcl_hash { #this is to store cache based on PHPSESSID or woocommerce cookie so cart doesn't show 0 if (req.http.cookie) { hash_data(req.http.cookie); } #fix flexible ssl css if (req.http.x-forwarded-proto) { hash_data(req.http.x-forwarded-proto); } } # HIT FUNCTION # ########################################################## sub vcl_hit { # IF THIS IS A PURGE REQUEST THEN DO THE PURGE # ########################################################## if (req.request == "PURGE") { purge; error 200 "Purged."; } return (deliver); } # MISS FUNCTION # ########################################################## sub vcl_miss { if (req.request == "PURGE") { purge; error 200 "Purged."; } return (fetch); } # FETCH FUNCTION # ########################################################## sub vcl_fetch { # I SET THE VARY TO ACCEPT-ENCODING, THIS OVERRIDES W3TC # TENDANCY TO SET VARY USER-AGENT. YOU MAY OR MAY NOT WANT # TO DO THIS # ########################################################## set beresp.http.Vary = "Accept-Encoding"; } # DELIVER FUNCTION # ########################################################## sub vcl_deliver { if ( req.url ~ "\?action=wpp_get_popular" && !req.http.cookie ~ "wordpress_logged_in" ) { # Remove headers preventing Cloudflare cache unset resp.http.Cache-Control; unset resp.http.Expires; unset resp.http.Pragma; unset resp.http.ETag; # Cache request in browser for 1 day (in seconds) set resp.http.Cache-Control = "max-age=259200"; } # IF THIS PAGE IS ALREADY CACHED THEN RETURN A 'HIT' TEXT # IN THE HEADER (GREAT FOR DEBUGGING) # ########################################################## if (obj.hits > 0) { set resp.http.X-Cache = "HIT"; # IF THIS IS A MISS RETURN THAT IN THE HEADER # ########################################################## } else { set resp.http.X-Cache = "MISS"; } } From guillaume at varnish-software.com Mon May 14 09:01:34 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 14 May 2018 11:01:34 +0200 Subject: Problem with Varnish X-Cache: MISS all the time In-Reply-To: References: Message-ID: Hi, you are hashing the cookie, that is pretty sure to kill your hit ratio. Regards, -- Guillaume Quintard On Mon, May 14, 2018 at 9:40 AM, jack Linux wrote: > I Have website running on cloudflare + nginx + varnish on centos 6.9 > arch 64bit Distribution > > > ============ > Header status = > ============ > > HTTP/1.1 200 OK > Date: Mon, 14 May 2018 07:25:34 GMT > Content-Type: text/html > Connection: keep-alive > Set-Cookie: __cfduid=d0d5a5c3e3423f286ca476d1fecbeeb7a1526282734; > expires=Tue, 14-May-19 07:25:34 GMT; path=/; domain=.site.com; > HttpOnly > X-Powered-By: PHP/5.4.45 > Set-Cookie: PHPSESSID=4a3847f117e1a757e8b3f92784987bfd; path=/ > Expires: Thu, 19 Nov 1981 08:52:00 GMT > Cache-Control: private, must-revalidate > Pragma: no-cache > Set-Cookie: detect_mobiles=20; expires=Tue, 15-May-2018 00:08:54 GMT; > path=/ > Vary: Accept-Encoding > X-Varnish: 1615241683 > Age: 0 > Via: 1.1 varnish > X-Cache: MISS > Set-Cookie: site.com=web01; path=/ > Server: cloudflare > CF-RAY: 41aba1b134769c35-AMS > > ============= > = varnish config = > ============= > > /* SET THE HOST AND PORT OF SITE > * *********************************************************/ > > backend default { > .host = "10.0.0.1"; > .port = "8080"; > } > > # SET THE ALLOWED IP OF PURGE REQUESTS > # ########################################################## > acl purge { > "localhost"; > "10.0.0.1"; > } > > #THE RECV FUNCTION > # ########################################################## > sub vcl_recv { > #remove HTTPOXY CGI vulnerability > unset req.http.proxy; > > #remove extraneous host ports > set req.http.host = regsub(req.http.Host, ":[0-9]+", ""); > > # set realIP by trimming CloudFlare IP which will be used for > various checks > set req.http.X-Actual-IP = regsub(req.http.X-Forwarded-For, > "[, ].*$", ""); > > # Enable smart refreshing > if (req.http.Cache-Control ~ "no-cache" && client.ip ~ purge) { > set req.hash_always_miss = true; > } > > # Unset cloudflare cookies > > # Remove has_js and CloudFlare/Google Analytics __* cookies. > set req.http.Cookie = regsuball(req.http.Cookie, > "(^|;\s*)(_[_a-z]+|has_js)=[^;]*", ""); > # Remove a ";" prefix, if present. > set req.http.Cookie = regsub(req.http.Cookie, "^;\s*", ""); > > # For Testing: If you want to test with Varnish passing (not > caching) uncomment > # return( pass ); > > # FORWARD THE IP OF THE REQUEST > if (req.restarts == 0) { > if (req.http.x-forwarded-for) { > set req.http.X-Forwarded-For = > req.http.X-Forwarded-For + ", " + client.ip; > } else { > set req.http.X-Forwarded-For = client.ip; > } > } > > # DO NOT CACHE RSS FEED > if (req.url ~ "/feed/") { > return ( pass ); > } > > ## Do not cache search results, comment these 3 lines if you do want > to cache them > > if (req.url ~ "/\?s\=") { > return ( pass ); > } > > # CLEAN UP THE ENCODING HEADER. > # SET TO GZIP, DEFLATE, OR REMOVE ENTIRELY. WITH VARY ACCEPT-ENCODING > # VARNISH WILL CREATE SEPARATE CACHES FOR EACH > # DO NOT ACCEPT-ENCODING IMAGES, ZIPPED FILES, AUDIO, ETC. > # ########################################################## > if (req.http.Accept-Encoding) { > if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") { > # No point in compressing these > remove req.http.Accept-Encoding; > } elsif (req.http.Accept-Encoding ~ "gzip") { > set req.http.Accept-Encoding = "gzip"; > } elsif (req.http.Accept-Encoding ~ "deflate") { > set req.http.Accept-Encoding = "deflate"; > } else { > # unknown algorithm > remove req.http.Accept-Encoding; > } > } > > # IF THIS IS A PURGE REQUEST, THEN CHECK THE IPS SET ABOVE > # BLOCK IF NOT ONE OF THOSE IPS > # ########################################################## > if (req.request == "PURGE") { > if ( !client.ip ~ purge ) { > error 405 "Not allowed."; > } > return (lookup); > } > > # PIPE ALL NON-STANDARD REQUESTS > # ########################################################## > if (req.request != "GET" && > req.request != "HEAD" && > req.request != "PUT" && > req.request != "POST" && > req.request != "TRACE" && > req.request != "OPTIONS" && > req.request != "DELETE") { > return (pipe); > } > > # ONLY CACHE GET AND HEAD REQUESTS > # ########################################################## > if (req.request != "GET" && req.request != "HEAD") { > return (pass); > } > > # IF YOU GET HERE THEN THIS REQUEST SHOULD BE CACHED > # ########################################################## > return (lookup); > } > > sub vcl_hash { > #this is to store cache based on PHPSESSID or woocommerce cookie so > cart doesn't show 0 > if (req.http.cookie) { > hash_data(req.http.cookie); > } > #fix flexible ssl css > if (req.http.x-forwarded-proto) { > hash_data(req.http.x-forwarded-proto); > } > } > > # HIT FUNCTION > # ########################################################## > sub vcl_hit { > # IF THIS IS A PURGE REQUEST THEN DO THE PURGE > # ########################################################## > if (req.request == "PURGE") { > purge; > error 200 "Purged."; > } > > return (deliver); > } > > # MISS FUNCTION > # ########################################################## > sub vcl_miss { > if (req.request == "PURGE") { > purge; > error 200 "Purged."; > } > > return (fetch); > } > > # FETCH FUNCTION > # ########################################################## > sub vcl_fetch { > # I SET THE VARY TO ACCEPT-ENCODING, THIS OVERRIDES W3TC > # TENDANCY TO SET VARY USER-AGENT. YOU MAY OR MAY NOT WANT > # TO DO THIS > # ########################################################## > > set beresp.http.Vary = "Accept-Encoding"; > > } > > # DELIVER FUNCTION # > ########################################################## > sub vcl_deliver { > > if ( req.url ~ "\?action=wpp_get_popular" && !req.http.cookie ~ > "wordpress_logged_in" ) { > # Remove headers preventing Cloudflare cache > unset resp.http.Cache-Control; > unset resp.http.Expires; > unset resp.http.Pragma; > unset resp.http.ETag; > # Cache request in browser for 1 day (in seconds) > set resp.http.Cache-Control = "max-age=259200"; > } > > > # IF THIS PAGE IS ALREADY CACHED THEN RETURN A 'HIT' TEXT > # IN THE HEADER (GREAT FOR DEBUGGING) > # ########################################################## > if (obj.hits > 0) { > set resp.http.X-Cache = "HIT"; > # IF THIS IS A MISS RETURN THAT IN THE HEADER > # ########################################################## > } else { > set resp.http.X-Cache = "MISS"; > } > } > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Mon May 14 09:13:43 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 14 May 2018 11:13:43 +0200 Subject: 503 (Service Unavailable) from Varnish on POST request using Digest Authentication In-Reply-To: References: Message-ID: Hi, Could it simply be your backend complaining? Notably, I'm pretty sure that WWW-Authenticate header doesn't come from Varnish, but I may be wrong. Cheers, -- Guillaume Quintard On Fri, May 11, 2018 at 4:38 PM, Sergio Rus wrote: > Varnish 3 is quite old, I know. But it's still supported by Ubuntu 14 LTS. > That's why I'm still using it. I will move to a recent version soon. > > In regards to the issue, I don't have any return(error) in the VCL I > wrote. So maybe it's coming from somewhere in the default VCL? > > I read about those parameters you mentioned already, and even increased > them just for testing, but didn't work at that time, testing bigger files. > So in any case that would be a partial solution, because it would fail at > some point with larger payloads. I'm currently using the default values for > those parameters, 8K, which is very close to the threshold I manually found > in the logs I posted. If you see the payload size, it's nearly 8K. Only > adding or removing a few bytes in the payload changes the behaviour in > Varnish. But the headers size in any case is the same, so that's why I > don't understand what's happening. The payload size seems to be affecting > here. > > Cheers > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Mon May 14 09:40:06 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 14 May 2018 11:40:06 +0200 Subject: Set cookie creating issues In-Reply-To: <3b74b839-8ff5-f5ea-23dc-41d6fd5113bc@waltzz.com> References: <3b74b839-8ff5-f5ea-23dc-41d6fd5113bc@waltzz.com> Message-ID: Hi, That's really going to be hard to debug without some logs. Anyway, your vcl doeesn't unset set-cookie headers, so you can expect you cached objects to wreck some havoc since they will be reused for multiple clients. Regards, -- Guillaume Quintard On Fri, Apr 27, 2018 at 4:57 PM, Pinakee BIswas wrote: > Hi, > > We have been using Varnish for caching our web pages. We have an ecommerce > site. Things have been working fine till today but suddenly things have > started breaking down and I am not sure why. Following is the issue: > > We use session cookie to store user sessions. The session cookie is > getting changed as Cached responses from varnish is having set-cookie > header which is messing up the session cookie. We are using varnish 4.8. > Following is a snippet of the VCL: > > > sub vcl_recv { > # Happens before we check if we have this in cache already. > # > # Typically you clean up the request here, removing cookies you don't > need, > # rewriting the request, etc. > set req.backend_hint = uwsgi; > > #if (req.http.cookie ~ "jivaana_country=") { > # Set the country header > # set req.http.X-CLIENT-COUNTRY = regsub(req.http.cookie, > ".*jivaana_country=([^;]+);.*", "\1"); > # } > > std.log("ga:" + ga.extract(req.url, mode = keep)); > set req.url = ga.apply(req.url); # remove Google Analytics parameters > > if (req.method == "GET") { > if ((req.url !~ "^/accounts/userheader") && > (req.url !~ "^/accounts/new-userheader") && > (req.url !~ "^/product/recently-viewed") && > (req.url !~ "^/product/recommended-products") && > (req.url !~ "^/product/addtobasket")) { > unset req.http.cookie; # strip the cookies - we don't need > them > } > } > > call devicedetect; > } > > sub vcl_backend_response { > # Happens after we have read the response headers from the backend. > # > # Here you clean the response headers, removing silly Set-Cookie > headers > # and other mistakes your backend does. > if (bereq.method == "GET") { > set beresp.do_esi = true; > if ((bereq.url !~ "^/accounts/userheader") && > (bereq.url !~ "^/accounts/new-userheader") && > (bereq.url !~ "^/product/recently-viewed") && > (bereq.url !~ "^/product/recommended-products") && > (bereq.url !~ "^/product/addtobasket")) { > #unset beresp.http.Set-Cookie; > set beresp.uncacheable = false; > #std.log("Caching the url : **********************" + > bereq.url); > } > } > > sub vcl_deliver { > # Happens when we have all the pieces we need, and are about to send > the > # response to the client. > # > # You can do accounting or modifying the final object here. > } > > sub vcl_hash { > } > > Would really appreciate any support as this is messing up our user > sessions. > > Thanks, > > Pinakee > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sergio at sergiorus.com Mon May 14 09:50:11 2018 From: sergio at sergiorus.com (Sergio Rus) Date: Mon, 14 May 2018 10:50:11 +0100 Subject: 503 (Service Unavailable) from Varnish on POST request using Digest Authentication In-Reply-To: References: Message-ID: Well, this issue is not really about the authentication process itself. As I said, the endpoint was using Digest Authentication, but I was intentionally not including the auth header, so I was expecting to get in the client a 401 UNAUTHORIZED response, but instead I was getting a 503 coming from Varnish. Everything works fine without Varnish: the client gets the 401 UNAUTHORIZED in every case, using any payload size. So I just wanted to understand why Varnish was failing with a 503, and after several tests I found that reducing the payload size a little bit, keeping it around 7K-8K, it worked. A few bytes bigger, close to 8K, it failed (503). To me, it looks like Varnish has some sort of size limit for the objects that can keep in memory that is not only applied for the headers size, but also for the payload in POST requests. Unfortunately I couldn't find in the documentation any parameter related to this. But as I said, increasing a parameter wouldn't be the right solution, because at some point it would fail again with bigger payloads. Cheers -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Mon May 14 09:54:39 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 14 May 2018 11:54:39 +0200 Subject: 503 (Service Unavailable) from Varnish on POST request using Digest Authentication In-Reply-To: References: Message-ID: Duh, right, I got really confused with the old log format. Sorry for the noise -- Guillaume Quintard On Mon, May 14, 2018 at 11:50 AM, Sergio Rus wrote: > Well, this issue is not really about the authentication process itself. As > I said, the endpoint was using Digest Authentication, but I was > intentionally not including the auth header, so I was expecting to get in > the client a 401 UNAUTHORIZED response, but instead I was getting a 503 > coming from Varnish. Everything works fine without Varnish: the client gets > the 401 UNAUTHORIZED in every case, using any payload size. So I just > wanted to understand why Varnish was failing with a 503, and after several > tests I found that reducing the payload size a little bit, keeping it > around 7K-8K, it worked. A few bytes bigger, close to 8K, it failed (503). > To me, it looks like Varnish has some sort of size limit for the objects > that can keep in memory that is not only applied for the headers size, but > also for the payload in POST requests. Unfortunately I couldn't find in the > documentation any parameter related to this. But as I said, increasing a > parameter wouldn't be the right solution, because at some point it would > fail again with bigger payloads. > > Cheers > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Mon May 14 15:44:50 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Mon, 14 May 2018 17:44:50 +0200 Subject: 503 (Service Unavailable) from Varnish on POST request using Digest Authentication In-Reply-To: References: Message-ID: On Fri, May 11, 2018 at 4:38 PM, Sergio Rus wrote: > Varnish 3 is quite old, I know. But it's still supported by Ubuntu 14 LTS. > That's why I'm still using it. I will move to a recent version soon. I don't think Ubuntu supports Varnish in any way. In my book that would mean you could open a ticket to them and have the package maintainer track down the bug and provide a fix (especially since _we_ won't do that considering it's EOL). > In regards to the issue, I don't have any return(error) in the VCL I wrote. > So maybe it's coming from somewhere in the default VCL? I don't think so, looking closer I don't see any hint at that (and don't remember any return(error) in the built-in off the top of my head). > I read about those parameters you mentioned already, and even increased them > just for testing, but didn't work at that time, testing bigger files. So in > any case that would be a partial solution, because it would fail at some > point with larger payloads. I'm currently using the default values for those > parameters, 8K, which is very close to the threshold I manually found in the > logs I posted. If you see the payload size, it's nearly 8K. Only adding or > removing a few bytes in the payload changes the behaviour in Varnish. But > the headers size in any case is the same, so that's why I don't understand > what's happening. The payload size seems to be affecting here. Very puzzling but sorry, I dipped my toes and don't feel like diving :( Dridi From sergio at sergiorus.com Mon May 14 16:51:56 2018 From: sergio at sergiorus.com (Sergio Rus) Date: Mon, 14 May 2018 17:51:56 +0100 Subject: 503 (Service Unavailable) from Varnish on POST request using Digest Authentication In-Reply-To: References: Message-ID: Ok don't mind. I guess it's quite a corner case. Or maybe not so unusual... Hope it's fixed (if needed) in recent versions of Varnish. I will test it when I make the move. Thanks From n.premkumar.me at gmail.com Thu May 17 15:44:04 2018 From: n.premkumar.me at gmail.com (Prem Kumar) Date: Thu, 17 May 2018 21:14:04 +0530 Subject: Strange behavior on varnish 5 Message-ID: Hi All, We are 2 behaviors on Varnish 5: - if we set ttl = 600s , we are seeing HIT is serving the content over 10s . Origin is not changing the content and we are NOT sending if_modified_since header. - VCL_return lookup - Hit 527698198 *-0.990171<<<< why it is a hit but ttl is negative value * 0.000000 0.000000 - VCL_call HIT - VCL_Log hit index: 305 - VCL_Log x-cache-hit:Buffer - VCL_Log rsp-source:Buffer - VCL_Log x-ingest:false - VCL_return deliver - Link bereq 288421900 bgfetch - Timestamp Fetch: 1526569247.693528 0.000478 0.000478 - RespProtocol HTTP/1.1 - RespStatus 403 - How to disable background fetch ?. Thanks, Prem -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu May 17 15:45:56 2018 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 17 May 2018 17:45:56 +0200 Subject: Strange behavior on varnish 5 In-Reply-To: References: Message-ID: Hello, You are discovering grace :-) https://varnish-cache.org/docs/trunk/users-guide/vcl-grace.html "set beresp.grace = 0s;" should do it. -- Guillaume Quintard On Thu, May 17, 2018 at 5:44 PM, Prem Kumar wrote: > Hi All, > > We are 2 behaviors on Varnish 5: > > - if we set ttl = 600s , we are seeing HIT is serving the content over 10s > . Origin is not changing the content and we are NOT sending > if_modified_since header. > > - VCL_return lookup > - Hit 527698198 *-0.990171<<<< why it is a hit but ttl is > negative value * 0.000000 0.000000 > - VCL_call HIT > - VCL_Log hit index: 305 > - VCL_Log x-cache-hit:Buffer > - VCL_Log rsp-source:Buffer > - VCL_Log x-ingest:false > - VCL_return deliver > - Link bereq 288421900 bgfetch > - Timestamp Fetch: 1526569247.693528 0.000478 0.000478 > - RespProtocol HTTP/1.1 > - RespStatus 403 > > > - How to disable background fetch ?. > > Thanks, > Prem > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From n.premkumar.me at gmail.com Thu May 17 16:03:36 2018 From: n.premkumar.me at gmail.com (Prem Kumar) Date: Thu, 17 May 2018 21:33:36 +0530 Subject: Strange behavior on varnish 5 In-Reply-To: References: Message-ID: Hit 527698198 *-0.990171<<<< why it is a hit but ttl is negative value * *0.000000 << Grace is 0* 0.000000 - VCL_call HIT We set to zero. Thanks, Prem On Thu, May 17, 2018 at 9:15 PM, Guillaume Quintard < guillaume at varnish-software.com> wrote: > Hello, > > You are discovering grace :-) > > https://varnish-cache.org/docs/trunk/users-guide/vcl-grace.html > > "set beresp.grace = 0s;" should do it. > > -- > Guillaume Quintard > > On Thu, May 17, 2018 at 5:44 PM, Prem Kumar > wrote: > >> Hi All, >> >> We are 2 behaviors on Varnish 5: >> >> - if we set ttl = 600s , we are seeing HIT is serving the content over >> 10s . Origin is not changing the content and we are NOT sending >> if_modified_since header. >> >> - VCL_return lookup >> - Hit 527698198 *-0.990171<<<< why it is a hit but ttl is >> negative value * 0.000000 0.000000 >> - VCL_call HIT >> - VCL_Log hit index: 305 >> - VCL_Log x-cache-hit:Buffer >> - VCL_Log rsp-source:Buffer >> - VCL_Log x-ingest:false >> - VCL_return deliver >> - Link bereq 288421900 bgfetch >> - Timestamp Fetch: 1526569247.693528 0.000478 0.000478 >> - RespProtocol HTTP/1.1 >> - RespStatus 403 >> >> >> - How to disable background fetch ?. >> >> Thanks, >> Prem >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Fri May 18 06:54:31 2018 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 18 May 2018 08:54:31 +0200 Subject: Strange behavior on varnish 5 In-Reply-To: References: Message-ID: On Thu, May 17, 2018 at 6:03 PM, Prem Kumar wrote: > Hit 527698198 -0.990171<<<< why it is a hit but ttl is negative > value 0.000000 << Grace is 0 0.000000 > - VCL_call HIT > > We set to zero. Are you actively setting it to zero? The default is 10s. This could be related to the recent changes in TTL calculation: https://github.com/varnishcache/varnish-cache/pull/2555 Dridi From hugues at betabrand.com Thu May 31 22:49:17 2018 From: hugues at betabrand.com (Hugues Alary) Date: Thu, 31 May 2018 15:49:17 -0700 Subject: BAN - Memory consumption exploding under load In-Reply-To: References: Message-ID: Hi there, Re-opening this thread from 2016! A few days ago, I ran into an issue where varnish (varnish-6.0.0 revision a068361dff0d25a0d85cf82a6e5fdaf315e06a7d) became completely unresponsive. While investigating, I noticed a lot of errors in varnishlog `VSL: store overflow`, but most importantly, I noticed our BAN list was starting to accumulate with somewhere around 130K BANs reported by varnishstat. The load on the website was pretty low. Restarting Varnish fixed the issue. It is possible I had been running some script generating a lot of BANs, but it still seems weird it would generate so many BANs that varnish would become unresponsive. After restarting Varnish, I watched `varnishstat -1 | grep bans` and could see MAIN.bans going up and down, but staying fairly stable on average. Today, I decided to check again and I noticed: MAIN.bans 66863 . Count of bans MAIN.bans_completed 66834 . Number of bans marked 'completed' MAIN.bans_obj 66849 . Number of bans using obj.* MAIN.bans_req 14 . Number of bans using req.* MAIN.bans_added 170939 1.18 Bans added MAIN.bans_deleted 104076 0.72 Bans deleted MAIN.bans_tested 3767852 26.08 Bans tested against objects (lookup) MAIN.bans_obj_killed 62024 0.43 Objects killed by bans (lookup) MAIN.bans_lurker_tested 3344073359 23148.62 Bans tested against objects (lurker) MAIN.bans_tests_tested 94711644 655.62 Ban tests tested against objects (lookup) MAIN.bans_lurker_tests_tested 4573910595 31661.91 Ban tests tested against objects (lurker) MAIN.bans_lurker_obj_killed 16627 0.12 Objects killed by bans (lurker) MAIN.bans_lurker_obj_killed_cutoff 0 0.00 Objects killed by bans for cutoff (lurker) MAIN.bans_dups 113845 0.79 Bans superseded by other bans MAIN.bans_lurker_contention 3 0.00 Lurker gave way for lookup MAIN.bans_persisted_bytes 163879330 . Bytes used by the persisted ban lists MAIN.bans_persisted_fragmentation 162795194 . Extra bytes in persisted ban lists due to fragmentation 1- MAIN.bans is almost at 70k and varnish has been running for 1 day 16 hours 2- MAIN.bans_req is > 0, but nowhere in my configuration do I ever submit bans using req.* So I have 3 questions: - is 70k+ bans un-reasonable and should I find a way to reduce them? - why am I seeing 14 MAIN.bans_req when my config doesn't ever emit such bans? Thanks, -Hugues On Thu, Nov 3, 2016 at 3:04 AM, Dridi Boukelmoune wrote: > On Thu, Oct 27, 2016 at 6:02 PM, Nils Goroll wrote: > > Hi, > > > > we've added a bunch of ban and ban lurker improvements which did not get > > backported to 4.1.3 > > > > Please upgrade to 5.0 or master. > > Hi Nils, > > I was thinking (against the too-many-knobs trend) that maybe we should > have some sort of ban_queue_limit parameter to avoid piling up too > many bans. It would also play nice with your std.ban() initiative > where hitting the queue limit could be one of the reasons returned by > the function when it fails. > > The default could be 0 for unlimited, or a fairly large number. > > Dridi > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: