From dridi at varni.sh Thu Dec 5 10:24:36 2019 From: dridi at varni.sh (Dridi Boukelmoune) Date: Thu, 5 Dec 2019 10:24:36 +0000 Subject: Caching a resource after specific number of requests In-Reply-To: <558895272.8253744.1575396235450@mail.yahoo.com> References: <558895272.8253744.1575396235450.ref@mail.yahoo.com> <558895272.8253744.1575396235450@mail.yahoo.com> Message-ID: On Tue, Dec 3, 2019 at 6:04 PM al sefid wrote: > > Hello there! > Is there any functionality like proxy_cache_min_uses in the Varnish cache that caches a resource after specific number of requests to that resource? > Thank you! Varnish is not capable of doing that by itself, but as Geoff pointed out on github you could rely on some form of throttling to emulate this functionality, but that requires a third-party module: https://github.com/varnishcache/varnish-cache/issues/3150#issuecomment-561307014 Be careful though, his example was a bit simplistic (I assume on purpose to show the idea) and shouldn't be used as-is. It might also not work if you have an object in your cache that outlives the throttling. Maybe the cat flap could work for this case, but here be dragons... I moved the discussion to the misc mailing list because this is not (yet?) dev material. While this may sound like a good idea, and while it works for nginx, it might not be a good fit for varnish. Of course unless we have that feature in place, it's also hard to tell whether this would benefit your workload, I suspect it would make things worse or force us to make other changes. In particular, making this a global configuration could be highly detrimental if you have some variety of contents in your cache (small, big, short- and long-lived etc) and the best place to start is to have the backend tell you precisely all about responses you may be able to cache or not and how instead of starting with an a priori criteria that a response shouldn't be cached unless it's asked at least 3 times (which is a very vague requirement). Dridi From geoff at uplex.de Thu Dec 5 10:30:06 2019 From: geoff at uplex.de (Geoff Simmons) Date: Thu, 5 Dec 2019 11:30:06 +0100 Subject: Caching a resource after specific number of requests In-Reply-To: References: <558895272.8253744.1575396235450.ref@mail.yahoo.com> <558895272.8253744.1575396235450@mail.yahoo.com> Message-ID: On 12/5/19 11:24, Dridi Boukelmoune wrote: > On Tue, Dec 3, 2019 at 6:04 PM al sefid wrote: >> >> Is there any functionality like proxy_cache_min_uses in the >> Varnish cache that caches a resource after specific number of >> requests to that resource? > > Varnish is not capable of doing that by itself, but as Geoff pointed > out on github you could rely on some form of throttling to emulate > this functionality, but that requires a third-party module: > > https://github.com/varnishcache/varnish-cache/issues/3150#issuecomment-561307014 > > Be careful though, his example was a bit simplistic (I assume on > purpose to show the idea) and shouldn't be used as-is. Right, and also not tested. HTH nevertheless, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From hrhosseini at hotmail.com Mon Dec 9 10:44:27 2019 From: hrhosseini at hotmail.com (hamidreza hosseini) Date: Mon, 9 Dec 2019 10:44:27 +0000 Subject: Varnish doesnt PURGE urls Message-ID: Hi, I've two varnish, that one of them use ram as backend and another file as backend, requests first come to varnish(ram) and then go to another server in varnish(file) and then go to nginx, this is my diagram: Req=====> [varnish ram] =======> [varnish file] =====> [Nginx] And following file is my vcl configuration file that i'm using, but when i want to purge a url it doesnt work and it steel remains in cache, Should i chage my config file? What's wrong with varnish? Vcl config file: ### Naming note: use [0-9a-zA-Z_] only. no -. Or use camelCase ;-) vcl 4.0; import directors; backend varnish_malloc_01 { .host = "192.168.200.12"; .port = "8080"; } sub vcl_init { new varnish_cluster = directors.round_robin(); varnish_cluster.add_backend(varnish_malloc_01); } acl purge { "localhost"; "192.168.200.0"/24; ; } sub vcl_recv { sub vcl_recv { #set req.backend_hint = varnish_cluster.backend(); unset req.http.Cookie; if (req.method == "PURGE") { ### TODO: also purge from the backend if (!client.ip ~ purge) { return(synth(405,"Not allowed.")); } return (purge); } } sub vcl_backend_response { set beresp.ttl=48h; # Don't cache 404 responses if ( beresp.status == 404 ) { set beresp.ttl = 120s; set beresp.uncacheable = true; return (deliver); } } -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Mon Dec 9 19:48:42 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 9 Dec 2019 11:48:42 -0800 Subject: Varnish doesnt PURGE urls In-Reply-To: References: Message-ID: Hi, How do you tell that it's not purging? Cheers, -- Guillaume Quintard On Mon, Dec 9, 2019 at 2:45 AM hamidreza hosseini wrote: > Hi, > > I've two varnish, that one of them use ram as backend and another file as > backend, requests first come to varnish(ram) and then go to another server > in varnish(file) and then go to nginx, this is my diagram: > > > Req=====> [varnish ram] =======> [varnish file] =====> [Nginx] > > And following file is my vcl configuration file that i'm using, > but when i want to purge a url it doesnt work and it steel remains in > cache, > Should i chage my config file? > What's wrong with varnish? > > Vcl config file: > > ### Naming note: use [0-9a-zA-Z_] only. no -. Or use camelCase ;-) > vcl 4.0; > > import directors; > > > backend varnish_malloc_01 { .host = "192.168.200.12"; .port = "8080"; } > > > sub vcl_init { > new varnish_cluster = directors.round_robin(); > varnish_cluster.add_backend(varnish_malloc_01); > > } > > > acl purge { > "localhost"; > "192.168.200.0"/24; > ; > } > > > sub vcl_recv { > > sub vcl_recv { > #set req.backend_hint = varnish_cluster.backend(); > unset req.http.Cookie; > if (req.method == "PURGE") { > ### TODO: also purge from the backend > if (!client.ip ~ purge) { > return(synth(405,"Not allowed.")); > } > return (purge); > } > } > > sub vcl_backend_response { > set beresp.ttl=48h; > # Don't cache 404 responses > if ( beresp.status == 404 ) { > set beresp.ttl = 120s; > set beresp.uncacheable = true; > return (deliver); > } > } > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrhosseini at hotmail.com Mon Dec 9 10:55:14 2019 From: hrhosseini at hotmail.com (hamidreza hosseini) Date: Mon, 9 Dec 2019 10:55:14 +0000 Subject: Varnish doesnt PURGE urls In-Reply-To: References: , Message-ID: because its steel in cach and i can download and see th urls that i purged before! its Interesting that it return 200 code from varnish!!! Best regards. ________________________________ From: Guillaume Quintard Sent: Monday, December 9, 2019 11:48 AM To: hamidreza hosseini Cc: varnish-misc at varnish-cache.org Subject: Re: Varnish doesnt PURGE urls Hi, How do you tell that it's not purging? Cheers, -- Guillaume Quintard On Mon, Dec 9, 2019 at 2:45 AM hamidreza hosseini > wrote: Hi, I've two varnish, that one of them use ram as backend and another file as backend, requests first come to varnish(ram) and then go to another server in varnish(file) and then go to nginx, this is my diagram: Req=====> [varnish ram] =======> [varnish file] =====> [Nginx] And following file is my vcl configuration file that i'm using, but when i want to purge a url it doesnt work and it steel remains in cache, Should i chage my config file? What's wrong with varnish? Vcl config file: ### Naming note: use [0-9a-zA-Z_] only. no -. Or use camelCase ;-) vcl 4.0; import directors; backend varnish_malloc_01 { .host = "192.168.200.12"; .port = "8080"; } sub vcl_init { new varnish_cluster = directors.round_robin(); varnish_cluster.add_backend(varnish_malloc_01); } acl purge { "localhost"; "192.168.200.0"/24; ; } sub vcl_recv { sub vcl_recv { #set req.backend_hint = varnish_cluster.backend(); unset req.http.Cookie; if (req.method == "PURGE") { ### TODO: also purge from the backend if (!client.ip ~ purge) { return(synth(405,"Not allowed.")); } return (purge); } } sub vcl_backend_response { set beresp.ttl=48h; # Don't cache 404 responses if ( beresp.status == 404 ) { set beresp.ttl = 120s; set beresp.uncacheable = true; return (deliver); } } _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Mon Dec 9 19:57:50 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 9 Dec 2019 11:57:50 -0800 Subject: Varnish doesnt PURGE urls In-Reply-To: References: Message-ID: Are you sure that you purge both Varnish layers (file and RAM)? can you pastebin a varnishlog of the purge and of the subsequent request? -- Guillaume Quintard On Mon, Dec 9, 2019 at 2:55 AM hamidreza hosseini wrote: > because its steel in cach and i can download and see th urls that i purged > before! > its Interesting that it return 200 code from varnish!!! > > Best regards. > ------------------------------ > *From:* Guillaume Quintard > *Sent:* Monday, December 9, 2019 11:48 AM > *To:* hamidreza hosseini > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt PURGE urls > > Hi, > > How do you tell that it's not purging? > > Cheers, > > -- > Guillaume Quintard > > > On Mon, Dec 9, 2019 at 2:45 AM hamidreza hosseini > wrote: > > Hi, > > I've two varnish, that one of them use ram as backend and another file as > backend, requests first come to varnish(ram) and then go to another server > in varnish(file) and then go to nginx, this is my diagram: > > > Req=====> [varnish ram] =======> [varnish file] =====> [Nginx] > > And following file is my vcl configuration file that i'm using, > but when i want to purge a url it doesnt work and it steel remains in > cache, > Should i chage my config file? > What's wrong with varnish? > > Vcl config file: > > ### Naming note: use [0-9a-zA-Z_] only. no -. Or use camelCase ;-) > vcl 4.0; > > import directors; > > > backend varnish_malloc_01 { .host = "192.168.200.12"; .port = "8080"; } > > > sub vcl_init { > new varnish_cluster = directors.round_robin(); > varnish_cluster.add_backend(varnish_malloc_01); > > } > > > acl purge { > "localhost"; > "192.168.200.0"/24; > ; > } > > > sub vcl_recv { > > sub vcl_recv { > #set req.backend_hint = varnish_cluster.backend(); > unset req.http.Cookie; > if (req.method == "PURGE") { > ### TODO: also purge from the backend > if (!client.ip ~ purge) { > return(synth(405,"Not allowed.")); > } > return (purge); > } > } > > sub vcl_backend_response { > set beresp.ttl=48h; > # Don't cache 404 responses > if ( beresp.status == 404 ) { > set beresp.ttl = 120s; > set beresp.uncacheable = true; > return (deliver); > } > } > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrhosseini at hotmail.com Mon Dec 9 11:14:44 2019 From: hrhosseini at hotmail.com (hamidreza hosseini) Date: Mon, 9 Dec 2019 11:14:44 +0000 Subject: Varnish doesnt PURGE urls In-Reply-To: References: , Message-ID: Yes, surely i purge both servers This is my purge request: $curl -v -k -X PURGE http://localhost/Naserfeiz.mp4 * Trying ::1... * Connected to localhost (::1) port 80 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: localhost > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 200 Purged < Date: Mon, 09 Dec 2019 11:12:54 GMT < Server: Varnish < X-Varnish: 98306 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 240 < Accept-Ranges: bytes < Connection: keep-alive < 200 Purged

Error 200 Purged

Purged

Guru Meditation:

XID: 98306


Varnish cache server

* Connection #0 to host localhost left intact And this is varnishlog resault: * << BeReq >> 32771 - Begin bereq 32770 fetch - Timestamp Start: 1575889894.491843 0.000000 0.000000 - BereqMethod GET - BereqURL / - BereqProtocol HTTP/1.1 - BereqHeader Host: 192.168.200.13 - BereqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0 - BereqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 - BereqHeader Accept-Language: en-US,en;q=0.5 - BereqHeader Upgrade-Insecure-Requests: 1 - BereqHeader X-Forwarded-For: 172.16.1.3 - BereqHeader Accept-Encoding: gzip - BereqHeader X-Varnish: 32771 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 18 boot.varnish_malloc_01 192.168.200.12 8080 192.168.200.13 14088 - Timestamp Bereq: 1575889894.492444 0.000601 0.000601 - Timestamp Beresp: 1575889894.493276 0.001433 0.000832 - BerespProtocol HTTP/1.1 - BerespStatus 200 - BerespReason OK - BerespHeader Server: nginx - BerespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT - BerespHeader Content-Type: text/html - BerespHeader Content-Length: 340 - BerespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT - BerespHeader ETag: "5dcaad34-154" - BerespHeader X-Varnish: 8 6 - BerespHeader Age: 4270 - BerespHeader Via: 1.1 varnish-v4 - BerespHeader Accept-Ranges: bytes - BerespHeader Connection: keep-alive - TTL RFC 120 10 -1 1575889894 1575885624 1575885629 0 0 - VCL_call BACKEND_RESPONSE - TTL VCL 177070 10 0 1575885624 - VCL_return deliver - Storage malloc s0 - ObjProtocol HTTP/1.1 - ObjStatus 200 - ObjReason OK - ObjHeader Server: nginx - ObjHeader Date: Mon, 09 Dec 2019 10:00:29 GMT - ObjHeader Content-Type: text/html - ObjHeader Content-Length: 340 - ObjHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT - ObjHeader ETag: "5dcaad34-154" - ObjHeader X-Varnish: 8 6 - ObjHeader Via: 1.1 varnish-v4 - Fetch_Body 3 length stream - BackendReuse 18 boot.varnish_malloc_01 - Timestamp BerespBody: 1575889894.493474 0.001631 0.000198 - Length 340 - BereqAcct 336 0 336 279 340 619 - End * << Request >> 32770 - Begin req 32769 rxreq - Timestamp Start: 1575889894.491659 0.000000 0.000000 - Timestamp Req: 1575889894.491659 0.000000 0.000000 - ReqStart 172.16.1.3 42502 - ReqMethod GET - ReqURL / - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.13 - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0 - ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 - ReqHeader Accept-Language: en-US,en;q=0.5 - ReqHeader Accept-Encoding: gzip, deflate - ReqHeader Connection: keep-alive - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - ReqHeader Upgrade-Insecure-Requests: 1 - ReqHeader If-Modified-Since: Tue, 12 Nov 2019 13:01:40 GMT - ReqHeader If-None-Match: "5dcaad34-154" - ReqHeader Cache-Control: max-age=0 - ReqHeader X-Forwarded-For: 172.16.1.3 - VCL_call RECV - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - VCL_return hash - ReqUnset Accept-Encoding: gzip, deflate - ReqHeader Accept-Encoding: gzip - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 32771 fetch - Timestamp Fetch: 1575889894.493501 0.001842 0.001842 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT - RespHeader Content-Type: text/html - RespHeader Content-Length: 340 - RespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT - RespHeader ETag: "5dcaad34-154" - RespHeader X-Varnish: 8 6 - RespHeader Via: 1.1 varnish-v4 - RespHeader X-Varnish: 32770 - RespHeader Age: 4270 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1575889894.493541 0.001881 0.000039 - RespProtocol HTTP/1.1 - RespStatus 304 - RespReason Not Modified - RespReason Not Modified - RespUnset Content-Length: 340 - Debug "RES_MODE 0" - RespHeader Connection: keep-alive - Timestamp Resp: 1575889894.493599 0.001940 0.000058 - ReqAcct 503 0 503 285 0 285 - End * << Request >> 65538 - Begin req 65537 rxreq - Timestamp Start: 1575889962.124164 0.000000 0.000000 - Timestamp Req: 1575889962.124164 0.000000 0.000000 - ReqStart ::1 62064 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575889962.124300 0.000136 0.000136 - RespHeader Date: Mon, 09 Dec 2019 11:12:42 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 65538 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575889962.124593 0.000428 0.000292 - ReqAcct 88 0 88 218 240 458 - End * << Session >> 65537 - Begin sess 0 HTTP/1 - SessOpen ::1 62064 :80 ::1 80 1575889962.124081 13 - Link req 65538 rxreq - SessClose REM_CLOSE 0.001 - End * << Request >> 98306 - Begin req 98305 rxreq - Timestamp Start: 1575889974.116017 0.000000 0.000000 - Timestamp Req: 1575889974.116017 0.000000 0.000000 - ReqStart ::1 62066 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575889974.116128 0.000111 0.000111 - RespHeader Date: Mon, 09 Dec 2019 11:12:54 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 98306 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575889974.116288 0.000271 0.000160 - ReqAcct 88 0 88 218 240 458 - End * << Session >> 98305 - Begin sess 0 HTTP/1 - SessOpen ::1 62066 :80 ::1 80 1575889974.115935 15 - Link req 98306 rxreq - SessClose REM_CLOSE 0.002 - End * << Session >> 32769 - Begin sess 0 HTTP/1 - SessOpen 172.16.1.3 42502 :80 192.168.200.13 80 1575889894.491572 14 - Link req 32770 rxreq - Link req 32772 rxreq - VSL timeout - End synth * << Request >> 32772 - Begin req 32769 rxreq - Timestamp Start: 1575889894.845556 0.000000 0.000000 - Timestamp Req: 1575889894.845556 0.000000 0.000000 - ReqStart 172.16.1.3 42502 - ReqMethod GET - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.13 - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0 - ReqHeader Accept: video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5 - ReqHeader Accept-Language: en-US,en;q=0.5 - ReqHeader Range: bytes=205560- - ReqHeader Connection: keep-alive - ReqHeader Referer: http://192.168.200.13/ - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - ReqHeader X-Forwarded-For: 172.16.1.3 - VCL_call RECV - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - VCL_return hash - VCL_call HASH - VCL_return lookup - Hit 3 - VCL_call HIT - VCL_return deliver - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Mon, 09 Dec 2019 11:06:51 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 32770 - RespHeader Via: 1.1 varnish-v4 - RespHeader X-Varnish: 32772 3 - RespHeader Age: 288 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1575889894.845683 0.000127 0.000127 - RespHeader Accept-Ranges: bytes - RespHeader Content-Range: bytes 205560-23521498/23521499 - RespProtocol HTTP/1.1 - RespStatus 206 - RespReason Partial Content - RespReason Partial Content - RespUnset Content-Length: 23521499 - RespHeader Content-Length: 23315939 - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Debug "Hit idle send timeout, wrote = 10899908/23316330; retrying" - Debug "Hit idle send timeout, wrote = 258432/12416422; retrying" - Debug "Write error, retval = -1, len = 12157990, errno = Resource temporarily unavailable" - Timestamp Resp: 1575890074.846999 180.001443 180.001316 - ReqAcct 419 0 419 391 23315939 23316330 - End ########################## ________________________________ From: Guillaume Quintard Sent: Monday, December 9, 2019 11:57 AM To: hamidreza hosseini Cc: varnish-misc at varnish-cache.org Subject: Re: Varnish doesnt PURGE urls Are you sure that you purge both Varnish layers (file and RAM)? can you pastebin a varnishlog of the purge and of the subsequent request? -- Guillaume Quintard On Mon, Dec 9, 2019 at 2:55 AM hamidreza hosseini > wrote: because its steel in cach and i can download and see th urls that i purged before! its Interesting that it return 200 code from varnish!!! Best regards. ________________________________ From: Guillaume Quintard > Sent: Monday, December 9, 2019 11:48 AM To: hamidreza hosseini > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt PURGE urls Hi, How do you tell that it's not purging? Cheers, -- Guillaume Quintard On Mon, Dec 9, 2019 at 2:45 AM hamidreza hosseini > wrote: Hi, I've two varnish, that one of them use ram as backend and another file as backend, requests first come to varnish(ram) and then go to another server in varnish(file) and then go to nginx, this is my diagram: Req=====> [varnish ram] =======> [varnish file] =====> [Nginx] And following file is my vcl configuration file that i'm using, but when i want to purge a url it doesnt work and it steel remains in cache, Should i chage my config file? What's wrong with varnish? Vcl config file: ### Naming note: use [0-9a-zA-Z_] only. no -. Or use camelCase ;-) vcl 4.0; import directors; backend varnish_malloc_01 { .host = "192.168.200.12"; .port = "8080"; } sub vcl_init { new varnish_cluster = directors.round_robin(); varnish_cluster.add_backend(varnish_malloc_01); } acl purge { "localhost"; "192.168.200.0"/24; ; } sub vcl_recv { sub vcl_recv { #set req.backend_hint = varnish_cluster.backend(); unset req.http.Cookie; if (req.method == "PURGE") { ### TODO: also purge from the backend if (!client.ip ~ purge) { return(synth(405,"Not allowed.")); } return (purge); } } sub vcl_backend_response { set beresp.ttl=48h; # Don't cache 404 responses if ( beresp.status == 404 ) { set beresp.ttl = 120s; set beresp.uncacheable = true; return (deliver); } } _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From shafeequeaslam at gmail.com Mon Dec 9 11:19:21 2019 From: shafeequeaslam at gmail.com (shafeeque aslam) Date: Mon, 9 Dec 2019 16:49:21 +0530 Subject: Varnish doesnt PURGE urls In-Reply-To: References: Message-ID: This is purge to 1st Varnish server only. Give another purge request to 2nd Varnish server giving it's port number. On Mon 9 Dec, 2019, 4:45 PM hamidreza hosseini, wrote: > Yes, surely i purge both servers > This is my purge request: > > $curl -v -k -X PURGE http://localhost/Naserfeiz.mp4 > > * Trying ::1... > * Connected to localhost (::1) port 80 (#0) > > PURGE /Naserfeiz.mp4 HTTP/1.1 > > Host: localhost > > User-Agent: curl/7.47.0 > > Accept: */* > > > < HTTP/1.1 200 Purged > < Date: Mon, 09 Dec 2019 11:12:54 GMT > < Server: Varnish > < X-Varnish: 98306 > < Content-Type: text/html; charset=utf-8 > < Retry-After: 5 > < Content-Length: 240 > < Accept-Ranges: bytes > < Connection: keep-alive > < > > > > 200 Purged > > >

Error 200 Purged

>

Purged

>

Guru Meditation:

>

XID: 98306

>
>

Varnish cache server

> > > * Connection #0 to host localhost left intact > > > And this is varnishlog resault: > > * << BeReq >> 32771 > - Begin bereq 32770 fetch > - Timestamp Start: 1575889894.491843 0.000000 0.000000 > - BereqMethod GET > - BereqURL / > - BereqProtocol HTTP/1.1 > - BereqHeader Host: 192.168.200.13 > - BereqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; > rv:70.0) Gecko/20100101 Firefox/70.0 > - BereqHeader Accept: > text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > - BereqHeader Accept-Language: en-US,en;q=0.5 > - BereqHeader Upgrade-Insecure-Requests: 1 > - BereqHeader X-Forwarded-For: 172.16.1.3 > - BereqHeader Accept-Encoding: gzip > - BereqHeader X-Varnish: 32771 > - VCL_call BACKEND_FETCH > - VCL_return fetch > - BackendOpen 18 boot.varnish_malloc_01 192.168.200.12 8080 > 192.168.200.13 14088 > - Timestamp Bereq: 1575889894.492444 0.000601 0.000601 > - Timestamp Beresp: 1575889894.493276 0.001433 0.000832 > - BerespProtocol HTTP/1.1 > - BerespStatus 200 > - BerespReason OK > - BerespHeader Server: nginx > - BerespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT > - BerespHeader Content-Type: text/html > - BerespHeader Content-Length: 340 > - BerespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT > - BerespHeader ETag: "5dcaad34-154" > - BerespHeader X-Varnish: 8 6 > - BerespHeader Age: 4270 > - BerespHeader Via: 1.1 varnish-v4 > - BerespHeader Accept-Ranges: bytes > - BerespHeader Connection: keep-alive > - TTL RFC 120 10 -1 1575889894 1575885624 1575885629 0 0 > - VCL_call BACKEND_RESPONSE > - TTL VCL 177070 10 0 1575885624 > - VCL_return deliver > - Storage malloc s0 > - ObjProtocol HTTP/1.1 > - ObjStatus 200 > - ObjReason OK > - ObjHeader Server: nginx > - ObjHeader Date: Mon, 09 Dec 2019 10:00:29 GMT > - ObjHeader Content-Type: text/html > - ObjHeader Content-Length: 340 > - ObjHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT > - ObjHeader ETag: "5dcaad34-154" > - ObjHeader X-Varnish: 8 6 > - ObjHeader Via: 1.1 varnish-v4 > - Fetch_Body 3 length stream > - BackendReuse 18 boot.varnish_malloc_01 > - Timestamp BerespBody: 1575889894.493474 0.001631 0.000198 > - Length 340 > - BereqAcct 336 0 336 279 340 619 > - End > > * << Request >> 32770 > - Begin req 32769 rxreq > - Timestamp Start: 1575889894.491659 0.000000 0.000000 > - Timestamp Req: 1575889894.491659 0.000000 0.000000 > - ReqStart 172.16.1.3 42502 > - ReqMethod GET > - ReqURL / > - ReqProtocol HTTP/1.1 > - ReqHeader Host: 192.168.200.13 > - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; > rv:70.0) Gecko/20100101 Firefox/70.0 > - ReqHeader Accept: > text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > - ReqHeader Accept-Language: en-US,en;q=0.5 > - ReqHeader Accept-Encoding: gzip, deflate > - ReqHeader Connection: keep-alive > - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; > _gid=GA1.1.117112441.1575877103 > - ReqHeader Upgrade-Insecure-Requests: 1 > - ReqHeader If-Modified-Since: Tue, 12 Nov 2019 13:01:40 GMT > - ReqHeader If-None-Match: "5dcaad34-154" > - ReqHeader Cache-Control: max-age=0 > - ReqHeader X-Forwarded-For: 172.16.1.3 > - VCL_call RECV > - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; > _gid=GA1.1.117112441.1575877103 > - VCL_return hash > - ReqUnset Accept-Encoding: gzip, deflate > - ReqHeader Accept-Encoding: gzip > - VCL_call HASH > - VCL_return lookup > - VCL_call MISS > - VCL_return fetch > - Link bereq 32771 fetch > - Timestamp Fetch: 1575889894.493501 0.001842 0.001842 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespHeader Server: nginx > - RespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT > - RespHeader Content-Type: text/html > - RespHeader Content-Length: 340 > - RespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT > - RespHeader ETag: "5dcaad34-154" > - RespHeader X-Varnish: 8 6 > - RespHeader Via: 1.1 varnish-v4 > - RespHeader X-Varnish: 32770 > - RespHeader Age: 4270 > - RespHeader Via: 1.1 varnish-v4 > - VCL_call DELIVER > - VCL_return deliver > - Timestamp Process: 1575889894.493541 0.001881 0.000039 > - RespProtocol HTTP/1.1 > - RespStatus 304 > - RespReason Not Modified > - RespReason Not Modified > - RespUnset Content-Length: 340 > - Debug "RES_MODE 0" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1575889894.493599 0.001940 0.000058 > - ReqAcct 503 0 503 285 0 285 > - End > > * << Request >> 65538 > - Begin req 65537 rxreq > - Timestamp Start: 1575889962.124164 0.000000 0.000000 > - Timestamp Req: 1575889962.124164 0.000000 0.000000 > - ReqStart ::1 62064 > - ReqMethod PURGE > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: localhost > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader X-Forwarded-For: ::1 > - VCL_call RECV > - VCL_acl MATCH purge "localhost" > - VCL_return purge > - VCL_call HASH > - VCL_return lookup > - VCL_call PURGE > - VCL_return synth > - Timestamp Process: 1575889962.124300 0.000136 0.000136 > - RespHeader Date: Mon, 09 Dec 2019 11:12:42 GMT > - RespHeader Server: Varnish > - RespHeader X-Varnish: 65538 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespReason Purged > - VCL_call SYNTH > - RespHeader Content-Type: text/html; charset=utf-8 > - RespHeader Retry-After: 5 > - VCL_return deliver > - RespHeader Content-Length: 240 > - Storage malloc Transient > - RespHeader Accept-Ranges: bytes > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1575889962.124593 0.000428 0.000292 > - ReqAcct 88 0 88 218 240 458 > - End > > * << Session >> 65537 > - Begin sess 0 HTTP/1 > - SessOpen ::1 62064 :80 ::1 80 1575889962.124081 13 > - Link req 65538 rxreq > - SessClose REM_CLOSE 0.001 > - End > > * << Request >> 98306 > - Begin req 98305 rxreq > - Timestamp Start: 1575889974.116017 0.000000 0.000000 > - Timestamp Req: 1575889974.116017 0.000000 0.000000 > - ReqStart ::1 62066 > - ReqMethod PURGE > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: localhost > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader X-Forwarded-For: ::1 > - VCL_call RECV > - VCL_acl MATCH purge "localhost" > - VCL_return purge > - VCL_call HASH > - VCL_return lookup > - VCL_call PURGE > - VCL_return synth > - Timestamp Process: 1575889974.116128 0.000111 0.000111 > - RespHeader Date: Mon, 09 Dec 2019 11:12:54 GMT > - RespHeader Server: Varnish > - RespHeader X-Varnish: 98306 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespReason Purged > - VCL_call SYNTH > - RespHeader Content-Type: text/html; charset=utf-8 > - RespHeader Retry-After: 5 > - VCL_return deliver > - RespHeader Content-Length: 240 > - Storage malloc Transient > - RespHeader Accept-Ranges: bytes > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1575889974.116288 0.000271 0.000160 > - ReqAcct 88 0 88 218 240 458 > - End > > * << Session >> 98305 > - Begin sess 0 HTTP/1 > - SessOpen ::1 62066 :80 ::1 80 1575889974.115935 15 > - Link req 98306 rxreq > - SessClose REM_CLOSE 0.002 > - End > > * << Session >> 32769 > - Begin sess 0 HTTP/1 > - SessOpen 172.16.1.3 42502 :80 192.168.200.13 80 > 1575889894.491572 14 > - Link req 32770 rxreq > - Link req 32772 rxreq > - VSL timeout > - End synth > > * << Request >> 32772 > - Begin req 32769 rxreq > - Timestamp Start: 1575889894.845556 0.000000 0.000000 > - Timestamp Req: 1575889894.845556 0.000000 0.000000 > - ReqStart 172.16.1.3 42502 > - ReqMethod GET > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: 192.168.200.13 > - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; > rv:70.0) Gecko/20100101 Firefox/70.0 > - ReqHeader Accept: > video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5 > - ReqHeader Accept-Language: en-US,en;q=0.5 > - ReqHeader Range: bytes=205560- > - ReqHeader Connection: keep-alive > - ReqHeader Referer: http://192.168.200.13/ > - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; > _gid=GA1.1.117112441.1575877103 > - ReqHeader X-Forwarded-For: 172.16.1.3 > - VCL_call RECV > - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; > _gid=GA1.1.117112441.1575877103 > - VCL_return hash > - VCL_call HASH > - VCL_return lookup > - Hit 3 > - VCL_call HIT > - VCL_return deliver > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespHeader Server: nginx > - RespHeader Date: Mon, 09 Dec 2019 11:06:51 GMT > - RespHeader Content-Type: text/plain > - RespHeader Content-Length: 23521499 > - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > - RespHeader ETag: "3a1794b0-166e8db" > - RespHeader X-Varnish: 32770 > - RespHeader Via: 1.1 varnish-v4 > - RespHeader X-Varnish: 32772 3 > - RespHeader Age: 288 > - RespHeader Via: 1.1 varnish-v4 > - VCL_call DELIVER > - VCL_return deliver > - Timestamp Process: 1575889894.845683 0.000127 0.000127 > - RespHeader Accept-Ranges: bytes > - RespHeader Content-Range: bytes 205560-23521498/23521499 > - RespProtocol HTTP/1.1 > - RespStatus 206 > - RespReason Partial Content > - RespReason Partial Content > - RespUnset Content-Length: 23521499 > - RespHeader Content-Length: 23315939 > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Debug "Hit idle send timeout, wrote = 10899908/23316330; > retrying" > - Debug "Hit idle send timeout, wrote = 258432/12416422; > retrying" > - Debug "Write error, retval = -1, len = 12157990, errno = > Resource temporarily unavailable" > - Timestamp Resp: 1575890074.846999 180.001443 180.001316 > - ReqAcct 419 0 419 391 23315939 23316330 > - End > > ########################## > ------------------------------ > *From:* Guillaume Quintard > *Sent:* Monday, December 9, 2019 11:57 AM > *To:* hamidreza hosseini > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt PURGE urls > > Are you sure that you purge both Varnish layers (file and RAM)? > > can you pastebin a varnishlog of the purge and of the subsequent request? > > -- > Guillaume Quintard > > > On Mon, Dec 9, 2019 at 2:55 AM hamidreza hosseini > wrote: > > because its steel in cach and i can download and see th urls that i purged > before! > its Interesting that it return 200 code from varnish!!! > > Best regards. > ------------------------------ > *From:* Guillaume Quintard > *Sent:* Monday, December 9, 2019 11:48 AM > *To:* hamidreza hosseini > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt PURGE urls > > Hi, > > How do you tell that it's not purging? > > Cheers, > > -- > Guillaume Quintard > > > On Mon, Dec 9, 2019 at 2:45 AM hamidreza hosseini > wrote: > > Hi, > > I've two varnish, that one of them use ram as backend and another file as > backend, requests first come to varnish(ram) and then go to another server > in varnish(file) and then go to nginx, this is my diagram: > > > Req=====> [varnish ram] =======> [varnish file] =====> [Nginx] > > And following file is my vcl configuration file that i'm using, > but when i want to purge a url it doesnt work and it steel remains in > cache, > Should i chage my config file? > What's wrong with varnish? > > Vcl config file: > > ### Naming note: use [0-9a-zA-Z_] only. no -. Or use camelCase ;-) > vcl 4.0; > > import directors; > > > backend varnish_malloc_01 { .host = "192.168.200.12"; .port = "8080"; } > > > sub vcl_init { > new varnish_cluster = directors.round_robin(); > varnish_cluster.add_backend(varnish_malloc_01); > > } > > > acl purge { > "localhost"; > "192.168.200.0"/24; > ; > } > > > sub vcl_recv { > > sub vcl_recv { > #set req.backend_hint = varnish_cluster.backend(); > unset req.http.Cookie; > if (req.method == "PURGE") { > ### TODO: also purge from the backend > if (!client.ip ~ purge) { > return(synth(405,"Not allowed.")); > } > return (purge); > } > } > > sub vcl_backend_response { > set beresp.ttl=48h; > # Don't cache 404 responses > if ( beresp.status == 404 ) { > set beresp.ttl = 120s; > set beresp.uncacheable = true; > return (deliver); > } > } > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrhosseini at hotmail.com Mon Dec 9 11:24:16 2019 From: hrhosseini at hotmail.com (hamidreza hosseini) Date: Mon, 9 Dec 2019 11:24:16 +0000 Subject: Varnish doesnt PURGE urls In-Reply-To: References: , Message-ID: This is purge to another server: $curl -v -k -X PURGE http://localhost:8080/Naserfeiz.mp4 * Trying ::1... * Connected to localhost (::1) port 8080 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: localhost:8080 > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 200 Purged < Date: Mon, 09 Dec 2019 11:20:52 GMT < Server: Varnish < X-Varnish: 10 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 237 < Accept-Ranges: bytes < Connection: keep-alive < 200 Purged

Error 200 Purged

Purged

Guru Meditation:

XID: 10


Varnish cache server

* Connection #0 to host localhost left intact And varnishlog resault: $ sudo varnishlog * << Request >> 32773 - Begin req 32772 rxreq - Timestamp Start: 1575890593.662132 0.000000 0.000000 - Timestamp Req: 1575890593.662132 0.000000 0.000000 - ReqStart ::1 63184 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575890593.662268 0.000135 0.000135 - RespHeader Date: Mon, 09 Dec 2019 11:23:13 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 32773 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575890593.662484 0.000352 0.000216 - ReqAcct 93 0 93 218 240 458 - End * << Session >> 32772 - Begin sess 0 HTTP/1 - SessOpen ::1 63184 :8080 ::1 8080 1575890593.662044 16 - Link req 32773 rxreq - SessClose REM_CLOSE 0.001 - End * << Request >> 12 - Begin req 11 rxreq - Timestamp Start: 1575890600.799371 0.000000 0.000000 - Timestamp Req: 1575890600.799371 0.000000 0.000000 - ReqStart ::1 63186 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575890600.799463 0.000092 0.000092 - RespHeader Date: Mon, 09 Dec 2019 11:23:20 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 12 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 237 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575890600.799611 0.000240 0.000147 - ReqAcct 93 0 93 215 237 452 - End * << Session >> 11 - Begin sess 0 HTTP/1 - SessOpen ::1 63186 :8080 ::1 8080 1575890600.799261 16 - Link req 12 rxreq - SessClose REM_CLOSE 0.001 - End ________________________________ From: shafeeque aslam Sent: Monday, December 9, 2019 3:19 AM To: hamidreza hosseini Cc: Guillaume Quintard ; varnish-misc at varnish-cache.org Subject: Re: Varnish doesnt PURGE urls This is purge to 1st Varnish server only. Give another purge request to 2nd Varnish server giving it's port number. On Mon 9 Dec, 2019, 4:45 PM hamidreza hosseini, > wrote: Yes, surely i purge both servers This is my purge request: $curl -v -k -X PURGE http://localhost/Naserfeiz.mp4 * Trying ::1... * Connected to localhost (::1) port 80 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: localhost > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 200 Purged < Date: Mon, 09 Dec 2019 11:12:54 GMT < Server: Varnish < X-Varnish: 98306 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 240 < Accept-Ranges: bytes < Connection: keep-alive < 200 Purged

Error 200 Purged

Purged

Guru Meditation:

XID: 98306


Varnish cache server

* Connection #0 to host localhost left intact And this is varnishlog resault: * << BeReq >> 32771 - Begin bereq 32770 fetch - Timestamp Start: 1575889894.491843 0.000000 0.000000 - BereqMethod GET - BereqURL / - BereqProtocol HTTP/1.1 - BereqHeader Host: 192.168.200.13 - BereqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0 - BereqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 - BereqHeader Accept-Language: en-US,en;q=0.5 - BereqHeader Upgrade-Insecure-Requests: 1 - BereqHeader X-Forwarded-For: 172.16.1.3 - BereqHeader Accept-Encoding: gzip - BereqHeader X-Varnish: 32771 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 18 boot.varnish_malloc_01 192.168.200.12 8080 192.168.200.13 14088 - Timestamp Bereq: 1575889894.492444 0.000601 0.000601 - Timestamp Beresp: 1575889894.493276 0.001433 0.000832 - BerespProtocol HTTP/1.1 - BerespStatus 200 - BerespReason OK - BerespHeader Server: nginx - BerespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT - BerespHeader Content-Type: text/html - BerespHeader Content-Length: 340 - BerespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT - BerespHeader ETag: "5dcaad34-154" - BerespHeader X-Varnish: 8 6 - BerespHeader Age: 4270 - BerespHeader Via: 1.1 varnish-v4 - BerespHeader Accept-Ranges: bytes - BerespHeader Connection: keep-alive - TTL RFC 120 10 -1 1575889894 1575885624 1575885629 0 0 - VCL_call BACKEND_RESPONSE - TTL VCL 177070 10 0 1575885624 - VCL_return deliver - Storage malloc s0 - ObjProtocol HTTP/1.1 - ObjStatus 200 - ObjReason OK - ObjHeader Server: nginx - ObjHeader Date: Mon, 09 Dec 2019 10:00:29 GMT - ObjHeader Content-Type: text/html - ObjHeader Content-Length: 340 - ObjHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT - ObjHeader ETag: "5dcaad34-154" - ObjHeader X-Varnish: 8 6 - ObjHeader Via: 1.1 varnish-v4 - Fetch_Body 3 length stream - BackendReuse 18 boot.varnish_malloc_01 - Timestamp BerespBody: 1575889894.493474 0.001631 0.000198 - Length 340 - BereqAcct 336 0 336 279 340 619 - End * << Request >> 32770 - Begin req 32769 rxreq - Timestamp Start: 1575889894.491659 0.000000 0.000000 - Timestamp Req: 1575889894.491659 0.000000 0.000000 - ReqStart 172.16.1.3 42502 - ReqMethod GET - ReqURL / - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.13 - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0 - ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 - ReqHeader Accept-Language: en-US,en;q=0.5 - ReqHeader Accept-Encoding: gzip, deflate - ReqHeader Connection: keep-alive - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - ReqHeader Upgrade-Insecure-Requests: 1 - ReqHeader If-Modified-Since: Tue, 12 Nov 2019 13:01:40 GMT - ReqHeader If-None-Match: "5dcaad34-154" - ReqHeader Cache-Control: max-age=0 - ReqHeader X-Forwarded-For: 172.16.1.3 - VCL_call RECV - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - VCL_return hash - ReqUnset Accept-Encoding: gzip, deflate - ReqHeader Accept-Encoding: gzip - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 32771 fetch - Timestamp Fetch: 1575889894.493501 0.001842 0.001842 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT - RespHeader Content-Type: text/html - RespHeader Content-Length: 340 - RespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT - RespHeader ETag: "5dcaad34-154" - RespHeader X-Varnish: 8 6 - RespHeader Via: 1.1 varnish-v4 - RespHeader X-Varnish: 32770 - RespHeader Age: 4270 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1575889894.493541 0.001881 0.000039 - RespProtocol HTTP/1.1 - RespStatus 304 - RespReason Not Modified - RespReason Not Modified - RespUnset Content-Length: 340 - Debug "RES_MODE 0" - RespHeader Connection: keep-alive - Timestamp Resp: 1575889894.493599 0.001940 0.000058 - ReqAcct 503 0 503 285 0 285 - End * << Request >> 65538 - Begin req 65537 rxreq - Timestamp Start: 1575889962.124164 0.000000 0.000000 - Timestamp Req: 1575889962.124164 0.000000 0.000000 - ReqStart ::1 62064 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575889962.124300 0.000136 0.000136 - RespHeader Date: Mon, 09 Dec 2019 11:12:42 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 65538 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575889962.124593 0.000428 0.000292 - ReqAcct 88 0 88 218 240 458 - End * << Session >> 65537 - Begin sess 0 HTTP/1 - SessOpen ::1 62064 :80 ::1 80 1575889962.124081 13 - Link req 65538 rxreq - SessClose REM_CLOSE 0.001 - End * << Request >> 98306 - Begin req 98305 rxreq - Timestamp Start: 1575889974.116017 0.000000 0.000000 - Timestamp Req: 1575889974.116017 0.000000 0.000000 - ReqStart ::1 62066 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575889974.116128 0.000111 0.000111 - RespHeader Date: Mon, 09 Dec 2019 11:12:54 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 98306 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575889974.116288 0.000271 0.000160 - ReqAcct 88 0 88 218 240 458 - End * << Session >> 98305 - Begin sess 0 HTTP/1 - SessOpen ::1 62066 :80 ::1 80 1575889974.115935 15 - Link req 98306 rxreq - SessClose REM_CLOSE 0.002 - End * << Session >> 32769 - Begin sess 0 HTTP/1 - SessOpen 172.16.1.3 42502 :80 192.168.200.13 80 1575889894.491572 14 - Link req 32770 rxreq - Link req 32772 rxreq - VSL timeout - End synth * << Request >> 32772 - Begin req 32769 rxreq - Timestamp Start: 1575889894.845556 0.000000 0.000000 - Timestamp Req: 1575889894.845556 0.000000 0.000000 - ReqStart 172.16.1.3 42502 - ReqMethod GET - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.13 - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0 - ReqHeader Accept: video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5 - ReqHeader Accept-Language: en-US,en;q=0.5 - ReqHeader Range: bytes=205560- - ReqHeader Connection: keep-alive - ReqHeader Referer: http://192.168.200.13/ - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - ReqHeader X-Forwarded-For: 172.16.1.3 - VCL_call RECV - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - VCL_return hash - VCL_call HASH - VCL_return lookup - Hit 3 - VCL_call HIT - VCL_return deliver - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Mon, 09 Dec 2019 11:06:51 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 32770 - RespHeader Via: 1.1 varnish-v4 - RespHeader X-Varnish: 32772 3 - RespHeader Age: 288 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1575889894.845683 0.000127 0.000127 - RespHeader Accept-Ranges: bytes - RespHeader Content-Range: bytes 205560-23521498/23521499 - RespProtocol HTTP/1.1 - RespStatus 206 - RespReason Partial Content - RespReason Partial Content - RespUnset Content-Length: 23521499 - RespHeader Content-Length: 23315939 - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Debug "Hit idle send timeout, wrote = 10899908/23316330; retrying" - Debug "Hit idle send timeout, wrote = 258432/12416422; retrying" - Debug "Write error, retval = -1, len = 12157990, errno = Resource temporarily unavailable" - Timestamp Resp: 1575890074.846999 180.001443 180.001316 - ReqAcct 419 0 419 391 23315939 23316330 - End ########################## ________________________________ From: Guillaume Quintard > Sent: Monday, December 9, 2019 11:57 AM To: hamidreza hosseini > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt PURGE urls Are you sure that you purge both Varnish layers (file and RAM)? can you pastebin a varnishlog of the purge and of the subsequent request? -- Guillaume Quintard On Mon, Dec 9, 2019 at 2:55 AM hamidreza hosseini > wrote: because its steel in cach and i can download and see th urls that i purged before! its Interesting that it return 200 code from varnish!!! Best regards. ________________________________ From: Guillaume Quintard > Sent: Monday, December 9, 2019 11:48 AM To: hamidreza hosseini > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt PURGE urls Hi, How do you tell that it's not purging? Cheers, -- Guillaume Quintard On Mon, Dec 9, 2019 at 2:45 AM hamidreza hosseini > wrote: Hi, I've two varnish, that one of them use ram as backend and another file as backend, requests first come to varnish(ram) and then go to another server in varnish(file) and then go to nginx, this is my diagram: Req=====> [varnish ram] =======> [varnish file] =====> [Nginx] And following file is my vcl configuration file that i'm using, but when i want to purge a url it doesnt work and it steel remains in cache, Should i chage my config file? What's wrong with varnish? Vcl config file: ### Naming note: use [0-9a-zA-Z_] only. no -. Or use camelCase ;-) vcl 4.0; import directors; backend varnish_malloc_01 { .host = "192.168.200.12"; .port = "8080"; } sub vcl_init { new varnish_cluster = directors.round_robin(); varnish_cluster.add_backend(varnish_malloc_01); } acl purge { "localhost"; "192.168.200.0"/24; ; } sub vcl_recv { sub vcl_recv { #set req.backend_hint = varnish_cluster.backend(); unset req.http.Cookie; if (req.method == "PURGE") { ### TODO: also purge from the backend if (!client.ip ~ purge) { return(synth(405,"Not allowed.")); } return (purge); } } sub vcl_backend_response { set beresp.ttl=48h; # Don't cache 404 responses if ( beresp.status == 404 ) { set beresp.ttl = 120s; set beresp.uncacheable = true; return (deliver); } } _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Mon Dec 9 20:31:30 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 9 Dec 2019 12:31:30 -0800 Subject: Varnish doesnt PURGE urls In-Reply-To: References: Message-ID: Purge uses the hash of the objects to clear them. In turn, the hash, by default depends on the host of the requests that created the object. So, when you purge using localhost/some/path.mp4, you dont purge the object referring to my.website.com/some/path.mp4 -- Guillaume Quintard On Mon, Dec 9, 2019 at 3:24 AM hamidreza hosseini wrote: > This is purge to another server: > > $curl -v -k -X PURGE http://localhost:8080/Naserfeiz.mp4 > > * Trying ::1... > * Connected to localhost (::1) port 8080 (#0) > > PURGE /Naserfeiz.mp4 HTTP/1.1 > > Host: localhost:8080 > > User-Agent: curl/7.47.0 > > Accept: */* > > > < HTTP/1.1 200 Purged > < Date: Mon, 09 Dec 2019 11:20:52 GMT > < Server: Varnish > < X-Varnish: 10 > < Content-Type: text/html; charset=utf-8 > < Retry-After: 5 > < Content-Length: 237 > < Accept-Ranges: bytes > < Connection: keep-alive > < > > > > 200 Purged > > >

Error 200 Purged

>

Purged

>

Guru Meditation:

>

XID: 10

>
>

Varnish cache server

> > > * Connection #0 to host localhost left intact > > And varnishlog resault: > > $ sudo varnishlog > > * << Request >> 32773 > - Begin req 32772 rxreq > - Timestamp Start: 1575890593.662132 0.000000 0.000000 > - Timestamp Req: 1575890593.662132 0.000000 0.000000 > - ReqStart ::1 63184 > - ReqMethod PURGE > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: localhost:8080 > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader X-Forwarded-For: ::1 > - VCL_call RECV > - VCL_acl MATCH purge "localhost" > - VCL_return purge > - VCL_call HASH > - VCL_return lookup > - VCL_call PURGE > - VCL_return synth > - Timestamp Process: 1575890593.662268 0.000135 0.000135 > - RespHeader Date: Mon, 09 Dec 2019 11:23:13 GMT > - RespHeader Server: Varnish > - RespHeader X-Varnish: 32773 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespReason Purged > - VCL_call SYNTH > - RespHeader Content-Type: text/html; charset=utf-8 > - RespHeader Retry-After: 5 > - VCL_return deliver > - RespHeader Content-Length: 240 > - Storage malloc Transient > - RespHeader Accept-Ranges: bytes > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1575890593.662484 0.000352 0.000216 > - ReqAcct 93 0 93 218 240 458 > - End > > * << Session >> 32772 > - Begin sess 0 HTTP/1 > - SessOpen ::1 63184 :8080 ::1 8080 1575890593.662044 16 > - Link req 32773 rxreq > - SessClose REM_CLOSE 0.001 > - End > > * << Request >> 12 > - Begin req 11 rxreq > - Timestamp Start: 1575890600.799371 0.000000 0.000000 > - Timestamp Req: 1575890600.799371 0.000000 0.000000 > - ReqStart ::1 63186 > - ReqMethod PURGE > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: localhost:8080 > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader X-Forwarded-For: ::1 > - VCL_call RECV > - VCL_acl MATCH purge "localhost" > - VCL_return purge > - VCL_call HASH > - VCL_return lookup > - VCL_call PURGE > - VCL_return synth > - Timestamp Process: 1575890600.799463 0.000092 0.000092 > - RespHeader Date: Mon, 09 Dec 2019 11:23:20 GMT > - RespHeader Server: Varnish > - RespHeader X-Varnish: 12 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespReason Purged > - VCL_call SYNTH > - RespHeader Content-Type: text/html; charset=utf-8 > - RespHeader Retry-After: 5 > - VCL_return deliver > - RespHeader Content-Length: 237 > - Storage malloc Transient > - RespHeader Accept-Ranges: bytes > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1575890600.799611 0.000240 0.000147 > - ReqAcct 93 0 93 215 237 452 > - End > > * << Session >> 11 > - Begin sess 0 HTTP/1 > - SessOpen ::1 63186 :8080 ::1 8080 1575890600.799261 16 > - Link req 12 rxreq > - SessClose REM_CLOSE 0.001 > - End > > ------------------------------ > *From:* shafeeque aslam > *Sent:* Monday, December 9, 2019 3:19 AM > *To:* hamidreza hosseini > *Cc:* Guillaume Quintard ; > varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt PURGE urls > > This is purge to 1st Varnish server only. Give another purge request to > 2nd Varnish server giving it's port number. > > On Mon 9 Dec, 2019, 4:45 PM hamidreza hosseini, > wrote: > > Yes, surely i purge both servers > This is my purge request: > > $curl -v -k -X PURGE http://localhost/Naserfeiz.mp4 > > * Trying ::1... > * Connected to localhost (::1) port 80 (#0) > > PURGE /Naserfeiz.mp4 HTTP/1.1 > > Host: localhost > > User-Agent: curl/7.47.0 > > Accept: */* > > > < HTTP/1.1 200 Purged > < Date: Mon, 09 Dec 2019 11:12:54 GMT > < Server: Varnish > < X-Varnish: 98306 > < Content-Type: text/html; charset=utf-8 > < Retry-After: 5 > < Content-Length: 240 > < Accept-Ranges: bytes > < Connection: keep-alive > < > > > > 200 Purged > > >

Error 200 Purged

>

Purged

>

Guru Meditation:

>

XID: 98306

>
>

Varnish cache server

> > > * Connection #0 to host localhost left intact > > > And this is varnishlog resault: > > * << BeReq >> 32771 > - Begin bereq 32770 fetch > - Timestamp Start: 1575889894.491843 0.000000 0.000000 > - BereqMethod GET > - BereqURL / > - BereqProtocol HTTP/1.1 > - BereqHeader Host: 192.168.200.13 > - BereqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; > rv:70.0) Gecko/20100101 Firefox/70.0 > - BereqHeader Accept: > text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > - BereqHeader Accept-Language: en-US,en;q=0.5 > - BereqHeader Upgrade-Insecure-Requests: 1 > - BereqHeader X-Forwarded-For: 172.16.1.3 > - BereqHeader Accept-Encoding: gzip > - BereqHeader X-Varnish: 32771 > - VCL_call BACKEND_FETCH > - VCL_return fetch > - BackendOpen 18 boot.varnish_malloc_01 192.168.200.12 8080 > 192.168.200.13 14088 > - Timestamp Bereq: 1575889894.492444 0.000601 0.000601 > - Timestamp Beresp: 1575889894.493276 0.001433 0.000832 > - BerespProtocol HTTP/1.1 > - BerespStatus 200 > - BerespReason OK > - BerespHeader Server: nginx > - BerespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT > - BerespHeader Content-Type: text/html > - BerespHeader Content-Length: 340 > - BerespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT > - BerespHeader ETag: "5dcaad34-154" > - BerespHeader X-Varnish: 8 6 > - BerespHeader Age: 4270 > - BerespHeader Via: 1.1 varnish-v4 > - BerespHeader Accept-Ranges: bytes > - BerespHeader Connection: keep-alive > - TTL RFC 120 10 -1 1575889894 1575885624 1575885629 0 0 > - VCL_call BACKEND_RESPONSE > - TTL VCL 177070 10 0 1575885624 > - VCL_return deliver > - Storage malloc s0 > - ObjProtocol HTTP/1.1 > - ObjStatus 200 > - ObjReason OK > - ObjHeader Server: nginx > - ObjHeader Date: Mon, 09 Dec 2019 10:00:29 GMT > - ObjHeader Content-Type: text/html > - ObjHeader Content-Length: 340 > - ObjHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT > - ObjHeader ETag: "5dcaad34-154" > - ObjHeader X-Varnish: 8 6 > - ObjHeader Via: 1.1 varnish-v4 > - Fetch_Body 3 length stream > - BackendReuse 18 boot.varnish_malloc_01 > - Timestamp BerespBody: 1575889894.493474 0.001631 0.000198 > - Length 340 > - BereqAcct 336 0 336 279 340 619 > - End > > * << Request >> 32770 > - Begin req 32769 rxreq > - Timestamp Start: 1575889894.491659 0.000000 0.000000 > - Timestamp Req: 1575889894.491659 0.000000 0.000000 > - ReqStart 172.16.1.3 42502 > - ReqMethod GET > - ReqURL / > - ReqProtocol HTTP/1.1 > - ReqHeader Host: 192.168.200.13 > - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; > rv:70.0) Gecko/20100101 Firefox/70.0 > - ReqHeader Accept: > text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > - ReqHeader Accept-Language: en-US,en;q=0.5 > - ReqHeader Accept-Encoding: gzip, deflate > - ReqHeader Connection: keep-alive > - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; > _gid=GA1.1.117112441.1575877103 > - ReqHeader Upgrade-Insecure-Requests: 1 > - ReqHeader If-Modified-Since: Tue, 12 Nov 2019 13:01:40 GMT > - ReqHeader If-None-Match: "5dcaad34-154" > - ReqHeader Cache-Control: max-age=0 > - ReqHeader X-Forwarded-For: 172.16.1.3 > - VCL_call RECV > - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; > _gid=GA1.1.117112441.1575877103 > - VCL_return hash > - ReqUnset Accept-Encoding: gzip, deflate > - ReqHeader Accept-Encoding: gzip > - VCL_call HASH > - VCL_return lookup > - VCL_call MISS > - VCL_return fetch > - Link bereq 32771 fetch > - Timestamp Fetch: 1575889894.493501 0.001842 0.001842 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespHeader Server: nginx > - RespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT > - RespHeader Content-Type: text/html > - RespHeader Content-Length: 340 > - RespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT > - RespHeader ETag: "5dcaad34-154" > - RespHeader X-Varnish: 8 6 > - RespHeader Via: 1.1 varnish-v4 > - RespHeader X-Varnish: 32770 > - RespHeader Age: 4270 > - RespHeader Via: 1.1 varnish-v4 > - VCL_call DELIVER > - VCL_return deliver > - Timestamp Process: 1575889894.493541 0.001881 0.000039 > - RespProtocol HTTP/1.1 > - RespStatus 304 > - RespReason Not Modified > - RespReason Not Modified > - RespUnset Content-Length: 340 > - Debug "RES_MODE 0" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1575889894.493599 0.001940 0.000058 > - ReqAcct 503 0 503 285 0 285 > - End > > * << Request >> 65538 > - Begin req 65537 rxreq > - Timestamp Start: 1575889962.124164 0.000000 0.000000 > - Timestamp Req: 1575889962.124164 0.000000 0.000000 > - ReqStart ::1 62064 > - ReqMethod PURGE > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: localhost > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader X-Forwarded-For: ::1 > - VCL_call RECV > - VCL_acl MATCH purge "localhost" > - VCL_return purge > - VCL_call HASH > - VCL_return lookup > - VCL_call PURGE > - VCL_return synth > - Timestamp Process: 1575889962.124300 0.000136 0.000136 > - RespHeader Date: Mon, 09 Dec 2019 11:12:42 GMT > - RespHeader Server: Varnish > - RespHeader X-Varnish: 65538 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespReason Purged > - VCL_call SYNTH > - RespHeader Content-Type: text/html; charset=utf-8 > - RespHeader Retry-After: 5 > - VCL_return deliver > - RespHeader Content-Length: 240 > - Storage malloc Transient > - RespHeader Accept-Ranges: bytes > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1575889962.124593 0.000428 0.000292 > - ReqAcct 88 0 88 218 240 458 > - End > > * << Session >> 65537 > - Begin sess 0 HTTP/1 > - SessOpen ::1 62064 :80 ::1 80 1575889962.124081 13 > - Link req 65538 rxreq > - SessClose REM_CLOSE 0.001 > - End > > * << Request >> 98306 > - Begin req 98305 rxreq > - Timestamp Start: 1575889974.116017 0.000000 0.000000 > - Timestamp Req: 1575889974.116017 0.000000 0.000000 > - ReqStart ::1 62066 > - ReqMethod PURGE > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: localhost > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader X-Forwarded-For: ::1 > - VCL_call RECV > - VCL_acl MATCH purge "localhost" > - VCL_return purge > - VCL_call HASH > - VCL_return lookup > - VCL_call PURGE > - VCL_return synth > - Timestamp Process: 1575889974.116128 0.000111 0.000111 > - RespHeader Date: Mon, 09 Dec 2019 11:12:54 GMT > - RespHeader Server: Varnish > - RespHeader X-Varnish: 98306 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespReason Purged > - VCL_call SYNTH > - RespHeader Content-Type: text/html; charset=utf-8 > - RespHeader Retry-After: 5 > - VCL_return deliver > - RespHeader Content-Length: 240 > - Storage malloc Transient > - RespHeader Accept-Ranges: bytes > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1575889974.116288 0.000271 0.000160 > - ReqAcct 88 0 88 218 240 458 > - End > > * << Session >> 98305 > - Begin sess 0 HTTP/1 > - SessOpen ::1 62066 :80 ::1 80 1575889974.115935 15 > - Link req 98306 rxreq > - SessClose REM_CLOSE 0.002 > - End > > * << Session >> 32769 > - Begin sess 0 HTTP/1 > - SessOpen 172.16.1.3 42502 :80 192.168.200.13 80 > 1575889894.491572 14 > - Link req 32770 rxreq > - Link req 32772 rxreq > - VSL timeout > - End synth > > * << Request >> 32772 > - Begin req 32769 rxreq > - Timestamp Start: 1575889894.845556 0.000000 0.000000 > - Timestamp Req: 1575889894.845556 0.000000 0.000000 > - ReqStart 172.16.1.3 42502 > - ReqMethod GET > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: 192.168.200.13 > - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; > rv:70.0) Gecko/20100101 Firefox/70.0 > - ReqHeader Accept: > video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5 > - ReqHeader Accept-Language: en-US,en;q=0.5 > - ReqHeader Range: bytes=205560- > - ReqHeader Connection: keep-alive > - ReqHeader Referer: http://192.168.200.13/ > - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; > _gid=GA1.1.117112441.1575877103 > - ReqHeader X-Forwarded-For: 172.16.1.3 > - VCL_call RECV > - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; > _gid=GA1.1.117112441.1575877103 > - VCL_return hash > - VCL_call HASH > - VCL_return lookup > - Hit 3 > - VCL_call HIT > - VCL_return deliver > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespHeader Server: nginx > - RespHeader Date: Mon, 09 Dec 2019 11:06:51 GMT > - RespHeader Content-Type: text/plain > - RespHeader Content-Length: 23521499 > - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > - RespHeader ETag: "3a1794b0-166e8db" > - RespHeader X-Varnish: 32770 > - RespHeader Via: 1.1 varnish-v4 > - RespHeader X-Varnish: 32772 3 > - RespHeader Age: 288 > - RespHeader Via: 1.1 varnish-v4 > - VCL_call DELIVER > - VCL_return deliver > - Timestamp Process: 1575889894.845683 0.000127 0.000127 > - RespHeader Accept-Ranges: bytes > - RespHeader Content-Range: bytes 205560-23521498/23521499 > - RespProtocol HTTP/1.1 > - RespStatus 206 > - RespReason Partial Content > - RespReason Partial Content > - RespUnset Content-Length: 23521499 > - RespHeader Content-Length: 23315939 > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Debug "Hit idle send timeout, wrote = 10899908/23316330; > retrying" > - Debug "Hit idle send timeout, wrote = 258432/12416422; > retrying" > - Debug "Write error, retval = -1, len = 12157990, errno = > Resource temporarily unavailable" > - Timestamp Resp: 1575890074.846999 180.001443 180.001316 > - ReqAcct 419 0 419 391 23315939 23316330 > - End > > ########################## > ------------------------------ > *From:* Guillaume Quintard > *Sent:* Monday, December 9, 2019 11:57 AM > *To:* hamidreza hosseini > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt PURGE urls > > Are you sure that you purge both Varnish layers (file and RAM)? > > can you pastebin a varnishlog of the purge and of the subsequent request? > > -- > Guillaume Quintard > > > On Mon, Dec 9, 2019 at 2:55 AM hamidreza hosseini > wrote: > > because its steel in cach and i can download and see th urls that i purged > before! > its Interesting that it return 200 code from varnish!!! > > Best regards. > ------------------------------ > *From:* Guillaume Quintard > *Sent:* Monday, December 9, 2019 11:48 AM > *To:* hamidreza hosseini > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt PURGE urls > > Hi, > > How do you tell that it's not purging? > > Cheers, > > -- > Guillaume Quintard > > > On Mon, Dec 9, 2019 at 2:45 AM hamidreza hosseini > wrote: > > Hi, > > I've two varnish, that one of them use ram as backend and another file as > backend, requests first come to varnish(ram) and then go to another server > in varnish(file) and then go to nginx, this is my diagram: > > > Req=====> [varnish ram] =======> [varnish file] =====> [Nginx] > > And following file is my vcl configuration file that i'm using, > but when i want to purge a url it doesnt work and it steel remains in > cache, > Should i chage my config file? > What's wrong with varnish? > > Vcl config file: > > ### Naming note: use [0-9a-zA-Z_] only. no -. Or use camelCase ;-) > vcl 4.0; > > import directors; > > > backend varnish_malloc_01 { .host = "192.168.200.12"; .port = "8080"; } > > > sub vcl_init { > new varnish_cluster = directors.round_robin(); > varnish_cluster.add_backend(varnish_malloc_01); > > } > > > acl purge { > "localhost"; > "192.168.200.0"/24; > ; > } > > > sub vcl_recv { > > sub vcl_recv { > #set req.backend_hint = varnish_cluster.backend(); > unset req.http.Cookie; > if (req.method == "PURGE") { > ### TODO: also purge from the backend > if (!client.ip ~ purge) { > return(synth(405,"Not allowed.")); > } > return (purge); > } > } > > sub vcl_backend_response { > set beresp.ttl=48h; > # Don't cache 404 responses > if ( beresp.status == 404 ) { > set beresp.ttl = 120s; > set beresp.uncacheable = true; > return (deliver); > } > } > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrhosseini at hotmail.com Mon Dec 9 11:34:35 2019 From: hrhosseini at hotmail.com (hamidreza hosseini) Date: Mon, 9 Dec 2019 11:34:35 +0000 Subject: Varnish doesnt PURGE urls In-Reply-To: References: , Message-ID: this is my test servers, I purge with ip in production but steel it doesnt work! ________________________________ From: Guillaume Quintard Sent: Monday, December 9, 2019 12:31 PM To: hamidreza hosseini Cc: shafeeque aslam ; varnish-misc at varnish-cache.org Subject: Re: Varnish doesnt PURGE urls Purge uses the hash of the objects to clear them. In turn, the hash, by default depends on the host of the requests that created the object. So, when you purge using localhost/some/path.mp4, you dont purge the object referring to my.website.com/some/path.mp4 -- Guillaume Quintard On Mon, Dec 9, 2019 at 3:24 AM hamidreza hosseini > wrote: This is purge to another server: $curl -v -k -X PURGE http://localhost:8080/Naserfeiz.mp4 * Trying ::1... * Connected to localhost (::1) port 8080 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: localhost:8080 > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 200 Purged < Date: Mon, 09 Dec 2019 11:20:52 GMT < Server: Varnish < X-Varnish: 10 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 237 < Accept-Ranges: bytes < Connection: keep-alive < 200 Purged

Error 200 Purged

Purged

Guru Meditation:

XID: 10


Varnish cache server

* Connection #0 to host localhost left intact And varnishlog resault: $ sudo varnishlog * << Request >> 32773 - Begin req 32772 rxreq - Timestamp Start: 1575890593.662132 0.000000 0.000000 - Timestamp Req: 1575890593.662132 0.000000 0.000000 - ReqStart ::1 63184 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575890593.662268 0.000135 0.000135 - RespHeader Date: Mon, 09 Dec 2019 11:23:13 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 32773 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575890593.662484 0.000352 0.000216 - ReqAcct 93 0 93 218 240 458 - End * << Session >> 32772 - Begin sess 0 HTTP/1 - SessOpen ::1 63184 :8080 ::1 8080 1575890593.662044 16 - Link req 32773 rxreq - SessClose REM_CLOSE 0.001 - End * << Request >> 12 - Begin req 11 rxreq - Timestamp Start: 1575890600.799371 0.000000 0.000000 - Timestamp Req: 1575890600.799371 0.000000 0.000000 - ReqStart ::1 63186 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575890600.799463 0.000092 0.000092 - RespHeader Date: Mon, 09 Dec 2019 11:23:20 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 12 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 237 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575890600.799611 0.000240 0.000147 - ReqAcct 93 0 93 215 237 452 - End * << Session >> 11 - Begin sess 0 HTTP/1 - SessOpen ::1 63186 :8080 ::1 8080 1575890600.799261 16 - Link req 12 rxreq - SessClose REM_CLOSE 0.001 - End ________________________________ From: shafeeque aslam > Sent: Monday, December 9, 2019 3:19 AM To: hamidreza hosseini > Cc: Guillaume Quintard >; varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt PURGE urls This is purge to 1st Varnish server only. Give another purge request to 2nd Varnish server giving it's port number. On Mon 9 Dec, 2019, 4:45 PM hamidreza hosseini, > wrote: Yes, surely i purge both servers This is my purge request: $curl -v -k -X PURGE http://localhost/Naserfeiz.mp4 * Trying ::1... * Connected to localhost (::1) port 80 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: localhost > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 200 Purged < Date: Mon, 09 Dec 2019 11:12:54 GMT < Server: Varnish < X-Varnish: 98306 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 240 < Accept-Ranges: bytes < Connection: keep-alive < 200 Purged

Error 200 Purged

Purged

Guru Meditation:

XID: 98306


Varnish cache server

* Connection #0 to host localhost left intact And this is varnishlog resault: * << BeReq >> 32771 - Begin bereq 32770 fetch - Timestamp Start: 1575889894.491843 0.000000 0.000000 - BereqMethod GET - BereqURL / - BereqProtocol HTTP/1.1 - BereqHeader Host: 192.168.200.13 - BereqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0 - BereqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 - BereqHeader Accept-Language: en-US,en;q=0.5 - BereqHeader Upgrade-Insecure-Requests: 1 - BereqHeader X-Forwarded-For: 172.16.1.3 - BereqHeader Accept-Encoding: gzip - BereqHeader X-Varnish: 32771 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 18 boot.varnish_malloc_01 192.168.200.12 8080 192.168.200.13 14088 - Timestamp Bereq: 1575889894.492444 0.000601 0.000601 - Timestamp Beresp: 1575889894.493276 0.001433 0.000832 - BerespProtocol HTTP/1.1 - BerespStatus 200 - BerespReason OK - BerespHeader Server: nginx - BerespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT - BerespHeader Content-Type: text/html - BerespHeader Content-Length: 340 - BerespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT - BerespHeader ETag: "5dcaad34-154" - BerespHeader X-Varnish: 8 6 - BerespHeader Age: 4270 - BerespHeader Via: 1.1 varnish-v4 - BerespHeader Accept-Ranges: bytes - BerespHeader Connection: keep-alive - TTL RFC 120 10 -1 1575889894 1575885624 1575885629 0 0 - VCL_call BACKEND_RESPONSE - TTL VCL 177070 10 0 1575885624 - VCL_return deliver - Storage malloc s0 - ObjProtocol HTTP/1.1 - ObjStatus 200 - ObjReason OK - ObjHeader Server: nginx - ObjHeader Date: Mon, 09 Dec 2019 10:00:29 GMT - ObjHeader Content-Type: text/html - ObjHeader Content-Length: 340 - ObjHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT - ObjHeader ETag: "5dcaad34-154" - ObjHeader X-Varnish: 8 6 - ObjHeader Via: 1.1 varnish-v4 - Fetch_Body 3 length stream - BackendReuse 18 boot.varnish_malloc_01 - Timestamp BerespBody: 1575889894.493474 0.001631 0.000198 - Length 340 - BereqAcct 336 0 336 279 340 619 - End * << Request >> 32770 - Begin req 32769 rxreq - Timestamp Start: 1575889894.491659 0.000000 0.000000 - Timestamp Req: 1575889894.491659 0.000000 0.000000 - ReqStart 172.16.1.3 42502 - ReqMethod GET - ReqURL / - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.13 - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0 - ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 - ReqHeader Accept-Language: en-US,en;q=0.5 - ReqHeader Accept-Encoding: gzip, deflate - ReqHeader Connection: keep-alive - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - ReqHeader Upgrade-Insecure-Requests: 1 - ReqHeader If-Modified-Since: Tue, 12 Nov 2019 13:01:40 GMT - ReqHeader If-None-Match: "5dcaad34-154" - ReqHeader Cache-Control: max-age=0 - ReqHeader X-Forwarded-For: 172.16.1.3 - VCL_call RECV - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - VCL_return hash - ReqUnset Accept-Encoding: gzip, deflate - ReqHeader Accept-Encoding: gzip - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 32771 fetch - Timestamp Fetch: 1575889894.493501 0.001842 0.001842 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT - RespHeader Content-Type: text/html - RespHeader Content-Length: 340 - RespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT - RespHeader ETag: "5dcaad34-154" - RespHeader X-Varnish: 8 6 - RespHeader Via: 1.1 varnish-v4 - RespHeader X-Varnish: 32770 - RespHeader Age: 4270 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1575889894.493541 0.001881 0.000039 - RespProtocol HTTP/1.1 - RespStatus 304 - RespReason Not Modified - RespReason Not Modified - RespUnset Content-Length: 340 - Debug "RES_MODE 0" - RespHeader Connection: keep-alive - Timestamp Resp: 1575889894.493599 0.001940 0.000058 - ReqAcct 503 0 503 285 0 285 - End * << Request >> 65538 - Begin req 65537 rxreq - Timestamp Start: 1575889962.124164 0.000000 0.000000 - Timestamp Req: 1575889962.124164 0.000000 0.000000 - ReqStart ::1 62064 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575889962.124300 0.000136 0.000136 - RespHeader Date: Mon, 09 Dec 2019 11:12:42 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 65538 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575889962.124593 0.000428 0.000292 - ReqAcct 88 0 88 218 240 458 - End * << Session >> 65537 - Begin sess 0 HTTP/1 - SessOpen ::1 62064 :80 ::1 80 1575889962.124081 13 - Link req 65538 rxreq - SessClose REM_CLOSE 0.001 - End * << Request >> 98306 - Begin req 98305 rxreq - Timestamp Start: 1575889974.116017 0.000000 0.000000 - Timestamp Req: 1575889974.116017 0.000000 0.000000 - ReqStart ::1 62066 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575889974.116128 0.000111 0.000111 - RespHeader Date: Mon, 09 Dec 2019 11:12:54 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 98306 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575889974.116288 0.000271 0.000160 - ReqAcct 88 0 88 218 240 458 - End * << Session >> 98305 - Begin sess 0 HTTP/1 - SessOpen ::1 62066 :80 ::1 80 1575889974.115935 15 - Link req 98306 rxreq - SessClose REM_CLOSE 0.002 - End * << Session >> 32769 - Begin sess 0 HTTP/1 - SessOpen 172.16.1.3 42502 :80 192.168.200.13 80 1575889894.491572 14 - Link req 32770 rxreq - Link req 32772 rxreq - VSL timeout - End synth * << Request >> 32772 - Begin req 32769 rxreq - Timestamp Start: 1575889894.845556 0.000000 0.000000 - Timestamp Req: 1575889894.845556 0.000000 0.000000 - ReqStart 172.16.1.3 42502 - ReqMethod GET - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.13 - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0 - ReqHeader Accept: video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5 - ReqHeader Accept-Language: en-US,en;q=0.5 - ReqHeader Range: bytes=205560- - ReqHeader Connection: keep-alive - ReqHeader Referer: http://192.168.200.13/ - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - ReqHeader X-Forwarded-For: 172.16.1.3 - VCL_call RECV - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - VCL_return hash - VCL_call HASH - VCL_return lookup - Hit 3 - VCL_call HIT - VCL_return deliver - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Mon, 09 Dec 2019 11:06:51 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 32770 - RespHeader Via: 1.1 varnish-v4 - RespHeader X-Varnish: 32772 3 - RespHeader Age: 288 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1575889894.845683 0.000127 0.000127 - RespHeader Accept-Ranges: bytes - RespHeader Content-Range: bytes 205560-23521498/23521499 - RespProtocol HTTP/1.1 - RespStatus 206 - RespReason Partial Content - RespReason Partial Content - RespUnset Content-Length: 23521499 - RespHeader Content-Length: 23315939 - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Debug "Hit idle send timeout, wrote = 10899908/23316330; retrying" - Debug "Hit idle send timeout, wrote = 258432/12416422; retrying" - Debug "Write error, retval = -1, len = 12157990, errno = Resource temporarily unavailable" - Timestamp Resp: 1575890074.846999 180.001443 180.001316 - ReqAcct 419 0 419 391 23315939 23316330 - End ########################## ________________________________ From: Guillaume Quintard > Sent: Monday, December 9, 2019 11:57 AM To: hamidreza hosseini > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt PURGE urls Are you sure that you purge both Varnish layers (file and RAM)? can you pastebin a varnishlog of the purge and of the subsequent request? -- Guillaume Quintard On Mon, Dec 9, 2019 at 2:55 AM hamidreza hosseini > wrote: because its steel in cach and i can download and see th urls that i purged before! its Interesting that it return 200 code from varnish!!! Best regards. ________________________________ From: Guillaume Quintard > Sent: Monday, December 9, 2019 11:48 AM To: hamidreza hosseini > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt PURGE urls Hi, How do you tell that it's not purging? Cheers, -- Guillaume Quintard On Mon, Dec 9, 2019 at 2:45 AM hamidreza hosseini > wrote: Hi, I've two varnish, that one of them use ram as backend and another file as backend, requests first come to varnish(ram) and then go to another server in varnish(file) and then go to nginx, this is my diagram: Req=====> [varnish ram] =======> [varnish file] =====> [Nginx] And following file is my vcl configuration file that i'm using, but when i want to purge a url it doesnt work and it steel remains in cache, Should i chage my config file? What's wrong with varnish? Vcl config file: ### Naming note: use [0-9a-zA-Z_] only. no -. Or use camelCase ;-) vcl 4.0; import directors; backend varnish_malloc_01 { .host = "192.168.200.12"; .port = "8080"; } sub vcl_init { new varnish_cluster = directors.round_robin(); varnish_cluster.add_backend(varnish_malloc_01); } acl purge { "localhost"; "192.168.200.0"/24; ; } sub vcl_recv { sub vcl_recv { #set req.backend_hint = varnish_cluster.backend(); unset req.http.Cookie; if (req.method == "PURGE") { ### TODO: also purge from the backend if (!client.ip ~ purge) { return(synth(405,"Not allowed.")); } return (purge); } } sub vcl_backend_response { set beresp.ttl=48h; # Don't cache 404 responses if ( beresp.status == 404 ) { set beresp.ttl = 120s; set beresp.uncacheable = true; return (deliver); } } _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrhosseini at hotmail.com Mon Dec 9 12:08:08 2019 From: hrhosseini at hotmail.com (hamidreza hosseini) Date: Mon, 9 Dec 2019 12:08:08 +0000 Subject: Varnish doesnt PURGE urls In-Reply-To: References: , , Message-ID: I test it again and in this test the fron server was purged but the backend didnt purge what is the problem? ________________________________ From: varnish-misc on behalf of hamidreza hosseini Sent: Monday, December 9, 2019 3:34 AM To: Guillaume Quintard Cc: varnish-misc at varnish-cache.org Subject: Re: Varnish doesnt PURGE urls this is my test servers, I purge with ip in production but steel it doesnt work! ________________________________ From: Guillaume Quintard Sent: Monday, December 9, 2019 12:31 PM To: hamidreza hosseini Cc: shafeeque aslam ; varnish-misc at varnish-cache.org Subject: Re: Varnish doesnt PURGE urls Purge uses the hash of the objects to clear them. In turn, the hash, by default depends on the host of the requests that created the object. So, when you purge using localhost/some/path.mp4, you dont purge the object referring to my.website.com/some/path.mp4 -- Guillaume Quintard On Mon, Dec 9, 2019 at 3:24 AM hamidreza hosseini > wrote: This is purge to another server: $curl -v -k -X PURGE http://localhost:8080/Naserfeiz.mp4 * Trying ::1... * Connected to localhost (::1) port 8080 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: localhost:8080 > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 200 Purged < Date: Mon, 09 Dec 2019 11:20:52 GMT < Server: Varnish < X-Varnish: 10 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 237 < Accept-Ranges: bytes < Connection: keep-alive < 200 Purged

Error 200 Purged

Purged

Guru Meditation:

XID: 10


Varnish cache server

* Connection #0 to host localhost left intact And varnishlog resault: $ sudo varnishlog * << Request >> 32773 - Begin req 32772 rxreq - Timestamp Start: 1575890593.662132 0.000000 0.000000 - Timestamp Req: 1575890593.662132 0.000000 0.000000 - ReqStart ::1 63184 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575890593.662268 0.000135 0.000135 - RespHeader Date: Mon, 09 Dec 2019 11:23:13 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 32773 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575890593.662484 0.000352 0.000216 - ReqAcct 93 0 93 218 240 458 - End * << Session >> 32772 - Begin sess 0 HTTP/1 - SessOpen ::1 63184 :8080 ::1 8080 1575890593.662044 16 - Link req 32773 rxreq - SessClose REM_CLOSE 0.001 - End * << Request >> 12 - Begin req 11 rxreq - Timestamp Start: 1575890600.799371 0.000000 0.000000 - Timestamp Req: 1575890600.799371 0.000000 0.000000 - ReqStart ::1 63186 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575890600.799463 0.000092 0.000092 - RespHeader Date: Mon, 09 Dec 2019 11:23:20 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 12 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 237 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575890600.799611 0.000240 0.000147 - ReqAcct 93 0 93 215 237 452 - End * << Session >> 11 - Begin sess 0 HTTP/1 - SessOpen ::1 63186 :8080 ::1 8080 1575890600.799261 16 - Link req 12 rxreq - SessClose REM_CLOSE 0.001 - End ________________________________ From: shafeeque aslam > Sent: Monday, December 9, 2019 3:19 AM To: hamidreza hosseini > Cc: Guillaume Quintard >; varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt PURGE urls This is purge to 1st Varnish server only. Give another purge request to 2nd Varnish server giving it's port number. On Mon 9 Dec, 2019, 4:45 PM hamidreza hosseini, > wrote: Yes, surely i purge both servers This is my purge request: $curl -v -k -X PURGE http://localhost/Naserfeiz.mp4 * Trying ::1... * Connected to localhost (::1) port 80 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: localhost > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 200 Purged < Date: Mon, 09 Dec 2019 11:12:54 GMT < Server: Varnish < X-Varnish: 98306 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 240 < Accept-Ranges: bytes < Connection: keep-alive < 200 Purged

Error 200 Purged

Purged

Guru Meditation:

XID: 98306


Varnish cache server

* Connection #0 to host localhost left intact And this is varnishlog resault: * << BeReq >> 32771 - Begin bereq 32770 fetch - Timestamp Start: 1575889894.491843 0.000000 0.000000 - BereqMethod GET - BereqURL / - BereqProtocol HTTP/1.1 - BereqHeader Host: 192.168.200.13 - BereqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0 - BereqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 - BereqHeader Accept-Language: en-US,en;q=0.5 - BereqHeader Upgrade-Insecure-Requests: 1 - BereqHeader X-Forwarded-For: 172.16.1.3 - BereqHeader Accept-Encoding: gzip - BereqHeader X-Varnish: 32771 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 18 boot.varnish_malloc_01 192.168.200.12 8080 192.168.200.13 14088 - Timestamp Bereq: 1575889894.492444 0.000601 0.000601 - Timestamp Beresp: 1575889894.493276 0.001433 0.000832 - BerespProtocol HTTP/1.1 - BerespStatus 200 - BerespReason OK - BerespHeader Server: nginx - BerespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT - BerespHeader Content-Type: text/html - BerespHeader Content-Length: 340 - BerespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT - BerespHeader ETag: "5dcaad34-154" - BerespHeader X-Varnish: 8 6 - BerespHeader Age: 4270 - BerespHeader Via: 1.1 varnish-v4 - BerespHeader Accept-Ranges: bytes - BerespHeader Connection: keep-alive - TTL RFC 120 10 -1 1575889894 1575885624 1575885629 0 0 - VCL_call BACKEND_RESPONSE - TTL VCL 177070 10 0 1575885624 - VCL_return deliver - Storage malloc s0 - ObjProtocol HTTP/1.1 - ObjStatus 200 - ObjReason OK - ObjHeader Server: nginx - ObjHeader Date: Mon, 09 Dec 2019 10:00:29 GMT - ObjHeader Content-Type: text/html - ObjHeader Content-Length: 340 - ObjHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT - ObjHeader ETag: "5dcaad34-154" - ObjHeader X-Varnish: 8 6 - ObjHeader Via: 1.1 varnish-v4 - Fetch_Body 3 length stream - BackendReuse 18 boot.varnish_malloc_01 - Timestamp BerespBody: 1575889894.493474 0.001631 0.000198 - Length 340 - BereqAcct 336 0 336 279 340 619 - End * << Request >> 32770 - Begin req 32769 rxreq - Timestamp Start: 1575889894.491659 0.000000 0.000000 - Timestamp Req: 1575889894.491659 0.000000 0.000000 - ReqStart 172.16.1.3 42502 - ReqMethod GET - ReqURL / - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.13 - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0 - ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 - ReqHeader Accept-Language: en-US,en;q=0.5 - ReqHeader Accept-Encoding: gzip, deflate - ReqHeader Connection: keep-alive - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - ReqHeader Upgrade-Insecure-Requests: 1 - ReqHeader If-Modified-Since: Tue, 12 Nov 2019 13:01:40 GMT - ReqHeader If-None-Match: "5dcaad34-154" - ReqHeader Cache-Control: max-age=0 - ReqHeader X-Forwarded-For: 172.16.1.3 - VCL_call RECV - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - VCL_return hash - ReqUnset Accept-Encoding: gzip, deflate - ReqHeader Accept-Encoding: gzip - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 32771 fetch - Timestamp Fetch: 1575889894.493501 0.001842 0.001842 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT - RespHeader Content-Type: text/html - RespHeader Content-Length: 340 - RespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT - RespHeader ETag: "5dcaad34-154" - RespHeader X-Varnish: 8 6 - RespHeader Via: 1.1 varnish-v4 - RespHeader X-Varnish: 32770 - RespHeader Age: 4270 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1575889894.493541 0.001881 0.000039 - RespProtocol HTTP/1.1 - RespStatus 304 - RespReason Not Modified - RespReason Not Modified - RespUnset Content-Length: 340 - Debug "RES_MODE 0" - RespHeader Connection: keep-alive - Timestamp Resp: 1575889894.493599 0.001940 0.000058 - ReqAcct 503 0 503 285 0 285 - End * << Request >> 65538 - Begin req 65537 rxreq - Timestamp Start: 1575889962.124164 0.000000 0.000000 - Timestamp Req: 1575889962.124164 0.000000 0.000000 - ReqStart ::1 62064 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575889962.124300 0.000136 0.000136 - RespHeader Date: Mon, 09 Dec 2019 11:12:42 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 65538 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575889962.124593 0.000428 0.000292 - ReqAcct 88 0 88 218 240 458 - End * << Session >> 65537 - Begin sess 0 HTTP/1 - SessOpen ::1 62064 :80 ::1 80 1575889962.124081 13 - Link req 65538 rxreq - SessClose REM_CLOSE 0.001 - End * << Request >> 98306 - Begin req 98305 rxreq - Timestamp Start: 1575889974.116017 0.000000 0.000000 - Timestamp Req: 1575889974.116017 0.000000 0.000000 - ReqStart ::1 62066 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575889974.116128 0.000111 0.000111 - RespHeader Date: Mon, 09 Dec 2019 11:12:54 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 98306 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575889974.116288 0.000271 0.000160 - ReqAcct 88 0 88 218 240 458 - End * << Session >> 98305 - Begin sess 0 HTTP/1 - SessOpen ::1 62066 :80 ::1 80 1575889974.115935 15 - Link req 98306 rxreq - SessClose REM_CLOSE 0.002 - End * << Session >> 32769 - Begin sess 0 HTTP/1 - SessOpen 172.16.1.3 42502 :80 192.168.200.13 80 1575889894.491572 14 - Link req 32770 rxreq - Link req 32772 rxreq - VSL timeout - End synth * << Request >> 32772 - Begin req 32769 rxreq - Timestamp Start: 1575889894.845556 0.000000 0.000000 - Timestamp Req: 1575889894.845556 0.000000 0.000000 - ReqStart 172.16.1.3 42502 - ReqMethod GET - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.13 - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0 - ReqHeader Accept: video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5 - ReqHeader Accept-Language: en-US,en;q=0.5 - ReqHeader Range: bytes=205560- - ReqHeader Connection: keep-alive - ReqHeader Referer: http://192.168.200.13/ - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - ReqHeader X-Forwarded-For: 172.16.1.3 - VCL_call RECV - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - VCL_return hash - VCL_call HASH - VCL_return lookup - Hit 3 - VCL_call HIT - VCL_return deliver - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Mon, 09 Dec 2019 11:06:51 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 32770 - RespHeader Via: 1.1 varnish-v4 - RespHeader X-Varnish: 32772 3 - RespHeader Age: 288 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1575889894.845683 0.000127 0.000127 - RespHeader Accept-Ranges: bytes - RespHeader Content-Range: bytes 205560-23521498/23521499 - RespProtocol HTTP/1.1 - RespStatus 206 - RespReason Partial Content - RespReason Partial Content - RespUnset Content-Length: 23521499 - RespHeader Content-Length: 23315939 - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Debug "Hit idle send timeout, wrote = 10899908/23316330; retrying" - Debug "Hit idle send timeout, wrote = 258432/12416422; retrying" - Debug "Write error, retval = -1, len = 12157990, errno = Resource temporarily unavailable" - Timestamp Resp: 1575890074.846999 180.001443 180.001316 - ReqAcct 419 0 419 391 23315939 23316330 - End ########################## ________________________________ From: Guillaume Quintard > Sent: Monday, December 9, 2019 11:57 AM To: hamidreza hosseini > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt PURGE urls Are you sure that you purge both Varnish layers (file and RAM)? can you pastebin a varnishlog of the purge and of the subsequent request? -- Guillaume Quintard On Mon, Dec 9, 2019 at 2:55 AM hamidreza hosseini > wrote: because its steel in cach and i can download and see th urls that i purged before! its Interesting that it return 200 code from varnish!!! Best regards. ________________________________ From: Guillaume Quintard > Sent: Monday, December 9, 2019 11:48 AM To: hamidreza hosseini > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt PURGE urls Hi, How do you tell that it's not purging? Cheers, -- Guillaume Quintard On Mon, Dec 9, 2019 at 2:45 AM hamidreza hosseini > wrote: Hi, I've two varnish, that one of them use ram as backend and another file as backend, requests first come to varnish(ram) and then go to another server in varnish(file) and then go to nginx, this is my diagram: Req=====> [varnish ram] =======> [varnish file] =====> [Nginx] And following file is my vcl configuration file that i'm using, but when i want to purge a url it doesnt work and it steel remains in cache, Should i chage my config file? What's wrong with varnish? Vcl config file: ### Naming note: use [0-9a-zA-Z_] only. no -. Or use camelCase ;-) vcl 4.0; import directors; backend varnish_malloc_01 { .host = "192.168.200.12"; .port = "8080"; } sub vcl_init { new varnish_cluster = directors.round_robin(); varnish_cluster.add_backend(varnish_malloc_01); } acl purge { "localhost"; "192.168.200.0"/24; ; } sub vcl_recv { sub vcl_recv { #set req.backend_hint = varnish_cluster.backend(); unset req.http.Cookie; if (req.method == "PURGE") { ### TODO: also purge from the backend if (!client.ip ~ purge) { return(synth(405,"Not allowed.")); } return (purge); } } sub vcl_backend_response { set beresp.ttl=48h; # Don't cache 404 responses if ( beresp.status == 404 ) { set beresp.ttl = 120s; set beresp.uncacheable = true; return (deliver); } } _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Mon Dec 9 21:17:06 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 9 Dec 2019 13:17:06 -0800 Subject: Varnish doesnt PURGE urls In-Reply-To: References: Message-ID: when you purge, varnish creates a synthetic response, and doesn't go to the backend, so if you wish to purge that one too, you need to send a request to it too. -- Guillaume Quintard On Mon, Dec 9, 2019 at 4:08 AM hamidreza hosseini wrote: > I test it again and in this test the fron server was purged but the > backend didnt purge > what is the problem? > ------------------------------ > *From:* varnish-misc hotmail.com at varnish-cache.org> on behalf of hamidreza hosseini < > hrhosseini at hotmail.com> > *Sent:* Monday, December 9, 2019 3:34 AM > *To:* Guillaume Quintard > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt PURGE urls > > this is my test servers, I purge with ip in production but steel it doesnt > work! > ------------------------------ > *From:* Guillaume Quintard > *Sent:* Monday, December 9, 2019 12:31 PM > *To:* hamidreza hosseini > *Cc:* shafeeque aslam ; > varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt PURGE urls > > Purge uses the hash of the objects to clear them. In turn, the hash, by > default depends on the host of the requests that created the object. > > So, when you purge using localhost/some/path.mp4, you dont purge the > object referring to my.website.com/some/path.mp4 > > -- > Guillaume Quintard > > > On Mon, Dec 9, 2019 at 3:24 AM hamidreza hosseini > wrote: > > This is purge to another server: > > $curl -v -k -X PURGE http://localhost:8080/Naserfeiz.mp4 > > * Trying ::1... > * Connected to localhost (::1) port 8080 (#0) > > PURGE /Naserfeiz.mp4 HTTP/1.1 > > Host: localhost:8080 > > User-Agent: curl/7.47.0 > > Accept: */* > > > < HTTP/1.1 200 Purged > < Date: Mon, 09 Dec 2019 11:20:52 GMT > < Server: Varnish > < X-Varnish: 10 > < Content-Type: text/html; charset=utf-8 > < Retry-After: 5 > < Content-Length: 237 > < Accept-Ranges: bytes > < Connection: keep-alive > < > > > > 200 Purged > > >

Error 200 Purged

>

Purged

>

Guru Meditation:

>

XID: 10

>
>

Varnish cache server

> > > * Connection #0 to host localhost left intact > > And varnishlog resault: > > $ sudo varnishlog > > * << Request >> 32773 > - Begin req 32772 rxreq > - Timestamp Start: 1575890593.662132 0.000000 0.000000 > - Timestamp Req: 1575890593.662132 0.000000 0.000000 > - ReqStart ::1 63184 > - ReqMethod PURGE > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: localhost:8080 > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader X-Forwarded-For: ::1 > - VCL_call RECV > - VCL_acl MATCH purge "localhost" > - VCL_return purge > - VCL_call HASH > - VCL_return lookup > - VCL_call PURGE > - VCL_return synth > - Timestamp Process: 1575890593.662268 0.000135 0.000135 > - RespHeader Date: Mon, 09 Dec 2019 11:23:13 GMT > - RespHeader Server: Varnish > - RespHeader X-Varnish: 32773 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespReason Purged > - VCL_call SYNTH > - RespHeader Content-Type: text/html; charset=utf-8 > - RespHeader Retry-After: 5 > - VCL_return deliver > - RespHeader Content-Length: 240 > - Storage malloc Transient > - RespHeader Accept-Ranges: bytes > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1575890593.662484 0.000352 0.000216 > - ReqAcct 93 0 93 218 240 458 > - End > > * << Session >> 32772 > - Begin sess 0 HTTP/1 > - SessOpen ::1 63184 :8080 ::1 8080 1575890593.662044 16 > - Link req 32773 rxreq > - SessClose REM_CLOSE 0.001 > - End > > * << Request >> 12 > - Begin req 11 rxreq > - Timestamp Start: 1575890600.799371 0.000000 0.000000 > - Timestamp Req: 1575890600.799371 0.000000 0.000000 > - ReqStart ::1 63186 > - ReqMethod PURGE > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: localhost:8080 > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader X-Forwarded-For: ::1 > - VCL_call RECV > - VCL_acl MATCH purge "localhost" > - VCL_return purge > - VCL_call HASH > - VCL_return lookup > - VCL_call PURGE > - VCL_return synth > - Timestamp Process: 1575890600.799463 0.000092 0.000092 > - RespHeader Date: Mon, 09 Dec 2019 11:23:20 GMT > - RespHeader Server: Varnish > - RespHeader X-Varnish: 12 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespReason Purged > - VCL_call SYNTH > - RespHeader Content-Type: text/html; charset=utf-8 > - RespHeader Retry-After: 5 > - VCL_return deliver > - RespHeader Content-Length: 237 > - Storage malloc Transient > - RespHeader Accept-Ranges: bytes > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1575890600.799611 0.000240 0.000147 > - ReqAcct 93 0 93 215 237 452 > - End > > * << Session >> 11 > - Begin sess 0 HTTP/1 > - SessOpen ::1 63186 :8080 ::1 8080 1575890600.799261 16 > - Link req 12 rxreq > - SessClose REM_CLOSE 0.001 > - End > > ------------------------------ > *From:* shafeeque aslam > *Sent:* Monday, December 9, 2019 3:19 AM > *To:* hamidreza hosseini > *Cc:* Guillaume Quintard ; > varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt PURGE urls > > This is purge to 1st Varnish server only. Give another purge request to > 2nd Varnish server giving it's port number. > > On Mon 9 Dec, 2019, 4:45 PM hamidreza hosseini, > wrote: > > Yes, surely i purge both servers > This is my purge request: > > $curl -v -k -X PURGE http://localhost/Naserfeiz.mp4 > > * Trying ::1... > * Connected to localhost (::1) port 80 (#0) > > PURGE /Naserfeiz.mp4 HTTP/1.1 > > Host: localhost > > User-Agent: curl/7.47.0 > > Accept: */* > > > < HTTP/1.1 200 Purged > < Date: Mon, 09 Dec 2019 11:12:54 GMT > < Server: Varnish > < X-Varnish: 98306 > < Content-Type: text/html; charset=utf-8 > < Retry-After: 5 > < Content-Length: 240 > < Accept-Ranges: bytes > < Connection: keep-alive > < > > > > 200 Purged > > >

Error 200 Purged

>

Purged

>

Guru Meditation:

>

XID: 98306

>
>

Varnish cache server

> > > * Connection #0 to host localhost left intact > > > And this is varnishlog resault: > > * << BeReq >> 32771 > - Begin bereq 32770 fetch > - Timestamp Start: 1575889894.491843 0.000000 0.000000 > - BereqMethod GET > - BereqURL / > - BereqProtocol HTTP/1.1 > - BereqHeader Host: 192.168.200.13 > - BereqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; > rv:70.0) Gecko/20100101 Firefox/70.0 > - BereqHeader Accept: > text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > - BereqHeader Accept-Language: en-US,en;q=0.5 > - BereqHeader Upgrade-Insecure-Requests: 1 > - BereqHeader X-Forwarded-For: 172.16.1.3 > - BereqHeader Accept-Encoding: gzip > - BereqHeader X-Varnish: 32771 > - VCL_call BACKEND_FETCH > - VCL_return fetch > - BackendOpen 18 boot.varnish_malloc_01 192.168.200.12 8080 > 192.168.200.13 14088 > - Timestamp Bereq: 1575889894.492444 0.000601 0.000601 > - Timestamp Beresp: 1575889894.493276 0.001433 0.000832 > - BerespProtocol HTTP/1.1 > - BerespStatus 200 > - BerespReason OK > - BerespHeader Server: nginx > - BerespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT > - BerespHeader Content-Type: text/html > - BerespHeader Content-Length: 340 > - BerespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT > - BerespHeader ETag: "5dcaad34-154" > - BerespHeader X-Varnish: 8 6 > - BerespHeader Age: 4270 > - BerespHeader Via: 1.1 varnish-v4 > - BerespHeader Accept-Ranges: bytes > - BerespHeader Connection: keep-alive > - TTL RFC 120 10 -1 1575889894 1575885624 1575885629 0 0 > - VCL_call BACKEND_RESPONSE > - TTL VCL 177070 10 0 1575885624 > - VCL_return deliver > - Storage malloc s0 > - ObjProtocol HTTP/1.1 > - ObjStatus 200 > - ObjReason OK > - ObjHeader Server: nginx > - ObjHeader Date: Mon, 09 Dec 2019 10:00:29 GMT > - ObjHeader Content-Type: text/html > - ObjHeader Content-Length: 340 > - ObjHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT > - ObjHeader ETag: "5dcaad34-154" > - ObjHeader X-Varnish: 8 6 > - ObjHeader Via: 1.1 varnish-v4 > - Fetch_Body 3 length stream > - BackendReuse 18 boot.varnish_malloc_01 > - Timestamp BerespBody: 1575889894.493474 0.001631 0.000198 > - Length 340 > - BereqAcct 336 0 336 279 340 619 > - End > > * << Request >> 32770 > - Begin req 32769 rxreq > - Timestamp Start: 1575889894.491659 0.000000 0.000000 > - Timestamp Req: 1575889894.491659 0.000000 0.000000 > - ReqStart 172.16.1.3 42502 > - ReqMethod GET > - ReqURL / > - ReqProtocol HTTP/1.1 > - ReqHeader Host: 192.168.200.13 > - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; > rv:70.0) Gecko/20100101 Firefox/70.0 > - ReqHeader Accept: > text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > - ReqHeader Accept-Language: en-US,en;q=0.5 > - ReqHeader Accept-Encoding: gzip, deflate > - ReqHeader Connection: keep-alive > - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; > _gid=GA1.1.117112441.1575877103 > - ReqHeader Upgrade-Insecure-Requests: 1 > - ReqHeader If-Modified-Since: Tue, 12 Nov 2019 13:01:40 GMT > - ReqHeader If-None-Match: "5dcaad34-154" > - ReqHeader Cache-Control: max-age=0 > - ReqHeader X-Forwarded-For: 172.16.1.3 > - VCL_call RECV > - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; > _gid=GA1.1.117112441.1575877103 > - VCL_return hash > - ReqUnset Accept-Encoding: gzip, deflate > - ReqHeader Accept-Encoding: gzip > - VCL_call HASH > - VCL_return lookup > - VCL_call MISS > - VCL_return fetch > - Link bereq 32771 fetch > - Timestamp Fetch: 1575889894.493501 0.001842 0.001842 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespHeader Server: nginx > - RespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT > - RespHeader Content-Type: text/html > - RespHeader Content-Length: 340 > - RespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT > - RespHeader ETag: "5dcaad34-154" > - RespHeader X-Varnish: 8 6 > - RespHeader Via: 1.1 varnish-v4 > - RespHeader X-Varnish: 32770 > - RespHeader Age: 4270 > - RespHeader Via: 1.1 varnish-v4 > - VCL_call DELIVER > - VCL_return deliver > - Timestamp Process: 1575889894.493541 0.001881 0.000039 > - RespProtocol HTTP/1.1 > - RespStatus 304 > - RespReason Not Modified > - RespReason Not Modified > - RespUnset Content-Length: 340 > - Debug "RES_MODE 0" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1575889894.493599 0.001940 0.000058 > - ReqAcct 503 0 503 285 0 285 > - End > > * << Request >> 65538 > - Begin req 65537 rxreq > - Timestamp Start: 1575889962.124164 0.000000 0.000000 > - Timestamp Req: 1575889962.124164 0.000000 0.000000 > - ReqStart ::1 62064 > - ReqMethod PURGE > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: localhost > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader X-Forwarded-For: ::1 > - VCL_call RECV > - VCL_acl MATCH purge "localhost" > - VCL_return purge > - VCL_call HASH > - VCL_return lookup > - VCL_call PURGE > - VCL_return synth > - Timestamp Process: 1575889962.124300 0.000136 0.000136 > - RespHeader Date: Mon, 09 Dec 2019 11:12:42 GMT > - RespHeader Server: Varnish > - RespHeader X-Varnish: 65538 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespReason Purged > - VCL_call SYNTH > - RespHeader Content-Type: text/html; charset=utf-8 > - RespHeader Retry-After: 5 > - VCL_return deliver > - RespHeader Content-Length: 240 > - Storage malloc Transient > - RespHeader Accept-Ranges: bytes > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1575889962.124593 0.000428 0.000292 > - ReqAcct 88 0 88 218 240 458 > - End > > * << Session >> 65537 > - Begin sess 0 HTTP/1 > - SessOpen ::1 62064 :80 ::1 80 1575889962.124081 13 > - Link req 65538 rxreq > - SessClose REM_CLOSE 0.001 > - End > > * << Request >> 98306 > - Begin req 98305 rxreq > - Timestamp Start: 1575889974.116017 0.000000 0.000000 > - Timestamp Req: 1575889974.116017 0.000000 0.000000 > - ReqStart ::1 62066 > - ReqMethod PURGE > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: localhost > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader X-Forwarded-For: ::1 > - VCL_call RECV > - VCL_acl MATCH purge "localhost" > - VCL_return purge > - VCL_call HASH > - VCL_return lookup > - VCL_call PURGE > - VCL_return synth > - Timestamp Process: 1575889974.116128 0.000111 0.000111 > - RespHeader Date: Mon, 09 Dec 2019 11:12:54 GMT > - RespHeader Server: Varnish > - RespHeader X-Varnish: 98306 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespReason Purged > - VCL_call SYNTH > - RespHeader Content-Type: text/html; charset=utf-8 > - RespHeader Retry-After: 5 > - VCL_return deliver > - RespHeader Content-Length: 240 > - Storage malloc Transient > - RespHeader Accept-Ranges: bytes > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1575889974.116288 0.000271 0.000160 > - ReqAcct 88 0 88 218 240 458 > - End > > * << Session >> 98305 > - Begin sess 0 HTTP/1 > - SessOpen ::1 62066 :80 ::1 80 1575889974.115935 15 > - Link req 98306 rxreq > - SessClose REM_CLOSE 0.002 > - End > > * << Session >> 32769 > - Begin sess 0 HTTP/1 > - SessOpen 172.16.1.3 42502 :80 192.168.200.13 80 > 1575889894.491572 14 > - Link req 32770 rxreq > - Link req 32772 rxreq > - VSL timeout > - End synth > > * << Request >> 32772 > - Begin req 32769 rxreq > - Timestamp Start: 1575889894.845556 0.000000 0.000000 > - Timestamp Req: 1575889894.845556 0.000000 0.000000 > - ReqStart 172.16.1.3 42502 > - ReqMethod GET > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: 192.168.200.13 > - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; > rv:70.0) Gecko/20100101 Firefox/70.0 > - ReqHeader Accept: > video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5 > - ReqHeader Accept-Language: en-US,en;q=0.5 > - ReqHeader Range: bytes=205560- > - ReqHeader Connection: keep-alive > - ReqHeader Referer: http://192.168.200.13/ > - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; > _gid=GA1.1.117112441.1575877103 > - ReqHeader X-Forwarded-For: 172.16.1.3 > - VCL_call RECV > - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; > _gid=GA1.1.117112441.1575877103 > - VCL_return hash > - VCL_call HASH > - VCL_return lookup > - Hit 3 > - VCL_call HIT > - VCL_return deliver > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespHeader Server: nginx > - RespHeader Date: Mon, 09 Dec 2019 11:06:51 GMT > - RespHeader Content-Type: text/plain > - RespHeader Content-Length: 23521499 > - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > - RespHeader ETag: "3a1794b0-166e8db" > - RespHeader X-Varnish: 32770 > - RespHeader Via: 1.1 varnish-v4 > - RespHeader X-Varnish: 32772 3 > - RespHeader Age: 288 > - RespHeader Via: 1.1 varnish-v4 > - VCL_call DELIVER > - VCL_return deliver > - Timestamp Process: 1575889894.845683 0.000127 0.000127 > - RespHeader Accept-Ranges: bytes > - RespHeader Content-Range: bytes 205560-23521498/23521499 > - RespProtocol HTTP/1.1 > - RespStatus 206 > - RespReason Partial Content > - RespReason Partial Content > - RespUnset Content-Length: 23521499 > - RespHeader Content-Length: 23315939 > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Debug "Hit idle send timeout, wrote = 10899908/23316330; > retrying" > - Debug "Hit idle send timeout, wrote = 258432/12416422; > retrying" > - Debug "Write error, retval = -1, len = 12157990, errno = > Resource temporarily unavailable" > - Timestamp Resp: 1575890074.846999 180.001443 180.001316 > - ReqAcct 419 0 419 391 23315939 23316330 > - End > > ########################## > ------------------------------ > *From:* Guillaume Quintard > *Sent:* Monday, December 9, 2019 11:57 AM > *To:* hamidreza hosseini > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt PURGE urls > > Are you sure that you purge both Varnish layers (file and RAM)? > > can you pastebin a varnishlog of the purge and of the subsequent request? > > -- > Guillaume Quintard > > > On Mon, Dec 9, 2019 at 2:55 AM hamidreza hosseini > wrote: > > because its steel in cach and i can download and see th urls that i purged > before! > its Interesting that it return 200 code from varnish!!! > > Best regards. > ------------------------------ > *From:* Guillaume Quintard > *Sent:* Monday, December 9, 2019 11:48 AM > *To:* hamidreza hosseini > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt PURGE urls > > Hi, > > How do you tell that it's not purging? > > Cheers, > > -- > Guillaume Quintard > > > On Mon, Dec 9, 2019 at 2:45 AM hamidreza hosseini > wrote: > > Hi, > > I've two varnish, that one of them use ram as backend and another file as > backend, requests first come to varnish(ram) and then go to another server > in varnish(file) and then go to nginx, this is my diagram: > > > Req=====> [varnish ram] =======> [varnish file] =====> [Nginx] > > And following file is my vcl configuration file that i'm using, > but when i want to purge a url it doesnt work and it steel remains in > cache, > Should i chage my config file? > What's wrong with varnish? > > Vcl config file: > > ### Naming note: use [0-9a-zA-Z_] only. no -. Or use camelCase ;-) > vcl 4.0; > > import directors; > > > backend varnish_malloc_01 { .host = "192.168.200.12"; .port = "8080"; } > > > sub vcl_init { > new varnish_cluster = directors.round_robin(); > varnish_cluster.add_backend(varnish_malloc_01); > > } > > > acl purge { > "localhost"; > "192.168.200.0"/24; > ; > } > > > sub vcl_recv { > > sub vcl_recv { > #set req.backend_hint = varnish_cluster.backend(); > unset req.http.Cookie; > if (req.method == "PURGE") { > ### TODO: also purge from the backend > if (!client.ip ~ purge) { > return(synth(405,"Not allowed.")); > } > return (purge); > } > } > > sub vcl_backend_response { > set beresp.ttl=48h; > # Don't cache 404 responses > if ( beresp.status == 404 ) { > set beresp.ttl = 120s; > set beresp.uncacheable = true; > return (deliver); > } > } > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrhosseini at hotmail.com Mon Dec 9 12:21:23 2019 From: hrhosseini at hotmail.com (hamidreza hosseini) Date: Mon, 9 Dec 2019 12:21:23 +0000 Subject: Varnish doesnt PURGE urls In-Reply-To: References: , Message-ID: I purge both of them even i delete the files from Nginx or swift so it purged from varnish front but it steel remains in varnish backend! ________________________________ From: Guillaume Quintard Sent: Monday, December 9, 2019 1:17 PM To: hamidreza hosseini Cc: varnish-misc at varnish-cache.org Subject: Re: Varnish doesnt PURGE urls when you purge, varnish creates a synthetic response, and doesn't go to the backend, so if you wish to purge that one too, you need to send a request to it too. -- Guillaume Quintard On Mon, Dec 9, 2019 at 4:08 AM hamidreza hosseini > wrote: I test it again and in this test the fron server was purged but the backend didnt purge what is the problem? ________________________________ From: varnish-misc > on behalf of hamidreza hosseini > Sent: Monday, December 9, 2019 3:34 AM To: Guillaume Quintard > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt PURGE urls this is my test servers, I purge with ip in production but steel it doesnt work! ________________________________ From: Guillaume Quintard > Sent: Monday, December 9, 2019 12:31 PM To: hamidreza hosseini > Cc: shafeeque aslam >; varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt PURGE urls Purge uses the hash of the objects to clear them. In turn, the hash, by default depends on the host of the requests that created the object. So, when you purge using localhost/some/path.mp4, you dont purge the object referring to my.website.com/some/path.mp4 -- Guillaume Quintard On Mon, Dec 9, 2019 at 3:24 AM hamidreza hosseini > wrote: This is purge to another server: $curl -v -k -X PURGE http://localhost:8080/Naserfeiz.mp4 * Trying ::1... * Connected to localhost (::1) port 8080 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: localhost:8080 > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 200 Purged < Date: Mon, 09 Dec 2019 11:20:52 GMT < Server: Varnish < X-Varnish: 10 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 237 < Accept-Ranges: bytes < Connection: keep-alive < 200 Purged

Error 200 Purged

Purged

Guru Meditation:

XID: 10


Varnish cache server

* Connection #0 to host localhost left intact And varnishlog resault: $ sudo varnishlog * << Request >> 32773 - Begin req 32772 rxreq - Timestamp Start: 1575890593.662132 0.000000 0.000000 - Timestamp Req: 1575890593.662132 0.000000 0.000000 - ReqStart ::1 63184 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575890593.662268 0.000135 0.000135 - RespHeader Date: Mon, 09 Dec 2019 11:23:13 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 32773 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575890593.662484 0.000352 0.000216 - ReqAcct 93 0 93 218 240 458 - End * << Session >> 32772 - Begin sess 0 HTTP/1 - SessOpen ::1 63184 :8080 ::1 8080 1575890593.662044 16 - Link req 32773 rxreq - SessClose REM_CLOSE 0.001 - End * << Request >> 12 - Begin req 11 rxreq - Timestamp Start: 1575890600.799371 0.000000 0.000000 - Timestamp Req: 1575890600.799371 0.000000 0.000000 - ReqStart ::1 63186 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575890600.799463 0.000092 0.000092 - RespHeader Date: Mon, 09 Dec 2019 11:23:20 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 12 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 237 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575890600.799611 0.000240 0.000147 - ReqAcct 93 0 93 215 237 452 - End * << Session >> 11 - Begin sess 0 HTTP/1 - SessOpen ::1 63186 :8080 ::1 8080 1575890600.799261 16 - Link req 12 rxreq - SessClose REM_CLOSE 0.001 - End ________________________________ From: shafeeque aslam > Sent: Monday, December 9, 2019 3:19 AM To: hamidreza hosseini > Cc: Guillaume Quintard >; varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt PURGE urls This is purge to 1st Varnish server only. Give another purge request to 2nd Varnish server giving it's port number. On Mon 9 Dec, 2019, 4:45 PM hamidreza hosseini, > wrote: Yes, surely i purge both servers This is my purge request: $curl -v -k -X PURGE http://localhost/Naserfeiz.mp4 * Trying ::1... * Connected to localhost (::1) port 80 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: localhost > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 200 Purged < Date: Mon, 09 Dec 2019 11:12:54 GMT < Server: Varnish < X-Varnish: 98306 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 240 < Accept-Ranges: bytes < Connection: keep-alive < 200 Purged

Error 200 Purged

Purged

Guru Meditation:

XID: 98306


Varnish cache server

* Connection #0 to host localhost left intact And this is varnishlog resault: * << BeReq >> 32771 - Begin bereq 32770 fetch - Timestamp Start: 1575889894.491843 0.000000 0.000000 - BereqMethod GET - BereqURL / - BereqProtocol HTTP/1.1 - BereqHeader Host: 192.168.200.13 - BereqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0 - BereqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 - BereqHeader Accept-Language: en-US,en;q=0.5 - BereqHeader Upgrade-Insecure-Requests: 1 - BereqHeader X-Forwarded-For: 172.16.1.3 - BereqHeader Accept-Encoding: gzip - BereqHeader X-Varnish: 32771 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 18 boot.varnish_malloc_01 192.168.200.12 8080 192.168.200.13 14088 - Timestamp Bereq: 1575889894.492444 0.000601 0.000601 - Timestamp Beresp: 1575889894.493276 0.001433 0.000832 - BerespProtocol HTTP/1.1 - BerespStatus 200 - BerespReason OK - BerespHeader Server: nginx - BerespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT - BerespHeader Content-Type: text/html - BerespHeader Content-Length: 340 - BerespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT - BerespHeader ETag: "5dcaad34-154" - BerespHeader X-Varnish: 8 6 - BerespHeader Age: 4270 - BerespHeader Via: 1.1 varnish-v4 - BerespHeader Accept-Ranges: bytes - BerespHeader Connection: keep-alive - TTL RFC 120 10 -1 1575889894 1575885624 1575885629 0 0 - VCL_call BACKEND_RESPONSE - TTL VCL 177070 10 0 1575885624 - VCL_return deliver - Storage malloc s0 - ObjProtocol HTTP/1.1 - ObjStatus 200 - ObjReason OK - ObjHeader Server: nginx - ObjHeader Date: Mon, 09 Dec 2019 10:00:29 GMT - ObjHeader Content-Type: text/html - ObjHeader Content-Length: 340 - ObjHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT - ObjHeader ETag: "5dcaad34-154" - ObjHeader X-Varnish: 8 6 - ObjHeader Via: 1.1 varnish-v4 - Fetch_Body 3 length stream - BackendReuse 18 boot.varnish_malloc_01 - Timestamp BerespBody: 1575889894.493474 0.001631 0.000198 - Length 340 - BereqAcct 336 0 336 279 340 619 - End * << Request >> 32770 - Begin req 32769 rxreq - Timestamp Start: 1575889894.491659 0.000000 0.000000 - Timestamp Req: 1575889894.491659 0.000000 0.000000 - ReqStart 172.16.1.3 42502 - ReqMethod GET - ReqURL / - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.13 - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0 - ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 - ReqHeader Accept-Language: en-US,en;q=0.5 - ReqHeader Accept-Encoding: gzip, deflate - ReqHeader Connection: keep-alive - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - ReqHeader Upgrade-Insecure-Requests: 1 - ReqHeader If-Modified-Since: Tue, 12 Nov 2019 13:01:40 GMT - ReqHeader If-None-Match: "5dcaad34-154" - ReqHeader Cache-Control: max-age=0 - ReqHeader X-Forwarded-For: 172.16.1.3 - VCL_call RECV - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - VCL_return hash - ReqUnset Accept-Encoding: gzip, deflate - ReqHeader Accept-Encoding: gzip - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 32771 fetch - Timestamp Fetch: 1575889894.493501 0.001842 0.001842 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT - RespHeader Content-Type: text/html - RespHeader Content-Length: 340 - RespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT - RespHeader ETag: "5dcaad34-154" - RespHeader X-Varnish: 8 6 - RespHeader Via: 1.1 varnish-v4 - RespHeader X-Varnish: 32770 - RespHeader Age: 4270 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1575889894.493541 0.001881 0.000039 - RespProtocol HTTP/1.1 - RespStatus 304 - RespReason Not Modified - RespReason Not Modified - RespUnset Content-Length: 340 - Debug "RES_MODE 0" - RespHeader Connection: keep-alive - Timestamp Resp: 1575889894.493599 0.001940 0.000058 - ReqAcct 503 0 503 285 0 285 - End * << Request >> 65538 - Begin req 65537 rxreq - Timestamp Start: 1575889962.124164 0.000000 0.000000 - Timestamp Req: 1575889962.124164 0.000000 0.000000 - ReqStart ::1 62064 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575889962.124300 0.000136 0.000136 - RespHeader Date: Mon, 09 Dec 2019 11:12:42 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 65538 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575889962.124593 0.000428 0.000292 - ReqAcct 88 0 88 218 240 458 - End * << Session >> 65537 - Begin sess 0 HTTP/1 - SessOpen ::1 62064 :80 ::1 80 1575889962.124081 13 - Link req 65538 rxreq - SessClose REM_CLOSE 0.001 - End * << Request >> 98306 - Begin req 98305 rxreq - Timestamp Start: 1575889974.116017 0.000000 0.000000 - Timestamp Req: 1575889974.116017 0.000000 0.000000 - ReqStart ::1 62066 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575889974.116128 0.000111 0.000111 - RespHeader Date: Mon, 09 Dec 2019 11:12:54 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 98306 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575889974.116288 0.000271 0.000160 - ReqAcct 88 0 88 218 240 458 - End * << Session >> 98305 - Begin sess 0 HTTP/1 - SessOpen ::1 62066 :80 ::1 80 1575889974.115935 15 - Link req 98306 rxreq - SessClose REM_CLOSE 0.002 - End * << Session >> 32769 - Begin sess 0 HTTP/1 - SessOpen 172.16.1.3 42502 :80 192.168.200.13 80 1575889894.491572 14 - Link req 32770 rxreq - Link req 32772 rxreq - VSL timeout - End synth * << Request >> 32772 - Begin req 32769 rxreq - Timestamp Start: 1575889894.845556 0.000000 0.000000 - Timestamp Req: 1575889894.845556 0.000000 0.000000 - ReqStart 172.16.1.3 42502 - ReqMethod GET - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.13 - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0 - ReqHeader Accept: video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5 - ReqHeader Accept-Language: en-US,en;q=0.5 - ReqHeader Range: bytes=205560- - ReqHeader Connection: keep-alive - ReqHeader Referer: http://192.168.200.13/ - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - ReqHeader X-Forwarded-For: 172.16.1.3 - VCL_call RECV - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - VCL_return hash - VCL_call HASH - VCL_return lookup - Hit 3 - VCL_call HIT - VCL_return deliver - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Mon, 09 Dec 2019 11:06:51 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 32770 - RespHeader Via: 1.1 varnish-v4 - RespHeader X-Varnish: 32772 3 - RespHeader Age: 288 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1575889894.845683 0.000127 0.000127 - RespHeader Accept-Ranges: bytes - RespHeader Content-Range: bytes 205560-23521498/23521499 - RespProtocol HTTP/1.1 - RespStatus 206 - RespReason Partial Content - RespReason Partial Content - RespUnset Content-Length: 23521499 - RespHeader Content-Length: 23315939 - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Debug "Hit idle send timeout, wrote = 10899908/23316330; retrying" - Debug "Hit idle send timeout, wrote = 258432/12416422; retrying" - Debug "Write error, retval = -1, len = 12157990, errno = Resource temporarily unavailable" - Timestamp Resp: 1575890074.846999 180.001443 180.001316 - ReqAcct 419 0 419 391 23315939 23316330 - End ########################## ________________________________ From: Guillaume Quintard > Sent: Monday, December 9, 2019 11:57 AM To: hamidreza hosseini > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt PURGE urls Are you sure that you purge both Varnish layers (file and RAM)? can you pastebin a varnishlog of the purge and of the subsequent request? -- Guillaume Quintard On Mon, Dec 9, 2019 at 2:55 AM hamidreza hosseini > wrote: because its steel in cach and i can download and see th urls that i purged before! its Interesting that it return 200 code from varnish!!! Best regards. ________________________________ From: Guillaume Quintard > Sent: Monday, December 9, 2019 11:48 AM To: hamidreza hosseini > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt PURGE urls Hi, How do you tell that it's not purging? Cheers, -- Guillaume Quintard On Mon, Dec 9, 2019 at 2:45 AM hamidreza hosseini > wrote: Hi, I've two varnish, that one of them use ram as backend and another file as backend, requests first come to varnish(ram) and then go to another server in varnish(file) and then go to nginx, this is my diagram: Req=====> [varnish ram] =======> [varnish file] =====> [Nginx] And following file is my vcl configuration file that i'm using, but when i want to purge a url it doesnt work and it steel remains in cache, Should i chage my config file? What's wrong with varnish? Vcl config file: ### Naming note: use [0-9a-zA-Z_] only. no -. Or use camelCase ;-) vcl 4.0; import directors; backend varnish_malloc_01 { .host = "192.168.200.12"; .port = "8080"; } sub vcl_init { new varnish_cluster = directors.round_robin(); varnish_cluster.add_backend(varnish_malloc_01); } acl purge { "localhost"; "192.168.200.0"/24; ; } sub vcl_recv { sub vcl_recv { #set req.backend_hint = varnish_cluster.backend(); unset req.http.Cookie; if (req.method == "PURGE") { ### TODO: also purge from the backend if (!client.ip ~ purge) { return(synth(405,"Not allowed.")); } return (purge); } } sub vcl_backend_response { set beresp.ttl=48h; # Don't cache 404 responses if ( beresp.status == 404 ) { set beresp.ttl = 120s; set beresp.uncacheable = true; return (deliver); } } _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrhosseini at hotmail.com Mon Dec 9 12:25:10 2019 From: hrhosseini at hotmail.com (hamidreza hosseini) Date: Mon, 9 Dec 2019 12:25:10 +0000 Subject: Varnish doesnt PURGE urls In-Reply-To: References: , , Message-ID: Oh my GOOOOOD... I'm going to get crazy!!! now i tested again and it didnt purge on both even the front! ________________________________ From: hamidreza hosseini Sent: Monday, December 9, 2019 4:21 AM To: Guillaume Quintard Cc: varnish-misc at varnish-cache.org Subject: Re: Varnish doesnt PURGE urls I purge both of them even i delete the files from Nginx or swift so it purged from varnish front but it steel remains in varnish backend! ________________________________ From: Guillaume Quintard Sent: Monday, December 9, 2019 1:17 PM To: hamidreza hosseini Cc: varnish-misc at varnish-cache.org Subject: Re: Varnish doesnt PURGE urls when you purge, varnish creates a synthetic response, and doesn't go to the backend, so if you wish to purge that one too, you need to send a request to it too. -- Guillaume Quintard On Mon, Dec 9, 2019 at 4:08 AM hamidreza hosseini > wrote: I test it again and in this test the fron server was purged but the backend didnt purge what is the problem? ________________________________ From: varnish-misc > on behalf of hamidreza hosseini > Sent: Monday, December 9, 2019 3:34 AM To: Guillaume Quintard > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt PURGE urls this is my test servers, I purge with ip in production but steel it doesnt work! ________________________________ From: Guillaume Quintard > Sent: Monday, December 9, 2019 12:31 PM To: hamidreza hosseini > Cc: shafeeque aslam >; varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt PURGE urls Purge uses the hash of the objects to clear them. In turn, the hash, by default depends on the host of the requests that created the object. So, when you purge using localhost/some/path.mp4, you dont purge the object referring to my.website.com/some/path.mp4 -- Guillaume Quintard On Mon, Dec 9, 2019 at 3:24 AM hamidreza hosseini > wrote: This is purge to another server: $curl -v -k -X PURGE http://localhost:8080/Naserfeiz.mp4 * Trying ::1... * Connected to localhost (::1) port 8080 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: localhost:8080 > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 200 Purged < Date: Mon, 09 Dec 2019 11:20:52 GMT < Server: Varnish < X-Varnish: 10 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 237 < Accept-Ranges: bytes < Connection: keep-alive < 200 Purged

Error 200 Purged

Purged

Guru Meditation:

XID: 10


Varnish cache server

* Connection #0 to host localhost left intact And varnishlog resault: $ sudo varnishlog * << Request >> 32773 - Begin req 32772 rxreq - Timestamp Start: 1575890593.662132 0.000000 0.000000 - Timestamp Req: 1575890593.662132 0.000000 0.000000 - ReqStart ::1 63184 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575890593.662268 0.000135 0.000135 - RespHeader Date: Mon, 09 Dec 2019 11:23:13 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 32773 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575890593.662484 0.000352 0.000216 - ReqAcct 93 0 93 218 240 458 - End * << Session >> 32772 - Begin sess 0 HTTP/1 - SessOpen ::1 63184 :8080 ::1 8080 1575890593.662044 16 - Link req 32773 rxreq - SessClose REM_CLOSE 0.001 - End * << Request >> 12 - Begin req 11 rxreq - Timestamp Start: 1575890600.799371 0.000000 0.000000 - Timestamp Req: 1575890600.799371 0.000000 0.000000 - ReqStart ::1 63186 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575890600.799463 0.000092 0.000092 - RespHeader Date: Mon, 09 Dec 2019 11:23:20 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 12 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 237 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575890600.799611 0.000240 0.000147 - ReqAcct 93 0 93 215 237 452 - End * << Session >> 11 - Begin sess 0 HTTP/1 - SessOpen ::1 63186 :8080 ::1 8080 1575890600.799261 16 - Link req 12 rxreq - SessClose REM_CLOSE 0.001 - End ________________________________ From: shafeeque aslam > Sent: Monday, December 9, 2019 3:19 AM To: hamidreza hosseini > Cc: Guillaume Quintard >; varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt PURGE urls This is purge to 1st Varnish server only. Give another purge request to 2nd Varnish server giving it's port number. On Mon 9 Dec, 2019, 4:45 PM hamidreza hosseini, > wrote: Yes, surely i purge both servers This is my purge request: $curl -v -k -X PURGE http://localhost/Naserfeiz.mp4 * Trying ::1... * Connected to localhost (::1) port 80 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: localhost > User-Agent: curl/7.47.0 > Accept: */* > < HTTP/1.1 200 Purged < Date: Mon, 09 Dec 2019 11:12:54 GMT < Server: Varnish < X-Varnish: 98306 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 240 < Accept-Ranges: bytes < Connection: keep-alive < 200 Purged

Error 200 Purged

Purged

Guru Meditation:

XID: 98306


Varnish cache server

* Connection #0 to host localhost left intact And this is varnishlog resault: * << BeReq >> 32771 - Begin bereq 32770 fetch - Timestamp Start: 1575889894.491843 0.000000 0.000000 - BereqMethod GET - BereqURL / - BereqProtocol HTTP/1.1 - BereqHeader Host: 192.168.200.13 - BereqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0 - BereqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 - BereqHeader Accept-Language: en-US,en;q=0.5 - BereqHeader Upgrade-Insecure-Requests: 1 - BereqHeader X-Forwarded-For: 172.16.1.3 - BereqHeader Accept-Encoding: gzip - BereqHeader X-Varnish: 32771 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 18 boot.varnish_malloc_01 192.168.200.12 8080 192.168.200.13 14088 - Timestamp Bereq: 1575889894.492444 0.000601 0.000601 - Timestamp Beresp: 1575889894.493276 0.001433 0.000832 - BerespProtocol HTTP/1.1 - BerespStatus 200 - BerespReason OK - BerespHeader Server: nginx - BerespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT - BerespHeader Content-Type: text/html - BerespHeader Content-Length: 340 - BerespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT - BerespHeader ETag: "5dcaad34-154" - BerespHeader X-Varnish: 8 6 - BerespHeader Age: 4270 - BerespHeader Via: 1.1 varnish-v4 - BerespHeader Accept-Ranges: bytes - BerespHeader Connection: keep-alive - TTL RFC 120 10 -1 1575889894 1575885624 1575885629 0 0 - VCL_call BACKEND_RESPONSE - TTL VCL 177070 10 0 1575885624 - VCL_return deliver - Storage malloc s0 - ObjProtocol HTTP/1.1 - ObjStatus 200 - ObjReason OK - ObjHeader Server: nginx - ObjHeader Date: Mon, 09 Dec 2019 10:00:29 GMT - ObjHeader Content-Type: text/html - ObjHeader Content-Length: 340 - ObjHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT - ObjHeader ETag: "5dcaad34-154" - ObjHeader X-Varnish: 8 6 - ObjHeader Via: 1.1 varnish-v4 - Fetch_Body 3 length stream - BackendReuse 18 boot.varnish_malloc_01 - Timestamp BerespBody: 1575889894.493474 0.001631 0.000198 - Length 340 - BereqAcct 336 0 336 279 340 619 - End * << Request >> 32770 - Begin req 32769 rxreq - Timestamp Start: 1575889894.491659 0.000000 0.000000 - Timestamp Req: 1575889894.491659 0.000000 0.000000 - ReqStart 172.16.1.3 42502 - ReqMethod GET - ReqURL / - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.13 - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0 - ReqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 - ReqHeader Accept-Language: en-US,en;q=0.5 - ReqHeader Accept-Encoding: gzip, deflate - ReqHeader Connection: keep-alive - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - ReqHeader Upgrade-Insecure-Requests: 1 - ReqHeader If-Modified-Since: Tue, 12 Nov 2019 13:01:40 GMT - ReqHeader If-None-Match: "5dcaad34-154" - ReqHeader Cache-Control: max-age=0 - ReqHeader X-Forwarded-For: 172.16.1.3 - VCL_call RECV - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - VCL_return hash - ReqUnset Accept-Encoding: gzip, deflate - ReqHeader Accept-Encoding: gzip - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 32771 fetch - Timestamp Fetch: 1575889894.493501 0.001842 0.001842 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT - RespHeader Content-Type: text/html - RespHeader Content-Length: 340 - RespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT - RespHeader ETag: "5dcaad34-154" - RespHeader X-Varnish: 8 6 - RespHeader Via: 1.1 varnish-v4 - RespHeader X-Varnish: 32770 - RespHeader Age: 4270 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1575889894.493541 0.001881 0.000039 - RespProtocol HTTP/1.1 - RespStatus 304 - RespReason Not Modified - RespReason Not Modified - RespUnset Content-Length: 340 - Debug "RES_MODE 0" - RespHeader Connection: keep-alive - Timestamp Resp: 1575889894.493599 0.001940 0.000058 - ReqAcct 503 0 503 285 0 285 - End * << Request >> 65538 - Begin req 65537 rxreq - Timestamp Start: 1575889962.124164 0.000000 0.000000 - Timestamp Req: 1575889962.124164 0.000000 0.000000 - ReqStart ::1 62064 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575889962.124300 0.000136 0.000136 - RespHeader Date: Mon, 09 Dec 2019 11:12:42 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 65538 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575889962.124593 0.000428 0.000292 - ReqAcct 88 0 88 218 240 458 - End * << Session >> 65537 - Begin sess 0 HTTP/1 - SessOpen ::1 62064 :80 ::1 80 1575889962.124081 13 - Link req 65538 rxreq - SessClose REM_CLOSE 0.001 - End * << Request >> 98306 - Begin req 98305 rxreq - Timestamp Start: 1575889974.116017 0.000000 0.000000 - Timestamp Req: 1575889974.116017 0.000000 0.000000 - ReqStart ::1 62066 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: localhost - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader X-Forwarded-For: ::1 - VCL_call RECV - VCL_acl MATCH purge "localhost" - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1575889974.116128 0.000111 0.000111 - RespHeader Date: Mon, 09 Dec 2019 11:12:54 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 98306 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Timestamp Resp: 1575889974.116288 0.000271 0.000160 - ReqAcct 88 0 88 218 240 458 - End * << Session >> 98305 - Begin sess 0 HTTP/1 - SessOpen ::1 62066 :80 ::1 80 1575889974.115935 15 - Link req 98306 rxreq - SessClose REM_CLOSE 0.002 - End * << Session >> 32769 - Begin sess 0 HTTP/1 - SessOpen 172.16.1.3 42502 :80 192.168.200.13 80 1575889894.491572 14 - Link req 32770 rxreq - Link req 32772 rxreq - VSL timeout - End synth * << Request >> 32772 - Begin req 32769 rxreq - Timestamp Start: 1575889894.845556 0.000000 0.000000 - Timestamp Req: 1575889894.845556 0.000000 0.000000 - ReqStart 172.16.1.3 42502 - ReqMethod GET - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.13 - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:70.0) Gecko/20100101 Firefox/70.0 - ReqHeader Accept: video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5 - ReqHeader Accept-Language: en-US,en;q=0.5 - ReqHeader Range: bytes=205560- - ReqHeader Connection: keep-alive - ReqHeader Referer: http://192.168.200.13/ - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - ReqHeader X-Forwarded-For: 172.16.1.3 - VCL_call RECV - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; _gid=GA1.1.117112441.1575877103 - VCL_return hash - VCL_call HASH - VCL_return lookup - Hit 3 - VCL_call HIT - VCL_return deliver - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Mon, 09 Dec 2019 11:06:51 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 32770 - RespHeader Via: 1.1 varnish-v4 - RespHeader X-Varnish: 32772 3 - RespHeader Age: 288 - RespHeader Via: 1.1 varnish-v4 - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1575889894.845683 0.000127 0.000127 - RespHeader Accept-Ranges: bytes - RespHeader Content-Range: bytes 205560-23521498/23521499 - RespProtocol HTTP/1.1 - RespStatus 206 - RespReason Partial Content - RespReason Partial Content - RespUnset Content-Length: 23521499 - RespHeader Content-Length: 23315939 - Debug "RES_MODE 2" - RespHeader Connection: keep-alive - Debug "Hit idle send timeout, wrote = 10899908/23316330; retrying" - Debug "Hit idle send timeout, wrote = 258432/12416422; retrying" - Debug "Write error, retval = -1, len = 12157990, errno = Resource temporarily unavailable" - Timestamp Resp: 1575890074.846999 180.001443 180.001316 - ReqAcct 419 0 419 391 23315939 23316330 - End ########################## ________________________________ From: Guillaume Quintard > Sent: Monday, December 9, 2019 11:57 AM To: hamidreza hosseini > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt PURGE urls Are you sure that you purge both Varnish layers (file and RAM)? can you pastebin a varnishlog of the purge and of the subsequent request? -- Guillaume Quintard On Mon, Dec 9, 2019 at 2:55 AM hamidreza hosseini > wrote: because its steel in cach and i can download and see th urls that i purged before! its Interesting that it return 200 code from varnish!!! Best regards. ________________________________ From: Guillaume Quintard > Sent: Monday, December 9, 2019 11:48 AM To: hamidreza hosseini > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt PURGE urls Hi, How do you tell that it's not purging? Cheers, -- Guillaume Quintard On Mon, Dec 9, 2019 at 2:45 AM hamidreza hosseini > wrote: Hi, I've two varnish, that one of them use ram as backend and another file as backend, requests first come to varnish(ram) and then go to another server in varnish(file) and then go to nginx, this is my diagram: Req=====> [varnish ram] =======> [varnish file] =====> [Nginx] And following file is my vcl configuration file that i'm using, but when i want to purge a url it doesnt work and it steel remains in cache, Should i chage my config file? What's wrong with varnish? Vcl config file: ### Naming note: use [0-9a-zA-Z_] only. no -. Or use camelCase ;-) vcl 4.0; import directors; backend varnish_malloc_01 { .host = "192.168.200.12"; .port = "8080"; } sub vcl_init { new varnish_cluster = directors.round_robin(); varnish_cluster.add_backend(varnish_malloc_01); } acl purge { "localhost"; "192.168.200.0"/24; ; } sub vcl_recv { sub vcl_recv { #set req.backend_hint = varnish_cluster.backend(); unset req.http.Cookie; if (req.method == "PURGE") { ### TODO: also purge from the backend if (!client.ip ~ purge) { return(synth(405,"Not allowed.")); } return (purge); } } sub vcl_backend_response { set beresp.ttl=48h; # Don't cache 404 responses if ( beresp.status == 404 ) { set beresp.ttl = 120s; set beresp.uncacheable = true; return (deliver); } } _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Mon Dec 9 21:27:48 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 9 Dec 2019 13:27:48 -0800 Subject: Varnish doesnt PURGE urls In-Reply-To: References: Message-ID: please, try to send fewer messages, with a bit more useful information. let's try this: curl -I http://localhost/Naserfeiz.mp4 -o /dev/null -H "track: yes" curl -v -k -X PURGE http://localhost/Naserfeiz.mp4 -o /dev/null -H "track: yes" curl -I http://localhost/Naserfeiz.mp4 -o /dev/null -H "track: yes" varnishlog -d -q "ReqHeader:track" -g request -- Guillaume Quintard On Mon, Dec 9, 2019 at 4:25 AM hamidreza hosseini wrote: > Oh my GOOOOOD... > I'm going to get crazy!!! > now i tested again and it didnt purge on both even the front! > ------------------------------ > *From:* hamidreza hosseini > *Sent:* Monday, December 9, 2019 4:21 AM > *To:* Guillaume Quintard > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt PURGE urls > > I purge both of them even i delete the files from Nginx or swift so it > purged from varnish front but it steel remains in varnish backend! > ------------------------------ > *From:* Guillaume Quintard > *Sent:* Monday, December 9, 2019 1:17 PM > *To:* hamidreza hosseini > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt PURGE urls > > when you purge, varnish creates a synthetic response, and doesn't go to > the backend, so if you wish to purge that one too, you need to send a > request to it too. > > -- > Guillaume Quintard > > > On Mon, Dec 9, 2019 at 4:08 AM hamidreza hosseini > wrote: > > I test it again and in this test the fron server was purged but the > backend didnt purge > what is the problem? > ------------------------------ > *From:* varnish-misc hotmail.com at varnish-cache.org> on behalf of hamidreza hosseini < > hrhosseini at hotmail.com> > *Sent:* Monday, December 9, 2019 3:34 AM > *To:* Guillaume Quintard > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt PURGE urls > > this is my test servers, I purge with ip in production but steel it doesnt > work! > ------------------------------ > *From:* Guillaume Quintard > *Sent:* Monday, December 9, 2019 12:31 PM > *To:* hamidreza hosseini > *Cc:* shafeeque aslam ; > varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt PURGE urls > > Purge uses the hash of the objects to clear them. In turn, the hash, by > default depends on the host of the requests that created the object. > > So, when you purge using localhost/some/path.mp4, you dont purge the > object referring to my.website.com/some/path.mp4 > > -- > Guillaume Quintard > > > On Mon, Dec 9, 2019 at 3:24 AM hamidreza hosseini > wrote: > > This is purge to another server: > > $curl -v -k -X PURGE http://localhost:8080/Naserfeiz.mp4 > > * Trying ::1... > * Connected to localhost (::1) port 8080 (#0) > > PURGE /Naserfeiz.mp4 HTTP/1.1 > > Host: localhost:8080 > > User-Agent: curl/7.47.0 > > Accept: */* > > > < HTTP/1.1 200 Purged > < Date: Mon, 09 Dec 2019 11:20:52 GMT > < Server: Varnish > < X-Varnish: 10 > < Content-Type: text/html; charset=utf-8 > < Retry-After: 5 > < Content-Length: 237 > < Accept-Ranges: bytes > < Connection: keep-alive > < > > > > 200 Purged > > >

Error 200 Purged

>

Purged

>

Guru Meditation:

>

XID: 10

>
>

Varnish cache server

> > > * Connection #0 to host localhost left intact > > And varnishlog resault: > > $ sudo varnishlog > > * << Request >> 32773 > - Begin req 32772 rxreq > - Timestamp Start: 1575890593.662132 0.000000 0.000000 > - Timestamp Req: 1575890593.662132 0.000000 0.000000 > - ReqStart ::1 63184 > - ReqMethod PURGE > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: localhost:8080 > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader X-Forwarded-For: ::1 > - VCL_call RECV > - VCL_acl MATCH purge "localhost" > - VCL_return purge > - VCL_call HASH > - VCL_return lookup > - VCL_call PURGE > - VCL_return synth > - Timestamp Process: 1575890593.662268 0.000135 0.000135 > - RespHeader Date: Mon, 09 Dec 2019 11:23:13 GMT > - RespHeader Server: Varnish > - RespHeader X-Varnish: 32773 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespReason Purged > - VCL_call SYNTH > - RespHeader Content-Type: text/html; charset=utf-8 > - RespHeader Retry-After: 5 > - VCL_return deliver > - RespHeader Content-Length: 240 > - Storage malloc Transient > - RespHeader Accept-Ranges: bytes > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1575890593.662484 0.000352 0.000216 > - ReqAcct 93 0 93 218 240 458 > - End > > * << Session >> 32772 > - Begin sess 0 HTTP/1 > - SessOpen ::1 63184 :8080 ::1 8080 1575890593.662044 16 > - Link req 32773 rxreq > - SessClose REM_CLOSE 0.001 > - End > > * << Request >> 12 > - Begin req 11 rxreq > - Timestamp Start: 1575890600.799371 0.000000 0.000000 > - Timestamp Req: 1575890600.799371 0.000000 0.000000 > - ReqStart ::1 63186 > - ReqMethod PURGE > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: localhost:8080 > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader X-Forwarded-For: ::1 > - VCL_call RECV > - VCL_acl MATCH purge "localhost" > - VCL_return purge > - VCL_call HASH > - VCL_return lookup > - VCL_call PURGE > - VCL_return synth > - Timestamp Process: 1575890600.799463 0.000092 0.000092 > - RespHeader Date: Mon, 09 Dec 2019 11:23:20 GMT > - RespHeader Server: Varnish > - RespHeader X-Varnish: 12 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespReason Purged > - VCL_call SYNTH > - RespHeader Content-Type: text/html; charset=utf-8 > - RespHeader Retry-After: 5 > - VCL_return deliver > - RespHeader Content-Length: 237 > - Storage malloc Transient > - RespHeader Accept-Ranges: bytes > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1575890600.799611 0.000240 0.000147 > - ReqAcct 93 0 93 215 237 452 > - End > > * << Session >> 11 > - Begin sess 0 HTTP/1 > - SessOpen ::1 63186 :8080 ::1 8080 1575890600.799261 16 > - Link req 12 rxreq > - SessClose REM_CLOSE 0.001 > - End > > ------------------------------ > *From:* shafeeque aslam > *Sent:* Monday, December 9, 2019 3:19 AM > *To:* hamidreza hosseini > *Cc:* Guillaume Quintard ; > varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt PURGE urls > > This is purge to 1st Varnish server only. Give another purge request to > 2nd Varnish server giving it's port number. > > On Mon 9 Dec, 2019, 4:45 PM hamidreza hosseini, > wrote: > > Yes, surely i purge both servers > This is my purge request: > > $curl -v -k -X PURGE http://localhost/Naserfeiz.mp4 > > * Trying ::1... > * Connected to localhost (::1) port 80 (#0) > > PURGE /Naserfeiz.mp4 HTTP/1.1 > > Host: localhost > > User-Agent: curl/7.47.0 > > Accept: */* > > > < HTTP/1.1 200 Purged > < Date: Mon, 09 Dec 2019 11:12:54 GMT > < Server: Varnish > < X-Varnish: 98306 > < Content-Type: text/html; charset=utf-8 > < Retry-After: 5 > < Content-Length: 240 > < Accept-Ranges: bytes > < Connection: keep-alive > < > > > > 200 Purged > > >

Error 200 Purged

>

Purged

>

Guru Meditation:

>

XID: 98306

>
>

Varnish cache server

> > > * Connection #0 to host localhost left intact > > > And this is varnishlog resault: > > * << BeReq >> 32771 > - Begin bereq 32770 fetch > - Timestamp Start: 1575889894.491843 0.000000 0.000000 > - BereqMethod GET > - BereqURL / > - BereqProtocol HTTP/1.1 > - BereqHeader Host: 192.168.200.13 > - BereqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; > rv:70.0) Gecko/20100101 Firefox/70.0 > - BereqHeader Accept: > text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > - BereqHeader Accept-Language: en-US,en;q=0.5 > - BereqHeader Upgrade-Insecure-Requests: 1 > - BereqHeader X-Forwarded-For: 172.16.1.3 > - BereqHeader Accept-Encoding: gzip > - BereqHeader X-Varnish: 32771 > - VCL_call BACKEND_FETCH > - VCL_return fetch > - BackendOpen 18 boot.varnish_malloc_01 192.168.200.12 8080 > 192.168.200.13 14088 > - Timestamp Bereq: 1575889894.492444 0.000601 0.000601 > - Timestamp Beresp: 1575889894.493276 0.001433 0.000832 > - BerespProtocol HTTP/1.1 > - BerespStatus 200 > - BerespReason OK > - BerespHeader Server: nginx > - BerespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT > - BerespHeader Content-Type: text/html > - BerespHeader Content-Length: 340 > - BerespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT > - BerespHeader ETag: "5dcaad34-154" > - BerespHeader X-Varnish: 8 6 > - BerespHeader Age: 4270 > - BerespHeader Via: 1.1 varnish-v4 > - BerespHeader Accept-Ranges: bytes > - BerespHeader Connection: keep-alive > - TTL RFC 120 10 -1 1575889894 1575885624 1575885629 0 0 > - VCL_call BACKEND_RESPONSE > - TTL VCL 177070 10 0 1575885624 > - VCL_return deliver > - Storage malloc s0 > - ObjProtocol HTTP/1.1 > - ObjStatus 200 > - ObjReason OK > - ObjHeader Server: nginx > - ObjHeader Date: Mon, 09 Dec 2019 10:00:29 GMT > - ObjHeader Content-Type: text/html > - ObjHeader Content-Length: 340 > - ObjHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT > - ObjHeader ETag: "5dcaad34-154" > - ObjHeader X-Varnish: 8 6 > - ObjHeader Via: 1.1 varnish-v4 > - Fetch_Body 3 length stream > - BackendReuse 18 boot.varnish_malloc_01 > - Timestamp BerespBody: 1575889894.493474 0.001631 0.000198 > - Length 340 > - BereqAcct 336 0 336 279 340 619 > - End > > * << Request >> 32770 > - Begin req 32769 rxreq > - Timestamp Start: 1575889894.491659 0.000000 0.000000 > - Timestamp Req: 1575889894.491659 0.000000 0.000000 > - ReqStart 172.16.1.3 42502 > - ReqMethod GET > - ReqURL / > - ReqProtocol HTTP/1.1 > - ReqHeader Host: 192.168.200.13 > - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; > rv:70.0) Gecko/20100101 Firefox/70.0 > - ReqHeader Accept: > text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 > - ReqHeader Accept-Language: en-US,en;q=0.5 > - ReqHeader Accept-Encoding: gzip, deflate > - ReqHeader Connection: keep-alive > - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; > _gid=GA1.1.117112441.1575877103 > - ReqHeader Upgrade-Insecure-Requests: 1 > - ReqHeader If-Modified-Since: Tue, 12 Nov 2019 13:01:40 GMT > - ReqHeader If-None-Match: "5dcaad34-154" > - ReqHeader Cache-Control: max-age=0 > - ReqHeader X-Forwarded-For: 172.16.1.3 > - VCL_call RECV > - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; > _gid=GA1.1.117112441.1575877103 > - VCL_return hash > - ReqUnset Accept-Encoding: gzip, deflate > - ReqHeader Accept-Encoding: gzip > - VCL_call HASH > - VCL_return lookup > - VCL_call MISS > - VCL_return fetch > - Link bereq 32771 fetch > - Timestamp Fetch: 1575889894.493501 0.001842 0.001842 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespHeader Server: nginx > - RespHeader Date: Mon, 09 Dec 2019 10:00:29 GMT > - RespHeader Content-Type: text/html > - RespHeader Content-Length: 340 > - RespHeader Last-Modified: Tue, 12 Nov 2019 13:01:40 GMT > - RespHeader ETag: "5dcaad34-154" > - RespHeader X-Varnish: 8 6 > - RespHeader Via: 1.1 varnish-v4 > - RespHeader X-Varnish: 32770 > - RespHeader Age: 4270 > - RespHeader Via: 1.1 varnish-v4 > - VCL_call DELIVER > - VCL_return deliver > - Timestamp Process: 1575889894.493541 0.001881 0.000039 > - RespProtocol HTTP/1.1 > - RespStatus 304 > - RespReason Not Modified > - RespReason Not Modified > - RespUnset Content-Length: 340 > - Debug "RES_MODE 0" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1575889894.493599 0.001940 0.000058 > - ReqAcct 503 0 503 285 0 285 > - End > > * << Request >> 65538 > - Begin req 65537 rxreq > - Timestamp Start: 1575889962.124164 0.000000 0.000000 > - Timestamp Req: 1575889962.124164 0.000000 0.000000 > - ReqStart ::1 62064 > - ReqMethod PURGE > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: localhost > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader X-Forwarded-For: ::1 > - VCL_call RECV > - VCL_acl MATCH purge "localhost" > - VCL_return purge > - VCL_call HASH > - VCL_return lookup > - VCL_call PURGE > - VCL_return synth > - Timestamp Process: 1575889962.124300 0.000136 0.000136 > - RespHeader Date: Mon, 09 Dec 2019 11:12:42 GMT > - RespHeader Server: Varnish > - RespHeader X-Varnish: 65538 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespReason Purged > - VCL_call SYNTH > - RespHeader Content-Type: text/html; charset=utf-8 > - RespHeader Retry-After: 5 > - VCL_return deliver > - RespHeader Content-Length: 240 > - Storage malloc Transient > - RespHeader Accept-Ranges: bytes > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1575889962.124593 0.000428 0.000292 > - ReqAcct 88 0 88 218 240 458 > - End > > * << Session >> 65537 > - Begin sess 0 HTTP/1 > - SessOpen ::1 62064 :80 ::1 80 1575889962.124081 13 > - Link req 65538 rxreq > - SessClose REM_CLOSE 0.001 > - End > > * << Request >> 98306 > - Begin req 98305 rxreq > - Timestamp Start: 1575889974.116017 0.000000 0.000000 > - Timestamp Req: 1575889974.116017 0.000000 0.000000 > - ReqStart ::1 62066 > - ReqMethod PURGE > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: localhost > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader X-Forwarded-For: ::1 > - VCL_call RECV > - VCL_acl MATCH purge "localhost" > - VCL_return purge > - VCL_call HASH > - VCL_return lookup > - VCL_call PURGE > - VCL_return synth > - Timestamp Process: 1575889974.116128 0.000111 0.000111 > - RespHeader Date: Mon, 09 Dec 2019 11:12:54 GMT > - RespHeader Server: Varnish > - RespHeader X-Varnish: 98306 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespReason Purged > - VCL_call SYNTH > - RespHeader Content-Type: text/html; charset=utf-8 > - RespHeader Retry-After: 5 > - VCL_return deliver > - RespHeader Content-Length: 240 > - Storage malloc Transient > - RespHeader Accept-Ranges: bytes > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Timestamp Resp: 1575889974.116288 0.000271 0.000160 > - ReqAcct 88 0 88 218 240 458 > - End > > * << Session >> 98305 > - Begin sess 0 HTTP/1 > - SessOpen ::1 62066 :80 ::1 80 1575889974.115935 15 > - Link req 98306 rxreq > - SessClose REM_CLOSE 0.002 > - End > > * << Session >> 32769 > - Begin sess 0 HTTP/1 > - SessOpen 172.16.1.3 42502 :80 192.168.200.13 80 > 1575889894.491572 14 > - Link req 32770 rxreq > - Link req 32772 rxreq > - VSL timeout > - End synth > > * << Request >> 32772 > - Begin req 32769 rxreq > - Timestamp Start: 1575889894.845556 0.000000 0.000000 > - Timestamp Req: 1575889894.845556 0.000000 0.000000 > - ReqStart 172.16.1.3 42502 > - ReqMethod GET > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: 192.168.200.13 > - ReqHeader User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; > rv:70.0) Gecko/20100101 Firefox/70.0 > - ReqHeader Accept: > video/webm,video/ogg,video/*;q=0.9,application/ogg;q=0.7,audio/*;q=0.6,*/*;q=0.5 > - ReqHeader Accept-Language: en-US,en;q=0.5 > - ReqHeader Range: bytes=205560- > - ReqHeader Connection: keep-alive > - ReqHeader Referer: http://192.168.200.13/ > - ReqHeader Cookie: _ga=GA1.1.1476719154.1572941620; > _gid=GA1.1.117112441.1575877103 > - ReqHeader X-Forwarded-For: 172.16.1.3 > - VCL_call RECV > - ReqUnset Cookie: _ga=GA1.1.1476719154.1572941620; > _gid=GA1.1.117112441.1575877103 > - VCL_return hash > - VCL_call HASH > - VCL_return lookup > - Hit 3 > - VCL_call HIT > - VCL_return deliver > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespHeader Server: nginx > - RespHeader Date: Mon, 09 Dec 2019 11:06:51 GMT > - RespHeader Content-Type: text/plain > - RespHeader Content-Length: 23521499 > - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > - RespHeader ETag: "3a1794b0-166e8db" > - RespHeader X-Varnish: 32770 > - RespHeader Via: 1.1 varnish-v4 > - RespHeader X-Varnish: 32772 3 > - RespHeader Age: 288 > - RespHeader Via: 1.1 varnish-v4 > - VCL_call DELIVER > - VCL_return deliver > - Timestamp Process: 1575889894.845683 0.000127 0.000127 > - RespHeader Accept-Ranges: bytes > - RespHeader Content-Range: bytes 205560-23521498/23521499 > - RespProtocol HTTP/1.1 > - RespStatus 206 > - RespReason Partial Content > - RespReason Partial Content > - RespUnset Content-Length: 23521499 > - RespHeader Content-Length: 23315939 > - Debug "RES_MODE 2" > - RespHeader Connection: keep-alive > - Debug "Hit idle send timeout, wrote = 10899908/23316330; > retrying" > - Debug "Hit idle send timeout, wrote = 258432/12416422; > retrying" > - Debug "Write error, retval = -1, len = 12157990, errno = > Resource temporarily unavailable" > - Timestamp Resp: 1575890074.846999 180.001443 180.001316 > - ReqAcct 419 0 419 391 23315939 23316330 > - End > > ########################## > ------------------------------ > *From:* Guillaume Quintard > *Sent:* Monday, December 9, 2019 11:57 AM > *To:* hamidreza hosseini > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt PURGE urls > > Are you sure that you purge both Varnish layers (file and RAM)? > > can you pastebin a varnishlog of the purge and of the subsequent request? > > -- > Guillaume Quintard > > > On Mon, Dec 9, 2019 at 2:55 AM hamidreza hosseini > wrote: > > because its steel in cach and i can download and see th urls that i purged > before! > its Interesting that it return 200 code from varnish!!! > > Best regards. > ------------------------------ > *From:* Guillaume Quintard > *Sent:* Monday, December 9, 2019 11:48 AM > *To:* hamidreza hosseini > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt PURGE urls > > Hi, > > How do you tell that it's not purging? > > Cheers, > > -- > Guillaume Quintard > > > On Mon, Dec 9, 2019 at 2:45 AM hamidreza hosseini > wrote: > > Hi, > > I've two varnish, that one of them use ram as backend and another file as > backend, requests first come to varnish(ram) and then go to another server > in varnish(file) and then go to nginx, this is my diagram: > > > Req=====> [varnish ram] =======> [varnish file] =====> [Nginx] > > And following file is my vcl configuration file that i'm using, > but when i want to purge a url it doesnt work and it steel remains in > cache, > Should i chage my config file? > What's wrong with varnish? > > Vcl config file: > > ### Naming note: use [0-9a-zA-Z_] only. no -. Or use camelCase ;-) > vcl 4.0; > > import directors; > > > backend varnish_malloc_01 { .host = "192.168.200.12"; .port = "8080"; } > > > sub vcl_init { > new varnish_cluster = directors.round_robin(); > varnish_cluster.add_backend(varnish_malloc_01); > > } > > > acl purge { > "localhost"; > "192.168.200.0"/24; > ; > } > > > sub vcl_recv { > > sub vcl_recv { > #set req.backend_hint = varnish_cluster.backend(); > unset req.http.Cookie; > if (req.method == "PURGE") { > ### TODO: also purge from the backend > if (!client.ip ~ purge) { > return(synth(405,"Not allowed.")); > } > return (purge); > } > } > > sub vcl_backend_response { > set beresp.ttl=48h; > # Don't cache 404 responses > if ( beresp.status == 404 ) { > set beresp.ttl = 120s; > set beresp.uncacheable = true; > return (deliver); > } > } > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Tue Dec 10 03:06:11 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Mon, 9 Dec 2019 19:06:11 -0800 Subject: Wrong VMOD_ABI_Version in 6.0.5 src rpm In-Reply-To: References: <1ea4dbfd-349d-5ff6-a992-4fadc8d0e540@uplex.de> Message-ID: Hum, interestingly the debs are ok... -- Guillaume Quintard On Wed, Nov 27, 2019 at 8:52 AM Guillaume Quintard < guillaume at varnish-software.com> wrote: > I stand corrected. I can have a look. > > -- > Guillaume Quintard > > > On Wed, Nov 27, 2019 at 8:40 AM Geoff Simmons wrote: > >> On 11/27/19 17:27, Guillaume Quintard wrote: >> >> >> >> >> https://packagecloud.io/varnishcache/varnish60lts/packages/el/7/varnish-6.0.5-1.el7.src.rpm >> >> Note version 6.0.5. >> >> >> contains a wrong VMOD_ABI_Version define. >> >> >> >> #define VMOD_ABI_Version "Varnish 6.0.4 >> >> 204a927f4a4283529fc89f5182fe8cc3f2d0f617" >> >> >> instead of >> >> >> #define VMOD_ABI_Version "Varnish 6.0.5 >> >> 3065ccaacc4bb537fb976a524bd808db42c5fe40" >> >> I think the problem here is not the usual hassle of updating VMODs with >> strict ABI compatibility. It looks like the wrong ABI version is declared. >> >> >> Best, >> Geoff >> -- >> ** * * UPLEX - Nils Goroll Systemoptimierung >> >> Scheffelstra?e 32 >> 22301 Hamburg >> >> Tel +49 40 2880 5731 >> Mob +49 176 636 90917 >> Fax +49 40 42949753 >> >> http://uplex.de >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrhosseini at hotmail.com Tue Dec 10 14:33:28 2019 From: hrhosseini at hotmail.com (hamidreza hosseini) Date: Tue, 10 Dec 2019 14:33:28 +0000 Subject: Varnish doesnt PURGE urls Message-ID: Hi, I've two varnish, that one of them use ram as backend and another file as backend, requests first come to varnish(ram) and then go to another server in varnish(file) and then go to nginx, this is my diagram: Req=====> [varnish ram (varnish fronted)] =======> [varnish file (varnish backend)] =====> [Nginx] And following file is my vcl configuration file that i'm using, when i purge a url there is no problem with frontend but it doesnt purge from backend I run varnishlog on both servers and i determines diffrences in link below: https://www.diffchecker.com/NRPsTEGO (Note that my problem is with varnish backend it doesnt purge the url, one of the wierd things is that logs show that the url purged successfully and returned 200 ok , but that url doesnt remove from cach ) How can i solve my problem? vcl config file: ### Naming note: use [0-9a-zA-Z_] only. no -. Or use camelCase ;-) vcl 4.0; import directors; backend varnish_malloc_01 { .host = "192.168.200.12"; .port = "8080"; } sub vcl_init { new varnish_cluster = directors.round_robin(); varnish_cluster.add_backend(varnish_malloc_01); } acl purge { "localhost"; "192.168.200.0"/24; ; } sub vcl_recv { sub vcl_recv { #set req.backend_hint = varnish_cluster.backend(); unset req.http.Cookie; if (req.method == "PURGE") { ### TODO: also purge from the backend if (!client.ip ~ purge) { return(synth(405,"Not allowed.")); } return (purge); } } sub vcl_backend_response { set beresp.ttl=48h; # Don't cache 404 responses if ( beresp.status == 404 ) { set beresp.ttl = 120s; set beresp.uncacheable = true; return (deliver); } } Best regards -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrhosseini at hotmail.com Wed Dec 11 14:06:51 2019 From: hrhosseini at hotmail.com (hamidreza hosseini) Date: Wed, 11 Dec 2019 14:06:51 +0000 Subject: Varnish doesnt purge Message-ID: Hi, please help me to solve this problem, this is the link that i explaine my question completely: https://varnish-cache.org/lists/pipermail/varnish-misc/2019-December/026754.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Wed Dec 11 17:44:22 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 11 Dec 2019 18:44:22 +0100 Subject: Varnish doesnt purge In-Reply-To: References: Message-ID: Yes it does, we have already proved it./ Each instance purges and will refetch the object for its respective backend. I explained how to test: start with the origin, change the data and check that you actually get the new object. Then add a varnish layer, purge, check, repeat. You are testing everything at once and are unable to isolate the issue because you refuse to take things step by step, and spamming won't change that. I'm sorry but until you have tested each layer individually and identified the one that fails, I won't be able to help. -- Guillaume Quintard On Wed, Dec 11, 2019 at 3:08 PM hamidreza hosseini wrote: > Hi, > please help me to solve this problem, this is the link that i explaine my > question completely: > > https://varnish-cache.org/lists/pipermail/varnish-misc/2019-December/026754.html > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrhosseini at hotmail.com Thu Dec 12 07:49:11 2019 From: hrhosseini at hotmail.com (hamidreza hosseini) Date: Thu, 12 Dec 2019 07:49:11 +0000 Subject: Varnish doesnt purge In-Reply-To: References: , Message-ID: I do all steps now again, this is the resault: STEP ONE: First of all i configure a varnish (with file backend) in front of swift/Nginx and i ask the url that exist in swift with curl, varnish-file$curl -I http://192.168.200.12:8080/Naserfeiz.mp4 -H "newtrack: yes" HTTP/1.1 200 OK Server: nginx Date: Thu, 12 Dec 2019 07:29:32 GMT Content-Type: text/plain Content-Length: 23521499 Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT ETag: "3a1794b0-166e8db" X-Varnish: 2 Age: 0 Via: 1.1 varnish (Varnish/6.0) Accept-Ranges: bytes Connection: keep-alive varnish cach that object and then i purge the url: varnish-file$ curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" * Trying 192.168.200.12... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.12 (192.168.200.12) port 8080 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: 192.168.200.12:8080 > User-Agent: curl/7.47.0 > Accept: */* > newtrack: yes > < HTTP/1.1 200 Purged < Date: Thu, 12 Dec 2019 07:29:16 GMT < Server: Varnish < X-Varnish: 32770 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 240 < Accept-Ranges: bytes < Connection: keep-alive < { [240 bytes data] 100 240 100 240 0 0 124k 0 --:--:-- --:--:-- --:--:-- 234k * Connection #0 to host 192.168.200.12 left intact The object purged successfully and it didnt exist in cach anymore. This is varnishlog resault: varnish-file$ sudo varnishlog -d -g request * << Request >> 2 - Begin req 1 rxreq - Timestamp Start: 1576135743.442646 0.000000 0.000000 - Timestamp Req: 1576135743.442646 0.000000 0.000000 - ReqStart 192.168.200.12 5432 a0 - ReqMethod HEAD - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 3 fetch - Timestamp Fetch: 1576135743.444347 0.001701 0.001701 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Thu, 12 Dec 2019 07:29:32 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 2 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish (Varnish/6.0) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1576135743.444382 0.001736 0.000036 - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135743.444474 0.001828 0.000092 - ReqAcct 112 0 112 295 0 295 - End ** << BeReq >> 3 -- Begin bereq 2 fetch -- VCL_use boot -- Timestamp Start: 1576135743.442807 0.000000 0.000000 -- BereqMethod HEAD -- BereqURL /Naserfeiz.mp4 -- BereqProtocol HTTP/1.1 -- BereqHeader Host: 192.168.200.12:8080 -- BereqHeader User-Agent: curl/7.47.0 -- BereqHeader Accept: */* -- BereqHeader newtrack: yes -- BereqHeader X-Forwarded-For: 192.168.200.12 -- BereqMethod GET -- BereqHeader Accept-Encoding: gzip -- BereqHeader X-Varnish: 3 -- VCL_call BACKEND_FETCH -- VCL_return fetch -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6846 -- BackendStart 192.168.200.11 8080 -- Timestamp Bereq: 1576135743.443586 0.000779 0.000779 -- Timestamp Beresp: 1576135743.444017 0.001210 0.000431 -- BerespProtocol HTTP/1.1 -- BerespStatus 200 -- BerespReason OK -- BerespHeader Server: nginx -- BerespHeader Date: Thu, 12 Dec 2019 07:29:32 GMT -- BerespHeader Content-Type: text/plain -- BerespHeader Content-Length: 23521499 -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT -- BerespHeader Connection: keep-alive -- BerespHeader ETag: "3a1794b0-166e8db" -- BerespHeader Accept-Ranges: bytes -- TTL RFC 120 10 0 1576135743 1576135743 1576135772 0 0 cacheable -- VCL_call BACKEND_RESPONSE -- TTL VCL 86400 10 0 1576135743 cacheable -- VCL_return deliver -- Storage file s0 -- Fetch_Body 3 length stream -- BackendReuse 27 boot.swift_proxy_1 -- Timestamp BerespBody: 1576135743.747835 0.305028 0.303818 -- Length 23521499 -- BereqAcct 181 0 181 241 23521499 23521740 -- End * << Request >> 32770 - Begin req 32769 rxreq - Timestamp Start: 1576135756.680792 0.000000 0.000000 - Timestamp Req: 1576135756.680792 0.000000 0.000000 - ReqStart 192.168.200.12 5438 a0 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1576135756.680982 0.000190 0.000190 - RespHeader Date: Thu, 12 Dec 2019 07:29:16 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 32770 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135756.681190 0.000399 0.000208 - ReqAcct 113 0 113 218 240 458 - End * << Request >> 32772 - Begin req 32771 rxreq - Timestamp Start: 1576135769.304679 0.000000 0.000000 - Timestamp Req: 1576135769.304679 0.000000 0.000000 - ReqStart 192.168.200.12 5440 a0 - ReqMethod HEAD - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 32773 fetch - Timestamp Fetch: 1576135769.306630 0.001951 0.001951 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Thu, 12 Dec 2019 07:29:58 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 32772 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish (Varnish/6.0) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1576135769.306657 0.001978 0.000027 - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135769.306748 0.002069 0.000091 - ReqAcct 112 0 112 299 0 299 - End ** << BeReq >> 32773 -- Begin bereq 32772 fetch -- VCL_use boot -- Timestamp Start: 1576135769.304834 0.000000 0.000000 -- BereqMethod HEAD -- BereqURL /Naserfeiz.mp4 -- BereqProtocol HTTP/1.1 -- BereqHeader Host: 192.168.200.12:8080 -- BereqHeader User-Agent: curl/7.47.0 -- BereqHeader Accept: */* -- BereqHeader newtrack: yes -- BereqHeader X-Forwarded-For: 192.168.200.12 -- BereqMethod GET -- BereqHeader Accept-Encoding: gzip -- BereqHeader X-Varnish: 32773 -- VCL_call BACKEND_FETCH -- VCL_return fetch -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6846 -- BackendStart 192.168.200.11 8080 -- Timestamp Bereq: 1576135769.304993 0.000160 0.000160 -- Timestamp Beresp: 1576135769.306170 0.001336 0.001176 -- BerespProtocol HTTP/1.1 -- BerespStatus 200 -- BerespReason OK -- BerespHeader Server: nginx -- BerespHeader Date: Thu, 12 Dec 2019 07:29:58 GMT -- BerespHeader Content-Type: text/plain -- BerespHeader Content-Length: 23521499 -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT -- BerespHeader Connection: keep-alive -- BerespHeader ETag: "3a1794b0-166e8db" -- BerespHeader Accept-Ranges: bytes -- TTL RFC 120 10 0 1576135769 1576135769 1576135798 0 0 cacheable -- VCL_call BACKEND_RESPONSE -- TTL VCL 86400 10 0 1576135769 cacheable -- VCL_return deliver -- Storage file s0 -- Fetch_Body 3 length stream -- BackendReuse 27 boot.swift_proxy_1 -- Timestamp BerespBody: 1576135769.645490 0.340657 0.339321 -- Length 23521499 -- BereqAcct 185 0 185 241 23521499 23521740 -- End ######## STEP TWO: First i restart varnish file and varnishlog then: I add another varnish (with ram backend) in fornt of the varnish file and then i ask the url again in varnish ram (because it is my frontend server and users can access media though this server): varnish-ram$ curl -I http://192.168.200.13/Naserfeiz.mp4 -H "newtrack: yes" HTTP/1.1 200 OK Server: nginx Date: Thu, 12 Dec 2019 07:33:05 GMT Content-Type: text/plain Content-Length: 23521499 Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT ETag: "3a1794b0-166e8db" X-Varnish: 2 Via: 1.1 varnish (Varnish/6.0) X-Varnish: 2 Age: 0 Via: 1.1 varnish-v4 Accept-Ranges: bytes Connection: keep-alive First varnish file cached the url and then Varnish ram fetched the object from varnish file and cached it , then i purged that url in varnish ram and it purged successfuly and it exactly removed from cached: varnish-ram:~$ curl -v -k -X PURGE http://192.168.200.13/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" * Trying 192.168.200.13... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.13 (192.168.200.13) port 80 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: 192.168.200.13 > User-Agent: curl/7.47.0 > Accept: */* > newtrack: yes > < HTTP/1.1 200 Purged < Date: Thu, 12 Dec 2019 07:33:46 GMT < Server: Varnish < X-Varnish: 32770 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 240 < Accept-Ranges: bytes < Connection: keep-alive < { [240 bytes data] 100 240 100 240 0 0 73846 0 --:--:-- --:--:-- --:--:-- 117k * Connection #0 to host 192.168.200.13 left intact Then i purged the url in varnish file it shows "200 purged" but it still exist in varnish file!!!!!!: varnish-file$ curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" * Trying 192.168.200.12... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.12 (192.168.200.12) port 8080 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: 192.168.200.12:8080 > User-Agent: curl/7.47.0 > Accept: */* > newtrack: yes > < HTTP/1.1 200 Purged < Date: Thu, 12 Dec 2019 07:33:39 GMT < Server: Varnish < X-Varnish: 5 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 236 < Accept-Ranges: bytes < Connection: keep-alive < { [236 bytes data] 100 236 100 236 0 0 110k 0 --:--:-- --:--:-- --:--:-- 230k * Connection #0 to host 192.168.200.12 left intact This is varnishlog resault in varnish file: varnish-file:~$ sudo varnishlog -d -g request * << Request >> 2 - Begin req 1 rxreq - Timestamp Start: 1576135956.460768 0.000000 0.000000 - Timestamp Req: 1576135956.460768 0.000000 0.000000 - ReqStart 192.168.200.13 22300 a0 - ReqMethod GET - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.13 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.13 - ReqHeader Accept-Encoding: gzip - ReqHeader X-Varnish: 3 - ReqUnset X-Forwarded-For: 192.168.200.13 - ReqHeader X-Forwarded-For: 192.168.200.13, 192.168.200.13 - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 3 fetch - Timestamp Fetch: 1576135956.463065 0.002297 0.002297 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Thu, 12 Dec 2019 07:33:05 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 2 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish (Varnish/6.0) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1576135956.463114 0.002346 0.000049 - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135956.758920 0.298152 0.295806 - ReqAcct 176 0 176 295 23521499 23521794 - End ** << BeReq >> 3 -- Begin bereq 2 fetch -- VCL_use boot -- Timestamp Start: 1576135956.461099 0.000000 0.000000 -- BereqMethod GET -- BereqURL /Naserfeiz.mp4 -- BereqProtocol HTTP/1.1 -- BereqHeader Host: 192.168.200.13 -- BereqHeader User-Agent: curl/7.47.0 -- BereqHeader Accept: */* -- BereqHeader newtrack: yes -- BereqHeader Accept-Encoding: gzip -- BereqHeader X-Varnish: 3 -- BereqHeader X-Forwarded-For: 192.168.200.13, 192.168.200.13 -- BereqHeader X-Varnish: 3 -- VCL_call BACKEND_FETCH -- VCL_return fetch -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6860 -- BackendStart 192.168.200.11 8080 -- Timestamp Bereq: 1576135956.461955 0.000856 0.000856 -- Timestamp Beresp: 1576135956.462607 0.001508 0.000652 -- BerespProtocol HTTP/1.1 -- BerespStatus 200 -- BerespReason OK -- BerespHeader Server: nginx -- BerespHeader Date: Thu, 12 Dec 2019 07:33:05 GMT -- BerespHeader Content-Type: text/plain -- BerespHeader Content-Length: 23521499 -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT -- BerespHeader Connection: keep-alive -- BerespHeader ETag: "3a1794b0-166e8db" -- BerespHeader Accept-Ranges: bytes -- TTL RFC 120 10 0 1576135956 1576135956 1576135985 0 0 cacheable -- VCL_call BACKEND_RESPONSE -- TTL VCL 86400 10 0 1576135956 cacheable -- VCL_return deliver -- Storage file s0 -- Fetch_Body 3 length stream -- BackendReuse 27 boot.swift_proxy_1 -- Timestamp BerespBody: 1576135956.758956 0.297858 0.296349 -- Length 23521499 -- BereqAcct 206 0 206 241 23521499 23521740 -- End * << Request >> 5 - Begin req 4 rxreq - Timestamp Start: 1576136019.230666 0.000000 0.000000 - Timestamp Req: 1576136019.230666 0.000000 0.000000 - ReqStart 192.168.200.12 5452 a0 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1576136019.230863 0.000198 0.000198 - RespHeader Date: Thu, 12 Dec 2019 07:33:39 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 5 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 236 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576136019.231138 0.000473 0.000275 - ReqAcct 113 0 113 214 236 450 - End ######## PLEASE test this scenario for yourself and you can see that i do correct and the object wont remove from varnish-file Please do this scenario for yourself ________________________________ From: Guillaume Quintard Sent: Wednesday, December 11, 2019 9:44 AM To: hamidreza hosseini Cc: varnish-misc at varnish-cache.org Subject: Re: Varnish doesnt purge Yes it does, we have already proved it./ Each instance purges and will refetch the object for its respective backend. I explained how to test: start with the origin, change the data and check that you actually get the new object. Then add a varnish layer, purge, check, repeat. You are testing everything at once and are unable to isolate the issue because you refuse to take things step by step, and spamming won't change that. I'm sorry but until you have tested each layer individually and identified the one that fails, I won't be able to help. -- Guillaume Quintard On Wed, Dec 11, 2019 at 3:08 PM hamidreza hosseini > wrote: Hi, please help me to solve this problem, this is the link that i explaine my question completely: https://varnish-cache.org/lists/pipermail/varnish-misc/2019-December/026754.html _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrhosseini at hotmail.com Sat Dec 14 07:13:37 2019 From: hrhosseini at hotmail.com (hamidreza hosseini) Date: Sat, 14 Dec 2019 07:13:37 +0000 Subject: Varnish doesnt purge In-Reply-To: References: , , Message-ID: Hi, Please annwer my question, I sent you my test ________________________________ From: varnish-misc on behalf of hamidreza hosseini Sent: Wednesday, December 11, 2019 11:49 PM To: Guillaume Quintard Cc: varnish-misc at varnish-cache.org Subject: Re: Varnish doesnt purge I do all steps now again, this is the resault: STEP ONE: First of all i configure a varnish (with file backend) in front of swift/Nginx and i ask the url that exist in swift with curl, varnish-file$curl -I http://192.168.200.12:8080/Naserfeiz.mp4 -H "newtrack: yes" HTTP/1.1 200 OK Server: nginx Date: Thu, 12 Dec 2019 07:29:32 GMT Content-Type: text/plain Content-Length: 23521499 Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT ETag: "3a1794b0-166e8db" X-Varnish: 2 Age: 0 Via: 1.1 varnish (Varnish/6.0) Accept-Ranges: bytes Connection: keep-alive varnish cach that object and then i purge the url: varnish-file$ curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" * Trying 192.168.200.12... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.12 (192.168.200.12) port 8080 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: 192.168.200.12:8080 > User-Agent: curl/7.47.0 > Accept: */* > newtrack: yes > < HTTP/1.1 200 Purged < Date: Thu, 12 Dec 2019 07:29:16 GMT < Server: Varnish < X-Varnish: 32770 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 240 < Accept-Ranges: bytes < Connection: keep-alive < { [240 bytes data] 100 240 100 240 0 0 124k 0 --:--:-- --:--:-- --:--:-- 234k * Connection #0 to host 192.168.200.12 left intact The object purged successfully and it didnt exist in cach anymore. This is varnishlog resault: varnish-file$ sudo varnishlog -d -g request * << Request >> 2 - Begin req 1 rxreq - Timestamp Start: 1576135743.442646 0.000000 0.000000 - Timestamp Req: 1576135743.442646 0.000000 0.000000 - ReqStart 192.168.200.12 5432 a0 - ReqMethod HEAD - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 3 fetch - Timestamp Fetch: 1576135743.444347 0.001701 0.001701 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Thu, 12 Dec 2019 07:29:32 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 2 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish (Varnish/6.0) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1576135743.444382 0.001736 0.000036 - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135743.444474 0.001828 0.000092 - ReqAcct 112 0 112 295 0 295 - End ** << BeReq >> 3 -- Begin bereq 2 fetch -- VCL_use boot -- Timestamp Start: 1576135743.442807 0.000000 0.000000 -- BereqMethod HEAD -- BereqURL /Naserfeiz.mp4 -- BereqProtocol HTTP/1.1 -- BereqHeader Host: 192.168.200.12:8080 -- BereqHeader User-Agent: curl/7.47.0 -- BereqHeader Accept: */* -- BereqHeader newtrack: yes -- BereqHeader X-Forwarded-For: 192.168.200.12 -- BereqMethod GET -- BereqHeader Accept-Encoding: gzip -- BereqHeader X-Varnish: 3 -- VCL_call BACKEND_FETCH -- VCL_return fetch -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6846 -- BackendStart 192.168.200.11 8080 -- Timestamp Bereq: 1576135743.443586 0.000779 0.000779 -- Timestamp Beresp: 1576135743.444017 0.001210 0.000431 -- BerespProtocol HTTP/1.1 -- BerespStatus 200 -- BerespReason OK -- BerespHeader Server: nginx -- BerespHeader Date: Thu, 12 Dec 2019 07:29:32 GMT -- BerespHeader Content-Type: text/plain -- BerespHeader Content-Length: 23521499 -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT -- BerespHeader Connection: keep-alive -- BerespHeader ETag: "3a1794b0-166e8db" -- BerespHeader Accept-Ranges: bytes -- TTL RFC 120 10 0 1576135743 1576135743 1576135772 0 0 cacheable -- VCL_call BACKEND_RESPONSE -- TTL VCL 86400 10 0 1576135743 cacheable -- VCL_return deliver -- Storage file s0 -- Fetch_Body 3 length stream -- BackendReuse 27 boot.swift_proxy_1 -- Timestamp BerespBody: 1576135743.747835 0.305028 0.303818 -- Length 23521499 -- BereqAcct 181 0 181 241 23521499 23521740 -- End * << Request >> 32770 - Begin req 32769 rxreq - Timestamp Start: 1576135756.680792 0.000000 0.000000 - Timestamp Req: 1576135756.680792 0.000000 0.000000 - ReqStart 192.168.200.12 5438 a0 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1576135756.680982 0.000190 0.000190 - RespHeader Date: Thu, 12 Dec 2019 07:29:16 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 32770 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135756.681190 0.000399 0.000208 - ReqAcct 113 0 113 218 240 458 - End * << Request >> 32772 - Begin req 32771 rxreq - Timestamp Start: 1576135769.304679 0.000000 0.000000 - Timestamp Req: 1576135769.304679 0.000000 0.000000 - ReqStart 192.168.200.12 5440 a0 - ReqMethod HEAD - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 32773 fetch - Timestamp Fetch: 1576135769.306630 0.001951 0.001951 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Thu, 12 Dec 2019 07:29:58 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 32772 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish (Varnish/6.0) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1576135769.306657 0.001978 0.000027 - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135769.306748 0.002069 0.000091 - ReqAcct 112 0 112 299 0 299 - End ** << BeReq >> 32773 -- Begin bereq 32772 fetch -- VCL_use boot -- Timestamp Start: 1576135769.304834 0.000000 0.000000 -- BereqMethod HEAD -- BereqURL /Naserfeiz.mp4 -- BereqProtocol HTTP/1.1 -- BereqHeader Host: 192.168.200.12:8080 -- BereqHeader User-Agent: curl/7.47.0 -- BereqHeader Accept: */* -- BereqHeader newtrack: yes -- BereqHeader X-Forwarded-For: 192.168.200.12 -- BereqMethod GET -- BereqHeader Accept-Encoding: gzip -- BereqHeader X-Varnish: 32773 -- VCL_call BACKEND_FETCH -- VCL_return fetch -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6846 -- BackendStart 192.168.200.11 8080 -- Timestamp Bereq: 1576135769.304993 0.000160 0.000160 -- Timestamp Beresp: 1576135769.306170 0.001336 0.001176 -- BerespProtocol HTTP/1.1 -- BerespStatus 200 -- BerespReason OK -- BerespHeader Server: nginx -- BerespHeader Date: Thu, 12 Dec 2019 07:29:58 GMT -- BerespHeader Content-Type: text/plain -- BerespHeader Content-Length: 23521499 -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT -- BerespHeader Connection: keep-alive -- BerespHeader ETag: "3a1794b0-166e8db" -- BerespHeader Accept-Ranges: bytes -- TTL RFC 120 10 0 1576135769 1576135769 1576135798 0 0 cacheable -- VCL_call BACKEND_RESPONSE -- TTL VCL 86400 10 0 1576135769 cacheable -- VCL_return deliver -- Storage file s0 -- Fetch_Body 3 length stream -- BackendReuse 27 boot.swift_proxy_1 -- Timestamp BerespBody: 1576135769.645490 0.340657 0.339321 -- Length 23521499 -- BereqAcct 185 0 185 241 23521499 23521740 -- End ######## STEP TWO: First i restart varnish file and varnishlog then: I add another varnish (with ram backend) in fornt of the varnish file and then i ask the url again in varnish ram (because it is my frontend server and users can access media though this server): varnish-ram$ curl -I http://192.168.200.13/Naserfeiz.mp4 -H "newtrack: yes" HTTP/1.1 200 OK Server: nginx Date: Thu, 12 Dec 2019 07:33:05 GMT Content-Type: text/plain Content-Length: 23521499 Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT ETag: "3a1794b0-166e8db" X-Varnish: 2 Via: 1.1 varnish (Varnish/6.0) X-Varnish: 2 Age: 0 Via: 1.1 varnish-v4 Accept-Ranges: bytes Connection: keep-alive First varnish file cached the url and then Varnish ram fetched the object from varnish file and cached it , then i purged that url in varnish ram and it purged successfuly and it exactly removed from cached: varnish-ram:~$ curl -v -k -X PURGE http://192.168.200.13/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" * Trying 192.168.200.13... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.13 (192.168.200.13) port 80 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: 192.168.200.13 > User-Agent: curl/7.47.0 > Accept: */* > newtrack: yes > < HTTP/1.1 200 Purged < Date: Thu, 12 Dec 2019 07:33:46 GMT < Server: Varnish < X-Varnish: 32770 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 240 < Accept-Ranges: bytes < Connection: keep-alive < { [240 bytes data] 100 240 100 240 0 0 73846 0 --:--:-- --:--:-- --:--:-- 117k * Connection #0 to host 192.168.200.13 left intact Then i purged the url in varnish file it shows "200 purged" but it still exist in varnish file!!!!!!: varnish-file$ curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" * Trying 192.168.200.12... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.12 (192.168.200.12) port 8080 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: 192.168.200.12:8080 > User-Agent: curl/7.47.0 > Accept: */* > newtrack: yes > < HTTP/1.1 200 Purged < Date: Thu, 12 Dec 2019 07:33:39 GMT < Server: Varnish < X-Varnish: 5 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 236 < Accept-Ranges: bytes < Connection: keep-alive < { [236 bytes data] 100 236 100 236 0 0 110k 0 --:--:-- --:--:-- --:--:-- 230k * Connection #0 to host 192.168.200.12 left intact This is varnishlog resault in varnish file: varnish-file:~$ sudo varnishlog -d -g request * << Request >> 2 - Begin req 1 rxreq - Timestamp Start: 1576135956.460768 0.000000 0.000000 - Timestamp Req: 1576135956.460768 0.000000 0.000000 - ReqStart 192.168.200.13 22300 a0 - ReqMethod GET - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.13 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.13 - ReqHeader Accept-Encoding: gzip - ReqHeader X-Varnish: 3 - ReqUnset X-Forwarded-For: 192.168.200.13 - ReqHeader X-Forwarded-For: 192.168.200.13, 192.168.200.13 - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 3 fetch - Timestamp Fetch: 1576135956.463065 0.002297 0.002297 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Thu, 12 Dec 2019 07:33:05 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 2 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish (Varnish/6.0) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1576135956.463114 0.002346 0.000049 - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135956.758920 0.298152 0.295806 - ReqAcct 176 0 176 295 23521499 23521794 - End ** << BeReq >> 3 -- Begin bereq 2 fetch -- VCL_use boot -- Timestamp Start: 1576135956.461099 0.000000 0.000000 -- BereqMethod GET -- BereqURL /Naserfeiz.mp4 -- BereqProtocol HTTP/1.1 -- BereqHeader Host: 192.168.200.13 -- BereqHeader User-Agent: curl/7.47.0 -- BereqHeader Accept: */* -- BereqHeader newtrack: yes -- BereqHeader Accept-Encoding: gzip -- BereqHeader X-Varnish: 3 -- BereqHeader X-Forwarded-For: 192.168.200.13, 192.168.200.13 -- BereqHeader X-Varnish: 3 -- VCL_call BACKEND_FETCH -- VCL_return fetch -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6860 -- BackendStart 192.168.200.11 8080 -- Timestamp Bereq: 1576135956.461955 0.000856 0.000856 -- Timestamp Beresp: 1576135956.462607 0.001508 0.000652 -- BerespProtocol HTTP/1.1 -- BerespStatus 200 -- BerespReason OK -- BerespHeader Server: nginx -- BerespHeader Date: Thu, 12 Dec 2019 07:33:05 GMT -- BerespHeader Content-Type: text/plain -- BerespHeader Content-Length: 23521499 -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT -- BerespHeader Connection: keep-alive -- BerespHeader ETag: "3a1794b0-166e8db" -- BerespHeader Accept-Ranges: bytes -- TTL RFC 120 10 0 1576135956 1576135956 1576135985 0 0 cacheable -- VCL_call BACKEND_RESPONSE -- TTL VCL 86400 10 0 1576135956 cacheable -- VCL_return deliver -- Storage file s0 -- Fetch_Body 3 length stream -- BackendReuse 27 boot.swift_proxy_1 -- Timestamp BerespBody: 1576135956.758956 0.297858 0.296349 -- Length 23521499 -- BereqAcct 206 0 206 241 23521499 23521740 -- End * << Request >> 5 - Begin req 4 rxreq - Timestamp Start: 1576136019.230666 0.000000 0.000000 - Timestamp Req: 1576136019.230666 0.000000 0.000000 - ReqStart 192.168.200.12 5452 a0 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1576136019.230863 0.000198 0.000198 - RespHeader Date: Thu, 12 Dec 2019 07:33:39 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 5 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 236 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576136019.231138 0.000473 0.000275 - ReqAcct 113 0 113 214 236 450 - End ######## PLEASE test this scenario for yourself and you can see that i do correct and the object wont remove from varnish-file Please do this scenario for yourself ________________________________ From: Guillaume Quintard Sent: Wednesday, December 11, 2019 9:44 AM To: hamidreza hosseini Cc: varnish-misc at varnish-cache.org Subject: Re: Varnish doesnt purge Yes it does, we have already proved it./ Each instance purges and will refetch the object for its respective backend. I explained how to test: start with the origin, change the data and check that you actually get the new object. Then add a varnish layer, purge, check, repeat. You are testing everything at once and are unable to isolate the issue because you refuse to take things step by step, and spamming won't change that. I'm sorry but until you have tested each layer individually and identified the one that fails, I won't be able to help. -- Guillaume Quintard On Wed, Dec 11, 2019 at 3:08 PM hamidreza hosseini > wrote: Hi, please help me to solve this problem, this is the link that i explaine my question completely: https://varnish-cache.org/lists/pipermail/varnish-misc/2019-December/026754.html _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From vlad.rusu at lola.tech Sun Dec 15 16:18:42 2019 From: vlad.rusu at lola.tech (Vlad Rusu) Date: Sun, 15 Dec 2019 18:18:42 +0200 Subject: Varnish doesnt purge In-Reply-To: References: Message-ID: I appreciate Guillaume?s diplomacy :) Your issue (one of them at least) is that in your cascaded varnish setup the hash key of the cached url, in both varnish storages, is the one you use to originally access the asset. Your vcl hash routine might also be a custom one.. that would be the next thing to look at if what I?m saying here doesn?t fix it. So, purge http://192.168.200.13/Naserfeiz.mp4 and NOT http://192.168.200.12:8080/Naserfeiz.mp4 when in the cascaded setup Cheers On Sat, 14 Dec 2019 at 09:16, hamidreza hosseini wrote: > Hi, > Please annwer my question, I sent you my test > > ------------------------------ > *From:* varnish-misc hotmail.com at varnish-cache.org> on behalf of hamidreza hosseini < > hrhosseini at hotmail.com> > *Sent:* Wednesday, December 11, 2019 11:49 PM > *To:* Guillaume Quintard > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt purge > > I do all steps now again, this is the resault: > STEP ONE: > First of all i configure a varnish (with file backend) in front of > swift/Nginx and i ask the url that exist in swift with curl, > > varnish-file$curl -I http://192.168.200.12:8080/Naserfeiz.mp4 -H > "newtrack: yes" > > HTTP/1.1 200 OK > Server: nginx > Date: Thu, 12 Dec 2019 07:29:32 GMT > Content-Type: text/plain > Content-Length: 23521499 > Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > ETag: "3a1794b0-166e8db" > X-Varnish: 2 > Age: 0 > Via: 1.1 varnish (Varnish/6.0) > Accept-Ranges: bytes > Connection: keep-alive > > > varnish cach that object and then i purge the url: > > varnish-file$ curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 > -o /dev/null -H "newtrack: yes" > > * Trying 192.168.200.12... > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- > 0* Connected to 192.168.200.12 (192.168.200.12) port 8080 (#0) > > PURGE /Naserfeiz.mp4 HTTP/1.1 > > Host: 192.168.200.12:8080 > > User-Agent: curl/7.47.0 > > Accept: */* > > newtrack: yes > > > < HTTP/1.1 200 Purged > < Date: Thu, 12 Dec 2019 07:29:16 GMT > < Server: Varnish > < X-Varnish: 32770 > < Content-Type: text/html; charset=utf-8 > < Retry-After: 5 > < Content-Length: 240 > < Accept-Ranges: bytes > < Connection: keep-alive > < > { [240 bytes data] > 100 240 100 240 0 0 124k 0 --:--:-- --:--:-- --:--:-- > 234k > * Connection #0 to host 192.168.200.12 left intact > > The object purged successfully and it didnt exist in cach anymore. > > This is varnishlog resault: > > varnish-file$ sudo varnishlog -d -g request > * << Request >> 2 > - Begin req 1 rxreq > - Timestamp Start: 1576135743.442646 0.000000 0.000000 > - Timestamp Req: 1576135743.442646 0.000000 0.000000 > - ReqStart 192.168.200.12 5432 a0 > - ReqMethod HEAD > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: 192.168.200.12:8080 > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader newtrack: yes > - ReqHeader X-Forwarded-For: 192.168.200.12 > - VCL_call RECV > - VCL_return hash > - VCL_call HASH > - VCL_return lookup > - VCL_call MISS > - VCL_return fetch > - Link bereq 3 fetch > - Timestamp Fetch: 1576135743.444347 0.001701 0.001701 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespHeader Server: nginx > - RespHeader Date: Thu, 12 Dec 2019 07:29:32 GMT > - RespHeader Content-Type: text/plain > - RespHeader Content-Length: 23521499 > - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > - RespHeader ETag: "3a1794b0-166e8db" > - RespHeader X-Varnish: 2 > - RespHeader Age: 0 > - RespHeader Via: 1.1 varnish (Varnish/6.0) > - VCL_call DELIVER > - VCL_return deliver > - Timestamp Process: 1576135743.444382 0.001736 0.000036 > - RespHeader Accept-Ranges: bytes > - RespHeader Connection: keep-alive > - Timestamp Resp: 1576135743.444474 0.001828 0.000092 > - ReqAcct 112 0 112 295 0 295 > - End > ** << BeReq >> 3 > -- Begin bereq 2 fetch > -- VCL_use boot > -- Timestamp Start: 1576135743.442807 0.000000 0.000000 > -- BereqMethod HEAD > -- BereqURL /Naserfeiz.mp4 > -- BereqProtocol HTTP/1.1 > -- BereqHeader Host: 192.168.200.12:8080 > -- BereqHeader User-Agent: curl/7.47.0 > -- BereqHeader Accept: */* > -- BereqHeader newtrack: yes > -- BereqHeader X-Forwarded-For: 192.168.200.12 > -- BereqMethod GET > -- BereqHeader Accept-Encoding: gzip > -- BereqHeader X-Varnish: 3 > -- VCL_call BACKEND_FETCH > -- VCL_return fetch > -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 > 192.168.200.12 6846 > -- BackendStart 192.168.200.11 8080 > -- Timestamp Bereq: 1576135743.443586 0.000779 0.000779 > -- Timestamp Beresp: 1576135743.444017 0.001210 0.000431 > -- BerespProtocol HTTP/1.1 > -- BerespStatus 200 > -- BerespReason OK > -- BerespHeader Server: nginx > -- BerespHeader Date: Thu, 12 Dec 2019 07:29:32 GMT > -- BerespHeader Content-Type: text/plain > -- BerespHeader Content-Length: 23521499 > -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > -- BerespHeader Connection: keep-alive > -- BerespHeader ETag: "3a1794b0-166e8db" > -- BerespHeader Accept-Ranges: bytes > -- TTL RFC 120 10 0 1576135743 1576135743 1576135772 0 0 > cacheable > -- VCL_call BACKEND_RESPONSE > -- TTL VCL 86400 10 0 1576135743 cacheable > -- VCL_return deliver > -- Storage file s0 > -- Fetch_Body 3 length stream > -- BackendReuse 27 boot.swift_proxy_1 > -- Timestamp BerespBody: 1576135743.747835 0.305028 0.303818 > -- Length 23521499 > -- BereqAcct 181 0 181 241 23521499 23521740 > -- End > > * << Request >> 32770 > - Begin req 32769 rxreq > - Timestamp Start: 1576135756.680792 0.000000 0.000000 > - Timestamp Req: 1576135756.680792 0.000000 0.000000 > - ReqStart 192.168.200.12 5438 a0 > - ReqMethod PURGE > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: 192.168.200.12:8080 > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader newtrack: yes > - ReqHeader X-Forwarded-For: 192.168.200.12 > - VCL_call RECV > - VCL_return purge > - VCL_call HASH > - VCL_return lookup > - VCL_call PURGE > - VCL_return synth > - Timestamp Process: 1576135756.680982 0.000190 0.000190 > - RespHeader Date: Thu, 12 Dec 2019 07:29:16 GMT > - RespHeader Server: Varnish > - RespHeader X-Varnish: 32770 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespReason Purged > - VCL_call SYNTH > - RespHeader Content-Type: text/html; charset=utf-8 > - RespHeader Retry-After: 5 > - VCL_return deliver > - RespHeader Content-Length: 240 > - Storage malloc Transient > - RespHeader Accept-Ranges: bytes > - RespHeader Connection: keep-alive > - Timestamp Resp: 1576135756.681190 0.000399 0.000208 > - ReqAcct 113 0 113 218 240 458 > - End > > * << Request >> 32772 > - Begin req 32771 rxreq > - Timestamp Start: 1576135769.304679 0.000000 0.000000 > - Timestamp Req: 1576135769.304679 0.000000 0.000000 > - ReqStart 192.168.200.12 5440 a0 > - ReqMethod HEAD > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: 192.168.200.12:8080 > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader newtrack: yes > - ReqHeader X-Forwarded-For: 192.168.200.12 > - VCL_call RECV > - VCL_return hash > - VCL_call HASH > - VCL_return lookup > - VCL_call MISS > - VCL_return fetch > - Link bereq 32773 fetch > - Timestamp Fetch: 1576135769.306630 0.001951 0.001951 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespHeader Server: nginx > - RespHeader Date: Thu, 12 Dec 2019 07:29:58 GMT > - RespHeader Content-Type: text/plain > - RespHeader Content-Length: 23521499 > - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > - RespHeader ETag: "3a1794b0-166e8db" > - RespHeader X-Varnish: 32772 > - RespHeader Age: 0 > - RespHeader Via: 1.1 varnish (Varnish/6.0) > - VCL_call DELIVER > - VCL_return deliver > - Timestamp Process: 1576135769.306657 0.001978 0.000027 > - RespHeader Accept-Ranges: bytes > - RespHeader Connection: keep-alive > - Timestamp Resp: 1576135769.306748 0.002069 0.000091 > - ReqAcct 112 0 112 299 0 299 > - End > ** << BeReq >> 32773 > -- Begin bereq 32772 fetch > -- VCL_use boot > -- Timestamp Start: 1576135769.304834 0.000000 0.000000 > -- BereqMethod HEAD > -- BereqURL /Naserfeiz.mp4 > -- BereqProtocol HTTP/1.1 > -- BereqHeader Host: 192.168.200.12:8080 > -- BereqHeader User-Agent: curl/7.47.0 > -- BereqHeader Accept: */* > -- BereqHeader newtrack: yes > -- BereqHeader X-Forwarded-For: 192.168.200.12 > -- BereqMethod GET > -- BereqHeader Accept-Encoding: gzip > -- BereqHeader X-Varnish: 32773 > -- VCL_call BACKEND_FETCH > -- VCL_return fetch > -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 > 192.168.200.12 6846 > -- BackendStart 192.168.200.11 8080 > -- Timestamp Bereq: 1576135769.304993 0.000160 0.000160 > -- Timestamp Beresp: 1576135769.306170 0.001336 0.001176 > -- BerespProtocol HTTP/1.1 > -- BerespStatus 200 > -- BerespReason OK > -- BerespHeader Server: nginx > -- BerespHeader Date: Thu, 12 Dec 2019 07:29:58 GMT > -- BerespHeader Content-Type: text/plain > -- BerespHeader Content-Length: 23521499 > -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > -- BerespHeader Connection: keep-alive > -- BerespHeader ETag: "3a1794b0-166e8db" > -- BerespHeader Accept-Ranges: bytes > -- TTL RFC 120 10 0 1576135769 1576135769 1576135798 0 0 > cacheable > -- VCL_call BACKEND_RESPONSE > -- TTL VCL 86400 10 0 1576135769 cacheable > -- VCL_return deliver > -- Storage file s0 > -- Fetch_Body 3 length stream > -- BackendReuse 27 boot.swift_proxy_1 > -- Timestamp BerespBody: 1576135769.645490 0.340657 0.339321 > -- Length 23521499 > -- BereqAcct 185 0 185 241 23521499 23521740 > -- End > > > ######## > > STEP TWO: > First i restart varnish file and varnishlog then: > I add another varnish (with ram backend) in fornt of the varnish file and > then i ask the url again in varnish ram (because it is my frontend server > and users can access media though this server): > > varnish-ram$ curl -I http://192.168.200.13/Naserfeiz.mp4 -H "newtrack: > yes" > HTTP/1.1 200 OK > Server: nginx > Date: Thu, 12 Dec 2019 07:33:05 GMT > Content-Type: text/plain > Content-Length: 23521499 > Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > ETag: "3a1794b0-166e8db" > X-Varnish: 2 > Via: 1.1 varnish (Varnish/6.0) > X-Varnish: 2 > Age: 0 > Via: 1.1 varnish-v4 > Accept-Ranges: bytes > Connection: keep-alive > > > First varnish file cached the url and then Varnish ram fetched the object > from varnish file and cached it , then i purged that url in varnish ram and > it purged successfuly and it exactly removed from cached: > > varnish-ram:~$ curl -v -k -X PURGE http://192.168.200.13/Naserfeiz.mp4 -o > /dev/null -H "newtrack: yes" > * Trying 192.168.200.13... > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- > 0* Connected to 192.168.200.13 (192.168.200.13) port 80 (#0) > > PURGE /Naserfeiz.mp4 HTTP/1.1 > > Host: 192.168.200.13 > > User-Agent: curl/7.47.0 > > Accept: */* > > newtrack: yes > > > < HTTP/1.1 200 Purged > < Date: Thu, 12 Dec 2019 07:33:46 GMT > < Server: Varnish > < X-Varnish: 32770 > < Content-Type: text/html; charset=utf-8 > < Retry-After: 5 > < Content-Length: 240 > < Accept-Ranges: bytes > < Connection: keep-alive > < > { [240 bytes data] > 100 240 100 240 0 0 73846 0 --:--:-- --:--:-- --:--:-- > 117k > * Connection #0 to host 192.168.200.13 left intact > > > Then i purged the url in varnish file it shows "200 purged" but it still > exist in varnish file!!!!!!: > > varnish-file$ curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 > -o /dev/null -H "newtrack: yes" > * Trying 192.168.200.12... > % Total % Received % Xferd Average Speed Time Time Time > Current > Dload Upload Total Spent Left > Speed > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- > 0* Connected to 192.168.200.12 (192.168.200.12) port 8080 (#0) > > PURGE /Naserfeiz.mp4 HTTP/1.1 > > Host: 192.168.200.12:8080 > > User-Agent: curl/7.47.0 > > Accept: */* > > newtrack: yes > > > < HTTP/1.1 200 Purged > < Date: Thu, 12 Dec 2019 07:33:39 GMT > < Server: Varnish > < X-Varnish: 5 > < Content-Type: text/html; charset=utf-8 > < Retry-After: 5 > < Content-Length: 236 > < Accept-Ranges: bytes > < Connection: keep-alive > < > { [236 bytes data] > 100 236 100 236 0 0 110k 0 --:--:-- --:--:-- --:--:-- > 230k > * Connection #0 to host 192.168.200.12 left intact > > > This is varnishlog resault in varnish file: > > varnish-file:~$ sudo varnishlog -d -g request > * << Request >> 2 > - Begin req 1 rxreq > - Timestamp Start: 1576135956.460768 0.000000 0.000000 > - Timestamp Req: 1576135956.460768 0.000000 0.000000 > - ReqStart 192.168.200.13 22300 a0 > - ReqMethod GET > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: 192.168.200.13 > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader newtrack: yes > - ReqHeader X-Forwarded-For: 192.168.200.13 > - ReqHeader Accept-Encoding: gzip > - ReqHeader X-Varnish: 3 > - ReqUnset X-Forwarded-For: 192.168.200.13 > - ReqHeader X-Forwarded-For: 192.168.200.13, 192.168.200.13 > - VCL_call RECV > - VCL_return hash > - VCL_call HASH > - VCL_return lookup > - VCL_call MISS > - VCL_return fetch > - Link bereq 3 fetch > - Timestamp Fetch: 1576135956.463065 0.002297 0.002297 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespHeader Server: nginx > - RespHeader Date: Thu, 12 Dec 2019 07:33:05 GMT > - RespHeader Content-Type: text/plain > - RespHeader Content-Length: 23521499 > - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > - RespHeader ETag: "3a1794b0-166e8db" > - RespHeader X-Varnish: 2 > - RespHeader Age: 0 > - RespHeader Via: 1.1 varnish (Varnish/6.0) > - VCL_call DELIVER > - VCL_return deliver > - Timestamp Process: 1576135956.463114 0.002346 0.000049 > - RespHeader Accept-Ranges: bytes > - RespHeader Connection: keep-alive > - Timestamp Resp: 1576135956.758920 0.298152 0.295806 > - ReqAcct 176 0 176 295 23521499 23521794 > - End > ** << BeReq >> 3 > -- Begin bereq 2 fetch > -- VCL_use boot > -- Timestamp Start: 1576135956.461099 0.000000 0.000000 > -- BereqMethod GET > -- BereqURL /Naserfeiz.mp4 > -- BereqProtocol HTTP/1.1 > -- BereqHeader Host: 192.168.200.13 > -- BereqHeader User-Agent: curl/7.47.0 > -- BereqHeader Accept: */* > -- BereqHeader newtrack: yes > -- BereqHeader Accept-Encoding: gzip > -- BereqHeader X-Varnish: 3 > -- BereqHeader X-Forwarded-For: 192.168.200.13, 192.168.200.13 > -- BereqHeader X-Varnish: 3 > -- VCL_call BACKEND_FETCH > -- VCL_return fetch > -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 > 192.168.200.12 6860 > -- BackendStart 192.168.200.11 8080 > -- Timestamp Bereq: 1576135956.461955 0.000856 0.000856 > -- Timestamp Beresp: 1576135956.462607 0.001508 0.000652 > -- BerespProtocol HTTP/1.1 > -- BerespStatus 200 > -- BerespReason OK > -- BerespHeader Server: nginx > -- BerespHeader Date: Thu, 12 Dec 2019 07:33:05 GMT > -- BerespHeader Content-Type: text/plain > -- BerespHeader Content-Length: 23521499 > -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > -- BerespHeader Connection: keep-alive > -- BerespHeader ETag: "3a1794b0-166e8db" > -- BerespHeader Accept-Ranges: bytes > -- TTL RFC 120 10 0 1576135956 1576135956 1576135985 0 0 > cacheable > -- VCL_call BACKEND_RESPONSE > -- TTL VCL 86400 10 0 1576135956 cacheable > -- VCL_return deliver > -- Storage file s0 > -- Fetch_Body 3 length stream > -- BackendReuse 27 boot.swift_proxy_1 > -- Timestamp BerespBody: 1576135956.758956 0.297858 0.296349 > -- Length 23521499 > -- BereqAcct 206 0 206 241 23521499 23521740 > -- End > > * << Request >> 5 > - Begin req 4 rxreq > - Timestamp Start: 1576136019.230666 0.000000 0.000000 > - Timestamp Req: 1576136019.230666 0.000000 0.000000 > - ReqStart 192.168.200.12 5452 a0 > - ReqMethod PURGE > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: 192.168.200.12:8080 > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader newtrack: yes > - ReqHeader X-Forwarded-For: 192.168.200.12 > - VCL_call RECV > - VCL_return purge > - VCL_call HASH > - VCL_return lookup > - VCL_call PURGE > - VCL_return synth > - Timestamp Process: 1576136019.230863 0.000198 0.000198 > - RespHeader Date: Thu, 12 Dec 2019 07:33:39 GMT > - RespHeader Server: Varnish > - RespHeader X-Varnish: 5 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespReason Purged > - VCL_call SYNTH > - RespHeader Content-Type: text/html; charset=utf-8 > - RespHeader Retry-After: 5 > - VCL_return deliver > - RespHeader Content-Length: 236 > - Storage malloc Transient > - RespHeader Accept-Ranges: bytes > - RespHeader Connection: keep-alive > - Timestamp Resp: 1576136019.231138 0.000473 0.000275 > - ReqAcct 113 0 113 214 236 450 > - End > > ######## > PLEASE test this scenario for yourself and you can see that i do correct > and the object wont remove from varnish-file > Please do this scenario for yourself > > ------------------------------ > *From:* Guillaume Quintard > *Sent:* Wednesday, December 11, 2019 9:44 AM > *To:* hamidreza hosseini > *Cc:* varnish-misc at varnish-cache.org > *Subject:* Re: Varnish doesnt purge > > Yes it does, we have already proved it./ > > Each instance purges and will refetch the object for its respective > backend. I explained how to test: start with the origin, change the data > and check that you actually get the new object. Then add a varnish layer, > purge, check, repeat. > > You are testing everything at once and are unable to isolate the issue > because you refuse to take things step by step, and spamming won't change > that. > > I'm sorry but until you have tested each layer individually and identified > the one that fails, I won't be able to help. > > -- > Guillaume Quintard > > > On Wed, Dec 11, 2019 at 3:08 PM hamidreza hosseini > wrote: > > Hi, > please help me to solve this problem, this is the link that i explaine my > question completely: > > https://varnish-cache.org/lists/pipermail/varnish-misc/2019-December/026754.html > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- Sent from my iPhone -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrhosseini at hotmail.com Sun Dec 15 19:31:04 2019 From: hrhosseini at hotmail.com (hamidreza hosseini) Date: Sun, 15 Dec 2019 19:31:04 +0000 Subject: Varnish doesnt purge In-Reply-To: References: , Message-ID: I dont understand your message, I cant solve this problem Instead of saying "appericiate to your diplomacy!!!!!!!!!!" you can help me! I dont have enough knoledge on Varnish, I dont get your text about hash, i didnt use hash. Best Regards ________________________________ From: Vlad Rusu Sent: Sunday, December 15, 2019 7:48:42 PM To: hamidreza hosseini Cc: Guillaume Quintard ; varnish-misc at varnish-cache.org Subject: Re: Varnish doesnt purge I appreciate Guillaume?s diplomacy :) Your issue (one of them at least) is that in your cascaded varnish setup the hash key of the cached url, in both varnish storages, is the one you use to originally access the asset. Your vcl hash routine might also be a custom one.. that would be the next thing to look at if what I?m saying here doesn?t fix it. So, purge http://192.168.200.13/Naserfeiz.mp4 and NOT http://192.168.200.12:8080/Naserfeiz.mp4 when in the cascaded setup Cheers On Sat, 14 Dec 2019 at 09:16, hamidreza hosseini > wrote: Hi, Please annwer my question, I sent you my test ________________________________ From: varnish-misc > on behalf of hamidreza hosseini > Sent: Wednesday, December 11, 2019 11:49 PM To: Guillaume Quintard > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt purge I do all steps now again, this is the resault: STEP ONE: First of all i configure a varnish (with file backend) in front of swift/Nginx and i ask the url that exist in swift with curl, varnish-file$curl -I http://192.168.200.12:8080/Naserfeiz.mp4 -H "newtrack: yes" HTTP/1.1 200 OK Server: nginx Date: Thu, 12 Dec 2019 07:29:32 GMT Content-Type: text/plain Content-Length: 23521499 Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT ETag: "3a1794b0-166e8db" X-Varnish: 2 Age: 0 Via: 1.1 varnish (Varnish/6.0) Accept-Ranges: bytes Connection: keep-alive varnish cach that object and then i purge the url: varnish-file$ curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" * Trying 192.168.200.12... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.12 (192.168.200.12) port 8080 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: 192.168.200.12:8080 > User-Agent: curl/7.47.0 > Accept: */* > newtrack: yes > < HTTP/1.1 200 Purged < Date: Thu, 12 Dec 2019 07:29:16 GMT < Server: Varnish < X-Varnish: 32770 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 240 < Accept-Ranges: bytes < Connection: keep-alive < { [240 bytes data] 100 240 100 240 0 0 124k 0 --:--:-- --:--:-- --:--:-- 234k * Connection #0 to host 192.168.200.12 left intact The object purged successfully and it didnt exist in cach anymore. This is varnishlog resault: varnish-file$ sudo varnishlog -d -g request * << Request >> 2 - Begin req 1 rxreq - Timestamp Start: 1576135743.442646 0.000000 0.000000 - Timestamp Req: 1576135743.442646 0.000000 0.000000 - ReqStart 192.168.200.12 5432 a0 - ReqMethod HEAD - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 3 fetch - Timestamp Fetch: 1576135743.444347 0.001701 0.001701 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Thu, 12 Dec 2019 07:29:32 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 2 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish (Varnish/6.0) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1576135743.444382 0.001736 0.000036 - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135743.444474 0.001828 0.000092 - ReqAcct 112 0 112 295 0 295 - End ** << BeReq >> 3 -- Begin bereq 2 fetch -- VCL_use boot -- Timestamp Start: 1576135743.442807 0.000000 0.000000 -- BereqMethod HEAD -- BereqURL /Naserfeiz.mp4 -- BereqProtocol HTTP/1.1 -- BereqHeader Host: 192.168.200.12:8080 -- BereqHeader User-Agent: curl/7.47.0 -- BereqHeader Accept: */* -- BereqHeader newtrack: yes -- BereqHeader X-Forwarded-For: 192.168.200.12 -- BereqMethod GET -- BereqHeader Accept-Encoding: gzip -- BereqHeader X-Varnish: 3 -- VCL_call BACKEND_FETCH -- VCL_return fetch -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6846 -- BackendStart 192.168.200.11 8080 -- Timestamp Bereq: 1576135743.443586 0.000779 0.000779 -- Timestamp Beresp: 1576135743.444017 0.001210 0.000431 -- BerespProtocol HTTP/1.1 -- BerespStatus 200 -- BerespReason OK -- BerespHeader Server: nginx -- BerespHeader Date: Thu, 12 Dec 2019 07:29:32 GMT -- BerespHeader Content-Type: text/plain -- BerespHeader Content-Length: 23521499 -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT -- BerespHeader Connection: keep-alive -- BerespHeader ETag: "3a1794b0-166e8db" -- BerespHeader Accept-Ranges: bytes -- TTL RFC 120 10 0 1576135743 1576135743 1576135772 0 0 cacheable -- VCL_call BACKEND_RESPONSE -- TTL VCL 86400 10 0 1576135743 cacheable -- VCL_return deliver -- Storage file s0 -- Fetch_Body 3 length stream -- BackendReuse 27 boot.swift_proxy_1 -- Timestamp BerespBody: 1576135743.747835 0.305028 0.303818 -- Length 23521499 -- BereqAcct 181 0 181 241 23521499 23521740 -- End * << Request >> 32770 - Begin req 32769 rxreq - Timestamp Start: 1576135756.680792 0.000000 0.000000 - Timestamp Req: 1576135756.680792 0.000000 0.000000 - ReqStart 192.168.200.12 5438 a0 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1576135756.680982 0.000190 0.000190 - RespHeader Date: Thu, 12 Dec 2019 07:29:16 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 32770 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135756.681190 0.000399 0.000208 - ReqAcct 113 0 113 218 240 458 - End * << Request >> 32772 - Begin req 32771 rxreq - Timestamp Start: 1576135769.304679 0.000000 0.000000 - Timestamp Req: 1576135769.304679 0.000000 0.000000 - ReqStart 192.168.200.12 5440 a0 - ReqMethod HEAD - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 32773 fetch - Timestamp Fetch: 1576135769.306630 0.001951 0.001951 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Thu, 12 Dec 2019 07:29:58 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 32772 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish (Varnish/6.0) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1576135769.306657 0.001978 0.000027 - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135769.306748 0.002069 0.000091 - ReqAcct 112 0 112 299 0 299 - End ** << BeReq >> 32773 -- Begin bereq 32772 fetch -- VCL_use boot -- Timestamp Start: 1576135769.304834 0.000000 0.000000 -- BereqMethod HEAD -- BereqURL /Naserfeiz.mp4 -- BereqProtocol HTTP/1.1 -- BereqHeader Host: 192.168.200.12:8080 -- BereqHeader User-Agent: curl/7.47.0 -- BereqHeader Accept: */* -- BereqHeader newtrack: yes -- BereqHeader X-Forwarded-For: 192.168.200.12 -- BereqMethod GET -- BereqHeader Accept-Encoding: gzip -- BereqHeader X-Varnish: 32773 -- VCL_call BACKEND_FETCH -- VCL_return fetch -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6846 -- BackendStart 192.168.200.11 8080 -- Timestamp Bereq: 1576135769.304993 0.000160 0.000160 -- Timestamp Beresp: 1576135769.306170 0.001336 0.001176 -- BerespProtocol HTTP/1.1 -- BerespStatus 200 -- BerespReason OK -- BerespHeader Server: nginx -- BerespHeader Date: Thu, 12 Dec 2019 07:29:58 GMT -- BerespHeader Content-Type: text/plain -- BerespHeader Content-Length: 23521499 -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT -- BerespHeader Connection: keep-alive -- BerespHeader ETag: "3a1794b0-166e8db" -- BerespHeader Accept-Ranges: bytes -- TTL RFC 120 10 0 1576135769 1576135769 1576135798 0 0 cacheable -- VCL_call BACKEND_RESPONSE -- TTL VCL 86400 10 0 1576135769 cacheable -- VCL_return deliver -- Storage file s0 -- Fetch_Body 3 length stream -- BackendReuse 27 boot.swift_proxy_1 -- Timestamp BerespBody: 1576135769.645490 0.340657 0.339321 -- Length 23521499 -- BereqAcct 185 0 185 241 23521499 23521740 -- End ######## STEP TWO: First i restart varnish file and varnishlog then: I add another varnish (with ram backend) in fornt of the varnish file and then i ask the url again in varnish ram (because it is my frontend server and users can access media though this server): varnish-ram$ curl -I http://192.168.200.13/Naserfeiz.mp4 -H "newtrack: yes" HTTP/1.1 200 OK Server: nginx Date: Thu, 12 Dec 2019 07:33:05 GMT Content-Type: text/plain Content-Length: 23521499 Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT ETag: "3a1794b0-166e8db" X-Varnish: 2 Via: 1.1 varnish (Varnish/6.0) X-Varnish: 2 Age: 0 Via: 1.1 varnish-v4 Accept-Ranges: bytes Connection: keep-alive First varnish file cached the url and then Varnish ram fetched the object from varnish file and cached it , then i purged that url in varnish ram and it purged successfuly and it exactly removed from cached: varnish-ram:~$ curl -v -k -X PURGE http://192.168.200.13/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" * Trying 192.168.200.13... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.13 (192.168.200.13) port 80 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: 192.168.200.13 > User-Agent: curl/7.47.0 > Accept: */* > newtrack: yes > < HTTP/1.1 200 Purged < Date: Thu, 12 Dec 2019 07:33:46 GMT < Server: Varnish < X-Varnish: 32770 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 240 < Accept-Ranges: bytes < Connection: keep-alive < { [240 bytes data] 100 240 100 240 0 0 73846 0 --:--:-- --:--:-- --:--:-- 117k * Connection #0 to host 192.168.200.13 left intact Then i purged the url in varnish file it shows "200 purged" but it still exist in varnish file!!!!!!: varnish-file$ curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" * Trying 192.168.200.12... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.12 (192.168.200.12) port 8080 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: 192.168.200.12:8080 > User-Agent: curl/7.47.0 > Accept: */* > newtrack: yes > < HTTP/1.1 200 Purged < Date: Thu, 12 Dec 2019 07:33:39 GMT < Server: Varnish < X-Varnish: 5 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 236 < Accept-Ranges: bytes < Connection: keep-alive < { [236 bytes data] 100 236 100 236 0 0 110k 0 --:--:-- --:--:-- --:--:-- 230k * Connection #0 to host 192.168.200.12 left intact This is varnishlog resault in varnish file: varnish-file:~$ sudo varnishlog -d -g request * << Request >> 2 - Begin req 1 rxreq - Timestamp Start: 1576135956.460768 0.000000 0.000000 - Timestamp Req: 1576135956.460768 0.000000 0.000000 - ReqStart 192.168.200.13 22300 a0 - ReqMethod GET - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.13 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.13 - ReqHeader Accept-Encoding: gzip - ReqHeader X-Varnish: 3 - ReqUnset X-Forwarded-For: 192.168.200.13 - ReqHeader X-Forwarded-For: 192.168.200.13, 192.168.200.13 - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 3 fetch - Timestamp Fetch: 1576135956.463065 0.002297 0.002297 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Thu, 12 Dec 2019 07:33:05 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 2 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish (Varnish/6.0) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1576135956.463114 0.002346 0.000049 - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135956.758920 0.298152 0.295806 - ReqAcct 176 0 176 295 23521499 23521794 - End ** << BeReq >> 3 -- Begin bereq 2 fetch -- VCL_use boot -- Timestamp Start: 1576135956.461099 0.000000 0.000000 -- BereqMethod GET -- BereqURL /Naserfeiz.mp4 -- BereqProtocol HTTP/1.1 -- BereqHeader Host: 192.168.200.13 -- BereqHeader User-Agent: curl/7.47.0 -- BereqHeader Accept: */* -- BereqHeader newtrack: yes -- BereqHeader Accept-Encoding: gzip -- BereqHeader X-Varnish: 3 -- BereqHeader X-Forwarded-For: 192.168.200.13, 192.168.200.13 -- BereqHeader X-Varnish: 3 -- VCL_call BACKEND_FETCH -- VCL_return fetch -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6860 -- BackendStart 192.168.200.11 8080 -- Timestamp Bereq: 1576135956.461955 0.000856 0.000856 -- Timestamp Beresp: 1576135956.462607 0.001508 0.000652 -- BerespProtocol HTTP/1.1 -- BerespStatus 200 -- BerespReason OK -- BerespHeader Server: nginx -- BerespHeader Date: Thu, 12 Dec 2019 07:33:05 GMT -- BerespHeader Content-Type: text/plain -- BerespHeader Content-Length: 23521499 -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT -- BerespHeader Connection: keep-alive -- BerespHeader ETag: "3a1794b0-166e8db" -- BerespHeader Accept-Ranges: bytes -- TTL RFC 120 10 0 1576135956 1576135956 1576135985 0 0 cacheable -- VCL_call BACKEND_RESPONSE -- TTL VCL 86400 10 0 1576135956 cacheable -- VCL_return deliver -- Storage file s0 -- Fetch_Body 3 length stream -- BackendReuse 27 boot.swift_proxy_1 -- Timestamp BerespBody: 1576135956.758956 0.297858 0.296349 -- Length 23521499 -- BereqAcct 206 0 206 241 23521499 23521740 -- End * << Request >> 5 - Begin req 4 rxreq - Timestamp Start: 1576136019.230666 0.000000 0.000000 - Timestamp Req: 1576136019.230666 0.000000 0.000000 - ReqStart 192.168.200.12 5452 a0 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1576136019.230863 0.000198 0.000198 - RespHeader Date: Thu, 12 Dec 2019 07:33:39 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 5 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 236 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576136019.231138 0.000473 0.000275 - ReqAcct 113 0 113 214 236 450 - End ######## PLEASE test this scenario for yourself and you can see that i do correct and the object wont remove from varnish-file Please do this scenario for yourself ________________________________ From: Guillaume Quintard > Sent: Wednesday, December 11, 2019 9:44 AM To: hamidreza hosseini > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt purge Yes it does, we have already proved it./ Each instance purges and will refetch the object for its respective backend. I explained how to test: start with the origin, change the data and check that you actually get the new object. Then add a varnish layer, purge, check, repeat. You are testing everything at once and are unable to isolate the issue because you refuse to take things step by step, and spamming won't change that. I'm sorry but until you have tested each layer individually and identified the one that fails, I won't be able to help. -- Guillaume Quintard On Wed, Dec 11, 2019 at 3:08 PM hamidreza hosseini > wrote: Hi, please help me to solve this problem, this is the link that i explaine my question completely: https://varnish-cache.org/lists/pipermail/varnish-misc/2019-December/026754.html _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -- Sent from my iPhone -------------- next part -------------- An HTML attachment was scrubbed... URL: From vlad.rusu at lola.tech Sun Dec 15 19:46:59 2019 From: vlad.rusu at lola.tech (Vlad Rusu) Date: Sun, 15 Dec 2019 21:46:59 +0200 Subject: Varnish doesnt purge In-Reply-To: References: Message-ID: Try this when purging varnish ram: curl -v -k -X PURGE http://192.168.200.13/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" varnish file (2nd one): curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes? -H "Host: 192.168.200.13 ? Notice the difference? You to understand HOW content is stored in varnish?s storage in order to understand how to evict it - my comment about the hash routine Thanks > On 15 Dec 2019, at 21:31, hamidreza hosseini wrote: > > I dont understand your message, > > I cant solve this problem > > Instead of saying "appericiate to your diplomacy!!!!!!!!!!" you can help me! > > I dont have enough knoledge on Varnish, > > I dont get your text about hash, i didnt use hash. > > Best Regards > > > From: Vlad Rusu > Sent: Sunday, December 15, 2019 7:48:42 PM > To: hamidreza hosseini > Cc: Guillaume Quintard ; varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt purge > > I appreciate Guillaume?s diplomacy :) > > Your issue (one of them at least) is that in your cascaded varnish setup the hash key of the cached url, in both varnish storages, is the one you use to originally access the asset. Your vcl hash routine might also be a custom one.. that would be the next thing to look at if what I?m saying here doesn?t fix it. > > So, purge > http://192.168.200.13/Naserfeiz.mp4 > and NOT > http://192.168.200.12:8080/Naserfeiz.mp4 when in the cascaded setup > > Cheers > > On Sat, 14 Dec 2019 at 09:16, hamidreza hosseini > wrote: > Hi, > Please annwer my question, I sent you my test > > From: varnish-misc > on behalf of hamidreza hosseini > > Sent: Wednesday, December 11, 2019 11:49 PM > To: Guillaume Quintard > > Cc: varnish-misc at varnish-cache.org > > Subject: Re: Varnish doesnt purge > > I do all steps now again, this is the resault: > STEP ONE: > First of all i configure a varnish (with file backend) in front of swift/Nginx and i ask the url that exist in swift with curl, > > varnish-file$curl -I http://192.168.200.12:8080/Naserfeiz.mp4 -H "newtrack: yes" > > HTTP/1.1 200 OK > Server: nginx > Date: Thu, 12 Dec 2019 07:29:32 GMT > Content-Type: text/plain > Content-Length: 23521499 > Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > ETag: "3a1794b0-166e8db" > X-Varnish: 2 > Age: 0 > Via: 1.1 varnish (Varnish/6.0) > Accept-Ranges: bytes > Connection: keep-alive > > > varnish cach that object and then i purge the url: > > varnish-file$ curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" > > * Trying 192.168.200.12... > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.12 (192.168.200.12) port 8080 (#0) > > PURGE /Naserfeiz.mp4 HTTP/1.1 > > Host: 192.168.200.12:8080 > > User-Agent: curl/7.47.0 > > Accept: */* > > newtrack: yes > > > < HTTP/1.1 200 Purged > < Date: Thu, 12 Dec 2019 07:29:16 GMT > < Server: Varnish > < X-Varnish: 32770 > < Content-Type: text/html; charset=utf-8 > < Retry-After: 5 > < Content-Length: 240 > < Accept-Ranges: bytes > < Connection: keep-alive > < > { [240 bytes data] > 100 240 100 240 0 0 124k 0 --:--:-- --:--:-- --:--:-- 234k > * Connection #0 to host 192.168.200.12 left intact > > The object purged successfully and it didnt exist in cach anymore. > > This is varnishlog resault: > > varnish-file$ sudo varnishlog -d -g request > * << Request >> 2 > - Begin req 1 rxreq > - Timestamp Start: 1576135743.442646 0.000000 0.000000 > - Timestamp Req: 1576135743.442646 0.000000 0.000000 > - ReqStart 192.168.200.12 5432 a0 > - ReqMethod HEAD > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: 192.168.200.12:8080 > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader newtrack: yes > - ReqHeader X-Forwarded-For: 192.168.200.12 > - VCL_call RECV > - VCL_return hash > - VCL_call HASH > - VCL_return lookup > - VCL_call MISS > - VCL_return fetch > - Link bereq 3 fetch > - Timestamp Fetch: 1576135743.444347 0.001701 0.001701 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespHeader Server: nginx > - RespHeader Date: Thu, 12 Dec 2019 07:29:32 GMT > - RespHeader Content-Type: text/plain > - RespHeader Content-Length: 23521499 > - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > - RespHeader ETag: "3a1794b0-166e8db" > - RespHeader X-Varnish: 2 > - RespHeader Age: 0 > - RespHeader Via: 1.1 varnish (Varnish/6.0) > - VCL_call DELIVER > - VCL_return deliver > - Timestamp Process: 1576135743.444382 0.001736 0.000036 > - RespHeader Accept-Ranges: bytes > - RespHeader Connection: keep-alive > - Timestamp Resp: 1576135743.444474 0.001828 0.000092 > - ReqAcct 112 0 112 295 0 295 > - End > ** << BeReq >> 3 > -- Begin bereq 2 fetch > -- VCL_use boot > -- Timestamp Start: 1576135743.442807 0.000000 0.000000 > -- BereqMethod HEAD > -- BereqURL /Naserfeiz.mp4 > -- BereqProtocol HTTP/1.1 > -- BereqHeader Host: 192.168.200.12:8080 > -- BereqHeader User-Agent: curl/7.47.0 > -- BereqHeader Accept: */* > -- BereqHeader newtrack: yes > -- BereqHeader X-Forwarded-For: 192.168.200.12 > -- BereqMethod GET > -- BereqHeader Accept-Encoding: gzip > -- BereqHeader X-Varnish: 3 > -- VCL_call BACKEND_FETCH > -- VCL_return fetch > -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6846 > -- BackendStart 192.168.200.11 8080 > -- Timestamp Bereq: 1576135743.443586 0.000779 0.000779 > -- Timestamp Beresp: 1576135743.444017 0.001210 0.000431 > -- BerespProtocol HTTP/1.1 > -- BerespStatus 200 > -- BerespReason OK > -- BerespHeader Server: nginx > -- BerespHeader Date: Thu, 12 Dec 2019 07:29:32 GMT > -- BerespHeader Content-Type: text/plain > -- BerespHeader Content-Length: 23521499 > -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > -- BerespHeader Connection: keep-alive > -- BerespHeader ETag: "3a1794b0-166e8db" > -- BerespHeader Accept-Ranges: bytes > -- TTL RFC 120 10 0 1576135743 1576135743 1576135772 0 0 cacheable > -- VCL_call BACKEND_RESPONSE > -- TTL VCL 86400 10 0 1576135743 cacheable > -- VCL_return deliver > -- Storage file s0 > -- Fetch_Body 3 length stream > -- BackendReuse 27 boot.swift_proxy_1 > -- Timestamp BerespBody: 1576135743.747835 0.305028 0.303818 > -- Length 23521499 > -- BereqAcct 181 0 181 241 23521499 23521740 > -- End > > * << Request >> 32770 > - Begin req 32769 rxreq > - Timestamp Start: 1576135756.680792 0.000000 0.000000 > - Timestamp Req: 1576135756.680792 0.000000 0.000000 > - ReqStart 192.168.200.12 5438 a0 > - ReqMethod PURGE > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: 192.168.200.12:8080 > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader newtrack: yes > - ReqHeader X-Forwarded-For: 192.168.200.12 > - VCL_call RECV > - VCL_return purge > - VCL_call HASH > - VCL_return lookup > - VCL_call PURGE > - VCL_return synth > - Timestamp Process: 1576135756.680982 0.000190 0.000190 > - RespHeader Date: Thu, 12 Dec 2019 07:29:16 GMT > - RespHeader Server: Varnish > - RespHeader X-Varnish: 32770 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespReason Purged > - VCL_call SYNTH > - RespHeader Content-Type: text/html; charset=utf-8 > - RespHeader Retry-After: 5 > - VCL_return deliver > - RespHeader Content-Length: 240 > - Storage malloc Transient > - RespHeader Accept-Ranges: bytes > - RespHeader Connection: keep-alive > - Timestamp Resp: 1576135756.681190 0.000399 0.000208 > - ReqAcct 113 0 113 218 240 458 > - End > > * << Request >> 32772 > - Begin req 32771 rxreq > - Timestamp Start: 1576135769.304679 0.000000 0.000000 > - Timestamp Req: 1576135769.304679 0.000000 0.000000 > - ReqStart 192.168.200.12 5440 a0 > - ReqMethod HEAD > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: 192.168.200.12:8080 > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader newtrack: yes > - ReqHeader X-Forwarded-For: 192.168.200.12 > - VCL_call RECV > - VCL_return hash > - VCL_call HASH > - VCL_return lookup > - VCL_call MISS > - VCL_return fetch > - Link bereq 32773 fetch > - Timestamp Fetch: 1576135769.306630 0.001951 0.001951 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespHeader Server: nginx > - RespHeader Date: Thu, 12 Dec 2019 07:29:58 GMT > - RespHeader Content-Type: text/plain > - RespHeader Content-Length: 23521499 > - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > - RespHeader ETag: "3a1794b0-166e8db" > - RespHeader X-Varnish: 32772 > - RespHeader Age: 0 > - RespHeader Via: 1.1 varnish (Varnish/6.0) > - VCL_call DELIVER > - VCL_return deliver > - Timestamp Process: 1576135769.306657 0.001978 0.000027 > - RespHeader Accept-Ranges: bytes > - RespHeader Connection: keep-alive > - Timestamp Resp: 1576135769.306748 0.002069 0.000091 > - ReqAcct 112 0 112 299 0 299 > - End > ** << BeReq >> 32773 > -- Begin bereq 32772 fetch > -- VCL_use boot > -- Timestamp Start: 1576135769.304834 0.000000 0.000000 > -- BereqMethod HEAD > -- BereqURL /Naserfeiz.mp4 > -- BereqProtocol HTTP/1.1 > -- BereqHeader Host: 192.168.200.12:8080 > -- BereqHeader User-Agent: curl/7.47.0 > -- BereqHeader Accept: */* > -- BereqHeader newtrack: yes > -- BereqHeader X-Forwarded-For: 192.168.200.12 > -- BereqMethod GET > -- BereqHeader Accept-Encoding: gzip > -- BereqHeader X-Varnish: 32773 > -- VCL_call BACKEND_FETCH > -- VCL_return fetch > -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6846 > -- BackendStart 192.168.200.11 8080 > -- Timestamp Bereq: 1576135769.304993 0.000160 0.000160 > -- Timestamp Beresp: 1576135769.306170 0.001336 0.001176 > -- BerespProtocol HTTP/1.1 > -- BerespStatus 200 > -- BerespReason OK > -- BerespHeader Server: nginx > -- BerespHeader Date: Thu, 12 Dec 2019 07:29:58 GMT > -- BerespHeader Content-Type: text/plain > -- BerespHeader Content-Length: 23521499 > -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > -- BerespHeader Connection: keep-alive > -- BerespHeader ETag: "3a1794b0-166e8db" > -- BerespHeader Accept-Ranges: bytes > -- TTL RFC 120 10 0 1576135769 1576135769 1576135798 0 0 cacheable > -- VCL_call BACKEND_RESPONSE > -- TTL VCL 86400 10 0 1576135769 cacheable > -- VCL_return deliver > -- Storage file s0 > -- Fetch_Body 3 length stream > -- BackendReuse 27 boot.swift_proxy_1 > -- Timestamp BerespBody: 1576135769.645490 0.340657 0.339321 > -- Length 23521499 > -- BereqAcct 185 0 185 241 23521499 23521740 > -- End > > > ######## > > STEP TWO: > First i restart varnish file and varnishlog then: > I add another varnish (with ram backend) in fornt of the varnish file and then i ask the url again in varnish ram (because it is my frontend server and users can access media though this server): > > varnish-ram$ curl -I http://192.168.200.13/Naserfeiz.mp4 -H "newtrack: yes" > HTTP/1.1 200 OK > Server: nginx > Date: Thu, 12 Dec 2019 07:33:05 GMT > Content-Type: text/plain > Content-Length: 23521499 > Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > ETag: "3a1794b0-166e8db" > X-Varnish: 2 > Via: 1.1 varnish (Varnish/6.0) > X-Varnish: 2 > Age: 0 > Via: 1.1 varnish-v4 > Accept-Ranges: bytes > Connection: keep-alive > > > First varnish file cached the url and then Varnish ram fetched the object from varnish file and cached it , then i purged that url in varnish ram and it purged successfuly and it exactly removed from cached: > > varnish-ram:~$ curl -v -k -X PURGE http://192.168.200.13/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" > * Trying 192.168.200.13... > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.13 (192.168.200.13) port 80 (#0) > > PURGE /Naserfeiz.mp4 HTTP/1.1 > > Host: 192.168.200.13 > > User-Agent: curl/7.47.0 > > Accept: */* > > newtrack: yes > > > < HTTP/1.1 200 Purged > < Date: Thu, 12 Dec 2019 07:33:46 GMT > < Server: Varnish > < X-Varnish: 32770 > < Content-Type: text/html; charset=utf-8 > < Retry-After: 5 > < Content-Length: 240 > < Accept-Ranges: bytes > < Connection: keep-alive > < > { [240 bytes data] > 100 240 100 240 0 0 73846 0 --:--:-- --:--:-- --:--:-- 117k > * Connection #0 to host 192.168.200.13 left intact > > > Then i purged the url in varnish file it shows "200 purged" but it still exist in varnish file!!!!!!: > > varnish-file$ curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" > * Trying 192.168.200.12... > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.12 (192.168.200.12) port 8080 (#0) > > PURGE /Naserfeiz.mp4 HTTP/1.1 > > Host: 192.168.200.12:8080 > > User-Agent: curl/7.47.0 > > Accept: */* > > newtrack: yes > > > < HTTP/1.1 200 Purged > < Date: Thu, 12 Dec 2019 07:33:39 GMT > < Server: Varnish > < X-Varnish: 5 > < Content-Type: text/html; charset=utf-8 > < Retry-After: 5 > < Content-Length: 236 > < Accept-Ranges: bytes > < Connection: keep-alive > < > { [236 bytes data] > 100 236 100 236 0 0 110k 0 --:--:-- --:--:-- --:--:-- 230k > * Connection #0 to host 192.168.200.12 left intact > > > This is varnishlog resault in varnish file: > > varnish-file:~$ sudo varnishlog -d -g request > * << Request >> 2 > - Begin req 1 rxreq > - Timestamp Start: 1576135956.460768 0.000000 0.000000 > - Timestamp Req: 1576135956.460768 0.000000 0.000000 > - ReqStart 192.168.200.13 22300 a0 > - ReqMethod GET > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: 192.168.200.13 > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader newtrack: yes > - ReqHeader X-Forwarded-For: 192.168.200.13 > - ReqHeader Accept-Encoding: gzip > - ReqHeader X-Varnish: 3 > - ReqUnset X-Forwarded-For: 192.168.200.13 > - ReqHeader X-Forwarded-For: 192.168.200.13, 192.168.200.13 > - VCL_call RECV > - VCL_return hash > - VCL_call HASH > - VCL_return lookup > - VCL_call MISS > - VCL_return fetch > - Link bereq 3 fetch > - Timestamp Fetch: 1576135956.463065 0.002297 0.002297 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespHeader Server: nginx > - RespHeader Date: Thu, 12 Dec 2019 07:33:05 GMT > - RespHeader Content-Type: text/plain > - RespHeader Content-Length: 23521499 > - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > - RespHeader ETag: "3a1794b0-166e8db" > - RespHeader X-Varnish: 2 > - RespHeader Age: 0 > - RespHeader Via: 1.1 varnish (Varnish/6.0) > - VCL_call DELIVER > - VCL_return deliver > - Timestamp Process: 1576135956.463114 0.002346 0.000049 > - RespHeader Accept-Ranges: bytes > - RespHeader Connection: keep-alive > - Timestamp Resp: 1576135956.758920 0.298152 0.295806 > - ReqAcct 176 0 176 295 23521499 23521794 > - End > ** << BeReq >> 3 > -- Begin bereq 2 fetch > -- VCL_use boot > -- Timestamp Start: 1576135956.461099 0.000000 0.000000 > -- BereqMethod GET > -- BereqURL /Naserfeiz.mp4 > -- BereqProtocol HTTP/1.1 > -- BereqHeader Host: 192.168.200.13 > -- BereqHeader User-Agent: curl/7.47.0 > -- BereqHeader Accept: */* > -- BereqHeader newtrack: yes > -- BereqHeader Accept-Encoding: gzip > -- BereqHeader X-Varnish: 3 > -- BereqHeader X-Forwarded-For: 192.168.200.13, 192.168.200.13 > -- BereqHeader X-Varnish: 3 > -- VCL_call BACKEND_FETCH > -- VCL_return fetch > -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6860 > -- BackendStart 192.168.200.11 8080 > -- Timestamp Bereq: 1576135956.461955 0.000856 0.000856 > -- Timestamp Beresp: 1576135956.462607 0.001508 0.000652 > -- BerespProtocol HTTP/1.1 > -- BerespStatus 200 > -- BerespReason OK > -- BerespHeader Server: nginx > -- BerespHeader Date: Thu, 12 Dec 2019 07:33:05 GMT > -- BerespHeader Content-Type: text/plain > -- BerespHeader Content-Length: 23521499 > -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT > -- BerespHeader Connection: keep-alive > -- BerespHeader ETag: "3a1794b0-166e8db" > -- BerespHeader Accept-Ranges: bytes > -- TTL RFC 120 10 0 1576135956 1576135956 1576135985 0 0 cacheable > -- VCL_call BACKEND_RESPONSE > -- TTL VCL 86400 10 0 1576135956 cacheable > -- VCL_return deliver > -- Storage file s0 > -- Fetch_Body 3 length stream > -- BackendReuse 27 boot.swift_proxy_1 > -- Timestamp BerespBody: 1576135956.758956 0.297858 0.296349 > -- Length 23521499 > -- BereqAcct 206 0 206 241 23521499 23521740 > -- End > > * << Request >> 5 > - Begin req 4 rxreq > - Timestamp Start: 1576136019.230666 0.000000 0.000000 > - Timestamp Req: 1576136019.230666 0.000000 0.000000 > - ReqStart 192.168.200.12 5452 a0 > - ReqMethod PURGE > - ReqURL /Naserfeiz.mp4 > - ReqProtocol HTTP/1.1 > - ReqHeader Host: 192.168.200.12:8080 > - ReqHeader User-Agent: curl/7.47.0 > - ReqHeader Accept: */* > - ReqHeader newtrack: yes > - ReqHeader X-Forwarded-For: 192.168.200.12 > - VCL_call RECV > - VCL_return purge > - VCL_call HASH > - VCL_return lookup > - VCL_call PURGE > - VCL_return synth > - Timestamp Process: 1576136019.230863 0.000198 0.000198 > - RespHeader Date: Thu, 12 Dec 2019 07:33:39 GMT > - RespHeader Server: Varnish > - RespHeader X-Varnish: 5 > - RespProtocol HTTP/1.1 > - RespStatus 200 > - RespReason OK > - RespReason Purged > - VCL_call SYNTH > - RespHeader Content-Type: text/html; charset=utf-8 > - RespHeader Retry-After: 5 > - VCL_return deliver > - RespHeader Content-Length: 236 > - Storage malloc Transient > - RespHeader Accept-Ranges: bytes > - RespHeader Connection: keep-alive > - Timestamp Resp: 1576136019.231138 0.000473 0.000275 > - ReqAcct 113 0 113 214 236 450 > - End > > ######## > PLEASE test this scenario for yourself and you can see that i do correct and the object wont remove from varnish-file > Please do this scenario for yourself > > From: Guillaume Quintard > > Sent: Wednesday, December 11, 2019 9:44 AM > To: hamidreza hosseini > > Cc: varnish-misc at varnish-cache.org > > Subject: Re: Varnish doesnt purge > > Yes it does, we have already proved it./ > > Each instance purges and will refetch the object for its respective backend. I explained how to test: start with the origin, change the data and check that you actually get the new object. Then add a varnish layer, purge, check, repeat. > > You are testing everything at once and are unable to isolate the issue because you refuse to take things step by step, and spamming won't change that. > > I'm sorry but until you have tested each layer individually and identified the one that fails, I won't be able to help. > > -- > Guillaume Quintard > > > On Wed, Dec 11, 2019 at 3:08 PM hamidreza hosseini > wrote: > Hi, > please help me to solve this problem, this is the link that i explaine my question completely: > https://varnish-cache.org/lists/pipermail/varnish-misc/2019-December/026754.html > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -- > Sent from my iPhone -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrhosseini at hotmail.com Sun Dec 15 19:54:58 2019 From: hrhosseini at hotmail.com (hamidreza hosseini) Date: Sun, 15 Dec 2019 19:54:58 +0000 Subject: Varnish doesnt purge In-Reply-To: References: , Message-ID: Many Thanks for your answer You mean that in production i should add additional header to my purge request because it stored this with the header content like the ip address of first varnish OK, for purging i need wich header to add to my purge request and how can i get that what are that? And how they stored and how can i purge them Best Regards Get Outlook for Android ________________________________ From: Vlad Rusu Sent: Sunday, December 15, 2019 11:16:59 PM To: hamidreza hosseini Cc: Guillaume Quintard ; varnish-misc at varnish-cache.org Subject: Re: Varnish doesnt purge Try this when purging varnish ram: curl -v -k -X PURGE http://192.168.200.13/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" varnish file (2nd one): curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes? -H "Host: 192.168.200.13? Notice the difference? You to understand HOW content is stored in varnish?s storage in order to understand how to evict it - my comment about the hash routine Thanks On 15 Dec 2019, at 21:31, hamidreza hosseini > wrote: I dont understand your message, I cant solve this problem Instead of saying "appericiate to your diplomacy!!!!!!!!!!" you can help me! I dont have enough knoledge on Varnish, I dont get your text about hash, i didnt use hash. Best Regards ________________________________ From: Vlad Rusu > Sent: Sunday, December 15, 2019 7:48:42 PM To: hamidreza hosseini > Cc: Guillaume Quintard >; varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt purge I appreciate Guillaume?s diplomacy :) Your issue (one of them at least) is that in your cascaded varnish setup the hash key of the cached url, in both varnish storages, is the one you use to originally access the asset. Your vcl hash routine might also be a custom one.. that would be the next thing to look at if what I?m saying here doesn?t fix it. So, purge http://192.168.200.13/Naserfeiz.mp4 and NOT http://192.168.200.12:8080/Naserfeiz.mp4 when in the cascaded setup Cheers On Sat, 14 Dec 2019 at 09:16, hamidreza hosseini > wrote: Hi, Please annwer my question, I sent you my test ________________________________ From: varnish-misc > on behalf of hamidreza hosseini > Sent: Wednesday, December 11, 2019 11:49 PM To: Guillaume Quintard > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt purge I do all steps now again, this is the resault: STEP ONE: First of all i configure a varnish (with file backend) in front of swift/Nginx and i ask the url that exist in swift with curl, varnish-file$curl -I http://192.168.200.12:8080/Naserfeiz.mp4 -H "newtrack: yes" HTTP/1.1 200 OK Server: nginx Date: Thu, 12 Dec 2019 07:29:32 GMT Content-Type: text/plain Content-Length: 23521499 Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT ETag: "3a1794b0-166e8db" X-Varnish: 2 Age: 0 Via: 1.1 varnish (Varnish/6.0) Accept-Ranges: bytes Connection: keep-alive varnish cach that object and then i purge the url: varnish-file$ curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" * Trying 192.168.200.12... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.12 (192.168.200.12) port 8080 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: 192.168.200.12:8080 > User-Agent: curl/7.47.0 > Accept: */* > newtrack: yes > < HTTP/1.1 200 Purged < Date: Thu, 12 Dec 2019 07:29:16 GMT < Server: Varnish < X-Varnish: 32770 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 240 < Accept-Ranges: bytes < Connection: keep-alive < { [240 bytes data] 100 240 100 240 0 0 124k 0 --:--:-- --:--:-- --:--:-- 234k * Connection #0 to host 192.168.200.12 left intact The object purged successfully and it didnt exist in cach anymore. This is varnishlog resault: varnish-file$ sudo varnishlog -d -g request * << Request >> 2 - Begin req 1 rxreq - Timestamp Start: 1576135743.442646 0.000000 0.000000 - Timestamp Req: 1576135743.442646 0.000000 0.000000 - ReqStart 192.168.200.12 5432 a0 - ReqMethod HEAD - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 3 fetch - Timestamp Fetch: 1576135743.444347 0.001701 0.001701 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Thu, 12 Dec 2019 07:29:32 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 2 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish (Varnish/6.0) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1576135743.444382 0.001736 0.000036 - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135743.444474 0.001828 0.000092 - ReqAcct 112 0 112 295 0 295 - End ** << BeReq >> 3 -- Begin bereq 2 fetch -- VCL_use boot -- Timestamp Start: 1576135743.442807 0.000000 0.000000 -- BereqMethod HEAD -- BereqURL /Naserfeiz.mp4 -- BereqProtocol HTTP/1.1 -- BereqHeader Host: 192.168.200.12:8080 -- BereqHeader User-Agent: curl/7.47.0 -- BereqHeader Accept: */* -- BereqHeader newtrack: yes -- BereqHeader X-Forwarded-For: 192.168.200.12 -- BereqMethod GET -- BereqHeader Accept-Encoding: gzip -- BereqHeader X-Varnish: 3 -- VCL_call BACKEND_FETCH -- VCL_return fetch -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6846 -- BackendStart 192.168.200.11 8080 -- Timestamp Bereq: 1576135743.443586 0.000779 0.000779 -- Timestamp Beresp: 1576135743.444017 0.001210 0.000431 -- BerespProtocol HTTP/1.1 -- BerespStatus 200 -- BerespReason OK -- BerespHeader Server: nginx -- BerespHeader Date: Thu, 12 Dec 2019 07:29:32 GMT -- BerespHeader Content-Type: text/plain -- BerespHeader Content-Length: 23521499 -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT -- BerespHeader Connection: keep-alive -- BerespHeader ETag: "3a1794b0-166e8db" -- BerespHeader Accept-Ranges: bytes -- TTL RFC 120 10 0 1576135743 1576135743 1576135772 0 0 cacheable -- VCL_call BACKEND_RESPONSE -- TTL VCL 86400 10 0 1576135743 cacheable -- VCL_return deliver -- Storage file s0 -- Fetch_Body 3 length stream -- BackendReuse 27 boot.swift_proxy_1 -- Timestamp BerespBody: 1576135743.747835 0.305028 0.303818 -- Length 23521499 -- BereqAcct 181 0 181 241 23521499 23521740 -- End * << Request >> 32770 - Begin req 32769 rxreq - Timestamp Start: 1576135756.680792 0.000000 0.000000 - Timestamp Req: 1576135756.680792 0.000000 0.000000 - ReqStart 192.168.200.12 5438 a0 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1576135756.680982 0.000190 0.000190 - RespHeader Date: Thu, 12 Dec 2019 07:29:16 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 32770 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135756.681190 0.000399 0.000208 - ReqAcct 113 0 113 218 240 458 - End * << Request >> 32772 - Begin req 32771 rxreq - Timestamp Start: 1576135769.304679 0.000000 0.000000 - Timestamp Req: 1576135769.304679 0.000000 0.000000 - ReqStart 192.168.200.12 5440 a0 - ReqMethod HEAD - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 32773 fetch - Timestamp Fetch: 1576135769.306630 0.001951 0.001951 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Thu, 12 Dec 2019 07:29:58 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 32772 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish (Varnish/6.0) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1576135769.306657 0.001978 0.000027 - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135769.306748 0.002069 0.000091 - ReqAcct 112 0 112 299 0 299 - End ** << BeReq >> 32773 -- Begin bereq 32772 fetch -- VCL_use boot -- Timestamp Start: 1576135769.304834 0.000000 0.000000 -- BereqMethod HEAD -- BereqURL /Naserfeiz.mp4 -- BereqProtocol HTTP/1.1 -- BereqHeader Host: 192.168.200.12:8080 -- BereqHeader User-Agent: curl/7.47.0 -- BereqHeader Accept: */* -- BereqHeader newtrack: yes -- BereqHeader X-Forwarded-For: 192.168.200.12 -- BereqMethod GET -- BereqHeader Accept-Encoding: gzip -- BereqHeader X-Varnish: 32773 -- VCL_call BACKEND_FETCH -- VCL_return fetch -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6846 -- BackendStart 192.168.200.11 8080 -- Timestamp Bereq: 1576135769.304993 0.000160 0.000160 -- Timestamp Beresp: 1576135769.306170 0.001336 0.001176 -- BerespProtocol HTTP/1.1 -- BerespStatus 200 -- BerespReason OK -- BerespHeader Server: nginx -- BerespHeader Date: Thu, 12 Dec 2019 07:29:58 GMT -- BerespHeader Content-Type: text/plain -- BerespHeader Content-Length: 23521499 -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT -- BerespHeader Connection: keep-alive -- BerespHeader ETag: "3a1794b0-166e8db" -- BerespHeader Accept-Ranges: bytes -- TTL RFC 120 10 0 1576135769 1576135769 1576135798 0 0 cacheable -- VCL_call BACKEND_RESPONSE -- TTL VCL 86400 10 0 1576135769 cacheable -- VCL_return deliver -- Storage file s0 -- Fetch_Body 3 length stream -- BackendReuse 27 boot.swift_proxy_1 -- Timestamp BerespBody: 1576135769.645490 0.340657 0.339321 -- Length 23521499 -- BereqAcct 185 0 185 241 23521499 23521740 -- End ######## STEP TWO: First i restart varnish file and varnishlog then: I add another varnish (with ram backend) in fornt of the varnish file and then i ask the url again in varnish ram (because it is my frontend server and users can access media though this server): varnish-ram$ curl -I http://192.168.200.13/Naserfeiz.mp4 -H "newtrack: yes" HTTP/1.1 200 OK Server: nginx Date: Thu, 12 Dec 2019 07:33:05 GMT Content-Type: text/plain Content-Length: 23521499 Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT ETag: "3a1794b0-166e8db" X-Varnish: 2 Via: 1.1 varnish (Varnish/6.0) X-Varnish: 2 Age: 0 Via: 1.1 varnish-v4 Accept-Ranges: bytes Connection: keep-alive First varnish file cached the url and then Varnish ram fetched the object from varnish file and cached it , then i purged that url in varnish ram and it purged successfuly and it exactly removed from cached: varnish-ram:~$ curl -v -k -X PURGE http://192.168.200.13/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" * Trying 192.168.200.13... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.13 (192.168.200.13) port 80 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: 192.168.200.13 > User-Agent: curl/7.47.0 > Accept: */* > newtrack: yes > < HTTP/1.1 200 Purged < Date: Thu, 12 Dec 2019 07:33:46 GMT < Server: Varnish < X-Varnish: 32770 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 240 < Accept-Ranges: bytes < Connection: keep-alive < { [240 bytes data] 100 240 100 240 0 0 73846 0 --:--:-- --:--:-- --:--:-- 117k * Connection #0 to host 192.168.200.13 left intact Then i purged the url in varnish file it shows "200 purged" but it still exist in varnish file!!!!!!: varnish-file$ curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" * Trying 192.168.200.12... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.12 (192.168.200.12) port 8080 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: 192.168.200.12:8080 > User-Agent: curl/7.47.0 > Accept: */* > newtrack: yes > < HTTP/1.1 200 Purged < Date: Thu, 12 Dec 2019 07:33:39 GMT < Server: Varnish < X-Varnish: 5 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 236 < Accept-Ranges: bytes < Connection: keep-alive < { [236 bytes data] 100 236 100 236 0 0 110k 0 --:--:-- --:--:-- --:--:-- 230k * Connection #0 to host 192.168.200.12 left intact This is varnishlog resault in varnish file: varnish-file:~$ sudo varnishlog -d -g request * << Request >> 2 - Begin req 1 rxreq - Timestamp Start: 1576135956.460768 0.000000 0.000000 - Timestamp Req: 1576135956.460768 0.000000 0.000000 - ReqStart 192.168.200.13 22300 a0 - ReqMethod GET - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.13 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.13 - ReqHeader Accept-Encoding: gzip - ReqHeader X-Varnish: 3 - ReqUnset X-Forwarded-For: 192.168.200.13 - ReqHeader X-Forwarded-For: 192.168.200.13, 192.168.200.13 - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 3 fetch - Timestamp Fetch: 1576135956.463065 0.002297 0.002297 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Thu, 12 Dec 2019 07:33:05 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 2 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish (Varnish/6.0) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1576135956.463114 0.002346 0.000049 - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135956.758920 0.298152 0.295806 - ReqAcct 176 0 176 295 23521499 23521794 - End ** << BeReq >> 3 -- Begin bereq 2 fetch -- VCL_use boot -- Timestamp Start: 1576135956.461099 0.000000 0.000000 -- BereqMethod GET -- BereqURL /Naserfeiz.mp4 -- BereqProtocol HTTP/1.1 -- BereqHeader Host: 192.168.200.13 -- BereqHeader User-Agent: curl/7.47.0 -- BereqHeader Accept: */* -- BereqHeader newtrack: yes -- BereqHeader Accept-Encoding: gzip -- BereqHeader X-Varnish: 3 -- BereqHeader X-Forwarded-For: 192.168.200.13, 192.168.200.13 -- BereqHeader X-Varnish: 3 -- VCL_call BACKEND_FETCH -- VCL_return fetch -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6860 -- BackendStart 192.168.200.11 8080 -- Timestamp Bereq: 1576135956.461955 0.000856 0.000856 -- Timestamp Beresp: 1576135956.462607 0.001508 0.000652 -- BerespProtocol HTTP/1.1 -- BerespStatus 200 -- BerespReason OK -- BerespHeader Server: nginx -- BerespHeader Date: Thu, 12 Dec 2019 07:33:05 GMT -- BerespHeader Content-Type: text/plain -- BerespHeader Content-Length: 23521499 -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT -- BerespHeader Connection: keep-alive -- BerespHeader ETag: "3a1794b0-166e8db" -- BerespHeader Accept-Ranges: bytes -- TTL RFC 120 10 0 1576135956 1576135956 1576135985 0 0 cacheable -- VCL_call BACKEND_RESPONSE -- TTL VCL 86400 10 0 1576135956 cacheable -- VCL_return deliver -- Storage file s0 -- Fetch_Body 3 length stream -- BackendReuse 27 boot.swift_proxy_1 -- Timestamp BerespBody: 1576135956.758956 0.297858 0.296349 -- Length 23521499 -- BereqAcct 206 0 206 241 23521499 23521740 -- End * << Request >> 5 - Begin req 4 rxreq - Timestamp Start: 1576136019.230666 0.000000 0.000000 - Timestamp Req: 1576136019.230666 0.000000 0.000000 - ReqStart 192.168.200.12 5452 a0 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1576136019.230863 0.000198 0.000198 - RespHeader Date: Thu, 12 Dec 2019 07:33:39 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 5 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 236 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576136019.231138 0.000473 0.000275 - ReqAcct 113 0 113 214 236 450 - End ######## PLEASE test this scenario for yourself and you can see that i do correct and the object wont remove from varnish-file Please do this scenario for yourself ________________________________ From: Guillaume Quintard > Sent: Wednesday, December 11, 2019 9:44 AM To: hamidreza hosseini > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt purge Yes it does, we have already proved it./ Each instance purges and will refetch the object for its respective backend. I explained how to test: start with the origin, change the data and check that you actually get the new object. Then add a varnish layer, purge, check, repeat. You are testing everything at once and are unable to isolate the issue because you refuse to take things step by step, and spamming won't change that. I'm sorry but until you have tested each layer individually and identified the one that fails, I won't be able to help. -- Guillaume Quintard On Wed, Dec 11, 2019 at 3:08 PM hamidreza hosseini > wrote: Hi, please help me to solve this problem, this is the link that i explaine my question completely: https://varnish-cache.org/lists/pipermail/varnish-misc/2019-December/026754.html _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -- Sent from my iPhone -------------- next part -------------- An HTML attachment was scrubbed... URL: From vlad.rusu at lola.tech Sun Dec 15 20:09:37 2019 From: vlad.rusu at lola.tech (Vlad Rusu) Date: Sun, 15 Dec 2019 22:09:37 +0200 Subject: Varnish doesnt purge In-Reply-To: References: Message-ID: I think the following will explain this in great detail http://varnish-cache.org/docs/6.3/users-guide/vcl-hashing.html https://github.com/varnishcache/varnish-cache/blob/6.3/bin/varnishd/builtin.vcl#L87-L95 By default (your case) any cache entry will be uniquely identified by the following combination: host header + url No matter how many varnish server you stack, unless you specifically alter the Host header in one of them, they will all store the object in their own storage using the same hash key If you call for https://google.com/a and you have stacked 2 varnishes, both of them will save the object as: google.com/a When you want to purge that object, you have to send the same data - which is why I gave you an example that altered the the Host header when sending the PURGE request to the 2nd varnish I?m personally a fan of BANNING when using varnish-cache (non commercial). Read more about this below. https://varnish-cache.org/docs/6.3/users-guide/purging.html + https://book.varnish-software.com/4.0/chapters/Cache_Invalidation.html#lurker-friendly-bans Also worth printing this one on your wall if you want to truly understand the journey of a request through varnish http://varnish-cache.org/docs/6.3/reference/states.html#reference-states Thanks, ? Vlad Rusu Cell: +40758066019 Lola Tech | lola.tech > On 15 Dec 2019, at 21:54, hamidreza hosseini wrote: > > Many Thanks for your answer > You mean that in production i should add additional header to my purge request because it stored this with the header content like the ip address of first varnish > OK, for purging i need wich header to add to my purge request and how can i get that what are that? And how they stored and how can i purge them > > Best Regards > > Get Outlook for Android > From: Vlad Rusu > Sent: Sunday, December 15, 2019 11:16:59 PM > To: hamidreza hosseini > Cc: Guillaume Quintard ; varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt purge > > Try this when purging > > varnish ram: curl -v -k -X PURGE http://192.168.200.13/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" > > varnish file (2nd one): curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes? -H "Host: 192.168.200.13 ? > > Notice the difference? You to understand HOW content is stored in varnish?s storage in order to understand how to evict it - my comment about the hash routine > > Thanks > >> On 15 Dec 2019, at 21:31, hamidreza hosseini > wrote: >> >> I dont understand your message, >> >> I cant solve this problem >> >> Instead of saying "appericiate to your diplomacy!!!!!!!!!!" you can help me! >> >> I dont have enough knoledge on Varnish, >> >> I dont get your text about hash, i didnt use hash. >> >> Best Regards >> >> >> From: Vlad Rusu > >> Sent: Sunday, December 15, 2019 7:48:42 PM >> To: hamidreza hosseini > >> Cc: Guillaume Quintard >; varnish-misc at varnish-cache.org > >> Subject: Re: Varnish doesnt purge >> >> I appreciate Guillaume?s diplomacy :) >> >> Your issue (one of them at least) is that in your cascaded varnish setup the hash key of the cached url, in both varnish storages, is the one you use to originally access the asset. Your vcl hash routine might also be a custom one.. that would be the next thing to look at if what I?m saying here doesn?t fix it. >> >> So, purge >> http://192.168.200.13/Naserfeiz.mp4 >> and NOT >> http://192.168.200.12:8080/Naserfeiz.mp4 when in the cascaded setup >> >> Cheers >> >> On Sat, 14 Dec 2019 at 09:16, hamidreza hosseini > wrote: >> Hi, >> Please annwer my question, I sent you my test >> >> From: varnish-misc > on behalf of hamidreza hosseini > >> Sent: Wednesday, December 11, 2019 11:49 PM >> To: Guillaume Quintard > >> Cc: varnish-misc at varnish-cache.org > >> Subject: Re: Varnish doesnt purge >> >> I do all steps now again, this is the resault: >> STEP ONE: >> First of all i configure a varnish (with file backend) in front of swift/Nginx and i ask the url that exist in swift with curl, >> >> varnish-file$curl -I http://192.168.200.12:8080/Naserfeiz.mp4 -H "newtrack: yes" >> >> HTTP/1.1 200 OK >> Server: nginx >> Date: Thu, 12 Dec 2019 07:29:32 GMT >> Content-Type: text/plain >> Content-Length: 23521499 >> Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT >> ETag: "3a1794b0-166e8db" >> X-Varnish: 2 >> Age: 0 >> Via: 1.1 varnish (Varnish/6.0) >> Accept-Ranges: bytes >> Connection: keep-alive >> >> >> varnish cach that object and then i purge the url: >> >> varnish-file$ curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" >> >> * Trying 192.168.200.12... >> % Total % Received % Xferd Average Speed Time Time Time Current >> Dload Upload Total Spent Left Speed >> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.12 (192.168.200.12) port 8080 (#0) >> > PURGE /Naserfeiz.mp4 HTTP/1.1 >> > Host: 192.168.200.12:8080 >> > User-Agent: curl/7.47.0 >> > Accept: */* >> > newtrack: yes >> > >> < HTTP/1.1 200 Purged >> < Date: Thu, 12 Dec 2019 07:29:16 GMT >> < Server: Varnish >> < X-Varnish: 32770 >> < Content-Type: text/html; charset=utf-8 >> < Retry-After: 5 >> < Content-Length: 240 >> < Accept-Ranges: bytes >> < Connection: keep-alive >> < >> { [240 bytes data] >> 100 240 100 240 0 0 124k 0 --:--:-- --:--:-- --:--:-- 234k >> * Connection #0 to host 192.168.200.12 left intact >> >> The object purged successfully and it didnt exist in cach anymore. >> >> This is varnishlog resault: >> >> varnish-file$ sudo varnishlog -d -g request >> * << Request >> 2 >> - Begin req 1 rxreq >> - Timestamp Start: 1576135743.442646 0.000000 0.000000 >> - Timestamp Req: 1576135743.442646 0.000000 0.000000 >> - ReqStart 192.168.200.12 5432 a0 >> - ReqMethod HEAD >> - ReqURL /Naserfeiz.mp4 >> - ReqProtocol HTTP/1.1 >> - ReqHeader Host: 192.168.200.12:8080 >> - ReqHeader User-Agent: curl/7.47.0 >> - ReqHeader Accept: */* >> - ReqHeader newtrack: yes >> - ReqHeader X-Forwarded-For: 192.168.200.12 >> - VCL_call RECV >> - VCL_return hash >> - VCL_call HASH >> - VCL_return lookup >> - VCL_call MISS >> - VCL_return fetch >> - Link bereq 3 fetch >> - Timestamp Fetch: 1576135743.444347 0.001701 0.001701 >> - RespProtocol HTTP/1.1 >> - RespStatus 200 >> - RespReason OK >> - RespHeader Server: nginx >> - RespHeader Date: Thu, 12 Dec 2019 07:29:32 GMT >> - RespHeader Content-Type: text/plain >> - RespHeader Content-Length: 23521499 >> - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT >> - RespHeader ETag: "3a1794b0-166e8db" >> - RespHeader X-Varnish: 2 >> - RespHeader Age: 0 >> - RespHeader Via: 1.1 varnish (Varnish/6.0) >> - VCL_call DELIVER >> - VCL_return deliver >> - Timestamp Process: 1576135743.444382 0.001736 0.000036 >> - RespHeader Accept-Ranges: bytes >> - RespHeader Connection: keep-alive >> - Timestamp Resp: 1576135743.444474 0.001828 0.000092 >> - ReqAcct 112 0 112 295 0 295 >> - End >> ** << BeReq >> 3 >> -- Begin bereq 2 fetch >> -- VCL_use boot >> -- Timestamp Start: 1576135743.442807 0.000000 0.000000 >> -- BereqMethod HEAD >> -- BereqURL /Naserfeiz.mp4 >> -- BereqProtocol HTTP/1.1 >> -- BereqHeader Host: 192.168.200.12:8080 >> -- BereqHeader User-Agent: curl/7.47.0 >> -- BereqHeader Accept: */* >> -- BereqHeader newtrack: yes >> -- BereqHeader X-Forwarded-For: 192.168.200.12 >> -- BereqMethod GET >> -- BereqHeader Accept-Encoding: gzip >> -- BereqHeader X-Varnish: 3 >> -- VCL_call BACKEND_FETCH >> -- VCL_return fetch >> -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6846 >> -- BackendStart 192.168.200.11 8080 >> -- Timestamp Bereq: 1576135743.443586 0.000779 0.000779 >> -- Timestamp Beresp: 1576135743.444017 0.001210 0.000431 >> -- BerespProtocol HTTP/1.1 >> -- BerespStatus 200 >> -- BerespReason OK >> -- BerespHeader Server: nginx >> -- BerespHeader Date: Thu, 12 Dec 2019 07:29:32 GMT >> -- BerespHeader Content-Type: text/plain >> -- BerespHeader Content-Length: 23521499 >> -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT >> -- BerespHeader Connection: keep-alive >> -- BerespHeader ETag: "3a1794b0-166e8db" >> -- BerespHeader Accept-Ranges: bytes >> -- TTL RFC 120 10 0 1576135743 1576135743 1576135772 0 0 cacheable >> -- VCL_call BACKEND_RESPONSE >> -- TTL VCL 86400 10 0 1576135743 cacheable >> -- VCL_return deliver >> -- Storage file s0 >> -- Fetch_Body 3 length stream >> -- BackendReuse 27 boot.swift_proxy_1 >> -- Timestamp BerespBody: 1576135743.747835 0.305028 0.303818 >> -- Length 23521499 >> -- BereqAcct 181 0 181 241 23521499 23521740 >> -- End >> >> * << Request >> 32770 >> - Begin req 32769 rxreq >> - Timestamp Start: 1576135756.680792 0.000000 0.000000 >> - Timestamp Req: 1576135756.680792 0.000000 0.000000 >> - ReqStart 192.168.200.12 5438 a0 >> - ReqMethod PURGE >> - ReqURL /Naserfeiz.mp4 >> - ReqProtocol HTTP/1.1 >> - ReqHeader Host: 192.168.200.12:8080 >> - ReqHeader User-Agent: curl/7.47.0 >> - ReqHeader Accept: */* >> - ReqHeader newtrack: yes >> - ReqHeader X-Forwarded-For: 192.168.200.12 >> - VCL_call RECV >> - VCL_return purge >> - VCL_call HASH >> - VCL_return lookup >> - VCL_call PURGE >> - VCL_return synth >> - Timestamp Process: 1576135756.680982 0.000190 0.000190 >> - RespHeader Date: Thu, 12 Dec 2019 07:29:16 GMT >> - RespHeader Server: Varnish >> - RespHeader X-Varnish: 32770 >> - RespProtocol HTTP/1.1 >> - RespStatus 200 >> - RespReason OK >> - RespReason Purged >> - VCL_call SYNTH >> - RespHeader Content-Type: text/html; charset=utf-8 >> - RespHeader Retry-After: 5 >> - VCL_return deliver >> - RespHeader Content-Length: 240 >> - Storage malloc Transient >> - RespHeader Accept-Ranges: bytes >> - RespHeader Connection: keep-alive >> - Timestamp Resp: 1576135756.681190 0.000399 0.000208 >> - ReqAcct 113 0 113 218 240 458 >> - End >> >> * << Request >> 32772 >> - Begin req 32771 rxreq >> - Timestamp Start: 1576135769.304679 0.000000 0.000000 >> - Timestamp Req: 1576135769.304679 0.000000 0.000000 >> - ReqStart 192.168.200.12 5440 a0 >> - ReqMethod HEAD >> - ReqURL /Naserfeiz.mp4 >> - ReqProtocol HTTP/1.1 >> - ReqHeader Host: 192.168.200.12:8080 >> - ReqHeader User-Agent: curl/7.47.0 >> - ReqHeader Accept: */* >> - ReqHeader newtrack: yes >> - ReqHeader X-Forwarded-For: 192.168.200.12 >> - VCL_call RECV >> - VCL_return hash >> - VCL_call HASH >> - VCL_return lookup >> - VCL_call MISS >> - VCL_return fetch >> - Link bereq 32773 fetch >> - Timestamp Fetch: 1576135769.306630 0.001951 0.001951 >> - RespProtocol HTTP/1.1 >> - RespStatus 200 >> - RespReason OK >> - RespHeader Server: nginx >> - RespHeader Date: Thu, 12 Dec 2019 07:29:58 GMT >> - RespHeader Content-Type: text/plain >> - RespHeader Content-Length: 23521499 >> - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT >> - RespHeader ETag: "3a1794b0-166e8db" >> - RespHeader X-Varnish: 32772 >> - RespHeader Age: 0 >> - RespHeader Via: 1.1 varnish (Varnish/6.0) >> - VCL_call DELIVER >> - VCL_return deliver >> - Timestamp Process: 1576135769.306657 0.001978 0.000027 >> - RespHeader Accept-Ranges: bytes >> - RespHeader Connection: keep-alive >> - Timestamp Resp: 1576135769.306748 0.002069 0.000091 >> - ReqAcct 112 0 112 299 0 299 >> - End >> ** << BeReq >> 32773 >> -- Begin bereq 32772 fetch >> -- VCL_use boot >> -- Timestamp Start: 1576135769.304834 0.000000 0.000000 >> -- BereqMethod HEAD >> -- BereqURL /Naserfeiz.mp4 >> -- BereqProtocol HTTP/1.1 >> -- BereqHeader Host: 192.168.200.12:8080 >> -- BereqHeader User-Agent: curl/7.47.0 >> -- BereqHeader Accept: */* >> -- BereqHeader newtrack: yes >> -- BereqHeader X-Forwarded-For: 192.168.200.12 >> -- BereqMethod GET >> -- BereqHeader Accept-Encoding: gzip >> -- BereqHeader X-Varnish: 32773 >> -- VCL_call BACKEND_FETCH >> -- VCL_return fetch >> -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6846 >> -- BackendStart 192.168.200.11 8080 >> -- Timestamp Bereq: 1576135769.304993 0.000160 0.000160 >> -- Timestamp Beresp: 1576135769.306170 0.001336 0.001176 >> -- BerespProtocol HTTP/1.1 >> -- BerespStatus 200 >> -- BerespReason OK >> -- BerespHeader Server: nginx >> -- BerespHeader Date: Thu, 12 Dec 2019 07:29:58 GMT >> -- BerespHeader Content-Type: text/plain >> -- BerespHeader Content-Length: 23521499 >> -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT >> -- BerespHeader Connection: keep-alive >> -- BerespHeader ETag: "3a1794b0-166e8db" >> -- BerespHeader Accept-Ranges: bytes >> -- TTL RFC 120 10 0 1576135769 1576135769 1576135798 0 0 cacheable >> -- VCL_call BACKEND_RESPONSE >> -- TTL VCL 86400 10 0 1576135769 cacheable >> -- VCL_return deliver >> -- Storage file s0 >> -- Fetch_Body 3 length stream >> -- BackendReuse 27 boot.swift_proxy_1 >> -- Timestamp BerespBody: 1576135769.645490 0.340657 0.339321 >> -- Length 23521499 >> -- BereqAcct 185 0 185 241 23521499 23521740 >> -- End >> >> >> ######## >> >> STEP TWO: >> First i restart varnish file and varnishlog then: >> I add another varnish (with ram backend) in fornt of the varnish file and then i ask the url again in varnish ram (because it is my frontend server and users can access media though this server): >> >> varnish-ram$ curl -I http://192.168.200.13/Naserfeiz.mp4 -H "newtrack: yes" >> HTTP/1.1 200 OK >> Server: nginx >> Date: Thu, 12 Dec 2019 07:33:05 GMT >> Content-Type: text/plain >> Content-Length: 23521499 >> Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT >> ETag: "3a1794b0-166e8db" >> X-Varnish: 2 >> Via: 1.1 varnish (Varnish/6.0) >> X-Varnish: 2 >> Age: 0 >> Via: 1.1 varnish-v4 >> Accept-Ranges: bytes >> Connection: keep-alive >> >> >> First varnish file cached the url and then Varnish ram fetched the object from varnish file and cached it , then i purged that url in varnish ram and it purged successfuly and it exactly removed from cached: >> >> varnish-ram:~$ curl -v -k -X PURGE http://192.168.200.13/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" >> * Trying 192.168.200.13... >> % Total % Received % Xferd Average Speed Time Time Time Current >> Dload Upload Total Spent Left Speed >> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.13 (192.168.200.13) port 80 (#0) >> > PURGE /Naserfeiz.mp4 HTTP/1.1 >> > Host: 192.168.200.13 >> > User-Agent: curl/7.47.0 >> > Accept: */* >> > newtrack: yes >> > >> < HTTP/1.1 200 Purged >> < Date: Thu, 12 Dec 2019 07:33:46 GMT >> < Server: Varnish >> < X-Varnish: 32770 >> < Content-Type: text/html; charset=utf-8 >> < Retry-After: 5 >> < Content-Length: 240 >> < Accept-Ranges: bytes >> < Connection: keep-alive >> < >> { [240 bytes data] >> 100 240 100 240 0 0 73846 0 --:--:-- --:--:-- --:--:-- 117k >> * Connection #0 to host 192.168.200.13 left intact >> >> >> Then i purged the url in varnish file it shows "200 purged" but it still exist in varnish file!!!!!!: >> >> varnish-file$ curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" >> * Trying 192.168.200.12... >> % Total % Received % Xferd Average Speed Time Time Time Current >> Dload Upload Total Spent Left Speed >> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.12 (192.168.200.12) port 8080 (#0) >> > PURGE /Naserfeiz.mp4 HTTP/1.1 >> > Host: 192.168.200.12:8080 >> > User-Agent: curl/7.47.0 >> > Accept: */* >> > newtrack: yes >> > >> < HTTP/1.1 200 Purged >> < Date: Thu, 12 Dec 2019 07:33:39 GMT >> < Server: Varnish >> < X-Varnish: 5 >> < Content-Type: text/html; charset=utf-8 >> < Retry-After: 5 >> < Content-Length: 236 >> < Accept-Ranges: bytes >> < Connection: keep-alive >> < >> { [236 bytes data] >> 100 236 100 236 0 0 110k 0 --:--:-- --:--:-- --:--:-- 230k >> * Connection #0 to host 192.168.200.12 left intact >> >> >> This is varnishlog resault in varnish file: >> >> varnish-file:~$ sudo varnishlog -d -g request >> * << Request >> 2 >> - Begin req 1 rxreq >> - Timestamp Start: 1576135956.460768 0.000000 0.000000 >> - Timestamp Req: 1576135956.460768 0.000000 0.000000 >> - ReqStart 192.168.200.13 22300 a0 >> - ReqMethod GET >> - ReqURL /Naserfeiz.mp4 >> - ReqProtocol HTTP/1.1 >> - ReqHeader Host: 192.168.200.13 >> - ReqHeader User-Agent: curl/7.47.0 >> - ReqHeader Accept: */* >> - ReqHeader newtrack: yes >> - ReqHeader X-Forwarded-For: 192.168.200.13 >> - ReqHeader Accept-Encoding: gzip >> - ReqHeader X-Varnish: 3 >> - ReqUnset X-Forwarded-For: 192.168.200.13 >> - ReqHeader X-Forwarded-For: 192.168.200.13, 192.168.200.13 >> - VCL_call RECV >> - VCL_return hash >> - VCL_call HASH >> - VCL_return lookup >> - VCL_call MISS >> - VCL_return fetch >> - Link bereq 3 fetch >> - Timestamp Fetch: 1576135956.463065 0.002297 0.002297 >> - RespProtocol HTTP/1.1 >> - RespStatus 200 >> - RespReason OK >> - RespHeader Server: nginx >> - RespHeader Date: Thu, 12 Dec 2019 07:33:05 GMT >> - RespHeader Content-Type: text/plain >> - RespHeader Content-Length: 23521499 >> - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT >> - RespHeader ETag: "3a1794b0-166e8db" >> - RespHeader X-Varnish: 2 >> - RespHeader Age: 0 >> - RespHeader Via: 1.1 varnish (Varnish/6.0) >> - VCL_call DELIVER >> - VCL_return deliver >> - Timestamp Process: 1576135956.463114 0.002346 0.000049 >> - RespHeader Accept-Ranges: bytes >> - RespHeader Connection: keep-alive >> - Timestamp Resp: 1576135956.758920 0.298152 0.295806 >> - ReqAcct 176 0 176 295 23521499 23521794 >> - End >> ** << BeReq >> 3 >> -- Begin bereq 2 fetch >> -- VCL_use boot >> -- Timestamp Start: 1576135956.461099 0.000000 0.000000 >> -- BereqMethod GET >> -- BereqURL /Naserfeiz.mp4 >> -- BereqProtocol HTTP/1.1 >> -- BereqHeader Host: 192.168.200.13 >> -- BereqHeader User-Agent: curl/7.47.0 >> -- BereqHeader Accept: */* >> -- BereqHeader newtrack: yes >> -- BereqHeader Accept-Encoding: gzip >> -- BereqHeader X-Varnish: 3 >> -- BereqHeader X-Forwarded-For: 192.168.200.13, 192.168.200.13 >> -- BereqHeader X-Varnish: 3 >> -- VCL_call BACKEND_FETCH >> -- VCL_return fetch >> -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6860 >> -- BackendStart 192.168.200.11 8080 >> -- Timestamp Bereq: 1576135956.461955 0.000856 0.000856 >> -- Timestamp Beresp: 1576135956.462607 0.001508 0.000652 >> -- BerespProtocol HTTP/1.1 >> -- BerespStatus 200 >> -- BerespReason OK >> -- BerespHeader Server: nginx >> -- BerespHeader Date: Thu, 12 Dec 2019 07:33:05 GMT >> -- BerespHeader Content-Type: text/plain >> -- BerespHeader Content-Length: 23521499 >> -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT >> -- BerespHeader Connection: keep-alive >> -- BerespHeader ETag: "3a1794b0-166e8db" >> -- BerespHeader Accept-Ranges: bytes >> -- TTL RFC 120 10 0 1576135956 1576135956 1576135985 0 0 cacheable >> -- VCL_call BACKEND_RESPONSE >> -- TTL VCL 86400 10 0 1576135956 cacheable >> -- VCL_return deliver >> -- Storage file s0 >> -- Fetch_Body 3 length stream >> -- BackendReuse 27 boot.swift_proxy_1 >> -- Timestamp BerespBody: 1576135956.758956 0.297858 0.296349 >> -- Length 23521499 >> -- BereqAcct 206 0 206 241 23521499 23521740 >> -- End >> >> * << Request >> 5 >> - Begin req 4 rxreq >> - Timestamp Start: 1576136019.230666 0.000000 0.000000 >> - Timestamp Req: 1576136019.230666 0.000000 0.000000 >> - ReqStart 192.168.200.12 5452 a0 >> - ReqMethod PURGE >> - ReqURL /Naserfeiz.mp4 >> - ReqProtocol HTTP/1.1 >> - ReqHeader Host: 192.168.200.12:8080 >> - ReqHeader User-Agent: curl/7.47.0 >> - ReqHeader Accept: */* >> - ReqHeader newtrack: yes >> - ReqHeader X-Forwarded-For: 192.168.200.12 >> - VCL_call RECV >> - VCL_return purge >> - VCL_call HASH >> - VCL_return lookup >> - VCL_call PURGE >> - VCL_return synth >> - Timestamp Process: 1576136019.230863 0.000198 0.000198 >> - RespHeader Date: Thu, 12 Dec 2019 07:33:39 GMT >> - RespHeader Server: Varnish >> - RespHeader X-Varnish: 5 >> - RespProtocol HTTP/1.1 >> - RespStatus 200 >> - RespReason OK >> - RespReason Purged >> - VCL_call SYNTH >> - RespHeader Content-Type: text/html; charset=utf-8 >> - RespHeader Retry-After: 5 >> - VCL_return deliver >> - RespHeader Content-Length: 236 >> - Storage malloc Transient >> - RespHeader Accept-Ranges: bytes >> - RespHeader Connection: keep-alive >> - Timestamp Resp: 1576136019.231138 0.000473 0.000275 >> - ReqAcct 113 0 113 214 236 450 >> - End >> >> ######## >> PLEASE test this scenario for yourself and you can see that i do correct and the object wont remove from varnish-file >> Please do this scenario for yourself >> >> From: Guillaume Quintard > >> Sent: Wednesday, December 11, 2019 9:44 AM >> To: hamidreza hosseini > >> Cc: varnish-misc at varnish-cache.org > >> Subject: Re: Varnish doesnt purge >> >> Yes it does, we have already proved it./ >> >> Each instance purges and will refetch the object for its respective backend. I explained how to test: start with the origin, change the data and check that you actually get the new object. Then add a varnish layer, purge, check, repeat. >> >> You are testing everything at once and are unable to isolate the issue because you refuse to take things step by step, and spamming won't change that. >> >> I'm sorry but until you have tested each layer individually and identified the one that fails, I won't be able to help. >> >> -- >> Guillaume Quintard >> >> >> On Wed, Dec 11, 2019 at 3:08 PM hamidreza hosseini > wrote: >> Hi, >> please help me to solve this problem, this is the link that i explaine my question completely: >> https://varnish-cache.org/lists/pipermail/varnish-misc/2019-December/026754.html >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> -- >> Sent from my iPhone > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrhosseini at hotmail.com Mon Dec 16 07:40:32 2019 From: hrhosseini at hotmail.com (hamidreza hosseini) Date: Mon, 16 Dec 2019 07:40:32 +0000 Subject: Varnish doesnt purge In-Reply-To: References: , Message-ID: Many Thanks to you (Vlad Rusu), my problem solved. Regards ________________________________ From: Vlad Rusu Sent: Sunday, December 15, 2019 12:09 PM To: hamidreza hosseini Cc: Guillaume Quintard ; varnish-misc at varnish-cache.org Subject: Re: Varnish doesnt purge I think the following will explain this in great detail http://varnish-cache.org/docs/6.3/users-guide/vcl-hashing.html https://github.com/varnishcache/varnish-cache/blob/6.3/bin/varnishd/builtin.vcl#L87-L95 By default (your case) any cache entry will be uniquely identified by the following combination: host header + url No matter how many varnish server you stack, unless you specifically alter the Host header in one of them, they will all store the object in their own storage using the same hash key If you call for https://google.com/a and you have stacked 2 varnishes, both of them will save the object as: google.com/a When you want to purge that object, you have to send the same data - which is why I gave you an example that altered the the Host header when sending the PURGE request to the 2nd varnish I?m personally a fan of BANNING when using varnish-cache (non commercial). Read more about this below. https://varnish-cache.org/docs/6.3/users-guide/purging.html + https://book.varnish-software.com/4.0/chapters/Cache_Invalidation.html#lurker-friendly-bans Also worth printing this one on your wall if you want to truly understand the journey of a request through varnish http://varnish-cache.org/docs/6.3/reference/states.html#reference-states Thanks, ? Vlad Rusu Cell: +40758066019 Lola Tech | lola.tech On 15 Dec 2019, at 21:54, hamidreza hosseini > wrote: Many Thanks for your answer You mean that in production i should add additional header to my purge request because it stored this with the header content like the ip address of first varnish OK, for purging i need wich header to add to my purge request and how can i get that what are that? And how they stored and how can i purge them Best Regards Get Outlook for Android ________________________________ From: Vlad Rusu > Sent: Sunday, December 15, 2019 11:16:59 PM To: hamidreza hosseini > Cc: Guillaume Quintard >; varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt purge Try this when purging varnish ram: curl -v -k -X PURGE http://192.168.200.13/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" varnish file (2nd one): curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes? -H "Host: 192.168.200.13? Notice the difference? You to understand HOW content is stored in varnish?s storage in order to understand how to evict it - my comment about the hash routine Thanks On 15 Dec 2019, at 21:31, hamidreza hosseini > wrote: I dont understand your message, I cant solve this problem Instead of saying "appericiate to your diplomacy!!!!!!!!!!" you can help me! I dont have enough knoledge on Varnish, I dont get your text about hash, i didnt use hash. Best Regards ________________________________ From: Vlad Rusu > Sent: Sunday, December 15, 2019 7:48:42 PM To: hamidreza hosseini > Cc: Guillaume Quintard >; varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt purge I appreciate Guillaume?s diplomacy :) Your issue (one of them at least) is that in your cascaded varnish setup the hash key of the cached url, in both varnish storages, is the one you use to originally access the asset. Your vcl hash routine might also be a custom one.. that would be the next thing to look at if what I?m saying here doesn?t fix it. So, purge http://192.168.200.13/Naserfeiz.mp4 and NOT http://192.168.200.12:8080/Naserfeiz.mp4 when in the cascaded setup Cheers On Sat, 14 Dec 2019 at 09:16, hamidreza hosseini > wrote: Hi, Please annwer my question, I sent you my test ________________________________ From: varnish-misc > on behalf of hamidreza hosseini > Sent: Wednesday, December 11, 2019 11:49 PM To: Guillaume Quintard > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt purge I do all steps now again, this is the resault: STEP ONE: First of all i configure a varnish (with file backend) in front of swift/Nginx and i ask the url that exist in swift with curl, varnish-file$curl -I http://192.168.200.12:8080/Naserfeiz.mp4 -H "newtrack: yes" HTTP/1.1 200 OK Server: nginx Date: Thu, 12 Dec 2019 07:29:32 GMT Content-Type: text/plain Content-Length: 23521499 Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT ETag: "3a1794b0-166e8db" X-Varnish: 2 Age: 0 Via: 1.1 varnish (Varnish/6.0) Accept-Ranges: bytes Connection: keep-alive varnish cach that object and then i purge the url: varnish-file$ curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" * Trying 192.168.200.12... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.12 (192.168.200.12) port 8080 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: 192.168.200.12:8080 > User-Agent: curl/7.47.0 > Accept: */* > newtrack: yes > < HTTP/1.1 200 Purged < Date: Thu, 12 Dec 2019 07:29:16 GMT < Server: Varnish < X-Varnish: 32770 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 240 < Accept-Ranges: bytes < Connection: keep-alive < { [240 bytes data] 100 240 100 240 0 0 124k 0 --:--:-- --:--:-- --:--:-- 234k * Connection #0 to host 192.168.200.12 left intact The object purged successfully and it didnt exist in cach anymore. This is varnishlog resault: varnish-file$ sudo varnishlog -d -g request * << Request >> 2 - Begin req 1 rxreq - Timestamp Start: 1576135743.442646 0.000000 0.000000 - Timestamp Req: 1576135743.442646 0.000000 0.000000 - ReqStart 192.168.200.12 5432 a0 - ReqMethod HEAD - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 3 fetch - Timestamp Fetch: 1576135743.444347 0.001701 0.001701 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Thu, 12 Dec 2019 07:29:32 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 2 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish (Varnish/6.0) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1576135743.444382 0.001736 0.000036 - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135743.444474 0.001828 0.000092 - ReqAcct 112 0 112 295 0 295 - End ** << BeReq >> 3 -- Begin bereq 2 fetch -- VCL_use boot -- Timestamp Start: 1576135743.442807 0.000000 0.000000 -- BereqMethod HEAD -- BereqURL /Naserfeiz.mp4 -- BereqProtocol HTTP/1.1 -- BereqHeader Host: 192.168.200.12:8080 -- BereqHeader User-Agent: curl/7.47.0 -- BereqHeader Accept: */* -- BereqHeader newtrack: yes -- BereqHeader X-Forwarded-For: 192.168.200.12 -- BereqMethod GET -- BereqHeader Accept-Encoding: gzip -- BereqHeader X-Varnish: 3 -- VCL_call BACKEND_FETCH -- VCL_return fetch -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6846 -- BackendStart 192.168.200.11 8080 -- Timestamp Bereq: 1576135743.443586 0.000779 0.000779 -- Timestamp Beresp: 1576135743.444017 0.001210 0.000431 -- BerespProtocol HTTP/1.1 -- BerespStatus 200 -- BerespReason OK -- BerespHeader Server: nginx -- BerespHeader Date: Thu, 12 Dec 2019 07:29:32 GMT -- BerespHeader Content-Type: text/plain -- BerespHeader Content-Length: 23521499 -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT -- BerespHeader Connection: keep-alive -- BerespHeader ETag: "3a1794b0-166e8db" -- BerespHeader Accept-Ranges: bytes -- TTL RFC 120 10 0 1576135743 1576135743 1576135772 0 0 cacheable -- VCL_call BACKEND_RESPONSE -- TTL VCL 86400 10 0 1576135743 cacheable -- VCL_return deliver -- Storage file s0 -- Fetch_Body 3 length stream -- BackendReuse 27 boot.swift_proxy_1 -- Timestamp BerespBody: 1576135743.747835 0.305028 0.303818 -- Length 23521499 -- BereqAcct 181 0 181 241 23521499 23521740 -- End * << Request >> 32770 - Begin req 32769 rxreq - Timestamp Start: 1576135756.680792 0.000000 0.000000 - Timestamp Req: 1576135756.680792 0.000000 0.000000 - ReqStart 192.168.200.12 5438 a0 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1576135756.680982 0.000190 0.000190 - RespHeader Date: Thu, 12 Dec 2019 07:29:16 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 32770 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 240 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135756.681190 0.000399 0.000208 - ReqAcct 113 0 113 218 240 458 - End * << Request >> 32772 - Begin req 32771 rxreq - Timestamp Start: 1576135769.304679 0.000000 0.000000 - Timestamp Req: 1576135769.304679 0.000000 0.000000 - ReqStart 192.168.200.12 5440 a0 - ReqMethod HEAD - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 32773 fetch - Timestamp Fetch: 1576135769.306630 0.001951 0.001951 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Thu, 12 Dec 2019 07:29:58 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 32772 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish (Varnish/6.0) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1576135769.306657 0.001978 0.000027 - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135769.306748 0.002069 0.000091 - ReqAcct 112 0 112 299 0 299 - End ** << BeReq >> 32773 -- Begin bereq 32772 fetch -- VCL_use boot -- Timestamp Start: 1576135769.304834 0.000000 0.000000 -- BereqMethod HEAD -- BereqURL /Naserfeiz.mp4 -- BereqProtocol HTTP/1.1 -- BereqHeader Host: 192.168.200.12:8080 -- BereqHeader User-Agent: curl/7.47.0 -- BereqHeader Accept: */* -- BereqHeader newtrack: yes -- BereqHeader X-Forwarded-For: 192.168.200.12 -- BereqMethod GET -- BereqHeader Accept-Encoding: gzip -- BereqHeader X-Varnish: 32773 -- VCL_call BACKEND_FETCH -- VCL_return fetch -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6846 -- BackendStart 192.168.200.11 8080 -- Timestamp Bereq: 1576135769.304993 0.000160 0.000160 -- Timestamp Beresp: 1576135769.306170 0.001336 0.001176 -- BerespProtocol HTTP/1.1 -- BerespStatus 200 -- BerespReason OK -- BerespHeader Server: nginx -- BerespHeader Date: Thu, 12 Dec 2019 07:29:58 GMT -- BerespHeader Content-Type: text/plain -- BerespHeader Content-Length: 23521499 -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT -- BerespHeader Connection: keep-alive -- BerespHeader ETag: "3a1794b0-166e8db" -- BerespHeader Accept-Ranges: bytes -- TTL RFC 120 10 0 1576135769 1576135769 1576135798 0 0 cacheable -- VCL_call BACKEND_RESPONSE -- TTL VCL 86400 10 0 1576135769 cacheable -- VCL_return deliver -- Storage file s0 -- Fetch_Body 3 length stream -- BackendReuse 27 boot.swift_proxy_1 -- Timestamp BerespBody: 1576135769.645490 0.340657 0.339321 -- Length 23521499 -- BereqAcct 185 0 185 241 23521499 23521740 -- End ######## STEP TWO: First i restart varnish file and varnishlog then: I add another varnish (with ram backend) in fornt of the varnish file and then i ask the url again in varnish ram (because it is my frontend server and users can access media though this server): varnish-ram$ curl -I http://192.168.200.13/Naserfeiz.mp4 -H "newtrack: yes" HTTP/1.1 200 OK Server: nginx Date: Thu, 12 Dec 2019 07:33:05 GMT Content-Type: text/plain Content-Length: 23521499 Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT ETag: "3a1794b0-166e8db" X-Varnish: 2 Via: 1.1 varnish (Varnish/6.0) X-Varnish: 2 Age: 0 Via: 1.1 varnish-v4 Accept-Ranges: bytes Connection: keep-alive First varnish file cached the url and then Varnish ram fetched the object from varnish file and cached it , then i purged that url in varnish ram and it purged successfuly and it exactly removed from cached: varnish-ram:~$ curl -v -k -X PURGE http://192.168.200.13/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" * Trying 192.168.200.13... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.13 (192.168.200.13) port 80 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: 192.168.200.13 > User-Agent: curl/7.47.0 > Accept: */* > newtrack: yes > < HTTP/1.1 200 Purged < Date: Thu, 12 Dec 2019 07:33:46 GMT < Server: Varnish < X-Varnish: 32770 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 240 < Accept-Ranges: bytes < Connection: keep-alive < { [240 bytes data] 100 240 100 240 0 0 73846 0 --:--:-- --:--:-- --:--:-- 117k * Connection #0 to host 192.168.200.13 left intact Then i purged the url in varnish file it shows "200 purged" but it still exist in varnish file!!!!!!: varnish-file$ curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4 -o /dev/null -H "newtrack: yes" * Trying 192.168.200.12... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to 192.168.200.12 (192.168.200.12) port 8080 (#0) > PURGE /Naserfeiz.mp4 HTTP/1.1 > Host: 192.168.200.12:8080 > User-Agent: curl/7.47.0 > Accept: */* > newtrack: yes > < HTTP/1.1 200 Purged < Date: Thu, 12 Dec 2019 07:33:39 GMT < Server: Varnish < X-Varnish: 5 < Content-Type: text/html; charset=utf-8 < Retry-After: 5 < Content-Length: 236 < Accept-Ranges: bytes < Connection: keep-alive < { [236 bytes data] 100 236 100 236 0 0 110k 0 --:--:-- --:--:-- --:--:-- 230k * Connection #0 to host 192.168.200.12 left intact This is varnishlog resault in varnish file: varnish-file:~$ sudo varnishlog -d -g request * << Request >> 2 - Begin req 1 rxreq - Timestamp Start: 1576135956.460768 0.000000 0.000000 - Timestamp Req: 1576135956.460768 0.000000 0.000000 - ReqStart 192.168.200.13 22300 a0 - ReqMethod GET - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.13 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.13 - ReqHeader Accept-Encoding: gzip - ReqHeader X-Varnish: 3 - ReqUnset X-Forwarded-For: 192.168.200.13 - ReqHeader X-Forwarded-For: 192.168.200.13, 192.168.200.13 - VCL_call RECV - VCL_return hash - VCL_call HASH - VCL_return lookup - VCL_call MISS - VCL_return fetch - Link bereq 3 fetch - Timestamp Fetch: 1576135956.463065 0.002297 0.002297 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespHeader Server: nginx - RespHeader Date: Thu, 12 Dec 2019 07:33:05 GMT - RespHeader Content-Type: text/plain - RespHeader Content-Length: 23521499 - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT - RespHeader ETag: "3a1794b0-166e8db" - RespHeader X-Varnish: 2 - RespHeader Age: 0 - RespHeader Via: 1.1 varnish (Varnish/6.0) - VCL_call DELIVER - VCL_return deliver - Timestamp Process: 1576135956.463114 0.002346 0.000049 - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576135956.758920 0.298152 0.295806 - ReqAcct 176 0 176 295 23521499 23521794 - End ** << BeReq >> 3 -- Begin bereq 2 fetch -- VCL_use boot -- Timestamp Start: 1576135956.461099 0.000000 0.000000 -- BereqMethod GET -- BereqURL /Naserfeiz.mp4 -- BereqProtocol HTTP/1.1 -- BereqHeader Host: 192.168.200.13 -- BereqHeader User-Agent: curl/7.47.0 -- BereqHeader Accept: */* -- BereqHeader newtrack: yes -- BereqHeader Accept-Encoding: gzip -- BereqHeader X-Varnish: 3 -- BereqHeader X-Forwarded-For: 192.168.200.13, 192.168.200.13 -- BereqHeader X-Varnish: 3 -- VCL_call BACKEND_FETCH -- VCL_return fetch -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080 192.168.200.12 6860 -- BackendStart 192.168.200.11 8080 -- Timestamp Bereq: 1576135956.461955 0.000856 0.000856 -- Timestamp Beresp: 1576135956.462607 0.001508 0.000652 -- BerespProtocol HTTP/1.1 -- BerespStatus 200 -- BerespReason OK -- BerespHeader Server: nginx -- BerespHeader Date: Thu, 12 Dec 2019 07:33:05 GMT -- BerespHeader Content-Type: text/plain -- BerespHeader Content-Length: 23521499 -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT -- BerespHeader Connection: keep-alive -- BerespHeader ETag: "3a1794b0-166e8db" -- BerespHeader Accept-Ranges: bytes -- TTL RFC 120 10 0 1576135956 1576135956 1576135985 0 0 cacheable -- VCL_call BACKEND_RESPONSE -- TTL VCL 86400 10 0 1576135956 cacheable -- VCL_return deliver -- Storage file s0 -- Fetch_Body 3 length stream -- BackendReuse 27 boot.swift_proxy_1 -- Timestamp BerespBody: 1576135956.758956 0.297858 0.296349 -- Length 23521499 -- BereqAcct 206 0 206 241 23521499 23521740 -- End * << Request >> 5 - Begin req 4 rxreq - Timestamp Start: 1576136019.230666 0.000000 0.000000 - Timestamp Req: 1576136019.230666 0.000000 0.000000 - ReqStart 192.168.200.12 5452 a0 - ReqMethod PURGE - ReqURL /Naserfeiz.mp4 - ReqProtocol HTTP/1.1 - ReqHeader Host: 192.168.200.12:8080 - ReqHeader User-Agent: curl/7.47.0 - ReqHeader Accept: */* - ReqHeader newtrack: yes - ReqHeader X-Forwarded-For: 192.168.200.12 - VCL_call RECV - VCL_return purge - VCL_call HASH - VCL_return lookup - VCL_call PURGE - VCL_return synth - Timestamp Process: 1576136019.230863 0.000198 0.000198 - RespHeader Date: Thu, 12 Dec 2019 07:33:39 GMT - RespHeader Server: Varnish - RespHeader X-Varnish: 5 - RespProtocol HTTP/1.1 - RespStatus 200 - RespReason OK - RespReason Purged - VCL_call SYNTH - RespHeader Content-Type: text/html; charset=utf-8 - RespHeader Retry-After: 5 - VCL_return deliver - RespHeader Content-Length: 236 - Storage malloc Transient - RespHeader Accept-Ranges: bytes - RespHeader Connection: keep-alive - Timestamp Resp: 1576136019.231138 0.000473 0.000275 - ReqAcct 113 0 113 214 236 450 - End ######## PLEASE test this scenario for yourself and you can see that i do correct and the object wont remove from varnish-file Please do this scenario for yourself ________________________________ From: Guillaume Quintard > Sent: Wednesday, December 11, 2019 9:44 AM To: hamidreza hosseini > Cc: varnish-misc at varnish-cache.org > Subject: Re: Varnish doesnt purge Yes it does, we have already proved it./ Each instance purges and will refetch the object for its respective backend. I explained how to test: start with the origin, change the data and check that you actually get the new object. Then add a varnish layer, purge, check, repeat. You are testing everything at once and are unable to isolate the issue because you refuse to take things step by step, and spamming won't change that. I'm sorry but until you have tested each layer individually and identified the one that fails, I won't be able to help. -- Guillaume Quintard On Wed, Dec 11, 2019 at 3:08 PM hamidreza hosseini > wrote: Hi, please help me to solve this problem, this is the link that i explaine my question completely: https://varnish-cache.org/lists/pipermail/varnish-misc/2019-December/026754.html _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -- Sent from my iPhone -------------- next part -------------- An HTML attachment was scrubbed... URL: From varunnath86 at gmail.com Wed Dec 18 12:54:55 2019 From: varunnath86 at gmail.com (Varun Nath) Date: Wed, 18 Dec 2019 18:24:55 +0530 Subject: Add Dummy Values to Varnish Message-ID: Hello, My requirement is to Purge the Varnish cache set from an application. I need to implement this without setting the target application. Is there anyway to add dummy values to Varnish cache. In Redis cache we have 'set' command to set values, is there something similar available in Varnish? Please let me know if more details are required VARUN NATH A V -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Wed Dec 18 13:06:40 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 18 Dec 2019 14:06:40 +0100 Subject: Add Dummy Values to Varnish In-Reply-To: References: Message-ID: I'm not sure, but I think you are looking for a solution to a non-problem. Let's tackle only purging for now. Can you have a look at https://docs.varnish-software.com/tutorials/cache-invalidation/#purge and see if it solves your issue? On Wed, Dec 18, 2019, 13:56 Varun Nath wrote: > Hello, > My requirement is to Purge the Varnish cache set from an application. I > need to implement this without setting the target application. Is there > anyway to add dummy values to Varnish cache. In Redis cache we have 'set' > command to set values, is there something similar available in Varnish? > Please let me know if more details are required > > > VARUN NATH A V > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From carlos.abalde at gmail.com Wed Dec 18 16:02:30 2019 From: carlos.abalde at gmail.com (Carlos Abalde) Date: Wed, 18 Dec 2019 17:02:30 +0100 Subject: How to discover manager & worker PIDs given a Varnish Cache instance name? Message-ID: <824D08A3-AFA5-478C-9A9D-19EBDABECABA@gmail.com> Hi, Simple question related with Varnish Cache and monitoring. Let's assume a single server running one or more Varnish Cache instances. Given the name of one instance (i.e. '-n' argument), is there any reasonable way (e.g. via varnishadm) to discover the PIDs of the associated manager and worker processes? The goal is to find those PIDs in order to fetch detailed memory usage stats (virtual, resident, shared, etc.) of the associated Varnish Cache instance and then feed the monitoring agent with that info. Thanks, -- Carlos Abalde -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Wed Dec 18 20:40:29 2019 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 18 Dec 2019 20:40:29 +0000 Subject: How to discover manager & worker PIDs given a Varnish Cache instance name? In-Reply-To: <824D08A3-AFA5-478C-9A9D-19EBDABECABA@gmail.com> References: <824D08A3-AFA5-478C-9A9D-19EBDABECABA@gmail.com> Message-ID: Hello Carlos, On Wed, Dec 18, 2019 at 4:04 PM Carlos Abalde wrote: > > Hi, > > Simple question related with Varnish Cache and monitoring. Let's assume a single server running one or more Varnish Cache instances. Given the name of one instance (i.e. '-n' argument), is there any reasonable way (e.g. via varnishadm) to discover the PIDs of the associated manager and worker processes? > > The goal is to find those PIDs in order to fetch detailed memory usage stats (virtual, resident, shared, etc.) of the associated Varnish Cache instance and then feed the monitoring agent with that info. You can try this: > $ cat test.vtc > varnishtest "dude, where's my pid?" > > varnish v1 -vcl {backend be { .host = "${bad_backend}"; }} -start > > shell { > # manager > cat ${v1_name}/_.pid > echo > # child > awk '$1 == "#" {print $2}' ${v1_name}/_.vsm_child/_.index > } > > $ varnishtest -v test.vtc | grep shell_out > **** top shell_out|2038076 > **** top shell_out|2038088 Not the best interface for the child, I must admit :) Dridi > Thanks, > > -- > Carlos Abalde > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From carlos.abalde at gmail.com Thu Dec 19 08:33:32 2019 From: carlos.abalde at gmail.com (Carlos Abalde) Date: Thu, 19 Dec 2019 09:33:32 +0100 Subject: How to discover manager & worker PIDs given a Varnish Cache instance name? In-Reply-To: References: <824D08A3-AFA5-478C-9A9D-19EBDABECABA@gmail.com> Message-ID: Hi Dridi, It's ugly as hell, but it works. Sadly not an option for 4.1. Thanks a lot, -- Carlos Abalde > On 18 Dec 2019, at 21:40, Dridi Boukelmoune wrote: > > Hello Carlos, > > On Wed, Dec 18, 2019 at 4:04 PM Carlos Abalde wrote: >> >> Hi, >> >> Simple question related with Varnish Cache and monitoring. Let's assume a single server running one or more Varnish Cache instances. Given the name of one instance (i.e. '-n' argument), is there any reasonable way (e.g. via varnishadm) to discover the PIDs of the associated manager and worker processes? >> >> The goal is to find those PIDs in order to fetch detailed memory usage stats (virtual, resident, shared, etc.) of the associated Varnish Cache instance and then feed the monitoring agent with that info. > > You can try this: > >> $ cat test.vtc >> varnishtest "dude, where's my pid?" >> >> varnish v1 -vcl {backend be { .host = "${bad_backend}"; }} -start >> >> shell { >> # manager >> cat ${v1_name}/_.pid >> echo >> # child >> awk '$1 == "#" {print $2}' ${v1_name}/_.vsm_child/_.index >> } >> >> $ varnishtest -v test.vtc | grep shell_out >> **** top shell_out|2038076 >> **** top shell_out|2038088 > > Not the best interface for the child, I must admit :) > > Dridi > >> Thanks, >> >> -- >> Carlos Abalde >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From varunnath86 at gmail.com Thu Dec 19 08:38:01 2019 From: varunnath86 at gmail.com (Varun Nath) Date: Thu, 19 Dec 2019 14:08:01 +0530 Subject: Connecting to remote host Message-ID: Hello, I need to connect to a remote host from local VCL config file. I have added the backend config as below backend default { .host = "remote.host.com"; } But am getting Page not available message. Do I miss something? Thanks in advance -- VARUN NATH A V -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Dec 19 08:49:46 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 19 Dec 2019 08:49:46 +0000 Subject: Connecting to remote host In-Reply-To: References: Message-ID: Hi, Use varnishlog -b (with possibly -d) to see the backend transactions and see what IP is being used (is it the right one). It could also be that you need to change the host header before sending the request to the backend. Cheers, On Thu, Dec 19, 2019, 08:39 Varun Nath wrote: > Hello, > I need to connect to a remote host from local VCL config file. I have > added the backend config as below > > backend default { > .host = "remote.host.com"; > } > > But am getting Page not available message. Do I miss something? > Thanks in advance > > -- > VARUN NATH A V > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Dec 19 08:59:23 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 19 Dec 2019 08:59:23 +0000 Subject: Connecting to remote host In-Reply-To: References: Message-ID: Note: please keep the mailing list CC'd Check the host header you are sending to the backend. Usually, a 404 arises because the URL and/or the host wrong. I'm thinking you possibly tested varnish requesting something like http://localhost/your/test/file, but your backend isn't configured to reply to the "localhost" host On Thu, Dec 19, 2019, 08:55 Varun Nath wrote: > Thanks again for the quick response. > I have checked the log and getting the expected IP address in *BackendStart > *parameter. Sorry I didnt get what you mean by you need to change *the > host header before sending the request to the backend. *Could you please > elaborate a little bit? > > On Thu, Dec 19, 2019 at 2:20 PM Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> Hi, >> >> Use varnishlog -b (with possibly -d) to see the backend transactions and >> see what IP is being used (is it the right one). It could also be that you >> need to change the host header before sending the request to the backend. >> >> Cheers, >> >> >> On Thu, Dec 19, 2019, 08:39 Varun Nath wrote: >> >>> Hello, >>> I need to connect to a remote host from local VCL config file. I have >>> added the backend config as below >>> >>> backend default { >>> .host = "remote.host.com"; >>> } >>> >>> But am getting Page not available message. Do I miss something? >>> Thanks in advance >>> >>> -- >>> VARUN NATH A V >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> > > -- > VARUN NATH A V > -------------- next part -------------- An HTML attachment was scrubbed... URL: From varunnath86 at gmail.com Thu Dec 19 09:02:37 2019 From: varunnath86 at gmail.com (Varun Nath) Date: Thu, 19 Dec 2019 14:32:37 +0530 Subject: Connecting to remote host In-Reply-To: References: Message-ID: Yes, I tried with http://locahost, but that is expected, right? since the backend server is configured in VCL file. Sorry if I am wrong On Thu, Dec 19, 2019 at 2:29 PM Guillaume Quintard < guillaume at varnish-software.com> wrote: > Note: please keep the mailing list CC'd > > Check the host header you are sending to the backend. Usually, a 404 > arises because the URL and/or the host wrong. > > I'm thinking you possibly tested varnish requesting something like > http://localhost/your/test/file, but your backend isn't configured to > reply to the "localhost" host > > On Thu, Dec 19, 2019, 08:55 Varun Nath wrote: > >> Thanks again for the quick response. >> I have checked the log and getting the expected IP address in *BackendStart >> *parameter. Sorry I didnt get what you mean by you need to change *the >> host header before sending the request to the backend. *Could you please >> elaborate a little bit? >> >> On Thu, Dec 19, 2019 at 2:20 PM Guillaume Quintard < >> guillaume at varnish-software.com> wrote: >> >>> Hi, >>> >>> Use varnishlog -b (with possibly -d) to see the backend transactions and >>> see what IP is being used (is it the right one). It could also be that you >>> need to change the host header before sending the request to the backend. >>> >>> Cheers, >>> >>> >>> On Thu, Dec 19, 2019, 08:39 Varun Nath wrote: >>> >>>> Hello, >>>> I need to connect to a remote host from local VCL config file. I have >>>> added the backend config as below >>>> >>>> backend default { >>>> .host = "remote.host.com"; >>>> } >>>> >>>> But am getting Page not available message. Do I miss something? >>>> Thanks in advance >>>> >>>> -- >>>> VARUN NATH A V >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>> >>> >> >> -- >> VARUN NATH A V >> > -- VARUN NATH A V -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Dec 19 09:19:44 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 19 Dec 2019 09:19:44 +0000 Subject: Connecting to remote host In-Reply-To: References: Message-ID: The backend portion of the vcl only handles the TCP part of the equation. If your backend expect a specific host, you need to either have the client give it, or have varnish override it. Try curl -I localhost/your/file -H "host: whatever.your.host.is.com" On Thu, Dec 19, 2019, 09:02 Varun Nath wrote: > Yes, I tried with http://locahost, but that is expected, right? since the > backend server is configured in VCL file. Sorry if I am wrong > > On Thu, Dec 19, 2019 at 2:29 PM Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> Note: please keep the mailing list CC'd >> >> Check the host header you are sending to the backend. Usually, a 404 >> arises because the URL and/or the host wrong. >> >> I'm thinking you possibly tested varnish requesting something like >> http://localhost/your/test/file, but your backend isn't configured to >> reply to the "localhost" host >> >> On Thu, Dec 19, 2019, 08:55 Varun Nath wrote: >> >>> Thanks again for the quick response. >>> I have checked the log and getting the expected IP address in *BackendStart >>> *parameter. Sorry I didnt get what you mean by you need to change *the >>> host header before sending the request to the backend. *Could you >>> please elaborate a little bit? >>> >>> On Thu, Dec 19, 2019 at 2:20 PM Guillaume Quintard < >>> guillaume at varnish-software.com> wrote: >>> >>>> Hi, >>>> >>>> Use varnishlog -b (with possibly -d) to see the backend transactions >>>> and see what IP is being used (is it the right one). It could also be that >>>> you need to change the host header before sending the request to the >>>> backend. >>>> >>>> Cheers, >>>> >>>> >>>> On Thu, Dec 19, 2019, 08:39 Varun Nath wrote: >>>> >>>>> Hello, >>>>> I need to connect to a remote host from local VCL config file. I have >>>>> added the backend config as below >>>>> >>>>> backend default { >>>>> .host = "remote.host.com"; >>>>> } >>>>> >>>>> But am getting Page not available message. Do I miss something? >>>>> Thanks in advance >>>>> >>>>> -- >>>>> VARUN NATH A V >>>>> _______________________________________________ >>>>> varnish-misc mailing list >>>>> varnish-misc at varnish-cache.org >>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>> >>>> >>> >>> -- >>> VARUN NATH A V >>> >> > > -- > VARUN NATH A V > -------------- next part -------------- An HTML attachment was scrubbed... URL: From varunnath86 at gmail.com Thu Dec 19 09:30:15 2019 From: varunnath86 at gmail.com (Varun Nath) Date: Thu, 19 Dec 2019 15:00:15 +0530 Subject: Connecting to remote host In-Reply-To: References: Message-ID: Thank You so much for your support. Will try this and let you know On Thu, Dec 19, 2019 at 2:49 PM Guillaume Quintard < guillaume at varnish-software.com> wrote: > The backend portion of the vcl only handles the TCP part of the equation. > If your backend expect a specific host, you need to either have the client > give it, or have varnish override it. > > Try curl -I localhost/your/file -H "host: whatever.your.host.is.com" > > On Thu, Dec 19, 2019, 09:02 Varun Nath wrote: > >> Yes, I tried with http://locahost, but that is expected, right? since >> the backend server is configured in VCL file. Sorry if I am wrong >> >> On Thu, Dec 19, 2019 at 2:29 PM Guillaume Quintard < >> guillaume at varnish-software.com> wrote: >> >>> Note: please keep the mailing list CC'd >>> >>> Check the host header you are sending to the backend. Usually, a 404 >>> arises because the URL and/or the host wrong. >>> >>> I'm thinking you possibly tested varnish requesting something like >>> http://localhost/your/test/file, but your backend isn't configured to >>> reply to the "localhost" host >>> >>> On Thu, Dec 19, 2019, 08:55 Varun Nath wrote: >>> >>>> Thanks again for the quick response. >>>> I have checked the log and getting the expected IP address in *BackendStart >>>> *parameter. Sorry I didnt get what you mean by you need to change *the >>>> host header before sending the request to the backend. *Could you >>>> please elaborate a little bit? >>>> >>>> On Thu, Dec 19, 2019 at 2:20 PM Guillaume Quintard < >>>> guillaume at varnish-software.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> Use varnishlog -b (with possibly -d) to see the backend transactions >>>>> and see what IP is being used (is it the right one). It could also be that >>>>> you need to change the host header before sending the request to the >>>>> backend. >>>>> >>>>> Cheers, >>>>> >>>>> >>>>> On Thu, Dec 19, 2019, 08:39 Varun Nath wrote: >>>>> >>>>>> Hello, >>>>>> I need to connect to a remote host from local VCL config file. I have >>>>>> added the backend config as below >>>>>> >>>>>> backend default { >>>>>> .host = "remote.host.com"; >>>>>> } >>>>>> >>>>>> But am getting Page not available message. Do I miss something? >>>>>> Thanks in advance >>>>>> >>>>>> -- >>>>>> VARUN NATH A V >>>>>> _______________________________________________ >>>>>> varnish-misc mailing list >>>>>> varnish-misc at varnish-cache.org >>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>> >>>>> >>>> >>>> -- >>>> VARUN NATH A V >>>> >>> >> >> -- >> VARUN NATH A V >> > -- VARUN NATH A V -------------- next part -------------- An HTML attachment was scrubbed... URL: From varunnath86 at gmail.com Thu Dec 19 11:35:55 2019 From: varunnath86 at gmail.com (Varun Nath) Date: Thu, 19 Dec 2019 17:05:55 +0530 Subject: Connecting to remote host In-Reply-To: References: Message-ID: Hello, I got some response via CURL. But getting 403 Forbidden message. Log is as below: * BerespStatus 403* * BerespReason Forbidden* Also tried setting *remove req.http.X-Forwarded-For;set req.http.x-forwarded-for = client.ip;* But didnt work. Could you please share your thoughts? On Thu, Dec 19, 2019 at 3:00 PM Varun Nath wrote: > Thank You so much for your support. Will try this and let you know > > On Thu, Dec 19, 2019 at 2:49 PM Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> The backend portion of the vcl only handles the TCP part of the equation. >> If your backend expect a specific host, you need to either have the client >> give it, or have varnish override it. >> >> Try curl -I localhost/your/file -H "host: whatever.your.host.is.com" >> >> On Thu, Dec 19, 2019, 09:02 Varun Nath wrote: >> >>> Yes, I tried with http://locahost, but that is expected, right? since >>> the backend server is configured in VCL file. Sorry if I am wrong >>> >>> On Thu, Dec 19, 2019 at 2:29 PM Guillaume Quintard < >>> guillaume at varnish-software.com> wrote: >>> >>>> Note: please keep the mailing list CC'd >>>> >>>> Check the host header you are sending to the backend. Usually, a 404 >>>> arises because the URL and/or the host wrong. >>>> >>>> I'm thinking you possibly tested varnish requesting something like >>>> http://localhost/your/test/file, but your backend isn't configured to >>>> reply to the "localhost" host >>>> >>>> On Thu, Dec 19, 2019, 08:55 Varun Nath wrote: >>>> >>>>> Thanks again for the quick response. >>>>> I have checked the log and getting the expected IP address in *BackendStart >>>>> *parameter. Sorry I didnt get what you mean by you need to change *the >>>>> host header before sending the request to the backend. *Could you >>>>> please elaborate a little bit? >>>>> >>>>> On Thu, Dec 19, 2019 at 2:20 PM Guillaume Quintard < >>>>> guillaume at varnish-software.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Use varnishlog -b (with possibly -d) to see the backend transactions >>>>>> and see what IP is being used (is it the right one). It could also be that >>>>>> you need to change the host header before sending the request to the >>>>>> backend. >>>>>> >>>>>> Cheers, >>>>>> >>>>>> >>>>>> On Thu, Dec 19, 2019, 08:39 Varun Nath wrote: >>>>>> >>>>>>> Hello, >>>>>>> I need to connect to a remote host from local VCL config file. I >>>>>>> have added the backend config as below >>>>>>> >>>>>>> backend default { >>>>>>> .host = "remote.host.com"; >>>>>>> } >>>>>>> >>>>>>> But am getting Page not available message. Do I miss something? >>>>>>> Thanks in advance >>>>>>> >>>>>>> -- >>>>>>> VARUN NATH A V >>>>>>> _______________________________________________ >>>>>>> varnish-misc mailing list >>>>>>> varnish-misc at varnish-cache.org >>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> VARUN NATH A V >>>>> >>>> >>> >>> -- >>> VARUN NATH A V >>> >> > > -- > VARUN NATH A V > -- VARUN NATH A V -------------- next part -------------- An HTML attachment was scrubbed... URL: From varunnath86 at gmail.com Thu Dec 19 12:03:02 2019 From: varunnath86 at gmail.com (Varun Nath) Date: Thu, 19 Dec 2019 17:33:02 +0530 Subject: Connecting to remote host In-Reply-To: References: Message-ID: This is the VCL file am using: Note: Host is not a valid one vcl 4.0; import std; import directors; # Default backend definition. Set this to point to your content server. backend default { .host = "remote.host.com"; .port = "80"; } sub vcl_init { new vdir = directors.round_robin(); vdir.add_backend(default); } sub vcl_recv { unset req.http.proxy; unset req.http.cookie; set req.http.x-host = "http://remote.host.com"; set req.http.x-clientip = client.ip; set req.http.x-serverip = server.ip; set req.http.x-localip = local.ip; set req.http.x-remoteip = remote.ip; unset req.http.X-Forwarded-For; set req.http.x-forwarded-for = client.ip; return(pass); } sub vcl_backend_response { # Happens after we have read the response headers from the backend. # # Here you clean the response headers, removing silly Set-Cookie headers # and other mistakes your backend does. } sub vcl_deliver { # Happens when we have all the pieces we need, and are about to send the # response to the client. } Please let me know if I need to add something more to make the remote host caching from localhost varnish On Thu, Dec 19, 2019 at 5:05 PM Varun Nath wrote: > Hello, > I got some response via CURL. But getting 403 Forbidden message. Log is as > below: > * BerespStatus 403* > * BerespReason Forbidden* > Also tried setting > > *remove req.http.X-Forwarded-For;set req.http.x-forwarded-for = client.ip;* > But didnt work. > Could you please share your thoughts? > > On Thu, Dec 19, 2019 at 3:00 PM Varun Nath wrote: > >> Thank You so much for your support. Will try this and let you know >> >> On Thu, Dec 19, 2019 at 2:49 PM Guillaume Quintard < >> guillaume at varnish-software.com> wrote: >> >>> The backend portion of the vcl only handles the TCP part of the >>> equation. If your backend expect a specific host, you need to either have >>> the client give it, or have varnish override it. >>> >>> Try curl -I localhost/your/file -H "host: whatever.your.host.is.com" >>> >>> On Thu, Dec 19, 2019, 09:02 Varun Nath wrote: >>> >>>> Yes, I tried with http://locahost, but that is expected, right? since >>>> the backend server is configured in VCL file. Sorry if I am wrong >>>> >>>> On Thu, Dec 19, 2019 at 2:29 PM Guillaume Quintard < >>>> guillaume at varnish-software.com> wrote: >>>> >>>>> Note: please keep the mailing list CC'd >>>>> >>>>> Check the host header you are sending to the backend. Usually, a 404 >>>>> arises because the URL and/or the host wrong. >>>>> >>>>> I'm thinking you possibly tested varnish requesting something like >>>>> http://localhost/your/test/file, but your backend isn't configured to >>>>> reply to the "localhost" host >>>>> >>>>> On Thu, Dec 19, 2019, 08:55 Varun Nath wrote: >>>>> >>>>>> Thanks again for the quick response. >>>>>> I have checked the log and getting the expected IP address in *BackendStart >>>>>> *parameter. Sorry I didnt get what you mean by you need to change *the >>>>>> host header before sending the request to the backend. *Could you >>>>>> please elaborate a little bit? >>>>>> >>>>>> On Thu, Dec 19, 2019 at 2:20 PM Guillaume Quintard < >>>>>> guillaume at varnish-software.com> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Use varnishlog -b (with possibly -d) to see the backend transactions >>>>>>> and see what IP is being used (is it the right one). It could also be that >>>>>>> you need to change the host header before sending the request to the >>>>>>> backend. >>>>>>> >>>>>>> Cheers, >>>>>>> >>>>>>> >>>>>>> On Thu, Dec 19, 2019, 08:39 Varun Nath >>>>>>> wrote: >>>>>>> >>>>>>>> Hello, >>>>>>>> I need to connect to a remote host from local VCL config file. I >>>>>>>> have added the backend config as below >>>>>>>> >>>>>>>> backend default { >>>>>>>> .host = "remote.host.com"; >>>>>>>> } >>>>>>>> >>>>>>>> But am getting Page not available message. Do I miss something? >>>>>>>> Thanks in advance >>>>>>>> >>>>>>>> -- >>>>>>>> VARUN NATH A V >>>>>>>> _______________________________________________ >>>>>>>> varnish-misc mailing list >>>>>>>> varnish-misc at varnish-cache.org >>>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> VARUN NATH A V >>>>>> >>>>> >>>> >>>> -- >>>> VARUN NATH A V >>>> >>> >> >> -- >> VARUN NATH A V >> > > > -- > VARUN NATH A V > -- VARUN NATH A V -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Dec 19 12:15:36 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 19 Dec 2019 13:15:36 +0100 Subject: Connecting to remote host In-Reply-To: References: Message-ID: check the backend transaction, and understand why your origin is sending back a 403. This is due to your backend configuration and I can't help you with that. -- Guillaume Quintard On Thu, Dec 19, 2019 at 1:03 PM Varun Nath wrote: > This is the VCL file am using: > > Note: Host is not a valid one > > vcl 4.0; > import std; > import directors; > > # Default backend definition. Set this to point to your content server. > backend default { > .host = "remote.host.com"; > .port = "80"; > } > > sub vcl_init { > new vdir = directors.round_robin(); > vdir.add_backend(default); > } > > sub vcl_recv { > unset req.http.proxy; > unset req.http.cookie; > set req.http.x-host = "http://remote.host.com"; > set req.http.x-clientip = client.ip; > set req.http.x-serverip = server.ip; > set req.http.x-localip = local.ip; > set req.http.x-remoteip = remote.ip; > unset req.http.X-Forwarded-For; > set req.http.x-forwarded-for = client.ip; > return(pass); > } > > sub vcl_backend_response { > # Happens after we have read the response headers from the backend. > # > # Here you clean the response headers, removing silly Set-Cookie > headers > # and other mistakes your backend does. > } > > sub vcl_deliver { > # Happens when we have all the pieces we need, and are about to send > the > # response to the client. > } > > Please let me know if I need to add something more to make the remote host > caching from localhost varnish > > On Thu, Dec 19, 2019 at 5:05 PM Varun Nath wrote: > >> Hello, >> I got some response via CURL. But getting 403 Forbidden message. Log is >> as below: >> * BerespStatus 403* >> * BerespReason Forbidden* >> Also tried setting >> >> *remove req.http.X-Forwarded-For;set req.http.x-forwarded-for = >> client.ip;* >> But didnt work. >> Could you please share your thoughts? >> >> On Thu, Dec 19, 2019 at 3:00 PM Varun Nath wrote: >> >>> Thank You so much for your support. Will try this and let you know >>> >>> On Thu, Dec 19, 2019 at 2:49 PM Guillaume Quintard < >>> guillaume at varnish-software.com> wrote: >>> >>>> The backend portion of the vcl only handles the TCP part of the >>>> equation. If your backend expect a specific host, you need to either have >>>> the client give it, or have varnish override it. >>>> >>>> Try curl -I localhost/your/file -H "host: whatever.your.host.is.com" >>>> >>>> On Thu, Dec 19, 2019, 09:02 Varun Nath wrote: >>>> >>>>> Yes, I tried with http://locahost, but that is expected, right? since >>>>> the backend server is configured in VCL file. Sorry if I am wrong >>>>> >>>>> On Thu, Dec 19, 2019 at 2:29 PM Guillaume Quintard < >>>>> guillaume at varnish-software.com> wrote: >>>>> >>>>>> Note: please keep the mailing list CC'd >>>>>> >>>>>> Check the host header you are sending to the backend. Usually, a 404 >>>>>> arises because the URL and/or the host wrong. >>>>>> >>>>>> I'm thinking you possibly tested varnish requesting something like >>>>>> http://localhost/your/test/file, but your backend isn't configured >>>>>> to reply to the "localhost" host >>>>>> >>>>>> On Thu, Dec 19, 2019, 08:55 Varun Nath wrote: >>>>>> >>>>>>> Thanks again for the quick response. >>>>>>> I have checked the log and getting the expected IP address in *BackendStart >>>>>>> *parameter. Sorry I didnt get what you mean by you need to change *the >>>>>>> host header before sending the request to the backend. *Could you >>>>>>> please elaborate a little bit? >>>>>>> >>>>>>> On Thu, Dec 19, 2019 at 2:20 PM Guillaume Quintard < >>>>>>> guillaume at varnish-software.com> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> Use varnishlog -b (with possibly -d) to see the backend >>>>>>>> transactions and see what IP is being used (is it the right one). It could >>>>>>>> also be that you need to change the host header before sending the request >>>>>>>> to the backend. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Dec 19, 2019, 08:39 Varun Nath >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Hello, >>>>>>>>> I need to connect to a remote host from local VCL config file. I >>>>>>>>> have added the backend config as below >>>>>>>>> >>>>>>>>> backend default { >>>>>>>>> .host = "remote.host.com"; >>>>>>>>> } >>>>>>>>> >>>>>>>>> But am getting Page not available message. Do I miss something? >>>>>>>>> Thanks in advance >>>>>>>>> >>>>>>>>> -- >>>>>>>>> VARUN NATH A V >>>>>>>>> _______________________________________________ >>>>>>>>> varnish-misc mailing list >>>>>>>>> varnish-misc at varnish-cache.org >>>>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> VARUN NATH A V >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> VARUN NATH A V >>>>> >>>> >>> >>> -- >>> VARUN NATH A V >>> >> >> >> -- >> VARUN NATH A V >> > > > -- > VARUN NATH A V > -------------- next part -------------- An HTML attachment was scrubbed... URL: From varunnath86 at gmail.com Thu Dec 19 12:18:10 2019 From: varunnath86 at gmail.com (Varun Nath) Date: Thu, 19 Dec 2019 17:48:10 +0530 Subject: Connecting to remote host In-Reply-To: References: Message-ID: Thank You. Could you please confirm whether the VCL config I have shared if fine? On Thu, Dec 19, 2019 at 5:46 PM Guillaume Quintard < guillaume at varnish-software.com> wrote: > check the backend transaction, and understand why your origin is sending > back a 403. This is due to your backend configuration and I can't help you > with that. > > -- > Guillaume Quintard > > > On Thu, Dec 19, 2019 at 1:03 PM Varun Nath wrote: > >> This is the VCL file am using: >> >> Note: Host is not a valid one >> >> vcl 4.0; >> import std; >> import directors; >> >> # Default backend definition. Set this to point to your content server. >> backend default { >> .host = "remote.host.com"; >> .port = "80"; >> } >> >> sub vcl_init { >> new vdir = directors.round_robin(); >> vdir.add_backend(default); >> } >> >> sub vcl_recv { >> unset req.http.proxy; >> unset req.http.cookie; >> set req.http.x-host = "http://remote.host.com"; >> set req.http.x-clientip = client.ip; >> set req.http.x-serverip = server.ip; >> set req.http.x-localip = local.ip; >> set req.http.x-remoteip = remote.ip; >> unset req.http.X-Forwarded-For; >> set req.http.x-forwarded-for = client.ip; >> return(pass); >> } >> >> sub vcl_backend_response { >> # Happens after we have read the response headers from the backend. >> # >> # Here you clean the response headers, removing silly Set-Cookie >> headers >> # and other mistakes your backend does. >> } >> >> sub vcl_deliver { >> # Happens when we have all the pieces we need, and are about to send >> the >> # response to the client. >> } >> >> Please let me know if I need to add something more to make the remote >> host caching from localhost varnish >> >> On Thu, Dec 19, 2019 at 5:05 PM Varun Nath wrote: >> >>> Hello, >>> I got some response via CURL. But getting 403 Forbidden message. Log is >>> as below: >>> * BerespStatus 403* >>> * BerespReason Forbidden* >>> Also tried setting >>> >>> *remove req.http.X-Forwarded-For;set req.http.x-forwarded-for = >>> client.ip;* >>> But didnt work. >>> Could you please share your thoughts? >>> >>> On Thu, Dec 19, 2019 at 3:00 PM Varun Nath >>> wrote: >>> >>>> Thank You so much for your support. Will try this and let you know >>>> >>>> On Thu, Dec 19, 2019 at 2:49 PM Guillaume Quintard < >>>> guillaume at varnish-software.com> wrote: >>>> >>>>> The backend portion of the vcl only handles the TCP part of the >>>>> equation. If your backend expect a specific host, you need to either have >>>>> the client give it, or have varnish override it. >>>>> >>>>> Try curl -I localhost/your/file -H "host: whatever.your.host.is.com" >>>>> >>>>> On Thu, Dec 19, 2019, 09:02 Varun Nath wrote: >>>>> >>>>>> Yes, I tried with http://locahost, but that is expected, right? >>>>>> since the backend server is configured in VCL file. Sorry if I am wrong >>>>>> >>>>>> On Thu, Dec 19, 2019 at 2:29 PM Guillaume Quintard < >>>>>> guillaume at varnish-software.com> wrote: >>>>>> >>>>>>> Note: please keep the mailing list CC'd >>>>>>> >>>>>>> Check the host header you are sending to the backend. Usually, a 404 >>>>>>> arises because the URL and/or the host wrong. >>>>>>> >>>>>>> I'm thinking you possibly tested varnish requesting something like >>>>>>> http://localhost/your/test/file, but your backend isn't configured >>>>>>> to reply to the "localhost" host >>>>>>> >>>>>>> On Thu, Dec 19, 2019, 08:55 Varun Nath >>>>>>> wrote: >>>>>>> >>>>>>>> Thanks again for the quick response. >>>>>>>> I have checked the log and getting the expected IP address in *BackendStart >>>>>>>> *parameter. Sorry I didnt get what you mean by you need to >>>>>>>> change *the host header before sending the request to the backend. >>>>>>>> *Could you please elaborate a little bit? >>>>>>>> >>>>>>>> On Thu, Dec 19, 2019 at 2:20 PM Guillaume Quintard < >>>>>>>> guillaume at varnish-software.com> wrote: >>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> Use varnishlog -b (with possibly -d) to see the backend >>>>>>>>> transactions and see what IP is being used (is it the right one). It could >>>>>>>>> also be that you need to change the host header before sending the request >>>>>>>>> to the backend. >>>>>>>>> >>>>>>>>> Cheers, >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Dec 19, 2019, 08:39 Varun Nath >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hello, >>>>>>>>>> I need to connect to a remote host from local VCL config file. I >>>>>>>>>> have added the backend config as below >>>>>>>>>> >>>>>>>>>> backend default { >>>>>>>>>> .host = "remote.host.com"; >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> But am getting Page not available message. Do I miss something? >>>>>>>>>> Thanks in advance >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> VARUN NATH A V >>>>>>>>>> _______________________________________________ >>>>>>>>>> varnish-misc mailing list >>>>>>>>>> varnish-misc at varnish-cache.org >>>>>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> VARUN NATH A V >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> VARUN NATH A V >>>>>> >>>>> >>>> >>>> -- >>>> VARUN NATH A V >>>> >>> >>> >>> -- >>> VARUN NATH A V >>> >> >> >> -- >> VARUN NATH A V >> > -- VARUN NATH A V -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Dec 19 12:45:10 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 19 Dec 2019 13:45:10 +0100 Subject: Connecting to remote host In-Reply-To: References: Message-ID: you need to understand what is going on why the backend is replying with 403, then we can fix the vcl. -- Guillaume Quintard On Thu, Dec 19, 2019 at 1:18 PM Varun Nath wrote: > Thank You. Could you please confirm whether the VCL config I have shared > if fine? > > On Thu, Dec 19, 2019 at 5:46 PM Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> check the backend transaction, and understand why your origin is sending >> back a 403. This is due to your backend configuration and I can't help you >> with that. >> >> -- >> Guillaume Quintard >> >> >> On Thu, Dec 19, 2019 at 1:03 PM Varun Nath wrote: >> >>> This is the VCL file am using: >>> >>> Note: Host is not a valid one >>> >>> vcl 4.0; >>> import std; >>> import directors; >>> >>> # Default backend definition. Set this to point to your content server. >>> backend default { >>> .host = "remote.host.com"; >>> .port = "80"; >>> } >>> >>> sub vcl_init { >>> new vdir = directors.round_robin(); >>> vdir.add_backend(default); >>> } >>> >>> sub vcl_recv { >>> unset req.http.proxy; >>> unset req.http.cookie; >>> set req.http.x-host = "http://remote.host.com"; >>> set req.http.x-clientip = client.ip; >>> set req.http.x-serverip = server.ip; >>> set req.http.x-localip = local.ip; >>> set req.http.x-remoteip = remote.ip; >>> unset req.http.X-Forwarded-For; >>> set req.http.x-forwarded-for = client.ip; >>> return(pass); >>> } >>> >>> sub vcl_backend_response { >>> # Happens after we have read the response headers from the backend. >>> # >>> # Here you clean the response headers, removing silly Set-Cookie >>> headers >>> # and other mistakes your backend does. >>> } >>> >>> sub vcl_deliver { >>> # Happens when we have all the pieces we need, and are about to send >>> the >>> # response to the client. >>> } >>> >>> Please let me know if I need to add something more to make the remote >>> host caching from localhost varnish >>> >>> On Thu, Dec 19, 2019 at 5:05 PM Varun Nath >>> wrote: >>> >>>> Hello, >>>> I got some response via CURL. But getting 403 Forbidden message. Log is >>>> as below: >>>> * BerespStatus 403* >>>> * BerespReason Forbidden* >>>> Also tried setting >>>> >>>> *remove req.http.X-Forwarded-For;set req.http.x-forwarded-for = >>>> client.ip;* >>>> But didnt work. >>>> Could you please share your thoughts? >>>> >>>> On Thu, Dec 19, 2019 at 3:00 PM Varun Nath >>>> wrote: >>>> >>>>> Thank You so much for your support. Will try this and let you know >>>>> >>>>> On Thu, Dec 19, 2019 at 2:49 PM Guillaume Quintard < >>>>> guillaume at varnish-software.com> wrote: >>>>> >>>>>> The backend portion of the vcl only handles the TCP part of the >>>>>> equation. If your backend expect a specific host, you need to either have >>>>>> the client give it, or have varnish override it. >>>>>> >>>>>> Try curl -I localhost/your/file -H "host: whatever.your.host.is.com" >>>>>> >>>>>> On Thu, Dec 19, 2019, 09:02 Varun Nath wrote: >>>>>> >>>>>>> Yes, I tried with http://locahost, but that is expected, right? >>>>>>> since the backend server is configured in VCL file. Sorry if I am wrong >>>>>>> >>>>>>> On Thu, Dec 19, 2019 at 2:29 PM Guillaume Quintard < >>>>>>> guillaume at varnish-software.com> wrote: >>>>>>> >>>>>>>> Note: please keep the mailing list CC'd >>>>>>>> >>>>>>>> Check the host header you are sending to the backend. Usually, a >>>>>>>> 404 arises because the URL and/or the host wrong. >>>>>>>> >>>>>>>> I'm thinking you possibly tested varnish requesting something like >>>>>>>> http://localhost/your/test/file, but your backend isn't configured >>>>>>>> to reply to the "localhost" host >>>>>>>> >>>>>>>> On Thu, Dec 19, 2019, 08:55 Varun Nath >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Thanks again for the quick response. >>>>>>>>> I have checked the log and getting the expected IP address in *BackendStart >>>>>>>>> *parameter. Sorry I didnt get what you mean by you need to >>>>>>>>> change *the host header before sending the request to the >>>>>>>>> backend. *Could you please elaborate a little bit? >>>>>>>>> >>>>>>>>> On Thu, Dec 19, 2019 at 2:20 PM Guillaume Quintard < >>>>>>>>> guillaume at varnish-software.com> wrote: >>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> Use varnishlog -b (with possibly -d) to see the backend >>>>>>>>>> transactions and see what IP is being used (is it the right one). It could >>>>>>>>>> also be that you need to change the host header before sending the request >>>>>>>>>> to the backend. >>>>>>>>>> >>>>>>>>>> Cheers, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Dec 19, 2019, 08:39 Varun Nath >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hello, >>>>>>>>>>> I need to connect to a remote host from local VCL config file. I >>>>>>>>>>> have added the backend config as below >>>>>>>>>>> >>>>>>>>>>> backend default { >>>>>>>>>>> .host = "remote.host.com"; >>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> But am getting Page not available message. Do I miss something? >>>>>>>>>>> Thanks in advance >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> VARUN NATH A V >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> varnish-misc mailing list >>>>>>>>>>> varnish-misc at varnish-cache.org >>>>>>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> VARUN NATH A V >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> VARUN NATH A V >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> VARUN NATH A V >>>>> >>>> >>>> >>>> -- >>>> VARUN NATH A V >>>> >>> >>> >>> -- >>> VARUN NATH A V >>> >> > > -- > VARUN NATH A V > -------------- next part -------------- An HTML attachment was scrubbed... URL: From varunnath86 at gmail.com Thu Dec 19 12:59:35 2019 From: varunnath86 at gmail.com (Varun Nath) Date: Thu, 19 Dec 2019 18:29:35 +0530 Subject: Connecting to remote host In-Reply-To: References: Message-ID: I am getting 502 Operation not permitted error. The log is as below: ObjStatus 502 ObjReason Operation not permitted Sorry for the inconvenience ?? On Thu, Dec 19, 2019 at 6:15 PM Guillaume Quintard < guillaume at varnish-software.com> wrote: > you need to understand what is going on why the backend is replying with > 403, then we can fix the vcl. > > -- > Guillaume Quintard > > > On Thu, Dec 19, 2019 at 1:18 PM Varun Nath wrote: > >> Thank You. Could you please confirm whether the VCL config I have shared >> if fine? >> >> On Thu, Dec 19, 2019 at 5:46 PM Guillaume Quintard < >> guillaume at varnish-software.com> wrote: >> >>> check the backend transaction, and understand why your origin is sending >>> back a 403. This is due to your backend configuration and I can't help you >>> with that. >>> >>> -- >>> Guillaume Quintard >>> >>> >>> On Thu, Dec 19, 2019 at 1:03 PM Varun Nath >>> wrote: >>> >>>> This is the VCL file am using: >>>> >>>> Note: Host is not a valid one >>>> >>>> vcl 4.0; >>>> import std; >>>> import directors; >>>> >>>> # Default backend definition. Set this to point to your content server. >>>> backend default { >>>> .host = "remote.host.com"; >>>> .port = "80"; >>>> } >>>> >>>> sub vcl_init { >>>> new vdir = directors.round_robin(); >>>> vdir.add_backend(default); >>>> } >>>> >>>> sub vcl_recv { >>>> unset req.http.proxy; >>>> unset req.http.cookie; >>>> set req.http.x-host = "http://remote.host.com"; >>>> set req.http.x-clientip = client.ip; >>>> set req.http.x-serverip = server.ip; >>>> set req.http.x-localip = local.ip; >>>> set req.http.x-remoteip = remote.ip; >>>> unset req.http.X-Forwarded-For; >>>> set req.http.x-forwarded-for = client.ip; >>>> return(pass); >>>> } >>>> >>>> sub vcl_backend_response { >>>> # Happens after we have read the response headers from the backend. >>>> # >>>> # Here you clean the response headers, removing silly Set-Cookie >>>> headers >>>> # and other mistakes your backend does. >>>> } >>>> >>>> sub vcl_deliver { >>>> # Happens when we have all the pieces we need, and are about to >>>> send the >>>> # response to the client. >>>> } >>>> >>>> Please let me know if I need to add something more to make the remote >>>> host caching from localhost varnish >>>> >>>> On Thu, Dec 19, 2019 at 5:05 PM Varun Nath >>>> wrote: >>>> >>>>> Hello, >>>>> I got some response via CURL. But getting 403 Forbidden message. Log >>>>> is as below: >>>>> * BerespStatus 403* >>>>> * BerespReason Forbidden* >>>>> Also tried setting >>>>> >>>>> *remove req.http.X-Forwarded-For;set req.http.x-forwarded-for = >>>>> client.ip;* >>>>> But didnt work. >>>>> Could you please share your thoughts? >>>>> >>>>> On Thu, Dec 19, 2019 at 3:00 PM Varun Nath >>>>> wrote: >>>>> >>>>>> Thank You so much for your support. Will try this and let you know >>>>>> >>>>>> On Thu, Dec 19, 2019 at 2:49 PM Guillaume Quintard < >>>>>> guillaume at varnish-software.com> wrote: >>>>>> >>>>>>> The backend portion of the vcl only handles the TCP part of the >>>>>>> equation. If your backend expect a specific host, you need to either have >>>>>>> the client give it, or have varnish override it. >>>>>>> >>>>>>> Try curl -I localhost/your/file -H "host: whatever.your.host.is.com" >>>>>>> >>>>>>> On Thu, Dec 19, 2019, 09:02 Varun Nath >>>>>>> wrote: >>>>>>> >>>>>>>> Yes, I tried with http://locahost, but that is expected, right? >>>>>>>> since the backend server is configured in VCL file. Sorry if I am wrong >>>>>>>> >>>>>>>> On Thu, Dec 19, 2019 at 2:29 PM Guillaume Quintard < >>>>>>>> guillaume at varnish-software.com> wrote: >>>>>>>> >>>>>>>>> Note: please keep the mailing list CC'd >>>>>>>>> >>>>>>>>> Check the host header you are sending to the backend. Usually, a >>>>>>>>> 404 arises because the URL and/or the host wrong. >>>>>>>>> >>>>>>>>> I'm thinking you possibly tested varnish requesting something like >>>>>>>>> http://localhost/your/test/file, but your backend isn't >>>>>>>>> configured to reply to the "localhost" host >>>>>>>>> >>>>>>>>> On Thu, Dec 19, 2019, 08:55 Varun Nath >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Thanks again for the quick response. >>>>>>>>>> I have checked the log and getting the expected IP address in *BackendStart >>>>>>>>>> *parameter. Sorry I didnt get what you mean by you need to >>>>>>>>>> change *the host header before sending the request to the >>>>>>>>>> backend. *Could you please elaborate a little bit? >>>>>>>>>> >>>>>>>>>> On Thu, Dec 19, 2019 at 2:20 PM Guillaume Quintard < >>>>>>>>>> guillaume at varnish-software.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> Use varnishlog -b (with possibly -d) to see the backend >>>>>>>>>>> transactions and see what IP is being used (is it the right one). It could >>>>>>>>>>> also be that you need to change the host header before sending the request >>>>>>>>>>> to the backend. >>>>>>>>>>> >>>>>>>>>>> Cheers, >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Thu, Dec 19, 2019, 08:39 Varun Nath >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hello, >>>>>>>>>>>> I need to connect to a remote host from local VCL config file. >>>>>>>>>>>> I have added the backend config as below >>>>>>>>>>>> >>>>>>>>>>>> backend default { >>>>>>>>>>>> .host = "remote.host.com"; >>>>>>>>>>>> } >>>>>>>>>>>> >>>>>>>>>>>> But am getting Page not available message. Do I miss something? >>>>>>>>>>>> Thanks in advance >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> VARUN NATH A V >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> varnish-misc mailing list >>>>>>>>>>>> varnish-misc at varnish-cache.org >>>>>>>>>>>> >>>>>>>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> VARUN NATH A V >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> VARUN NATH A V >>>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> VARUN NATH A V >>>>>> >>>>> >>>>> >>>>> -- >>>>> VARUN NATH A V >>>>> >>>> >>>> >>>> -- >>>> VARUN NATH A V >>>> >>> >> >> -- >> VARUN NATH A V >> > -- VARUN NATH A V -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Dec 19 13:17:29 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 19 Dec 2019 14:17:29 +0100 Subject: Connecting to remote host In-Reply-To: References: Message-ID: varnish doesn't generate 502s, that's your backend misbehaving. Again, look at what you are sending to it using varnishlog, and understand what is happening, but at this time, this is not a varnish issue. -- Guillaume Quintard On Thu, Dec 19, 2019 at 1:59 PM Varun Nath wrote: > I am getting 502 Operation not permitted error. The log is as below: > > ObjStatus 502 > ObjReason Operation not permitted > > Sorry for the inconvenience ?? > > On Thu, Dec 19, 2019 at 6:15 PM Guillaume Quintard < > guillaume at varnish-software.com> wrote: > >> you need to understand what is going on why the backend is replying with >> 403, then we can fix the vcl. >> >> -- >> Guillaume Quintard >> >> >> On Thu, Dec 19, 2019 at 1:18 PM Varun Nath wrote: >> >>> Thank You. Could you please confirm whether the VCL config I have shared >>> if fine? >>> >>> On Thu, Dec 19, 2019 at 5:46 PM Guillaume Quintard < >>> guillaume at varnish-software.com> wrote: >>> >>>> check the backend transaction, and understand why your origin is >>>> sending back a 403. This is due to your backend configuration and I can't >>>> help you with that. >>>> >>>> -- >>>> Guillaume Quintard >>>> >>>> >>>> On Thu, Dec 19, 2019 at 1:03 PM Varun Nath >>>> wrote: >>>> >>>>> This is the VCL file am using: >>>>> >>>>> Note: Host is not a valid one >>>>> >>>>> vcl 4.0; >>>>> import std; >>>>> import directors; >>>>> >>>>> # Default backend definition. Set this to point to your content server. >>>>> backend default { >>>>> .host = "remote.host.com"; >>>>> .port = "80"; >>>>> } >>>>> >>>>> sub vcl_init { >>>>> new vdir = directors.round_robin(); >>>>> vdir.add_backend(default); >>>>> } >>>>> >>>>> sub vcl_recv { >>>>> unset req.http.proxy; >>>>> unset req.http.cookie; >>>>> set req.http.x-host = "http://remote.host.com"; >>>>> set req.http.x-clientip = client.ip; >>>>> set req.http.x-serverip = server.ip; >>>>> set req.http.x-localip = local.ip; >>>>> set req.http.x-remoteip = remote.ip; >>>>> unset req.http.X-Forwarded-For; >>>>> set req.http.x-forwarded-for = client.ip; >>>>> return(pass); >>>>> } >>>>> >>>>> sub vcl_backend_response { >>>>> # Happens after we have read the response headers from the backend. >>>>> # >>>>> # Here you clean the response headers, removing silly Set-Cookie >>>>> headers >>>>> # and other mistakes your backend does. >>>>> } >>>>> >>>>> sub vcl_deliver { >>>>> # Happens when we have all the pieces we need, and are about to >>>>> send the >>>>> # response to the client. >>>>> } >>>>> >>>>> Please let me know if I need to add something more to make the remote >>>>> host caching from localhost varnish >>>>> >>>>> On Thu, Dec 19, 2019 at 5:05 PM Varun Nath >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> I got some response via CURL. But getting 403 Forbidden message. Log >>>>>> is as below: >>>>>> * BerespStatus 403* >>>>>> * BerespReason Forbidden* >>>>>> Also tried setting >>>>>> >>>>>> *remove req.http.X-Forwarded-For;set req.http.x-forwarded-for = >>>>>> client.ip;* >>>>>> But didnt work. >>>>>> Could you please share your thoughts? >>>>>> >>>>>> On Thu, Dec 19, 2019 at 3:00 PM Varun Nath >>>>>> wrote: >>>>>> >>>>>>> Thank You so much for your support. Will try this and let you know >>>>>>> >>>>>>> On Thu, Dec 19, 2019 at 2:49 PM Guillaume Quintard < >>>>>>> guillaume at varnish-software.com> wrote: >>>>>>> >>>>>>>> The backend portion of the vcl only handles the TCP part of the >>>>>>>> equation. If your backend expect a specific host, you need to either have >>>>>>>> the client give it, or have varnish override it. >>>>>>>> >>>>>>>> Try curl -I localhost/your/file -H "host: whatever.your.host.is.com >>>>>>>> " >>>>>>>> >>>>>>>> On Thu, Dec 19, 2019, 09:02 Varun Nath >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Yes, I tried with http://locahost, but that is expected, right? >>>>>>>>> since the backend server is configured in VCL file. Sorry if I am wrong >>>>>>>>> >>>>>>>>> On Thu, Dec 19, 2019 at 2:29 PM Guillaume Quintard < >>>>>>>>> guillaume at varnish-software.com> wrote: >>>>>>>>> >>>>>>>>>> Note: please keep the mailing list CC'd >>>>>>>>>> >>>>>>>>>> Check the host header you are sending to the backend. Usually, a >>>>>>>>>> 404 arises because the URL and/or the host wrong. >>>>>>>>>> >>>>>>>>>> I'm thinking you possibly tested varnish requesting something >>>>>>>>>> like http://localhost/your/test/file, but your backend isn't >>>>>>>>>> configured to reply to the "localhost" host >>>>>>>>>> >>>>>>>>>> On Thu, Dec 19, 2019, 08:55 Varun Nath >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Thanks again for the quick response. >>>>>>>>>>> I have checked the log and getting the expected IP address in *BackendStart >>>>>>>>>>> *parameter. Sorry I didnt get what you mean by you need to >>>>>>>>>>> change *the host header before sending the request to the >>>>>>>>>>> backend. *Could you please elaborate a little bit? >>>>>>>>>>> >>>>>>>>>>> On Thu, Dec 19, 2019 at 2:20 PM Guillaume Quintard < >>>>>>>>>>> guillaume at varnish-software.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> Use varnishlog -b (with possibly -d) to see the backend >>>>>>>>>>>> transactions and see what IP is being used (is it the right one). It could >>>>>>>>>>>> also be that you need to change the host header before sending the request >>>>>>>>>>>> to the backend. >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Dec 19, 2019, 08:39 Varun Nath >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hello, >>>>>>>>>>>>> I need to connect to a remote host from local VCL config file. >>>>>>>>>>>>> I have added the backend config as below >>>>>>>>>>>>> >>>>>>>>>>>>> backend default { >>>>>>>>>>>>> .host = "remote.host.com"; >>>>>>>>>>>>> } >>>>>>>>>>>>> >>>>>>>>>>>>> But am getting Page not available message. Do I miss something? >>>>>>>>>>>>> Thanks in advance >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> VARUN NATH A V >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> varnish-misc mailing list >>>>>>>>>>>>> varnish-misc at varnish-cache.org >>>>>>>>>>>>> >>>>>>>>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> VARUN NATH A V >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> VARUN NATH A V >>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> -- >>>>>>> VARUN NATH A V >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> VARUN NATH A V >>>>>> >>>>> >>>>> >>>>> -- >>>>> VARUN NATH A V >>>>> >>>> >>> >>> -- >>> VARUN NATH A V >>> >> > > -- > VARUN NATH A V > -------------- next part -------------- An HTML attachment was scrubbed... URL: From varunnath86 at gmail.com Thu Dec 19 13:24:53 2019 From: varunnath86 at gmail.com (Varun Nath) Date: Thu, 19 Dec 2019 18:54:53 +0530 Subject: Connecting to remote host In-Reply-To: References: Message-ID: Okie let me check. No idea what's going wrong. Trying with different backend servers and all are getting 502. On Thu, Dec 19, 2019, 6:48 PM Guillaume Quintard < guillaume at varnish-software.com> wrote: > varnish doesn't generate 502s, that's your backend misbehaving. Again, > look at what you are sending to it using varnishlog, and understand what is > happening, but at this time, this is not a varnish issue. > -- > Guillaume Quintard > > > On Thu, Dec 19, 2019 at 1:59 PM Varun Nath wrote: > >> I am getting 502 Operation not permitted error. The log is as below: >> >> ObjStatus 502 >> ObjReason Operation not permitted >> >> Sorry for the inconvenience ?? >> >> On Thu, Dec 19, 2019 at 6:15 PM Guillaume Quintard < >> guillaume at varnish-software.com> wrote: >> >>> you need to understand what is going on why the backend is replying with >>> 403, then we can fix the vcl. >>> >>> -- >>> Guillaume Quintard >>> >>> >>> On Thu, Dec 19, 2019 at 1:18 PM Varun Nath >>> wrote: >>> >>>> Thank You. Could you please confirm whether the VCL config I have >>>> shared if fine? >>>> >>>> On Thu, Dec 19, 2019 at 5:46 PM Guillaume Quintard < >>>> guillaume at varnish-software.com> wrote: >>>> >>>>> check the backend transaction, and understand why your origin is >>>>> sending back a 403. This is due to your backend configuration and I can't >>>>> help you with that. >>>>> >>>>> -- >>>>> Guillaume Quintard >>>>> >>>>> >>>>> On Thu, Dec 19, 2019 at 1:03 PM Varun Nath >>>>> wrote: >>>>> >>>>>> This is the VCL file am using: >>>>>> >>>>>> Note: Host is not a valid one >>>>>> >>>>>> vcl 4.0; >>>>>> import std; >>>>>> import directors; >>>>>> >>>>>> # Default backend definition. Set this to point to your content >>>>>> server. >>>>>> backend default { >>>>>> .host = "remote.host.com"; >>>>>> .port = "80"; >>>>>> } >>>>>> >>>>>> sub vcl_init { >>>>>> new vdir = directors.round_robin(); >>>>>> vdir.add_backend(default); >>>>>> } >>>>>> >>>>>> sub vcl_recv { >>>>>> unset req.http.proxy; >>>>>> unset req.http.cookie; >>>>>> set req.http.x-host = "http://remote.host.com"; >>>>>> set req.http.x-clientip = client.ip; >>>>>> set req.http.x-serverip = server.ip; >>>>>> set req.http.x-localip = local.ip; >>>>>> set req.http.x-remoteip = remote.ip; >>>>>> unset req.http.X-Forwarded-For; >>>>>> set req.http.x-forwarded-for = client.ip; >>>>>> return(pass); >>>>>> } >>>>>> >>>>>> sub vcl_backend_response { >>>>>> # Happens after we have read the response headers from the >>>>>> backend. >>>>>> # >>>>>> # Here you clean the response headers, removing silly Set-Cookie >>>>>> headers >>>>>> # and other mistakes your backend does. >>>>>> } >>>>>> >>>>>> sub vcl_deliver { >>>>>> # Happens when we have all the pieces we need, and are about to >>>>>> send the >>>>>> # response to the client. >>>>>> } >>>>>> >>>>>> Please let me know if I need to add something more to make the remote >>>>>> host caching from localhost varnish >>>>>> >>>>>> On Thu, Dec 19, 2019 at 5:05 PM Varun Nath >>>>>> wrote: >>>>>> >>>>>>> Hello, >>>>>>> I got some response via CURL. But getting 403 Forbidden message. Log >>>>>>> is as below: >>>>>>> * BerespStatus 403* >>>>>>> * BerespReason Forbidden* >>>>>>> Also tried setting >>>>>>> >>>>>>> *remove req.http.X-Forwarded-For;set req.http.x-forwarded-for = >>>>>>> client.ip;* >>>>>>> But didnt work. >>>>>>> Could you please share your thoughts? >>>>>>> >>>>>>> On Thu, Dec 19, 2019 at 3:00 PM Varun Nath >>>>>>> wrote: >>>>>>> >>>>>>>> Thank You so much for your support. Will try this and let you know >>>>>>>> >>>>>>>> On Thu, Dec 19, 2019 at 2:49 PM Guillaume Quintard < >>>>>>>> guillaume at varnish-software.com> wrote: >>>>>>>> >>>>>>>>> The backend portion of the vcl only handles the TCP part of the >>>>>>>>> equation. If your backend expect a specific host, you need to either have >>>>>>>>> the client give it, or have varnish override it. >>>>>>>>> >>>>>>>>> Try curl -I localhost/your/file -H "host: >>>>>>>>> whatever.your.host.is.com" >>>>>>>>> >>>>>>>>> On Thu, Dec 19, 2019, 09:02 Varun Nath >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Yes, I tried with http://locahost, but that is expected, right? >>>>>>>>>> since the backend server is configured in VCL file. Sorry if I am wrong >>>>>>>>>> >>>>>>>>>> On Thu, Dec 19, 2019 at 2:29 PM Guillaume Quintard < >>>>>>>>>> guillaume at varnish-software.com> wrote: >>>>>>>>>> >>>>>>>>>>> Note: please keep the mailing list CC'd >>>>>>>>>>> >>>>>>>>>>> Check the host header you are sending to the backend. Usually, a >>>>>>>>>>> 404 arises because the URL and/or the host wrong. >>>>>>>>>>> >>>>>>>>>>> I'm thinking you possibly tested varnish requesting something >>>>>>>>>>> like http://localhost/your/test/file, but your backend isn't >>>>>>>>>>> configured to reply to the "localhost" host >>>>>>>>>>> >>>>>>>>>>> On Thu, Dec 19, 2019, 08:55 Varun Nath >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Thanks again for the quick response. >>>>>>>>>>>> I have checked the log and getting the expected IP address in *BackendStart >>>>>>>>>>>> *parameter. Sorry I didnt get what you mean by you need to >>>>>>>>>>>> change *the host header before sending the request to the >>>>>>>>>>>> backend. *Could you please elaborate a little bit? >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Dec 19, 2019 at 2:20 PM Guillaume Quintard < >>>>>>>>>>>> guillaume at varnish-software.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> Use varnishlog -b (with possibly -d) to see the backend >>>>>>>>>>>>> transactions and see what IP is being used (is it the right one). It could >>>>>>>>>>>>> also be that you need to change the host header before sending the request >>>>>>>>>>>>> to the backend. >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers, >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Dec 19, 2019, 08:39 Varun Nath >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hello, >>>>>>>>>>>>>> I need to connect to a remote host from local VCL config >>>>>>>>>>>>>> file. I have added the backend config as below >>>>>>>>>>>>>> >>>>>>>>>>>>>> backend default { >>>>>>>>>>>>>> .host = "remote.host.com"; >>>>>>>>>>>>>> } >>>>>>>>>>>>>> >>>>>>>>>>>>>> But am getting Page not available message. Do I miss >>>>>>>>>>>>>> something? >>>>>>>>>>>>>> Thanks in advance >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- >>>>>>>>>>>>>> VARUN NATH A V >>>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>>> varnish-misc mailing list >>>>>>>>>>>>>> varnish-misc at varnish-cache.org >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> VARUN NATH A V >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> VARUN NATH A V >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> VARUN NATH A V >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> VARUN NATH A V >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> VARUN NATH A V >>>>>> >>>>> >>>> >>>> -- >>>> VARUN NATH A V >>>> >>> >> >> -- >> VARUN NATH A V >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adrien.bigot at smile.fr Thu Dec 19 15:06:25 2019 From: adrien.bigot at smile.fr (Adrien Bigot) Date: Thu, 19 Dec 2019 16:06:25 +0100 Subject: Labels and purge scope Message-ID: <2c2d0371-4c3f-a44e-6843-523103b1f082@smile.fr> Hello, I plan to use labels on a Varnish stack in front of multiple backends. I have a question about ban with multiple labels. My main.vcl will route requests in function of Host header for example like this : if (req.http.host == "front.mydomain.com") { ????? return (vcl(l_varnish_fronts)); ??? } ??? if (req.http.host == "api.mydomain.com") { ????? return (vcl(l_varnish_api)); ??? } When we want to ban objects we do it with a HTTP BAN method on each varnish server. Varnish handle it the BAN request like that? in each label's VCL file : sub vcl_recv { ??????? if (req.method == "BAN") { ??????? if (req.http.Cache-BanExpression) { ??????????? ban(req.http.Cache-BanExpression); ??????? } ??????? return (synth(200, "Banned")); } My question is if I want to purge contents like this : curl -XBAN --header 'Cache-BanExpression: obj.http.Cache-Tags-TeamCompetitionReference ~ (^|,)6847(,|$)' ? http://myVarnishIP Will them be banned for all my Labels (if I include the proper vcl code in main.vcl), or do I have to make a request for each label like : curl -XBAN --header 'Host: front.mydomain.com' --header 'Cache-BanExpression: obj.http.Cache-Tags-TeamCompetitionReference ~ (^|,)6847(,|$)' ? http://myVarnishIP curl -XBAN --header 'Host: api.mydomain.com'??? --header 'Cache-BanExpression: obj.http.Cache-Tags-TeamCompetitionReference ~ (^|,)6847(,|$)' ? http://myVarnishIP In fact, I want to know if the cache is mutualized between labels ? If yes, is a solution exists to purge contents in all labels without sending multiple BAN requests ? Thanks in advance. Best regards, -- Adrien Bigot -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Dec 19 15:14:16 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 19 Dec 2019 15:14:16 +0000 Subject: Labels and purge scope In-Reply-To: <2c2d0371-4c3f-a44e-6843-523103b1f082@smile.fr> References: <2c2d0371-4c3f-a44e-6843-523103b1f082@smile.fr> Message-ID: Hi, Yes, labels are going to access the same cache , exactly like a reloaded vcl is going to use the same cache as the vcl it replaced. So, sending one request on any domain is going to work, but wouldn't it be more straightforward to just have the ban block in the main vcl, before you dispatch the request to a sub vcl? Cheers, On Thu, Dec 19, 2019, 15:07 Adrien Bigot wrote: > Hello, > > I plan to use labels on a Varnish stack in front of multiple backends. > I have a question about ban with multiple labels. > > My main.vcl will route requests in function of Host header for example > like this : > > if (req.http.host == "front.mydomain.com") { > return (vcl(l_varnish_fronts)); > } > if (req.http.host == "api.mydomain.com") { > return (vcl(l_varnish_api)); > } > > > When we want to ban objects we do it with a HTTP BAN method on each > varnish server. Varnish handle it the BAN request like that in each > label's VCL file : > sub vcl_recv { > if (req.method == "BAN") { > > if (req.http.Cache-BanExpression) { > ban(req.http.Cache-BanExpression); > } > return (synth(200, "Banned")); > } > > > My question is if I want to purge contents like this : > > curl -XBAN --header 'Cache-BanExpression: > obj.http.Cache-Tags-TeamCompetitionReference ~ (^|,)6847(,|$)' > http://myVarnishIP > > Will them be banned for all my Labels (if I include the proper vcl code in > main.vcl), or do I have to make a request for each label like : > curl -XBAN --header 'Host: front.mydomain.com' --header > 'Cache-BanExpression: obj.http.Cache-Tags-TeamCompetitionReference ~ > (^|,)6847(,|$)' http://myVarnishIP > curl -XBAN --header 'Host: api.mydomain.com' --header > 'Cache-BanExpression: obj.http.Cache-Tags-TeamCompetitionReference ~ > (^|,)6847(,|$)' http://myVarnishIP > > > In fact, I want to know if the cache is mutualized between labels ? If > yes, is a solution exists to purge contents in all labels without sending > multiple BAN requests ? > > Thanks in advance. > > Best regards, > > -- > > Adrien Bigot > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From adrien.bigot at smile.fr Thu Dec 19 15:44:51 2019 From: adrien.bigot at smile.fr (Adrien Bigot) Date: Thu, 19 Dec 2019 16:44:51 +0100 Subject: Labels and purge scope In-Reply-To: References: <2c2d0371-4c3f-a44e-6843-523103b1f082@smile.fr> Message-ID: <691621ea-1b07-92a5-5c66-536f9a200e4f@smile.fr> Thank you for this confirmation ! Now I know this is the same cache, I will move the ban block in the main VCL ! Thanks again ! Regards, -- Adrien Bigot Le 19/12/2019 ? 16:14, Guillaume Quintard a ?crit?: > Hi, > > Yes, labels are going to access the same cache , exactly like a > reloaded vcl is going to use the same cache as the vcl it replaced. > > So, sending one request on any domain is going to work, but wouldn't > it be more straightforward to just have the ban block in the main vcl, > before you dispatch the request to a sub vcl? > > Cheers, > > > On Thu, Dec 19, 2019, 15:07 Adrien Bigot > wrote: > > Hello, > > I plan to use labels on a Varnish stack in front of multiple backends. > I have a question about ban with multiple labels. > > My main.vcl will route requests in function of Host header for > example like this : > > if (req.http.host == "front.mydomain.com > ") { > ????? return (vcl(l_varnish_fronts)); > ??? } > ??? if (req.http.host == "api.mydomain.com > ") { > ????? return (vcl(l_varnish_api)); > ??? } > > > When we want to ban objects we do it with a HTTP BAN method on > each varnish server. Varnish handle it the BAN request like that? > in each label's VCL file : > sub vcl_recv { > ??????? if (req.method == "BAN") { > > ??????? if (req.http.Cache-BanExpression) { > ??????????? ban(req.http.Cache-BanExpression); > ??????? } > ??????? return (synth(200, "Banned")); > } > > > My question is if I want to purge contents like this : > > curl -XBAN --header 'Cache-BanExpression: > obj.http.Cache-Tags-TeamCompetitionReference ~ (^|,)6847(,|$)' ? > http://myVarnishIP > > Will them be banned for all my Labels (if I include the proper vcl > code in main.vcl), or do I have to make a request for each label > like : > curl -XBAN --header 'Host: front.mydomain.com > ' --header 'Cache-BanExpression: > obj.http.Cache-Tags-TeamCompetitionReference ~ (^|,)6847(,|$)' ? > http://myVarnishIP > curl -XBAN --header 'Host: api.mydomain.com > '??? --header 'Cache-BanExpression: > obj.http.Cache-Tags-TeamCompetitionReference ~ (^|,)6847(,|$)' ? > http://myVarnishIP > > > In fact, I want to know if the cache is mutualized between labels > ? If yes, is a solution exists to purge contents in all labels > without sending multiple BAN requests ? > > Thanks in advance. > > Best regards, > > -- > > Adrien Bigot > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mandys at gmail.com Thu Dec 26 06:32:06 2019 From: mandys at gmail.com (Maninder Singh) Date: Thu, 26 Dec 2019 12:02:06 +0530 Subject: Varnish 503 on POST requests once in a while Message-ID: Hi, I am facing a strange problem. I am running varnish 4.1.11 on port 80 which is behind a LB. Varnish then connects to apache 2.4 ( on port 8080 ) and also have php-fpm running. Now, everything works fine but once in a while I am seeing 503 response from varnish for a POST request. During this time there is no log in apache also ( neither access log nor error log ). Where does this request vanish ? How do I debug this issue ? Some more details: * I have varnishncsa turned on with following flags. ExecStart=/usr/bin/varnishncsa -q 'not (ReqURL ~ "\.(js|css|png|jpeg|jpg|ico|gif|swf|woff)" or ReqHeader:User-Agent ~ "HTTP-Monitor" or ReqHeader:User-Agent ~ "Monitoring/1.1")' -a -w /var/log/varnish/varnishncsa.log -D -f /etc/sysconfig/varnishncsa Here is the entry in varnish log during that time. X.X.X.X, LB.IP.0.0 - - [25/Dec/2019:00:26:40 -0600] 2629363 "POST http://mydomain.com/LetsCelebrate HTTP/1.1" 503 1288 " https://referringdomain.com/LetsCelebrate?hostingUrl=https%3A%2F%2Fxy.abc.com" "Mozilla/5.0 (Linux; Android 9; SM-A920F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.93 Mobile Safari/537.36" "mydomain.com" miss In .vcl file I have following set. .first_byte_timeout = 120s; Now, I am not logging varnish to apache requests ( which backend fetch ). As you can see above I am only logging client requests. Is their a way to log client requests like above but then also log backend fetch failures to another log file ? [ Running 2 instances of varnishncsa ? ] Any pointers ? Please let me know. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Dec 26 09:40:29 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 26 Dec 2019 09:40:29 +0000 Subject: Varnish 503 on POST requests once in a while In-Reply-To: References: Message-ID: Hi, Most probably, varnishncsa isn't going to give you the full picture, so let's go for the exhaustive way: varnishlog -q "BerespStatus eq 503" -w /whatever/file you can then read the logs with varnishlog -r /whatever/file (more logging pointers here: https://docs.varnish-software.com/tutorials/vsl-query/) If you can get your hands on a failed transaction, we should be able to tell you what went wrong. Most probably, you hit a timeout of some sort. Or it could be that the backend was declared sick, if you have probes. -- Guillaume Quintard On Thu, Dec 26, 2019 at 6:33 AM Maninder Singh wrote: > Hi, > > I am facing a strange problem. > > I am running varnish 4.1.11 on port 80 which is behind a LB. > > Varnish then connects to apache 2.4 ( on port 8080 ) and also have php-fpm > running. > > Now, everything works fine but once in a while I am seeing 503 response > from varnish for a POST request. During this time there is no log in apache > also ( neither access log nor error log ). > > Where does this request vanish ? > > How do I debug this issue ? > > Some more details: > > * I have varnishncsa turned on with following flags. > > ExecStart=/usr/bin/varnishncsa -q 'not (ReqURL ~ > "\.(js|css|png|jpeg|jpg|ico|gif|swf|woff)" or ReqHeader:User-Agent ~ > "HTTP-Monitor" or ReqHeader:User-Agent ~ "Monitoring/1.1")' -a -w > /var/log/varnish/varnishncsa.log -D -f /etc/sysconfig/varnishncsa > > Here is the entry in varnish log during that time. > > X.X.X.X, LB.IP.0.0 - - [25/Dec/2019:00:26:40 -0600] 2629363 "POST > http://mydomain.com/LetsCelebrate HTTP/1.1" 503 1288 " > https://referringdomain.com/LetsCelebrate?hostingUrl=https%3A%2F%2Fxy.abc.com" > "Mozilla/5.0 (Linux; Android 9; SM-A920F) AppleWebKit/537.36 (KHTML, like > Gecko) Chrome/79.0.3945.93 Mobile Safari/537.36" "mydomain.com" miss > > In .vcl file I have following set. > .first_byte_timeout = 120s; > > Now, I am not logging varnish to apache requests ( which backend fetch ). > As you can see above I am only logging client requests. > > Is their a way to log client requests like above but then also log backend > fetch failures to another log file ? > [ Running 2 instances of varnishncsa ? ] > > Any pointers ? > > Please let me know. > > Thanks. > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mandys at gmail.com Thu Dec 26 09:51:19 2019 From: mandys at gmail.com (Maninder Singh) Date: Thu, 26 Dec 2019 15:21:19 +0530 Subject: Varnish 503 on POST requests once in a while In-Reply-To: References: Message-ID: Okay. Let me set it up and get back to you once i see something in that log. It happens only once/twice a day so could be a while before i get back. On Thu, 26 Dec, 2019, 3:10 pm Guillaume Quintard, < guillaume at varnish-software.com> wrote: > Hi, > > Most probably, varnishncsa isn't going to give you the full picture, so > let's go for the exhaustive way: > > varnishlog -q "BerespStatus eq 503" -w /whatever/file > > > you can then read the logs with > > varnishlog -r /whatever/file > > > (more logging pointers here: > https://docs.varnish-software.com/tutorials/vsl-query/) > > If you can get your hands on a failed transaction, we should be able to > tell you what went wrong. Most probably, you hit a timeout of some sort. Or > it could be that the backend was declared sick, if you have probes. > > -- > Guillaume Quintard > > > On Thu, Dec 26, 2019 at 6:33 AM Maninder Singh wrote: > >> Hi, >> >> I am facing a strange problem. >> >> I am running varnish 4.1.11 on port 80 which is behind a LB. >> >> Varnish then connects to apache 2.4 ( on port 8080 ) and also have >> php-fpm running. >> >> Now, everything works fine but once in a while I am seeing 503 response >> from varnish for a POST request. During this time there is no log in apache >> also ( neither access log nor error log ). >> >> Where does this request vanish ? >> >> How do I debug this issue ? >> >> Some more details: >> >> * I have varnishncsa turned on with following flags. >> >> ExecStart=/usr/bin/varnishncsa -q 'not (ReqURL ~ >> "\.(js|css|png|jpeg|jpg|ico|gif|swf|woff)" or ReqHeader:User-Agent ~ >> "HTTP-Monitor" or ReqHeader:User-Agent ~ "Monitoring/1.1")' -a -w >> /var/log/varnish/varnishncsa.log -D -f /etc/sysconfig/varnishncsa >> >> Here is the entry in varnish log during that time. >> >> X.X.X.X, LB.IP.0.0 - - [25/Dec/2019:00:26:40 -0600] 2629363 "POST >> http://mydomain.com/LetsCelebrate HTTP/1.1" 503 1288 " >> https://referringdomain.com/LetsCelebrate?hostingUrl=https%3A%2F%2Fxy.abc.com" >> "Mozilla/5.0 (Linux; Android 9; SM-A920F) AppleWebKit/537.36 (KHTML, like >> Gecko) Chrome/79.0.3945.93 Mobile Safari/537.36" "mydomain.com" miss >> >> In .vcl file I have following set. >> .first_byte_timeout = 120s; >> >> Now, I am not logging varnish to apache requests ( which backend fetch ). >> As you can see above I am only logging client requests. >> >> Is their a way to log client requests like above but then also log >> backend fetch failures to another log file ? >> [ Running 2 instances of varnishncsa ? ] >> >> Any pointers ? >> >> Please let me know. >> >> Thanks. >> >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mandys at gmail.com Thu Dec 26 11:57:10 2019 From: mandys at gmail.com (Maninder Singh) Date: Thu, 26 Dec 2019 17:27:10 +0530 Subject: Varnish 503 on POST requests once in a while In-Reply-To: References: Message-ID: Got one in the log. varnishlog -r /var/log/varnish/503.log * << BeReq >> 21993384 - Begin bereq 21993383 pass - Timestamp Start: 1577360704.829258 0.000000 0.000000 - BereqMethod POST - BereqURL /corp/contact.php?cf=exitpopup - BereqProtocol HTTP/1.1 - BereqHeader User-Agent: Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko - BereqHeader Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, */* - BereqHeader SSLClientCertStatus: NoClientCert - BereqHeader X-Forwarded-Proto: https - BereqHeader Content-Type: application/x-www-form-urlencoded - BereqHeader SSLClientCipher: SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA, version=TLSv1, bits=256 - BereqHeader Host: www.mydomain.com - BereqHeader Referer: http://mydomain.com/solutions/enterprise.php - BereqHeader SSLSessionID: 3C119EF3FE70318789F82E9B583581C876B4DDE520C0182C33CD6AF1AA46E63A - BereqHeader X-Cluster-Client-Ip: X.X.X.X - BereqHeader Cookie: X-Mapping-fjhppofk=E8F494E9A630955F1975EEDF11EBC4D7 - BereqHeader X-Forwarded-Port: 443 - BereqHeader Accept-Encoding: gzip, deflate - BereqHeader Content-Length: 1366 - BereqHeader X-Forwarded-For: X.X.X.X, 10.187.187.36 - BereqHeader browser: other - BereqHeader serverIp: 10.208.224.192 - BereqHeader X-Varnish: 21993384 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 26 reload_2019-12-18T054253.default 127.0.0.1 8080 127.0.0.1 59998 - BackendStart 127.0.0.1 8080 - Timestamp Bereq: 1577360705.664932 0.835674 0.835674 - FetchError http first read error: EOF - BackendClose 26 reload_2019-12-18T054253.default - Timestamp Beresp: 1577360705.665020 0.835763 0.000089 - Timestamp Error: 1577360705.665028 0.835770 0.000008 - BerespProtocol HTTP/1.1 - BerespStatus 503 - BerespReason Service Unavailable - BerespReason Backend fetch failed - BerespHeader Date: Thu, 26 Dec 2019 11:45:05 GMT - BerespHeader Server: Varnish - VCL_call BACKEND_ERROR - BerespHeader Content-Type: text/html; charset=utf-8 - BerespHeader Retry-After: 5 - VCL_return deliver - Storage malloc Transient - ObjProtocol HTTP/1.1 - ObjStatus 503 - ObjReason Backend fetch failed - ObjHeader Date: Thu, 26 Dec 2019 11:45:05 GMT - ObjHeader Server: Varnish - ObjHeader Content-Type: text/html; charset=utf-8 - ObjHeader Retry-After: 5 - Length 1288 - BereqAcct 849 1366 2215 0 0 0 - End On Thu, 26 Dec 2019 at 15:21, Maninder Singh wrote: > Okay. Let me set it up and get back to you once i see something in that > log. It happens only once/twice a day so could be a while before i get > back. > > On Thu, 26 Dec, 2019, 3:10 pm Guillaume Quintard, < > guillaume at varnish-software.com> wrote: > >> Hi, >> >> Most probably, varnishncsa isn't going to give you the full picture, so >> let's go for the exhaustive way: >> >> varnishlog -q "BerespStatus eq 503" -w /whatever/file >> >> >> you can then read the logs with >> >> varnishlog -r /whatever/file >> >> >> (more logging pointers here: >> https://docs.varnish-software.com/tutorials/vsl-query/) >> >> If you can get your hands on a failed transaction, we should be able to >> tell you what went wrong. Most probably, you hit a timeout of some sort. Or >> it could be that the backend was declared sick, if you have probes. >> >> -- >> Guillaume Quintard >> >> >> On Thu, Dec 26, 2019 at 6:33 AM Maninder Singh wrote: >> >>> Hi, >>> >>> I am facing a strange problem. >>> >>> I am running varnish 4.1.11 on port 80 which is behind a LB. >>> >>> Varnish then connects to apache 2.4 ( on port 8080 ) and also have >>> php-fpm running. >>> >>> Now, everything works fine but once in a while I am seeing 503 response >>> from varnish for a POST request. During this time there is no log in apache >>> also ( neither access log nor error log ). >>> >>> Where does this request vanish ? >>> >>> How do I debug this issue ? >>> >>> Some more details: >>> >>> * I have varnishncsa turned on with following flags. >>> >>> ExecStart=/usr/bin/varnishncsa -q 'not (ReqURL ~ >>> "\.(js|css|png|jpeg|jpg|ico|gif|swf|woff)" or ReqHeader:User-Agent ~ >>> "HTTP-Monitor" or ReqHeader:User-Agent ~ "Monitoring/1.1")' -a -w >>> /var/log/varnish/varnishncsa.log -D -f /etc/sysconfig/varnishncsa >>> >>> Here is the entry in varnish log during that time. >>> >>> X.X.X.X, LB.IP.0.0 - - [25/Dec/2019:00:26:40 -0600] 2629363 "POST >>> http://mydomain.com/LetsCelebrate HTTP/1.1" 503 1288 " >>> https://referringdomain.com/LetsCelebrate?hostingUrl=https%3A%2F%2Fxy.abc.com" >>> "Mozilla/5.0 (Linux; Android 9; SM-A920F) AppleWebKit/537.36 (KHTML, like >>> Gecko) Chrome/79.0.3945.93 Mobile Safari/537.36" "mydomain.com" miss >>> >>> In .vcl file I have following set. >>> .first_byte_timeout = 120s; >>> >>> Now, I am not logging varnish to apache requests ( which backend fetch ). >>> As you can see above I am only logging client requests. >>> >>> Is their a way to log client requests like above but then also log >>> backend fetch failures to another log file ? >>> [ Running 2 instances of varnishncsa ? ] >>> >>> Any pointers ? >>> >>> Please let me know. >>> >>> Thanks. >>> >>> _______________________________________________ >>> varnish-misc mailing list >>> varnish-misc at varnish-cache.org >>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mandys at gmail.com Thu Dec 26 15:12:41 2019 From: mandys at gmail.com (Maninder Singh) Date: Thu, 26 Dec 2019 20:42:41 +0530 Subject: Varnish 503 on POST requests once in a while In-Reply-To: References: Message-ID: Another one. This one is diff error. * << BeReq >> 19327398 - Begin bereq 19327397 pass - Timestamp Start: 1577370022.344818 0.000000 0.000000 - BereqMethod POST - BereqURL /LetsCelebrate - BereqProtocol HTTP/1.1 - BereqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36 - BereqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 - BereqHeader Cache-Control: max-age=0 - BereqHeader SSLClientCertStatus: NoClientCert - BereqHeader X-Forwarded-Proto: https - BereqHeader Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryBBVOJciq1v4rbZs9 - BereqHeader Accept-Language: en-GB,en-US;q=0.9,en;q=0.8 - BereqHeader SSLClientCipher: SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384, version=TLSv1.2, bits=256 - BereqHeader Host: mydomain.com - BereqHeader Referer: https://mydomain.com/LetsCelebrate - BereqHeader SSLSessionID: 97258494A3DD008A855B552F701E9F95C25CDC7B1CF5FD63D382ED09391585C7 - BereqHeader Origin: https://mydomain.com - BereqHeader X-Cluster-Client-Ip: X.X.X.X - BereqHeader X-Forwarded-Port: 443 - BereqHeader Upgrade-Insecure-Requests: 1 - BereqHeader Sec-Fetch-User: ?1 - BereqHeader Sec-Fetch-Site: same-origin - BereqHeader Sec-Fetch-Mode: nested-navigate - BereqHeader Accept-Encoding: gzip, deflate, br - BereqHeader Content-Length: 2076 - BereqHeader X-Forwarded-For: X.X.X.X, 10.187.187.36 - BereqHeader browser: other - BereqHeader serverIp: 10.208.225.235 - BereqHeader X-Varnish: 19327398 - VCL_call BACKEND_FETCH - VCL_return fetch - BackendOpen 23 reload_2019-12-18T054238.default 127.0.0.1 8080 127.0.0.1 39568 - BackendStart 127.0.0.1 8080 - FetchError req.body read error: 11 (Resource temporarily unavailable) - FetchError backend write error: 11 (Resource temporarily unavailable) - Timestamp Bereq: 1577370027.344908 5.000090 5.000090 - BackendClose 23 reload_2019-12-18T054238.default - Timestamp Beresp: 1577370027.345014 5.000196 0.000105 - Timestamp Error: 1577370027.345025 5.000206 0.000011 - BerespProtocol HTTP/1.1 - BerespStatus 503 - BerespReason Service Unavailable - BerespReason Backend fetch failed - BerespHeader Date: Thu, 26 Dec 2019 14:20:27 GMT - BerespHeader Server: Varnish - VCL_call BACKEND_ERROR - BerespHeader Content-Type: text/html; charset=utf-8 - BerespHeader Retry-After: 5 - VCL_return deliver - Storage malloc Transient - ObjProtocol HTTP/1.1 - ObjStatus 503 - ObjReason Backend fetch failed - ObjHeader Date: Thu, 26 Dec 2019 14:20:27 GMT - ObjHeader Server: Varnish - ObjHeader Content-Type: text/html; charset=utf-8 - ObjHeader Retry-After: 5 - Length 1288 - BereqAcct 1276 0 1276 0 0 0 - End On Thu, 26 Dec 2019 at 17:27, Maninder Singh wrote: > Got one in the log. > > varnishlog -r /var/log/varnish/503.log > * << BeReq >> 21993384 > - Begin bereq 21993383 pass > - Timestamp Start: 1577360704.829258 0.000000 0.000000 > - BereqMethod POST > - BereqURL /corp/contact.php?cf=exitpopup > - BereqProtocol HTTP/1.1 > - BereqHeader User-Agent: Mozilla/5.0 (Windows NT 6.1; Trident/7.0; > rv:11.0) like Gecko > - BereqHeader Accept: image/gif, image/x-xbitmap, image/jpeg, > image/pjpeg, application/x-shockwave-flash, */* > - BereqHeader SSLClientCertStatus: NoClientCert > - BereqHeader X-Forwarded-Proto: https > - BereqHeader Content-Type: application/x-www-form-urlencoded > - BereqHeader SSLClientCipher: SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA, > version=TLSv1, bits=256 > - BereqHeader Host: www.mydomain.com > - BereqHeader Referer: http://mydomain.com/solutions/enterprise.php > - BereqHeader SSLSessionID: > 3C119EF3FE70318789F82E9B583581C876B4DDE520C0182C33CD6AF1AA46E63A > - BereqHeader X-Cluster-Client-Ip: X.X.X.X > - BereqHeader Cookie: > X-Mapping-fjhppofk=E8F494E9A630955F1975EEDF11EBC4D7 > - BereqHeader X-Forwarded-Port: 443 > - BereqHeader Accept-Encoding: gzip, deflate > - BereqHeader Content-Length: 1366 > - BereqHeader X-Forwarded-For: X.X.X.X, 10.187.187.36 > - BereqHeader browser: other > - BereqHeader serverIp: 10.208.224.192 > - BereqHeader X-Varnish: 21993384 > - VCL_call BACKEND_FETCH > - VCL_return fetch > - BackendOpen 26 reload_2019-12-18T054253.default 127.0.0.1 8080 > 127.0.0.1 59998 > - BackendStart 127.0.0.1 8080 > - Timestamp Bereq: 1577360705.664932 0.835674 0.835674 > - FetchError http first read error: EOF > - BackendClose 26 reload_2019-12-18T054253.default > - Timestamp Beresp: 1577360705.665020 0.835763 0.000089 > - Timestamp Error: 1577360705.665028 0.835770 0.000008 > - BerespProtocol HTTP/1.1 > - BerespStatus 503 > - BerespReason Service Unavailable > - BerespReason Backend fetch failed > - BerespHeader Date: Thu, 26 Dec 2019 11:45:05 GMT > - BerespHeader Server: Varnish > - VCL_call BACKEND_ERROR > - BerespHeader Content-Type: text/html; charset=utf-8 > - BerespHeader Retry-After: 5 > - VCL_return deliver > - Storage malloc Transient > - ObjProtocol HTTP/1.1 > - ObjStatus 503 > - ObjReason Backend fetch failed > - ObjHeader Date: Thu, 26 Dec 2019 11:45:05 GMT > - ObjHeader Server: Varnish > - ObjHeader Content-Type: text/html; charset=utf-8 > - ObjHeader Retry-After: 5 > - Length 1288 > - BereqAcct 849 1366 2215 0 0 0 > - End > > On Thu, 26 Dec 2019 at 15:21, Maninder Singh wrote: > >> Okay. Let me set it up and get back to you once i see something in that >> log. It happens only once/twice a day so could be a while before i get >> back. >> >> On Thu, 26 Dec, 2019, 3:10 pm Guillaume Quintard, < >> guillaume at varnish-software.com> wrote: >> >>> Hi, >>> >>> Most probably, varnishncsa isn't going to give you the full picture, so >>> let's go for the exhaustive way: >>> >>> varnishlog -q "BerespStatus eq 503" -w /whatever/file >>> >>> >>> you can then read the logs with >>> >>> varnishlog -r /whatever/file >>> >>> >>> (more logging pointers here: >>> https://docs.varnish-software.com/tutorials/vsl-query/) >>> >>> If you can get your hands on a failed transaction, we should be able to >>> tell you what went wrong. Most probably, you hit a timeout of some sort. Or >>> it could be that the backend was declared sick, if you have probes. >>> >>> -- >>> Guillaume Quintard >>> >>> >>> On Thu, Dec 26, 2019 at 6:33 AM Maninder Singh wrote: >>> >>>> Hi, >>>> >>>> I am facing a strange problem. >>>> >>>> I am running varnish 4.1.11 on port 80 which is behind a LB. >>>> >>>> Varnish then connects to apache 2.4 ( on port 8080 ) and also have >>>> php-fpm running. >>>> >>>> Now, everything works fine but once in a while I am seeing 503 response >>>> from varnish for a POST request. During this time there is no log in apache >>>> also ( neither access log nor error log ). >>>> >>>> Where does this request vanish ? >>>> >>>> How do I debug this issue ? >>>> >>>> Some more details: >>>> >>>> * I have varnishncsa turned on with following flags. >>>> >>>> ExecStart=/usr/bin/varnishncsa -q 'not (ReqURL ~ >>>> "\.(js|css|png|jpeg|jpg|ico|gif|swf|woff)" or ReqHeader:User-Agent ~ >>>> "HTTP-Monitor" or ReqHeader:User-Agent ~ "Monitoring/1.1")' -a -w >>>> /var/log/varnish/varnishncsa.log -D -f /etc/sysconfig/varnishncsa >>>> >>>> Here is the entry in varnish log during that time. >>>> >>>> X.X.X.X, LB.IP.0.0 - - [25/Dec/2019:00:26:40 -0600] 2629363 "POST >>>> http://mydomain.com/LetsCelebrate HTTP/1.1" 503 1288 " >>>> https://referringdomain.com/LetsCelebrate?hostingUrl=https%3A%2F%2Fxy.abc.com" >>>> "Mozilla/5.0 (Linux; Android 9; SM-A920F) AppleWebKit/537.36 (KHTML, like >>>> Gecko) Chrome/79.0.3945.93 Mobile Safari/537.36" "mydomain.com" miss >>>> >>>> In .vcl file I have following set. >>>> .first_byte_timeout = 120s; >>>> >>>> Now, I am not logging varnish to apache requests ( which backend fetch >>>> ). >>>> As you can see above I am only logging client requests. >>>> >>>> Is their a way to log client requests like above but then also log >>>> backend fetch failures to another log file ? >>>> [ Running 2 instances of varnishncsa ? ] >>>> >>>> Any pointers ? >>>> >>>> Please let me know. >>>> >>>> Thanks. >>>> >>>> _______________________________________________ >>>> varnish-misc mailing list >>>> varnish-misc at varnish-cache.org >>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Thu Dec 26 17:14:43 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Thu, 26 Dec 2019 17:14:43 +0000 Subject: Varnish 503 on POST requests once in a while In-Reply-To: References: Message-ID: The first looks like the backend is just closing the door on you. The other one (Resource temporarily unavailable), because of the Timestamp:Beresp, is most probably a timeout (did you set anything to 5s?) -- Guillaume Quintard On Thu, Dec 26, 2019 at 3:12 PM Maninder Singh wrote: > Another one. > This one is diff error. > > * << BeReq >> 19327398 > - Begin bereq 19327397 pass > - Timestamp Start: 1577370022.344818 0.000000 0.000000 > - BereqMethod POST > - BereqURL /LetsCelebrate > - BereqProtocol HTTP/1.1 > - BereqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X > 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 > Safari/537.36 > - BereqHeader Accept: > text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 > - BereqHeader Cache-Control: max-age=0 > - BereqHeader SSLClientCertStatus: NoClientCert > - BereqHeader X-Forwarded-Proto: https > - BereqHeader Content-Type: multipart/form-data; > boundary=----WebKitFormBoundaryBBVOJciq1v4rbZs9 > - BereqHeader Accept-Language: en-GB,en-US;q=0.9,en;q=0.8 > - BereqHeader SSLClientCipher: SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384, > version=TLSv1.2, bits=256 > - BereqHeader Host: mydomain.com > - BereqHeader Referer: https://mydomain.com/LetsCelebrate > - BereqHeader SSLSessionID: > 97258494A3DD008A855B552F701E9F95C25CDC7B1CF5FD63D382ED09391585C7 > - BereqHeader Origin: https://mydomain.com > - BereqHeader X-Cluster-Client-Ip: X.X.X.X > - BereqHeader X-Forwarded-Port: 443 > - BereqHeader Upgrade-Insecure-Requests: 1 > - BereqHeader Sec-Fetch-User: ?1 > - BereqHeader Sec-Fetch-Site: same-origin > - BereqHeader Sec-Fetch-Mode: nested-navigate > - BereqHeader Accept-Encoding: gzip, deflate, br > - BereqHeader Content-Length: 2076 > - BereqHeader X-Forwarded-For: X.X.X.X, 10.187.187.36 > - BereqHeader browser: other > - BereqHeader serverIp: 10.208.225.235 > - BereqHeader X-Varnish: 19327398 > - VCL_call BACKEND_FETCH > - VCL_return fetch > - BackendOpen 23 reload_2019-12-18T054238.default 127.0.0.1 8080 > 127.0.0.1 39568 > - BackendStart 127.0.0.1 8080 > - FetchError req.body read error: 11 (Resource temporarily > unavailable) > - FetchError backend write error: 11 (Resource temporarily > unavailable) > - Timestamp Bereq: 1577370027.344908 5.000090 5.000090 > - BackendClose 23 reload_2019-12-18T054238.default > - Timestamp Beresp: 1577370027.345014 5.000196 0.000105 > - Timestamp Error: 1577370027.345025 5.000206 0.000011 > - BerespProtocol HTTP/1.1 > - BerespStatus 503 > - BerespReason Service Unavailable > - BerespReason Backend fetch failed > - BerespHeader Date: Thu, 26 Dec 2019 14:20:27 GMT > - BerespHeader Server: Varnish > - VCL_call BACKEND_ERROR > - BerespHeader Content-Type: text/html; charset=utf-8 > - BerespHeader Retry-After: 5 > - VCL_return deliver > - Storage malloc Transient > - ObjProtocol HTTP/1.1 > - ObjStatus 503 > - ObjReason Backend fetch failed > - ObjHeader Date: Thu, 26 Dec 2019 14:20:27 GMT > - ObjHeader Server: Varnish > - ObjHeader Content-Type: text/html; charset=utf-8 > - ObjHeader Retry-After: 5 > - Length 1288 > - BereqAcct 1276 0 1276 0 0 0 > - End > > On Thu, 26 Dec 2019 at 17:27, Maninder Singh wrote: > >> Got one in the log. >> >> varnishlog -r /var/log/varnish/503.log >> * << BeReq >> 21993384 >> - Begin bereq 21993383 pass >> - Timestamp Start: 1577360704.829258 0.000000 0.000000 >> - BereqMethod POST >> - BereqURL /corp/contact.php?cf=exitpopup >> - BereqProtocol HTTP/1.1 >> - BereqHeader User-Agent: Mozilla/5.0 (Windows NT 6.1; Trident/7.0; >> rv:11.0) like Gecko >> - BereqHeader Accept: image/gif, image/x-xbitmap, image/jpeg, >> image/pjpeg, application/x-shockwave-flash, */* >> - BereqHeader SSLClientCertStatus: NoClientCert >> - BereqHeader X-Forwarded-Proto: https >> - BereqHeader Content-Type: application/x-www-form-urlencoded >> - BereqHeader SSLClientCipher: SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA, >> version=TLSv1, bits=256 >> - BereqHeader Host: www.mydomain.com >> - BereqHeader Referer: http://mydomain.com/solutions/enterprise.php >> - BereqHeader SSLSessionID: >> 3C119EF3FE70318789F82E9B583581C876B4DDE520C0182C33CD6AF1AA46E63A >> - BereqHeader X-Cluster-Client-Ip: X.X.X.X >> - BereqHeader Cookie: >> X-Mapping-fjhppofk=E8F494E9A630955F1975EEDF11EBC4D7 >> - BereqHeader X-Forwarded-Port: 443 >> - BereqHeader Accept-Encoding: gzip, deflate >> - BereqHeader Content-Length: 1366 >> - BereqHeader X-Forwarded-For: X.X.X.X, 10.187.187.36 >> - BereqHeader browser: other >> - BereqHeader serverIp: 10.208.224.192 >> - BereqHeader X-Varnish: 21993384 >> - VCL_call BACKEND_FETCH >> - VCL_return fetch >> - BackendOpen 26 reload_2019-12-18T054253.default 127.0.0.1 8080 >> 127.0.0.1 59998 >> - BackendStart 127.0.0.1 8080 >> - Timestamp Bereq: 1577360705.664932 0.835674 0.835674 >> - FetchError http first read error: EOF >> - BackendClose 26 reload_2019-12-18T054253.default >> - Timestamp Beresp: 1577360705.665020 0.835763 0.000089 >> - Timestamp Error: 1577360705.665028 0.835770 0.000008 >> - BerespProtocol HTTP/1.1 >> - BerespStatus 503 >> - BerespReason Service Unavailable >> - BerespReason Backend fetch failed >> - BerespHeader Date: Thu, 26 Dec 2019 11:45:05 GMT >> - BerespHeader Server: Varnish >> - VCL_call BACKEND_ERROR >> - BerespHeader Content-Type: text/html; charset=utf-8 >> - BerespHeader Retry-After: 5 >> - VCL_return deliver >> - Storage malloc Transient >> - ObjProtocol HTTP/1.1 >> - ObjStatus 503 >> - ObjReason Backend fetch failed >> - ObjHeader Date: Thu, 26 Dec 2019 11:45:05 GMT >> - ObjHeader Server: Varnish >> - ObjHeader Content-Type: text/html; charset=utf-8 >> - ObjHeader Retry-After: 5 >> - Length 1288 >> - BereqAcct 849 1366 2215 0 0 0 >> - End >> >> On Thu, 26 Dec 2019 at 15:21, Maninder Singh wrote: >> >>> Okay. Let me set it up and get back to you once i see something in that >>> log. It happens only once/twice a day so could be a while before i get >>> back. >>> >>> On Thu, 26 Dec, 2019, 3:10 pm Guillaume Quintard, < >>> guillaume at varnish-software.com> wrote: >>> >>>> Hi, >>>> >>>> Most probably, varnishncsa isn't going to give you the full picture, so >>>> let's go for the exhaustive way: >>>> >>>> varnishlog -q "BerespStatus eq 503" -w /whatever/file >>>> >>>> >>>> you can then read the logs with >>>> >>>> varnishlog -r /whatever/file >>>> >>>> >>>> (more logging pointers here: >>>> https://docs.varnish-software.com/tutorials/vsl-query/) >>>> >>>> If you can get your hands on a failed transaction, we should be able to >>>> tell you what went wrong. Most probably, you hit a timeout of some sort. Or >>>> it could be that the backend was declared sick, if you have probes. >>>> >>>> -- >>>> Guillaume Quintard >>>> >>>> >>>> On Thu, Dec 26, 2019 at 6:33 AM Maninder Singh >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I am facing a strange problem. >>>>> >>>>> I am running varnish 4.1.11 on port 80 which is behind a LB. >>>>> >>>>> Varnish then connects to apache 2.4 ( on port 8080 ) and also have >>>>> php-fpm running. >>>>> >>>>> Now, everything works fine but once in a while I am seeing 503 >>>>> response from varnish for a POST request. During this time there is no log >>>>> in apache also ( neither access log nor error log ). >>>>> >>>>> Where does this request vanish ? >>>>> >>>>> How do I debug this issue ? >>>>> >>>>> Some more details: >>>>> >>>>> * I have varnishncsa turned on with following flags. >>>>> >>>>> ExecStart=/usr/bin/varnishncsa -q 'not (ReqURL ~ >>>>> "\.(js|css|png|jpeg|jpg|ico|gif|swf|woff)" or ReqHeader:User-Agent ~ >>>>> "HTTP-Monitor" or ReqHeader:User-Agent ~ "Monitoring/1.1")' -a -w >>>>> /var/log/varnish/varnishncsa.log -D -f /etc/sysconfig/varnishncsa >>>>> >>>>> Here is the entry in varnish log during that time. >>>>> >>>>> X.X.X.X, LB.IP.0.0 - - [25/Dec/2019:00:26:40 -0600] 2629363 "POST >>>>> http://mydomain.com/LetsCelebrate HTTP/1.1" 503 1288 " >>>>> https://referringdomain.com/LetsCelebrate?hostingUrl=https%3A%2F%2Fxy.abc.com" >>>>> "Mozilla/5.0 (Linux; Android 9; SM-A920F) AppleWebKit/537.36 (KHTML, like >>>>> Gecko) Chrome/79.0.3945.93 Mobile Safari/537.36" "mydomain.com" miss >>>>> >>>>> In .vcl file I have following set. >>>>> .first_byte_timeout = 120s; >>>>> >>>>> Now, I am not logging varnish to apache requests ( which backend fetch >>>>> ). >>>>> As you can see above I am only logging client requests. >>>>> >>>>> Is their a way to log client requests like above but then also log >>>>> backend fetch failures to another log file ? >>>>> [ Running 2 instances of varnishncsa ? ] >>>>> >>>>> Any pointers ? >>>>> >>>>> Please let me know. >>>>> >>>>> Thanks. >>>>> >>>>> _______________________________________________ >>>>> varnish-misc mailing list >>>>> varnish-misc at varnish-cache.org >>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mandys at gmail.com Fri Dec 27 00:57:56 2019 From: mandys at gmail.com (Maninder Singh) Date: Fri, 27 Dec 2019 06:27:56 +0530 Subject: Varnish 503 on POST requests once in a while In-Reply-To: References: Message-ID: I only have php-fpm slow logging for requests greater than 5secs. Otherwise it should be all defaults for varnish/apache/fpm. Does varnish or apache have anything defaulting to 5secs? Otherwise fpm might be closing the connection for logging a slow request ? Strange? On Thu, 26 Dec, 2019, 10:44 pm Guillaume Quintard, < guillaume at varnish-software.com> wrote: > The first looks like the backend is just closing the door on you. The > other one (Resource temporarily unavailable), because of the > Timestamp:Beresp, is most probably a timeout (did you set anything to 5s?) > > -- > Guillaume Quintard > > > On Thu, Dec 26, 2019 at 3:12 PM Maninder Singh wrote: > >> Another one. >> This one is diff error. >> >> * << BeReq >> 19327398 >> - Begin bereq 19327397 pass >> - Timestamp Start: 1577370022.344818 0.000000 0.000000 >> - BereqMethod POST >> - BereqURL /LetsCelebrate >> - BereqProtocol HTTP/1.1 >> - BereqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X >> 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 >> Safari/537.36 >> - BereqHeader Accept: >> text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 >> - BereqHeader Cache-Control: max-age=0 >> - BereqHeader SSLClientCertStatus: NoClientCert >> - BereqHeader X-Forwarded-Proto: https >> - BereqHeader Content-Type: multipart/form-data; >> boundary=----WebKitFormBoundaryBBVOJciq1v4rbZs9 >> - BereqHeader Accept-Language: en-GB,en-US;q=0.9,en;q=0.8 >> - BereqHeader SSLClientCipher: >> SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384, version=TLSv1.2, bits=256 >> - BereqHeader Host: mydomain.com >> - BereqHeader Referer: https://mydomain.com/LetsCelebrate >> - BereqHeader SSLSessionID: >> 97258494A3DD008A855B552F701E9F95C25CDC7B1CF5FD63D382ED09391585C7 >> - BereqHeader Origin: https://mydomain.com >> - BereqHeader X-Cluster-Client-Ip: X.X.X.X >> - BereqHeader X-Forwarded-Port: 443 >> - BereqHeader Upgrade-Insecure-Requests: 1 >> - BereqHeader Sec-Fetch-User: ?1 >> - BereqHeader Sec-Fetch-Site: same-origin >> - BereqHeader Sec-Fetch-Mode: nested-navigate >> - BereqHeader Accept-Encoding: gzip, deflate, br >> - BereqHeader Content-Length: 2076 >> - BereqHeader X-Forwarded-For: X.X.X.X, 10.187.187.36 >> - BereqHeader browser: other >> - BereqHeader serverIp: 10.208.225.235 >> - BereqHeader X-Varnish: 19327398 >> - VCL_call BACKEND_FETCH >> - VCL_return fetch >> - BackendOpen 23 reload_2019-12-18T054238.default 127.0.0.1 8080 >> 127.0.0.1 39568 >> - BackendStart 127.0.0.1 8080 >> - FetchError req.body read error: 11 (Resource temporarily >> unavailable) >> - FetchError backend write error: 11 (Resource temporarily >> unavailable) >> - Timestamp Bereq: 1577370027.344908 5.000090 5.000090 >> - BackendClose 23 reload_2019-12-18T054238.default >> - Timestamp Beresp: 1577370027.345014 5.000196 0.000105 >> - Timestamp Error: 1577370027.345025 5.000206 0.000011 >> - BerespProtocol HTTP/1.1 >> - BerespStatus 503 >> - BerespReason Service Unavailable >> - BerespReason Backend fetch failed >> - BerespHeader Date: Thu, 26 Dec 2019 14:20:27 GMT >> - BerespHeader Server: Varnish >> - VCL_call BACKEND_ERROR >> - BerespHeader Content-Type: text/html; charset=utf-8 >> - BerespHeader Retry-After: 5 >> - VCL_return deliver >> - Storage malloc Transient >> - ObjProtocol HTTP/1.1 >> - ObjStatus 503 >> - ObjReason Backend fetch failed >> - ObjHeader Date: Thu, 26 Dec 2019 14:20:27 GMT >> - ObjHeader Server: Varnish >> - ObjHeader Content-Type: text/html; charset=utf-8 >> - ObjHeader Retry-After: 5 >> - Length 1288 >> - BereqAcct 1276 0 1276 0 0 0 >> - End >> >> On Thu, 26 Dec 2019 at 17:27, Maninder Singh wrote: >> >>> Got one in the log. >>> >>> varnishlog -r /var/log/varnish/503.log >>> * << BeReq >> 21993384 >>> - Begin bereq 21993383 pass >>> - Timestamp Start: 1577360704.829258 0.000000 0.000000 >>> - BereqMethod POST >>> - BereqURL /corp/contact.php?cf=exitpopup >>> - BereqProtocol HTTP/1.1 >>> - BereqHeader User-Agent: Mozilla/5.0 (Windows NT 6.1; Trident/7.0; >>> rv:11.0) like Gecko >>> - BereqHeader Accept: image/gif, image/x-xbitmap, image/jpeg, >>> image/pjpeg, application/x-shockwave-flash, */* >>> - BereqHeader SSLClientCertStatus: NoClientCert >>> - BereqHeader X-Forwarded-Proto: https >>> - BereqHeader Content-Type: application/x-www-form-urlencoded >>> - BereqHeader SSLClientCipher: SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA, >>> version=TLSv1, bits=256 >>> - BereqHeader Host: www.mydomain.com >>> - BereqHeader Referer: http://mydomain.com/solutions/enterprise.php >>> - BereqHeader SSLSessionID: >>> 3C119EF3FE70318789F82E9B583581C876B4DDE520C0182C33CD6AF1AA46E63A >>> - BereqHeader X-Cluster-Client-Ip: X.X.X.X >>> - BereqHeader Cookie: >>> X-Mapping-fjhppofk=E8F494E9A630955F1975EEDF11EBC4D7 >>> - BereqHeader X-Forwarded-Port: 443 >>> - BereqHeader Accept-Encoding: gzip, deflate >>> - BereqHeader Content-Length: 1366 >>> - BereqHeader X-Forwarded-For: X.X.X.X, 10.187.187.36 >>> - BereqHeader browser: other >>> - BereqHeader serverIp: 10.208.224.192 >>> - BereqHeader X-Varnish: 21993384 >>> - VCL_call BACKEND_FETCH >>> - VCL_return fetch >>> - BackendOpen 26 reload_2019-12-18T054253.default 127.0.0.1 8080 >>> 127.0.0.1 59998 >>> - BackendStart 127.0.0.1 8080 >>> - Timestamp Bereq: 1577360705.664932 0.835674 0.835674 >>> - FetchError http first read error: EOF >>> - BackendClose 26 reload_2019-12-18T054253.default >>> - Timestamp Beresp: 1577360705.665020 0.835763 0.000089 >>> - Timestamp Error: 1577360705.665028 0.835770 0.000008 >>> - BerespProtocol HTTP/1.1 >>> - BerespStatus 503 >>> - BerespReason Service Unavailable >>> - BerespReason Backend fetch failed >>> - BerespHeader Date: Thu, 26 Dec 2019 11:45:05 GMT >>> - BerespHeader Server: Varnish >>> - VCL_call BACKEND_ERROR >>> - BerespHeader Content-Type: text/html; charset=utf-8 >>> - BerespHeader Retry-After: 5 >>> - VCL_return deliver >>> - Storage malloc Transient >>> - ObjProtocol HTTP/1.1 >>> - ObjStatus 503 >>> - ObjReason Backend fetch failed >>> - ObjHeader Date: Thu, 26 Dec 2019 11:45:05 GMT >>> - ObjHeader Server: Varnish >>> - ObjHeader Content-Type: text/html; charset=utf-8 >>> - ObjHeader Retry-After: 5 >>> - Length 1288 >>> - BereqAcct 849 1366 2215 0 0 0 >>> - End >>> >>> On Thu, 26 Dec 2019 at 15:21, Maninder Singh wrote: >>> >>>> Okay. Let me set it up and get back to you once i see something in that >>>> log. It happens only once/twice a day so could be a while before i get >>>> back. >>>> >>>> On Thu, 26 Dec, 2019, 3:10 pm Guillaume Quintard, < >>>> guillaume at varnish-software.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> Most probably, varnishncsa isn't going to give you the full picture, >>>>> so let's go for the exhaustive way: >>>>> >>>>> varnishlog -q "BerespStatus eq 503" -w /whatever/file >>>>> >>>>> >>>>> you can then read the logs with >>>>> >>>>> varnishlog -r /whatever/file >>>>> >>>>> >>>>> (more logging pointers here: >>>>> https://docs.varnish-software.com/tutorials/vsl-query/) >>>>> >>>>> If you can get your hands on a failed transaction, we should be able >>>>> to tell you what went wrong. Most probably, you hit a timeout of some sort. >>>>> Or it could be that the backend was declared sick, if you have probes. >>>>> >>>>> -- >>>>> Guillaume Quintard >>>>> >>>>> >>>>> On Thu, Dec 26, 2019 at 6:33 AM Maninder Singh >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I am facing a strange problem. >>>>>> >>>>>> I am running varnish 4.1.11 on port 80 which is behind a LB. >>>>>> >>>>>> Varnish then connects to apache 2.4 ( on port 8080 ) and also have >>>>>> php-fpm running. >>>>>> >>>>>> Now, everything works fine but once in a while I am seeing 503 >>>>>> response from varnish for a POST request. During this time there is no log >>>>>> in apache also ( neither access log nor error log ). >>>>>> >>>>>> Where does this request vanish ? >>>>>> >>>>>> How do I debug this issue ? >>>>>> >>>>>> Some more details: >>>>>> >>>>>> * I have varnishncsa turned on with following flags. >>>>>> >>>>>> ExecStart=/usr/bin/varnishncsa -q 'not (ReqURL ~ >>>>>> "\.(js|css|png|jpeg|jpg|ico|gif|swf|woff)" or ReqHeader:User-Agent ~ >>>>>> "HTTP-Monitor" or ReqHeader:User-Agent ~ "Monitoring/1.1")' -a -w >>>>>> /var/log/varnish/varnishncsa.log -D -f /etc/sysconfig/varnishncsa >>>>>> >>>>>> Here is the entry in varnish log during that time. >>>>>> >>>>>> X.X.X.X, LB.IP.0.0 - - [25/Dec/2019:00:26:40 -0600] 2629363 "POST >>>>>> http://mydomain.com/LetsCelebrate HTTP/1.1" 503 1288 " >>>>>> https://referringdomain.com/LetsCelebrate?hostingUrl=https%3A%2F%2Fxy.abc.com" >>>>>> "Mozilla/5.0 (Linux; Android 9; SM-A920F) AppleWebKit/537.36 (KHTML, like >>>>>> Gecko) Chrome/79.0.3945.93 Mobile Safari/537.36" "mydomain.com" miss >>>>>> >>>>>> In .vcl file I have following set. >>>>>> .first_byte_timeout = 120s; >>>>>> >>>>>> Now, I am not logging varnish to apache requests ( which backend >>>>>> fetch ). >>>>>> As you can see above I am only logging client requests. >>>>>> >>>>>> Is their a way to log client requests like above but then also log >>>>>> backend fetch failures to another log file ? >>>>>> [ Running 2 instances of varnishncsa ? ] >>>>>> >>>>>> Any pointers ? >>>>>> >>>>>> Please let me know. >>>>>> >>>>>> Thanks. >>>>>> >>>>>> _______________________________________________ >>>>>> varnish-misc mailing list >>>>>> varnish-misc at varnish-cache.org >>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Fri Dec 27 07:45:17 2019 From: dridi at varni.sh (Dridi Boukelmoune) Date: Fri, 27 Dec 2019 07:45:17 +0000 Subject: Varnish 503 on POST requests once in a while In-Reply-To: References: Message-ID: On Fri, Dec 27, 2019 at 1:00 AM Maninder Singh wrote: > > I only have php-fpm slow logging for requests greater than 5secs. Otherwise it should be all defaults for varnish/apache/fpm. > > Does varnish or apache have anything defaulting to 5secs? timeout_idle defaults to 5s > Otherwise fpm might be closing the connection for logging a slow request ? Strange? > > On Thu, 26 Dec, 2019, 10:44 pm Guillaume Quintard, wrote: >> >> The first looks like the backend is just closing the door on you. The other one (Resource temporarily unavailable), because of the Timestamp:Beresp, is most probably a timeout (did you set anything to 5s?) >> >> -- >> Guillaume Quintard >> >> >> On Thu, Dec 26, 2019 at 3:12 PM Maninder Singh wrote: >>> >>> Another one. >>> This one is diff error. >>> >>> * << BeReq >> 19327398 >>> - Begin bereq 19327397 pass >>> - Timestamp Start: 1577370022.344818 0.000000 0.000000 >>> - BereqMethod POST >>> - BereqURL /LetsCelebrate >>> - BereqProtocol HTTP/1.1 >>> - BereqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36 >>> - BereqHeader Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 >>> - BereqHeader Cache-Control: max-age=0 >>> - BereqHeader SSLClientCertStatus: NoClientCert >>> - BereqHeader X-Forwarded-Proto: https >>> - BereqHeader Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryBBVOJciq1v4rbZs9 >>> - BereqHeader Accept-Language: en-GB,en-US;q=0.9,en;q=0.8 >>> - BereqHeader SSLClientCipher: SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384, version=TLSv1.2, bits=256 >>> - BereqHeader Host: mydomain.com >>> - BereqHeader Referer: https://mydomain.com/LetsCelebrate >>> - BereqHeader SSLSessionID: 97258494A3DD008A855B552F701E9F95C25CDC7B1CF5FD63D382ED09391585C7 >>> - BereqHeader Origin: https://mydomain.com >>> - BereqHeader X-Cluster-Client-Ip: X.X.X.X >>> - BereqHeader X-Forwarded-Port: 443 >>> - BereqHeader Upgrade-Insecure-Requests: 1 >>> - BereqHeader Sec-Fetch-User: ?1 >>> - BereqHeader Sec-Fetch-Site: same-origin >>> - BereqHeader Sec-Fetch-Mode: nested-navigate >>> - BereqHeader Accept-Encoding: gzip, deflate, br >>> - BereqHeader Content-Length: 2076 >>> - BereqHeader X-Forwarded-For: X.X.X.X, 10.187.187.36 >>> - BereqHeader browser: other >>> - BereqHeader serverIp: 10.208.225.235 >>> - BereqHeader X-Varnish: 19327398 >>> - VCL_call BACKEND_FETCH >>> - VCL_return fetch >>> - BackendOpen 23 reload_2019-12-18T054238.default 127.0.0.1 8080 127.0.0.1 39568 >>> - BackendStart 127.0.0.1 8080 >>> - FetchError req.body read error: 11 (Resource temporarily unavailable) >>> - FetchError backend write error: 11 (Resource temporarily unavailable) >>> - Timestamp Bereq: 1577370027.344908 5.000090 5.000090 >>> - BackendClose 23 reload_2019-12-18T054238.default >>> - Timestamp Beresp: 1577370027.345014 5.000196 0.000105 >>> - Timestamp Error: 1577370027.345025 5.000206 0.000011 >>> - BerespProtocol HTTP/1.1 >>> - BerespStatus 503 >>> - BerespReason Service Unavailable >>> - BerespReason Backend fetch failed >>> - BerespHeader Date: Thu, 26 Dec 2019 14:20:27 GMT >>> - BerespHeader Server: Varnish >>> - VCL_call BACKEND_ERROR >>> - BerespHeader Content-Type: text/html; charset=utf-8 >>> - BerespHeader Retry-After: 5 >>> - VCL_return deliver >>> - Storage malloc Transient >>> - ObjProtocol HTTP/1.1 >>> - ObjStatus 503 >>> - ObjReason Backend fetch failed >>> - ObjHeader Date: Thu, 26 Dec 2019 14:20:27 GMT >>> - ObjHeader Server: Varnish >>> - ObjHeader Content-Type: text/html; charset=utf-8 >>> - ObjHeader Retry-After: 5 >>> - Length 1288 >>> - BereqAcct 1276 0 1276 0 0 0 >>> - End >>> >>> On Thu, 26 Dec 2019 at 17:27, Maninder Singh wrote: >>>> >>>> Got one in the log. >>>> >>>> varnishlog -r /var/log/varnish/503.log >>>> * << BeReq >> 21993384 >>>> - Begin bereq 21993383 pass >>>> - Timestamp Start: 1577360704.829258 0.000000 0.000000 >>>> - BereqMethod POST >>>> - BereqURL /corp/contact.php?cf=exitpopup >>>> - BereqProtocol HTTP/1.1 >>>> - BereqHeader User-Agent: Mozilla/5.0 (Windows NT 6.1; Trident/7.0; rv:11.0) like Gecko >>>> - BereqHeader Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, */* >>>> - BereqHeader SSLClientCertStatus: NoClientCert >>>> - BereqHeader X-Forwarded-Proto: https >>>> - BereqHeader Content-Type: application/x-www-form-urlencoded >>>> - BereqHeader SSLClientCipher: SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA, version=TLSv1, bits=256 >>>> - BereqHeader Host: www.mydomain.com >>>> - BereqHeader Referer: http://mydomain.com/solutions/enterprise.php >>>> - BereqHeader SSLSessionID: 3C119EF3FE70318789F82E9B583581C876B4DDE520C0182C33CD6AF1AA46E63A >>>> - BereqHeader X-Cluster-Client-Ip: X.X.X.X >>>> - BereqHeader Cookie: X-Mapping-fjhppofk=E8F494E9A630955F1975EEDF11EBC4D7 >>>> - BereqHeader X-Forwarded-Port: 443 >>>> - BereqHeader Accept-Encoding: gzip, deflate >>>> - BereqHeader Content-Length: 1366 >>>> - BereqHeader X-Forwarded-For: X.X.X.X, 10.187.187.36 >>>> - BereqHeader browser: other >>>> - BereqHeader serverIp: 10.208.224.192 >>>> - BereqHeader X-Varnish: 21993384 >>>> - VCL_call BACKEND_FETCH >>>> - VCL_return fetch >>>> - BackendOpen 26 reload_2019-12-18T054253.default 127.0.0.1 8080 127.0.0.1 59998 >>>> - BackendStart 127.0.0.1 8080 >>>> - Timestamp Bereq: 1577360705.664932 0.835674 0.835674 >>>> - FetchError http first read error: EOF >>>> - BackendClose 26 reload_2019-12-18T054253.default >>>> - Timestamp Beresp: 1577360705.665020 0.835763 0.000089 >>>> - Timestamp Error: 1577360705.665028 0.835770 0.000008 >>>> - BerespProtocol HTTP/1.1 >>>> - BerespStatus 503 >>>> - BerespReason Service Unavailable >>>> - BerespReason Backend fetch failed >>>> - BerespHeader Date: Thu, 26 Dec 2019 11:45:05 GMT >>>> - BerespHeader Server: Varnish >>>> - VCL_call BACKEND_ERROR >>>> - BerespHeader Content-Type: text/html; charset=utf-8 >>>> - BerespHeader Retry-After: 5 >>>> - VCL_return deliver >>>> - Storage malloc Transient >>>> - ObjProtocol HTTP/1.1 >>>> - ObjStatus 503 >>>> - ObjReason Backend fetch failed >>>> - ObjHeader Date: Thu, 26 Dec 2019 11:45:05 GMT >>>> - ObjHeader Server: Varnish >>>> - ObjHeader Content-Type: text/html; charset=utf-8 >>>> - ObjHeader Retry-After: 5 >>>> - Length 1288 >>>> - BereqAcct 849 1366 2215 0 0 0 >>>> - End >>>> >>>> On Thu, 26 Dec 2019 at 15:21, Maninder Singh wrote: >>>>> >>>>> Okay. Let me set it up and get back to you once i see something in that log. It happens only once/twice a day so could be a while before i get back. >>>>> >>>>> On Thu, 26 Dec, 2019, 3:10 pm Guillaume Quintard, wrote: >>>>>> >>>>>> Hi, >>>>>> >>>>>> Most probably, varnishncsa isn't going to give you the full picture, so let's go for the exhaustive way: >>>>>> >>>>>> varnishlog -q "BerespStatus eq 503" -w /whatever/file >>>>>> >>>>>> >>>>>> you can then read the logs with >>>>>> >>>>>> varnishlog -r /whatever/file >>>>>> >>>>>> >>>>>> (more logging pointers here: https://docs.varnish-software.com/tutorials/vsl-query/) >>>>>> >>>>>> If you can get your hands on a failed transaction, we should be able to tell you what went wrong. Most probably, you hit a timeout of some sort. Or it could be that the backend was declared sick, if you have probes. >>>>>> >>>>>> -- >>>>>> Guillaume Quintard >>>>>> >>>>>> >>>>>> On Thu, Dec 26, 2019 at 6:33 AM Maninder Singh wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am facing a strange problem. >>>>>>> >>>>>>> I am running varnish 4.1.11 on port 80 which is behind a LB. >>>>>>> >>>>>>> Varnish then connects to apache 2.4 ( on port 8080 ) and also have php-fpm running. >>>>>>> >>>>>>> Now, everything works fine but once in a while I am seeing 503 response from varnish for a POST request. During this time there is no log in apache also ( neither access log nor error log ). >>>>>>> >>>>>>> Where does this request vanish ? >>>>>>> >>>>>>> How do I debug this issue ? >>>>>>> >>>>>>> Some more details: >>>>>>> >>>>>>> * I have varnishncsa turned on with following flags. >>>>>>> >>>>>>> ExecStart=/usr/bin/varnishncsa -q 'not (ReqURL ~ "\.(js|css|png|jpeg|jpg|ico|gif|swf|woff)" or ReqHeader:User-Agent ~ "HTTP-Monitor" or ReqHeader:User-Agent ~ "Monitoring/1.1")' -a -w /var/log/varnish/varnishncsa.log -D -f /etc/sysconfig/varnishncsa >>>>>>> >>>>>>> Here is the entry in varnish log during that time. >>>>>>> >>>>>>> X.X.X.X, LB.IP.0.0 - - [25/Dec/2019:00:26:40 -0600] 2629363 "POST http://mydomain.com/LetsCelebrate HTTP/1.1" 503 1288 "https://referringdomain.com/LetsCelebrate?hostingUrl=https%3A%2F%2Fxy.abc.com" "Mozilla/5.0 (Linux; Android 9; SM-A920F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.93 Mobile Safari/537.36" "mydomain.com" miss >>>>>>> >>>>>>> In .vcl file I have following set. >>>>>>> .first_byte_timeout = 120s; >>>>>>> >>>>>>> Now, I am not logging varnish to apache requests ( which backend fetch ). >>>>>>> As you can see above I am only logging client requests. >>>>>>> >>>>>>> Is their a way to log client requests like above but then also log backend fetch failures to another log file ? >>>>>>> [ Running 2 instances of varnishncsa ? ] >>>>>>> >>>>>>> Any pointers ? >>>>>>> >>>>>>> Please let me know. >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> _______________________________________________ >>>>>>> varnish-misc mailing list >>>>>>> varnish-misc at varnish-cache.org >>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From mandys at gmail.com Fri Dec 27 07:46:20 2019 From: mandys at gmail.com (Maninder Singh) Date: Fri, 27 Dec 2019 13:16:20 +0530 Subject: Varnish 503 on POST requests once in a while In-Reply-To: References: Message-ID: When I got the Resource temporarily unavailable error, I see the following in apache error_log error_80_log.26:[Thu Dec 26 21:24:18 2019] [error] [pid 16739] [client x.x.x.x] AH01075: Error dispatching request to :8080: And almost everytime request seems to be POST and taking more than 5 seconds. Somewhere a 5 sec timeout is imposed. Not sure if apaches mod_reqtimeout has any effect. https://httpd.apache.org/docs/current/mod/mod_reqtimeout.html#requestreadtimeout This is enabled by default on apache 2.4. On Fri, 27 Dec 2019 at 06:27, Maninder Singh wrote: > I only have php-fpm slow logging for requests greater than 5secs. > Otherwise it should be all defaults for varnish/apache/fpm. > > Does varnish or apache have anything defaulting to 5secs? > > Otherwise fpm might be closing the connection for logging a slow request ? > Strange? > > On Thu, 26 Dec, 2019, 10:44 pm Guillaume Quintard, < > guillaume at varnish-software.com> wrote: > >> The first looks like the backend is just closing the door on you. The >> other one (Resource temporarily unavailable), because of the >> Timestamp:Beresp, is most probably a timeout (did you set anything to 5s?) >> >> -- >> Guillaume Quintard >> >> >> On Thu, Dec 26, 2019 at 3:12 PM Maninder Singh wrote: >> >>> Another one. >>> This one is diff error. >>> >>> * << BeReq >> 19327398 >>> - Begin bereq 19327397 pass >>> - Timestamp Start: 1577370022.344818 0.000000 0.000000 >>> - BereqMethod POST >>> - BereqURL /LetsCelebrate >>> - BereqProtocol HTTP/1.1 >>> - BereqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X >>> 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 >>> Safari/537.36 >>> - BereqHeader Accept: >>> text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 >>> - BereqHeader Cache-Control: max-age=0 >>> - BereqHeader SSLClientCertStatus: NoClientCert >>> - BereqHeader X-Forwarded-Proto: https >>> - BereqHeader Content-Type: multipart/form-data; >>> boundary=----WebKitFormBoundaryBBVOJciq1v4rbZs9 >>> - BereqHeader Accept-Language: en-GB,en-US;q=0.9,en;q=0.8 >>> - BereqHeader SSLClientCipher: >>> SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384, version=TLSv1.2, bits=256 >>> - BereqHeader Host: mydomain.com >>> - BereqHeader Referer: https://mydomain.com/LetsCelebrate >>> - BereqHeader SSLSessionID: >>> 97258494A3DD008A855B552F701E9F95C25CDC7B1CF5FD63D382ED09391585C7 >>> - BereqHeader Origin: https://mydomain.com >>> - BereqHeader X-Cluster-Client-Ip: X.X.X.X >>> - BereqHeader X-Forwarded-Port: 443 >>> - BereqHeader Upgrade-Insecure-Requests: 1 >>> - BereqHeader Sec-Fetch-User: ?1 >>> - BereqHeader Sec-Fetch-Site: same-origin >>> - BereqHeader Sec-Fetch-Mode: nested-navigate >>> - BereqHeader Accept-Encoding: gzip, deflate, br >>> - BereqHeader Content-Length: 2076 >>> - BereqHeader X-Forwarded-For: X.X.X.X, 10.187.187.36 >>> - BereqHeader browser: other >>> - BereqHeader serverIp: 10.208.225.235 >>> - BereqHeader X-Varnish: 19327398 >>> - VCL_call BACKEND_FETCH >>> - VCL_return fetch >>> - BackendOpen 23 reload_2019-12-18T054238.default 127.0.0.1 8080 >>> 127.0.0.1 39568 >>> - BackendStart 127.0.0.1 8080 >>> - FetchError req.body read error: 11 (Resource temporarily >>> unavailable) >>> - FetchError backend write error: 11 (Resource temporarily >>> unavailable) >>> - Timestamp Bereq: 1577370027.344908 5.000090 5.000090 >>> - BackendClose 23 reload_2019-12-18T054238.default >>> - Timestamp Beresp: 1577370027.345014 5.000196 0.000105 >>> - Timestamp Error: 1577370027.345025 5.000206 0.000011 >>> - BerespProtocol HTTP/1.1 >>> - BerespStatus 503 >>> - BerespReason Service Unavailable >>> - BerespReason Backend fetch failed >>> - BerespHeader Date: Thu, 26 Dec 2019 14:20:27 GMT >>> - BerespHeader Server: Varnish >>> - VCL_call BACKEND_ERROR >>> - BerespHeader Content-Type: text/html; charset=utf-8 >>> - BerespHeader Retry-After: 5 >>> - VCL_return deliver >>> - Storage malloc Transient >>> - ObjProtocol HTTP/1.1 >>> - ObjStatus 503 >>> - ObjReason Backend fetch failed >>> - ObjHeader Date: Thu, 26 Dec 2019 14:20:27 GMT >>> - ObjHeader Server: Varnish >>> - ObjHeader Content-Type: text/html; charset=utf-8 >>> - ObjHeader Retry-After: 5 >>> - Length 1288 >>> - BereqAcct 1276 0 1276 0 0 0 >>> - End >>> >>> On Thu, 26 Dec 2019 at 17:27, Maninder Singh wrote: >>> >>>> Got one in the log. >>>> >>>> varnishlog -r /var/log/varnish/503.log >>>> * << BeReq >> 21993384 >>>> - Begin bereq 21993383 pass >>>> - Timestamp Start: 1577360704.829258 0.000000 0.000000 >>>> - BereqMethod POST >>>> - BereqURL /corp/contact.php?cf=exitpopup >>>> - BereqProtocol HTTP/1.1 >>>> - BereqHeader User-Agent: Mozilla/5.0 (Windows NT 6.1; >>>> Trident/7.0; rv:11.0) like Gecko >>>> - BereqHeader Accept: image/gif, image/x-xbitmap, image/jpeg, >>>> image/pjpeg, application/x-shockwave-flash, */* >>>> - BereqHeader SSLClientCertStatus: NoClientCert >>>> - BereqHeader X-Forwarded-Proto: https >>>> - BereqHeader Content-Type: application/x-www-form-urlencoded >>>> - BereqHeader SSLClientCipher: SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA, >>>> version=TLSv1, bits=256 >>>> - BereqHeader Host: www.mydomain.com >>>> - BereqHeader Referer: >>>> http://mydomain.com/solutions/enterprise.php >>>> - BereqHeader SSLSessionID: >>>> 3C119EF3FE70318789F82E9B583581C876B4DDE520C0182C33CD6AF1AA46E63A >>>> - BereqHeader X-Cluster-Client-Ip: X.X.X.X >>>> - BereqHeader Cookie: >>>> X-Mapping-fjhppofk=E8F494E9A630955F1975EEDF11EBC4D7 >>>> - BereqHeader X-Forwarded-Port: 443 >>>> - BereqHeader Accept-Encoding: gzip, deflate >>>> - BereqHeader Content-Length: 1366 >>>> - BereqHeader X-Forwarded-For: X.X.X.X, 10.187.187.36 >>>> - BereqHeader browser: other >>>> - BereqHeader serverIp: 10.208.224.192 >>>> - BereqHeader X-Varnish: 21993384 >>>> - VCL_call BACKEND_FETCH >>>> - VCL_return fetch >>>> - BackendOpen 26 reload_2019-12-18T054253.default 127.0.0.1 8080 >>>> 127.0.0.1 59998 >>>> - BackendStart 127.0.0.1 8080 >>>> - Timestamp Bereq: 1577360705.664932 0.835674 0.835674 >>>> - FetchError http first read error: EOF >>>> - BackendClose 26 reload_2019-12-18T054253.default >>>> - Timestamp Beresp: 1577360705.665020 0.835763 0.000089 >>>> - Timestamp Error: 1577360705.665028 0.835770 0.000008 >>>> - BerespProtocol HTTP/1.1 >>>> - BerespStatus 503 >>>> - BerespReason Service Unavailable >>>> - BerespReason Backend fetch failed >>>> - BerespHeader Date: Thu, 26 Dec 2019 11:45:05 GMT >>>> - BerespHeader Server: Varnish >>>> - VCL_call BACKEND_ERROR >>>> - BerespHeader Content-Type: text/html; charset=utf-8 >>>> - BerespHeader Retry-After: 5 >>>> - VCL_return deliver >>>> - Storage malloc Transient >>>> - ObjProtocol HTTP/1.1 >>>> - ObjStatus 503 >>>> - ObjReason Backend fetch failed >>>> - ObjHeader Date: Thu, 26 Dec 2019 11:45:05 GMT >>>> - ObjHeader Server: Varnish >>>> - ObjHeader Content-Type: text/html; charset=utf-8 >>>> - ObjHeader Retry-After: 5 >>>> - Length 1288 >>>> - BereqAcct 849 1366 2215 0 0 0 >>>> - End >>>> >>>> On Thu, 26 Dec 2019 at 15:21, Maninder Singh wrote: >>>> >>>>> Okay. Let me set it up and get back to you once i see something in >>>>> that log. It happens only once/twice a day so could be a while before i get >>>>> back. >>>>> >>>>> On Thu, 26 Dec, 2019, 3:10 pm Guillaume Quintard, < >>>>> guillaume at varnish-software.com> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Most probably, varnishncsa isn't going to give you the full picture, >>>>>> so let's go for the exhaustive way: >>>>>> >>>>>> varnishlog -q "BerespStatus eq 503" -w /whatever/file >>>>>> >>>>>> >>>>>> you can then read the logs with >>>>>> >>>>>> varnishlog -r /whatever/file >>>>>> >>>>>> >>>>>> (more logging pointers here: >>>>>> https://docs.varnish-software.com/tutorials/vsl-query/) >>>>>> >>>>>> If you can get your hands on a failed transaction, we should be able >>>>>> to tell you what went wrong. Most probably, you hit a timeout of some sort. >>>>>> Or it could be that the backend was declared sick, if you have probes. >>>>>> >>>>>> -- >>>>>> Guillaume Quintard >>>>>> >>>>>> >>>>>> On Thu, Dec 26, 2019 at 6:33 AM Maninder Singh >>>>>> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> I am facing a strange problem. >>>>>>> >>>>>>> I am running varnish 4.1.11 on port 80 which is behind a LB. >>>>>>> >>>>>>> Varnish then connects to apache 2.4 ( on port 8080 ) and also have >>>>>>> php-fpm running. >>>>>>> >>>>>>> Now, everything works fine but once in a while I am seeing 503 >>>>>>> response from varnish for a POST request. During this time there is no log >>>>>>> in apache also ( neither access log nor error log ). >>>>>>> >>>>>>> Where does this request vanish ? >>>>>>> >>>>>>> How do I debug this issue ? >>>>>>> >>>>>>> Some more details: >>>>>>> >>>>>>> * I have varnishncsa turned on with following flags. >>>>>>> >>>>>>> ExecStart=/usr/bin/varnishncsa -q 'not (ReqURL ~ >>>>>>> "\.(js|css|png|jpeg|jpg|ico|gif|swf|woff)" or ReqHeader:User-Agent ~ >>>>>>> "HTTP-Monitor" or ReqHeader:User-Agent ~ "Monitoring/1.1")' -a -w >>>>>>> /var/log/varnish/varnishncsa.log -D -f /etc/sysconfig/varnishncsa >>>>>>> >>>>>>> Here is the entry in varnish log during that time. >>>>>>> >>>>>>> X.X.X.X, LB.IP.0.0 - - [25/Dec/2019:00:26:40 -0600] 2629363 "POST >>>>>>> http://mydomain.com/LetsCelebrate HTTP/1.1" 503 1288 " >>>>>>> https://referringdomain.com/LetsCelebrate?hostingUrl=https%3A%2F%2Fxy.abc.com" >>>>>>> "Mozilla/5.0 (Linux; Android 9; SM-A920F) AppleWebKit/537.36 (KHTML, like >>>>>>> Gecko) Chrome/79.0.3945.93 Mobile Safari/537.36" "mydomain.com" miss >>>>>>> >>>>>>> In .vcl file I have following set. >>>>>>> .first_byte_timeout = 120s; >>>>>>> >>>>>>> Now, I am not logging varnish to apache requests ( which backend >>>>>>> fetch ). >>>>>>> As you can see above I am only logging client requests. >>>>>>> >>>>>>> Is their a way to log client requests like above but then also log >>>>>>> backend fetch failures to another log file ? >>>>>>> [ Running 2 instances of varnishncsa ? ] >>>>>>> >>>>>>> Any pointers ? >>>>>>> >>>>>>> Please let me know. >>>>>>> >>>>>>> Thanks. >>>>>>> >>>>>>> _______________________________________________ >>>>>>> varnish-misc mailing list >>>>>>> varnish-misc at varnish-cache.org >>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >>>>>>> >>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From mandys at gmail.com Fri Dec 27 08:52:30 2019 From: mandys at gmail.com (Maninder Singh) Date: Fri, 27 Dec 2019 14:22:30 +0530 Subject: Varnish 503 on POST requests once in a while In-Reply-To: References: Message-ID: Here are the different timeouts: *timeout_idle* (default: 5 seconds): How long we will wait from we `accept()` the connection, until the client must have sent us a non-whitespace character. (typically this will be the G in "GET /url HTTP/1.1") Isn't idle_timeout for client to varnish requests ?? Also, just so that you know, these requests dont make it to apache logs also. backend_idle_timeout 60.000 [seconds] (default) between_bytes_timeout 60.000 [seconds] (default) cli_timeout 60.000 [seconds] (default) connect_timeout 3.500 [seconds] (default) first_byte_timeout 60.000 [seconds] (default) idle_send_timeout 60.000 [seconds] (default) pipe_timeout 60.000 [seconds] (default) send_timeout 600.000 [seconds] (default) thread_pool_timeout 300.000 [seconds] (default) timeout_idle 5.000 [seconds] (default) timeout_linger 0.050 [seconds] (default) On Fri, 27 Dec 2019 at 13:15, Dridi Boukelmoune wrote: > On Fri, Dec 27, 2019 at 1:00 AM Maninder Singh wrote: > > > > I only have php-fpm slow logging for requests greater than 5secs. > Otherwise it should be all defaults for varnish/apache/fpm. > > > > Does varnish or apache have anything defaulting to 5secs? > > timeout_idle defaults to 5s > > > Otherwise fpm might be closing the connection for logging a slow request > ? Strange? > > > > On Thu, 26 Dec, 2019, 10:44 pm Guillaume Quintard, < > guillaume at varnish-software.com> wrote: > >> > >> The first looks like the backend is just closing the door on you. The > other one (Resource temporarily unavailable), because of the > Timestamp:Beresp, is most probably a timeout (did you set anything to 5s?) > >> > >> -- > >> Guillaume Quintard > >> > >> > >> On Thu, Dec 26, 2019 at 3:12 PM Maninder Singh > wrote: > >>> > >>> Another one. > >>> This one is diff error. > >>> > >>> * << BeReq >> 19327398 > >>> - Begin bereq 19327397 pass > >>> - Timestamp Start: 1577370022.344818 0.000000 0.000000 > >>> - BereqMethod POST > >>> - BereqURL /LetsCelebrate > >>> - BereqProtocol HTTP/1.1 > >>> - BereqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X > 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 > Safari/537.36 > >>> - BereqHeader Accept: > text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 > >>> - BereqHeader Cache-Control: max-age=0 > >>> - BereqHeader SSLClientCertStatus: NoClientCert > >>> - BereqHeader X-Forwarded-Proto: https > >>> - BereqHeader Content-Type: multipart/form-data; > boundary=----WebKitFormBoundaryBBVOJciq1v4rbZs9 > >>> - BereqHeader Accept-Language: en-GB,en-US;q=0.9,en;q=0.8 > >>> - BereqHeader SSLClientCipher: > SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384, version=TLSv1.2, bits=256 > >>> - BereqHeader Host: mydomain.com > >>> - BereqHeader Referer: https://mydomain.com/LetsCelebrate > >>> - BereqHeader SSLSessionID: > 97258494A3DD008A855B552F701E9F95C25CDC7B1CF5FD63D382ED09391585C7 > >>> - BereqHeader Origin: https://mydomain.com > >>> - BereqHeader X-Cluster-Client-Ip: X.X.X.X > >>> - BereqHeader X-Forwarded-Port: 443 > >>> - BereqHeader Upgrade-Insecure-Requests: 1 > >>> - BereqHeader Sec-Fetch-User: ?1 > >>> - BereqHeader Sec-Fetch-Site: same-origin > >>> - BereqHeader Sec-Fetch-Mode: nested-navigate > >>> - BereqHeader Accept-Encoding: gzip, deflate, br > >>> - BereqHeader Content-Length: 2076 > >>> - BereqHeader X-Forwarded-For: X.X.X.X, 10.187.187.36 > >>> - BereqHeader browser: other > >>> - BereqHeader serverIp: 10.208.225.235 > >>> - BereqHeader X-Varnish: 19327398 > >>> - VCL_call BACKEND_FETCH > >>> - VCL_return fetch > >>> - BackendOpen 23 reload_2019-12-18T054238.default 127.0.0.1 8080 > 127.0.0.1 39568 > >>> - BackendStart 127.0.0.1 8080 > >>> - FetchError req.body read error: 11 (Resource temporarily > unavailable) > >>> - FetchError backend write error: 11 (Resource temporarily > unavailable) > >>> - Timestamp Bereq: 1577370027.344908 5.000090 5.000090 > >>> - BackendClose 23 reload_2019-12-18T054238.default > >>> - Timestamp Beresp: 1577370027.345014 5.000196 0.000105 > >>> - Timestamp Error: 1577370027.345025 5.000206 0.000011 > >>> - BerespProtocol HTTP/1.1 > >>> - BerespStatus 503 > >>> - BerespReason Service Unavailable > >>> - BerespReason Backend fetch failed > >>> - BerespHeader Date: Thu, 26 Dec 2019 14:20:27 GMT > >>> - BerespHeader Server: Varnish > >>> - VCL_call BACKEND_ERROR > >>> - BerespHeader Content-Type: text/html; charset=utf-8 > >>> - BerespHeader Retry-After: 5 > >>> - VCL_return deliver > >>> - Storage malloc Transient > >>> - ObjProtocol HTTP/1.1 > >>> - ObjStatus 503 > >>> - ObjReason Backend fetch failed > >>> - ObjHeader Date: Thu, 26 Dec 2019 14:20:27 GMT > >>> - ObjHeader Server: Varnish > >>> - ObjHeader Content-Type: text/html; charset=utf-8 > >>> - ObjHeader Retry-After: 5 > >>> - Length 1288 > >>> - BereqAcct 1276 0 1276 0 0 0 > >>> - End > >>> > >>> On Thu, 26 Dec 2019 at 17:27, Maninder Singh wrote: > >>>> > >>>> Got one in the log. > >>>> > >>>> varnishlog -r /var/log/varnish/503.log > >>>> * << BeReq >> 21993384 > >>>> - Begin bereq 21993383 pass > >>>> - Timestamp Start: 1577360704.829258 0.000000 0.000000 > >>>> - BereqMethod POST > >>>> - BereqURL /corp/contact.php?cf=exitpopup > >>>> - BereqProtocol HTTP/1.1 > >>>> - BereqHeader User-Agent: Mozilla/5.0 (Windows NT 6.1; > Trident/7.0; rv:11.0) like Gecko > >>>> - BereqHeader Accept: image/gif, image/x-xbitmap, image/jpeg, > image/pjpeg, application/x-shockwave-flash, */* > >>>> - BereqHeader SSLClientCertStatus: NoClientCert > >>>> - BereqHeader X-Forwarded-Proto: https > >>>> - BereqHeader Content-Type: application/x-www-form-urlencoded > >>>> - BereqHeader SSLClientCipher: > SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA, version=TLSv1, bits=256 > >>>> - BereqHeader Host: www.mydomain.com > >>>> - BereqHeader Referer: > http://mydomain.com/solutions/enterprise.php > >>>> - BereqHeader SSLSessionID: > 3C119EF3FE70318789F82E9B583581C876B4DDE520C0182C33CD6AF1AA46E63A > >>>> - BereqHeader X-Cluster-Client-Ip: X.X.X.X > >>>> - BereqHeader Cookie: > X-Mapping-fjhppofk=E8F494E9A630955F1975EEDF11EBC4D7 > >>>> - BereqHeader X-Forwarded-Port: 443 > >>>> - BereqHeader Accept-Encoding: gzip, deflate > >>>> - BereqHeader Content-Length: 1366 > >>>> - BereqHeader X-Forwarded-For: X.X.X.X, 10.187.187.36 > >>>> - BereqHeader browser: other > >>>> - BereqHeader serverIp: 10.208.224.192 > >>>> - BereqHeader X-Varnish: 21993384 > >>>> - VCL_call BACKEND_FETCH > >>>> - VCL_return fetch > >>>> - BackendOpen 26 reload_2019-12-18T054253.default 127.0.0.1 8080 > 127.0.0.1 59998 > >>>> - BackendStart 127.0.0.1 8080 > >>>> - Timestamp Bereq: 1577360705.664932 0.835674 0.835674 > >>>> - FetchError http first read error: EOF > >>>> - BackendClose 26 reload_2019-12-18T054253.default > >>>> - Timestamp Beresp: 1577360705.665020 0.835763 0.000089 > >>>> - Timestamp Error: 1577360705.665028 0.835770 0.000008 > >>>> - BerespProtocol HTTP/1.1 > >>>> - BerespStatus 503 > >>>> - BerespReason Service Unavailable > >>>> - BerespReason Backend fetch failed > >>>> - BerespHeader Date: Thu, 26 Dec 2019 11:45:05 GMT > >>>> - BerespHeader Server: Varnish > >>>> - VCL_call BACKEND_ERROR > >>>> - BerespHeader Content-Type: text/html; charset=utf-8 > >>>> - BerespHeader Retry-After: 5 > >>>> - VCL_return deliver > >>>> - Storage malloc Transient > >>>> - ObjProtocol HTTP/1.1 > >>>> - ObjStatus 503 > >>>> - ObjReason Backend fetch failed > >>>> - ObjHeader Date: Thu, 26 Dec 2019 11:45:05 GMT > >>>> - ObjHeader Server: Varnish > >>>> - ObjHeader Content-Type: text/html; charset=utf-8 > >>>> - ObjHeader Retry-After: 5 > >>>> - Length 1288 > >>>> - BereqAcct 849 1366 2215 0 0 0 > >>>> - End > >>>> > >>>> On Thu, 26 Dec 2019 at 15:21, Maninder Singh > wrote: > >>>>> > >>>>> Okay. Let me set it up and get back to you once i see something in > that log. It happens only once/twice a day so could be a while before i get > back. > >>>>> > >>>>> On Thu, 26 Dec, 2019, 3:10 pm Guillaume Quintard, < > guillaume at varnish-software.com> wrote: > >>>>>> > >>>>>> Hi, > >>>>>> > >>>>>> Most probably, varnishncsa isn't going to give you the full > picture, so let's go for the exhaustive way: > >>>>>> > >>>>>> varnishlog -q "BerespStatus eq 503" -w /whatever/file > >>>>>> > >>>>>> > >>>>>> you can then read the logs with > >>>>>> > >>>>>> varnishlog -r /whatever/file > >>>>>> > >>>>>> > >>>>>> (more logging pointers here: > https://docs.varnish-software.com/tutorials/vsl-query/) > >>>>>> > >>>>>> If you can get your hands on a failed transaction, we should be > able to tell you what went wrong. Most probably, you hit a timeout of some > sort. Or it could be that the backend was declared sick, if you have probes. > >>>>>> > >>>>>> -- > >>>>>> Guillaume Quintard > >>>>>> > >>>>>> > >>>>>> On Thu, Dec 26, 2019 at 6:33 AM Maninder Singh > wrote: > >>>>>>> > >>>>>>> Hi, > >>>>>>> > >>>>>>> I am facing a strange problem. > >>>>>>> > >>>>>>> I am running varnish 4.1.11 on port 80 which is behind a LB. > >>>>>>> > >>>>>>> Varnish then connects to apache 2.4 ( on port 8080 ) and also have > php-fpm running. > >>>>>>> > >>>>>>> Now, everything works fine but once in a while I am seeing 503 > response from varnish for a POST request. During this time there is no log > in apache also ( neither access log nor error log ). > >>>>>>> > >>>>>>> Where does this request vanish ? > >>>>>>> > >>>>>>> How do I debug this issue ? > >>>>>>> > >>>>>>> Some more details: > >>>>>>> > >>>>>>> * I have varnishncsa turned on with following flags. > >>>>>>> > >>>>>>> ExecStart=/usr/bin/varnishncsa -q 'not (ReqURL ~ > "\.(js|css|png|jpeg|jpg|ico|gif|swf|woff)" or ReqHeader:User-Agent ~ > "HTTP-Monitor" or ReqHeader:User-Agent ~ "Monitoring/1.1")' -a -w > /var/log/varnish/varnishncsa.log -D -f /etc/sysconfig/varnishncsa > >>>>>>> > >>>>>>> Here is the entry in varnish log during that time. > >>>>>>> > >>>>>>> X.X.X.X, LB.IP.0.0 - - [25/Dec/2019:00:26:40 -0600] 2629363 "POST > http://mydomain.com/LetsCelebrate HTTP/1.1" 503 1288 " > https://referringdomain.com/LetsCelebrate?hostingUrl=https%3A%2F%2Fxy.abc.com" > "Mozilla/5.0 (Linux; Android 9; SM-A920F) AppleWebKit/537.36 (KHTML, like > Gecko) Chrome/79.0.3945.93 Mobile Safari/537.36" "mydomain.com" miss > >>>>>>> > >>>>>>> In .vcl file I have following set. > >>>>>>> .first_byte_timeout = 120s; > >>>>>>> > >>>>>>> Now, I am not logging varnish to apache requests ( which backend > fetch ). > >>>>>>> As you can see above I am only logging client requests. > >>>>>>> > >>>>>>> Is their a way to log client requests like above but then also log > backend fetch failures to another log file ? > >>>>>>> [ Running 2 instances of varnishncsa ? ] > >>>>>>> > >>>>>>> Any pointers ? > >>>>>>> > >>>>>>> Please let me know. > >>>>>>> > >>>>>>> Thanks. > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> varnish-misc mailing list > >>>>>>> varnish-misc at varnish-cache.org > >>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > > > > _______________________________________________ > > varnish-misc mailing list > > varnish-misc at varnish-cache.org > > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mandys at gmail.com Mon Dec 30 06:24:06 2019 From: mandys at gmail.com (Maninder Singh) Date: Mon, 30 Dec 2019 11:54:06 +0530 Subject: Varnish 503 on POST requests once in a while In-Reply-To: References: Message-ID: Hi Guillaume, Any idea on what the 5 sec timeout might be ( which happens only 3-4 times a day ) ? Is it the timeout_idle as mentioned by Dridi ? What can be done to fix it ? On Fri, 27 Dec 2019 at 14:22, Maninder Singh wrote: > Here are the different timeouts: > > *timeout_idle* (default: 5 seconds): How long we will wait from we > `accept()` the connection, until the client must have sent us a > non-whitespace character. (typically this will be the G in "GET /url > HTTP/1.1") > > Isn't idle_timeout for client to varnish requests ?? > > Also, just so that you know, these requests dont make it to apache logs > also. > > > > backend_idle_timeout 60.000 [seconds] (default) > between_bytes_timeout 60.000 [seconds] (default) > cli_timeout 60.000 [seconds] (default) > connect_timeout 3.500 [seconds] (default) > first_byte_timeout 60.000 [seconds] (default) > idle_send_timeout 60.000 [seconds] (default) > pipe_timeout 60.000 [seconds] (default) > send_timeout 600.000 [seconds] (default) > thread_pool_timeout 300.000 [seconds] (default) > timeout_idle 5.000 [seconds] (default) > timeout_linger 0.050 [seconds] (default) > > > On Fri, 27 Dec 2019 at 13:15, Dridi Boukelmoune wrote: > >> On Fri, Dec 27, 2019 at 1:00 AM Maninder Singh wrote: >> > >> > I only have php-fpm slow logging for requests greater than 5secs. >> Otherwise it should be all defaults for varnish/apache/fpm. >> > >> > Does varnish or apache have anything defaulting to 5secs? >> >> timeout_idle defaults to 5s >> >> > Otherwise fpm might be closing the connection for logging a slow >> request ? Strange? >> > >> > On Thu, 26 Dec, 2019, 10:44 pm Guillaume Quintard, < >> guillaume at varnish-software.com> wrote: >> >> >> >> The first looks like the backend is just closing the door on you. The >> other one (Resource temporarily unavailable), because of the >> Timestamp:Beresp, is most probably a timeout (did you set anything to 5s?) >> >> >> >> -- >> >> Guillaume Quintard >> >> >> >> >> >> On Thu, Dec 26, 2019 at 3:12 PM Maninder Singh >> wrote: >> >>> >> >>> Another one. >> >>> This one is diff error. >> >>> >> >>> * << BeReq >> 19327398 >> >>> - Begin bereq 19327397 pass >> >>> - Timestamp Start: 1577370022.344818 0.000000 0.000000 >> >>> - BereqMethod POST >> >>> - BereqURL /LetsCelebrate >> >>> - BereqProtocol HTTP/1.1 >> >>> - BereqHeader User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X >> 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 >> Safari/537.36 >> >>> - BereqHeader Accept: >> text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 >> >>> - BereqHeader Cache-Control: max-age=0 >> >>> - BereqHeader SSLClientCertStatus: NoClientCert >> >>> - BereqHeader X-Forwarded-Proto: https >> >>> - BereqHeader Content-Type: multipart/form-data; >> boundary=----WebKitFormBoundaryBBVOJciq1v4rbZs9 >> >>> - BereqHeader Accept-Language: en-GB,en-US;q=0.9,en;q=0.8 >> >>> - BereqHeader SSLClientCipher: >> SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384, version=TLSv1.2, bits=256 >> >>> - BereqHeader Host: mydomain.com >> >>> - BereqHeader Referer: https://mydomain.com/LetsCelebrate >> >>> - BereqHeader SSLSessionID: >> 97258494A3DD008A855B552F701E9F95C25CDC7B1CF5FD63D382ED09391585C7 >> >>> - BereqHeader Origin: https://mydomain.com >> >>> - BereqHeader X-Cluster-Client-Ip: X.X.X.X >> >>> - BereqHeader X-Forwarded-Port: 443 >> >>> - BereqHeader Upgrade-Insecure-Requests: 1 >> >>> - BereqHeader Sec-Fetch-User: ?1 >> >>> - BereqHeader Sec-Fetch-Site: same-origin >> >>> - BereqHeader Sec-Fetch-Mode: nested-navigate >> >>> - BereqHeader Accept-Encoding: gzip, deflate, br >> >>> - BereqHeader Content-Length: 2076 >> >>> - BereqHeader X-Forwarded-For: X.X.X.X, 10.187.187.36 >> >>> - BereqHeader browser: other >> >>> - BereqHeader serverIp: 10.208.225.235 >> >>> - BereqHeader X-Varnish: 19327398 >> >>> - VCL_call BACKEND_FETCH >> >>> - VCL_return fetch >> >>> - BackendOpen 23 reload_2019-12-18T054238.default 127.0.0.1 8080 >> 127.0.0.1 39568 >> >>> - BackendStart 127.0.0.1 8080 >> >>> - FetchError req.body read error: 11 (Resource temporarily >> unavailable) >> >>> - FetchError backend write error: 11 (Resource temporarily >> unavailable) >> >>> - Timestamp Bereq: 1577370027.344908 5.000090 5.000090 >> >>> - BackendClose 23 reload_2019-12-18T054238.default >> >>> - Timestamp Beresp: 1577370027.345014 5.000196 0.000105 >> >>> - Timestamp Error: 1577370027.345025 5.000206 0.000011 >> >>> - BerespProtocol HTTP/1.1 >> >>> - BerespStatus 503 >> >>> - BerespReason Service Unavailable >> >>> - BerespReason Backend fetch failed >> >>> - BerespHeader Date: Thu, 26 Dec 2019 14:20:27 GMT >> >>> - BerespHeader Server: Varnish >> >>> - VCL_call BACKEND_ERROR >> >>> - BerespHeader Content-Type: text/html; charset=utf-8 >> >>> - BerespHeader Retry-After: 5 >> >>> - VCL_return deliver >> >>> - Storage malloc Transient >> >>> - ObjProtocol HTTP/1.1 >> >>> - ObjStatus 503 >> >>> - ObjReason Backend fetch failed >> >>> - ObjHeader Date: Thu, 26 Dec 2019 14:20:27 GMT >> >>> - ObjHeader Server: Varnish >> >>> - ObjHeader Content-Type: text/html; charset=utf-8 >> >>> - ObjHeader Retry-After: 5 >> >>> - Length 1288 >> >>> - BereqAcct 1276 0 1276 0 0 0 >> >>> - End >> >>> >> >>> On Thu, 26 Dec 2019 at 17:27, Maninder Singh >> wrote: >> >>>> >> >>>> Got one in the log. >> >>>> >> >>>> varnishlog -r /var/log/varnish/503.log >> >>>> * << BeReq >> 21993384 >> >>>> - Begin bereq 21993383 pass >> >>>> - Timestamp Start: 1577360704.829258 0.000000 0.000000 >> >>>> - BereqMethod POST >> >>>> - BereqURL /corp/contact.php?cf=exitpopup >> >>>> - BereqProtocol HTTP/1.1 >> >>>> - BereqHeader User-Agent: Mozilla/5.0 (Windows NT 6.1; >> Trident/7.0; rv:11.0) like Gecko >> >>>> - BereqHeader Accept: image/gif, image/x-xbitmap, image/jpeg, >> image/pjpeg, application/x-shockwave-flash, */* >> >>>> - BereqHeader SSLClientCertStatus: NoClientCert >> >>>> - BereqHeader X-Forwarded-Proto: https >> >>>> - BereqHeader Content-Type: application/x-www-form-urlencoded >> >>>> - BereqHeader SSLClientCipher: >> SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA, version=TLSv1, bits=256 >> >>>> - BereqHeader Host: www.mydomain.com >> >>>> - BereqHeader Referer: >> http://mydomain.com/solutions/enterprise.php >> >>>> - BereqHeader SSLSessionID: >> 3C119EF3FE70318789F82E9B583581C876B4DDE520C0182C33CD6AF1AA46E63A >> >>>> - BereqHeader X-Cluster-Client-Ip: X.X.X.X >> >>>> - BereqHeader Cookie: >> X-Mapping-fjhppofk=E8F494E9A630955F1975EEDF11EBC4D7 >> >>>> - BereqHeader X-Forwarded-Port: 443 >> >>>> - BereqHeader Accept-Encoding: gzip, deflate >> >>>> - BereqHeader Content-Length: 1366 >> >>>> - BereqHeader X-Forwarded-For: X.X.X.X, 10.187.187.36 >> >>>> - BereqHeader browser: other >> >>>> - BereqHeader serverIp: 10.208.224.192 >> >>>> - BereqHeader X-Varnish: 21993384 >> >>>> - VCL_call BACKEND_FETCH >> >>>> - VCL_return fetch >> >>>> - BackendOpen 26 reload_2019-12-18T054253.default 127.0.0.1 >> 8080 127.0.0.1 59998 >> >>>> - BackendStart 127.0.0.1 8080 >> >>>> - Timestamp Bereq: 1577360705.664932 0.835674 0.835674 >> >>>> - FetchError http first read error: EOF >> >>>> - BackendClose 26 reload_2019-12-18T054253.default >> >>>> - Timestamp Beresp: 1577360705.665020 0.835763 0.000089 >> >>>> - Timestamp Error: 1577360705.665028 0.835770 0.000008 >> >>>> - BerespProtocol HTTP/1.1 >> >>>> - BerespStatus 503 >> >>>> - BerespReason Service Unavailable >> >>>> - BerespReason Backend fetch failed >> >>>> - BerespHeader Date: Thu, 26 Dec 2019 11:45:05 GMT >> >>>> - BerespHeader Server: Varnish >> >>>> - VCL_call BACKEND_ERROR >> >>>> - BerespHeader Content-Type: text/html; charset=utf-8 >> >>>> - BerespHeader Retry-After: 5 >> >>>> - VCL_return deliver >> >>>> - Storage malloc Transient >> >>>> - ObjProtocol HTTP/1.1 >> >>>> - ObjStatus 503 >> >>>> - ObjReason Backend fetch failed >> >>>> - ObjHeader Date: Thu, 26 Dec 2019 11:45:05 GMT >> >>>> - ObjHeader Server: Varnish >> >>>> - ObjHeader Content-Type: text/html; charset=utf-8 >> >>>> - ObjHeader Retry-After: 5 >> >>>> - Length 1288 >> >>>> - BereqAcct 849 1366 2215 0 0 0 >> >>>> - End >> >>>> >> >>>> On Thu, 26 Dec 2019 at 15:21, Maninder Singh >> wrote: >> >>>>> >> >>>>> Okay. Let me set it up and get back to you once i see something in >> that log. It happens only once/twice a day so could be a while before i get >> back. >> >>>>> >> >>>>> On Thu, 26 Dec, 2019, 3:10 pm Guillaume Quintard, < >> guillaume at varnish-software.com> wrote: >> >>>>>> >> >>>>>> Hi, >> >>>>>> >> >>>>>> Most probably, varnishncsa isn't going to give you the full >> picture, so let's go for the exhaustive way: >> >>>>>> >> >>>>>> varnishlog -q "BerespStatus eq 503" -w /whatever/file >> >>>>>> >> >>>>>> >> >>>>>> you can then read the logs with >> >>>>>> >> >>>>>> varnishlog -r /whatever/file >> >>>>>> >> >>>>>> >> >>>>>> (more logging pointers here: >> https://docs.varnish-software.com/tutorials/vsl-query/) >> >>>>>> >> >>>>>> If you can get your hands on a failed transaction, we should be >> able to tell you what went wrong. Most probably, you hit a timeout of some >> sort. Or it could be that the backend was declared sick, if you have probes. >> >>>>>> >> >>>>>> -- >> >>>>>> Guillaume Quintard >> >>>>>> >> >>>>>> >> >>>>>> On Thu, Dec 26, 2019 at 6:33 AM Maninder Singh >> wrote: >> >>>>>>> >> >>>>>>> Hi, >> >>>>>>> >> >>>>>>> I am facing a strange problem. >> >>>>>>> >> >>>>>>> I am running varnish 4.1.11 on port 80 which is behind a LB. >> >>>>>>> >> >>>>>>> Varnish then connects to apache 2.4 ( on port 8080 ) and also >> have php-fpm running. >> >>>>>>> >> >>>>>>> Now, everything works fine but once in a while I am seeing 503 >> response from varnish for a POST request. During this time there is no log >> in apache also ( neither access log nor error log ). >> >>>>>>> >> >>>>>>> Where does this request vanish ? >> >>>>>>> >> >>>>>>> How do I debug this issue ? >> >>>>>>> >> >>>>>>> Some more details: >> >>>>>>> >> >>>>>>> * I have varnishncsa turned on with following flags. >> >>>>>>> >> >>>>>>> ExecStart=/usr/bin/varnishncsa -q 'not (ReqURL ~ >> "\.(js|css|png|jpeg|jpg|ico|gif|swf|woff)" or ReqHeader:User-Agent ~ >> "HTTP-Monitor" or ReqHeader:User-Agent ~ "Monitoring/1.1")' -a -w >> /var/log/varnish/varnishncsa.log -D -f /etc/sysconfig/varnishncsa >> >>>>>>> >> >>>>>>> Here is the entry in varnish log during that time. >> >>>>>>> >> >>>>>>> X.X.X.X, LB.IP.0.0 - - [25/Dec/2019:00:26:40 -0600] 2629363 "POST >> http://mydomain.com/LetsCelebrate HTTP/1.1" 503 1288 " >> https://referringdomain.com/LetsCelebrate?hostingUrl=https%3A%2F%2Fxy.abc.com" >> "Mozilla/5.0 (Linux; Android 9; SM-A920F) AppleWebKit/537.36 (KHTML, like >> Gecko) Chrome/79.0.3945.93 Mobile Safari/537.36" "mydomain.com" miss >> >>>>>>> >> >>>>>>> In .vcl file I have following set. >> >>>>>>> .first_byte_timeout = 120s; >> >>>>>>> >> >>>>>>> Now, I am not logging varnish to apache requests ( which backend >> fetch ). >> >>>>>>> As you can see above I am only logging client requests. >> >>>>>>> >> >>>>>>> Is their a way to log client requests like above but then also >> log backend fetch failures to another log file ? >> >>>>>>> [ Running 2 instances of varnishncsa ? ] >> >>>>>>> >> >>>>>>> Any pointers ? >> >>>>>>> >> >>>>>>> Please let me know. >> >>>>>>> >> >>>>>>> Thanks. >> >>>>>>> >> >>>>>>> _______________________________________________ >> >>>>>>> varnish-misc mailing list >> >>>>>>> varnish-misc at varnish-cache.org >> >>>>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > >> > _______________________________________________ >> > varnish-misc mailing list >> > varnish-misc at varnish-cache.org >> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: