Varnish doesnt purge
Vlad Rusu
vlad.rusu at lola.tech
Sun Dec 15 16:18:42 UTC 2019
I appreciate Guillaume’s diplomacy :)
Your issue (one of them at least) is that in your cascaded varnish setup
the hash key of the cached url, in both varnish storages, is the one you
use to originally access the asset. Your vcl hash routine might also be a
custom one.. that would be the next thing to look at if what I’m saying
here doesn’t fix it.
So, purge
http://192.168.200.13/Naserfeiz.mp4
and NOT
http://192.168.200.12:8080/Naserfeiz.mp4
when in the cascaded setup
Cheers
On Sat, 14 Dec 2019 at 09:16, hamidreza hosseini <hrhosseini at hotmail.com>
wrote:
> Hi,
> Please annwer my question, I sent you my test
>
> ------------------------------
> *From:* varnish-misc <varnish-misc-bounces+hrhosseini=
> hotmail.com at varnish-cache.org> on behalf of hamidreza hosseini <
> hrhosseini at hotmail.com>
> *Sent:* Wednesday, December 11, 2019 11:49 PM
> *To:* Guillaume Quintard <guillaume at varnish-software.com>
> *Cc:* varnish-misc at varnish-cache.org <varnish-misc at varnish-cache.org>
> *Subject:* Re: Varnish doesnt purge
>
> I do all steps now again, this is the resault:
> STEP ONE:
> First of all i configure a varnish (with file backend) in front of
> swift/Nginx and i ask the url that exist in swift with curl,
>
> varnish-file$curl -I http://192.168.200.12:8080/Naserfeiz.mp4 -H
> "newtrack: yes"
>
> HTTP/1.1 200 OK
> Server: nginx
> Date: Thu, 12 Dec 2019 07:29:32 GMT
> Content-Type: text/plain
> Content-Length: 23521499
> Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT
> ETag: "3a1794b0-166e8db"
> X-Varnish: 2
> Age: 0
> Via: 1.1 varnish (Varnish/6.0)
> Accept-Ranges: bytes
> Connection: keep-alive
>
>
> varnish cach that object and then i purge the url:
>
> varnish-file$ curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4
> -o /dev/null -H "newtrack: yes"
>
> * Trying 192.168.200.12...
> % Total % Received % Xferd Average Speed Time Time Time
> Current
> Dload Upload Total Spent Left
> Speed
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
> 0* Connected to 192.168.200.12 (192.168.200.12) port 8080 (#0)
> > PURGE /Naserfeiz.mp4 HTTP/1.1
> > Host: 192.168.200.12:8080
> > User-Agent: curl/7.47.0
> > Accept: */*
> > newtrack: yes
> >
> < HTTP/1.1 200 Purged
> < Date: Thu, 12 Dec 2019 07:29:16 GMT
> < Server: Varnish
> < X-Varnish: 32770
> < Content-Type: text/html; charset=utf-8
> < Retry-After: 5
> < Content-Length: 240
> < Accept-Ranges: bytes
> < Connection: keep-alive
> <
> { [240 bytes data]
> 100 240 100 240 0 0 124k 0 --:--:-- --:--:-- --:--:--
> 234k
> * Connection #0 to host 192.168.200.12 left intact
>
> The object purged successfully and it didnt exist in cach anymore.
>
> This is varnishlog resault:
>
> varnish-file$ sudo varnishlog -d -g request
> * << Request >> 2
> - Begin req 1 rxreq
> - Timestamp Start: 1576135743.442646 0.000000 0.000000
> - Timestamp Req: 1576135743.442646 0.000000 0.000000
> - ReqStart 192.168.200.12 5432 a0
> - ReqMethod HEAD
> - ReqURL /Naserfeiz.mp4
> - ReqProtocol HTTP/1.1
> - ReqHeader Host: 192.168.200.12:8080
> - ReqHeader User-Agent: curl/7.47.0
> - ReqHeader Accept: */*
> - ReqHeader newtrack: yes
> - ReqHeader X-Forwarded-For: 192.168.200.12
> - VCL_call RECV
> - VCL_return hash
> - VCL_call HASH
> - VCL_return lookup
> - VCL_call MISS
> - VCL_return fetch
> - Link bereq 3 fetch
> - Timestamp Fetch: 1576135743.444347 0.001701 0.001701
> - RespProtocol HTTP/1.1
> - RespStatus 200
> - RespReason OK
> - RespHeader Server: nginx
> - RespHeader Date: Thu, 12 Dec 2019 07:29:32 GMT
> - RespHeader Content-Type: text/plain
> - RespHeader Content-Length: 23521499
> - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT
> - RespHeader ETag: "3a1794b0-166e8db"
> - RespHeader X-Varnish: 2
> - RespHeader Age: 0
> - RespHeader Via: 1.1 varnish (Varnish/6.0)
> - VCL_call DELIVER
> - VCL_return deliver
> - Timestamp Process: 1576135743.444382 0.001736 0.000036
> - RespHeader Accept-Ranges: bytes
> - RespHeader Connection: keep-alive
> - Timestamp Resp: 1576135743.444474 0.001828 0.000092
> - ReqAcct 112 0 112 295 0 295
> - End
> ** << BeReq >> 3
> -- Begin bereq 2 fetch
> -- VCL_use boot
> -- Timestamp Start: 1576135743.442807 0.000000 0.000000
> -- BereqMethod HEAD
> -- BereqURL /Naserfeiz.mp4
> -- BereqProtocol HTTP/1.1
> -- BereqHeader Host: 192.168.200.12:8080
> -- BereqHeader User-Agent: curl/7.47.0
> -- BereqHeader Accept: */*
> -- BereqHeader newtrack: yes
> -- BereqHeader X-Forwarded-For: 192.168.200.12
> -- BereqMethod GET
> -- BereqHeader Accept-Encoding: gzip
> -- BereqHeader X-Varnish: 3
> -- VCL_call BACKEND_FETCH
> -- VCL_return fetch
> -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080
> 192.168.200.12 6846
> -- BackendStart 192.168.200.11 8080
> -- Timestamp Bereq: 1576135743.443586 0.000779 0.000779
> -- Timestamp Beresp: 1576135743.444017 0.001210 0.000431
> -- BerespProtocol HTTP/1.1
> -- BerespStatus 200
> -- BerespReason OK
> -- BerespHeader Server: nginx
> -- BerespHeader Date: Thu, 12 Dec 2019 07:29:32 GMT
> -- BerespHeader Content-Type: text/plain
> -- BerespHeader Content-Length: 23521499
> -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT
> -- BerespHeader Connection: keep-alive
> -- BerespHeader ETag: "3a1794b0-166e8db"
> -- BerespHeader Accept-Ranges: bytes
> -- TTL RFC 120 10 0 1576135743 1576135743 1576135772 0 0
> cacheable
> -- VCL_call BACKEND_RESPONSE
> -- TTL VCL 86400 10 0 1576135743 cacheable
> -- VCL_return deliver
> -- Storage file s0
> -- Fetch_Body 3 length stream
> -- BackendReuse 27 boot.swift_proxy_1
> -- Timestamp BerespBody: 1576135743.747835 0.305028 0.303818
> -- Length 23521499
> -- BereqAcct 181 0 181 241 23521499 23521740
> -- End
>
> * << Request >> 32770
> - Begin req 32769 rxreq
> - Timestamp Start: 1576135756.680792 0.000000 0.000000
> - Timestamp Req: 1576135756.680792 0.000000 0.000000
> - ReqStart 192.168.200.12 5438 a0
> - ReqMethod PURGE
> - ReqURL /Naserfeiz.mp4
> - ReqProtocol HTTP/1.1
> - ReqHeader Host: 192.168.200.12:8080
> - ReqHeader User-Agent: curl/7.47.0
> - ReqHeader Accept: */*
> - ReqHeader newtrack: yes
> - ReqHeader X-Forwarded-For: 192.168.200.12
> - VCL_call RECV
> - VCL_return purge
> - VCL_call HASH
> - VCL_return lookup
> - VCL_call PURGE
> - VCL_return synth
> - Timestamp Process: 1576135756.680982 0.000190 0.000190
> - RespHeader Date: Thu, 12 Dec 2019 07:29:16 GMT
> - RespHeader Server: Varnish
> - RespHeader X-Varnish: 32770
> - RespProtocol HTTP/1.1
> - RespStatus 200
> - RespReason OK
> - RespReason Purged
> - VCL_call SYNTH
> - RespHeader Content-Type: text/html; charset=utf-8
> - RespHeader Retry-After: 5
> - VCL_return deliver
> - RespHeader Content-Length: 240
> - Storage malloc Transient
> - RespHeader Accept-Ranges: bytes
> - RespHeader Connection: keep-alive
> - Timestamp Resp: 1576135756.681190 0.000399 0.000208
> - ReqAcct 113 0 113 218 240 458
> - End
>
> * << Request >> 32772
> - Begin req 32771 rxreq
> - Timestamp Start: 1576135769.304679 0.000000 0.000000
> - Timestamp Req: 1576135769.304679 0.000000 0.000000
> - ReqStart 192.168.200.12 5440 a0
> - ReqMethod HEAD
> - ReqURL /Naserfeiz.mp4
> - ReqProtocol HTTP/1.1
> - ReqHeader Host: 192.168.200.12:8080
> - ReqHeader User-Agent: curl/7.47.0
> - ReqHeader Accept: */*
> - ReqHeader newtrack: yes
> - ReqHeader X-Forwarded-For: 192.168.200.12
> - VCL_call RECV
> - VCL_return hash
> - VCL_call HASH
> - VCL_return lookup
> - VCL_call MISS
> - VCL_return fetch
> - Link bereq 32773 fetch
> - Timestamp Fetch: 1576135769.306630 0.001951 0.001951
> - RespProtocol HTTP/1.1
> - RespStatus 200
> - RespReason OK
> - RespHeader Server: nginx
> - RespHeader Date: Thu, 12 Dec 2019 07:29:58 GMT
> - RespHeader Content-Type: text/plain
> - RespHeader Content-Length: 23521499
> - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT
> - RespHeader ETag: "3a1794b0-166e8db"
> - RespHeader X-Varnish: 32772
> - RespHeader Age: 0
> - RespHeader Via: 1.1 varnish (Varnish/6.0)
> - VCL_call DELIVER
> - VCL_return deliver
> - Timestamp Process: 1576135769.306657 0.001978 0.000027
> - RespHeader Accept-Ranges: bytes
> - RespHeader Connection: keep-alive
> - Timestamp Resp: 1576135769.306748 0.002069 0.000091
> - ReqAcct 112 0 112 299 0 299
> - End
> ** << BeReq >> 32773
> -- Begin bereq 32772 fetch
> -- VCL_use boot
> -- Timestamp Start: 1576135769.304834 0.000000 0.000000
> -- BereqMethod HEAD
> -- BereqURL /Naserfeiz.mp4
> -- BereqProtocol HTTP/1.1
> -- BereqHeader Host: 192.168.200.12:8080
> -- BereqHeader User-Agent: curl/7.47.0
> -- BereqHeader Accept: */*
> -- BereqHeader newtrack: yes
> -- BereqHeader X-Forwarded-For: 192.168.200.12
> -- BereqMethod GET
> -- BereqHeader Accept-Encoding: gzip
> -- BereqHeader X-Varnish: 32773
> -- VCL_call BACKEND_FETCH
> -- VCL_return fetch
> -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080
> 192.168.200.12 6846
> -- BackendStart 192.168.200.11 8080
> -- Timestamp Bereq: 1576135769.304993 0.000160 0.000160
> -- Timestamp Beresp: 1576135769.306170 0.001336 0.001176
> -- BerespProtocol HTTP/1.1
> -- BerespStatus 200
> -- BerespReason OK
> -- BerespHeader Server: nginx
> -- BerespHeader Date: Thu, 12 Dec 2019 07:29:58 GMT
> -- BerespHeader Content-Type: text/plain
> -- BerespHeader Content-Length: 23521499
> -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT
> -- BerespHeader Connection: keep-alive
> -- BerespHeader ETag: "3a1794b0-166e8db"
> -- BerespHeader Accept-Ranges: bytes
> -- TTL RFC 120 10 0 1576135769 1576135769 1576135798 0 0
> cacheable
> -- VCL_call BACKEND_RESPONSE
> -- TTL VCL 86400 10 0 1576135769 cacheable
> -- VCL_return deliver
> -- Storage file s0
> -- Fetch_Body 3 length stream
> -- BackendReuse 27 boot.swift_proxy_1
> -- Timestamp BerespBody: 1576135769.645490 0.340657 0.339321
> -- Length 23521499
> -- BereqAcct 185 0 185 241 23521499 23521740
> -- End
>
>
> ########
>
> STEP TWO:
> First i restart varnish file and varnishlog then:
> I add another varnish (with ram backend) in fornt of the varnish file and
> then i ask the url again in varnish ram (because it is my frontend server
> and users can access media though this server):
>
> varnish-ram$ curl -I http://192.168.200.13/Naserfeiz.mp4 -H "newtrack:
> yes"
> HTTP/1.1 200 OK
> Server: nginx
> Date: Thu, 12 Dec 2019 07:33:05 GMT
> Content-Type: text/plain
> Content-Length: 23521499
> Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT
> ETag: "3a1794b0-166e8db"
> X-Varnish: 2
> Via: 1.1 varnish (Varnish/6.0)
> X-Varnish: 2
> Age: 0
> Via: 1.1 varnish-v4
> Accept-Ranges: bytes
> Connection: keep-alive
>
>
> First varnish file cached the url and then Varnish ram fetched the object
> from varnish file and cached it , then i purged that url in varnish ram and
> it purged successfuly and it exactly removed from cached:
>
> varnish-ram:~$ curl -v -k -X PURGE http://192.168.200.13/Naserfeiz.mp4 -o
> /dev/null -H "newtrack: yes"
> * Trying 192.168.200.13...
> % Total % Received % Xferd Average Speed Time Time Time
> Current
> Dload Upload Total Spent Left
> Speed
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
> 0* Connected to 192.168.200.13 (192.168.200.13) port 80 (#0)
> > PURGE /Naserfeiz.mp4 HTTP/1.1
> > Host: 192.168.200.13
> > User-Agent: curl/7.47.0
> > Accept: */*
> > newtrack: yes
> >
> < HTTP/1.1 200 Purged
> < Date: Thu, 12 Dec 2019 07:33:46 GMT
> < Server: Varnish
> < X-Varnish: 32770
> < Content-Type: text/html; charset=utf-8
> < Retry-After: 5
> < Content-Length: 240
> < Accept-Ranges: bytes
> < Connection: keep-alive
> <
> { [240 bytes data]
> 100 240 100 240 0 0 73846 0 --:--:-- --:--:-- --:--:--
> 117k
> * Connection #0 to host 192.168.200.13 left intact
>
>
> Then i purged the url in varnish file it shows "200 purged" but it still
> exist in varnish file!!!!!!:
>
> varnish-file$ curl -v -k -X PURGE http://192.168.200.12:8080/Naserfeiz.mp4
> -o /dev/null -H "newtrack: yes"
> * Trying 192.168.200.12...
> % Total % Received % Xferd Average Speed Time Time Time
> Current
> Dload Upload Total Spent Left
> Speed
> 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:--
> 0* Connected to 192.168.200.12 (192.168.200.12) port 8080 (#0)
> > PURGE /Naserfeiz.mp4 HTTP/1.1
> > Host: 192.168.200.12:8080
> > User-Agent: curl/7.47.0
> > Accept: */*
> > newtrack: yes
> >
> < HTTP/1.1 200 Purged
> < Date: Thu, 12 Dec 2019 07:33:39 GMT
> < Server: Varnish
> < X-Varnish: 5
> < Content-Type: text/html; charset=utf-8
> < Retry-After: 5
> < Content-Length: 236
> < Accept-Ranges: bytes
> < Connection: keep-alive
> <
> { [236 bytes data]
> 100 236 100 236 0 0 110k 0 --:--:-- --:--:-- --:--:--
> 230k
> * Connection #0 to host 192.168.200.12 left intact
>
>
> This is varnishlog resault in varnish file:
>
> varnish-file:~$ sudo varnishlog -d -g request
> * << Request >> 2
> - Begin req 1 rxreq
> - Timestamp Start: 1576135956.460768 0.000000 0.000000
> - Timestamp Req: 1576135956.460768 0.000000 0.000000
> - ReqStart 192.168.200.13 22300 a0
> - ReqMethod GET
> - ReqURL /Naserfeiz.mp4
> - ReqProtocol HTTP/1.1
> - ReqHeader Host: 192.168.200.13
> - ReqHeader User-Agent: curl/7.47.0
> - ReqHeader Accept: */*
> - ReqHeader newtrack: yes
> - ReqHeader X-Forwarded-For: 192.168.200.13
> - ReqHeader Accept-Encoding: gzip
> - ReqHeader X-Varnish: 3
> - ReqUnset X-Forwarded-For: 192.168.200.13
> - ReqHeader X-Forwarded-For: 192.168.200.13, 192.168.200.13
> - VCL_call RECV
> - VCL_return hash
> - VCL_call HASH
> - VCL_return lookup
> - VCL_call MISS
> - VCL_return fetch
> - Link bereq 3 fetch
> - Timestamp Fetch: 1576135956.463065 0.002297 0.002297
> - RespProtocol HTTP/1.1
> - RespStatus 200
> - RespReason OK
> - RespHeader Server: nginx
> - RespHeader Date: Thu, 12 Dec 2019 07:33:05 GMT
> - RespHeader Content-Type: text/plain
> - RespHeader Content-Length: 23521499
> - RespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT
> - RespHeader ETag: "3a1794b0-166e8db"
> - RespHeader X-Varnish: 2
> - RespHeader Age: 0
> - RespHeader Via: 1.1 varnish (Varnish/6.0)
> - VCL_call DELIVER
> - VCL_return deliver
> - Timestamp Process: 1576135956.463114 0.002346 0.000049
> - RespHeader Accept-Ranges: bytes
> - RespHeader Connection: keep-alive
> - Timestamp Resp: 1576135956.758920 0.298152 0.295806
> - ReqAcct 176 0 176 295 23521499 23521794
> - End
> ** << BeReq >> 3
> -- Begin bereq 2 fetch
> -- VCL_use boot
> -- Timestamp Start: 1576135956.461099 0.000000 0.000000
> -- BereqMethod GET
> -- BereqURL /Naserfeiz.mp4
> -- BereqProtocol HTTP/1.1
> -- BereqHeader Host: 192.168.200.13
> -- BereqHeader User-Agent: curl/7.47.0
> -- BereqHeader Accept: */*
> -- BereqHeader newtrack: yes
> -- BereqHeader Accept-Encoding: gzip
> -- BereqHeader X-Varnish: 3
> -- BereqHeader X-Forwarded-For: 192.168.200.13, 192.168.200.13
> -- BereqHeader X-Varnish: 3
> -- VCL_call BACKEND_FETCH
> -- VCL_return fetch
> -- BackendOpen 27 boot.swift_proxy_1 192.168.200.11 8080
> 192.168.200.12 6860
> -- BackendStart 192.168.200.11 8080
> -- Timestamp Bereq: 1576135956.461955 0.000856 0.000856
> -- Timestamp Beresp: 1576135956.462607 0.001508 0.000652
> -- BerespProtocol HTTP/1.1
> -- BerespStatus 200
> -- BerespReason OK
> -- BerespHeader Server: nginx
> -- BerespHeader Date: Thu, 12 Dec 2019 07:33:05 GMT
> -- BerespHeader Content-Type: text/plain
> -- BerespHeader Content-Length: 23521499
> -- BerespHeader Last-Modified: Sun, 19 Nov 2000 08:52:00 GMT
> -- BerespHeader Connection: keep-alive
> -- BerespHeader ETag: "3a1794b0-166e8db"
> -- BerespHeader Accept-Ranges: bytes
> -- TTL RFC 120 10 0 1576135956 1576135956 1576135985 0 0
> cacheable
> -- VCL_call BACKEND_RESPONSE
> -- TTL VCL 86400 10 0 1576135956 cacheable
> -- VCL_return deliver
> -- Storage file s0
> -- Fetch_Body 3 length stream
> -- BackendReuse 27 boot.swift_proxy_1
> -- Timestamp BerespBody: 1576135956.758956 0.297858 0.296349
> -- Length 23521499
> -- BereqAcct 206 0 206 241 23521499 23521740
> -- End
>
> * << Request >> 5
> - Begin req 4 rxreq
> - Timestamp Start: 1576136019.230666 0.000000 0.000000
> - Timestamp Req: 1576136019.230666 0.000000 0.000000
> - ReqStart 192.168.200.12 5452 a0
> - ReqMethod PURGE
> - ReqURL /Naserfeiz.mp4
> - ReqProtocol HTTP/1.1
> - ReqHeader Host: 192.168.200.12:8080
> - ReqHeader User-Agent: curl/7.47.0
> - ReqHeader Accept: */*
> - ReqHeader newtrack: yes
> - ReqHeader X-Forwarded-For: 192.168.200.12
> - VCL_call RECV
> - VCL_return purge
> - VCL_call HASH
> - VCL_return lookup
> - VCL_call PURGE
> - VCL_return synth
> - Timestamp Process: 1576136019.230863 0.000198 0.000198
> - RespHeader Date: Thu, 12 Dec 2019 07:33:39 GMT
> - RespHeader Server: Varnish
> - RespHeader X-Varnish: 5
> - RespProtocol HTTP/1.1
> - RespStatus 200
> - RespReason OK
> - RespReason Purged
> - VCL_call SYNTH
> - RespHeader Content-Type: text/html; charset=utf-8
> - RespHeader Retry-After: 5
> - VCL_return deliver
> - RespHeader Content-Length: 236
> - Storage malloc Transient
> - RespHeader Accept-Ranges: bytes
> - RespHeader Connection: keep-alive
> - Timestamp Resp: 1576136019.231138 0.000473 0.000275
> - ReqAcct 113 0 113 214 236 450
> - End
>
> ########
> PLEASE test this scenario for yourself and you can see that i do correct
> and the object wont remove from varnish-file
> Please do this scenario for yourself
>
> ------------------------------
> *From:* Guillaume Quintard <guillaume at varnish-software.com>
> *Sent:* Wednesday, December 11, 2019 9:44 AM
> *To:* hamidreza hosseini <hrhosseini at hotmail.com>
> *Cc:* varnish-misc at varnish-cache.org <varnish-misc at varnish-cache.org>
> *Subject:* Re: Varnish doesnt purge
>
> Yes it does, we have already proved it./
>
> Each instance purges and will refetch the object for its respective
> backend. I explained how to test: start with the origin, change the data
> and check that you actually get the new object. Then add a varnish layer,
> purge, check, repeat.
>
> You are testing everything at once and are unable to isolate the issue
> because you refuse to take things step by step, and spamming won't change
> that.
>
> I'm sorry but until you have tested each layer individually and identified
> the one that fails, I won't be able to help.
>
> --
> Guillaume Quintard
>
>
> On Wed, Dec 11, 2019 at 3:08 PM hamidreza hosseini <hrhosseini at hotmail.com>
> wrote:
>
> Hi,
> please help me to solve this problem, this is the link that i explaine my
> question completely:
>
> https://varnish-cache.org/lists/pipermail/varnish-misc/2019-December/026754.html
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
--
Sent from my iPhone
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20191215/93e87d94/attachment-0001.html>
More information about the varnish-misc
mailing list