From lcruzero at gmail.com Tue Jul 2 20:48:44 2019 From: lcruzero at gmail.com (L Cruzero) Date: Tue, 2 Jul 2019 16:48:44 -0400 Subject: Pull Hash_data into header Message-ID: Hi, I'm trying to put hash_data info onto a header for debugging purposes: varnishd -V varnishd (varnish-plus-6.0.3r6 revision bd7ae68d24df8ec166bd40b47726313c235b70a1) I'm able to see the hash values via varnishlog after setting these startup parameters: DAEMON_OPTS=" -p vsl_mask=+Hash" - VCL_return hash - VCL_call HASH - Hash "/ps01/3000_complete.m3u8%00" - Hash "/ps01/3000_complete.m3u8%00" however I'm not able to load the data onto a header under sub vcl_deliver with: set resp.http.X.MY-DEBUG = req.http.X-MY-DEBUG + " HASH: " + req.hash; Thanks, Luis -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume at varnish-software.com Tue Jul 2 22:12:03 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Tue, 2 Jul 2019 15:12:03 -0700 Subject: Pull Hash_data into header In-Reply-To: References: Message-ID: Hi, This is because req.hash is a BLOB, and isn't coverted implicitly into a string. Good news is you have access to vmod_blob that's bundled with varnish. Here a self-contained test, you can run it using "varnishtest foo.vtc": varnishtest "req.hash example" server s1 { rxreq txresp } -start varnish v1 -vcl+backend { import blob; sub vcl_deliver { set resp.http.hash = blob.encode(HEX, blob=req.hash); } } -start client c1 { txreq -url /url/1 rxresp expect resp.http.hash == "fd1d762b1bfbb23067ee15d0d0a8b026d51a99147b26d01f95215b4b277d6e8f" } -run Three things to note: - vsl_mask only enable the logging side of things, so it's independent from the VCL code - please show us an error message, "it doesn't work" isn't an amazing explanation of what failed - while a bit hardcore, you usually can look at https://github.com/varnishcache/varnish-cache/tree/master/bin/varnishtest/tests for VCL examples cheers, -- Guillaume Quintard On Tue, Jul 2, 2019 at 1:50 PM L Cruzero wrote: > Hi, > > I'm trying to put hash_data info onto a header for debugging purposes: > > varnishd -V > varnishd (varnish-plus-6.0.3r6 revision > bd7ae68d24df8ec166bd40b47726313c235b70a1) > > I'm able to see the hash values via varnishlog after setting these startup > parameters: DAEMON_OPTS=" -p vsl_mask=+Hash" > > - VCL_return hash > - VCL_call HASH > - Hash "/ps01/3000_complete.m3u8%00" > - Hash "/ps01/3000_complete.m3u8%00" > > however I'm not able to load the data onto a header under sub vcl_deliver > with: > > set resp.http.X.MY-DEBUG = req.http.X-MY-DEBUG + " HASH: " + req.hash; > > > > > Thanks, > Luis > > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kokoniimasu at gmail.com Wed Jul 3 06:19:51 2019 From: kokoniimasu at gmail.com (kokoniimasu) Date: Wed, 3 Jul 2019 15:19:51 +0900 Subject: varnishncsa show ESI request Message-ID: Hi, I'm using varnishncsa, but it's difficult to investigate because does not show ESI sub-request in association. I added show ESI option (-e) and Varnish:vxid_parent x format. wip patch https://gist.github.com/xcir/78c02d50681a4a78bc8c79ffd8421c8c - Output sample # varnishncsa -c -e -g request -F '%{Varnish:side}x %D v:%{Varnish:vxid}x vp:%{Varnish:vxid_parent}x %{Varnish:hitmiss}x "%r" %s %b' c 1547310 v:193913332 vp:0 miss "GET http://**** HTTP/1.1" 200 7985 <-- slow base page c 577 v:193913334 vp:193913332 hit "GET http://**** HTTP/1.1" 200 163 c 48233 v:193913336 vp:193913332 miss "GET http://**** HTTP/1.1" 200 1561 <-- fast sub-request c 527 v:193913338 vp:193913332 hit "GET http://**** HTTP/1.1" 200 164 # varnishncsa -b -c -e -g request -F '%{Varnish:side}x %D v:%{Varnish:vxid}x vp:%{Varnish:vxid_parent}x %{Varnish:hitmiss}x "%r" %s %b' c 2262459 v:221188118 vp:0 miss "GET http://**** HTTP/1.1" 200 18168 b 817749 v:221188119 vp:221188118 - "GET http://**** HTTP/1.1" 200 0 c 550 v:221188120 vp:221188118 hit "GET http://**** HTTP/1.1" 200 163 c 295 v:221188122 vp:221188118 hit "GET http://**** HTTP/1.1" 200 1565 c 494 v:221188123 vp:221188118 hit "GET http://**** HTTP/1.1" 200 162 c 262 v:221188125 vp:221188118 hit "GET http://**** HTTP/1.1" 200 0 c 1242 v:221188127 vp:221188118 hit "GET http://**** HTTP/1.1" 200 1488 c 211 v:221188129 vp:221188118 hit "GET http://**** HTTP/1.1" 200 44 c 1316055 v:221188130 vp:221188118 miss "GET http://**** HTTP/1.1" 200 4353 <-- slow sub-request c 34039 v:221188132 vp:221188118 hit "GET http://**** HTTP/1.1" 200 788 c 18784 v:221188134 vp:221188118 hit "GET http://**** HTTP/1.1" 200 0 c 32942 v:221188135 vp:221188118 hit "GET http://**** HTTP/1.1" 200 774 c 22181 v:221188136 vp:221188118 hit "GET http://**** HTTP/1.1" 200 163 c 5766 v:221188138 vp:221188118 hit "GET http://**** HTTP/1.1" 200 0 c 8005 v:221188140 vp:221188118 miss "GET http://**** HTTP/1.1" 200 163 <-- fast sub-request c 1626 v:221188142 vp:221188118 hit "GET http://**** HTTP/1.1" 200 163 c 230 v:221188144 vp:221188118 hit "GET http://**** HTTP/1.1" 200 739 b 1315672 v:221188131 vp:221188130 - "GET http://**** HTTP/1.1" 200 0 b 1187 v:221188141 vp:221188140 - "GET http://**** HTTP/1.1" 200 0 I?m thinking about whether to put PR. Can I get your opinion? Regards, -- Shohei Tanaka(@xcir) http://blog.xcir.net/ (JP) From guillaume at varnish-software.com Wed Jul 3 14:29:47 2019 From: guillaume at varnish-software.com (Guillaume Quintard) Date: Wed, 3 Jul 2019 07:29:47 -0700 Subject: varnishncsa show ESI request In-Reply-To: References: Message-ID: Hi, Unless I'm mistaken, what you are asking for is the first item in the EXAMPLE section of "man varnishncsa": Log the second field of the Begin record, corresponding to the VXID of the parent transaction: varnishncsa -F "%{VSL:Begin[2]}x" Am I missing something? -- Guillaume Quintard On Tue, Jul 2, 2019 at 11:21 PM kokoniimasu wrote: > Hi, > > I'm using varnishncsa, but it's difficult to investigate because does > not show ESI sub-request in association. > I added show ESI option (-e) and Varnish:vxid_parent x format. > > wip patch > https://gist.github.com/xcir/78c02d50681a4a78bc8c79ffd8421c8c > > - Output sample > > # varnishncsa -c -e -g request -F '%{Varnish:side}x %D > v:%{Varnish:vxid}x vp:%{Varnish:vxid_parent}x %{Varnish:hitmiss}x "%r" > %s %b' > c 1547310 v:193913332 vp:0 miss "GET http://**** HTTP/1.1" 200 7985 > <-- slow base page > c 577 v:193913334 vp:193913332 hit "GET http://**** HTTP/1.1" 200 163 > c 48233 v:193913336 vp:193913332 miss "GET http://**** HTTP/1.1" 200 > 1561 <-- fast sub-request > c 527 v:193913338 vp:193913332 hit "GET http://**** HTTP/1.1" 200 164 > > # varnishncsa -b -c -e -g request -F '%{Varnish:side}x %D > v:%{Varnish:vxid}x vp:%{Varnish:vxid_parent}x %{Varnish:hitmiss}x "%r" > %s %b' > c 2262459 v:221188118 vp:0 miss "GET http://**** HTTP/1.1" 200 18168 > b 817749 v:221188119 vp:221188118 - "GET http://**** HTTP/1.1" 200 0 > c 550 v:221188120 vp:221188118 hit "GET http://**** HTTP/1.1" 200 163 > c 295 v:221188122 vp:221188118 hit "GET http://**** HTTP/1.1" 200 1565 > c 494 v:221188123 vp:221188118 hit "GET http://**** HTTP/1.1" 200 162 > c 262 v:221188125 vp:221188118 hit "GET http://**** HTTP/1.1" 200 0 > c 1242 v:221188127 vp:221188118 hit "GET http://**** HTTP/1.1" 200 1488 > c 211 v:221188129 vp:221188118 hit "GET http://**** HTTP/1.1" 200 44 > c 1316055 v:221188130 vp:221188118 miss "GET http://**** HTTP/1.1" 200 > 4353 <-- slow sub-request > c 34039 v:221188132 vp:221188118 hit "GET http://**** HTTP/1.1" 200 788 > c 18784 v:221188134 vp:221188118 hit "GET http://**** HTTP/1.1" 200 0 > c 32942 v:221188135 vp:221188118 hit "GET http://**** HTTP/1.1" 200 774 > c 22181 v:221188136 vp:221188118 hit "GET http://**** HTTP/1.1" 200 163 > c 5766 v:221188138 vp:221188118 hit "GET http://**** HTTP/1.1" 200 0 > c 8005 v:221188140 vp:221188118 miss "GET http://**** HTTP/1.1" 200 > 163 <-- fast sub-request > c 1626 v:221188142 vp:221188118 hit "GET http://**** HTTP/1.1" 200 163 > c 230 v:221188144 vp:221188118 hit "GET http://**** HTTP/1.1" 200 739 > b 1315672 v:221188131 vp:221188130 - "GET http://**** HTTP/1.1" 200 0 > b 1187 v:221188141 vp:221188140 - "GET http://**** HTTP/1.1" 200 0 > > I?m thinking about whether to put PR. > Can I get your opinion? > > Regards, > -- > Shohei Tanaka(@xcir) > http://blog.xcir.net/ (JP) > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kokoniimasu at gmail.com Wed Jul 3 15:00:25 2019 From: kokoniimasu at gmail.com (kokoniimasu) Date: Thu, 4 Jul 2019 00:00:25 +0900 Subject: varnishncsa show ESI request In-Reply-To: References: Message-ID: Hi, Guillaume. Thank You for your advice. > varnishncsa -F "%{VSL:Begin[2]}x" As you pointed out, I was mistaken about that. I'm remove vxid_parent format. 2019?7?3?(?) 23:29 Guillaume Quintard : > > Hi, > > Unless I'm mistaken, what you are asking for is the first item in the EXAMPLE section of "man varnishncsa": > > Log the second field of the Begin record, corresponding to the VXID of the parent transaction: > > varnishncsa -F "%{VSL:Begin[2]}x" > > Am I missing something? > > -- > Guillaume Quintard > > > On Tue, Jul 2, 2019 at 11:21 PM kokoniimasu wrote: >> >> Hi, >> >> I'm using varnishncsa, but it's difficult to investigate because does >> not show ESI sub-request in association. >> I added show ESI option (-e) and Varnish:vxid_parent x format. >> >> wip patch >> https://gist.github.com/xcir/78c02d50681a4a78bc8c79ffd8421c8c >> >> - Output sample >> >> # varnishncsa -c -e -g request -F '%{Varnish:side}x %D >> v:%{Varnish:vxid}x vp:%{Varnish:vxid_parent}x %{Varnish:hitmiss}x "%r" >> %s %b' >> c 1547310 v:193913332 vp:0 miss "GET http://**** HTTP/1.1" 200 7985 >> <-- slow base page >> c 577 v:193913334 vp:193913332 hit "GET http://**** HTTP/1.1" 200 163 >> c 48233 v:193913336 vp:193913332 miss "GET http://**** HTTP/1.1" 200 >> 1561 <-- fast sub-request >> c 527 v:193913338 vp:193913332 hit "GET http://**** HTTP/1.1" 200 164 >> >> # varnishncsa -b -c -e -g request -F '%{Varnish:side}x %D >> v:%{Varnish:vxid}x vp:%{Varnish:vxid_parent}x %{Varnish:hitmiss}x "%r" >> %s %b' >> c 2262459 v:221188118 vp:0 miss "GET http://**** HTTP/1.1" 200 18168 >> b 817749 v:221188119 vp:221188118 - "GET http://**** HTTP/1.1" 200 0 >> c 550 v:221188120 vp:221188118 hit "GET http://**** HTTP/1.1" 200 163 >> c 295 v:221188122 vp:221188118 hit "GET http://**** HTTP/1.1" 200 1565 >> c 494 v:221188123 vp:221188118 hit "GET http://**** HTTP/1.1" 200 162 >> c 262 v:221188125 vp:221188118 hit "GET http://**** HTTP/1.1" 200 0 >> c 1242 v:221188127 vp:221188118 hit "GET http://**** HTTP/1.1" 200 1488 >> c 211 v:221188129 vp:221188118 hit "GET http://**** HTTP/1.1" 200 44 >> c 1316055 v:221188130 vp:221188118 miss "GET http://**** HTTP/1.1" 200 >> 4353 <-- slow sub-request >> c 34039 v:221188132 vp:221188118 hit "GET http://**** HTTP/1.1" 200 788 >> c 18784 v:221188134 vp:221188118 hit "GET http://**** HTTP/1.1" 200 0 >> c 32942 v:221188135 vp:221188118 hit "GET http://**** HTTP/1.1" 200 774 >> c 22181 v:221188136 vp:221188118 hit "GET http://**** HTTP/1.1" 200 163 >> c 5766 v:221188138 vp:221188118 hit "GET http://**** HTTP/1.1" 200 0 >> c 8005 v:221188140 vp:221188118 miss "GET http://**** HTTP/1.1" 200 >> 163 <-- fast sub-request >> c 1626 v:221188142 vp:221188118 hit "GET http://**** HTTP/1.1" 200 163 >> c 230 v:221188144 vp:221188118 hit "GET http://**** HTTP/1.1" 200 739 >> b 1315672 v:221188131 vp:221188130 - "GET http://**** HTTP/1.1" 200 0 >> b 1187 v:221188141 vp:221188140 - "GET http://**** HTTP/1.1" 200 0 >> >> I?m thinking about whether to put PR. >> Can I get your opinion? >> >> Regards, >> -- >> Shohei Tanaka(@xcir) >> http://blog.xcir.net/ (JP) >> _______________________________________________ >> varnish-misc mailing list >> varnish-misc at varnish-cache.org >> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From nikolay.bogdanov at flant.com Tue Jul 9 08:10:21 2019 From: nikolay.bogdanov at flant.com (Nikolay Bogdanov) Date: Tue, 9 Jul 2019 11:10:21 +0300 Subject: Varnish and failed upstream Message-ID: Hello. I have one case and I can not find good solution for it. In varnish3 req.backend was saved during restarts, so I can compare old backend property and set other backend after first restart. But in varnish 5 and newer req.backend_hint in vcl_recv is default always. I can not use retry in vcl_backend_response, because this function did not called. How can I fix it? My bad solution: sub vcl_init { new upstream_rr = directors.round_robin(); key_rr.add_backend(srv1); key_rr.add_backend(srv2); } sub vcl_recv { if (req.restarts == 0) { set req.backend_hint = upstream_rr.backend(); } else { if (req.restarts == 1) { std.syslog(180, "RECV IN PROG: set backend srv1"); set req.backend_hint = srv1; } elseif (req.restarts == 2) { std.syslog(180, "RECV IN PROG: set backend srv2"); set req.backend_hint = srv2; } elseif (req.restarts > 2) { std.syslog(180, "RECV IN PROG: use other DC backend default"); set req.backend_hint = default; } } return (hash); } -- -- ??????? ???????? ??????? ?? ??????? http://flant.ru/ +7 (495) 721-10-27, ???. 422 +7 (926) 125-39-69 -------------- next part -------------- An HTML attachment was scrubbed... URL: From dridi at varni.sh Wed Jul 10 16:53:19 2019 From: dridi at varni.sh (Dridi Boukelmoune) Date: Wed, 10 Jul 2019 16:53:19 +0000 Subject: Varnish and failed upstream In-Reply-To: References: Message-ID: On Tue, Jul 9, 2019 at 8:12 AM Nikolay Bogdanov wrote: > > Hello. I have one case and I can not find good solution for it. > In varnish3 req.backend was saved during restarts, so I can compare old backend property and set other backend after first restart. > But in varnish 5 and newer req.backend_hint in vcl_recv is default always. I can not use retry in vcl_backend_response, because this function did not called. > How can I fix it? What you are looking for is probably a circuit breaker, you can implement one with vmod-saintmod: https://github.com/varnish/varnish-modules/blob/master/docs/vmod_saintmode.rst#vmod_saintmode However instead of using it with req, it's probably more efficient to work on the backend side and retry from vcl_backend_fetch or vcl_backend_error. Best, Dridi From Alexandre.Thaveau at mister-auto.com Mon Jul 22 13:29:28 2019 From: Alexandre.Thaveau at mister-auto.com (Alexandre Thaveau) Date: Mon, 22 Jul 2019 15:29:28 +0200 Subject: Very strange varnish statistics after ESI activation Message-ID: Hi, a few weeks ago, i asked the mailing list about my varnish statistics, tracked under prometheus using varnish exporter (varnishstat wrapper). 2 weeks ago, one of our developpers activated ESI using these changes (second modification is for a specific case but i put it in case of) ------------------------------------ # vcl_backend_response is called after a document has been successfully retrieved from the backend sub vcl_backend_response { set beresp.do_esi = true; ------------------------------------ if(req.http.Cookie ~ "[^?]*custom_data_cookie=.*"){ - if(req.url !~ "^/([a-z]{2}/)$" && req.url !~ "^/([a-z]{2}/)(&|\?)" && req.url !~ "^/(&|\?)" && req.url != "/" && req.url !~ "^\/([a-z]{2}+\/)?((?!(ofertas|offres|aanbiedingen|angebote|offers)\b)[^\/]+\/)+([^\/]+\/)+([^\/]+\/)+([^\/]+\/)+$"){ + if(req.url !~ "^/([a-z]{2}/)$" && req.url !~ "^/([a-z]{2}/)(&|\?)" && req.url !~ "^/(&|\?)" && req.url != "/" && req.url !~ "^\/([a-z]{2}+\/)?((?!(ofertas|offres|aanbiedingen|angebote|offers)\b)[^\/]+\/)+([^\/]+\/)+([^\/]+\/)+([^\/]+\/)+$" && req.url !~ "^\/([a-z]{2}\/)?esi\/(.+)?"){ return(pass); } } ------------------------------------ and configure our website menu to be edge side included (at maximum 50 combinations max using country and language used in parameter). After this modification, multiple metrics changed in the supervision which i do not understand : - my number of varnish objects (main_n_object) went from 337 000 entries to 475 000 entries ... i really don't understand this since i thought my cache memory was full before and since we only added 50 ESI keys in the cache with this update - my cache hit (main_cache_hit) went from 2.5 % to 22% ... but pass, miss and synth rates did not move, so the sum of all my varnish outputs are now upper than 110 % ... (before they were equals to 100%) - my backend output (beresp_hdrbytes) total dropped from 2MB to 85KB ... however our websites are working fine Can you help me understand this ? Thanks -- Alexandre THAVEAU *04 72 89 22 11* alexandre.thaveau at mister-auto.com -------------- next part -------------- An HTML attachment was scrubbed... URL: