From lcruzero at gmail.com  Tue Jul  2 20:48:44 2019
From: lcruzero at gmail.com (L Cruzero)
Date: Tue, 2 Jul 2019 16:48:44 -0400
Subject: Pull Hash_data into header
Message-ID: <CAO+Gocc6Jii8TKATucuR3NxiB2+5fFxReN=ZNZ8A=EtyK9wKQA@mail.gmail.com>

Hi,

I'm trying to put hash_data info onto a header for debugging purposes:

varnishd -V
varnishd (varnish-plus-6.0.3r6 revision
bd7ae68d24df8ec166bd40b47726313c235b70a1)

I'm able to see the hash values via varnishlog after setting these startup
parameters:  DAEMON_OPTS=" -p vsl_mask=+Hash"

-   VCL_return     hash
-   VCL_call       HASH
-   Hash           "/ps01/3000_complete.m3u8%00"
-   Hash           "/ps01/3000_complete.m3u8%00"

however I'm not able to load the data onto a header under sub vcl_deliver
with:

set resp.http.X.MY-DEBUG = req.http.X-MY-DEBUG + " HASH: "  + req.hash;




Thanks,
Luis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20190702/fc33588b/attachment.html>

From guillaume at varnish-software.com  Tue Jul  2 22:12:03 2019
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Tue, 2 Jul 2019 15:12:03 -0700
Subject: Pull Hash_data into header
In-Reply-To: <CAO+Gocc6Jii8TKATucuR3NxiB2+5fFxReN=ZNZ8A=EtyK9wKQA@mail.gmail.com>
References: <CAO+Gocc6Jii8TKATucuR3NxiB2+5fFxReN=ZNZ8A=EtyK9wKQA@mail.gmail.com>
Message-ID: <CAJ6ZYQx_DB=YWVyiZpP8oyJx61D=umimczMoLNN6XpD7t3c5YA@mail.gmail.com>

Hi,

This is because req.hash is a BLOB, and isn't coverted implicitly into a
string. Good news is you have access to vmod_blob that's bundled with
varnish.

Here a self-contained test, you can run it using "varnishtest foo.vtc":

varnishtest "req.hash example"

server s1 {
  rxreq
  txresp
} -start

varnish v1 -vcl+backend {
  import blob;
  sub vcl_deliver {
    set resp.http.hash = blob.encode(HEX, blob=req.hash);
  }
} -start


client c1 {
  txreq -url /url/1
  rxresp
  expect resp.http.hash ==
"fd1d762b1bfbb23067ee15d0d0a8b026d51a99147b26d01f95215b4b277d6e8f"
} -run


Three things to note:
- vsl_mask only enable the logging side of things, so it's independent from
the VCL code
- please show us an error message, "it doesn't work" isn't an amazing
explanation of what failed
- while a bit hardcore, you usually can look at
https://github.com/varnishcache/varnish-cache/tree/master/bin/varnishtest/tests
for
VCL examples

cheers,
-- 
Guillaume Quintard


On Tue, Jul 2, 2019 at 1:50 PM L Cruzero <lcruzero at gmail.com> wrote:

> Hi,
>
> I'm trying to put hash_data info onto a header for debugging purposes:
>
> varnishd -V
> varnishd (varnish-plus-6.0.3r6 revision
> bd7ae68d24df8ec166bd40b47726313c235b70a1)
>
> I'm able to see the hash values via varnishlog after setting these startup
> parameters:  DAEMON_OPTS=" -p vsl_mask=+Hash"
>
> -   VCL_return     hash
> -   VCL_call       HASH
> -   Hash           "/ps01/3000_complete.m3u8%00"
> -   Hash           "/ps01/3000_complete.m3u8%00"
>
> however I'm not able to load the data onto a header under sub vcl_deliver
> with:
>
> set resp.http.X.MY-DEBUG = req.http.X-MY-DEBUG + " HASH: "  + req.hash;
>
>
>
>
> Thanks,
> Luis
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20190702/4ec2393d/attachment.html>

From kokoniimasu at gmail.com  Wed Jul  3 06:19:51 2019
From: kokoniimasu at gmail.com (kokoniimasu)
Date: Wed, 3 Jul 2019 15:19:51 +0900
Subject: varnishncsa show ESI request
Message-ID: <CAJihfGCS8Xn1LSi7bDc_BYh4ZWzDUJqPzbYees8F4zCfgjaZqQ@mail.gmail.com>

Hi,

I'm using varnishncsa, but it's difficult to investigate because does
not show ESI sub-request in association.
I added show ESI option (-e) and Varnish:vxid_parent x format.

wip patch
https://gist.github.com/xcir/78c02d50681a4a78bc8c79ffd8421c8c

- Output sample

# varnishncsa -c -e -g request -F '%{Varnish:side}x %D
v:%{Varnish:vxid}x vp:%{Varnish:vxid_parent}x %{Varnish:hitmiss}x "%r"
%s %b'
c 1547310 v:193913332 vp:0 miss "GET http://**** HTTP/1.1" 200 7985
   <-- slow base page
c 577 v:193913334 vp:193913332 hit "GET http://**** HTTP/1.1" 200 163
c 48233 v:193913336 vp:193913332 miss "GET http://**** HTTP/1.1" 200
1561 <-- fast sub-request
c 527 v:193913338 vp:193913332 hit "GET http://**** HTTP/1.1" 200 164

# varnishncsa -b -c -e -g request -F '%{Varnish:side}x %D
v:%{Varnish:vxid}x vp:%{Varnish:vxid_parent}x %{Varnish:hitmiss}x "%r"
%s %b'
c 2262459 v:221188118 vp:0 miss "GET http://**** HTTP/1.1" 200 18168
b 817749 v:221188119 vp:221188118 - "GET http://**** HTTP/1.1" 200 0
c 550 v:221188120 vp:221188118 hit "GET http://**** HTTP/1.1" 200 163
c 295 v:221188122 vp:221188118 hit "GET http://**** HTTP/1.1" 200 1565
c 494 v:221188123 vp:221188118 hit "GET http://**** HTTP/1.1" 200 162
c 262 v:221188125 vp:221188118 hit "GET http://**** HTTP/1.1" 200 0
c 1242 v:221188127 vp:221188118 hit "GET http://**** HTTP/1.1" 200 1488
c 211 v:221188129 vp:221188118 hit "GET http://**** HTTP/1.1" 200 44
c 1316055 v:221188130 vp:221188118 miss "GET http://**** HTTP/1.1" 200
4353 <-- slow sub-request
c 34039 v:221188132 vp:221188118 hit "GET http://**** HTTP/1.1" 200 788
c 18784 v:221188134 vp:221188118 hit "GET http://**** HTTP/1.1" 200 0
c 32942 v:221188135 vp:221188118 hit "GET http://**** HTTP/1.1" 200 774
c 22181 v:221188136 vp:221188118 hit "GET http://**** HTTP/1.1" 200 163
c 5766 v:221188138 vp:221188118 hit "GET http://**** HTTP/1.1" 200 0
c 8005 v:221188140 vp:221188118 miss "GET http://**** HTTP/1.1" 200
163 <-- fast sub-request
c 1626 v:221188142 vp:221188118 hit "GET http://**** HTTP/1.1" 200 163
c 230 v:221188144 vp:221188118 hit "GET http://**** HTTP/1.1" 200 739
b 1315672 v:221188131 vp:221188130 - "GET http://**** HTTP/1.1" 200 0
b 1187 v:221188141 vp:221188140 - "GET http://**** HTTP/1.1" 200 0

I?m thinking about whether to put PR.
Can I get your opinion?

Regards,
--
Shohei Tanaka(@xcir)
http://blog.xcir.net/ (JP)

From guillaume at varnish-software.com  Wed Jul  3 14:29:47 2019
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Wed, 3 Jul 2019 07:29:47 -0700
Subject: varnishncsa show ESI request
In-Reply-To: <CAJihfGCS8Xn1LSi7bDc_BYh4ZWzDUJqPzbYees8F4zCfgjaZqQ@mail.gmail.com>
References: <CAJihfGCS8Xn1LSi7bDc_BYh4ZWzDUJqPzbYees8F4zCfgjaZqQ@mail.gmail.com>
Message-ID: <CAJ6ZYQzct1Ms4hzCy0x3qRZEGM+Zn2j6jLtAjjU8jOOBM2bZpw@mail.gmail.com>

Hi,

Unless I'm mistaken, what you are asking for is the first item in the
EXAMPLE section of "man varnishncsa":

       Log the second field of the Begin record, corresponding to the VXID
of the parent transaction:

          varnishncsa -F "%{VSL:Begin[2]}x"

Am I missing something?

-- 
Guillaume Quintard


On Tue, Jul 2, 2019 at 11:21 PM kokoniimasu <kokoniimasu at gmail.com> wrote:

> Hi,
>
> I'm using varnishncsa, but it's difficult to investigate because does
> not show ESI sub-request in association.
> I added show ESI option (-e) and Varnish:vxid_parent x format.
>
> wip patch
> https://gist.github.com/xcir/78c02d50681a4a78bc8c79ffd8421c8c
>
> - Output sample
>
> # varnishncsa -c -e -g request -F '%{Varnish:side}x %D
> v:%{Varnish:vxid}x vp:%{Varnish:vxid_parent}x %{Varnish:hitmiss}x "%r"
> %s %b'
> c 1547310 v:193913332 vp:0 miss "GET http://**** HTTP/1.1" 200 7985
>    <-- slow base page
> c 577 v:193913334 vp:193913332 hit "GET http://**** HTTP/1.1" 200 163
> c 48233 v:193913336 vp:193913332 miss "GET http://**** HTTP/1.1" 200
> 1561 <-- fast sub-request
> c 527 v:193913338 vp:193913332 hit "GET http://**** HTTP/1.1" 200 164
>
> # varnishncsa -b -c -e -g request -F '%{Varnish:side}x %D
> v:%{Varnish:vxid}x vp:%{Varnish:vxid_parent}x %{Varnish:hitmiss}x "%r"
> %s %b'
> c 2262459 v:221188118 vp:0 miss "GET http://**** HTTP/1.1" 200 18168
> b 817749 v:221188119 vp:221188118 - "GET http://**** HTTP/1.1" 200 0
> c 550 v:221188120 vp:221188118 hit "GET http://**** HTTP/1.1" 200 163
> c 295 v:221188122 vp:221188118 hit "GET http://**** HTTP/1.1" 200 1565
> c 494 v:221188123 vp:221188118 hit "GET http://**** HTTP/1.1" 200 162
> c 262 v:221188125 vp:221188118 hit "GET http://**** HTTP/1.1" 200 0
> c 1242 v:221188127 vp:221188118 hit "GET http://**** HTTP/1.1" 200 1488
> c 211 v:221188129 vp:221188118 hit "GET http://**** HTTP/1.1" 200 44
> c 1316055 v:221188130 vp:221188118 miss "GET http://**** HTTP/1.1" 200
> 4353 <-- slow sub-request
> c 34039 v:221188132 vp:221188118 hit "GET http://**** HTTP/1.1" 200 788
> c 18784 v:221188134 vp:221188118 hit "GET http://**** HTTP/1.1" 200 0
> c 32942 v:221188135 vp:221188118 hit "GET http://**** HTTP/1.1" 200 774
> c 22181 v:221188136 vp:221188118 hit "GET http://**** HTTP/1.1" 200 163
> c 5766 v:221188138 vp:221188118 hit "GET http://**** HTTP/1.1" 200 0
> c 8005 v:221188140 vp:221188118 miss "GET http://**** HTTP/1.1" 200
> 163 <-- fast sub-request
> c 1626 v:221188142 vp:221188118 hit "GET http://**** HTTP/1.1" 200 163
> c 230 v:221188144 vp:221188118 hit "GET http://**** HTTP/1.1" 200 739
> b 1315672 v:221188131 vp:221188130 - "GET http://**** HTTP/1.1" 200 0
> b 1187 v:221188141 vp:221188140 - "GET http://**** HTTP/1.1" 200 0
>
> I?m thinking about whether to put PR.
> Can I get your opinion?
>
> Regards,
> --
> Shohei Tanaka(@xcir)
> http://blog.xcir.net/ (JP)
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20190703/994fabfc/attachment-0001.html>

From kokoniimasu at gmail.com  Wed Jul  3 15:00:25 2019
From: kokoniimasu at gmail.com (kokoniimasu)
Date: Thu, 4 Jul 2019 00:00:25 +0900
Subject: varnishncsa show ESI request
In-Reply-To: <CAJ6ZYQzct1Ms4hzCy0x3qRZEGM+Zn2j6jLtAjjU8jOOBM2bZpw@mail.gmail.com>
References: <CAJihfGCS8Xn1LSi7bDc_BYh4ZWzDUJqPzbYees8F4zCfgjaZqQ@mail.gmail.com>
 <CAJ6ZYQzct1Ms4hzCy0x3qRZEGM+Zn2j6jLtAjjU8jOOBM2bZpw@mail.gmail.com>
Message-ID: <CAJihfGDgUPxL+Ea7eWYWGDA+ESCDTK7YFLGjruf-8u5u9pocYQ@mail.gmail.com>

Hi, Guillaume.

Thank You for your advice.

> varnishncsa -F "%{VSL:Begin[2]}x"

As you pointed out, I was mistaken about that.

I'm remove vxid_parent format.

2019?7?3?(?) 23:29 Guillaume Quintard <guillaume at varnish-software.com>:
>
> Hi,
>
> Unless I'm mistaken, what you are asking for is the first item in the EXAMPLE section of "man varnishncsa":
>
>        Log the second field of the Begin record, corresponding to the VXID of the parent transaction:
>
>           varnishncsa -F "%{VSL:Begin[2]}x"
>
> Am I missing something?
>
> --
> Guillaume Quintard
>
>
> On Tue, Jul 2, 2019 at 11:21 PM kokoniimasu <kokoniimasu at gmail.com> wrote:
>>
>> Hi,
>>
>> I'm using varnishncsa, but it's difficult to investigate because does
>> not show ESI sub-request in association.
>> I added show ESI option (-e) and Varnish:vxid_parent x format.
>>
>> wip patch
>> https://gist.github.com/xcir/78c02d50681a4a78bc8c79ffd8421c8c
>>
>> - Output sample
>>
>> # varnishncsa -c -e -g request -F '%{Varnish:side}x %D
>> v:%{Varnish:vxid}x vp:%{Varnish:vxid_parent}x %{Varnish:hitmiss}x "%r"
>> %s %b'
>> c 1547310 v:193913332 vp:0 miss "GET http://**** HTTP/1.1" 200 7985
>>    <-- slow base page
>> c 577 v:193913334 vp:193913332 hit "GET http://**** HTTP/1.1" 200 163
>> c 48233 v:193913336 vp:193913332 miss "GET http://**** HTTP/1.1" 200
>> 1561 <-- fast sub-request
>> c 527 v:193913338 vp:193913332 hit "GET http://**** HTTP/1.1" 200 164
>>
>> # varnishncsa -b -c -e -g request -F '%{Varnish:side}x %D
>> v:%{Varnish:vxid}x vp:%{Varnish:vxid_parent}x %{Varnish:hitmiss}x "%r"
>> %s %b'
>> c 2262459 v:221188118 vp:0 miss "GET http://**** HTTP/1.1" 200 18168
>> b 817749 v:221188119 vp:221188118 - "GET http://**** HTTP/1.1" 200 0
>> c 550 v:221188120 vp:221188118 hit "GET http://**** HTTP/1.1" 200 163
>> c 295 v:221188122 vp:221188118 hit "GET http://**** HTTP/1.1" 200 1565
>> c 494 v:221188123 vp:221188118 hit "GET http://**** HTTP/1.1" 200 162
>> c 262 v:221188125 vp:221188118 hit "GET http://**** HTTP/1.1" 200 0
>> c 1242 v:221188127 vp:221188118 hit "GET http://**** HTTP/1.1" 200 1488
>> c 211 v:221188129 vp:221188118 hit "GET http://**** HTTP/1.1" 200 44
>> c 1316055 v:221188130 vp:221188118 miss "GET http://**** HTTP/1.1" 200
>> 4353 <-- slow sub-request
>> c 34039 v:221188132 vp:221188118 hit "GET http://**** HTTP/1.1" 200 788
>> c 18784 v:221188134 vp:221188118 hit "GET http://**** HTTP/1.1" 200 0
>> c 32942 v:221188135 vp:221188118 hit "GET http://**** HTTP/1.1" 200 774
>> c 22181 v:221188136 vp:221188118 hit "GET http://**** HTTP/1.1" 200 163
>> c 5766 v:221188138 vp:221188118 hit "GET http://**** HTTP/1.1" 200 0
>> c 8005 v:221188140 vp:221188118 miss "GET http://**** HTTP/1.1" 200
>> 163 <-- fast sub-request
>> c 1626 v:221188142 vp:221188118 hit "GET http://**** HTTP/1.1" 200 163
>> c 230 v:221188144 vp:221188118 hit "GET http://**** HTTP/1.1" 200 739
>> b 1315672 v:221188131 vp:221188130 - "GET http://**** HTTP/1.1" 200 0
>> b 1187 v:221188141 vp:221188140 - "GET http://**** HTTP/1.1" 200 0
>>
>> I?m thinking about whether to put PR.
>> Can I get your opinion?
>>
>> Regards,
>> --
>> Shohei Tanaka(@xcir)
>> http://blog.xcir.net/ (JP)
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc

From nikolay.bogdanov at flant.com  Tue Jul  9 08:10:21 2019
From: nikolay.bogdanov at flant.com (Nikolay Bogdanov)
Date: Tue, 9 Jul 2019 11:10:21 +0300
Subject: Varnish and failed upstream
Message-ID: <CAFDre75dTHHza48fyAKHFDmcd02UgOajum=LgOLokHQAXjMt4Q@mail.gmail.com>

Hello. I have one case and I can not find good solution for it.
In varnish3 req.backend was saved during restarts, so I can compare old
backend property and set other backend after first restart.
But in varnish 5 and newer req.backend_hint in vcl_recv is default always.
I can not use retry in vcl_backend_response, because this function did not
called.
How can I fix it?

My bad solution:

sub vcl_init {
    new upstream_rr = directors.round_robin();
    key_rr.add_backend(srv1);
    key_rr.add_backend(srv2);
}

sub vcl_recv {
    if (req.restarts == 0) {
      set req.backend_hint = upstream_rr.backend();
    } else {
      if (req.restarts == 1) {
        std.syslog(180, "RECV IN PROG: set backend srv1");
        set req.backend_hint = srv1;
      } elseif (req.restarts == 2) {
        std.syslog(180, "RECV IN PROG: set backend srv2");
        set req.backend_hint = srv2;
      } elseif (req.restarts > 2) {
        std.syslog(180, "RECV IN PROG: use other DC backend default");
        set req.backend_hint = default;
      }
    }
    return (hash);
}

-- 
-- 
??????? ????????
??????? ?? ???????
http://flant.ru/
+7 (495) 721-10-27, ???. 422
+7 (926) 125-39-69
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20190709/5911c58b/attachment.html>

From dridi at varni.sh  Wed Jul 10 16:53:19 2019
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Wed, 10 Jul 2019 16:53:19 +0000
Subject: Varnish and failed upstream
In-Reply-To: <CAFDre75dTHHza48fyAKHFDmcd02UgOajum=LgOLokHQAXjMt4Q@mail.gmail.com>
References: <CAFDre75dTHHza48fyAKHFDmcd02UgOajum=LgOLokHQAXjMt4Q@mail.gmail.com>
Message-ID: <CABoVN9AyrNMMEkrgi00C8pcHE6KZhNnd-t7qkRXgkX+YdggXYg@mail.gmail.com>

On Tue, Jul 9, 2019 at 8:12 AM Nikolay Bogdanov
<nikolay.bogdanov at flant.com> wrote:
>
> Hello. I have one case and I can not find good solution for it.
> In varnish3 req.backend was saved during restarts, so I can compare old backend property and set other backend after first restart.
> But in varnish 5 and newer req.backend_hint in vcl_recv is default always. I can not use retry in vcl_backend_response, because this function did not called.
> How can I fix it?

What you are looking for is probably a circuit breaker, you can
implement one with vmod-saintmod:

https://github.com/varnish/varnish-modules/blob/master/docs/vmod_saintmode.rst#vmod_saintmode

However instead of using it with req, it's probably more efficient to
work on the backend side and retry from vcl_backend_fetch or
vcl_backend_error.

Best,
Dridi

From Alexandre.Thaveau at mister-auto.com  Mon Jul 22 13:29:28 2019
From: Alexandre.Thaveau at mister-auto.com (Alexandre Thaveau)
Date: Mon, 22 Jul 2019 15:29:28 +0200
Subject: Very strange varnish statistics after ESI activation
Message-ID: <CANqTgraohzzxhhnBarq8pqf8Km=p6TD8S_xJnSdhfwgf8qeBTQ@mail.gmail.com>

Hi,

a few weeks ago, i asked the mailing list about my varnish statistics,
tracked under prometheus using varnish exporter (varnishstat wrapper).
2 weeks ago, one of our developpers activated ESI using these changes
(second modification is for a specific case but i put it in case of)
------------------------------------
# vcl_backend_response is called after a document has been successfully
retrieved from the backend
sub vcl_backend_response {
        set beresp.do_esi = true;
------------------------------------
    if(req.http.Cookie ~ "[^?]*custom_data_cookie=.*"){
    -    if(req.url !~ "^/([a-z]{2}/)$" && req.url !~ "^/([a-z]{2}/)(&|\?)"
&& req.url !~ "^/(&|\?)" && req.url != "/" && req.url !~
"^\/([a-z]{2}+\/)?((?!(ofertas|offres|aanbiedingen|angebote|offers)\b)[^\/]+\/)+([^\/]+\/)+([^\/]+\/)+([^\/]+\/)+$"){
    +    if(req.url !~ "^/([a-z]{2}/)$" && req.url !~ "^/([a-z]{2}/)(&|\?)"
&& req.url !~ "^/(&|\?)" && req.url != "/" && req.url !~
"^\/([a-z]{2}+\/)?((?!(ofertas|offres|aanbiedingen|angebote|offers)\b)[^\/]+\/)+([^\/]+\/)+([^\/]+\/)+([^\/]+\/)+$"
&& req.url !~ "^\/([a-z]{2}\/)?esi\/(.+)?"){
            return(pass);
        }
    }
------------------------------------
and configure our website menu to be edge side included (at maximum 50
combinations max using country and language used in parameter).

After this modification, multiple metrics changed in the supervision which
i do not understand :
- my number of varnish objects (main_n_object) went from 337 000 entries to
475 000 entries ... i really don't understand this since i thought my cache
memory
was full before and since we only added 50 ESI keys in the cache with this
update
- my cache hit (main_cache_hit) went from 2.5 % to 22% ... but pass, miss
and synth rates did not move, so the sum of all my varnish outputs are now
upper than 110 % ... (before they were equals to 100%)
- my backend output (beresp_hdrbytes) total dropped from 2MB to 85KB ...
however our websites are working fine

Can you help me understand this ?

Thanks

-- 
Alexandre THAVEAU
*04 72 89 22 11*
alexandre.thaveau at mister-auto.com <rgz at mister-auto.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20190722/41463ec7/attachment.html>