From arvind at cs.umn.edu  Thu Feb  1 20:46:45 2018
From: arvind at cs.umn.edu (Arvind Narayanan)
Date: Thu, 1 Feb 2018 14:46:45 -0600
Subject: Query about HSH_EXPBUSY & HSH_EXP in HSH_Lookup
Message-ID: <CAOssFL6_5sW65PKcsf+LBFoYcPA0npi7KdC7JhrMJZFYgUpisQ@mail.gmail.com>

Hi,

I was trying to understand the function

 enum lookup_e
HSH_Lookup(struct req *req, struct objcore **ocp, struct objcore **bocp,
    int always_insert)

located in bin/varnishd/cache/cache_hash.c

My understanding is, if the lookedup-object in the cache is expired, the
code seems to look if the object is busy or not and accordingly returns
HSH_EXPBUSY or HSH_BUSY.

lines 474 to 501 in cache_hash.c
    if (exp_oc != NULL && exp_oc->flags & OC_F_PASS) {
        wrk->stats->cache_hitmiss++;
        VSLb(req->vsl, SLT_HitMiss, "%u %.6f", ObjGetXID(wrk, exp_oc),
            EXP_Dttl(req, exp_oc));
        exp_oc = NULL;
        busy_found = 0;
    }

    if (exp_oc != NULL) {
        assert(oh->refcnt > 1);
        assert(exp_oc->objhead == oh);
        exp_oc->refcnt++;

        if (!busy_found) {
            *bocp = hsh_insert_busyobj(wrk, oh);
            retval = HSH_EXPBUSY;
        } else {
            AZ(req->hash_ignore_busy);
            retval = HSH_EXP;
        }
        if (exp_oc->hits < LONG_MAX)
            exp_oc->hits++;
        Lck_Unlock(&oh->mtx);
        if (retval == HSH_EXP)
            assert(HSH_DerefObjHead(wrk, &oh));
        *ocp = exp_oc;
        return (retval);
    }

    if (!busy_found) {
        /* Insert objcore in objecthead and release mutex */
        *bocp = hsh_insert_busyobj(wrk, oh);
        /* NB: no deref of objhead, new object inherits reference */
        Lck_Unlock(&oh->mtx);
        return (HSH_MISS);
    }


If busy, is the variable busy_found set to 0 or 1? For expired objects, it
looks like if it is not busy, then you set the return value to HSH_EXPBUSY,
shouldn't it be HSH_EXP? Can anyone help me understand this piece of code?
Because below, the code returns HSH_MISS

?Thanks,
Arvind
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20180201/c5b89e1f/attachment.html>

From dridi at varni.sh  Thu Feb  1 21:36:02 2018
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Thu, 1 Feb 2018 22:36:02 +0100
Subject: Query about HSH_EXPBUSY & HSH_EXP in HSH_Lookup
In-Reply-To: <CAOssFL6_5sW65PKcsf+LBFoYcPA0npi7KdC7JhrMJZFYgUpisQ@mail.gmail.com>
References: <CAOssFL6_5sW65PKcsf+LBFoYcPA0npi7KdC7JhrMJZFYgUpisQ@mail.gmail.com>
Message-ID: <CABoVN9D=r2Nk_30T709YghyH7AxBqYhJYe6tKoTFCPZHghfd=g@mail.gmail.com>

On Thu, Feb 1, 2018 at 9:46 PM, Arvind Narayanan <arvind at cs.umn.edu> wrote:
<snip>
> If busy, is the variable busy_found set to 0 or 1? For expired objects, it
> looks like if it is not busy, then you set the return value to HSH_EXPBUSY,
> shouldn't it be HSH_EXP? Can anyone help me understand this piece of code?
> Because below, the code returns HSH_MISS

It's a bit embarrassing because I've stared hard at this code a couple
months ago, and don't remember the details. I think that HSH_EXPBUSY
means that the object you found during the lookup was expired but
there is an ongoing transaction also matching (hash and variants) your
lookup. So I think in that case your hit is actually a graced object.

But I only tried to remember from the snippet you posted, I didn't
look at the full picture.

Dridi

From hermunn at varnish-software.com  Fri Feb  2 12:23:36 2018
From: hermunn at varnish-software.com (=?UTF-8?Q?P=C3=A5l_Hermunn_Johansen?=)
Date: Fri, 2 Feb 2018 13:23:36 +0100
Subject: Query about HSH_EXPBUSY & HSH_EXP in HSH_Lookup
In-Reply-To: <CABoVN9D=r2Nk_30T709YghyH7AxBqYhJYe6tKoTFCPZHghfd=g@mail.gmail.com>
References: <CAOssFL6_5sW65PKcsf+LBFoYcPA0npi7KdC7JhrMJZFYgUpisQ@mail.gmail.com>
 <CABoVN9D=r2Nk_30T709YghyH7AxBqYhJYe6tKoTFCPZHghfd=g@mail.gmail.com>
Message-ID: <CAMNFQgPtoST-9+zBjPncM66SGsXVeWTM_Oxm=aj0NbdrJLBYPQ@mail.gmail.com>

2018-02-01 22:36 GMT+01:00 Dridi Boukelmoune <dridi at varni.sh>:
> On Thu, Feb 1, 2018 at 9:46 PM, Arvind Narayanan <arvind at cs.umn.edu> wrote:
> <snip>
>> If busy, is the variable busy_found set to 0 or 1? For expired objects, it
>> looks like if it is not busy, then you set the return value to HSH_EXPBUSY,
>> shouldn't it be HSH_EXP? Can anyone help me understand this piece of code?
>> Because below, the code returns HSH_MISS
>
> It's a bit embarrassing because I've stared hard at this code a couple
> months ago, and don't remember the details.

I had to spend a lot of time with this code, and I think I remember
most of it. I did a small essay on an old bug,

https://github.com/varnishcache/varnish-cache/pull/2519

and, as you can see, I propose a change in the code that will make it
even harder to follow. Later I discovered that my PR was not enough
to fix the problems, and I am working on a new PR that will change
how TTLs are calculated.

> I think that HSH_EXPBUSY means that the object you found during the lookup
> was expired but  there is an ongoing transaction also matching (hash and
> variants) your lookup.

This is correct.

> So I think in that case your hit is actually a graced object.

It can also be an expired object that is in "keep". However, even if
you have 0s keep and 0s grace, you can get an object here. This
happens when the expiry thread is lagging a bit, or when you are
affected by the very slightly different ways TTLs are calculated in
VCL and the core.

> But I only tried to remember from the snippet you posted, I didn't
> look at the full picture.

If you look at the code that calls HSH_Lookup, you will see that the
return value is not always considered, but the result of the lookup
is deduced from the values that HSH_Lookup did or did not set
(struct objcore **ocp, struct objcore **bocp).

When reading the code, maybe looking at the variables ocp and
bocp is key to understanding what is going on.


P?l

From n.premkumar.me at gmail.com  Mon Feb  5 07:29:09 2018
From: n.premkumar.me at gmail.com (Prem Kumar)
Date: Mon, 5 Feb 2018 12:59:09 +0530
Subject: Varnish 5.x - Streaming + range request
Message-ID: <CAC3PTxL8OnV49X7YE+kJnC6JNiSXuWrqcvmmsTS-0CWA9qz72g@mail.gmail.com>

Just  want to understand the behavior of varnish if range request while
streaming of entire object in progress. lets say client1 fetches complete
object  of size of 300 bytes (do_stream == true)  and client 2 is trying to
fetch range request 100-200 bytes at the same time. the hash functions will
match because we are using the same url+host for hashing. .How the clients
will be served?.  Will wait for entire object to streamed or make it miss
and fetch from origin ?. Would expect to deliver the object from origin
instead of wait whole object may take time.

Thanks,
Prem
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20180205/e5c837d6/attachment.html>

From dridi at varni.sh  Mon Feb  5 07:45:32 2018
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 5 Feb 2018 08:45:32 +0100
Subject: Varnish 5.x - Streaming + range request
In-Reply-To: <CAC3PTxL8OnV49X7YE+kJnC6JNiSXuWrqcvmmsTS-0CWA9qz72g@mail.gmail.com>
References: <CAC3PTxL8OnV49X7YE+kJnC6JNiSXuWrqcvmmsTS-0CWA9qz72g@mail.gmail.com>
Message-ID: <CABoVN9DGrvWEx4UUce3ud-NOgM0Am8m7eOviJWLA_SBssPmH9w@mail.gmail.com>

On Mon, Feb 5, 2018 at 8:29 AM, Prem Kumar <n.premkumar.me at gmail.com> wrote:
> Just  want to understand the behavior of varnish if range request while
> streaming of entire object in progress. lets say client1 fetches complete
> object  of size of 300 bytes (do_stream == true)  and client 2 is trying to
> fetch range request 100-200 bytes at the same time. the hash functions will
> match because we are using the same url+host for hashing. .How the clients
> will be served?.  Will wait for entire object to streamed or make it miss
> and fetch from origin ?. Would expect to deliver the object from origin
> instead of wait whole object may take time.

Varnish will only make range requests for pass transactions (unless
you force it in VCL) so in the case of a miss, either a complete fetch
is done or the client transactions attaches itself to an ongoing
fetch, and on the delivery side a partial response is delivered (I
think, as soon as the requested range is available in storage).

Dridi

From guillaume at varnish-software.com  Tue Feb  6 21:51:16 2018
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Tue, 6 Feb 2018 22:51:16 +0100
Subject: Any way to invoke code from VCL after a request has been serviced?
In-Reply-To: <CACqP_wzTYs7niv_7iy+oOmogoEDk7JVt+W_0+UE3qPETVodCSw@mail.gmail.com>
References: <CACqP_wwFPwrCtqNVGi4n152BZLudx0_pmVJgXtGUf7nY+W0Rcg@mail.gmail.com>
 <CAAARNrHYUWVxoCbCcQPSkCK2dbqh_6A9MDBOeGexbQNXMMqtew@mail.gmail.com>
 <CAAARNrE8Ry9_Xtd7k_3ZU54o-80o8DVUfnpYsXYsGphkDMiELA@mail.gmail.com>
 <CACqP_wyBr_AemDBYisw2cvcu5MnDGDB6ed1byKDnunV30xy2dg@mail.gmail.com>
 <CAJ6ZYQxRCb658V2a5aMx4LO3bYf9_tqC8FvSwurS6pasBsuKQw@mail.gmail.com>
 <CACqP_wzC0uERzK1tS+s3rh_Yo=c7heqooJEyxfBn6OdhDedRYA@mail.gmail.com>
 <CAJ6ZYQwTk9Ox9vhi6_w2Y=BaXGbyG5id+RHBPzGR2V0=bX4oiQ@mail.gmail.com>
 <CACqP_wzZf8BQydVAqTgN9uA0CxFO6=KOdLjk6VicO3m=XMkdvQ@mail.gmail.com>
 <CABoVN9D+xNfXWFc21dLgVLoD6aSb6yiDbhp0j0syW7m8H6cuCQ@mail.gmail.com>
 <CACqP_ww-ZK=fMG=pMJALRXHF3KeAxNe7guo+FMJy4fy3U2GhjQ@mail.gmail.com>
 <CABoVN9ADiHjdace3cktLo7gruW+eMSDYjXciochhpyx5=j_Dww@mail.gmail.com>
 <CACqP_wxcDbce+-d=KmPLt3C68WO9hR8KiP_PJCYRXySrsH+WmA@mail.gmail.com>
 <CABoVN9Dytd7A64zr2vH1qKfC+cqKRtXuyZBVO+V6CtMzReX8=g@mail.gmail.com>
 <CACqP_wzTYs7niv_7iy+oOmogoEDk7JVt+W_0+UE3qPETVodCSw@mail.gmail.com>
Message-ID: <CAJ6ZYQyFMsLbB6Q1xrWtFBG1MgCVXcUJYOvTFRz-n4upTH3tzQ@mail.gmail.com>

Sorry to dig that one up, but I stumble upon it again, and it got me
thinking: can't we just put the point to req->err_code in priv? That's not
super clean, but from what I re-read in the code, it should still be there
when it's time to clean the priv.

-- 
Guillaume Quintard

On Tue, May 23, 2017 at 11:15 PM, Ryan Burn <rnickb731 at gmail.com> wrote:

> > When it comes to header manipulations (including method, status etc)
> > they are already logged so a VUT can already pick that up and save
> > some work from the VMOD.
>
> It doesn't need to just view the headers; it needs to add headers to
> encode the
> active span context. See
> http://opentracing.io/documentation/pages/api/cross-process-tracing.html
> And the active span needs to be started before those headers can be added.
>
> > I'm not familiar with OT but what I described is how Zipnish does its
> > tracing. Except that Zipnish relies on the X-Varnish header to get a
> > unique-ish id, so no blocking call needs to be made. So maybe they are
> > highly different systems, I chimed in because I saw a Zipkin example
> > while briefly skimming through the docs.
>
> Zipkin is a distributed tracing system that provides OpenTracing
> implementations, but Zipnish is just using it as a monitor for
> varnish. It's not doing context propagation. If you're only interested
> in monitoring varnish, that's fine; but if you want to see how a
> request is processed in an entire distributed system (i.e. not just
> varnish, but the backend servers it's sitting in front of or any other
> service that might be in front of varnish), then you need to do
> context propagation.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20180206/d1d7bdb7/attachment.html>

From noelle at uni-wuppertal.de  Wed Feb  7 07:24:51 2018
From: noelle at uni-wuppertal.de (=?UTF-8?Q?Christian_N=c3=b6lle?=)
Date: Wed, 7 Feb 2018 08:24:51 +0100
Subject: munin plugin not working with 5.2
In-Reply-To: <09db60c6-522b-9293-f573-acd7d9ee8068@uni-wuppertal.de>
References: <e4aca6c9-b7b0-c006-a687-87880ee24c62@uni-wuppertal.de>
 <09db60c6-522b-9293-f573-acd7d9ee8068@uni-wuppertal.de>
Message-ID: <a3f5bd27-38f9-75dd-f326-861ff805ed3e@uni-wuppertal.de>

Hi everybody,

so if anybody is looking for a varnish 5.2 compatible munin, we made a 
pull request, which can be found here: 
https://github.com/munin-monitoring/contrib/pull/895

Take care:

-christian

-- 
-c

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5397 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20180207/8a0ec29d/attachment-0001.bin>

From pinakee at waltzz.com  Wed Feb  7 12:34:50 2018
From: pinakee at waltzz.com (Pinakee BIswas)
Date: Wed, 7 Feb 2018 18:04:50 +0530
Subject: Varnish Memory usage increase
Message-ID: <6ae5745c-7520-8e97-4f02-0f69104ae418@waltzz.com>

Hi,

We have been using varnish since more than a year for our ecommerce site.

Current version is 4.1.8.

OS is Linux. RAM is 8GB.

I am observing that the varnish memory usage is increasing (right now 
7%) and so is resident memory usage increasing.

The storage being used is file:

-s file,/tmp/varnish/,${storage}

I am not sure about the reason for the increase in varnish memory usage:

  * Is there a way to limit the varnish memory usage?
  * How can I diagnose what is consuming memory?
  * Most of our web pages have maximum 2 days of cache. Also, some of
    the pages might be least visited.
  * Is there a way to manipulate varnish mmap (when file storage is used)?

Would appreciate any help on the above for efficient use of varnish.

Thanks,

Pinakee


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20180207/07e6b337/attachment.html>

From guillaume at varnish-software.com  Wed Feb  7 12:51:19 2018
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Wed, 7 Feb 2018 13:51:19 +0100
Subject: Varnish Memory usage increase
In-Reply-To: <6ae5745c-7520-8e97-4f02-0f69104ae418@waltzz.com>
References: <6ae5745c-7520-8e97-4f02-0f69104ae418@waltzz.com>
Message-ID: <CAJ6ZYQzjZ_Lq6oZZ1RWjo7H_4ihPeoLqyJzNYeM+1LjoWiUSVw@mail.gmail.com>

Hi there,

So, varnish memory usage will mostly come from three directions:
- storage, include Transient, so check the g_bytes fields in varnishstat.
Passes and shortlived objects will use Transient, so you can either reduce
those, or limit the Transient storage (unbounded by defaukt).
- thread workspaces, one thread typically uses one workspace, so you can
limit the number of threads, or reduce the workspace size
- memory fragmentation: jemalloc will fragment up to 25%, not much you can
do here.

And no, no way to modify the mmap without restarting varnish.

May I ask how much data you are caching?

-- 
Guillaume Quintard

On Wed, Feb 7, 2018 at 1:34 PM, Pinakee BIswas <pinakee at waltzz.com> wrote:

> Hi,
>
> We have been using varnish since more than a year for our ecommerce site.
>
> Current version is 4.1.8.
>
> OS is Linux. RAM is 8GB.
>
> I am observing that the varnish memory usage is increasing (right now 7%)
> and so is resident memory usage increasing.
>
> The storage being used is file:
>
> -s file,/tmp/varnish/,${storage}
>
> I am not sure about the reason for the increase in varnish memory usage:
>
>    - Is there a way to limit the varnish memory usage?
>    - How can I diagnose what is consuming memory?
>    - Most of our web pages have maximum 2 days of cache. Also, some of
>    the pages might be least visited.
>    - Is there a way to manipulate varnish mmap (when file storage is
>    used)?
>
> Would appreciate any help on the above for efficient use of varnish.
>
> Thanks,
>
> Pinakee
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20180207/6038a3c9/attachment.html>

From pinakee at waltzz.com  Wed Feb  7 13:12:29 2018
From: pinakee at waltzz.com (Pinakee BIswas)
Date: Wed, 7 Feb 2018 18:42:29 +0530
Subject: Varnish Memory usage increase
In-Reply-To: <CAJ6ZYQzjZ_Lq6oZZ1RWjo7H_4ihPeoLqyJzNYeM+1LjoWiUSVw@mail.gmail.com>
References: <6ae5745c-7520-8e97-4f02-0f69104ae418@waltzz.com>
 <CAJ6ZYQzjZ_Lq6oZZ1RWjo7H_4ihPeoLqyJzNYeM+1LjoWiUSVw@mail.gmail.com>
Message-ID: <e716b3a2-c285-1042-c825-bc5254dd42ea@waltzz.com>

Hi Guillaume,

Thanks for your response and details.

Please find my comments inline:


On 07/02/18 6:21 pm, Guillaume Quintard wrote:
> Hi there,
>
> So, varnish memory usage will mostly come from three directions:
> - storage, include Transient, so check the g_bytes fields in 
> varnishstat. Passes and shortlived objects will use Transient, so you 
> can either reduce those, or limit the Transient storage (unbounded by 
> defaukt).
Checked the g_bytes for transient in varnishstat. It's in Kbs.
> - thread workspaces, one thread typically uses one workspace, so you 
> can limit the number of threads, or reduce the workspace size
> - memory fragmentation: jemalloc will fragment up to 25%, not much you 
> can do here.
>
> And no, no way to modify the mmap without restarting varnish.
>
> May I ask how much data you are caching?
Could you please let me know how to figure out the amount of data cached?
>
> -- 
> Guillaume Quintard
>
> On Wed, Feb 7, 2018 at 1:34 PM, Pinakee BIswas <pinakee at waltzz.com 
> <mailto:pinakee at waltzz.com>> wrote:
>
>     Hi,
>
>     We have been using varnish since more than a year for our
>     ecommerce site.
>
>     Current version is 4.1.8.
>
>     OS is Linux. RAM is 8GB.
>
>     I am observing that the varnish memory usage is increasing (right
>     now 7%) and so is resident memory usage increasing.
>
>     The storage being used is file:
>
>     -s file,/tmp/varnish/,${storage}
>
>     I am not sure about the reason for the increase in varnish memory
>     usage:
>
>       * Is there a way to limit the varnish memory usage?
>       * How can I diagnose what is consuming memory?
>       * Most of our web pages have maximum 2 days of cache. Also, some
>         of the pages might be least visited.
>       * Is there a way to manipulate varnish mmap (when file storage
>         is used)?
>
>     Would appreciate any help on the above for efficient use of varnish.
>
>     Thanks,
>
>     Pinakee
>
>
>
>     _______________________________________________
>     varnish-misc mailing list
>     varnish-misc at varnish-cache.org <mailto:varnish-misc at varnish-cache.org>
>     https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>     <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20180207/968e8e67/attachment.html>

From guillaume at varnish-software.com  Wed Feb  7 13:21:35 2018
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Wed, 7 Feb 2018 14:21:35 +0100
Subject: Varnish Memory usage increase
In-Reply-To: <e716b3a2-c285-1042-c825-bc5254dd42ea@waltzz.com>
References: <6ae5745c-7520-8e97-4f02-0f69104ae418@waltzz.com>
 <CAJ6ZYQzjZ_Lq6oZZ1RWjo7H_4ihPeoLqyJzNYeM+1LjoWiUSVw@mail.gmail.com>
 <e716b3a2-c285-1042-c825-bc5254dd42ea@waltzz.com>
Message-ID: <CAJ6ZYQz105M6mUCvRH3MzNgEY5_zYwC1N8KCUNTEkp_1z9CrGw@mail.gmail.com>

Amount of storage used is just the sum of all the g_bytes fields

-- 
Guillaume Quintard

On Feb 7, 2018 14:12, "Pinakee BIswas" <pinakee at waltzz.com> wrote:

> Hi Guillaume,
>
> Thanks for your response and details.
>
> Please find my comments inline:
>
> On 07/02/18 6:21 pm, Guillaume Quintard wrote:
>
> Hi there,
>
> So, varnish memory usage will mostly come from three directions:
> - storage, include Transient, so check the g_bytes fields in varnishstat.
> Passes and shortlived objects will use Transient, so you can either reduce
> those, or limit the Transient storage (unbounded by defaukt).
>
> Checked the g_bytes for transient in varnishstat. It's in Kbs.
>
> - thread workspaces, one thread typically uses one workspace, so you can
> limit the number of threads, or reduce the workspace size
> - memory fragmentation: jemalloc will fragment up to 25%, not much you can
> do here.
>
> And no, no way to modify the mmap without restarting varnish.
>
> May I ask how much data you are caching?
>
> Could you please let me know how to figure out the amount of data cached?
>
>
> --
> Guillaume Quintard
>
> On Wed, Feb 7, 2018 at 1:34 PM, Pinakee BIswas <pinakee at waltzz.com> wrote:
>
>> Hi,
>>
>> We have been using varnish since more than a year for our ecommerce site.
>>
>> Current version is 4.1.8.
>>
>> OS is Linux. RAM is 8GB.
>>
>> I am observing that the varnish memory usage is increasing (right now 7%)
>> and so is resident memory usage increasing.
>>
>> The storage being used is file:
>>
>> -s file,/tmp/varnish/,${storage}
>>
>> I am not sure about the reason for the increase in varnish memory usage:
>>
>>    - Is there a way to limit the varnish memory usage?
>>    - How can I diagnose what is consuming memory?
>>    - Most of our web pages have maximum 2 days of cache. Also, some of
>>    the pages might be least visited.
>>    - Is there a way to manipulate varnish mmap (when file storage is
>>    used)?
>>
>> Would appreciate any help on the above for efficient use of varnish.
>>
>> Thanks,
>>
>> Pinakee
>>
>>
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20180207/da5e88d9/attachment-0001.html>

From pinakee at waltzz.com  Wed Feb  7 13:35:21 2018
From: pinakee at waltzz.com (Pinakee BIswas)
Date: Wed, 7 Feb 2018 19:05:21 +0530
Subject: Varnish Memory usage increase
In-Reply-To: <CAJ6ZYQz105M6mUCvRH3MzNgEY5_zYwC1N8KCUNTEkp_1z9CrGw@mail.gmail.com>
References: <6ae5745c-7520-8e97-4f02-0f69104ae418@waltzz.com>
 <CAJ6ZYQzjZ_Lq6oZZ1RWjo7H_4ihPeoLqyJzNYeM+1LjoWiUSVw@mail.gmail.com>
 <e716b3a2-c285-1042-c825-bc5254dd42ea@waltzz.com>
 <CAJ6ZYQz105M6mUCvRH3MzNgEY5_zYwC1N8KCUNTEkp_1z9CrGw@mail.gmail.com>
Message-ID: <3be7adfb-4f78-7cc9-7a89-e7ebefec2d88@waltzz.com>

Following is the data from varnishstat:

SMA.Transient.g_bytes 15.76K        -2.19K          .           18.53K 
15.50K        15.24K

SMF.s0.g_bytes 673.26M        47.95K          .          672.87M 
669.89M       668.90M


On 07/02/18 6:51 pm, Guillaume Quintard wrote:
> Amount of storage used is just the sum of all the g_bytes fields
>
> -- 
> Guillaume Quintard
>
> On Feb 7, 2018 14:12, "Pinakee BIswas" <pinakee at waltzz.com 
> <mailto:pinakee at waltzz.com>> wrote:
>
>     Hi Guillaume,
>
>     Thanks for your response and details.
>
>     Please find my comments inline:
>
>
>     On 07/02/18 6:21 pm, Guillaume Quintard wrote:
>>     Hi there,
>>
>>     So, varnish memory usage will mostly come from three directions:
>>     - storage, include Transient, so check the g_bytes fields in
>>     varnishstat. Passes and shortlived objects will use Transient, so
>>     you can either reduce those, or limit the Transient storage
>>     (unbounded by defaukt).
>     Checked the g_bytes for transient in varnishstat. It's in Kbs.
>>     - thread workspaces, one thread typically uses one workspace, so
>>     you can limit the number of threads, or reduce the workspace size
>>     - memory fragmentation: jemalloc will fragment up to 25%, not
>>     much you can do here.
>>
>>     And no, no way to modify the mmap without restarting varnish.
>>
>>     May I ask how much data you are caching?
>     Could you please let me know how to figure out the amount of data
>     cached?
>>
>>     -- 
>>     Guillaume Quintard
>>
>>     On Wed, Feb 7, 2018 at 1:34 PM, Pinakee BIswas
>>     <pinakee at waltzz.com <mailto:pinakee at waltzz.com>> wrote:
>>
>>         Hi,
>>
>>         We have been using varnish since more than a year for our
>>         ecommerce site.
>>
>>         Current version is 4.1.8.
>>
>>         OS is Linux. RAM is 8GB.
>>
>>         I am observing that the varnish memory usage is increasing
>>         (right now 7%) and so is resident memory usage increasing.
>>
>>         The storage being used is file:
>>
>>         -s file,/tmp/varnish/,${storage}
>>
>>         I am not sure about the reason for the increase in varnish
>>         memory usage:
>>
>>           * Is there a way to limit the varnish memory usage?
>>           * How can I diagnose what is consuming memory?
>>           * Most of our web pages have maximum 2 days of cache. Also,
>>             some of the pages might be least visited.
>>           * Is there a way to manipulate varnish mmap (when file
>>             storage is used)?
>>
>>         Would appreciate any help on the above for efficient use of
>>         varnish.
>>
>>         Thanks,
>>
>>         Pinakee
>>
>>
>>
>>         _______________________________________________
>>         varnish-misc mailing list
>>         varnish-misc at varnish-cache.org
>>         <mailto:varnish-misc at varnish-cache.org>
>>         https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>         <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20180207/e38e7f7d/attachment.html>

From guillaume at varnish-software.com  Wed Feb  7 13:39:52 2018
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Wed, 7 Feb 2018 14:39:52 +0100
Subject: Varnish Memory usage increase
In-Reply-To: <3be7adfb-4f78-7cc9-7a89-e7ebefec2d88@waltzz.com>
References: <6ae5745c-7520-8e97-4f02-0f69104ae418@waltzz.com>
 <CAJ6ZYQzjZ_Lq6oZZ1RWjo7H_4ihPeoLqyJzNYeM+1LjoWiUSVw@mail.gmail.com>
 <e716b3a2-c285-1042-c825-bc5254dd42ea@waltzz.com>
 <CAJ6ZYQz105M6mUCvRH3MzNgEY5_zYwC1N8KCUNTEkp_1z9CrGw@mail.gmail.com>
 <3be7adfb-4f78-7cc9-7a89-e7ebefec2d88@waltzz.com>
Message-ID: <CAJ6ZYQzC-dOp69kMNz3iRFTF_N9y4-ExeTgAW3WGW9kVWNhoxQ@mail.gmail.com>

So, it's probably not related to your issue, but it seems your cached data
easily fits in memory, why use the file storage?

-- 
Guillaume Quintard

On Feb 7, 2018 14:35, "Pinakee BIswas" <pinakee at waltzz.com> wrote:

> Following is the data from varnishstat:
>
> SMA.Transient.g_bytes
> 15.76K        -2.19K          .           18.53K        15.50K        15.24K
>
> SMF.s0.g_bytes
> 673.26M        47.95K          .          672.87M       669.89M
> 668.90M
>
>
> On 07/02/18 6:51 pm, Guillaume Quintard wrote:
>
> Amount of storage used is just the sum of all the g_bytes fields
>
> --
> Guillaume Quintard
>
> On Feb 7, 2018 14:12, "Pinakee BIswas" <pinakee at waltzz.com> wrote:
>
>> Hi Guillaume,
>>
>> Thanks for your response and details.
>>
>> Please find my comments inline:
>>
>> On 07/02/18 6:21 pm, Guillaume Quintard wrote:
>>
>> Hi there,
>>
>> So, varnish memory usage will mostly come from three directions:
>> - storage, include Transient, so check the g_bytes fields in varnishstat.
>> Passes and shortlived objects will use Transient, so you can either reduce
>> those, or limit the Transient storage (unbounded by defaukt).
>>
>> Checked the g_bytes for transient in varnishstat. It's in Kbs.
>>
>> - thread workspaces, one thread typically uses one workspace, so you can
>> limit the number of threads, or reduce the workspace size
>> - memory fragmentation: jemalloc will fragment up to 25%, not much you
>> can do here.
>>
>> And no, no way to modify the mmap without restarting varnish.
>>
>> May I ask how much data you are caching?
>>
>> Could you please let me know how to figure out the amount of data cached?
>>
>>
>> --
>> Guillaume Quintard
>>
>> On Wed, Feb 7, 2018 at 1:34 PM, Pinakee BIswas <pinakee at waltzz.com>
>> wrote:
>>
>>> Hi,
>>>
>>> We have been using varnish since more than a year for our ecommerce
>>> site.
>>>
>>> Current version is 4.1.8.
>>>
>>> OS is Linux. RAM is 8GB.
>>>
>>> I am observing that the varnish memory usage is increasing (right now
>>> 7%) and so is resident memory usage increasing.
>>>
>>> The storage being used is file:
>>>
>>> -s file,/tmp/varnish/,${storage}
>>>
>>> I am not sure about the reason for the increase in varnish memory usage:
>>>
>>>    - Is there a way to limit the varnish memory usage?
>>>    - How can I diagnose what is consuming memory?
>>>    - Most of our web pages have maximum 2 days of cache. Also, some of
>>>    the pages might be least visited.
>>>    - Is there a way to manipulate varnish mmap (when file storage is
>>>    used)?
>>>
>>> Would appreciate any help on the above for efficient use of varnish.
>>>
>>> Thanks,
>>>
>>> Pinakee
>>>
>>>
>>>
>>> _______________________________________________
>>> varnish-misc mailing list
>>> varnish-misc at varnish-cache.org
>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20180207/30606fd0/attachment.html>

From pinakee at waltzz.com  Wed Feb  7 13:49:14 2018
From: pinakee at waltzz.com (Pinakee BIswas)
Date: Wed, 7 Feb 2018 19:19:14 +0530
Subject: Varnish Memory usage increase
In-Reply-To: <CAJ6ZYQzC-dOp69kMNz3iRFTF_N9y4-ExeTgAW3WGW9kVWNhoxQ@mail.gmail.com>
References: <6ae5745c-7520-8e97-4f02-0f69104ae418@waltzz.com>
 <CAJ6ZYQzjZ_Lq6oZZ1RWjo7H_4ihPeoLqyJzNYeM+1LjoWiUSVw@mail.gmail.com>
 <e716b3a2-c285-1042-c825-bc5254dd42ea@waltzz.com>
 <CAJ6ZYQz105M6mUCvRH3MzNgEY5_zYwC1N8KCUNTEkp_1z9CrGw@mail.gmail.com>
 <3be7adfb-4f78-7cc9-7a89-e7ebefec2d88@waltzz.com>
 <CAJ6ZYQzC-dOp69kMNz3iRFTF_N9y4-ExeTgAW3WGW9kVWNhoxQ@mail.gmail.com>
Message-ID: <4f298fd0-fd0f-282e-6cae-107a2832bfd0@waltzz.com>

I agree but the reason being we have other applications like DB servers, 
backend server, cache server etc running on the same machine (as our 
scale is not that big). Hence, memory is a sought after resource and 
needs to be optimized as much as possible.

So, if you suggest malloc would be a better storage still, I can change 
the storage to the same.

On 07/02/18 7:09 pm, Guillaume Quintard wrote:
> So, it's probably not related to your issue, but it seems your cached 
> data easily fits in memory, why use the file storage?
>
> -- 
> Guillaume Quintard
>
> On Feb 7, 2018 14:35, "Pinakee BIswas" <pinakee at waltzz.com 
> <mailto:pinakee at waltzz.com>> wrote:
>
>     Following is the data from varnishstat:
>
>     SMA.Transient.g_bytes 15.76K        -2.19K          .          
>     18.53K 15.50K        15.24K
>
>     SMF.s0.g_bytes 673.26M        47.95K          .          672.87M
>     669.89M       668.90M
>
>
>     On 07/02/18 6:51 pm, Guillaume Quintard wrote:
>>     Amount of storage used is just the sum of all the g_bytes fields
>>
>>     -- 
>>     Guillaume Quintard
>>
>>     On Feb 7, 2018 14:12, "Pinakee BIswas" <pinakee at waltzz.com
>>     <mailto:pinakee at waltzz.com>> wrote:
>>
>>         Hi Guillaume,
>>
>>         Thanks for your response and details.
>>
>>         Please find my comments inline:
>>
>>
>>         On 07/02/18 6:21 pm, Guillaume Quintard wrote:
>>>         Hi there,
>>>
>>>         So, varnish memory usage will mostly come from three directions:
>>>         - storage, include Transient, so check the g_bytes fields in
>>>         varnishstat. Passes and shortlived objects will use
>>>         Transient, so you can either reduce those, or limit the
>>>         Transient storage (unbounded by defaukt).
>>         Checked the g_bytes for transient in varnishstat. It's in Kbs.
>>>         - thread workspaces, one thread typically uses one
>>>         workspace, so you can limit the number of threads, or reduce
>>>         the workspace size
>>>         - memory fragmentation: jemalloc will fragment up to 25%,
>>>         not much you can do here.
>>>
>>>         And no, no way to modify the mmap without restarting varnish.
>>>
>>>         May I ask how much data you are caching?
>>         Could you please let me know how to figure out the amount of
>>         data cached?
>>>
>>>         -- 
>>>         Guillaume Quintard
>>>
>>>         On Wed, Feb 7, 2018 at 1:34 PM, Pinakee BIswas
>>>         <pinakee at waltzz.com <mailto:pinakee at waltzz.com>> wrote:
>>>
>>>             Hi,
>>>
>>>             We have been using varnish since more than a year for
>>>             our ecommerce site.
>>>
>>>             Current version is 4.1.8.
>>>
>>>             OS is Linux. RAM is 8GB.
>>>
>>>             I am observing that the varnish memory usage is
>>>             increasing (right now 7%) and so is resident memory
>>>             usage increasing.
>>>
>>>             The storage being used is file:
>>>
>>>             -s file,/tmp/varnish/,${storage}
>>>
>>>             I am not sure about the reason for the increase in
>>>             varnish memory usage:
>>>
>>>               * Is there a way to limit the varnish memory usage?
>>>               * How can I diagnose what is consuming memory?
>>>               * Most of our web pages have maximum 2 days of cache.
>>>                 Also, some of the pages might be least visited.
>>>               * Is there a way to manipulate varnish mmap (when file
>>>                 storage is used)?
>>>
>>>             Would appreciate any help on the above for efficient use
>>>             of varnish.
>>>
>>>             Thanks,
>>>
>>>             Pinakee
>>>
>>>
>>>
>>>             _______________________________________________
>>>             varnish-misc mailing list
>>>             varnish-misc at varnish-cache.org
>>>             <mailto:varnish-misc at varnish-cache.org>
>>>             https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>             <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
>>>
>>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20180207/e14cd0f2/attachment-0001.html>

From pinakee at waltzz.com  Thu Feb  8 06:26:59 2018
From: pinakee at waltzz.com (Pinakee BIswas)
Date: Thu, 8 Feb 2018 11:56:59 +0530
Subject: Varnish Memory usage increase
In-Reply-To: <4f298fd0-fd0f-282e-6cae-107a2832bfd0@waltzz.com>
References: <6ae5745c-7520-8e97-4f02-0f69104ae418@waltzz.com>
 <CAJ6ZYQzjZ_Lq6oZZ1RWjo7H_4ihPeoLqyJzNYeM+1LjoWiUSVw@mail.gmail.com>
 <e716b3a2-c285-1042-c825-bc5254dd42ea@waltzz.com>
 <CAJ6ZYQz105M6mUCvRH3MzNgEY5_zYwC1N8KCUNTEkp_1z9CrGw@mail.gmail.com>
 <3be7adfb-4f78-7cc9-7a89-e7ebefec2d88@waltzz.com>
 <CAJ6ZYQzC-dOp69kMNz3iRFTF_N9y4-ExeTgAW3WGW9kVWNhoxQ@mail.gmail.com>
 <4f298fd0-fd0f-282e-6cae-107a2832bfd0@waltzz.com>
Message-ID: <471d6a97-1348-8b55-4f52-137f5fb6d3fe@waltzz.com>

Hi,

Could you please suggest what would be the optimal configuration for 
varnishd in my scenario - in terms of memory usage and performance?

Thanks,

Pinakee


On 07/02/18 7:19 pm, Pinakee BIswas wrote:
>
> I agree but the reason being we have other applications like DB 
> servers, backend server, cache server etc running on the same machine 
> (as our scale is not that big). Hence, memory is a sought after 
> resource and needs to be optimized as much as possible.
>
> So, if you suggest malloc would be a better storage still, I can 
> change the storage to the same.
>
> On 07/02/18 7:09 pm, Guillaume Quintard wrote:
>> So, it's probably not related to your issue, but it seems your cached 
>> data easily fits in memory, why use the file storage?
>>
>> -- 
>> Guillaume Quintard
>>
>> On Feb 7, 2018 14:35, "Pinakee BIswas" <pinakee at waltzz.com 
>> <mailto:pinakee at waltzz.com>> wrote:
>>
>>     Following is the data from varnishstat:
>>
>>     SMA.Transient.g_bytes 15.76K        -2.19K          .
>>     18.53K        15.50K        15.24K
>>
>>     SMF.s0.g_bytes 673.26M        47.95K          . 672.87M      
>>     669.89M       668.90M
>>
>>
>>     On 07/02/18 6:51 pm, Guillaume Quintard wrote:
>>>     Amount of storage used is just the sum of all the g_bytes fields
>>>
>>>     -- 
>>>     Guillaume Quintard
>>>
>>>     On Feb 7, 2018 14:12, "Pinakee BIswas" <pinakee at waltzz.com
>>>     <mailto:pinakee at waltzz.com>> wrote:
>>>
>>>         Hi Guillaume,
>>>
>>>         Thanks for your response and details.
>>>
>>>         Please find my comments inline:
>>>
>>>
>>>         On 07/02/18 6:21 pm, Guillaume Quintard wrote:
>>>>         Hi there,
>>>>
>>>>         So, varnish memory usage will mostly come from three
>>>>         directions:
>>>>         - storage, include Transient, so check the g_bytes fields
>>>>         in varnishstat. Passes and shortlived objects will use
>>>>         Transient, so you can either reduce those, or limit the
>>>>         Transient storage (unbounded by defaukt).
>>>         Checked the g_bytes for transient in varnishstat. It's in Kbs.
>>>>         - thread workspaces, one thread typically uses one
>>>>         workspace, so you can limit the number of threads, or
>>>>         reduce the workspace size
>>>>         - memory fragmentation: jemalloc will fragment up to 25%,
>>>>         not much you can do here.
>>>>
>>>>         And no, no way to modify the mmap without restarting varnish.
>>>>
>>>>         May I ask how much data you are caching?
>>>         Could you please let me know how to figure out the amount of
>>>         data cached?
>>>>
>>>>         -- 
>>>>         Guillaume Quintard
>>>>
>>>>         On Wed, Feb 7, 2018 at 1:34 PM, Pinakee BIswas
>>>>         <pinakee at waltzz.com <mailto:pinakee at waltzz.com>> wrote:
>>>>
>>>>             Hi,
>>>>
>>>>             We have been using varnish since more than a year for
>>>>             our ecommerce site.
>>>>
>>>>             Current version is 4.1.8.
>>>>
>>>>             OS is Linux. RAM is 8GB.
>>>>
>>>>             I am observing that the varnish memory usage is
>>>>             increasing (right now 7%) and so is resident memory
>>>>             usage increasing.
>>>>
>>>>             The storage being used is file:
>>>>
>>>>             -s file,/tmp/varnish/,${storage}
>>>>
>>>>             I am not sure about the reason for the increase in
>>>>             varnish memory usage:
>>>>
>>>>               * Is there a way to limit the varnish memory usage?
>>>>               * How can I diagnose what is consuming memory?
>>>>               * Most of our web pages have maximum 2 days of cache.
>>>>                 Also, some of the pages might be least visited.
>>>>               * Is there a way to manipulate varnish mmap (when
>>>>                 file storage is used)?
>>>>
>>>>             Would appreciate any help on the above for efficient
>>>>             use of varnish.
>>>>
>>>>             Thanks,
>>>>
>>>>             Pinakee
>>>>
>>>>
>>>>
>>>>             _______________________________________________
>>>>             varnish-misc mailing list
>>>>             varnish-misc at varnish-cache.org
>>>>             <mailto:varnish-misc at varnish-cache.org>
>>>>             https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>             <https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc>
>>>>
>>>>
>>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20180208/94b5fbca/attachment.html>

From guillaume at varnish-software.com  Thu Feb  8 08:31:31 2018
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Thu, 8 Feb 2018 09:31:31 +0100
Subject: Varnish Memory usage increase
In-Reply-To: <471d6a97-1348-8b55-4f52-137f5fb6d3fe@waltzz.com>
References: <6ae5745c-7520-8e97-4f02-0f69104ae418@waltzz.com>
 <CAJ6ZYQzjZ_Lq6oZZ1RWjo7H_4ihPeoLqyJzNYeM+1LjoWiUSVw@mail.gmail.com>
 <e716b3a2-c285-1042-c825-bc5254dd42ea@waltzz.com>
 <CAJ6ZYQz105M6mUCvRH3MzNgEY5_zYwC1N8KCUNTEkp_1z9CrGw@mail.gmail.com>
 <3be7adfb-4f78-7cc9-7a89-e7ebefec2d88@waltzz.com>
 <CAJ6ZYQzC-dOp69kMNz3iRFTF_N9y4-ExeTgAW3WGW9kVWNhoxQ@mail.gmail.com>
 <4f298fd0-fd0f-282e-6cae-107a2832bfd0@waltzz.com>
 <471d6a97-1348-8b55-4f52-137f5fb6d3fe@waltzz.com>
Message-ID: <CAJ6ZYQxXFqE11MgYT9WgvvTdaw=aP7bMMbrs2xY+5ms-FPLbdg@mail.gmail.com>

There's no optimal configuration, only compromises, but I can point you
there:
https://info.varnish-software.com/blog/understanding-varnish-cache-memory-usage

-- 
Guillaume Quintard

On Thu, Feb 8, 2018 at 7:26 AM, Pinakee BIswas <pinakee at waltzz.com> wrote:

> Hi,
>
> Could you please suggest what would be the optimal configuration for
> varnishd in my scenario - in terms of memory usage and performance?
>
> Thanks,
>
> Pinakee
>
> On 07/02/18 7:19 pm, Pinakee BIswas wrote:
>
> I agree but the reason being we have other applications like DB servers,
> backend server, cache server etc running on the same machine (as our scale
> is not that big). Hence, memory is a sought after resource and needs to be
> optimized as much as possible.
>
> So, if you suggest malloc would be a better storage still, I can change
> the storage to the same.
> On 07/02/18 7:09 pm, Guillaume Quintard wrote:
>
> So, it's probably not related to your issue, but it seems your cached data
> easily fits in memory, why use the file storage?
>
> --
> Guillaume Quintard
>
> On Feb 7, 2018 14:35, "Pinakee BIswas" <pinakee at waltzz.com> wrote:
>
>> Following is the data from varnishstat:
>>
>> SMA.Transient.g_bytes
>> 15.76K        -2.19K          .           18.53K        15.50K        15.24K
>>
>> SMF.s0.g_bytes
>> 673.26M        47.95K          .          672.87M       669.89M
>> 668.90M
>>
>>
>> On 07/02/18 6:51 pm, Guillaume Quintard wrote:
>>
>> Amount of storage used is just the sum of all the g_bytes fields
>>
>> --
>> Guillaume Quintard
>>
>> On Feb 7, 2018 14:12, "Pinakee BIswas" <pinakee at waltzz.com> wrote:
>>
>>> Hi Guillaume,
>>>
>>> Thanks for your response and details.
>>>
>>> Please find my comments inline:
>>>
>>> On 07/02/18 6:21 pm, Guillaume Quintard wrote:
>>>
>>> Hi there,
>>>
>>> So, varnish memory usage will mostly come from three directions:
>>> - storage, include Transient, so check the g_bytes fields in
>>> varnishstat. Passes and shortlived objects will use Transient, so you can
>>> either reduce those, or limit the Transient storage (unbounded by defaukt).
>>>
>>> Checked the g_bytes for transient in varnishstat. It's in Kbs.
>>>
>>> - thread workspaces, one thread typically uses one workspace, so you can
>>> limit the number of threads, or reduce the workspace size
>>> - memory fragmentation: jemalloc will fragment up to 25%, not much you
>>> can do here.
>>>
>>> And no, no way to modify the mmap without restarting varnish.
>>>
>>> May I ask how much data you are caching?
>>>
>>> Could you please let me know how to figure out the amount of data cached?
>>>
>>>
>>> --
>>> Guillaume Quintard
>>>
>>> On Wed, Feb 7, 2018 at 1:34 PM, Pinakee BIswas <pinakee at waltzz.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> We have been using varnish since more than a year for our ecommerce
>>>> site.
>>>>
>>>> Current version is 4.1.8.
>>>>
>>>> OS is Linux. RAM is 8GB.
>>>>
>>>> I am observing that the varnish memory usage is increasing (right now
>>>> 7%) and so is resident memory usage increasing.
>>>>
>>>> The storage being used is file:
>>>>
>>>> -s file,/tmp/varnish/,${storage}
>>>>
>>>> I am not sure about the reason for the increase in varnish memory usage:
>>>>
>>>>    - Is there a way to limit the varnish memory usage?
>>>>    - How can I diagnose what is consuming memory?
>>>>    - Most of our web pages have maximum 2 days of cache. Also, some of
>>>>    the pages might be least visited.
>>>>    - Is there a way to manipulate varnish mmap (when file storage is
>>>>    used)?
>>>>
>>>> Would appreciate any help on the above for efficient use of varnish.
>>>>
>>>> Thanks,
>>>>
>>>> Pinakee
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> varnish-misc mailing list
>>>> varnish-misc at varnish-cache.org
>>>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>>>
>>>
>>>
>>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20180208/6ada6693/attachment-0001.html>

From ilinuxer.85 at gmail.com  Sun Feb 11 23:56:08 2018
From: ilinuxer.85 at gmail.com (jack Linux)
Date: Mon, 12 Feb 2018 01:56:08 +0200
Subject: Cannot start varnish i am getting error collect2: cannot find 'ld'
 Running C-compiler failed, exited with 1
Message-ID: <CAGHcRxZ0QPJNUMrHBHaVYpMGqfpZJvqdqb_1mhymtTz0zhq-0w@mail.gmail.com>

#varnishd -C -f /etc/varnish/default.vcl

Message from C-compiler:
collect2: cannot find 'ld'
Running C-compiler failed, exited with 1

LD command already exist in the server

# whereis ld
ld: /usr/bin/ld /usr/share/man/man1/ld.1.gz

# ld
ld: no input files

I didn't make any custom configuration in /etc/varnish/default.vcl or
/etc/sysconfig/varnish , this is just a new fresh install by yum
install varnish , no error or warring shows up during the
installations

My OS : CentOS release 6.9 (Final) 64bit

From guillaume at varnish-software.com  Thu Feb 15 10:25:51 2018
From: guillaume at varnish-software.com (Guillaume Quintard)
Date: Thu, 15 Feb 2018 11:25:51 +0100
Subject: Cannot start varnish i am getting error collect2: cannot find
 'ld' Running C-compiler failed, exited with 1
In-Reply-To: <CAGHcRxZ0QPJNUMrHBHaVYpMGqfpZJvqdqb_1mhymtTz0zhq-0w@mail.gmail.com>
References: <CAGHcRxZ0QPJNUMrHBHaVYpMGqfpZJvqdqb_1mhymtTz0zhq-0w@mail.gmail.com>
Message-ID: <CAJ6ZYQyz5k=PnqThKQm0eZF_a+dxHCn8fihMowirC0W-14dRmg@mail.gmail.com>

Hi,

I'm a bit puzzled on that one, what versions of gcc and varnish are you
using?

-- 
Guillaume Quintard

On Mon, Feb 12, 2018 at 12:56 AM, jack Linux <ilinuxer.85 at gmail.com> wrote:

> #varnishd -C -f /etc/varnish/default.vcl
>
> Message from C-compiler:
> collect2: cannot find 'ld'
> Running C-compiler failed, exited with 1
>
> LD command already exist in the server
>
> # whereis ld
> ld: /usr/bin/ld /usr/share/man/man1/ld.1.gz
>
> # ld
> ld: no input files
>
> I didn't make any custom configuration in /etc/varnish/default.vcl or
> /etc/sysconfig/varnish , this is just a new fresh install by yum
> install varnish , no error or warring shows up during the
> installations
>
> My OS : CentOS release 6.9 (Final) 64bit
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20180215/371b6438/attachment.html>

From kokoniimasu at gmail.com  Fri Feb 16 06:42:37 2018
From: kokoniimasu at gmail.com (kokoniimasu)
Date: Fri, 16 Feb 2018 15:42:37 +0900
Subject: Add nuke_limited counter(Number of times reached nuke_limit.)
Message-ID: <CAJihfGAdYrz2-S9QoNw9c6SyQrLm_TL0gGEWbvh6Wahurq7m7Q@mail.gmail.com>

Hi,

I got a strange error the other day.
Status code is 200, but body was incomplete.
This error occurred because nuke_limit was reached.(It does not 503,
because beresp.do_stream=true is default)
I think this error is hard to notice.
I would like to add a new counter to make notice it.

https://github.com/xcir/varnish-cache/commit/3a1f45d05dcdc0d22a71148c25ae2dedcf93e6ea

I think it can see that need to increase nuke_limit by this counter.
Can I get your opinion?

Regards,
--
Shohei Tanaka(@xcir)
http://blog.xcir.net/ (JP)

From dridi at varni.sh  Fri Feb 16 08:21:49 2018
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 16 Feb 2018 09:21:49 +0100
Subject: Add nuke_limited counter(Number of times reached nuke_limit.)
In-Reply-To: <CAJihfGAdYrz2-S9QoNw9c6SyQrLm_TL0gGEWbvh6Wahurq7m7Q@mail.gmail.com>
References: <CAJihfGAdYrz2-S9QoNw9c6SyQrLm_TL0gGEWbvh6Wahurq7m7Q@mail.gmail.com>
Message-ID: <CABoVN9D+2HGajEQgZcMCUqb_8-cc--SHpzL1uAsMxKL=joKDeQ@mail.gmail.com>

On Fri, Feb 16, 2018 at 7:42 AM, kokoniimasu <kokoniimasu at gmail.com> wrote:
> Hi,
>
> I got a strange error the other day.
> Status code is 200, but body was incomplete.
> This error occurred because nuke_limit was reached.(It does not 503,
> because beresp.do_stream=true is default)
> I think this error is hard to notice.
> I would like to add a new counter to make notice it.
>
> https://github.com/xcir/varnish-cache/commit/3a1f45d05dcdc0d22a71148c25ae2dedcf93e6ea
>
> I think it can see that need to increase nuke_limit by this counter.
> Can I get your opinion?

Hi,

Sounds like a great idea to me, the patch looks good too!

Dridi

From kokoniimasu at gmail.com  Fri Feb 16 09:56:23 2018
From: kokoniimasu at gmail.com (kokoniimasu)
Date: Fri, 16 Feb 2018 18:56:23 +0900
Subject: Add nuke_limited counter(Number of times reached nuke_limit.)
In-Reply-To: <CABoVN9D+2HGajEQgZcMCUqb_8-cc--SHpzL1uAsMxKL=joKDeQ@mail.gmail.com>
References: <CAJihfGAdYrz2-S9QoNw9c6SyQrLm_TL0gGEWbvh6Wahurq7m7Q@mail.gmail.com>
 <CABoVN9D+2HGajEQgZcMCUqb_8-cc--SHpzL1uAsMxKL=joKDeQ@mail.gmail.com>
Message-ID: <CAJihfGCCfTiSqLQ6CBNRX7ETq=JnAjmPFGQAz-z6gJ_z-zTUEg@mail.gmail.com>

Hi, Dridi.

Thank you for your comment!
I will create pull-request.

2018-02-16 17:21 GMT+09:00 Dridi Boukelmoune <dridi at varni.sh>:
> On Fri, Feb 16, 2018 at 7:42 AM, kokoniimasu <kokoniimasu at gmail.com> wrote:
>> Hi,
>>
>> I got a strange error the other day.
>> Status code is 200, but body was incomplete.
>> This error occurred because nuke_limit was reached.(It does not 503,
>> because beresp.do_stream=true is default)
>> I think this error is hard to notice.
>> I would like to add a new counter to make notice it.
>>
>> https://github.com/xcir/varnish-cache/commit/3a1f45d05dcdc0d22a71148c25ae2dedcf93e6ea
>>
>> I think it can see that need to increase nuke_limit by this counter.
>> Can I get your opinion?
>
> Hi,
>
> Sounds like a great idea to me, the patch looks good too!
>
> Dridi

From dridi at varni.sh  Fri Feb 16 10:10:58 2018
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Fri, 16 Feb 2018 11:10:58 +0100
Subject: Add nuke_limited counter(Number of times reached nuke_limit.)
In-Reply-To: <CAJihfGCCfTiSqLQ6CBNRX7ETq=JnAjmPFGQAz-z6gJ_z-zTUEg@mail.gmail.com>
References: <CAJihfGAdYrz2-S9QoNw9c6SyQrLm_TL0gGEWbvh6Wahurq7m7Q@mail.gmail.com>
 <CABoVN9D+2HGajEQgZcMCUqb_8-cc--SHpzL1uAsMxKL=joKDeQ@mail.gmail.com>
 <CAJihfGCCfTiSqLQ6CBNRX7ETq=JnAjmPFGQAz-z6gJ_z-zTUEg@mail.gmail.com>
Message-ID: <CABoVN9Cy++9T6oNWjB-vpqfU7U714_nXq3Ghbj82Jr_Agdh5Xw@mail.gmail.com>

On Fri, Feb 16, 2018 at 10:56 AM, kokoniimasu <kokoniimasu at gmail.com> wrote:
> Hi, Dridi.
>
> Thank you for your comment!
> I will create pull-request.

You may want to name the counter n_lru_limited or something similar to
be consistent with related counters.

Cheers

From miguel_3_gonzalez at yahoo.es  Fri Feb 16 21:21:33 2018
From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=)
Date: Fri, 16 Feb 2018 22:21:33 +0100
Subject: redirect url and banning
Message-ID: <10371643-80bc-c813-a7d6-d5974bd3a209@yahoo.es>

Hi,

  I have changed an URL from:

  https://www.mysite.com/word1-word2/

  https://www.mysite.com/word3-word2/

  and changed my .htaccess accordingly.

  I have successfully banned www.mysite.com/word1-word2 without the
ending / but not with / ending using:

  ban req.http.host ~ www.mysite.com/word1-word2

  and

  ban req.http.host ~ www.mysite.com/word1-word2/

  So option without ending /, 301 redirect works but with / doesn?t.

  I have googled around but I haven?t found any solution.

  Regards,

  Miguel

---
This email has been checked for viruses by AVG.
http://www.avg.com


From miguel_3_gonzalez at yahoo.es  Fri Feb 16 22:21:52 2018
From: miguel_3_gonzalez at yahoo.es (=?UTF-8?Q?Miguel_Gonz=c3=a1lez?=)
Date: Fri, 16 Feb 2018 23:21:52 +0100
Subject: redirect url and banning
In-Reply-To: <10371643-80bc-c813-a7d6-d5974bd3a209@yahoo.es>
References: <10371643-80bc-c813-a7d6-d5974bd3a209@yahoo.es>
Message-ID: <b158e97e-3898-9ce2-51d6-94a639ab4c75@yahoo.es>

On 02/16/18 10:21 PM, Miguel Gonz?lez wrote:
> Hi,
> 
>   I have changed an URL from:
> 
>   https://www.mysite.com/word1-word2/
> 
>   https://www.mysite.com/word3-word2/
> 
>   and changed my .htaccess accordingly.
> 
>   I have successfully banned www.mysite.com/word1-word2 without the
> ending / but not with / ending using:
> 
>   ban req.http.host ~ www.mysite.com/word1-word2
> 
>   and
> 
>   ban req.http.host ~ www.mysite.com/word1-word2/
> 
>   So option without ending /, 301 redirect works but with / doesn?t.
> 
>   I have googled around but I haven?t found any solution.
> 
>   Regards,
> 
>   Miguel
> 


The issue was on the .htaccess redirect, I had to use:

RewriteRule ^word1\-word2/?$ "https\:\/\/www\.mysite\.com\/word3\-word2
[R=301,L]

instead of:

RewriteRule ^word1\-word2$ "https\:\/\/www\.mysite\.com\/word3\-word2
[R=301,L]


Sorry!

Miguel



---
This email has been checked for viruses by AVG.
http://www.avg.com


From dridi at varni.sh  Mon Feb 19 12:57:15 2018
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Mon, 19 Feb 2018 13:57:15 +0100
Subject: redirect url and banning
In-Reply-To: <b158e97e-3898-9ce2-51d6-94a639ab4c75@yahoo.es>
References: <10371643-80bc-c813-a7d6-d5974bd3a209@yahoo.es>
 <b158e97e-3898-9ce2-51d6-94a639ab4c75@yahoo.es>
Message-ID: <CABoVN9B7H3XwXtzFf1=ATo07jvXvUxud+kRSmH-wR8Wkh5q9Xg@mail.gmail.com>

On Fri, Feb 16, 2018 at 11:21 PM, Miguel Gonz?lez
<miguel_3_gonzalez at yahoo.es> wrote:
> On 02/16/18 10:21 PM, Miguel Gonz?lez wrote:
>> Hi,
>>
>>   I have changed an URL from:
>>
>>   https://www.mysite.com/word1-word2/
>>
>>   https://www.mysite.com/word3-word2/
>>
>>   and changed my .htaccess accordingly.
>>
>>   I have successfully banned www.mysite.com/word1-word2 without the
>> ending / but not with / ending using:
>>
>>   ban req.http.host ~ www.mysite.com/word1-word2
>>
>>   and
>>
>>   ban req.http.host ~ www.mysite.com/word1-word2/
>>
>>   So option without ending /, 301 redirect works but with / doesn?t.
>>
>>   I have googled around but I haven?t found any solution.

None worked, it was probably a coincidence that one appeared to work.
The host header should only contain the authority, so the ban you are
looking for is rather:

    ban req.http.host == www.mysite.com && req.url ~ /word1-word2/?

That should take care of both cases in a single ban.

Dridi

From Naveen.Goswami at springer.com  Tue Feb 20 14:07:42 2018
From: Naveen.Goswami at springer.com (Goswami, Naveen, Springer SBM DE)
Date: Tue, 20 Feb 2018 14:07:42 +0000
Subject: Health Check on different port
Message-ID: <1519135662592.18028@springer.com>

Dear Varnish-Team,


First of all I would like to congratulate you for such an awesome product.


We wanted to send traffic directly to go-routers, which require us to send traffic to service port 80, but, do the health-check on port 8080. As doing the health check on port 80 is deprecated on go-routers.


We came across one such vmod oob_probe, which let us to do exactly the same what we are looking for. But this vmod is only compatible with varnish 4.1 and we are currently using the varnish version 5.2.


The another method we tried, we defined 2 backends in VCL one for port 80 and another for port 8080 for each go-router.


Then we applied round-robin director on backends wit?h port 8080. In assumption that director_round_robin.backend() will give us the name of the selected backend, where the incoming request will be forwarded to. Thereafter, we can modify the backend for example in vcl_backend_fetch() and set the correct definition of the backend, if the backend selected is go-router1:8080 by the round_robin director, we would set the backend go-router1:80.

But, this was also problematic as director_round_robin.backend() just return the name of director object, but, not the name of the chosen backend . Is there a way to fetch the name of backend chosen by the director before the request is sent to the backend? I am familiar it is possible to do so in vcl_backend_response. However, this is not very useful for us, in this scenario.


Best regards,

Naveen Goswami



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20180220/236860da/attachment.html>

From dridi at varni.sh  Tue Feb 20 14:25:57 2018
From: dridi at varni.sh (Dridi Boukelmoune)
Date: Tue, 20 Feb 2018 15:25:57 +0100
Subject: Health Check on different port
In-Reply-To: <1519135662592.18028@springer.com>
References: <1519135662592.18028@springer.com>
Message-ID: <CABoVN9DA=2ovh0kkhNoPNGyGrj53=urE5wnGJd_-TF8G0cGCKg@mail.gmail.com>

On Tue, Feb 20, 2018 at 3:07 PM, Goswami, Naveen, Springer SBM DE
<Naveen.Goswami at springer.com> wrote:
> Dear Varnish-Team,
>
> First of all I would like to congratulate you for such an awesome product.
>
> We wanted to send traffic directly to go-routers, which require us to send
> traffic to service port 80, but, do the health-check on port 8080. As doing
> the health check on port 80 is deprecated on go-routers.

You should reach out to Geoff (CC'd) as he's actually part to see
whether he or UPLEX could make oob_probe compatible with 5.2.

According to https://varnish-cache.org/vmods/ they can provide
commercial support for this module.

Otherwise one option is to have your own "probes" (or monitoring)
running outside of Varnish and use the CLI to set the backend heath,
for example via varnishadm.

Dridi

From Naveen.Goswami at springer.com  Tue Feb 20 14:35:56 2018
From: Naveen.Goswami at springer.com (Goswami, Naveen, Springer SBM DE)
Date: Tue, 20 Feb 2018 14:35:56 +0000
Subject: Health Check on different port
In-Reply-To: <CABoVN9DA=2ovh0kkhNoPNGyGrj53=urE5wnGJd_-TF8G0cGCKg@mail.gmail.com>
References: <1519135662592.18028@springer.com>,
 <CABoVN9DA=2ovh0kkhNoPNGyGrj53=urE5wnGJd_-TF8G0cGCKg@mail.gmail.com>
Message-ID: <1519137356116.91523@springer.com>

Thanks Dridi for quick and helpful advice. Much appreciate it. 

@Geoff, would that be possible to make oob_probe compatible with 5.2 ? 

Naveen 
________________________________________
From: Dridi Boukelmoune <dridi at varni.sh>
Sent: Tuesday, February 20, 2018 3:25 PM
To: Goswami, Naveen, Springer SBM DE
Cc: varnish-misc at varnish-cache.org; Engineering Enablement; Geoff Simmons
Subject: Re: Health Check on different port

On Tue, Feb 20, 2018 at 3:07 PM, Goswami, Naveen, Springer SBM DE
<Naveen.Goswami at springer.com> wrote:
> Dear Varnish-Team,
>
> First of all I would like to congratulate you for such an awesome product.
>
> We wanted to send traffic directly to go-routers, which require us to send
> traffic to service port 80, but, do the health-check on port 8080. As doing
> the health check on port 80 is deprecated on go-routers.

You should reach out to Geoff (CC'd) as he's actually part to see
whether he or UPLEX could make oob_probe compatible with 5.2.

According to https://varnish-cache.org/vmods/ they can provide
commercial support for this module.

Otherwise one option is to have your own "probes" (or monitoring)
running outside of Varnish and use the CLI to set the backend heath,
for example via varnishadm.

Dridi