Varnish malloc issue

Thu Sep 15 08:48:44 CEST 2016

That can't, hurt.

On Sep 15, 2016 03:31, "sujith pv" <sujithnss at gmail.com> wrote:

> No..I'm planning to try that next. So during that time also so do we need
> to cap transient storage ??
>
> On 15-Sep-2016 01:18, "Guillaume Quintard" <guillaume at varnish-software.com>
> wrote:
>
>> Have you tried the packages from your distribution?
>>
>> On Sep 14, 2016 20:09, "sujith pv" <sujithnss at gmail.com> wrote:
>>
>>> After the full load of requests..g_space became minimal and g_bytes
>>> became like 8 Gb..I had assigned malloc as 8 gb
>>>
>>> On 14-Sep-2016 22:42, "Guillaume Quintard" <
>>> guillaume at varnish-software.com> wrote:
>>>
>>>> What about the other g_space/g_bytes?
>>>>
>>>> On Sep 14, 2016 18:33, "sujith pv" <sujithnss at gmail.com> wrote:
>>>>
>>>>> It was showing as 0 for Transient_g_bytes and 1Gb for g_space as I
>>>>> mentioned in my mail.
>>>>>
>>>>> On Wed, Sep 14, 2016 at 7:45 PM, Guillaume Quintard <
>>>>> guillaume at varnish-software.com> wrote:
>>>>>
>>>>>> please keep using the mailing list, some other people may have ideas.
>>>>>>
>>>>>> what's the output of "varnishstat -1 | grep g_bytes" ?
>>>>>>
>>>>>> --
>>>>>> Guillaume Quintard
>>>>>>
>>>>>> On Wed, Sep 14, 2016 at 3:50 PM, sujith pv <sujithnss at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Guillaume
>>>>>>>
>>>>>>> I have tried capping Transient and I could see the respective values
>>>>>>> under SMA.transient.g_bytes/g_space. But still on load I didnt see this
>>>>>>> g_bytes being used and was showing 0 only.
>>>>>>>
>>>>>>> Best Regards
>>>>>>> Sujith P V
>>>>>>>
>>>>>>> On Tue, Sep 6, 2016 at 7:39 PM, Guillaume Quintard <
>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>
>>>>>>>> Maybe, not sure. Unless you have a **good** reason to do so, use
>>>>>>>> the packages.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Guillaume Quintard
>>>>>>>>
>>>>>>>> On Tue, Sep 6, 2016 at 3:19 PM, sujith pv <sujithnss at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks Guillaume. Actually in my end Varnish was compiled here and
>>>>>>>>> installed (I was not part of that initially :-)). Will this have any impact
>>>>>>>>> compared to as you have mentioned like try from a packaged version.
>>>>>>>>>
>>>>>>>>> On Tue, Sep 6, 2016 at 6:46 PM, Guillaume Quintard <
>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>
>>>>>>>>>> It's just a question of storage, shortlive objects got into
>>>>>>>>>> transient storage instead of the one(s) you defined in the command line.
>>>>>>>>>> But it's still a cache storage, based on malloc, that is by default
>>>>>>>>>> unlimited in size.
>>>>>>>>>>
>>>>>>>>>> The rationale is that for object with such short ttl+grace+keep,
>>>>>>>>>> a simple malloc is good enough.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 6, 2016 at 3:04 PM, sujith pv <sujithnss at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Actually I'm confused around the short lived objects. Is these
>>>>>>>>>>> objects are something like cached objects or who is creating these objects.
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 6, 2016 at 1:57 PM, Guillaume Quintard <
>>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Sorry, I didn't get your question.
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Sep 6, 2016 at 8:31 AM, sujith pv <sujithnss at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks Guillaume. May be Im asking a very basic question , but
>>>>>>>>>>>>> still how this short lived objects are created and I'm not giving any
>>>>>>>>>>>>> settings for the same as well.
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Sep 6, 2016 at 11:54 AM, Guillaume Quintard <
>>>>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> https://www.varnish-cache.org/docs/4.1/reference/varnishd.ht
>>>>>>>>>>>>>> ml#shortlived
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sep 6, 2016 08:07, "sujith pv" <sujithnss at gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks Guillaume. I will try the transient settings first
>>>>>>>>>>>>>>> and verify the same. Also could you please elaborate on the short lived
>>>>>>>>>>>>>>> objects.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Best Regards
>>>>>>>>>>>>>>> Sujith P V
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Fri, Sep 2, 2016 at 11:24 AM, Guillaume Quintard <
>>>>>>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Viktor,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Have a look here https://www.varnish-cache
>>>>>>>>>>>>>>>> .org/docs/trunk/users-guide/st
>>>>>>>>>>>>>>>> orage-backends.html#transient-storage
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Sujith, please try to use a packaged version from your
>>>>>>>>>>>>>>>> distribution.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Fri, Sep 2, 2016 at 1:18 AM, Viktor Villafuerte <
>>>>>>>>>>>>>>>> viktor.villafuerte at optusnet.com.au> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi Guillaume,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Thu 01 Sep 2016 09:53:22, Guillaume Quintard wrote:
>>>>>>>>>>>>>>>>> > Is that virtual or real memory?
>>>>>>>>>>>>>>>>> > Try to cap your Transient storage to 1GB, you may be
>>>>>>>>>>>>>>>>> overloaded with short
>>>>>>>>>>>>>>>>> > lived objects.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Could you elaborate on this bit more? I've got very
>>>>>>>>>>>>>>>>> similar problem
>>>>>>>>>>>>>>>>> here. Eg: Total memory 384g
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> VIRT: 487g
>>>>>>>>>>>>>>>>> RES:  311g
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> free says: 46g free
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> but SWAP is 91.3% used :(
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> How can this be?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> v
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > On Sep 1, 2016 9:40 AM, "sujith pv" <sujithnss at gmail.com>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>>>> > > Hi All
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > > I had already mailed this query long back but this
>>>>>>>>>>>>>>>>> time putting in a
>>>>>>>>>>>>>>>>> > > different manner.
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > >    - We are using Varnish 4.0 in our end. We have a
>>>>>>>>>>>>>>>>> machine with memory
>>>>>>>>>>>>>>>>> > >    of 16G with 8G being malloc for Varnish. We have a
>>>>>>>>>>>>>>>>> TTL for 2 hrs as well.
>>>>>>>>>>>>>>>>> > >    - During our peak traffic, when we see the total
>>>>>>>>>>>>>>>>> memory of the machine
>>>>>>>>>>>>>>>>> > >    reaching 90% and like varnishd process is taking
>>>>>>>>>>>>>>>>> some 89% .
>>>>>>>>>>>>>>>>> > >    - So I'm just confused even though we had allocated
>>>>>>>>>>>>>>>>> just 8G malloc
>>>>>>>>>>>>>>>>> > >    which is like 50% of the total memory, how the
>>>>>>>>>>>>>>>>> process is eating up 89% of
>>>>>>>>>>>>>>>>> > >    memory and the memory is not releasing even after
>>>>>>>>>>>>>>>>> TTL.
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > > Any help please...
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > > Best Regards
>>>>>>>>>>>>>>>>> > > Sujith P V
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>> > > _______________________________________________
>>>>>>>>>>>>>>>>> > > varnish-misc mailing list
>>>>>>>>>>>>>>>>> > > varnish-misc at varnish-cache.org
>>>>>>>>>>>>>>>>> > > https://www.varnish-cache.org/
>>>>>>>>>>>>>>>>> lists/mailman/listinfo/varnish-misc
>>>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>>>>>>>> > varnish-misc mailing list
>>>>>>>>>>>>>>>>> > varnish-misc at varnish-cache.org
>>>>>>>>>>>>>>>>> > https://www.varnish-cache.org/
>>>>>>>>>>>>>>>>> lists/mailman/listinfo/varnish-misc
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Viktor Villafuerte
>>>>>>>>>>>>>>>>> Optus Internet Engineering
>>>>>>>>>>>>>>>>> t: +61 2 80825265
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20160915/799ff909/attachment.html>