Varnish malloc issue

Wed Sep 14 19:12:08 CEST 2016

What about the other g_space/g_bytes?

On Sep 14, 2016 18:33, "sujith pv" <sujithnss at gmail.com> wrote:

> It was showing as 0 for Transient_g_bytes and 1Gb for g_space as I
> mentioned in my mail.
>
> On Wed, Sep 14, 2016 at 7:45 PM, Guillaume Quintard <
> guillaume at varnish-software.com> wrote:
>
>> please keep using the mailing list, some other people may have ideas.
>>
>> what's the output of "varnishstat -1 | grep g_bytes" ?
>>
>> --
>> Guillaume Quintard
>>
>> On Wed, Sep 14, 2016 at 3:50 PM, sujith pv <sujithnss at gmail.com> wrote:
>>
>>> Hi Guillaume
>>>
>>> I have tried capping Transient and I could see the respective values
>>> under SMA.transient.g_bytes/g_space. But still on load I didnt see this
>>> g_bytes being used and was showing 0 only.
>>>
>>> Best Regards
>>> Sujith P V
>>>
>>> On Tue, Sep 6, 2016 at 7:39 PM, Guillaume Quintard <
>>> guillaume at varnish-software.com> wrote:
>>>
>>>> Maybe, not sure. Unless you have a **good** reason to do so, use the
>>>> packages.
>>>>
>>>> --
>>>> Guillaume Quintard
>>>>
>>>> On Tue, Sep 6, 2016 at 3:19 PM, sujith pv <sujithnss at gmail.com> wrote:
>>>>
>>>>> Thanks Guillaume. Actually in my end Varnish was compiled here and
>>>>> installed (I was not part of that initially :-)). Will this have any impact
>>>>> compared to as you have mentioned like try from a packaged version.
>>>>>
>>>>> On Tue, Sep 6, 2016 at 6:46 PM, Guillaume Quintard <
>>>>> guillaume at varnish-software.com> wrote:
>>>>>
>>>>>> It's just a question of storage, shortlive objects got into transient
>>>>>> storage instead of the one(s) you defined in the command line. But it's
>>>>>> still a cache storage, based on malloc, that is by default unlimited in
>>>>>> size.
>>>>>>
>>>>>> The rationale is that for object with such short ttl+grace+keep, a
>>>>>> simple malloc is good enough.
>>>>>>
>>>>>> --
>>>>>> Guillaume Quintard
>>>>>>
>>>>>> On Tue, Sep 6, 2016 at 3:04 PM, sujith pv <sujithnss at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Actually I'm confused around the short lived objects. Is these
>>>>>>> objects are something like cached objects or who is creating these objects.
>>>>>>>
>>>>>>> On Tue, Sep 6, 2016 at 1:57 PM, Guillaume Quintard <
>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>
>>>>>>>> Sorry, I didn't get your question.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Guillaume Quintard
>>>>>>>>
>>>>>>>> On Tue, Sep 6, 2016 at 8:31 AM, sujith pv <sujithnss at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Thanks Guillaume. May be Im asking a very basic question , but
>>>>>>>>> still how this short lived objects are created and I'm not giving any
>>>>>>>>> settings for the same as well.
>>>>>>>>>
>>>>>>>>> On Tue, Sep 6, 2016 at 11:54 AM, Guillaume Quintard <
>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>
>>>>>>>>>> https://www.varnish-cache.org/docs/4.1/reference/varnishd.ht
>>>>>>>>>> ml#shortlived
>>>>>>>>>>
>>>>>>>>>> On Sep 6, 2016 08:07, "sujith pv" <sujithnss at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks Guillaume. I will try the transient settings first and
>>>>>>>>>>> verify the same. Also could you please elaborate on the short lived objects.
>>>>>>>>>>>
>>>>>>>>>>> Best Regards
>>>>>>>>>>> Sujith P V
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Sep 2, 2016 at 11:24 AM, Guillaume Quintard <
>>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Viktor,
>>>>>>>>>>>>
>>>>>>>>>>>> Have a look here https://www.varnish-cache
>>>>>>>>>>>> .org/docs/trunk/users-guide/storage-backends.html#transient-
>>>>>>>>>>>> storage
>>>>>>>>>>>>
>>>>>>>>>>>> Sujith, please try to use a packaged version from your
>>>>>>>>>>>> distribution.
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Sep 2, 2016 at 1:18 AM, Viktor Villafuerte <
>>>>>>>>>>>> viktor.villafuerte at optusnet.com.au> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Guillaume,
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Thu 01 Sep 2016 09:53:22, Guillaume Quintard wrote:
>>>>>>>>>>>>> > Is that virtual or real memory?
>>>>>>>>>>>>> > Try to cap your Transient storage to 1GB, you may be
>>>>>>>>>>>>> overloaded with short
>>>>>>>>>>>>> > lived objects.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Could you elaborate on this bit more? I've got very similar
>>>>>>>>>>>>> problem
>>>>>>>>>>>>> here. Eg: Total memory 384g
>>>>>>>>>>>>>
>>>>>>>>>>>>> VIRT: 487g
>>>>>>>>>>>>> RES:  311g
>>>>>>>>>>>>>
>>>>>>>>>>>>> free says: 46g free
>>>>>>>>>>>>>
>>>>>>>>>>>>> but SWAP is 91.3% used :(
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> How can this be?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>
>>>>>>>>>>>>> v
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > On Sep 1, 2016 9:40 AM, "sujith pv" <sujithnss at gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> >
>>>>>>>>>>>>> > > Hi All
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > > I had already mailed this query long back but this time
>>>>>>>>>>>>> putting in a
>>>>>>>>>>>>> > > different manner.
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > >    - We are using Varnish 4.0 in our end. We have a
>>>>>>>>>>>>> machine with memory
>>>>>>>>>>>>> > >    of 16G with 8G being malloc for Varnish. We have a TTL
>>>>>>>>>>>>> for 2 hrs as well.
>>>>>>>>>>>>> > >    - During our peak traffic, when we see the total memory
>>>>>>>>>>>>> of the machine
>>>>>>>>>>>>> > >    reaching 90% and like varnishd process is taking some
>>>>>>>>>>>>> 89% .
>>>>>>>>>>>>> > >    - So I'm just confused even though we had allocated
>>>>>>>>>>>>> just 8G malloc
>>>>>>>>>>>>> > >    which is like 50% of the total memory, how the process
>>>>>>>>>>>>> is eating up 89% of
>>>>>>>>>>>>> > >    memory and the memory is not releasing even after TTL.
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > > Any help please...
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > > Best Regards
>>>>>>>>>>>>> > > Sujith P V
>>>>>>>>>>>>> > >
>>>>>>>>>>>>> > > _______________________________________________
>>>>>>>>>>>>> > > varnish-misc mailing list
>>>>>>>>>>>>> > > varnish-misc at varnish-cache.org
>>>>>>>>>>>>> > > https://www.varnish-cache.org/
>>>>>>>>>>>>> lists/mailman/listinfo/varnish-misc
>>>>>>>>>>>>> > >
>>>>>>>>>>>>>
>>>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>>>> > varnish-misc mailing list
>>>>>>>>>>>>> > varnish-misc at varnish-cache.org
>>>>>>>>>>>>> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish
>>>>>>>>>>>>> -misc
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>
>>>>>>>>>>>>> Viktor Villafuerte
>>>>>>>>>>>>> Optus Internet Engineering
>>>>>>>>>>>>> t: +61 2 80825265
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20160914/53f54cc8/attachment-0001.html>