Varnish malloc issue

Wed Sep 14 18:33:39 CEST 2016

It was showing as 0 for Transient_g_bytes and 1Gb for g_space as I
mentioned in my mail.

On Wed, Sep 14, 2016 at 7:45 PM, Guillaume Quintard <
guillaume at varnish-software.com> wrote:

> please keep using the mailing list, some other people may have ideas.
>
> what's the output of "varnishstat -1 | grep g_bytes" ?
>
> --
> Guillaume Quintard
>
> On Wed, Sep 14, 2016 at 3:50 PM, sujith pv <sujithnss at gmail.com> wrote:
>
>> Hi Guillaume
>>
>> I have tried capping Transient and I could see the respective values
>> under SMA.transient.g_bytes/g_space. But still on load I didnt see this
>> g_bytes being used and was showing 0 only.
>>
>> Best Regards
>> Sujith P V
>>
>> On Tue, Sep 6, 2016 at 7:39 PM, Guillaume Quintard <
>> guillaume at varnish-software.com> wrote:
>>
>>> Maybe, not sure. Unless you have a **good** reason to do so, use the
>>> packages.
>>>
>>> --
>>> Guillaume Quintard
>>>
>>> On Tue, Sep 6, 2016 at 3:19 PM, sujith pv <sujithnss at gmail.com> wrote:
>>>
>>>> Thanks Guillaume. Actually in my end Varnish was compiled here and
>>>> installed (I was not part of that initially :-)). Will this have any impact
>>>> compared to as you have mentioned like try from a packaged version.
>>>>
>>>> On Tue, Sep 6, 2016 at 6:46 PM, Guillaume Quintard <
>>>> guillaume at varnish-software.com> wrote:
>>>>
>>>>> It's just a question of storage, shortlive objects got into transient
>>>>> storage instead of the one(s) you defined in the command line. But it's
>>>>> still a cache storage, based on malloc, that is by default unlimited in
>>>>> size.
>>>>>
>>>>> The rationale is that for object with such short ttl+grace+keep, a
>>>>> simple malloc is good enough.
>>>>>
>>>>> --
>>>>> Guillaume Quintard
>>>>>
>>>>> On Tue, Sep 6, 2016 at 3:04 PM, sujith pv <sujithnss at gmail.com> wrote:
>>>>>
>>>>>> Actually I'm confused around the short lived objects. Is these
>>>>>> objects are something like cached objects or who is creating these objects.
>>>>>>
>>>>>> On Tue, Sep 6, 2016 at 1:57 PM, Guillaume Quintard <
>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>
>>>>>>> Sorry, I didn't get your question.
>>>>>>>
>>>>>>> --
>>>>>>> Guillaume Quintard
>>>>>>>
>>>>>>> On Tue, Sep 6, 2016 at 8:31 AM, sujith pv <sujithnss at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thanks Guillaume. May be Im asking a very basic question , but
>>>>>>>> still how this short lived objects are created and I'm not giving any
>>>>>>>> settings for the same as well.
>>>>>>>>
>>>>>>>> On Tue, Sep 6, 2016 at 11:54 AM, Guillaume Quintard <
>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>
>>>>>>>>> https://www.varnish-cache.org/docs/4.1/reference/varnishd.ht
>>>>>>>>> ml#shortlived
>>>>>>>>>
>>>>>>>>> On Sep 6, 2016 08:07, "sujith pv" <sujithnss at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks Guillaume. I will try the transient settings first and
>>>>>>>>>> verify the same. Also could you please elaborate on the short lived objects.
>>>>>>>>>>
>>>>>>>>>> Best Regards
>>>>>>>>>> Sujith P V
>>>>>>>>>>
>>>>>>>>>> On Fri, Sep 2, 2016 at 11:24 AM, Guillaume Quintard <
>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Viktor,
>>>>>>>>>>>
>>>>>>>>>>> Have a look here https://www.varnish-cache
>>>>>>>>>>> .org/docs/trunk/users-guide/storage-backends.html#transient-
>>>>>>>>>>> storage
>>>>>>>>>>>
>>>>>>>>>>> Sujith, please try to use a packaged version from your
>>>>>>>>>>> distribution.
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Sep 2, 2016 at 1:18 AM, Viktor Villafuerte <
>>>>>>>>>>> viktor.villafuerte at optusnet.com.au> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Guillaume,
>>>>>>>>>>>>
>>>>>>>>>>>> On Thu 01 Sep 2016 09:53:22, Guillaume Quintard wrote:
>>>>>>>>>>>> > Is that virtual or real memory?
>>>>>>>>>>>> > Try to cap your Transient storage to 1GB, you may be
>>>>>>>>>>>> overloaded with short
>>>>>>>>>>>> > lived objects.
>>>>>>>>>>>>
>>>>>>>>>>>> Could you elaborate on this bit more? I've got very similar
>>>>>>>>>>>> problem
>>>>>>>>>>>> here. Eg: Total memory 384g
>>>>>>>>>>>>
>>>>>>>>>>>> VIRT: 487g
>>>>>>>>>>>> RES:  311g
>>>>>>>>>>>>
>>>>>>>>>>>> free says: 46g free
>>>>>>>>>>>>
>>>>>>>>>>>> but SWAP is 91.3% used :(
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> How can this be?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> thanks
>>>>>>>>>>>>
>>>>>>>>>>>> v
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> >
>>>>>>>>>>>> > On Sep 1, 2016 9:40 AM, "sujith pv" <sujithnss at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>> >
>>>>>>>>>>>> > > Hi All
>>>>>>>>>>>> > >
>>>>>>>>>>>> > > I had already mailed this query long back but this time
>>>>>>>>>>>> putting in a
>>>>>>>>>>>> > > different manner.
>>>>>>>>>>>> > >
>>>>>>>>>>>> > >
>>>>>>>>>>>> > >    - We are using Varnish 4.0 in our end. We have a machine
>>>>>>>>>>>> with memory
>>>>>>>>>>>> > >    of 16G with 8G being malloc for Varnish. We have a TTL
>>>>>>>>>>>> for 2 hrs as well.
>>>>>>>>>>>> > >    - During our peak traffic, when we see the total memory
>>>>>>>>>>>> of the machine
>>>>>>>>>>>> > >    reaching 90% and like varnishd process is taking some
>>>>>>>>>>>> 89% .
>>>>>>>>>>>> > >    - So I'm just confused even though we had allocated just
>>>>>>>>>>>> 8G malloc
>>>>>>>>>>>> > >    which is like 50% of the total memory, how the process
>>>>>>>>>>>> is eating up 89% of
>>>>>>>>>>>> > >    memory and the memory is not releasing even after TTL.
>>>>>>>>>>>> > >
>>>>>>>>>>>> > >
>>>>>>>>>>>> > > Any help please...
>>>>>>>>>>>> > >
>>>>>>>>>>>> > > Best Regards
>>>>>>>>>>>> > > Sujith P V
>>>>>>>>>>>> > >
>>>>>>>>>>>> > > _______________________________________________
>>>>>>>>>>>> > > varnish-misc mailing list
>>>>>>>>>>>> > > varnish-misc at varnish-cache.org
>>>>>>>>>>>> > > https://www.varnish-cache.org/
>>>>>>>>>>>> lists/mailman/listinfo/varnish-misc
>>>>>>>>>>>> > >
>>>>>>>>>>>>
>>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>>> > varnish-misc mailing list
>>>>>>>>>>>> > varnish-misc at varnish-cache.org
>>>>>>>>>>>> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish
>>>>>>>>>>>> -misc
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Regards
>>>>>>>>>>>>
>>>>>>>>>>>> Viktor Villafuerte
>>>>>>>>>>>> Optus Internet Engineering
>>>>>>>>>>>> t: +61 2 80825265
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20160914/05089516/attachment.html>