Varnish malloc issue

Wed Sep 14 20:09:25 CEST 2016

After the full load of requests..g_space became minimal and g_bytes became
like 8 Gb..I had assigned malloc as 8 gb

On 14-Sep-2016 22:42, "Guillaume Quintard" <guillaume at varnish-software.com>
wrote:

> What about the other g_space/g_bytes?
>
> On Sep 14, 2016 18:33, "sujith pv" <sujithnss at gmail.com> wrote:
>
>> It was showing as 0 for Transient_g_bytes and 1Gb for g_space as I
>> mentioned in my mail.
>>
>> On Wed, Sep 14, 2016 at 7:45 PM, Guillaume Quintard <
>> guillaume at varnish-software.com> wrote:
>>
>>> please keep using the mailing list, some other people may have ideas.
>>>
>>> what's the output of "varnishstat -1 | grep g_bytes" ?
>>>
>>> --
>>> Guillaume Quintard
>>>
>>> On Wed, Sep 14, 2016 at 3:50 PM, sujith pv <sujithnss at gmail.com> wrote:
>>>
>>>> Hi Guillaume
>>>>
>>>> I have tried capping Transient and I could see the respective values
>>>> under SMA.transient.g_bytes/g_space. But still on load I didnt see this
>>>> g_bytes being used and was showing 0 only.
>>>>
>>>> Best Regards
>>>> Sujith P V
>>>>
>>>> On Tue, Sep 6, 2016 at 7:39 PM, Guillaume Quintard <
>>>> guillaume at varnish-software.com> wrote:
>>>>
>>>>> Maybe, not sure. Unless you have a **good** reason to do so, use the
>>>>> packages.
>>>>>
>>>>> --
>>>>> Guillaume Quintard
>>>>>
>>>>> On Tue, Sep 6, 2016 at 3:19 PM, sujith pv <sujithnss at gmail.com> wrote:
>>>>>
>>>>>> Thanks Guillaume. Actually in my end Varnish was compiled here and
>>>>>> installed (I was not part of that initially :-)). Will this have any impact
>>>>>> compared to as you have mentioned like try from a packaged version.
>>>>>>
>>>>>> On Tue, Sep 6, 2016 at 6:46 PM, Guillaume Quintard <
>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>
>>>>>>> It's just a question of storage, shortlive objects got into
>>>>>>> transient storage instead of the one(s) you defined in the command line.
>>>>>>> But it's still a cache storage, based on malloc, that is by default
>>>>>>> unlimited in size.
>>>>>>>
>>>>>>> The rationale is that for object with such short ttl+grace+keep, a
>>>>>>> simple malloc is good enough.
>>>>>>>
>>>>>>> --
>>>>>>> Guillaume Quintard
>>>>>>>
>>>>>>> On Tue, Sep 6, 2016 at 3:04 PM, sujith pv <sujithnss at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Actually I'm confused around the short lived objects. Is these
>>>>>>>> objects are something like cached objects or who is creating these objects.
>>>>>>>>
>>>>>>>> On Tue, Sep 6, 2016 at 1:57 PM, Guillaume Quintard <
>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>
>>>>>>>>> Sorry, I didn't get your question.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Guillaume Quintard
>>>>>>>>>
>>>>>>>>> On Tue, Sep 6, 2016 at 8:31 AM, sujith pv <sujithnss at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks Guillaume. May be Im asking a very basic question , but
>>>>>>>>>> still how this short lived objects are created and I'm not giving any
>>>>>>>>>> settings for the same as well.
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 6, 2016 at 11:54 AM, Guillaume Quintard <
>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> https://www.varnish-cache.org/docs/4.1/reference/varnishd.ht
>>>>>>>>>>> ml#shortlived
>>>>>>>>>>>
>>>>>>>>>>> On Sep 6, 2016 08:07, "sujith pv" <sujithnss at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thanks Guillaume. I will try the transient settings first and
>>>>>>>>>>>> verify the same. Also could you please elaborate on the short lived objects.
>>>>>>>>>>>>
>>>>>>>>>>>> Best Regards
>>>>>>>>>>>> Sujith P V
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Sep 2, 2016 at 11:24 AM, Guillaume Quintard <
>>>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Viktor,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Have a look here https://www.varnish-cache
>>>>>>>>>>>>> .org/docs/trunk/users-guide/storage-backends.html#transient-
>>>>>>>>>>>>> storage
>>>>>>>>>>>>>
>>>>>>>>>>>>> Sujith, please try to use a packaged version from your
>>>>>>>>>>>>> distribution.
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Sep 2, 2016 at 1:18 AM, Viktor Villafuerte <
>>>>>>>>>>>>> viktor.villafuerte at optusnet.com.au> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Guillaume,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Thu 01 Sep 2016 09:53:22, Guillaume Quintard wrote:
>>>>>>>>>>>>>> > Is that virtual or real memory?
>>>>>>>>>>>>>> > Try to cap your Transient storage to 1GB, you may be
>>>>>>>>>>>>>> overloaded with short
>>>>>>>>>>>>>> > lived objects.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Could you elaborate on this bit more? I've got very similar
>>>>>>>>>>>>>> problem
>>>>>>>>>>>>>> here. Eg: Total memory 384g
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> VIRT: 487g
>>>>>>>>>>>>>> RES:  311g
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> free says: 46g free
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> but SWAP is 91.3% used :(
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> How can this be?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> v
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > On Sep 1, 2016 9:40 AM, "sujith pv" <sujithnss at gmail.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> >
>>>>>>>>>>>>>> > > Hi All
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > > I had already mailed this query long back but this time
>>>>>>>>>>>>>> putting in a
>>>>>>>>>>>>>> > > different manner.
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >    - We are using Varnish 4.0 in our end. We have a
>>>>>>>>>>>>>> machine with memory
>>>>>>>>>>>>>> > >    of 16G with 8G being malloc for Varnish. We have a TTL
>>>>>>>>>>>>>> for 2 hrs as well.
>>>>>>>>>>>>>> > >    - During our peak traffic, when we see the total
>>>>>>>>>>>>>> memory of the machine
>>>>>>>>>>>>>> > >    reaching 90% and like varnishd process is taking some
>>>>>>>>>>>>>> 89% .
>>>>>>>>>>>>>> > >    - So I'm just confused even though we had allocated
>>>>>>>>>>>>>> just 8G malloc
>>>>>>>>>>>>>> > >    which is like 50% of the total memory, how the process
>>>>>>>>>>>>>> is eating up 89% of
>>>>>>>>>>>>>> > >    memory and the memory is not releasing even after TTL.
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > > Any help please...
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > > Best Regards
>>>>>>>>>>>>>> > > Sujith P V
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>> > > _______________________________________________
>>>>>>>>>>>>>> > > varnish-misc mailing list
>>>>>>>>>>>>>> > > varnish-misc at varnish-cache.org
>>>>>>>>>>>>>> > > https://www.varnish-cache.org/
>>>>>>>>>>>>>> lists/mailman/listinfo/varnish-misc
>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>>>>> > varnish-misc mailing list
>>>>>>>>>>>>>> > varnish-misc at varnish-cache.org
>>>>>>>>>>>>>> > https://www.varnish-cache.org/
>>>>>>>>>>>>>> lists/mailman/listinfo/varnish-misc
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Viktor Villafuerte
>>>>>>>>>>>>>> Optus Internet Engineering
>>>>>>>>>>>>>> t: +61 2 80825265
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20160914/4718e5a3/attachment.html>