Varnish malloc issue

Wed Sep 14 21:48:19 CEST 2016

Have you tried the packages from your distribution?

On Sep 14, 2016 20:09, "sujith pv" <sujithnss at gmail.com> wrote:

> After the full load of requests..g_space became minimal and g_bytes became
> like 8 Gb..I had assigned malloc as 8 gb
>
> On 14-Sep-2016 22:42, "Guillaume Quintard" <guillaume at varnish-software.com>
> wrote:
>
>> What about the other g_space/g_bytes?
>>
>> On Sep 14, 2016 18:33, "sujith pv" <sujithnss at gmail.com> wrote:
>>
>>> It was showing as 0 for Transient_g_bytes and 1Gb for g_space as I
>>> mentioned in my mail.
>>>
>>> On Wed, Sep 14, 2016 at 7:45 PM, Guillaume Quintard <
>>> guillaume at varnish-software.com> wrote:
>>>
>>>> please keep using the mailing list, some other people may have ideas.
>>>>
>>>> what's the output of "varnishstat -1 | grep g_bytes" ?
>>>>
>>>> --
>>>> Guillaume Quintard
>>>>
>>>> On Wed, Sep 14, 2016 at 3:50 PM, sujith pv <sujithnss at gmail.com> wrote:
>>>>
>>>>> Hi Guillaume
>>>>>
>>>>> I have tried capping Transient and I could see the respective values
>>>>> under SMA.transient.g_bytes/g_space. But still on load I didnt see this
>>>>> g_bytes being used and was showing 0 only.
>>>>>
>>>>> Best Regards
>>>>> Sujith P V
>>>>>
>>>>> On Tue, Sep 6, 2016 at 7:39 PM, Guillaume Quintard <
>>>>> guillaume at varnish-software.com> wrote:
>>>>>
>>>>>> Maybe, not sure. Unless you have a **good** reason to do so, use the
>>>>>> packages.
>>>>>>
>>>>>> --
>>>>>> Guillaume Quintard
>>>>>>
>>>>>> On Tue, Sep 6, 2016 at 3:19 PM, sujith pv <sujithnss at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks Guillaume. Actually in my end Varnish was compiled here and
>>>>>>> installed (I was not part of that initially :-)). Will this have any impact
>>>>>>> compared to as you have mentioned like try from a packaged version.
>>>>>>>
>>>>>>> On Tue, Sep 6, 2016 at 6:46 PM, Guillaume Quintard <
>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>
>>>>>>>> It's just a question of storage, shortlive objects got into
>>>>>>>> transient storage instead of the one(s) you defined in the command line.
>>>>>>>> But it's still a cache storage, based on malloc, that is by default
>>>>>>>> unlimited in size.
>>>>>>>>
>>>>>>>> The rationale is that for object with such short ttl+grace+keep, a
>>>>>>>> simple malloc is good enough.
>>>>>>>>
>>>>>>>> --
>>>>>>>> Guillaume Quintard
>>>>>>>>
>>>>>>>> On Tue, Sep 6, 2016 at 3:04 PM, sujith pv <sujithnss at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Actually I'm confused around the short lived objects. Is these
>>>>>>>>> objects are something like cached objects or who is creating these objects.
>>>>>>>>>
>>>>>>>>> On Tue, Sep 6, 2016 at 1:57 PM, Guillaume Quintard <
>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>
>>>>>>>>>> Sorry, I didn't get your question.
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>
>>>>>>>>>> On Tue, Sep 6, 2016 at 8:31 AM, sujith pv <sujithnss at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thanks Guillaume. May be Im asking a very basic question , but
>>>>>>>>>>> still how this short lived objects are created and I'm not giving any
>>>>>>>>>>> settings for the same as well.
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Sep 6, 2016 at 11:54 AM, Guillaume Quintard <
>>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> https://www.varnish-cache.org/docs/4.1/reference/varnishd.ht
>>>>>>>>>>>> ml#shortlived
>>>>>>>>>>>>
>>>>>>>>>>>> On Sep 6, 2016 08:07, "sujith pv" <sujithnss at gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks Guillaume. I will try the transient settings first and
>>>>>>>>>>>>> verify the same. Also could you please elaborate on the short lived objects.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best Regards
>>>>>>>>>>>>> Sujith P V
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Fri, Sep 2, 2016 at 11:24 AM, Guillaume Quintard <
>>>>>>>>>>>>> guillaume at varnish-software.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Viktor,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Have a look here https://www.varnish-cache
>>>>>>>>>>>>>> .org/docs/trunk/users-guide/storage-backends.html#transient-
>>>>>>>>>>>>>> storage
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Sujith, please try to use a packaged version from your
>>>>>>>>>>>>>> distribution.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> Guillaume Quintard
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Sep 2, 2016 at 1:18 AM, Viktor Villafuerte <
>>>>>>>>>>>>>> viktor.villafuerte at optusnet.com.au> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Guillaume,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Thu 01 Sep 2016 09:53:22, Guillaume Quintard wrote:
>>>>>>>>>>>>>>> > Is that virtual or real memory?
>>>>>>>>>>>>>>> > Try to cap your Transient storage to 1GB, you may be
>>>>>>>>>>>>>>> overloaded with short
>>>>>>>>>>>>>>> > lived objects.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Could you elaborate on this bit more? I've got very similar
>>>>>>>>>>>>>>> problem
>>>>>>>>>>>>>>> here. Eg: Total memory 384g
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> VIRT: 487g
>>>>>>>>>>>>>>> RES:  311g
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> free says: 46g free
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> but SWAP is 91.3% used :(
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> How can this be?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> thanks
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> v
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > On Sep 1, 2016 9:40 AM, "sujith pv" <sujithnss at gmail.com>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> >
>>>>>>>>>>>>>>> > > Hi All
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > > I had already mailed this query long back but this time
>>>>>>>>>>>>>>> putting in a
>>>>>>>>>>>>>>> > > different manner.
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >    - We are using Varnish 4.0 in our end. We have a
>>>>>>>>>>>>>>> machine with memory
>>>>>>>>>>>>>>> > >    of 16G with 8G being malloc for Varnish. We have a
>>>>>>>>>>>>>>> TTL for 2 hrs as well.
>>>>>>>>>>>>>>> > >    - During our peak traffic, when we see the total
>>>>>>>>>>>>>>> memory of the machine
>>>>>>>>>>>>>>> > >    reaching 90% and like varnishd process is taking some
>>>>>>>>>>>>>>> 89% .
>>>>>>>>>>>>>>> > >    - So I'm just confused even though we had allocated
>>>>>>>>>>>>>>> just 8G malloc
>>>>>>>>>>>>>>> > >    which is like 50% of the total memory, how the
>>>>>>>>>>>>>>> process is eating up 89% of
>>>>>>>>>>>>>>> > >    memory and the memory is not releasing even after TTL.
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > > Any help please...
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > > Best Regards
>>>>>>>>>>>>>>> > > Sujith P V
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>> > > _______________________________________________
>>>>>>>>>>>>>>> > > varnish-misc mailing list
>>>>>>>>>>>>>>> > > varnish-misc at varnish-cache.org
>>>>>>>>>>>>>>> > > https://www.varnish-cache.org/
>>>>>>>>>>>>>>> lists/mailman/listinfo/varnish-misc
>>>>>>>>>>>>>>> > >
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>>>>>> > varnish-misc mailing list
>>>>>>>>>>>>>>> > varnish-misc at varnish-cache.org
>>>>>>>>>>>>>>> > https://www.varnish-cache.org/
>>>>>>>>>>>>>>> lists/mailman/listinfo/varnish-misc
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Viktor Villafuerte
>>>>>>>>>>>>>>> Optus Internet Engineering
>>>>>>>>>>>>>>> t: +61 2 80825265
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20160914/09e1b909/attachment-0001.html>