Child Process Killed

[Tejaswi Nadahalli]

Thanks for the detailed explanation of why the OOM Killer strikes. I have
dome some reading about it, and am tinkering with how to stop it from
killing varnishd.

What I am curious about is - how did the OOM killer get invoked at all? My
python process is fairly basic, and wouldn't have consumed any memory at
all. When varnish reaches it's malloc limit, I thought cached objects would
start getting Nuked. My LRU nuke counters were 0 through the process. So,
instead of nuking objects gracefully, I had a varnish-child-restart. This is
what I am worried about. If I can get nuking by reducing the overall memory
footprint by reducing the malloc limits, I will gladly do it.

Do you think that might help?

-T

On Wed, Mar 23, 2011 at 4:33 PM, Thiago Figueiro <TFigueiro at au.westfield.com
> wrote:

> From: Tejaswi Nadahalli
> > I am running my Python origin-server on the same machine. It seems like
> > the Python interpreter caused the OOM killer to kill Varnish. If that's
> > the case, is there anything I can do prevent this from happening?
>
>
> I've been meaning to write-up a blog entry regarding the OOM killer in
> Linux (what a dumb idea) but in the mean time this should get you started.
>
> The OOM Killer is there because Linux, by default in most distros,
> allocates more memory than available (swap+ram) on the assumption that
> applications will never need it (this is called overcommiting).  Mostly this
> is true but when it's not the oom_kill is called to free-up some memory so
> the kernel can keep its promise.  Usually it does a shit job (as you just
> noticed) and I hate it so much.
>
> One way to solve this is to tweak oom_kill so it doesn't kill varnish
> processes.  It's a bit cumbersome because you need to do that based on the
> PID, which you only learn after the process has started, leaving room for
> some nifty race conditions.  Still, adding these to Varnish's init scripts
> should do what you need - look up online for details.
>
> The other way is to disable memory overcommit.  Add to /etc/sysctl.conf:
>
> # Disables memory overcommit
> vm.overcommit_memory = 2
> # Tweak to fool VM (read manual for setting above)
> vm.overcommit_ratio = 100
> # swap only if really needed
> vm.swappiness = 10
>
> and sudo /sbin/sysctl -e -p /etc/sysctl.conf
>
> The problem with setting overcommit_memory to 2 is that the VM will not
> allocate more memory than you have available (the actual rule is a function
> of RAM, swap and overcommit_ratio, hence the tweak above).
>
> This could be a problem for Varnish depending on the storage used.  The
> file storage will mmap the file, resulting in a VM size as large as the
> file.  If you don't have enough RAM the kernel will deny memory allocation
> and varnish will fail to start.  At this point you either buy more RAM or
> tweak your swap size to account for greedy processes (ie.: processes that
> allocate a lot of memory but never use it).
>
>
> TL;DR: buy more memory; get rid of memory hungry scripts in your varnish
> box
>
>
> Good luck.
>
>
> ______________________________________________________
>    CONFIDENTIALITY NOTICE
> This electronic mail message, including any and/or all attachments, is for
> the sole use of the intended recipient(s), and may contain confidential
> and/or privileged information, pertaining to business conducted under the
> direction and supervision of the sending organization. All electronic mail
> messages, which may have been established as expressed views and/or opinions
> (stated either within the electronic mail message or any of its
> attachments), are left to the sole responsibility of that of the sender, and
> are not necessarily attributed to the sending organization. Unauthorized
> interception, review, use, disclosure or distribution of any such
> information contained within this electronic mail message and/or its
> attachment(s), is (are) strictly prohibited. If you are not the intended
> recipient, please contact the sender by replying to this electronic mail
> message, along with the destruction all copies of the original electronic
> mail message (along with any attachments).
> ______________________________________________________
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110323/39ee760e/attachment-0003.html>