Child Process Killed

[Thiago Figueiro]

From: Tejaswi Nadahalli
> I am running my Python origin-server on the same machine. It seems like 
> the Python interpreter caused the OOM killer to kill Varnish. If that's 
> the case, is there anything I can do prevent this from happening?


I've been meaning to write-up a blog entry regarding the OOM killer in Linux (what a dumb idea) but in the mean time this should get you started.

The OOM Killer is there because Linux, by default in most distros, allocates more memory than available (swap+ram) on the assumption that applications will never need it (this is called overcommiting).  Mostly this is true but when it's not the oom_kill is called to free-up some memory so the kernel can keep its promise.  Usually it does a shit job (as you just noticed) and I hate it so much.

One way to solve this is to tweak oom_kill so it doesn't kill varnish processes.  It's a bit cumbersome because you need to do that based on the PID, which you only learn after the process has started, leaving room for some nifty race conditions.  Still, adding these to Varnish's init scripts should do what you need - look up online for details.

The other way is to disable memory overcommit.  Add to /etc/sysctl.conf:

# Disables memory overcommit
vm.overcommit_memory = 2
# Tweak to fool VM (read manual for setting above)
vm.overcommit_ratio = 100
# swap only if really needed
vm.swappiness = 10

and sudo /sbin/sysctl -e -p /etc/sysctl.conf

The problem with setting overcommit_memory to 2 is that the VM will not allocate more memory than you have available (the actual rule is a function of RAM, swap and overcommit_ratio, hence the tweak above).

This could be a problem for Varnish depending on the storage used.  The file storage will mmap the file, resulting in a VM size as large as the file.  If you don't have enough RAM the kernel will deny memory allocation and varnish will fail to start.  At this point you either buy more RAM or tweak your swap size to account for greedy processes (ie.: processes that allocate a lot of memory but never use it).


TL;DR: buy more memory; get rid of memory hungry scripts in your varnish box


Good luck.


______________________________________________________
    CONFIDENTIALITY NOTICE    
This electronic mail message, including any and/or all attachments, is for the sole use of the intended recipient(s), and may contain confidential and/or privileged information, pertaining to business conducted under the direction and supervision of the sending organization. All electronic mail messages, which may have been established as expressed views and/or opinions (stated either within the electronic mail message or any of its attachments), are left to the sole responsibility of that of the sender, and are not necessarily attributed to the sending organization. Unauthorized interception, review, use, disclosure or distribution of any such information contained within this electronic mail message and/or its attachment(s), is (are) strictly prohibited. If you are not the intended recipient, please contact the sender by replying to this electronic mail message, along with the destruction all copies of the original electronic mail message (along with any attachments).
______________________________________________________