<!DOCTYPE html PUBLIC '-//W3C//DTD HTML 4.01 Transitional//EN'>
<html><head><meta http-equiv="Content-Type" content="text/html;charset=utf-8">
<style>BODY{font:10pt Tahoma,Verdana,sans-serif} .MsoNormal{line-height:120%;margin:0}</style></head><body>
<div><font face="Tahoma, Verdana, sans-serif" size="2">Dear Varnish List,</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"><br></font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">Yesterday I upgraded memory of one of our Varnish servers (from 32GB to 96GB).</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">Everything was running perfectly fine (hitrate increased from 75% to 85%), until today when I tried to load new VCL using varnishadm:</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"><br></font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> Reloading varnish config: live_20130625_131055</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> VCL compiled.dlopen(./vcl.ZpQp7cNR.so): ./vcl.ZpQp7cNR.so: failed to map segment from shared object: Cannot allocate memory</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"><br></font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> Command failed with error code 106</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"><br></font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">Immediately after vcl.load failed, the following errors started popping up in syslog:</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"><br></font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> Jun 25 13:11:19 host /var/www/varnish[10425]: Child (10426) said <jemalloc>: (malloc) Error in munmap(): P</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> Jun 25 13:11:31 host /var/www/varnish[10425]: Child (10426) said <jemalloc>: (malloc) Error in munmap(): #001</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> Jun 25 13:11:31 host /var/www/varnish[10425]: Child (10426) said <jemalloc>: (malloc) Error in munmap(): #020���X#177</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> Jun 25 13:11:31 host /var/www/varnish[10425]: Child (10426) said <jemalloc>: (malloc) Error in munmap(): #020���X#177</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"><br></font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">Shortly after, Varnish crashed completely and lost the whole cache. Before Varnish crashed, there was 16GB memory free on the server.</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"><br></font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">I have no idea yet what caused the crash, so hopefully someone on the list can shed some light on it.</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"><br></font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">Varnish is started with the following settings:</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"><br></font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">#!/bin/sh</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">echo starting varnish daemon</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">ulimit -n 131072</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">ulimit -l unlimited</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">sysctl -p /path/to/sysctl.conf</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">/path/to/varnishd \</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> -s malloc,70G \</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> -a 0.0.0.0:80 \</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> -T 127.0.0.1:7777 \</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> -p thread_pools=2 \</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> -p thread_pool_add_delay=2 \</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> -p thread_pool_min=500 \</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> -p thread_pool_max=3000 \</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> -p session_linger=50 \</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> -p sess_workspace=65536 \</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> -p connect_timeout=1 \</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> -p lru_interval=10 \</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> -n /var/www/varnish \</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"> -f /path/to/server.vcl</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">echo done</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"><br></font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"><br></font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">varnishstat showed the following:</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"><br></font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">$ varnishstat -1 | grep -i trans</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">SMA.Transient.c_req 845311 10.32 Allocator requests</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">SMA.Transient.c_fail 104 0.00 Allocator failures</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">SMA.Transient.c_bytes 10835594005 132281.74 Bytes allocated</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">SMA.Transient.c_freed 10835063135 132275.26 Bytes freed</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">SMA.Transient.g_alloc 68 . Allocations outstanding</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">SMA.Transient.g_bytes 530870 . Bytes outstanding</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">SMA.Transient.g_space 0 . Bytes available</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"><br></font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">104 Allocator failures on this instance, while we normally never see Allocator failures.</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"><br></font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">Any ideas what caused this, or how I can fix it?</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"><br></font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">Kind regards,</font></div><div><font face="Tahoma, Verdana, sans-serif" size="2"><br></font></div><div><font face="Tahoma, Verdana, sans-serif" size="2">Enno van Amerongen</font></div><div style="font-family: Tahoma, Verdana, sans-serif; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: normal; line-height: normal;"><br></div></body></html>