[Varnish] #1374: High SWAP usage

[Varnish]

#1374: High SWAP usage
-------------------+----------------------
 Reporter:  jammy  |       Type:  defect
   Status:  new    |   Priority:  high
Milestone:         |  Component:  varnishd
  Version:  3.0.3  |   Severity:  critical
 Keywords:         |
-------------------+----------------------
 We are running varnish 3.0.3 in production. but recently, we are
 experiencing high SWAP usage recently. As you can see from the following
 information, we configured varnished to use 4G malloc storage, but not
 sure why it ate SWAP so heavily.

 I'm looking forward to hearing from the community.

 #1. uname -a
 Linux ip-10-36-1-238 2.6.32-358.14.1.el6.x86_64 #1 SMP Tue Jul 16 23:51:20
 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

 #2.
 /usr/sbin/varnishd -P /var/run/varnish.pid -a :80 -f
 /etc/varnish/yottaa.vcl -T 127.0.0.1:6082 -t 120 -w 500,1000,360 -p
 thread_pool_add_delay 2 -p listen_depth 65535 -p ban_lurker_sleep 1 -p
 sess_workspace 204800 -p thread_pool_workspace 307200 -p http_req_size
 102400 -p http_resp_size 102400 -p http_req_hdr_len 51200 -p
 http_resp_hdr_len 51200 -p http_max_hdr 500 -S /etc/varnish/secret -s
 malloc,4066M

 #3.  cat /proc/14464/status
 VmPeak: 19214032 kB
 VmSize: 19211984 kB
 VmLck: 0 kB
 VmHWM: 5540916 kB
 VmRSS: 4494520 kB
 VmData: 19055864 kB
 VmStk: 152 kB
 VmExe: 492 kB
 VmLib: 5820 kB
 VmPTE: 33332 kB
 VmSwap: '''3383040''' kB


 #4. varnishstat -1

 n_sess_mem 919 . N struct sess_mem
 n_sess 12 . N struct sess
 n_object 94417 . N struct object
 n_vampireobject 0 . N unresurrected objects
 n_objectcore 94622 . N struct objectcore
 n_objecthead 95253 . N struct objecthead
 n_waitinglist 720 . N struct waitinglist
 n_vbc 23 . N struct vbc
 n_wrk 1000 . N worker threads
 n_wrk_create 1002 0.00 N worker threads created

 n_expired 8899069 . N expired objects
 n_lru_nuked 227237 . N LRU nuked objects
 n_lru_moved 4688309 . N LRU moved objects
 losthdr 0 0.00 HTTP header overflows
 n_objsendfile 0 0.00 Objects sent with sendfile
 n_objwrite 5670271 2.97 Objects sent with write
 n_objoverflow 0 0.00 Objects overflowing workspace
 s_sess 7342009 3.84 Total Sessions
 s_req 14921754 7.81 Total Requests
 s_pipe 1057 0.00 Total pipe
 s_pass 446621 0.23 Total pass
 s_fetch 10027602 5.25 Total fetch
 s_hdrbytes 9425706241 4932.10 Total header bytes
 s_bodybytes 168048482086 87933.18 Total body bytes

 n_ban 15133 . N total active bans
 n_ban_gone 10716 . N total gone bans
 n_ban_add 21766 0.01 N new bans added
 n_ban_retire 6633 0.00 N old bans deleted
 n_ban_obj_test 3386265 1.77 N objects tested
 n_ban_re_test 422921545 221.30 N regexps tested against
 n_ban_dups 5514 0.00 N duplicate bans removed

 SMA.s0.c_req 1662338 0.87 Allocator requests
 SMA.s0.c_fail 33874757152 17725.33 Allocator failures
 SMA.s0.c_bytes 70233567950 36750.47 Bytes allocated
 SMA.s0.c_freed 66209262888 34644.71 Bytes freed
 SMA.s0.g_alloc 195376 . Allocations outstanding
 SMA.s0.g_bytes 4024305062 . Bytes outstanding
 SMA.s0.g_space 239204954 . Bytes available
 SMA.Transient.c_req 106362180 55.66 Allocator requests
 SMA.Transient.c_fail 0 0.00 Allocator failures
 SMA.Transient.c_bytes 12698410805901 6644580.25 Bytes allocated
 SMA.Transient.c_freed 12698409165965 6644579.39 Bytes freed
 SMA.Transient.g_alloc 1139 . Allocations outstanding
 SMA.Transient.g_bytes 1639936 . Bytes outstanding
 SMA.Transient.g_space 0 . Bytes available

-- 
Ticket URL: <https://www.varnish-cache.org/trac/ticket/1374>
Varnish <https://varnish-cache.org/>
The Varnish HTTP Accelerator