Varnish 2.06 running away with memory

Sam Crawford samcrawford at gmail.com
Tue May 18 10:47:42 CEST 2010


Morning all,

A bit of a newbie question I fear... We've been successfully running
Varnish 2.04 for over a year now on our intranet, and have found in
the past two days that our instances in one particular region are
eating up a lot of system memory. We've upgraded to 2.06 a couple of
days ago, and are still experiencing the same issue.

[root at webserver ~]# ps aux
USER       PID %CPU %MEM   VSZ  RSS TTY      STAT START   TIME COMMAND
....
root     32637  0.0  0.0 97268 1096 ?        Ss   05:13   0:00
/local/0/opt/varnish/sbin/varnishd -P /var/run/varnish.pid -a
10.131.100.36:8080 -T 127.0.0.1:8
082 -n internal -f /local/0/opt/varnish/etc/varnish/internal.vcl -s
file,/local/0/opt/varnish/cache/internal.cache,512M
nobody   32638 23.9 81.7 14641440 13418764 ? Sl   05:13 177:35
/local/0/opt/varnish/sbin/varnishd -P /var/run/varnish.pid -a
10.131.100.36:8080 -T 127.0.0.1:8
082 -n internal -f /local/0/opt/varnish/etc/varnish/internal.vcl -s
file,/local/0/opt/varnish/cache/internal.cache,512M

See above that PID 32638 is using 14GB of RAM. We only have a
relatively small number of items (in the order of 10's of thousands),
but they're hit very frequently. Therefore a cache size of 512M to 1GB
is sufficient for our needs.

Below is the output from varnishstat:

[root at webserver bin]# ./varnishstat -n internal -1
uptime                  45005          .   Child uptime
client_conn            411358         9.14 Client connections accepted
client_drop                 0         0.00 Connection dropped, no sess
client_req            1513928        33.64 Client requests received
cache_hit              222603         4.95 Cache hits
cache_hitpass           59894         1.33 Cache hits for pass
cache_miss              15566         0.35 Cache misses
backend_conn            39422         0.88 Backend conn. success
backend_unhealthy            0         0.00 Backend conn. not attempted
backend_busy                0         0.00 Backend conn. too many
backend_fail                0         0.00 Backend conn. failures
backend_reuse         1251666        27.81 Backend conn. reuses
backend_toolate         25078         0.56 Backend conn. was closed
backend_recycle       1276976        28.37 Backend conn. recycles
backend_unused              0         0.00 Backend conn. unused
fetch_head                354         0.01 Fetch head
fetch_length           594722        13.21 Fetch with Length
fetch_chunked          631083        14.02 Fetch chunked
fetch_eof                   0         0.00 Fetch EOF
fetch_bad                   0         0.00 Fetch had bad headers
fetch_close               395         0.01 Fetch wanted close
fetch_oldhttp               0         0.00 Fetch pre HTTP/1.1 closed
fetch_zero              64637         1.44 Fetch zero len
fetch_failed                0         0.00 Fetch failed
n_srcaddr                   0          .   N struct srcaddr
n_srcaddr_act               0          .   N active struct srcaddr
n_sess_mem                447          .   N struct sess_mem
n_sess                     94          .   N struct sess
n_object                 1460          .   N struct object
n_objecthead             2370          .   N struct objecthead
n_smf                    3034          .   N struct smf
n_smf_frag                144          .   N small free smf
n_smf_large                13          .   N large free smf
n_vbe_conn                 11          .   N struct vbe_conn
n_bereq                   299          .   N struct bereq
n_wrk                      43          .   N worker threads
n_wrk_create              975         0.02 N worker threads created
n_wrk_failed                0         0.00 N worker threads not created
n_wrk_max                   0         0.00 N worker threads limited
n_wrk_queue                 0         0.00 N queued work requests
n_wrk_overflow           2857         0.06 N overflowed work requests
n_wrk_drop                  0         0.00 N dropped work requests
n_backend                   1          .   N backends
n_expired               14134          .   N expired objects
n_lru_nuked                 0          .   N LRU nuked objects
n_lru_saved                 0          .   N LRU saved objects
n_lru_moved            184533          .   N LRU moved objects
n_deathrow                  0          .   N objects on deathrow
losthdr                     0         0.00 HTTP header overflows
n_objsendfile               0         0.00 Objects sent with sendfile
n_objwrite            1262710        28.06 Objects sent with write
n_objoverflow               0         0.00 Objects overflowing workspace
s_sess                 411349         9.14 Total Sessions
s_req                 1513941        33.64 Total Requests
s_pipe                      0         0.00 Total pipe
s_pass                1275725        28.35 Total pass
s_fetch               1291202        28.69 Total fetch
s_hdrbytes          559253905     12426.48 Total header bytes
s_bodybytes        6015944416    133672.80 Total body bytes
sess_closed             58624         1.30 Session Closed
sess_pipeline               0         0.00 Session Pipeline
sess_readahead              0         0.00 Session Read Ahead
sess_linger           1483437        32.96 Session Linger
sess_herd             1207272        26.83 Session herd
shm_records         106754812      2372.07 SHM records
shm_writes            5312192       118.04 SHM writes
shm_flushes               240         0.01 SHM flushes due to overflow
shm_cont                 2364         0.05 SHM MTX contention
shm_cycles                 50         0.00 SHM cycles through buffer
sm_nreq               2519582        55.98 allocator requests
sm_nobj                  2877          .   outstanding allocations
sm_balloc            24608768          .   bytes allocated
sm_bfree            512262144          .   bytes free
sma_nreq                    0         0.00 SMA allocator requests
sma_nobj                    0          .   SMA outstanding allocations
sma_nbytes                  0          .   SMA outstanding bytes
sma_balloc                  0          .   SMA bytes allocated
sma_bfree                   0          .   SMA bytes free
sms_nreq                   89         0.00 SMS allocator requests
sms_nobj                    0          .   SMS outstanding allocations
sms_nbytes                  0          .   SMS outstanding bytes
sms_balloc              43343          .   SMS bytes allocated
sms_bfree               43343          .   SMS bytes freed
backend_req           1291192        28.69 Backend requests made
n_vcl                       1         0.00 N vcl total
n_vcl_avail                 1         0.00 N vcl available
n_vcl_discard               0         0.00 N vcl discarded
n_purge                246879          .   N total active purges
n_purge_add            252285         5.61 N new purges added
n_purge_retire           5406         0.12 N old purges deleted
n_purge_obj_test       245731         5.46 N objects tested
n_purge_re_test     214813964      4773.11 N regexps tested against
n_purge_dups             3227         0.07 N duplicate purges removed
hcb_nolock                  0         0.00 HCB Lookups without lock
hcb_lock                    0         0.00 HCB Lookups with lock
hcb_insert                  0         0.00 HCB Inserts
esi_parse                   0         0.00 Objects ESI parsed (unlock)
esi_errors                  0         0.00 ESI parse errors (unlock)

Are we just missing a simple parameter to varnishd, or is there
something else here?

Thanks,

Sam




More information about the varnish-misc mailing list