Varnish only using some of reserved ram, IO-trashing

Audun Ytterdal ay at vg.no
Tue Feb 19 14:59:52 CET 2008


I've got 3 varnish servers with 32GB of RAM

Two of them run CentOS4.5 and one runs CentOS5

The problem is with the CentOS5 box

Varnish sees all of the 32 GB of ram, but only uses 3-4GB of it, while 
trashing the server with high IO-wait and load.

Varnish running as follows
/usr/sbin/varnishd -a :80 -f /etc/varnish/xxxx.vcl -T 127.0.0.1:82 -t 
120 -w 2,1000,30 -u varnish -g varnish -p thread_pools 1 -p 
client_http11 on -p rush_exponent 10 -s 
file,/var/varnish/varnish_storage.bin,30G -P /var/run/varnish.pid

Anyone care to take a look?

Working server (CentOS4.5)
==========================
cpu(s):  3.0% us,  1.0% sy,  0.0% ni, 95.3% id,  0.0% wa,  0.1% hi, 0.7%si
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
19139 varnish   16   0 32.1g  19g  18g S   18 61.5   1333:30 varnishd

uname -r; ulimit -a
2.6.9-55.0.2.ELsmp
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
pending signals                 (-i) 1024
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 268287
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

/usr/sbin/varnishd -a :80 -f /etc/varnish/nettby.vcl -T 127.0.0.1:82 -t 
120 -w 2,1000,30 -u varnish -g varnish -p thread_pools 1 -p 
client_http11 on -p rush_exponent 10 -s 
file,/var/varnish/varnish_storage.bin,30G -P /var/run/varnish.pid

# varnishstat -1
client_conn         162710315       332.27 Client connections accepted
client_req          415626452       848.75 Client requests received
cache_hit           410058375       837.38 Cache hits
cache_hitpass         3187366         6.51 Cache hits for pass
cache_miss            2377826         4.86 Cache misses
backend_conn          5565194        11.36 Backend connections success
backend_fail              558         0.00 Backend connections failures
backend_reuse               0         0.00 Backend connections reuses
backend_recycle             0         0.00 Backend connections recycles
backend_unused            991         0.00 Backend connections unused
n_srcaddr                5558          .   N struct srcaddr
n_srcaddr_act            1260          .   N active struct srcaddr
n_sess_mem               8097          .   N struct sess_mem
n_sess                   2968          .   N struct sess
n_object              1249250          .   N struct object
n_objecthead          1249804          .   N struct objecthead
n_smf                 2389608          .   N struct smf
n_smf_frag                 16          .   N small free smf
n_smf_large                 1          .   N large free smf
n_vbe_conn                  0          .   N struct vbe_conn
n_wrk                      65          .   N worker threads
n_wrk_create            57012         0.12 N worker threads created
n_wrk_failed                0         0.00 N worker threads not created
n_wrk_max               39549         0.08 N worker threads limited
n_wrk_queue                 0         0.00 N queued work requests
n_wrk_overflow          96561         0.20 N overflowed work requests
n_wrk_drop                  0         0.00 N dropped work requests
n_expired             1183055          .   N expired objects
n_lru_nuked                 0          .   N LRU nuked objects
n_lru_saved                 0          .   N LRU saved objects
n_deathrow                  0          .   N objects on deathrow
losthdr                   171         0.00 HTTP header overflows
n_objsendfile               0         0.00 Objects sent with sendfile
n_objwrite          280398080       572.60 Objects sent with write
s_sess              162710284       332.27 Total Sessions
s_req               415627912       848.75 Total Requests
s_pipe                      0         0.00 Total pipe
s_pass                      0         0.00 Total pass
s_fetch               5565187        11.36 Total fetch
s_hdrbytes       122863296770    250899.13 Total header bytes
s_bodybytes      578238928802   1180821.68 Total body bytes
sess_closed           3290353         6.72 Session Closed
sess_pipeline               0         0.00 Session Pipeline
sess_readahead        1571272         3.21 Session Read Ahead
sess_herd           413387877       844.18 Session herd
shm_records       17029260317     34775.45 SHM records
shm_writes          908226596      1854.69 SHM writes
shm_cont               504250         1.03 SHM MTX contention
sm_nreq               7997521        16.33 allocator requests
sm_nobj               2389591          .   outstanding allocations
sm_balloc         21814378496          .   bytes allocated
sm_bfree          10397876224          .   bytes free
backend_req           5565190        11.36 Backend requests made


Problem Server (CentOS5)
========================
Cpu(s):  2.3%us,  0.9%sy,  0.0%ni, 59.3%id, 36.6%wa,  0.3%hi,  0.7%si, 
0.0%st
PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
9353 varnish   15   0 33.0g 3.0g 2.8g S   15  9.6  16:47.07 varnishd

Notice 36% IO-wait. It's using the file instead of mapping memory

# uname -r; ulimit -a
2.6.18-53.1.13.el5
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
max nice                        (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 268287
max locked memory       (kbytes, -l) 32
max memory size         (kbytes, -m) unlimited
open files                      (-n) 327680
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
max rt priority                 (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 268287
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

# varnishstat -1
client_conn           4474131       327.99 Client connections accepted
client_req            9736045       713.73 Client requests received
cache_hit             9439400       691.99 Cache hits
cache_hitpass               0         0.00 Cache hits for pass
cache_miss             295479        21.66 Cache misses
backend_conn           295485        21.66 Backend connections success
backend_fail                0         0.00 Backend connections failures
backend_reuse               0         0.00 Backend connections reuses
backend_recycle             0         0.00 Backend connections recycles
backend_unused              0         0.00 Backend connections unused
n_srcaddr                5361          .   N struct srcaddr
n_srcaddr_act            1297          .   N active struct srcaddr
n_sess_mem               2909          .   N struct sess_mem
n_sess                   2356          .   N struct sess
n_object               292447          .   N struct object
n_objecthead           292486          .   N struct objecthead
n_smf                  579235          .   N struct smf
n_smf_frag                  1          .   N small free smf
n_smf_large                 1          .   N large free smf
n_vbe_conn                 21          .   N struct vbe_conn
n_bereq                   126          .   N struct bereq
n_wrk                     225          .   N worker threads
n_wrk_create             3298         0.24 N worker threads created
n_wrk_failed                0         0.00 N worker threads not created
n_wrk_max                   0         0.00 N worker threads limited
n_wrk_queue                 0         0.00 N queued work requests
n_wrk_overflow           3298         0.24 N overflowed work requests
n_wrk_drop                  0         0.00 N dropped work requests
n_backend                   1          .   N backends
n_expired                5976          .   N expired objects
n_lru_nuked                 0          .   N LRU nuked objects
n_lru_saved                 0          .   N LRU saved objects
n_lru_moved           7282571          .   N LRU moved objects
n_deathrow                  0          .   N objects on deathrow
losthdr                     0         0.00 HTTP header overflows
n_objsendfile               0         0.00 Objects sent with sendfile
n_objwrite            7526263       551.74 Objects sent with write
s_sess                4474103       327.99 Total Sessions
s_req                 9736204       713.75 Total Requests
s_pipe                      0         0.00 Total pipe
s_pass                      0         0.00 Total pass
s_fetch                295483        21.66 Total fetch
s_hdrbytes         3017925993    221239.35 Total header bytes
s_bodybytes       15705176320   1151321.48 Total body bytes
sess_closed            125264         9.18 Session Closed
sess_pipeline               0         0.00 Session Pipeline
sess_readahead          42404         3.11 Session Read Ahead
sess_herd             9622386       705.40 Session herd
shm_records         403217255     29559.22 SHM records
shm_writes           23235793      1703.38 SHM writes
shm_cont               114977         8.43 SHM MTX contention
sm_nreq                593953        43.54 allocator requests
sm_nobj                579233          .   outstanding allocations
sm_balloc          4619284480          .   bytes allocated
sm_bfree          27592970240          .   bytes free
backend_req            295485        21.66 Backend requests made


*****************************************************************
Denne fotnoten bekrefter at denne e-postmeldingen ble
skannet av MailSweeper og funnet fri for virus.
*****************************************************************
This footnote confirms that this email message has been 
swept by MailSweeper for the presence of computer viruses.
*****************************************************************




More information about the varnish-dev mailing list