Varnish only using some of reserved ram, IO-trashing
Audun Ytterdal
ay at vg.no
Tue Feb 19 14:59:52 CET 2008
I've got 3 varnish servers with 32GB of RAM
Two of them run CentOS4.5 and one runs CentOS5
The problem is with the CentOS5 box
Varnish sees all of the 32 GB of ram, but only uses 3-4GB of it, while
trashing the server with high IO-wait and load.
Varnish running as follows
/usr/sbin/varnishd -a :80 -f /etc/varnish/xxxx.vcl -T 127.0.0.1:82 -t
120 -w 2,1000,30 -u varnish -g varnish -p thread_pools 1 -p
client_http11 on -p rush_exponent 10 -s
file,/var/varnish/varnish_storage.bin,30G -P /var/run/varnish.pid
Anyone care to take a look?
Working server (CentOS4.5)
==========================
cpu(s): 3.0% us, 1.0% sy, 0.0% ni, 95.3% id, 0.0% wa, 0.1% hi, 0.7%si
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19139 varnish 16 0 32.1g 19g 18g S 18 61.5 1333:30 varnishd
uname -r; ulimit -a
2.6.9-55.0.2.ELsmp
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 1024
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 268287
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
/usr/sbin/varnishd -a :80 -f /etc/varnish/nettby.vcl -T 127.0.0.1:82 -t
120 -w 2,1000,30 -u varnish -g varnish -p thread_pools 1 -p
client_http11 on -p rush_exponent 10 -s
file,/var/varnish/varnish_storage.bin,30G -P /var/run/varnish.pid
# varnishstat -1
client_conn 162710315 332.27 Client connections accepted
client_req 415626452 848.75 Client requests received
cache_hit 410058375 837.38 Cache hits
cache_hitpass 3187366 6.51 Cache hits for pass
cache_miss 2377826 4.86 Cache misses
backend_conn 5565194 11.36 Backend connections success
backend_fail 558 0.00 Backend connections failures
backend_reuse 0 0.00 Backend connections reuses
backend_recycle 0 0.00 Backend connections recycles
backend_unused 991 0.00 Backend connections unused
n_srcaddr 5558 . N struct srcaddr
n_srcaddr_act 1260 . N active struct srcaddr
n_sess_mem 8097 . N struct sess_mem
n_sess 2968 . N struct sess
n_object 1249250 . N struct object
n_objecthead 1249804 . N struct objecthead
n_smf 2389608 . N struct smf
n_smf_frag 16 . N small free smf
n_smf_large 1 . N large free smf
n_vbe_conn 0 . N struct vbe_conn
n_wrk 65 . N worker threads
n_wrk_create 57012 0.12 N worker threads created
n_wrk_failed 0 0.00 N worker threads not created
n_wrk_max 39549 0.08 N worker threads limited
n_wrk_queue 0 0.00 N queued work requests
n_wrk_overflow 96561 0.20 N overflowed work requests
n_wrk_drop 0 0.00 N dropped work requests
n_expired 1183055 . N expired objects
n_lru_nuked 0 . N LRU nuked objects
n_lru_saved 0 . N LRU saved objects
n_deathrow 0 . N objects on deathrow
losthdr 171 0.00 HTTP header overflows
n_objsendfile 0 0.00 Objects sent with sendfile
n_objwrite 280398080 572.60 Objects sent with write
s_sess 162710284 332.27 Total Sessions
s_req 415627912 848.75 Total Requests
s_pipe 0 0.00 Total pipe
s_pass 0 0.00 Total pass
s_fetch 5565187 11.36 Total fetch
s_hdrbytes 122863296770 250899.13 Total header bytes
s_bodybytes 578238928802 1180821.68 Total body bytes
sess_closed 3290353 6.72 Session Closed
sess_pipeline 0 0.00 Session Pipeline
sess_readahead 1571272 3.21 Session Read Ahead
sess_herd 413387877 844.18 Session herd
shm_records 17029260317 34775.45 SHM records
shm_writes 908226596 1854.69 SHM writes
shm_cont 504250 1.03 SHM MTX contention
sm_nreq 7997521 16.33 allocator requests
sm_nobj 2389591 . outstanding allocations
sm_balloc 21814378496 . bytes allocated
sm_bfree 10397876224 . bytes free
backend_req 5565190 11.36 Backend requests made
Problem Server (CentOS5)
========================
Cpu(s): 2.3%us, 0.9%sy, 0.0%ni, 59.3%id, 36.6%wa, 0.3%hi, 0.7%si,
0.0%st
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9353 varnish 15 0 33.0g 3.0g 2.8g S 15 9.6 16:47.07 varnishd
Notice 36% IO-wait. It's using the file instead of mapping memory
# uname -r; ulimit -a
2.6.18-53.1.13.el5
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
max nice (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 268287
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 327680
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
max rt priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 268287
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
# varnishstat -1
client_conn 4474131 327.99 Client connections accepted
client_req 9736045 713.73 Client requests received
cache_hit 9439400 691.99 Cache hits
cache_hitpass 0 0.00 Cache hits for pass
cache_miss 295479 21.66 Cache misses
backend_conn 295485 21.66 Backend connections success
backend_fail 0 0.00 Backend connections failures
backend_reuse 0 0.00 Backend connections reuses
backend_recycle 0 0.00 Backend connections recycles
backend_unused 0 0.00 Backend connections unused
n_srcaddr 5361 . N struct srcaddr
n_srcaddr_act 1297 . N active struct srcaddr
n_sess_mem 2909 . N struct sess_mem
n_sess 2356 . N struct sess
n_object 292447 . N struct object
n_objecthead 292486 . N struct objecthead
n_smf 579235 . N struct smf
n_smf_frag 1 . N small free smf
n_smf_large 1 . N large free smf
n_vbe_conn 21 . N struct vbe_conn
n_bereq 126 . N struct bereq
n_wrk 225 . N worker threads
n_wrk_create 3298 0.24 N worker threads created
n_wrk_failed 0 0.00 N worker threads not created
n_wrk_max 0 0.00 N worker threads limited
n_wrk_queue 0 0.00 N queued work requests
n_wrk_overflow 3298 0.24 N overflowed work requests
n_wrk_drop 0 0.00 N dropped work requests
n_backend 1 . N backends
n_expired 5976 . N expired objects
n_lru_nuked 0 . N LRU nuked objects
n_lru_saved 0 . N LRU saved objects
n_lru_moved 7282571 . N LRU moved objects
n_deathrow 0 . N objects on deathrow
losthdr 0 0.00 HTTP header overflows
n_objsendfile 0 0.00 Objects sent with sendfile
n_objwrite 7526263 551.74 Objects sent with write
s_sess 4474103 327.99 Total Sessions
s_req 9736204 713.75 Total Requests
s_pipe 0 0.00 Total pipe
s_pass 0 0.00 Total pass
s_fetch 295483 21.66 Total fetch
s_hdrbytes 3017925993 221239.35 Total header bytes
s_bodybytes 15705176320 1151321.48 Total body bytes
sess_closed 125264 9.18 Session Closed
sess_pipeline 0 0.00 Session Pipeline
sess_readahead 42404 3.11 Session Read Ahead
sess_herd 9622386 705.40 Session herd
shm_records 403217255 29559.22 SHM records
shm_writes 23235793 1703.38 SHM writes
shm_cont 114977 8.43 SHM MTX contention
sm_nreq 593953 43.54 allocator requests
sm_nobj 579233 . outstanding allocations
sm_balloc 4619284480 . bytes allocated
sm_bfree 27592970240 . bytes free
backend_req 295485 21.66 Backend requests made
*****************************************************************
Denne fotnoten bekrefter at denne e-postmeldingen ble
skannet av MailSweeper og funnet fri for virus.
*****************************************************************
This footnote confirms that this email message has been
swept by MailSweeper for the presence of computer viruses.
*****************************************************************
More information about the varnish-dev
mailing list