Varnish 3.0.5 RAM/SWAP usage

[geodni at free.fr]

Hi all,

I am working on a FreeBSD 10 amd64 system with Varnish 3.0.5 official port.
The hardware I use is :
- HP Proliant DL360 G8 
- one Intel(R) Xeon(R) CPU E5-2667 0 @ 2.90GHz
- 32GB of RAM
- 8GB of swap
- Smart Array P420i with 1GB not BBU
- 1 big RAID5 of 4x4TB Constellation ES.3 SAS2
- dedicated gpt freebsd-ufs partition of 3.6TB for Varnish mounted on /var/cache/varnish (no atime, no softs updates no journal at all)

Starting parameters for varnishd are default ones except storage :
-s file,/var/cache/varnish/file.bin,90%

I am stressing Varnish with big injection (trying to raise 200 millions of 12k average size objects from 51B to 250kB for a future production environment). It runs quite until all memory becomes full and then all swap is consumed, system simply runs out of swap. Performance is still convenient but varnishd child process is killed and a new one is started. Injection process full filled the RAM+SWAP 28+8 after 15718051 transactions in 1h15, it could be good but as the storage is not persistent, all the work done is unusable.

The sytem only runs FreeBSD (65MB used), nginx (35MB and no log) as the backend server for the tests and varnishncsa daemon (105MB).

I tried to find why all the RAM+SWAP are consumed so quickly on the net but i am at a lost. Documentation says 1kb overhead by objects, 1 million means 1GB RAM used. OK it stores "only" 15 millions of objects it should consume 15GB and several GB for SHM log and so on but not all the 13+8 (RAM+SWAP).

Can someone provide some welcome and good tips or misses I made ?

Best regards,
Denis

I can provide more detailed information if required, just let me know.

varnish> param.show
200        
acceptor_sleep_decay        0.900000 []
acceptor_sleep_incr         0.001000 [s]
acceptor_sleep_max          0.050000 [s]
auto_restart                on [bool]
ban_dups                    on [bool]
ban_lurker_sleep            0.010000 [s]
between_bytes_timeout       60.000000 [s]
cc_command                  "exec cc -std=gnu99 -O2 -pipe -fno-strict-aliasing -D_THREAD_SAFE -pthread -fpic -shared -Wl,-x -o %o %s"
cli_buffer                  8192 [bytes]
cli_timeout                 10 [seconds]
clock_skew                  10 [s]
connect_timeout             0.700000 [s]
critbit_cooloff             180.000000 [s]
default_grace               10.000000 [seconds]
default_keep                0.000000 [seconds]
default_ttl                 120.000000 [seconds]
diag_bitmap                 0x0 [bitmap]
esi_syntax                  0 [bitmap]
expiry_sleep                1.000000 [seconds]
fetch_chunksize             128 [kilobytes]
fetch_maxchunksize          262144 [kilobytes]
first_byte_timeout          60.000000 [s]
group                       www (80)
gzip_level                  6 []
gzip_memlevel               8 []
gzip_stack_buffer           32768 [Bytes]
gzip_tmp_space              0 []
gzip_window                 15 []
http_gzip_support           on [bool]
http_max_hdr                64 [header lines]
http_range_support          on [bool]
http_req_hdr_len            8192 [bytes]
http_req_size               32768 [bytes]
http_resp_hdr_len           8192 [bytes]
http_resp_size              32768 [bytes]
idle_send_timeout           60 [seconds]
listen_address              :8080
listen_depth                1024 [connections]
log_hashstring              on [bool]
log_local_address           off [bool]
lru_interval                2 [seconds]
max_esi_depth               5 [levels]
max_restarts                4 [restarts]
nuke_limit                  50 [allocations]
pcre_match_limit            10000 []
pcre_match_limit_recursion  10000 []
ping_interval               3 [seconds]
pipe_timeout                60 [seconds]
prefer_ipv6                 off [bool]
queue_max                   100 [%]
rush_exponent               3 [requests per request]
saintmode_threshold         10 [objects]
send_timeout                600 [seconds]
sendfile_threshold          unlimited [bytes]
sess_timeout                5 [seconds]
sess_workspace              65536 [bytes]
session_linger              50 [ms]
session_max                 100000 [sessions]
shm_reclen                  255 [bytes]
shm_workspace               8192 [bytes]
shortlived                  10.000000 [s]
syslog_cli_traffic          on [bool]
thread_pool_add_delay       2 [milliseconds]
thread_pool_add_threshold   2 [requests]
thread_pool_fail_delay      200 [milliseconds]
thread_pool_max             500 [threads]
thread_pool_min             5 [threads]
thread_pool_purge_delay     1000 [milliseconds]
thread_pool_stack           unlimited [bytes]
thread_pool_timeout         300 [seconds]
thread_pool_workspace       65536 [bytes]
thread_pools                2 [pools]
thread_stats_rate           10 [requests]
user                        www (80)
vcc_err_unref               on [bool]
vcl_dir                     /usr/local/etc/varnish
vcl_trace                   off [bool]
vmod_dir                    /usr/local/lib/varnish/vmods
waiter                      default (kqueue, poll)