<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><span id="OLK_SRC_BODY_SECTION" style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;"><div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt"><span style="font-weight:bold">From: </span> Per Buer <<a href="mailto:perbu@varnish-software.com">perbu@varnish-software.com</a>><br><br></div><blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;"><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><br><div><br></div><div>Wrt, Dridis point about threads might also not be that relevant as only 700-800 threads are being created. Session workspace is virtual until allocated so I don't think that is where your memory is disappearing. </div><div><br></div><div>24 thread pools is a bit odd (you only need two) but don't know of any leaks it might trigger. </div><div><br></div><div>I'd stay away from classic hashing as cribit is stable and you're the only one I know that uses it.</div><div><br></div><div> When is the varnishstat taken? Right before running out of memory? That would probably be useful.<br></div></div></div></div></div></blockquote></span><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;"><br></div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">There are 24 thread pools because there are 24 cores in the server and I’m sure someone read somewhere that a pool per was the way to go. Probably that was a long time ago in a galaxy far, far away….</div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;"><br></div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">What are the significant differences between classic and cribit hashing? </div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;"><br></div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">The system is currently fully using available physical memory and has burned through about 15% of the swap space. The varnishstat was taken after the physical memory was exhausted and with just a few Gigs of swap in use. Here is a current snapshot from top and varnishstat –1 if that helps. </div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;"><br></div><div><div><font face="Consolas">Mem: 24G Active, 2857M Inact, 3757M Wired, 433M Cache, 510M Free</font></div><div><font face="Consolas">ARC: 1715M Total, 555M MFU, 274M MRU, 1042K Anon, 216M Header, 670M Other</font></div><div><font face="Consolas">Swap: 32G Total, 5174M Used, 27G Free, 15% Inuse</font></div><div><font face="Consolas"><br></font></div><div><font face="Consolas"> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND</font></div><div><font face="Consolas">87309 www 803 20 0 49189M 23483M select 22 130.1H 5.27% varnishd</font></div></div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;"><br></div><div><div><font face="Consolas">client_conn 115492418 106.06 Client connections accepted</font></div><div><font face="Consolas">client_drop 0 0.00 Connection dropped, no sess/wrk</font></div><div><font face="Consolas">client_req 195339845 179.39 Client requests received</font></div><div><font face="Consolas">cache_hit 94748999 87.01 Cache hits</font></div><div><font face="Consolas">cache_hitpass 8249240 7.58 Cache hits for pass</font></div><div><font face="Consolas">cache_miss 79893031 73.37 Cache misses</font></div><div><font face="Consolas">backend_conn 39130304 35.93 Backend conn. success</font></div><div><font face="Consolas">backend_unhealthy 0 0.00 Backend conn. not attempted</font></div><div><font face="Consolas">backend_busy 132086 0.12 Backend conn. too many</font></div><div><font face="Consolas">backend_fail 0 0.00 Backend conn. failures</font></div><div><font face="Consolas">backend_reuse 98773816 90.71 Backend conn. reuses</font></div><div><font face="Consolas">backend_toolate 6237866 5.73 Backend conn. was closed</font></div><div><font face="Consolas">backend_recycle 105011602 96.44 Backend conn. recycles</font></div><div><font face="Consolas">backend_retry 11220 0.01 Backend conn. retry</font></div><div><font face="Consolas">fetch_head 577 0.00 Fetch head</font></div><div><font face="Consolas">fetch_length 48290509 44.35 Fetch with Length</font></div><div><font face="Consolas">fetch_chunked 89008238 81.74 Fetch chunked</font></div><div><font face="Consolas">fetch_eof 0 0.00 Fetch EOF</font></div><div><font face="Consolas">fetch_bad 0 0.00 Fetch had bad headers</font></div><div><font face="Consolas">fetch_close 509908 0.47 Fetch wanted close</font></div><div><font face="Consolas">fetch_oldhttp 0 0.00 Fetch pre HTTP/1.1 closed</font></div><div><font face="Consolas">fetch_zero 0 0.00 Fetch zero len</font></div><div><font face="Consolas">fetch_failed 45097 0.04 Fetch failed</font></div><div><font face="Consolas">fetch_1xx 0 0.00 Fetch no body (1xx)</font></div><div><font face="Consolas">fetch_204 0 0.00 Fetch no body (204)</font></div><div><font face="Consolas">fetch_304 68372 0.06 Fetch no body (304)</font></div><div><font face="Consolas">n_sess_mem 1150 . N struct sess_mem</font></div><div><font face="Consolas">n_sess 722 . N struct sess</font></div><div><font face="Consolas">n_object 942029 . N struct object</font></div><div><font face="Consolas">n_vampireobject 0 . N unresurrected objects</font></div><div><font face="Consolas">n_objectcore 942471 . N struct objectcore</font></div><div><font face="Consolas">n_objecthead 936744 . N struct objecthead</font></div><div><font face="Consolas">n_waitinglist 624 . N struct waitinglist</font></div><div><font face="Consolas">n_vbc 117 . N struct vbc</font></div><div><font face="Consolas">n_wrk 792 . N worker threads</font></div><div><font face="Consolas">n_wrk_create 792 0.00 N worker threads created</font></div><div><font face="Consolas">n_wrk_failed 0 0.00 N worker threads not created</font></div><div><font face="Consolas">n_wrk_max 0 0.00 N worker threads limited</font></div><div><font face="Consolas">n_wrk_lqueue 0 0.00 work request queue length</font></div><div><font face="Consolas">n_wrk_queued 0 0.00 N queued work requests</font></div><div><font face="Consolas">n_wrk_drop 0 0.00 N dropped work requests</font></div><div><font face="Consolas">n_backend 7 . N backends</font></div><div><font face="Consolas">n_expired 50061996 . N expired objects</font></div><div><font face="Consolas">n_lru_nuked 28851249 . N LRU nuked objects</font></div><div><font face="Consolas">n_lru_moved 56196515 . N LRU moved objects</font></div><div><font face="Consolas">losthdr 7 0.00 HTTP header overflows</font></div><div><font face="Consolas">n_objsendfile 0 0.00 Objects sent with sendfile</font></div><div><font face="Consolas">n_objwrite 139062543 127.71 Objects sent with write</font></div><div><font face="Consolas">n_objoverflow 0 0.00 Objects overflowing workspace</font></div><div><font face="Consolas">s_sess 115493404 106.06 Total Sessions</font></div><div><font face="Consolas">s_req 195339845 179.39 Total Requests</font></div><div><font face="Consolas">s_pipe 0 0.00 Total pipe</font></div><div><font face="Consolas">s_pass 58017202 53.28 Total pass</font></div><div><font face="Consolas">s_fetch 137832710 126.58 Total fetch</font></div><div><font face="Consolas">s_hdrbytes 73466348331 67467.23 Total header bytes</font></div><div><font face="Consolas">s_bodybytes 1475428907061 1354948.26 Total body bytes</font></div><div><font face="Consolas">sess_closed 11135893 10.23 Session Closed</font></div><div><font face="Consolas">sess_pipeline 8340 0.01 Session Pipeline</font></div><div><font face="Consolas">sess_readahead 3402 0.00 Session Read Ahead</font></div><div><font face="Consolas">sess_linger 186082340 170.89 Session Linger</font></div><div><font face="Consolas">sess_herd 193483825 177.68 Session herd</font></div><div><font face="Consolas">shm_records 17182889691 15779.77 SHM records</font></div><div><font face="Consolas">shm_writes 1058678573 972.23 SHM writes</font></div><div><font face="Consolas">shm_flushes 4719 0.00 SHM flushes due to overflow</font></div><div><font face="Consolas">shm_cont 335494 0.31 SHM MTX contention</font></div><div><font face="Consolas">shm_cycles 7346 0.01 SHM cycles through buffer</font></div><div><font face="Consolas">sms_nreq 405919 0.37 SMS allocator requests</font></div><div><font face="Consolas">sms_nobj 0 . SMS outstanding allocations</font></div><div><font face="Consolas">sms_nbytes 0 . SMS outstanding bytes</font></div><div><font face="Consolas">sms_balloc 1127850835 . SMS bytes allocated</font></div><div><font face="Consolas">sms_bfree 1127850835 . SMS bytes freed</font></div><div><font face="Consolas">backend_req 137878631 126.62 Backend requests made</font></div><div><font face="Consolas">n_vcl 1 0.00 N vcl total</font></div><div><font face="Consolas">n_vcl_avail 1 0.00 N vcl available</font></div><div><font face="Consolas">n_vcl_discard 0 0.00 N vcl discarded</font></div><div><font face="Consolas">n_ban 1 . N total active bans</font></div><div><font face="Consolas">n_ban_gone 1 . N total gone bans</font></div><div><font face="Consolas">n_ban_add 1 0.00 N new bans added</font></div><div><font face="Consolas">n_ban_retire 0 0.00 N old bans deleted</font></div><div><font face="Consolas">n_ban_obj_test 0 0.00 N objects tested</font></div><div><font face="Consolas">n_ban_re_test 0 0.00 N regexps tested against</font></div><div><font face="Consolas">n_ban_dups 0 0.00 N duplicate bans removed</font></div><div><font face="Consolas">hcb_nolock 0 0.00 HCB Lookups without lock</font></div><div><font face="Consolas">hcb_lock 0 0.00 HCB Lookups with lock</font></div><div><font face="Consolas">hcb_insert 0 0.00 HCB Inserts</font></div><div><font face="Consolas">esi_errors 0 0.00 ESI parse errors (unlock)</font></div><div><font face="Consolas">esi_warnings 0 0.00 ESI parse warnings (unlock)</font></div><div><font face="Consolas">accept_fail 547 0.00 Accept failures</font></div><div><font face="Consolas">client_drop_late 0 0.00 Connection dropped late</font></div><div><font face="Consolas">uptime 1088919 1.00 Client uptime</font></div><div><font face="Consolas">dir_dns_lookups 0 0.00 DNS director lookups</font></div><div><font face="Consolas">dir_dns_failed 0 0.00 DNS director failed lookups</font></div><div><font face="Consolas">dir_dns_hit 0 0.00 DNS director cached lookups hit</font></div><div><font face="Consolas">dir_dns_cache_full 0 0.00 DNS director full dnscache</font></div><div><font face="Consolas">vmods 0 . Loaded VMODs</font></div><div><font face="Consolas">n_gzip 92821054 85.24 Gzip operations</font></div><div><font face="Consolas">n_gunzip 98789399 90.72 Gunzip operations</font></div><div><font face="Consolas">sess_pipe_overflow 0 . Dropped sessions due to session pipe overflow</font></div><div><font face="Consolas">LCK.sms.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.sms.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.sms.locks 1217757 1.12 Lock Operations</font></div><div><font face="Consolas">LCK.sms.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.smp.creat 0 0.00 Created locks</font></div><div><font face="Consolas">LCK.smp.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.smp.locks 0 0.00 Lock Operations</font></div><div><font face="Consolas">LCK.smp.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.sma.creat 2 0.00 Created locks</font></div><div><font face="Consolas">LCK.sma.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.sma.locks 762080017 699.85 Lock Operations</font></div><div><font face="Consolas">LCK.sma.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.smf.creat 0 0.00 Created locks</font></div><div><font face="Consolas">LCK.smf.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.smf.locks 0 0.00 Lock Operations</font></div><div><font face="Consolas">LCK.smf.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.hsl.creat 0 0.00 Created locks</font></div><div><font face="Consolas">LCK.hsl.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.hsl.locks 0 0.00 Lock Operations</font></div><div><font face="Consolas">LCK.hsl.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.hcb.creat 0 0.00 Created locks</font></div><div><font face="Consolas">LCK.hcb.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.hcb.locks 0 0.00 Lock Operations</font></div><div><font face="Consolas">LCK.hcb.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.hcl.creat 250007 0.23 Created locks</font></div><div><font face="Consolas">LCK.hcl.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.hcl.locks 364838330 335.05 Lock Operations</font></div><div><font face="Consolas">LCK.hcl.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.vcl.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.vcl.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.vcl.locks 1404720 1.29 Lock Operations</font></div><div><font face="Consolas">LCK.vcl.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.stat.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.stat.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.stat.locks 115493914 106.06 Lock Operations</font></div><div><font face="Consolas">LCK.stat.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.sessmem.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.sessmem.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.sessmem.locks 115660082 106.22 Lock Operations</font></div><div><font face="Consolas">LCK.sessmem.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.wstat.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.wstat.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.wstat.locks 3815488 3.50 Lock Operations</font></div><div><font face="Consolas">LCK.wstat.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.herder.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.herder.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.herder.locks 1 0.00 Lock Operations</font></div><div><font face="Consolas">LCK.herder.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.wq.creat 24 0.00 Created locks</font></div><div><font face="Consolas">LCK.wq.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.wq.locks 436653782 401.00 Lock Operations</font></div><div><font face="Consolas">LCK.wq.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.objhdr.creat 69117038 63.47 Created locks</font></div><div><font face="Consolas">LCK.objhdr.destroy 68180433 62.61 Destroyed locks</font></div><div><font face="Consolas">LCK.objhdr.locks 605204037 555.78 Lock Operations</font></div><div><font face="Consolas">LCK.objhdr.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.exp.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.exp.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.exp.locks 159855422 146.80 Lock Operations</font></div><div><font face="Consolas">LCK.exp.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.lru.creat 2 0.00 Created locks</font></div><div><font face="Consolas">LCK.lru.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.lru.locks 108706511 99.83 Lock Operations</font></div><div><font face="Consolas">LCK.lru.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.cli.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.cli.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.cli.locks 544317 0.50 Lock Operations</font></div><div><font face="Consolas">LCK.cli.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.ban.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.ban.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.ban.locks 159911741 146.85 Lock Operations</font></div><div><font face="Consolas">LCK.ban.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.vbp.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.vbp.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.vbp.locks 826695 0.76 Lock Operations</font></div><div><font face="Consolas">LCK.vbp.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.vbe.creat 1 0.00 Created locks</font></div><div><font face="Consolas">LCK.vbe.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.vbe.locks 78260491 71.87 Lock Operations</font></div><div><font face="Consolas">LCK.vbe.colls 0 0.00 Collisions</font></div><div><font face="Consolas">LCK.backend.creat 7 0.00 Created locks</font></div><div><font face="Consolas">LCK.backend.destroy 0 0.00 Destroyed locks</font></div><div><font face="Consolas">LCK.backend.locks 327541647 300.80 Lock Operations</font></div><div><font face="Consolas">LCK.backend.colls 0 0.00 Collisions</font></div><div><font face="Consolas">SMA.s0.c_req 211189969 193.94 Allocator requests</font></div><div><font face="Consolas">SMA.s0.c_fail 28921596 26.56 Allocator failures</font></div><div><font face="Consolas">SMA.s0.c_bytes 9340561806222 8577829.76 Bytes allocated</font></div><div><font face="Consolas">SMA.s0.c_freed 9334120384877 8571914.33 Bytes freed</font></div><div><font face="Consolas">SMA.s0.g_alloc 1924886 . Allocations outstanding</font></div><div><font face="Consolas">SMA.s0.g_bytes 6441310878 . Bytes outstanding</font></div><div><font face="Consolas">SMA.s0.g_space 1140066 . Bytes available</font></div><div><font face="Consolas">SMA.Transient.c_req 122683830 112.67 Allocator requests</font></div><div><font face="Consolas">SMA.Transient.c_fail 0 0.00 Allocator failures</font></div><div><font face="Consolas">SMA.Transient.c_bytes 7588011522470 6968389.31 Bytes allocated</font></div><div><font face="Consolas">SMA.Transient.c_freed 7588004757463 6968383.10 Bytes freed</font></div><div><font face="Consolas">SMA.Transient.g_alloc 6530 . Allocations outstanding</font></div><div><font face="Consolas">SMA.Transient.g_bytes 6765007 . Bytes outstanding</font></div><div><font face="Consolas">SMA.Transient.g_space 0 . Bytes available</font></div></div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;"><br></div><span id="OLK_SRC_BODY_SECTION" style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;"><blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;"><div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;color:rgb(0,0,0);font-family:Calibri,sans-serif;font-size:14px"><span class=""><span style="font-size: 14px; font-family: Calibri, sans-serif;"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div>>As a hotfix I would cap Transient at, say, 2G. "varnishd -s Transient=malloc,2G". You can probably up your main storage quite a bit with this in place.</div></blockquote></div></div></div></span><div><br></div></span><div>I’ll give it a shot and let you know. It usually takes a couple days before filling up all the physical memory.</div></div></blockquote><div><br></div><div>Well, I'm somewhat doubtful it it will actually work. </div></div><div><br></div><div dir="ltr"><div style="color:rgb(136,136,136)"></div></div></div></div></div></blockquote></span><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;"><br></div><div style="color: rgb(0, 0, 0); font-family: Calibri, sans-serif; font-size: 14px;">I guess it is a good thing I hadn’t actually made the change then. Since this is a production system, I was going to wait until our maintenance period tonight.</div></body></html>