Varnish crashes periodically with high disk read and write
Hamidreza Hosseini
hrhosseini at hotmail.com
Mon Sep 28 08:39:26 UTC 2020
Hi
I'm using varnish-6.0.5,
1.
3nodes of varnish crashes every day and I should restart it else the whole of my system doesnt work anymore
I don't know the reason but my disks at that time shows that It has 50 MB/s (400Mb/s) Read and 48 MB/s write but my outbound network shows 120Mb/s
I put the link of Netdata picture at that time,
http://uupload.ir/files/c60r_varnish.png
How can I solve this problem?
2.
My next question is that I have another varnish that I set malloc to 68Gb (the server has 86 Gb ram) but it will pass 68g till 86 and server crash, Why does it exceed from 68g through I have adjusted the accurate and permitted number?
But really I'm confused and annoyed with first problem because it destroys my cluster
```
Sep 27 13:46:19 varnish-16 varnishd[121910]: Error: Manager got SIGTERM
Sep 27 13:46:19 varnish-16 varnishd[121910]: Manager got SIGTERM
Sep 27 13:46:19 varnish-16 varnishd[121910]: Debug: Stopping Child
Sep 27 13:46:19 varnish-16 varnishd[121910]: Stopping Child
Sep 27 13:46:30 varnish-16 varnishd[121910]: Error: Child (121932) died signal=15
Sep 27 13:46:30 varnish-16 varnishd[121910]: Child (121932) died signal=15
Sep 27 13:46:30 varnish-16 varnishd[121910]: Debug: Child cleanup complete
Sep 27 13:46:30 varnish-16 varnishd[121910]: Child cleanup complete
Sep 27 13:46:30 varnish-16 varnishd[121910]: Info: manager stopping child
Sep 27 13:46:30 varnish-16 varnishd[121910]: Info: manager dies
Sep 27 13:46:30 varnish-16 varnishd[121910]: manager stopping child
Sep 27 13:46:30 varnish-16 varnishd[121910]: manager dies
Sep 27 13:44:46 varnish-16 kernel: lowmem_reserve[]: 0 2934 15972 15972 15972
Sep 27 13:44:46 varnish-16 kernel: Node 0 DMA32 free:55712kB min:2968kB low:3708kB high:4452kB active_anon:0kB inactive_anon:2674740kB active_file:104kB inactive_file:0kB unevictable:15528kB isolated(ano
Sep 27 13:44:47 varnish-16 kernel: lowmem_reserve[]: 0 0 13037 13037 13037
Sep 27 13:44:47 varnish-16 kernel: Node 0 Normal free:21044kB min:13196kB low:16492kB high:19792kB active_anon:492kB inactive_anon:12208096kB active_file:548kB inactive_file:1300kB unevictable:70304kB is
Sep 27 13:44:47 varnish-16 kernel: lowmem_reserve[]: 0 0 0 0 0
Sep 27 13:44:47 varnish-16 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 1*32kB (E) 3*64kB (UE) 2*128kB (UE) 2*256kB (UE) 0*512kB 1*1024kB (U) 2*2048kB (ME) 2*4096kB (M) = 14332kB
Sep 27 13:44:47 varnish-16 kernel: Node 0 DMA32: 7633*4kB (UME) 2869*8kB (UMEH) 39*16kB (UM) 14*32kB (UM) 12*64kB (U) 2*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 55580kB
Sep 27 13:44:47 varnish-16 kernel: Node 0 Normal: 5132*4kB (UMEH) 69*8kB (UMH) 1*16kB (H) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 21096kB
Sep 27 13:44:47 varnish-16 kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Sep 27 13:44:47 varnish-16 kernel: 3742574 total pagecache pages
Sep 27 13:44:47 varnish-16 kernel: 3720854 pages in swap cache
Sep 27 13:44:47 varnish-16 kernel: Swap cache stats: add 473619640, delete 469898786, find 188596514/233894822
Sep 27 13:44:47 varnish-16 kernel: Free swap = 216675600kB
Sep 27 13:44:47 varnish-16 kernel: Total swap = 524287996kB
Sep 27 13:44:47 varnish-16 kernel: 4194157 pages RAM
Sep 27 13:44:47 varnish-16 kernel: 3720854 pages in swap cache
Sep 27 13:44:47 varnish-16 kernel: Swap cache stats: add 473619640, delete 469898786, find 188596514/233894822
Sep 27 13:44:47 varnish-16 kernel: Free swap = 216675600kB
Sep 27 13:44:47 varnish-16 kernel: Total swap = 524287996kB
Sep 27 13:44:47 varnish-16 kernel: 4194157 pages RAM
Sep 27 13:44:47 varnish-16 kernel: 0 pages HighMem/MovableOnly
Sep 27 13:44:47 varnish-16 kernel: 90468 pages reserved
Sep 27 13:44:47 varnish-16 kernel: 0 pages cma reserved
Sep 27 13:44:47 varnish-16 kernel: 0 pages hwpoisoned
Sep 27 13:44:47 varnish-16 kernel: [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Sep 27 13:44:47 varnish-16 kernel: [ 509] 0 509 10968 358 22 3 607 0 systemd-journal
Sep 27 13:44:47 varnish-16 kernel: [ 546] 0 546 23692 130 16 3 51 0 lvmetad
Sep 27 13:44:47 varnish-16 kernel: [ 557] 0 557 11194 307 22 3 327 -1000 systemd-udevd
Sep 27 13:44:47 varnish-16 kernel: [ 639] 0 639 28742 364 49 3 320 0 vmtoolsd
Sep 27 13:44:47 varnish-16 kernel: [ 703] 100 703 25080 235 19 3 85 0 systemd-timesyn
Sep 27 13:44:47 varnish-16 kernel: [ 969] 0 969 6510 350 18 3 59 0 atd
Sep 27 13:44:47 varnish-16 kernel: [ 983] 107 983 10721 208 28 3 119 -900 dbus-daemon
Sep 27 13:44:47 varnish-16 kernel: [ 1010] 0 1010 6931 401 19 3 94 0 cron
Sep 27 13:44:47 varnish-16 kernel: [ 1012] 104 1012 64097 156 28 3 324 0 rsyslogd
Sep 27 13:44:47 varnish-16 kernel: [ 1013] 0 1013 21359 372 33 3 351 0 VGAuthService
Sep 27 13:44:47 varnish-16 kernel: [ 1019] 0 1019 68649 76 37 3 211 0 accounts-daemon
Sep 27 13:44:47 varnish-16 kernel: [ 1025] 0 1025 1098 288 8 3 35 0 acpid
Sep 27 13:44:47 varnish-16 kernel: [ 1028] 0 1028 7164 366 18 3 86 0 systemd-logind
Sep 27 13:44:47 varnish-16 kernel: [ 1038] 0 1038 40225 209 15 3 114 0 lxcfs
Sep 27 13:44:47 varnish-16 kernel: [ 1100] 0 1100 3342 255 11 3 43 0 mdadm
Sep 27 13:44:47 varnish-16 kernel: [ 1114] 0 1114 69271 223 40 3 180 0 polkitd
Sep 27 13:44:47 varnish-16 kernel: [ 1224] 0 1224 1304 359 8 3 31 0 iscsid
Sep 27 13:44:47 varnish-16 kernel: [ 1225] 0 1225 1429 879 8 3 0 -17 iscsid
Sep 27 13:44:47 varnish-16 kernel: [ 1251] 0 1251 16377 348 36 3 190 -1000 sshd
Sep 27 13:44:47 varnish-16 kernel: [ 1303] 0 1303 3663 312 13 3 39 0 agetty
Sep 27 13:44:47 varnish-16 kernel: [ 1314] 0 1314 4903 314 15 3 125 0 irqbalance
Sep 27 13:44:47 varnish-16 kernel: [ 1358] 999 1358 13541 0 29 2 11886 1000 netdata
Sep 27 13:44:47 varnish-16 kernel: [121910] 0 121910 9067 422 19 3 113 0 varnishd
Sep 27 13:44:47 varnish-16 kernel: [121932] 112 121932 86615131 7577 159392 334 76899838 0 cache-main
Sep 27 13:44:47 varnish-16 kernel: [25236] 999 25236 1253 0 6 2 857 1000 apps.plugin
Sep 27 13:44:47 varnish-16 kernel: [35166] 999 35166 406 0 5 2 91 1000 bash
Sep 27 13:44:47 varnish-16 kernel: [44898] 999 44898 24372 337 35 3 3600 1000 python
Sep 27 13:44:47 varnish-16 kernel: Out of memory: Kill process 1358 (netdata) score 1000 or sacrifice child
Sep 27 13:44:47 varnish-16 kernel: Killed process 25236 (apps.plugin) total-vm:5012kB, anon-rss:0kB, file-rss:0kB
```
```
varnishd (varnish-6.0.5 revision 3065ccaacc4bb537fb976a524bd808db42c5fe40)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2019 Varnish Software AS
```
Best regards.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20200928/e80edefb/attachment-0001.html>
More information about the varnish-misc
mailing list