Varnish crashes periodically with high disk read and write

Guillaume Quintard guillaume at varnish-software.com
Mon Sep 28 18:45:17 UTC 2020


Hi,

This post should help understand a couple of points:
https://info.varnish-software.com/blog/understanding-varnish-cache-memory-usage
On top of this, you are probably using Transient:
https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage


-- 
Guillaume Quintard


On Mon, Sep 28, 2020 at 1:40 AM Hamidreza Hosseini <hrhosseini at hotmail.com>
wrote:

>
> Hi
> I'm using varnish-6.0.5,
> 1.
> 3nodes of varnish crashes every day and I should restart it else the whole
> of my system doesnt work anymore
> I don't know the reason but my disks at that time shows that It has 50
> MB/s (400Mb/s) Read and 48 MB/s write but my outbound network shows 120Mb/s
> I put the link of Netdata picture at that time,
> http://uupload.ir/files/c60r_varnish.png
> How can I solve this problem?
>
> 2.
> My next question is that I have another varnish that I set malloc to 68Gb
> (the server has 86 Gb ram) but it will pass 68g till 86 and server crash,
> Why does it exceed from 68g through I have adjusted the accurate and
> permitted number?
>
> But really I'm confused and annoyed with first problem because it destroys
> my cluster
>
> ```
>
> Sep 27 13:46:19 varnish-16 varnishd[121910]: Error: Manager got SIGTERM
> Sep 27 13:46:19 varnish-16 varnishd[121910]: Manager got SIGTERM
> Sep 27 13:46:19 varnish-16 varnishd[121910]: Debug: Stopping Child
> Sep 27 13:46:19 varnish-16 varnishd[121910]: Stopping Child
> Sep 27 13:46:30 varnish-16 varnishd[121910]: Error: Child (121932) died
> signal=15
> Sep 27 13:46:30 varnish-16 varnishd[121910]: Child (121932) died signal=15
> Sep 27 13:46:30 varnish-16 varnishd[121910]: Debug: Child cleanup complete
> Sep 27 13:46:30 varnish-16 varnishd[121910]: Child cleanup complete
> Sep 27 13:46:30 varnish-16 varnishd[121910]: Info: manager stopping child
> Sep 27 13:46:30 varnish-16 varnishd[121910]: Info: manager dies
> Sep 27 13:46:30 varnish-16 varnishd[121910]: manager stopping child
> Sep 27 13:46:30 varnish-16 varnishd[121910]: manager dies
> Sep 27 13:44:46 varnish-16 kernel: lowmem_reserve[]: 0 2934 15972 15972
> 15972
> Sep 27 13:44:46 varnish-16 kernel: Node 0 DMA32 free:55712kB min:2968kB
> low:3708kB high:4452kB active_anon:0kB inactive_anon:2674740kB
> active_file:104kB inactive_file:0kB unevictable:15528kB isolated(ano
> Sep 27 13:44:47 varnish-16 kernel: lowmem_reserve[]: 0 0 13037 13037 13037
> Sep 27 13:44:47 varnish-16 kernel: Node 0 Normal free:21044kB min:13196kB
> low:16492kB high:19792kB active_anon:492kB inactive_anon:12208096kB
> active_file:548kB inactive_file:1300kB unevictable:70304kB is
> Sep 27 13:44:47 varnish-16 kernel: lowmem_reserve[]: 0 0 0 0 0
> Sep 27 13:44:47 varnish-16 kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB
> (U) 1*32kB (E) 3*64kB (UE) 2*128kB (UE) 2*256kB (UE) 0*512kB 1*1024kB (U)
> 2*2048kB (ME) 2*4096kB (M) = 14332kB
> Sep 27 13:44:47 varnish-16 kernel: Node 0 DMA32: 7633*4kB (UME) 2869*8kB
> (UMEH) 39*16kB (UM) 14*32kB (UM) 12*64kB (U) 2*128kB (U) 0*256kB 0*512kB
> 0*1024kB 0*2048kB 0*4096kB = 55580kB
> Sep 27 13:44:47 varnish-16 kernel: Node 0 Normal: 5132*4kB (UMEH) 69*8kB
> (UMH) 1*16kB (H) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB
> 0*4096kB = 21096kB
> Sep 27 13:44:47 varnish-16 kernel: Node 0 hugepages_total=0
> hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
> Sep 27 13:44:47 varnish-16 kernel: 3742574 total pagecache pages
> Sep 27 13:44:47 varnish-16 kernel: 3720854 pages in swap cache
> Sep 27 13:44:47 varnish-16 kernel: Swap cache stats: add 473619640, delete
> 469898786, find 188596514/233894822
> Sep 27 13:44:47 varnish-16 kernel: Free swap  = 216675600kB
> Sep 27 13:44:47 varnish-16 kernel: Total swap = 524287996kB
> Sep 27 13:44:47 varnish-16 kernel: 4194157 pages RAM
> Sep 27 13:44:47 varnish-16 kernel: 3720854 pages in swap cache
> Sep 27 13:44:47 varnish-16 kernel: Swap cache stats: add 473619640, delete
> 469898786, find 188596514/233894822
> Sep 27 13:44:47 varnish-16 kernel: Free swap  = 216675600kB
> Sep 27 13:44:47 varnish-16 kernel: Total swap = 524287996kB
> Sep 27 13:44:47 varnish-16 kernel: 4194157 pages RAM
> Sep 27 13:44:47 varnish-16 kernel: 0 pages HighMem/MovableOnly
> Sep 27 13:44:47 varnish-16 kernel: 90468 pages reserved
> Sep 27 13:44:47 varnish-16 kernel: 0 pages cma reserved
> Sep 27 13:44:47 varnish-16 kernel: 0 pages hwpoisoned
> Sep 27 13:44:47 varnish-16 kernel: [ pid ]   uid  tgid total_vm      rss
> nr_ptes nr_pmds swapents oom_score_adj name
> Sep 27 13:44:47 varnish-16 kernel: [  509]     0   509    10968      358
>    22       3      607             0 systemd-journal
> Sep 27 13:44:47 varnish-16 kernel: [  546]     0   546    23692      130
>    16       3       51             0 lvmetad
> Sep 27 13:44:47 varnish-16 kernel: [  557]     0   557    11194      307
>    22       3      327         -1000 systemd-udevd
> Sep 27 13:44:47 varnish-16 kernel: [  639]     0   639    28742      364
>    49       3      320             0 vmtoolsd
> Sep 27 13:44:47 varnish-16 kernel: [  703]   100   703    25080      235
>    19       3       85             0 systemd-timesyn
> Sep 27 13:44:47 varnish-16 kernel: [  969]     0   969     6510      350
>    18       3       59             0 atd
> Sep 27 13:44:47 varnish-16 kernel: [  983]   107   983    10721      208
>    28       3      119          -900 dbus-daemon
> Sep 27 13:44:47 varnish-16 kernel: [ 1010]     0  1010     6931      401
>    19       3       94             0 cron
> Sep 27 13:44:47 varnish-16 kernel: [ 1012]   104  1012    64097      156
>    28       3      324             0 rsyslogd
> Sep 27 13:44:47 varnish-16 kernel: [ 1013]     0  1013    21359      372
>    33       3      351             0 VGAuthService
> Sep 27 13:44:47 varnish-16 kernel: [ 1019]     0  1019    68649       76
>    37       3      211             0 accounts-daemon
> Sep 27 13:44:47 varnish-16 kernel: [ 1025]     0  1025     1098      288
>     8       3       35             0 acpid
> Sep 27 13:44:47 varnish-16 kernel: [ 1028]     0  1028     7164      366
>    18       3       86             0 systemd-logind
> Sep 27 13:44:47 varnish-16 kernel: [ 1038]     0  1038    40225      209
>    15       3      114             0 lxcfs
> Sep 27 13:44:47 varnish-16 kernel: [ 1100]     0  1100     3342      255
>    11       3       43             0 mdadm
> Sep 27 13:44:47 varnish-16 kernel: [ 1114]     0  1114    69271      223
>    40       3      180             0 polkitd
> Sep 27 13:44:47 varnish-16 kernel: [ 1224]     0  1224     1304      359
>     8       3       31             0 iscsid
> Sep 27 13:44:47 varnish-16 kernel: [ 1225]     0  1225     1429      879
>     8       3        0           -17 iscsid
> Sep 27 13:44:47 varnish-16 kernel: [ 1251]     0  1251    16377      348
>    36       3      190         -1000 sshd
> Sep 27 13:44:47 varnish-16 kernel: [ 1303]     0  1303     3663      312
>    13       3       39             0 agetty
> Sep 27 13:44:47 varnish-16 kernel: [ 1314]     0  1314     4903      314
>    15       3      125             0 irqbalance
> Sep 27 13:44:47 varnish-16 kernel: [ 1358]   999  1358    13541        0
>    29       2    11886          1000 netdata
> Sep 27 13:44:47 varnish-16 kernel: [121910]     0 121910     9067      422
>      19       3      113             0 varnishd
> Sep 27 13:44:47 varnish-16 kernel: [121932]   112 121932 86615131     7577
>  159392     334 76899838             0 cache-main
> Sep 27 13:44:47 varnish-16 kernel: [25236]   999 25236     1253        0
>     6       2      857          1000 apps.plugin
> Sep 27 13:44:47 varnish-16 kernel: [35166]   999 35166      406        0
>     5       2       91          1000 bash
> Sep 27 13:44:47 varnish-16 kernel: [44898]   999 44898    24372      337
>    35       3     3600          1000 python
> Sep 27 13:44:47 varnish-16 kernel: Out of memory: Kill process 1358
> (netdata) score 1000 or sacrifice child
> Sep 27 13:44:47 varnish-16 kernel: Killed process 25236 (apps.plugin)
> total-vm:5012kB, anon-rss:0kB, file-rss:0kB
>
>
> ```
>
> ```
>
> varnishd (varnish-6.0.5 revision 3065ccaacc4bb537fb976a524bd808db42c5fe40)
> Copyright (c) 2006 Verdens Gang AS
> Copyright (c) 2006-2019 Varnish Software AS
>
> ```
>
> Best regards.
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20200928/9aa53aca/attachment.html>


More information about the varnish-misc mailing list