100% CPU IOwait

[Per Buer]

Hi,

On Tue, Nov 3, 2015 at 4:08 PM, Caires Vinicius <cairesvs at gmail.com> wrote:

> We had some problems with malloc with the same kind of aws instance and
> the -s malloc,5.8G(80% of the memory total). The only trace of the error
> was a cannot fork cannot allocate memory into syslog. We're probably
> missing some point, maybe the instance size ins't the right fit for us.
>


This sounds like your running out of virtual memory. Maybe you're running
without swap space?

Per.


>
>
> On Tue, Nov 3, 2015 at 10:56 AM Per Buer <perbu at varnish-software.com>
> wrote:
>
>> Hi,
>>
>>
>> On Tue, Nov 3, 2015 at 1:42 PM, Caires Vinicius <cairesvs at gmail.com>
>> wrote:
>>
>>> We've started to use Varnish 4 with Amazon Linux with EBS SSD of 40GB,
>>> memory of 7.5GB. We use the file storage with 20G allocated with ttl of 11
>>> minutes and grace of 5 hours, all the other configs are standard.
>>>
>> I would, on a general basis, recommend against using the file backend. It
>> will start to struggle with fragmentation relatively quickly and the
>> performance isn't all that great (lots of unnecessary synchronous reads).
>>
>>> Sometimes when we have a lot of request that result into cache miss we
>>> started to notice that our request latency grows and the iowait stays at
>>> 100%, something similar to this
>>> https://www.varnish-cache.org/lists/pipermail/varnish-misc/2008-April/01...
>>> <https://www.varnish-cache.org/lists/pipermail/varnish-misc/2008-April/016139.html>.
>>> And our threads reaches the maximum (1000).
>>>
>>> Do you guys have any idea why is that?
>>>
>> Yeah. New objects get assign a piece of memory, starts writing, triggers
>> pagefault, kernel takes over and reads/merges the underlying page, varnish
>> then overwrites that page which then gets written back to disk. This
>> naturally slows down delivery so Varnish spawns new threads.
>>
>> Try malloc. You should start with -s malloc,30G or there about - if you
>> have lots of small objects you might need to go a bit down to avoid
>> swapping.
>>
>> Not related: You should also move /var/lib/varnish onto tempfs. Linux
>> will do a lot of writing if the shared memory segment is visible on a
>> filesystem that is backed by a disk.
>> --
>> *Per Buer*
>> CTO | Varnish Software AS
>> Cell: +47 95839117
>> We Make Websites Fly!
>> www.varnish-software.com
>> <http://info.varnish-software.com/signature>
>>
>


-- 
*Per Buer*
CTO | Varnish Software AS
Cell: +47 95839117
We Make Websites Fly!
www.varnish-software.com
<http://info.varnish-software.com/signature>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20151103/b032609a/attachment-0001.html>