file vs malloc

[Per Buer]

On Fri, Nov 26, 2010 at 4:41 PM, Angelo Höngens <A.Hongens at netmatch.nl>wrote:

> Per wrote in his survey result: "62% use -s file, which I find somewhat
> surprising. -s file doesn't really perform that well under pressure on
> rotating hard drives due to some changes in recent Linux and FreeBSD
> kernels."
>
> But does anyone have a link to (or write up) some explanation for dummies
> on cache file types? I know there's 'file' and 'malloc', but I don't know
> the difference or the pros and cons.
>

In Varnish 1.0 we had -s file and everything was fine. People where using
Varnish on  Linux 2.6.9 (and thereabout) and FreeBSD6. Then came 2.6.18 with
RHEL5 and some other Linux distros and Varnish started behaving a bit
strange. The IO pressure had increased and we saw load skyrocket with
Varnish being stuck IO sleep a lot of the time, waiting for writes to
finish. Then FreeBSD7 came out and showed the same behavior. Obviously the
kernel developers of the world where ganging up on us.

Alterations in vm behavior are difficult things to track down and the
chances of changing the behavior back might be slim - Varnish being
something of an oddball doing crazy stuff like mmaping several gigabytes and
accessing it in a random fasion (and most of them saying things like "Wow,
that actually works?" when hearing what we do).

Anyway, Poul came up with -s malloc, which just allocates slabs of ordinary
memory and stores everything there. Since ordinary memory isn't backed by
disk a write to memory doesn't trigger a write to disk so there are no sync
disk writes to slow us down. On the flip side - at least on Linux, if your
data set doesn't dit in memory - the moment you start to actively use your
swapping space things grind to a halt. FreeBSD seems to cope better with
programs being larger then the amount of memory.

Several people are running high traffic sites with -s file backed by SSD.
That seem to perform very well on Linux. Our main FreeBSD user, Anders N,
uses -s malloc so I don't know much about how -s file on FreeBSD anymore.

So, to sum up:
 - If you have SSD drives you can mostly do what you want. I don't know of
anyone running huge varnish process that swap alot - but it _might_ work. If
you are, let me know.
 - If your varnish process doesn't write that much you're probably in the
clear no matter what you'll do.
 - If you have a small data set that fits neatly in memory, use -s malloc.
 - If you you have a large data set that doesn't fit in memory you have to
get SSD drives.

Oh, and at least on Linux there are a bunch of tunables that might alter or
even rectify some of the misbehavior we've seen.


I should clean this up, get some tests done and write it up somewhere.
-- 
Per Buer,  Varnish Software
Phone: +47 21 98 92 61 / Mobile: +47 958 39 117 / skype: per.buer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20101126/283de4f7/attachment-0003.html>