Varnish caching issues, cache size, expires, cache resets, whats going on?

[Anders Nordby]

Hi,

Varnish will only have the most requested objects in RAM, so why is it a
problem to have a large cache?

I've been trying to run with large data sets for a long time, and have
had a lot of problems with it. But it has gotten better, due to
Poul-Hennings and also Dag-Erlings help. Did you check the Performance
page on the wiki (http://varnish.projects.linpro.no/wiki/Performance),
particularly the section about running with many objects? By lowering
obj_workspace, you can reduce how much space Varnish uses to fit HTTP
headers. You also may want to switch to run trunk instead of 1.1.2.

Cheers,
Anders.

On Tue, May 06, 2008 at 02:25:18PM -0700, Demitrious Kelly wrote:
> We have a varnish (varnish-1.1.2) server handling about 2,500 requests 
> per second.  The nature of these requests are extraordinarily random, 
> and the disks cannot keep up with the demand, we therefor must keep the 
> cache in RAM.  Under highest traffic conditions this generates about 3GB 
> per hour of cached data.  The expires on the data being generated are 
> set to a random number of seconds between 1800 and 3600.  So 
> theoretically the cache size should never reach more than 4gb or so, 
> well within the servers RAM capacity of 8Gb.  What we are seeing however 
> is that the cache will continue to grow to (and beyond if we allow it,) 
> the 8gb mark.  At which point we have to invalidate the cache by 
> restarting the processes (naturally this causes the undesired effect of 
> pegging the back-end servers with very high loads until enough data is 
> back in the cache).  We have tried limiting the cache size to 7gb, 
> however once the 7gb mark is reached the varnish cache restarts... with 
> the undesired effect of pegging the back-end servers with very high 
> loads until enough data is back in the cache.
> 
> One definite oddity in our particular usage patterns is that 30% or more 
> of our cached data is comprised of headers, and not content.  Currently, 
> for example, varnishstat tells me:
>     3424095028    743539.36    851553.10 Total header bytes
>     7830249685   1718569.76   1947338.89 Total body bytes
> 
> Does anybody have any insight into this behavior?  We would like to not 
> have to resort to multiple varnish servers with mickey mouse hacks like 
> staggering restarts to reduce the effect on the back end servers.
> 
> System details below:
> ---------------------------------------
> 
> root at foo.com:~# uname -a
>     Linux foo.com 2.6.18-6-amd64 #1 SMP Sun Feb 10 17:50:19 UTC 2008 
> x86_64 GNU/Linux
> 
> root at foo.com:~# free -m
>                  total       used       free     shared    buffers     
> cached
>     Mem:          7991       4708       3282          0         91       
> 2317
>     -/+ buffers/cache:       2300       5691
>     Swap:            0          0          0
> 
> root at foo.com:~# cat /proc/cpuinfo  | grep -E 'mhz|model name' -i
>     model name      : Dual-Core AMD Opteron(tm) Processor 2218
>     cpu MHz         : 2593.784
>     model name      : Dual-Core AMD Opteron(tm) Processor 2218
>     cpu MHz         : 2593.784
>     model name      : Dual-Core AMD Opteron(tm) Processor 2218
>     cpu MHz         : 2593.784
>     model name      : Dual-Core AMD Opteron(tm) Processor 2218
>     cpu MHz         : 2593.784
> 
> Start Command:
>     /usr/local/sbin/varnishd -a 0.0.0.0:80 -f /etc/varnish/varnish.vcl 
> -P /var/run/varnishd.pid -T 192.x.x.x:6969 -t 600 -w
> 
> /etc/varnish/varnish.vcl
>     backend default {
>       set backend.host = "foo.lan";
>       set backend.port = "http";
>     }
> 
>     sub vcl_recv {
>       if ( req.url ~ "^/(part1|part2)" && req.http.cookie )  {
>         lookup;
>       }
>     }
> 
>     sub vcl_fetch {
>       if (obj.ttl < 600s) {
>         set obj.ttl = 600s;
>       }
>       if (obj.status == 404) {
>         set obj.ttl = 10s;
>       }
>       if (obj.status == 500 || obj.status == 503) {
>         pass;
>       }
>       insert;
>     }
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/varnish-misc


-- 
Anders.