efficient strategy to cache warm multiple varnish servers

Hettwer, Marian mhettwer at team.mobile.de
Mon Sep 12 13:44:28 CEST 2011

Hi there,

On 13.07.11 08:28, "Lukas Kahwe Smith" <mls at pooteeweet.org> wrote:

>If we ever have to restart our setup from scratch because of some big
>crash or because we are rolling out a new version of the site, I am
>wondering how to most efficiently warm all the varnish servers. Do I just
>warm one server and then rsync? Searching for "varnish cache warm" and
>didnt find anything very detailed.
>This link seems useful in the general context of cache warming scripts:

Since I know which url's I need to cache, I'm using a script based on the
example above.

Basically I'm running all the time against varnish and compare the Max-Age
header and the Age header.
If Max-Age - Age less than 600 seconds, my script fires a second request
against varnish with the preconfigured special X- header.
Therefor I always have a warm cache.

However, this only works since I can ask my application "hej, gimme a list
of url's we want to cache". And additionally I mess up my cache hit / miss
ratio since my script will very often generate cache hits.



cat $TEMP_FILE | while read _url; do
   for i in "$HEADER_1" "$HEADER_2"; do
        HEADER_CACHE=$(wget -O /dev/null -S --header "$i" --header
"Host:xxxxxx" "http://localhost/${_url}" 2>&1)
        MAX_AGE=$(echo "${HEADER_CACHE}"|awk -F\= '/max-age/{print $2}')
        AGE=$(echo "${HEADER_CACHE}"|awk -F\  '/Age/{print $2}')
        [ ! -z $DEBUG ] && echo $_url $MAX_AGE $AGE $AGE_DIFF
        [ $AGE -eq 0 ] || [ $AGE_DIFF -le $TIME_AFTER ] && \
                  [ ! -z $DEBUG ] && echo -e $RED "^ cache is being
updated" $END 
                  wget -O /dev/null -S --header "${MAGIC}" --header "$i"
--header "Host:xxxxx" "http://localhost/${_url}" 2>/dev/nulli

More information about the varnish-misc mailing list