efficient strategy to cache warm multiple varnish servers

[Hettwer, Marian]

Hi there,



On 13.07.11 08:28, "Lukas Kahwe Smith" <mls at pooteeweet.org> wrote:

>Hi,
>
>If we ever have to restart our setup from scratch because of some big
>crash or because we are rolling out a new version of the site, I am
>wondering how to most efficiently warm all the varnish servers. Do I just
>warm one server and then rsync? Searching for "varnish cache warm" and
>didnt find anything very detailed.
>
>This link seems useful in the general context of cache warming scripts:
>https://www.varnish-cache.org/trac/wiki/VCLExampleHashAlwaysMiss

Since I know which url's I need to cache, I'm using a script based on the
example above.

Basically I'm running all the time against varnish and compare the Max-Age
header and the Age header.
If Max-Age - Age less than 600 seconds, my script fires a second request
against varnish with the preconfigured special X- header.
Therefor I always have a warm cache.

However, this only works since I can ask my application "hej, gimme a list
of url's we want to cache". And additionally I mess up my cache hit / miss
ratio since my script will very often generate cache hits.

HTH,
Marian

Example:

cat $TEMP_FILE | while read _url; do
   for i in "$HEADER_1" "$HEADER_2"; do
        HEADER_CACHE=$(wget -O /dev/null -S --header "$i" --header
"Host:xxxxxx" "http://localhost/${_url}" 2>&1)
        MAX_AGE=$(echo "${HEADER_CACHE}"|awk -F\= '/max-age/{print $2}')
        AGE=$(echo "${HEADER_CACHE}"|awk -F\  '/Age/{print $2}')
        AGE_DIFF=$(($MAX_AGE-$AGE))
        [ ! -z $DEBUG ] && echo $_url $MAX_AGE $AGE $AGE_DIFF
        [ $AGE -eq 0 ] || [ $AGE_DIFF -le $TIME_AFTER ] && \
               (
                  [ ! -z $DEBUG ] && echo -e $RED "^ cache is being
updated" $END 
                  wget -O /dev/null -S --header "${MAGIC}" --header "$i"
--header "Host:xxxxx" "http://localhost/${_url}" 2>/dev/nulli
               )
   done
Done