Compressed and uncompressed cached object handling

Daniel Rodriguez coolbomb at gmail.com
Tue Nov 17 11:31:52 CET 2009


Hi guys,

I'm having a problem with a varnish implementation that we are testing
to replace an ugly appliance. We were almost ready to place our server
in a more real  environment (some of our production sites), but I
found out that there is something not working properly with the
compression handling in my varnishd (varnish-2.0.5 - Debian)

Varnish its returning the first object cached no matter if i ask for a
clear object (no Accept-encoding specified) or a gzip/deflate object.
If the object cached is a gzip object it will return that no matter if
I later ask for a clear one later.

According to what I have seen in the documentation varnish should keep
both object versions (compressed and no-compressed) in the cache and
deliver the one that its asked by the client.

Step 1

I ask for a non-compressed object (no Accept-encoding specified). This
works "great"

GET -H "TE:" -sed "http://foo.bar/test/prueba.php"
200 OK
Cache-Control: max-age=20
Connection: close
Date: Mon, 16 Nov 2009 16:56:06 GMT
Via: 1.1 varnish
Age: 0
Server: Apache
Content-Length: 11013
Content-Type: text/html; charset=iso-8859-15
Last-Modified: Mon, 16 Nov 2009 16:56:06 GMT
Client-Date: Mon, 16 Nov 2009 16:56:06 GMT
Client-Peer: 10.10.10.10:80
Client-Response-Num: 1
X-Varnish: 1655545411

The request goes like this in the log:

  12 SessionOpen  c 10.20.20.20 57909 :80
    12 ReqStart     c 10.20.20.20 57909 1655545411
    12 RxRequest    c GET
    12 RxURL        c /test/prueba.php
    12 RxProtocol   c HTTP/1.1
    12 RxHeader     c Connection: TE, close
    12 RxHeader     c Host: foo.bar
    12 RxHeader     c TE:
    12 RxHeader     c User-Agent: lwp-request/5.827 libwww-perl/5.831
    12 VCL_call     c recv
    12 VCL_return   c lookup
    12 VCL_call     c hash
    12 VCL_return   c hash
    12 VCL_call     c miss
    12 VCL_return   c fetch
    14 BackendOpen  b default 10.10.10.10 33484 10.30.30.30 80
    12 Backend      c 14 default default
    14 TxRequest    b GET
    14 TxURL        b /test/prueba.php
    14 TxProtocol   b HTTP/1.1
    14 TxHeader     b Host: foo.bar
    14 TxHeader     b User-Agent: lwp-request/5.827 libwww-perl/5.831
    14 TxHeader     b X-Varnish: 1655545411
    14 TxHeader     b X-Forwarded-For: 10.20.20.20
     0 CLI          - Rd ping
     0 CLI          - Wr 0 200 PONG 1258390564 1.0
    14 RxProtocol   b HTTP/1.1
    14 RxStatus     b 200
    14 RxResponse   b OK
    14 RxHeader     b Date: Mon, 16 Nov 2009 16:56:01 GMT
    14 RxHeader     b Server: Apache
    14 RxHeader     b Cache-control: max-age=20
    14 RxHeader     b Last-Modified: Mon, 16 Nov 2009 16:56:06 GMT
    14 RxHeader     b Connection: close
    14 RxHeader     b Transfer-Encoding: chunked
    14 RxHeader     b Content-Type: text/html; charset=iso-8859-15
    12 ObjProtocol  c HTTP/1.1
    12 ObjStatus    c 200
    12 ObjResponse  c OK
    12 ObjHeader    c Date: Mon, 16 Nov 2009 16:56:01 GMT
    12 ObjHeader    c Server: Apache
    12 ObjHeader    c Cache-control: max-age=20
    12 ObjHeader    c Last-Modified: Mon, 16 Nov 2009 16:56:06 GMT
    12 ObjHeader    c Content-Type: text/html; charset=iso-8859-15
    14 BackendClose b default
    12 TTL          c 1655545411 RFC 20 1258390566 0 0 20 0
    12 VCL_call     c fetch
    12 VCL_return   c deliver
    12 Length       c 11013
    12 VCL_call     c deliver
    12 VCL_return   c deliver
    12 TxProtocol   c HTTP/1.1
    12 TxStatus     c 200
    12 TxResponse   c OK
    12 TxHeader     c Server: Apache
    12 TxHeader     c Cache-control: max-age=20
    12 TxHeader     c Last-Modified: Mon, 16 Nov 2009 16:56:06 GMT
    12 TxHeader     c Content-Type: text/html; charset=iso-8859-15
    12 TxHeader     c Content-Length: 11013
    12 TxHeader     c Date: Mon, 16 Nov 2009 16:56:06 GMT
    12 TxHeader     c X-Varnish: 1655545411
    12 TxHeader     c Age: 0
    12 TxHeader     c Via: 1.1 varnish
    12 TxHeader     c Connection: close
    12 ReqEnd       c 1655545411 1258390561.316438675
1258390566.327898026 0.000134945 5.010995150 0.000464201
    12 SessionClose c Connection: close
    12 StatSess     c 10.20.20.20 57909 5 1 1 0 0 1 282 11013

Step 2

Then the next request will go with a (Accept-encoding: gzip), and
returns me a clear object :(

GET -H "Accept-encoding: gzip" -H "TE:" -sed "http://foo.bar/test/prueba.php"
200 OK
Cache-Control: max-age=20
Connection: close
Date: Mon, 16 Nov 2009 16:56:09 GMT
Via: 1.1 varnish
Age: 3
Server: Apache
Content-Length: 11013
Content-Type: text/html; charset=iso-8859-15
Last-Modified: Mon, 16 Nov 2009 16:56:06 GMT
Client-Date: Mon, 16 Nov 2009 16:56:08 GMT
Client-Peer: 10.10.10.10:80
Client-Response-Num: 1
X-Varnish: 1655545412 1655545411

    12 SessionOpen  c 10.20.20.20 57910 :80
    12 ReqStart     c 10.20.20.20 57910 1655545412
    12 RxRequest    c GET
    12 RxURL        c /test/prueba.php
    12 RxProtocol   c HTTP/1.1
    12 RxHeader     c Connection: TE, close
    12 RxHeader     c Accept-Encoding: gzip
    12 RxHeader     c Host: foo.bar
    12 RxHeader     c TE:
    12 RxHeader     c User-Agent: lwp-request/5.827 libwww-perl/5.831
    12 VCL_call     c recv
    12 VCL_return   c lookup
    12 VCL_call     c hash
    12 VCL_return   c hash
    12 Hit          c 1655545411
    12 VCL_call     c hit
    12 VCL_return   c deliver
    12 Length       c 11013
    12 VCL_call     c deliver
    12 VCL_return   c deliver
    12 TxProtocol   c HTTP/1.1
    12 TxStatus     c 200
    12 TxResponse   c OK
    12 TxHeader     c Server: Apache
    12 TxHeader     c Cache-control: max-age=20
    12 TxHeader     c Last-Modified: Mon, 16 Nov 2009 16:56:06 GMT
    12 TxHeader     c Content-Type: text/html; charset=iso-8859-15
    12 TxHeader     c Content-Length: 11013
    12 TxHeader     c Date: Mon, 16 Nov 2009 16:56:09 GMT
    12 TxHeader     c X-Varnish: 1655545412 1655545411
    12 TxHeader     c Age: 3
    12 TxHeader     c Via: 1.1 varnish
    12 TxHeader     c Connection: close
    12 ReqEnd       c 1655545412 1258390569.036545277
1258390569.036923647 0.000121355 0.000098705 0.000279665
    12 SessionClose c Connection: close
    12 StatSess     c 10.20.20.20 57910 0 1 1 0 0 0 293 11013

My config:

backend default {
        .host = "10.30.30.30";
	.port = "80";
}

acl purge {
        "localhost";
}

sub vcl_recv {
   set req.grace = 2m;
   if (req.http.Accept-Encoding) {
      if (req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp3|ogg)$") {
         # No se comprimen estos, no tiene logica
            remove req.http.Accept-Encoding;
      } elsif (req.http.Accept-Encoding ~ "gzip") {
         set req.http.Accept-Encoding = "gzip";
      } elsif (req.http.Accept-Encoding ~ "deflate") {
         set req.http.Accept-Encoding = "deflate";
      } else {
         # No se conoce el algoritmo
        remove req.http.Accept-Encoding;
      }
   }
   if (req.request == "PURGE") {
      if (!client.ip ~ purge) {
         error 405 "Not allowed.";
      }
      purge("req.url == " req.url);
   }
   if (req.http.Authorization) {
       return (pass);
   }
   return(lookup);
}

sub vcl_fetch {
    set obj.grace = 2m;
    if(obj.http.Pragma ~ "no-cache" || obj.http.Cache-Control ~
"no-cache" || obj.http.Cache-Control ~ "private" ||
obj.http.Cache-Control ~ "max-age=0" || obj.http.Cache-Control ~
"must-revalidate" || obj.http.Cache-Control ~ "private" ) {
       pass;
    }
    if (!obj.cacheable) {
        return (pass);
    }
    if (obj.http.Set-Cookie) {
        return (pass);
    }
    set obj.prefetch =  -3s;
    if (req.http.Authorization && !obj.http.Cache-Control ~ "public") {
       pass;
    }
    return (deliver);
}

I'm doing something wrong?...

Best Regards,



More information about the varnish-misc mailing list