Conditional requests for cached 404 responses

[Guillaume Quintard]

Hi Mark,

You are correct:
https://github.com/varnishcache/varnish-cache/blob/varnish-7.3.0/bin/varnishd/cache/cache_fetch.c#L699-L703

We only set the OF_IMSCAND flag (that we use to say that we can conditional
download) if:
- the object is not a Hit-For-Miss (HFM)
- if the status is 200
- we either have a convincing Last-modified, or an Etag header

You can also test it with this VTC:
varnishtest "conditional requests"

server s1 {
        rxreq
        txresp -status 200 -hdr "ETag: 1234" -hdr "Last-Modified: Wed, 21
Oct 2015 07:28:00 GMT" -body "dad"

        rxreq
        expect req.http.if-none-match == "1234"
        expect req.http.if-modified-since == "Wed, 21 Oct 2015 07:28:00 GMT"
        txresp
} -start

varnish v1 -vcl+backend {
        sub vcl_backend_response {
                set beresp.ttl = 0.1s;
                set beresp.grace = 0s;
                set beresp.keep = 1y;
                return (deliver);
        }
} -start

client c1 {
        txreq
        rxresp

        delay 0.2

        txreq
        rxresp
} -run

Change the 200 to a 404 and the test will now fail.

I quickly skimmed the HTTP spec and see no reason for us to actually check
the status, but I'm sure somebody closer to the code will pop up to provide
some light on the topic.

Cheers,

-- 
Guillaume Quintard


On Fri, Jul 14, 2023 at 7:30 AM Mark Slater <mark.slater at mail.com> wrote:

> Hi,
>
> I'm running Varnish in front of a back end that has to do some work to
> determine whether a request should receive a 404 response.  However, it can
> cheaply determine whether a previous 404 is still valid.
>
> I see Varnish issuing conditional requests for cached 200 responses, but I
> haven't managed to achieve the same for cached 404 responses.
>
> Here's my sample VCL:
>
>
> vcl 4.1;
>
> backend default {
>     .host = "localhost";
>     .port = "8081";
> }
>
> sub vcl_backend_response {
>      set beresp.keep = 5m;
> }
>
>
> I'm testing with canned responses on port 8081  For the working 200 case,
> I return:
>
>
> HTTP/1.1 200 OK
> cache-control: max-age=5
> etag: "foo"
> content-length: 13
> connection: close
>
> Hello, World!
>
>
> When I make requests to Varnish, I see, as expected, a first request to
> the back end, followed by five seconds of nothing to the back end, because
> Varnish is responding with its cached copy, followed by a conditional
> request to the back end:
>
>
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.68.0
> Accept: */*
> X-Forwarded-For: 127.0.0.1
> Accept-Encoding: gzip
> X-Varnish: 3
>
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.68.0
> Accept: */*
> X-Forwarded-For: 127.0.0.1
> Accept-Encoding: gzip
> If-None-Match: "foo"
> X-Varnish: 32773
>
>
> For the failing 404 case, my canned back end responds:
>
>
> HTTP/1.1 404 Not Found
> cache-control: max-age=5
> etag: "foo"
> content-length: 13
> connection: close
>
> Hello, World!
>
>
> Now when I make requests to Varnish, I get a cached response for five
> seconds as before, but when the response goes stale, rather than issuing a
> conditional request to revalidate it, Varnish is issuing unconditional
> requests:
>
>
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.68.0
> Accept: */*
> X-Forwarded-For: 127.0.0.1
> Accept-Encoding: gzip
> X-Varnish: 3
>
> GET / HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.68.0
> Accept: */*
> X-Forwarded-For: 127.0.0.1
> Accept-Encoding: gzip
> X-Varnish: 32771
>
>
> Is that something I can adjust with configuration?  If it's relevant, I'm
> running:
>
> Debug: Version: varnish-6.2.1 revision
> 9f8588e4ab785244e06c3446fe09bf9db5dd8753
> Debug: Platform:
> Linux,5.4.0-153-generic,x86_64,-jnone,-sdefault,-sdefault,-hcritbit
>
> Incidentally, 200 responses with content-length 0 also seem to exhibit
> this behaviour.
>
> Thanks in advance,
>
> Mark
>
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230714/d2a4f30d/attachment-0001.html>