Best practice for caching scenario with different backend servers but same content

[Dridi Boukelmoune]

On Sat, Aug 7, 2021 at 9:21 AM Hamidreza Hosseini
<hrhosseini at hotmail.com> wrote:
>
> Hi,
> I read the sample config that you sent but Can I use "bereq.url"  in this way:
> for example I want to shard my requests for live streams based on the url's that clients enter, for example if the following url's are different live streams (stream1 and stream2 are the name of different streams ):
> ```
> mydomain.com/live/australia/stream1/chunk_43212123.ts
> mydomain.com/live/australia/stream2/chunk_43212123.ts
> mydomain.com/live/australia/stream3/chunk_43212123.ts
> ```
> Now think I want just the url excluded with chunk file becomes hashed and sharded:
> Just this part: "/live/australia/stream{1,2,3}/"
> Not :"/live/australia/stream{1,2,3}/chunk_43212123.ts"
>
> So by adjusting " p.set(by=KEY, key=bereq.url) " it would shard "bereq.url",  it means "/live/australia/stream{1,2,3}/chunk_43212123.ts"

If you want to do that, you need to extract the relevant part and
compute a key from it:

> set bereq.http.my-key = bereq.http.host + regsub(bereq.url, "/chunk_.*", "");
> set bereq.backend = dir.backend(by=KEY, key=dir.key(bereq.http.my-key));

Below is a test case where both requests go through the same
backend:

---
    varnishtest "shard by subset of url"

    server s1 {
        loop 2 {
            rxreq
            txresp
        }
    } -start

    server s2 {
        loop 2 {
            rxreq
            txresp
        }
    } -start

    varnish v1 -vcl+backend {
        import directors;

        sub vcl_init {
            new dir = directors.shard();
            dir.add_backend(s1);
            dir.add_backend(s2);
        }

        sub vcl_backend_fetch {
            set bereq.http.shard-key = bereq.http.host +
                regsub(bereq.url, "/chunk_.*", "");
            set bereq.backend = dir.backend(by=KEY,
                key=dir.key(bereq.http.shard-key));
        }

        sub vcl_backend_response {
            set beresp.http.backend = beresp.backend;
        }
    } -start

    client c1 {
        txreq -url "/live/australia/stream1/chunk_123.ts"
        rxresp
        expect resp.http.backend == s1

        txreq -url "/live/australia/stream1/chunk_456.ts"
        rxresp
        expect resp.http.backend == s1
    } -run
---

Dridi