varnish and mp4 files
Karim Ayari
karim.ayari at univ-lyon1.fr
Tue Feb 21 08:05:12 UTC 2023
sorry I thought I noted it in my previous email.
here are the good lines for using pipe :
/if (req.url ~ "(?i)^/[^?]+\.mp4($|\?)") {//
// std.log("ispiped:true");//
// return (pipe);//
//}/
thank you!
Karim.
Le 20/02/2023 à 21:50, Guillaume Quintard a écrit :
> Looks like Rainer replied directly to you and not to the list, would
> you mind sharing/highlighting the fix for people having the same issue?
>
> Cheers,
>
> --
> Guillaume Quintard
>
>
> On Mon, Feb 20, 2023 at 12:36 PM Karim Ayari
> <karim.ayari at univ-lyon1.fr> wrote:
>
> thank you both for your replies.I forgot the most important
> thing:the varnish server has 16 GB of ram and the cache is 1 GB.
>
> the cache never seems full
>
>
> the command used : */usr/sbin/varnishd -j unix,user=vcache -F
> -a:8080 -T localhost:6082 -f /etc/varnish/moodle.vcl -S
> /etc/varnish/secret -s malloc,1g****-p http_max_hdr=96***
>
> I had tried using pipe it didn't work, but Rainer's lines works fine.
>
> this solution should suffice because the videos will soon have to
> be hosted on our video platform.
>
> /
> /
>
> my vcl (file found on github) :
>
> /
> /
>
> /....
> /
>
> /sub vcl_recv {//
> //
> // # Keep client IP//
> // if (req.restarts == 0) {//
> // if (req.http.x-forwarded-for) {//
> // set req.http.X-Forwarded-For =
> req.http.X-Forwarded-For + ", " + client.ip;//
> // } else {//
> // unset req.http.X-Forwarded-For;//
> // set req.http.X-Forwarded-For = client.ip;//
> // }//
> // }//
> //
> // if (req.http.X-Real-IP) {//
> // set req.http.X-Forwarded-For = req.http.X-Real-IP;//
> // } else {//
> // set req.http.X-Forwarded-For = client.ip;//
> // }//
> ////
> // # Only deal with "normal" types//
> // if (req.method != "GET" &&//
> // req.method != "HEAD" &&//
> // req.method != "PUT" &&//
> // req.method != "POST" &&//
> // req.method != "TRACE" &&//
> // req.method != "OPTIONS" &&//
> // req.method != "PATCH" &&//
> // req.method != "DELETE") {//
> // /* Non-RFC2616 or CONNECT which is weird. *///
> // /*Why send the packet upstream, while the visitor is using a
> non-valid HTTP method? *///
> // return (synth(404, "Non-valid HTTP method!"));//
> // }//
> ////
> // # Varnish don't mess with healthchecks//
> // if (req.url ~ "^/admin/tool/heartbeat" || req.url ~
> "^/healthcheck.php") {//
> // return (pass);//
> // }//
> // # Pipe requests to backup.php straight to backend - prevents
> problem with progress bar long polling 503 problem//
> // # This is here because backup.php is POSTing to itself -
> Filter before !GET&&!HEAD//
> // if (req.url ~ "^/backup/backup.php")//
> // {//
> // return (pipe);//
> // }//
> //
> // # Varnish only deals with GET and HEAD by default. If
> request method is not GET or HEAD, pass request to backend//
> // if (req.method != "GET" && req.method != "HEAD") {//
> // return (pass);//
> // }//
> //
> // if (req.http.Cookie) {//
> // # Remove any Google Analytics based cookies//
> // set req.http.Cookie = regsuball(req.http.Cookie, "^_ga$",
> "");//
> // set req.http.Cookie = regsuball(req.http.Cookie, "^_gid$",
> "");//
> // set req.http.Cookie = regsuball(req.http.Cookie,
> "__gads=[^;]+(; )?", "");//
> // set req.http.Cookie = regsuball(req.http.Cookie,
> "__qc.=[^;]+(; )?", "");//
> // set req.http.Cookie = regsuball(req.http.Cookie,
> "__atuv.=[^;]+(; )?", "");//
> // set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*",
> "");//
> // if (req.http.Cookie ~ "^\s*$") {//
> // unset req.http.Cookie;//
> // }//
> // }//
> //
> // ### Rules for Moodle ###//
> //
> // # Perform lookup for selected assets that we know are static
> but Moodle still needs a Cookie//
> // if( req.url ~ "^/theme/.+\.?" ||//
> // req.url ~ "^/webservice/pluginfile.php/.+\.(png|jpg)$" ||//
> // req.url ~ "^/lib/.+\.(png|jpg|jpeg|gif|css|js|webp)$" ||//
> // req.url ~
> "^/pluginfile.php/[0-9]+/course/.+\.(?i)(png|jpg)$" ||//
> // req.url ~
> "^/pluginfile.php/[0-9]+/theme_moove/.+\.(?i)(png|jpg)$"//
> // )//
> // {//
> // # Set internal temporary header, based on which we will
> do things in vcl_backend_response//
> // set req.http.X-Long-TTL = "86400";//
> // return (hash);//
> // }//
> // # Requests containing "Cookie" or "Authorization" headers
> will not be cached//
> // if (req.http.Authorization || req.http.Cookie) {//
> // return (pass);//
> // }//
> // # Almost everything in Moodle correctly serves Cache-Control
> headers, if//
> // # needed, which varnish will honor, but there are some which
> don't. Rather//
> // # than explicitly finding them all and listing them here we
> just fail safe//
> // # and don't cache unknown urls that get this far.//
> // return (pass);//
> //
> //}//
> //
> //sub vcl_backend_response {//
> // # Set backend name//
> // set beresp.http.X-Backend = beresp.backend.name
> <http://beresp.backend.name>;//
> //
> // if (beresp.http.Cache-Control && bereq.http.X-Long-TTL &&
> beresp.ttl < std.duration(bereq.http.X-Long-TTL + "s", 1s) &&
> !beresp.http.WWW-Authenticate ) { # If max-age < defined in
> X-Long-TTL header//
> // set beresp.http.X-Orig-Pragma = beresp.http.Pragma;
> unset beresp.http.Pragma;//
> // set beresp.http.X-Orig-Cache-Control =
> beresp.http.Cache-Control;//
> // set beresp.http.Cache-Control = "public,
> max-age="+bereq.http.X-Long-TTL+", no-transform";//
> // set beresp.ttl = std.duration(bereq.http.X-Long-TTL +
> "s", 1s);//
> // unset bereq.http.X-Long-TTL;//
> // }//
> // else if (!beresp.http.Cache-Control && bereq.http.X-Long-TTL
> && !beresp.http.WWW-Authenticate ) {//
> // set beresp.http.X-Orig-Pragma = beresp.http.Pragma;
> unset beresp.http.Pragma;//
> // set beresp.http.Cache-Control = "public,
> max-age="+bereq.http.X-Long-TTL+", no-transform";//
> // set beresp.ttl = std.duration(bereq.http.X-Long-TTL +
> "s", 1s);//
> // unset bereq.http.X-Long-TTL;//
> // }//
> // else { # Don't touch headers if max-age > defined in
> X-Long-TTL header//
> // unset bereq.http.X-Long-TTL;//
> // }//
> // # Here we set X-Trace header, prepending it to X-Trace
> header received from backend. Useful for troubleshooting//
> // if (beresp.http.x-trace && !beresp.was_304) {//
> // set beresp.http.X-Trace = regsub(server.identity,
> "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name
> <http://beresp.backend.name>,
> "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)")+"->"+beresp.http.X-Trace;//
> // }//
> // else {//
> // set beresp.http.X-Trace = regsub(server.identity,
> "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name
> <http://beresp.backend.name>,
> "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)");//
> // }//
> //}//
> //
> //sub vcl_deliver {//
> //
> //# Revert back to original Cache-Control header before delivery
> to client//
> // if (resp.http.X-Orig-Cache-Control)//
> // {//
> // set resp.http.Cache-Control =
> resp.http.X-Orig-Cache-Control;//
> // unset resp.http.X-Orig-Cache-Control;//
> // }//
> // # Revert back to original Pragma header before delivery to
> client//
> // if (resp.http.X-Orig-Pragma)//
> // {//
> // set resp.http.Pragma = resp.http.X-Orig-Pragma;//
> // unset resp.http.X-Orig-Pragma;//
> // }//
> //
> // if (obj.hits > 0) { # Add debug header to see if it's a
> HIT/MISS and the number of hits, disable when not needed//
> // set resp.http.X-Cache = "HIT";//
> // } else {//
> // set resp.http.X-Cache = "MISS";//
> // }//
> // set resp.http.X-Cache-Hits = obj.hits;//
> //
> //# If desired "Via: 1.1 Varnish-v4" response header can be
> removed from response//
> // unset resp.http.Via;//
> // unset resp.http.Server;//
> //
> // return (deliver);//
> //}//
> //
> //sub vcl_backend_error {//
> // # More comprehensive varnish error page. Display time,
> instance hostname, host header, url for easier troubleshooting.//
> // set beresp.http.Content-Type = "text/html; charset=utf-8";//
> // set beresp.http.Retry-After = "5";//
> // synthetic( {"//
> // <!DOCTYPE html>//
> // <html>//
> // <head>//
> // <title>"} + beresp.status + " " + beresp.reason + {"</title>//
> // </head>//
> // <body>//
> // <h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>//
> // <p>"} + beresp.reason + {"</p>//
> // <h3>Guru Meditation:</h3>//
> // <p>Time: "} + now + {"</p>//
> // <p>Node: "} + server.hostname + {"</p>//
> // <p>Host: "} + bereq.http.host + {"</p>//
> // <p>URL: "} + bereq.url + {"</p>//
> // <p>XID: "} + bereq.xid + {"</p>//
> // <hr>//
> // <p>Varnish cache server//
> // </body>//
> // </html>//
> // "} );//
> // return (deliver);//
> //}//
> //
> //sub vcl_synth {//
> // #Redirect using '301 - Permanent Redirect', permanent redirect//
> // if (resp.status == 851) { //
> // set resp.http.Location = req.http.x-redir;//
> // set resp.http.X-Varnish-Redirect = true;//
> // set resp.status = 301;//
> // return (deliver);//
> // }//
> // #Redirect using '302 - Found', temporary redirect//
> // if (resp.status == 852) { //
> // set resp.http.Location = req.http.x-redir;//
> // set resp.http.X-Varnish-Redirect = true;//
> // set resp.status = 302;//
> // return (deliver);//
> // }//
> // #Redirect using '307 - Temporary Redirect', !GET&&!HEAD
> requests, dont change method on redirected requests//
> // if (resp.status == 857) { //
> // set resp.http.Location = req.http.x-redir;//
> // set resp.http.X-Varnish-Redirect = true;//
> // set resp.status = 307;//
> // return (deliver);//
> // }//
> // #Respond with 403 - Forbidden//
> // if (resp.status == 863) {//
> // set resp.http.X-Varnish-Error = true;//
> // set resp.status = 403;//
> // return (deliver);//
> // }//
> //}//
> //
> //sub vcl_purge {//
> // if (req.method != "PURGE") {//
> // set req.http.X-Purge = "Yes";//
> // return (restart);//
> // }//
> //}/
>
> ...
>
>
> Le 20/02/2023 à 18:55, Guillaume Quintard a écrit :
>> Hello Karim,
>>
>> You VCL would be useful to debug this (as well as the command
>> line you are running Varnish with), but it sounds like Varnish is
>> using the Transient storage
>> (https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage)
>> to store the file, and as the storage isn't bounded, it explodes.
>> We can fix this in a couple of ways, from storing the file in the
>> regular cache storage, to using pipe, to waiting a few days for
>> https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643
>> to be released.
>>
>> Question is: should that file be cached?
>>
>> Cheers,
>>
>> --
>> Guillaume Quintard
>>
>>
>> On Mon, Feb 20, 2023 at 7:14 AM Karim Ayari
>> <karim.ayari at univ-lyon1.fr> wrote:
>>
>> Hi!
>>
>> I am currently experiencing a memory load problem with video
>> playback.
>>
>> here is the infrastructure :
>>
>> client --> haproxy --> varnish --> moodle workers (x5)
>>
>> a teacher uploaded a 400MB video to Moodle, when we start
>> playing the video with browser player, Varnish consumes all
>> the memory until it runs out and oom killer to kill
>> varnishd.i have no configuration for mp4 files in my vcl
>> file, so by default they are not hidden (?). I can't find a
>> solution :(
>>
>> I can give my vcl file if necessary.
>>
>> (I am a beginner on varnish :))
>>
>> thank you for your support.
>>
>> Karim
>>
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230221/64db32ee/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: psCUnmMoUHIDeDdC.png
Type: image/png
Size: 49126 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230221/64db32ee/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dKCCKWp0GtXF0le5.png
Type: image/png
Size: 42447 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230221/64db32ee/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4295 bytes
Desc: Signature cryptographique S/MIME
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230221/64db32ee/attachment-0001.bin>
More information about the varnish-misc
mailing list