varnish and mp4 files
Karim Ayari
karim.ayari at univ-lyon1.fr
Mon Feb 20 20:36:12 UTC 2023
thank you both for your replies.I forgot the most important thing:the
varnish server has 16 GB of ram and the cache is 1 GB.
the cache never seems full
the command used : */usr/sbin/varnishd -j unix,user=vcache -F -a:8080 -T
localhost:6082 -f /etc/varnish/moodle.vcl -S /etc/varnish/secret -s
malloc,1g****-p http_max_hdr=96***
I had tried using pipe it didn't work, but Rainer's lines works fine.
this solution should suffice because the videos will soon have to be
hosted on our video platform.
/
/
my vcl (file found on github) :
/
/
/....
/
/sub vcl_recv {//
//
// # Keep client IP//
// if (req.restarts == 0) {//
// if (req.http.x-forwarded-for) {//
// set req.http.X-Forwarded-For = req.http.X-Forwarded-For +
", " + client.ip;//
// } else {//
// unset req.http.X-Forwarded-For;//
// set req.http.X-Forwarded-For = client.ip;//
// }//
// }//
//
// if (req.http.X-Real-IP) {//
// set req.http.X-Forwarded-For = req.http.X-Real-IP;//
// } else {//
// set req.http.X-Forwarded-For = client.ip;//
// }//
////
// # Only deal with "normal" types//
// if (req.method != "GET" &&//
// req.method != "HEAD" &&//
// req.method != "PUT" &&//
// req.method != "POST" &&//
// req.method != "TRACE" &&//
// req.method != "OPTIONS" &&//
// req.method != "PATCH" &&//
// req.method != "DELETE") {//
// /* Non-RFC2616 or CONNECT which is weird. *///
// /*Why send the packet upstream, while the visitor is using a
non-valid HTTP method? *///
// return (synth(404, "Non-valid HTTP method!"));//
// }//
////
// # Varnish don't mess with healthchecks//
// if (req.url ~ "^/admin/tool/heartbeat" || req.url ~
"^/healthcheck.php") {//
// return (pass);//
// }//
// # Pipe requests to backup.php straight to backend - prevents
problem with progress bar long polling 503 problem//
// # This is here because backup.php is POSTing to itself - Filter
before !GET&&!HEAD//
// if (req.url ~ "^/backup/backup.php")//
// {//
// return (pipe);//
// }//
//
// # Varnish only deals with GET and HEAD by default. If request
method is not GET or HEAD, pass request to backend//
// if (req.method != "GET" && req.method != "HEAD") {//
// return (pass);//
// }//
//
// if (req.http.Cookie) {//
// # Remove any Google Analytics based cookies//
// set req.http.Cookie = regsuball(req.http.Cookie, "^_ga$", "");//
// set req.http.Cookie = regsuball(req.http.Cookie, "^_gid$", "");//
// set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(;
)?", "");//
// set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(;
)?", "");//
// set req.http.Cookie = regsuball(req.http.Cookie,
"__atuv.=[^;]+(; )?", "");//
// set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");//
// if (req.http.Cookie ~ "^\s*$") {//
// unset req.http.Cookie;//
// }//
// }//
//
// ### Rules for Moodle ###//
//
// # Perform lookup for selected assets that we know are static but
Moodle still needs a Cookie//
// if( req.url ~ "^/theme/.+\.?" ||//
// req.url ~ "^/webservice/pluginfile.php/.+\.(png|jpg)$" ||//
// req.url ~ "^/lib/.+\.(png|jpg|jpeg|gif|css|js|webp)$" ||//
// req.url ~ "^/pluginfile.php/[0-9]+/course/.+\.(?i)(png|jpg)$"
||//
// req.url ~ "^/pluginfile.php/[0-9]+/theme_moove/.+\.(?i)(png|jpg)$"//
// )//
// {//
// # Set internal temporary header, based on which we will do
things in vcl_backend_response//
// set req.http.X-Long-TTL = "86400";//
// return (hash);//
// }//
// # Requests containing "Cookie" or "Authorization" headers will not
be cached//
// if (req.http.Authorization || req.http.Cookie) {//
// return (pass);//
// }//
// # Almost everything in Moodle correctly serves Cache-Control
headers, if//
// # needed, which varnish will honor, but there are some which
don't. Rather//
// # than explicitly finding them all and listing them here we just
fail safe//
// # and don't cache unknown urls that get this far.//
// return (pass);//
//
//}//
//
//sub vcl_backend_response {//
// # Set backend name//
// set beresp.http.X-Backend = beresp.backend.name;//
//
// if (beresp.http.Cache-Control && bereq.http.X-Long-TTL &&
beresp.ttl < std.duration(bereq.http.X-Long-TTL + "s", 1s) &&
!beresp.http.WWW-Authenticate ) { # If max-age < defined in X-Long-TTL
header//
// set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset
beresp.http.Pragma;//
// set beresp.http.X-Orig-Cache-Control =
beresp.http.Cache-Control;//
// set beresp.http.Cache-Control = "public,
max-age="+bereq.http.X-Long-TTL+", no-transform";//
// set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);//
// unset bereq.http.X-Long-TTL;//
// }//
// else if (!beresp.http.Cache-Control && bereq.http.X-Long-TTL &&
!beresp.http.WWW-Authenticate ) {//
// set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset
beresp.http.Pragma;//
// set beresp.http.Cache-Control = "public,
max-age="+bereq.http.X-Long-TTL+", no-transform";//
// set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);//
// unset bereq.http.X-Long-TTL;//
// }//
// else { # Don't touch headers if max-age > defined in X-Long-TTL
header//
// unset bereq.http.X-Long-TTL;//
// }//
// # Here we set X-Trace header, prepending it to X-Trace header
received from backend. Useful for troubleshooting//
// if (beresp.http.x-trace && !beresp.was_304) {//
// set beresp.http.X-Trace = regsub(server.identity,
"^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name,
"^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)")+"->"+beresp.http.X-Trace;//
// }//
// else {//
// set beresp.http.X-Trace = regsub(server.identity,
"^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name,
"^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)");//
// }//
//}//
//
//sub vcl_deliver {//
//
//# Revert back to original Cache-Control header before delivery to client//
// if (resp.http.X-Orig-Cache-Control)//
// {//
// set resp.http.Cache-Control = resp.http.X-Orig-Cache-Control;//
// unset resp.http.X-Orig-Cache-Control;//
// }//
// # Revert back to original Pragma header before delivery to client//
// if (resp.http.X-Orig-Pragma)//
// {//
// set resp.http.Pragma = resp.http.X-Orig-Pragma;//
// unset resp.http.X-Orig-Pragma;//
// }//
//
// if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and
the number of hits, disable when not needed//
// set resp.http.X-Cache = "HIT";//
// } else {//
// set resp.http.X-Cache = "MISS";//
// }//
// set resp.http.X-Cache-Hits = obj.hits;//
//
//# If desired "Via: 1.1 Varnish-v4" response header can be removed from
response//
// unset resp.http.Via;//
// unset resp.http.Server;//
//
// return (deliver);//
//}//
//
//sub vcl_backend_error {//
// # More comprehensive varnish error page. Display time, instance
hostname, host header, url for easier troubleshooting.//
// set beresp.http.Content-Type = "text/html; charset=utf-8";//
// set beresp.http.Retry-After = "5";//
// synthetic( {"//
// <!DOCTYPE html>//
// <html>//
// <head>//
// <title>"} + beresp.status + " " + beresp.reason + {"</title>//
// </head>//
// <body>//
// <h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>//
// <p>"} + beresp.reason + {"</p>//
// <h3>Guru Meditation:</h3>//
// <p>Time: "} + now + {"</p>//
// <p>Node: "} + server.hostname + {"</p>//
// <p>Host: "} + bereq.http.host + {"</p>//
// <p>URL: "} + bereq.url + {"</p>//
// <p>XID: "} + bereq.xid + {"</p>//
// <hr>//
// <p>Varnish cache server//
// </body>//
// </html>//
// "} );//
// return (deliver);//
//}//
//
//sub vcl_synth {//
// #Redirect using '301 - Permanent Redirect', permanent redirect//
// if (resp.status == 851) { //
// set resp.http.Location = req.http.x-redir;//
// set resp.http.X-Varnish-Redirect = true;//
// set resp.status = 301;//
// return (deliver);//
// }//
// #Redirect using '302 - Found', temporary redirect//
// if (resp.status == 852) { //
// set resp.http.Location = req.http.x-redir;//
// set resp.http.X-Varnish-Redirect = true;//
// set resp.status = 302;//
// return (deliver);//
// }//
// #Redirect using '307 - Temporary Redirect', !GET&&!HEAD requests,
dont change method on redirected requests//
// if (resp.status == 857) { //
// set resp.http.Location = req.http.x-redir;//
// set resp.http.X-Varnish-Redirect = true;//
// set resp.status = 307;//
// return (deliver);//
// }//
// #Respond with 403 - Forbidden//
// if (resp.status == 863) {//
// set resp.http.X-Varnish-Error = true;//
// set resp.status = 403;//
// return (deliver);//
// }//
//}//
//
//sub vcl_purge {//
// if (req.method != "PURGE") {//
// set req.http.X-Purge = "Yes";//
// return (restart);//
// }//
//}/
...
Karim Ayari
Université Claude Bernard - Lyon 1
Service ICAP
Ingénieur systèmes
Le 20/02/2023 à 18:55, Guillaume Quintard a écrit :
> Hello Karim,
>
> You VCL would be useful to debug this (as well as the command line you
> are running Varnish with), but it sounds like Varnish is using the
> Transient storage
> (https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage)
> to store the file, and as the storage isn't bounded, it explodes.
> We can fix this in a couple of ways, from storing the file in the
> regular cache storage, to using pipe, to waiting a few days for
> https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643
> to be released.
>
> Question is: should that file be cached?
>
> Cheers,
>
> --
> Guillaume Quintard
>
>
> On Mon, Feb 20, 2023 at 7:14 AM Karim Ayari
> <karim.ayari at univ-lyon1.fr> wrote:
>
> Hi!
>
> I am currently experiencing a memory load problem with video playback.
>
> here is the infrastructure :
>
> client --> haproxy --> varnish --> moodle workers (x5)
>
> a teacher uploaded a 400MB video to Moodle, when we start playing
> the video with browser player, Varnish consumes all the memory
> until it runs out and oom killer to kill varnishd.i have no
> configuration for mp4 files in my vcl file, so by default they are
> not hidden (?). I can't find a solution :(
>
> I can give my vcl file if necessary.
>
> (I am a beginner on varnish :))
>
> thank you for your support.
>
> Karim
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230220/5d80b4c2/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: psCUnmMoUHIDeDdC.png
Type: image/png
Size: 49126 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230220/5d80b4c2/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dKCCKWp0GtXF0le5.png
Type: image/png
Size: 42447 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230220/5d80b4c2/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4295 bytes
Desc: Signature cryptographique S/MIME
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230220/5d80b4c2/attachment-0001.bin>
More information about the varnish-misc
mailing list