varnish and mp4 files
Guillaume Quintard
guillaume.quintard at gmail.com
Mon Feb 20 20:50:19 UTC 2023
Looks like Rainer replied directly to you and not to the list, would you
mind sharing/highlighting the fix for people having the same issue?
Cheers,
--
Guillaume Quintard
On Mon, Feb 20, 2023 at 12:36 PM Karim Ayari <karim.ayari at univ-lyon1.fr>
wrote:
> thank you both for your replies. I forgot the most important thing: the
> varnish server has 16 GB of ram and the cache is 1 GB.
>
> the cache never seems full
>
>
> the command used : */usr/sbin/varnishd -j unix,user=vcache -F -a:8080 -T
> localhost:6082 -f /etc/varnish/moodle.vcl -S /etc/varnish/secret -s
> malloc,1g* *-p http_max_hdr=96*
>
> I had tried using pipe it didn't work, but Rainer's lines works fine.
>
> this solution should suffice because the videos will soon have to be
> hosted on our video platform.
>
>
> my vcl (file found on github) :
>
>
>
> *.... *
>
> *sub vcl_recv {*
>
> * # Keep client IP*
> * if (req.restarts == 0) {*
> * if (req.http.x-forwarded-for) {*
> * set req.http.X-Forwarded-For = req.http.X-Forwarded-For + ",
> " + client.ip;*
> * } else {*
> * unset req.http.X-Forwarded-For;*
> * set req.http.X-Forwarded-For = client.ip;*
> * }*
> * }*
>
> * if (req.http.X-Real-IP) {*
> * set req.http.X-Forwarded-For = req.http.X-Real-IP;*
> * } else {*
> * set req.http.X-Forwarded-For = client.ip;*
> * }*
>
> * # Only deal with "normal" types*
> * if (req.method != "GET" &&*
> * req.method != "HEAD" &&*
> * req.method != "PUT" &&*
> * req.method != "POST" &&*
> * req.method != "TRACE" &&*
> * req.method != "OPTIONS" &&*
> * req.method != "PATCH" &&*
> * req.method != "DELETE") {*
> * /* Non-RFC2616 or CONNECT which is weird. */*
> * /*Why send the packet upstream, while the visitor is using a
> non-valid HTTP method? */*
> * return (synth(404, "Non-valid HTTP method!"));*
> * }*
>
> * # Varnish don't mess with healthchecks*
> * if (req.url ~ "^/admin/tool/heartbeat" || req.url ~
> "^/healthcheck.php") {*
> * return (pass);*
> * }*
> * # Pipe requests to backup.php straight to backend - prevents problem
> with progress bar long polling 503 problem*
> * # This is here because backup.php is POSTing to itself - Filter
> before !GET&&!HEAD*
> * if (req.url ~ "^/backup/backup.php")*
> * {*
> * return (pipe);*
> * }*
>
> * # Varnish only deals with GET and HEAD by default. If request method
> is not GET or HEAD, pass request to backend*
> * if (req.method != "GET" && req.method != "HEAD") {*
> * return (pass);*
> * }*
>
> * if (req.http.Cookie) {*
> * # Remove any Google Analytics based cookies*
> * set req.http.Cookie = regsuball(req.http.Cookie, "^_ga$", "");*
> * set req.http.Cookie = regsuball(req.http.Cookie, "^_gid$", "");*
> * set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(;
> )?", "");*
> * set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(;
> )?", "");*
> * set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(;
> )?", "");*
> * set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");*
> * if (req.http.Cookie ~ "^\s*$") {*
> * unset req.http.Cookie;*
> * }*
> * }*
>
> * ### Rules for Moodle ###*
>
> * # Perform lookup for selected assets that we know are static but
> Moodle still needs a Cookie*
> * if( req.url ~ "^/theme/.+\.?" ||*
> * req.url ~ "^/webservice/pluginfile.php/.+\.(png|jpg)$" ||*
> * req.url ~ "^/lib/.+\.(png|jpg|jpeg|gif|css|js|webp)$" ||*
> * req.url ~ "^/pluginfile.php/[0-9]+/course/.+\.(?i)(png|jpg)$" ||*
> * req.url ~ "^/pluginfile.php/[0-9]+/theme_moove/.+\.(?i)(png|jpg)$"*
> * )*
> * {*
> * # Set internal temporary header, based on which we will do
> things in vcl_backend_response*
> * set req.http.X-Long-TTL = "86400";*
> * return (hash);*
> * }*
> * # Requests containing "Cookie" or "Authorization" headers will not be
> cached*
> * if (req.http.Authorization || req.http.Cookie) {*
> * return (pass);*
> * }*
> * # Almost everything in Moodle correctly serves Cache-Control headers,
> if*
> * # needed, which varnish will honor, but there are some which don't.
> Rather*
> * # than explicitly finding them all and listing them here we just fail
> safe*
> * # and don't cache unknown urls that get this far.*
> * return (pass);*
>
> *}*
>
> *sub vcl_backend_response {*
> * # Set backend name*
> * set beresp.http.X-Backend = beresp.backend.name
> <http://beresp.backend.name>;*
>
> * if (beresp.http.Cache-Control && bereq.http.X-Long-TTL && beresp.ttl
> < std.duration(bereq.http.X-Long-TTL + "s", 1s) &&
> !beresp.http.WWW-Authenticate ) { # If max-age < defined in X-Long-TTL
> header*
> * set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset
> beresp.http.Pragma;*
> * set beresp.http.X-Orig-Cache-Control = beresp.http.Cache-Control;*
> * set beresp.http.Cache-Control = "public,
> max-age="+bereq.http.X-Long-TTL+", no-transform";*
> * set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);*
> * unset bereq.http.X-Long-TTL;*
> * }*
> * else if (!beresp.http.Cache-Control && bereq.http.X-Long-TTL &&
> !beresp.http.WWW-Authenticate ) {*
> * set beresp.http.X-Orig-Pragma = beresp.http.Pragma; unset
> beresp.http.Pragma;*
> * set beresp.http.Cache-Control = "public,
> max-age="+bereq.http.X-Long-TTL+", no-transform";*
> * set beresp.ttl = std.duration(bereq.http.X-Long-TTL + "s", 1s);*
> * unset bereq.http.X-Long-TTL;*
> * }*
> * else { # Don't touch headers if max-age > defined in X-Long-TTL
> header*
> * unset bereq.http.X-Long-TTL;*
> * }*
> * # Here we set X-Trace header, prepending it to X-Trace header
> received from backend. Useful for troubleshooting*
> * if (beresp.http.x-trace && !beresp.was_304) {*
> * set beresp.http.X-Trace = regsub(server.identity,
> "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name
> <http://beresp.backend.name>,
> "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)")+"->"+beresp.http.X-Trace;*
> * }*
> * else {*
> * set beresp.http.X-Trace = regsub(server.identity,
> "^([^.]+),?.*$", "\1")+"->"+regsub(beresp.backend.name
> <http://beresp.backend.name>,
> "^(.+)\((?:[0-9]{1,3}\.){3}([0-9]{1,3})\)","\1(\2)");*
> * }*
> *}*
>
> *sub vcl_deliver {*
>
> *# Revert back to original Cache-Control header before delivery to client*
> * if (resp.http.X-Orig-Cache-Control)*
> * {*
> * set resp.http.Cache-Control = resp.http.X-Orig-Cache-Control;*
> * unset resp.http.X-Orig-Cache-Control;*
> * }*
> * # Revert back to original Pragma header before delivery to client*
> * if (resp.http.X-Orig-Pragma)*
> * {*
> * set resp.http.Pragma = resp.http.X-Orig-Pragma;*
> * unset resp.http.X-Orig-Pragma;*
> * }*
>
> * if (obj.hits > 0) { # Add debug header to see if it's a HIT/MISS and
> the number of hits, disable when not needed*
> * set resp.http.X-Cache = "HIT";*
> * } else {*
> * set resp.http.X-Cache = "MISS";*
> * }*
> * set resp.http.X-Cache-Hits = obj.hits;*
>
> *# If desired "Via: 1.1 Varnish-v4" response header can be removed from
> response*
> * unset resp.http.Via;*
> * unset resp.http.Server;*
>
> * return (deliver);*
> *}*
>
> *sub vcl_backend_error {*
> * # More comprehensive varnish error page. Display time, instance
> hostname, host header, url for easier troubleshooting.*
> * set beresp.http.Content-Type = "text/html; charset=utf-8";*
> * set beresp.http.Retry-After = "5";*
> * synthetic( {"*
> * <!DOCTYPE html>*
> * <html>*
> * <head>*
> * <title>"} + beresp.status + " " + beresp.reason + {"</title>*
> * </head>*
> * <body>*
> * <h1>Error "} + beresp.status + " " + beresp.reason + {"</h1>*
> * <p>"} + beresp.reason + {"</p>*
> * <h3>Guru Meditation:</h3>*
> * <p>Time: "} + now + {"</p>*
> * <p>Node: "} + server.hostname + {"</p>*
> * <p>Host: "} + bereq.http.host + {"</p>*
> * <p>URL: "} + bereq.url + {"</p>*
> * <p>XID: "} + bereq.xid + {"</p>*
> * <hr>*
> * <p>Varnish cache server*
> * </body>*
> * </html>*
> * "} );*
> * return (deliver);*
> *}*
>
> *sub vcl_synth {*
> * #Redirect using '301 - Permanent Redirect', permanent redirect*
> * if (resp.status == 851) { *
> * set resp.http.Location = req.http.x-redir;*
> * set resp.http.X-Varnish-Redirect = true;*
> * set resp.status = 301;*
> * return (deliver);*
> * }*
> * #Redirect using '302 - Found', temporary redirect*
> * if (resp.status == 852) { *
> * set resp.http.Location = req.http.x-redir;*
> * set resp.http.X-Varnish-Redirect = true;*
> * set resp.status = 302;*
> * return (deliver);*
> * }*
> * #Redirect using '307 - Temporary Redirect', !GET&&!HEAD requests,
> dont change method on redirected requests*
> * if (resp.status == 857) { *
> * set resp.http.Location = req.http.x-redir;*
> * set resp.http.X-Varnish-Redirect = true;*
> * set resp.status = 307;*
> * return (deliver);*
> * }*
> * #Respond with 403 - Forbidden*
> * if (resp.status == 863) {*
> * set resp.http.X-Varnish-Error = true;*
> * set resp.status = 403;*
> * return (deliver);*
> * }*
> *}*
>
> *sub vcl_purge {*
> * if (req.method != "PURGE") {*
> * set req.http.X-Purge = "Yes";*
> * return (restart);*
> * }*
> *}*
>
> ...
>
>
> Karim Ayari
> Université Claude Bernard - Lyon 1
> Service ICAP
> Ingénieur systèmes
>
> Le 20/02/2023 à 18:55, Guillaume Quintard a écrit :
>
> Hello Karim,
>
> You VCL would be useful to debug this (as well as the command line you are
> running Varnish with), but it sounds like Varnish is using the Transient
> storage (
> https://varnish-cache.org/docs/trunk/users-guide/storage-backends.html#transient-storage)
> to store the file, and as the storage isn't bounded, it explodes.
> We can fix this in a couple of ways, from storing the file in the regular
> cache storage, to using pipe, to waiting a few days for
> https://github.com/varnishcache/varnish-cache/pull/3572#issuecomment-1305736643
> to be released.
>
> Question is: should that file be cached?
>
> Cheers,
>
> --
> Guillaume Quintard
>
>
> On Mon, Feb 20, 2023 at 7:14 AM Karim Ayari <karim.ayari at univ-lyon1.fr>
> wrote:
>
>> Hi!
>>
>> I am currently experiencing a memory load problem with video playback.
>>
>> here is the infrastructure :
>>
>> client --> haproxy --> varnish --> moodle workers (x5)
>>
>> a teacher uploaded a 400MB video to Moodle, when we start playing the
>> video with browser player, Varnish consumes all the memory until it runs
>> out and oom killer to kill varnishd. i have no configuration for mp4
>> files in my vcl file, so by default they are not hidden (?). I can't
>> find a solution :(
>>
>> I can give my vcl file if necessary.
>>
>> (I am a beginner on varnish :))
>>
>> thank you for your support.
>>
>> Karim
>> _______________________________________________
>> varnish-misc mailing list
>> varnish-misc at varnish-cache.org
>> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230220/a31ee9fe/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: psCUnmMoUHIDeDdC.png
Type: image/png
Size: 49126 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230220/a31ee9fe/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: dKCCKWp0GtXF0le5.png
Type: image/png
Size: 42447 bytes
Desc: not available
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20230220/a31ee9fe/attachment-0003.png>
More information about the varnish-misc
mailing list