From luca.gervasi at gmail.com Mon Jul 1 15:00:58 2024 From: luca.gervasi at gmail.com (Luca Gervasi) Date: Mon, 1 Jul 2024 17:00:58 +0200 Subject: Varnish memory usage Message-ID: Hello, I'm following an issue that caused some OOMs during a huge spike. Currently I have the following properties set: -p thread_pool_max=2000 \ -p thread_pool_min=35 \ -p feature=+esi_disable_xml_check \ -p feature=+esi_ignore_other_elements \ -p feature=+esi_remove_bom \ -p feature=+http2 \ -p pcre_match_limit_recursion=64 \ -p syslog_cli_traffic=off \ -p sigsegv_handler=on \ -p workspace_client=2m \ -p workspace_backend=2m \ -p http_max_hdr=128 \ -p http_req_hdr_len=32k \ -p http_resp_hdr_len=32k \ -p http_req_size=2m \ -p http_resp_size=2m \ -p thread_pool_stack=256k \ -s malloc,20G \ -S /config/varnish/secret \ -f /config/varnish/default.vcl \ -p vsl_reclen=700 And I'm using this formula to somehow calculate how much memory is varnish going to use: thread_pool_max*thread_pool_stack+thread_pool_max*(workspace_client+http_req_hdr_len+http_req_size) + malloc And it is somewhere close to where I get OOM. I suspect that what did screw up was the high esi-to-request ration that I normally have: MAIN.client_req 469.22 MAIN.esi_req 5489.25 Hit rate on that esi requests is pretty high as we use high ttls and invalidations, but I think that those ESIs have to be taken into account too (as well as the backend requests too...). I wasn't able to find an official memory sizing guide, sorry. How do you size correctly memory usage? I know I should use MSE, at the moment is not on my list. Thanks Luca -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Mon Jul 1 15:46:09 2024 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Mon, 1 Jul 2024 17:46:09 +0200 Subject: Varnish memory usage In-Reply-To: References: Message-ID: Hello Luca, This should give you some information: https://info.varnish-software.com/blog/understanding-varnish-cache-memory-usage . Notably, you need to account for some memory fragmentation. Note that your formula is probably a bit optimistic: each thread is allowed to consume as much as workspace_client/workspace_backend, and workspace will be consumed by VCL operations, like regex matching and substitutions. One thing you can do is use vmod_vtc: https://varnish-cache.org/docs/trunk/reference/vmod_vtc.html#int-workspace-free-enum-client-backend-session-thread . You could log the remaining size for all requests and see how close you get to zero. I'd like to ask: which Varnish packages are you using? (i.e. version and origin). I know some people have been hurt before, by the jemalloc package I think, and depending on how Varnish is built, you can get better a smaller memory footprint. -- Guillaume Quintard On Mon, Jul 1, 2024 at 5:02?PM Luca Gervasi wrote: > Hello, > I'm following an issue that caused some OOMs during a huge spike. > Currently I have the following properties set: > > -p thread_pool_max=2000 \ > -p thread_pool_min=35 \ > -p feature=+esi_disable_xml_check \ > -p feature=+esi_ignore_other_elements \ > -p feature=+esi_remove_bom \ > -p feature=+http2 \ > -p pcre_match_limit_recursion=64 \ > -p syslog_cli_traffic=off \ > -p sigsegv_handler=on \ > -p workspace_client=2m \ > -p workspace_backend=2m \ > -p http_max_hdr=128 \ > -p http_req_hdr_len=32k \ > -p http_resp_hdr_len=32k \ > -p http_req_size=2m \ > -p http_resp_size=2m \ > -p thread_pool_stack=256k \ > -s malloc,20G \ > -S /config/varnish/secret \ > -f /config/varnish/default.vcl \ > -p vsl_reclen=700 > > And I'm using this formula to somehow calculate how much memory is varnish > going to use: > thread_pool_max*thread_pool_stack+thread_pool_max*(workspace_client+http_req_hdr_len+http_req_size) + > malloc > > And it is somewhere close to where I get OOM. I suspect that what did > screw up was the high esi-to-request ration that I normally have: > > MAIN.client_req 469.22 > MAIN.esi_req 5489.25 > > Hit rate on that esi requests is pretty high as we use high ttls and > invalidations, but I think that those ESIs have to be taken into account > too (as well as the backend requests too...). > > I wasn't able to find an official memory sizing guide, sorry. How do you > size correctly memory usage? > I know I should use MSE, at the moment is not on my list. > > Thanks > Luca > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From justinl at arena.net Mon Jul 15 16:42:17 2024 From: justinl at arena.net (Justin Lloyd) Date: Mon, 15 Jul 2024 16:42:17 +0000 Subject: 400 Bad Request and whitespace in headers Message-ID: Hi all, I'm trying to figure out what the requests are that are resulting in the following Varnish responses and how to block them: * << Request >> 39071654 - Begin req 39071653 rxreq - Timestamp Start: 1721059686.537197 0.000000 0.000000 - Timestamp Req: 1721059686.537197 0.000000 0.000000 - BogoHeader Illegal char 0x20 in header name - HttpGarbage "GET%00" - RespProtocol HTTP/1.1 - RespStatus 400 - RespReason Bad Request - ReqAcct 535 0 535 28 0 28 - End These are on AWS EC2 instances that are behind an Application Load Balancer (ALB) that is connected to a Web Application Firewall (WAF), so in theory I should be able to figure out a rule to add to the WAF to block these. I'd just need to get more information to do so, and AWS support could probably help, but I wanted to check here first if there's any way to get further information about such requests out of Varnish. FWIW, the 0x20 is a space character, but there are also similar requests reporting 0x09 (horizontal tab) characters. Thanks, Justin -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaume.quintard at gmail.com Tue Jul 16 01:07:28 2024 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Mon, 15 Jul 2024 18:07:28 -0700 Subject: 400 Bad Request and whitespace in headers In-Reply-To: References: Message-ID: Hi Justin! What do you mean by "blocking" those requests? As you can see from the logs, thye don't even reach vcl_recv before they are thrown out, so they are technically already being rejected. Kind regards, -- Guillaume Quintard On Mon, Jul 15, 2024 at 9:44?AM Justin Lloyd wrote: > Hi all, > > > > I?m trying to figure out what the requests are that are resulting in the > following Varnish responses and how to block them: > > > > * << Request >> 39071654 > > - Begin req 39071653 rxreq > > - Timestamp Start: 1721059686.537197 0.000000 0.000000 > > - Timestamp Req: 1721059686.537197 0.000000 0.000000 > > - BogoHeader Illegal char 0x20 in header name > > - HttpGarbage "GET%00" > > - RespProtocol HTTP/1.1 > > - RespStatus 400 > > - RespReason Bad Request > > - ReqAcct 535 0 535 28 0 28 > > - End > > > > These are on AWS EC2 instances that are behind an Application Load > Balancer (ALB) that is connected to a Web Application Firewall (WAF), so in > theory I should be able to figure out a rule to add to the WAF to block > these. I?d just need to get more information to do so, and AWS support > could probably help, but I wanted to check here first if there?s any way to > get further information about such requests out of Varnish. > > > > FWIW, the 0x20 is a space character, but there are also similar requests > reporting 0x09 (horizontal tab) characters. > > > > Thanks, > > Justin > > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From justinl at arena.net Tue Jul 16 01:15:47 2024 From: justinl at arena.net (Justin Lloyd) Date: Tue, 16 Jul 2024 01:15:47 +0000 Subject: 400 Bad Request and whitespace in headers In-Reply-To: References: Message-ID: Hi Guillaume, I meant blocking them at the AWS WAF, before they even get to any of the web servers, i.e. less work for Varnish. I?d need to get the raw headers and I wasn?t having luck with that so far in the WAF CloudTrail logs, so I?ve opened up a support case about it, but I was hoping to possibly get some insight here, as well, since I don?t know whether the WAF support specialists will know much about using Varnish. Thanks, Justin From: Guillaume Quintard Sent: Monday, July 15, 2024 6:07 PM To: Justin Lloyd Cc: varnish-misc at varnish-cache.org Subject: Re: 400 Bad Request and whitespace in headers Hi Justin! What do you mean by "blocking" those requests? As you can see from the logs, thye don't even reach vcl_recv before they are thrown out, so they are technically already being rejected. Kind regards, -- Guillaume Quintard On Mon, Jul 15, 2024 at 9:44?AM Justin Lloyd > wrote: Hi all, I?m trying to figure out what the requests are that are resulting in the following Varnish responses and how to block them: * << Request >> 39071654 - Begin req 39071653 rxreq - Timestamp Start: 1721059686.537197 0.000000 0.000000 - Timestamp Req: 1721059686.537197 0.000000 0.000000 - BogoHeader Illegal char 0x20 in header name - HttpGarbage "GET%00" - RespProtocol HTTP/1.1 - RespStatus 400 - RespReason Bad Request - ReqAcct 535 0 535 28 0 28 - End These are on AWS EC2 instances that are behind an Application Load Balancer (ALB) that is connected to a Web Application Firewall (WAF), so in theory I should be able to figure out a rule to add to the WAF to block these. I?d just need to get more information to do so, and AWS support could probably help, but I wanted to check here first if there?s any way to get further information about such requests out of Varnish. FWIW, the 0x20 is a space character, but there are also similar requests reporting 0x09 (horizontal tab) characters. Thanks, Justin _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc -------------- next part -------------- An HTML attachment was scrubbed... URL: From geoff at uplex.de Tue Jul 16 08:18:33 2024 From: geoff at uplex.de (Geoff Simmons) Date: Tue, 16 Jul 2024 10:18:33 +0200 Subject: 400 Bad Request and whitespace in headers In-Reply-To: References: Message-ID: <9ba35676-6363-4059-a137-9289d9f1a8b5@uplex.de> On 7/16/24 03:15, Justin Lloyd wrote: > > I meant blocking them at the AWS WAF, before they even get to any of > the web servers, i.e. less work for Varnish. I?d need to get the raw > headers and I wasn?t having luck with that so far in the WAF CloudTrail > logs, so I?ve opened up a support case about it, but I was hoping to > possibly get some insight here, as well, since I don?t know whether the > WAF support specialists will know much about using Varnish. From what you've described, there were evidently requests with whitespace in header field names, a violation of HTTP syntax. That should be intelligible to WAF support, without any reference to Varnish at all. Why isn't a WAF rejecting requests like that by default? The invalid header names, and also your previous Varnish log excerpt showing "GET" followed by a nul byte, have the whiff of someone attempting a request smuggling attack. But it could be just a de-synchronized HTTP client. Either way, I would have expected a WAF to filter such requests, without having to ask support. And to agree with what Guillaume said, Varnish is not getting much additional work when it rejects those requests. The one in your previous example was probably taken care of in single-digit microseconds. It is true that the client connection would be spared if the request hadn't been forwarded at all. And it helps to use connections efficiently at a heavily loaded site. Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature.asc Type: application/pgp-signature Size: 840 bytes Desc: OpenPGP digital signature URL: From justinl at arena.net Tue Jul 16 13:28:21 2024 From: justinl at arena.net (Justin Lloyd) Date: Tue, 16 Jul 2024 13:28:21 +0000 Subject: 400 Bad Request and whitespace in headers In-Reply-To: <9ba35676-6363-4059-a137-9289d9f1a8b5@uplex.de> References: <9ba35676-6363-4059-a137-9289d9f1a8b5@uplex.de> Message-ID: > That should be intelligible to WAF support, without any reference to Varnish at all. I would hope so, I just wanted to cover all the bases. I do have a number of managed WAF rulesets in use, but maybe there's one that would cover this particular issue. I would think this should be checked for in all cases, but perhaps not. Hopefully Support can help explain why these would be getting through. As for the additional work, there is so much garbage that still gets through, despite my best efforts with the WAF configuration so far, the more obvious stuff I can reasonably block, the better. I appreciate the feedback! I'll respond once I get more info from Support, especially regarding the nul byte issue. Justin -----Original Message----- From: Geoff Simmons Sent: Tuesday, July 16, 2024 1:19 AM To: Justin Lloyd ; Guillaume Quintard Cc: varnish-misc at varnish-cache.org Subject: Re: 400 Bad Request and whitespace in headers On 7/16/24 03:15, Justin Lloyd wrote: > > I meant blocking them at the AWS WAF, before they even get to any of > the web servers, i.e. less work for Varnish. I?d need to get the raw > headers and I wasn?t having luck with that so far in the WAF > CloudTrail logs, so I?ve opened up a support case about it, but I was > hoping to possibly get some insight here, as well, since I don?t know > whether the WAF support specialists will know much about using Varnish. From what you've described, there were evidently requests with whitespace in header field names, a violation of HTTP syntax. That should be intelligible to WAF support, without any reference to Varnish at all. Why isn't a WAF rejecting requests like that by default? The invalid header names, and also your previous Varnish log excerpt showing "GET" followed by a nul byte, have the whiff of someone attempting a request smuggling attack. But it could be just a de-synchronized HTTP client. Either way, I would have expected a WAF to filter such requests, without having to ask support. And to agree with what Guillaume said, Varnish is not getting much additional work when it rejects those requests. The one in your previous example was probably taken care of in single-digit microseconds. It is true that the client connection would be spared if the request hadn't been forwarded at all. And it helps to use connections efficiently at a heavily loaded site. Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de From justinl at arena.net Tue Jul 16 17:01:35 2024 From: justinl at arena.net (Justin Lloyd) Date: Tue, 16 Jul 2024 17:01:35 +0000 Subject: 400 Bad Request and whitespace in headers In-Reply-To: References: <9ba35676-6363-4059-a137-9289d9f1a8b5@uplex.de> Message-ID: I got a response but it hasn't been helpful so far, basically just saying "look at ALB an WAF logs". The problem is that the varnishlog output doesn't give me enough to use to filter the WAF logs (ALB logging is currently not enabled), and even during 1 minute the number of requests is too much to filter through via the AWS console (and I don't yet have a good system for parsing the actual logs via Splunk or some other tool). Is there any way, for example, to get Varnish to display the IP and/or URL requested in the type of bad request output I showed in my original message? The command I used for that output was just varnishlog -q 'RespReason ~ "Bad Request"' Thanks, Justin -----Original Message----- From: varnish-misc On Behalf Of Justin Lloyd Sent: Tuesday, July 16, 2024 6:28 AM To: geoff at uplex.de; Guillaume Quintard Cc: varnish-misc at varnish-cache.org Subject: RE: 400 Bad Request and whitespace in headers > That should be intelligible to WAF support, without any reference to Varnish at all. I would hope so, I just wanted to cover all the bases. I do have a number of managed WAF rulesets in use, but maybe there's one that would cover this particular issue. I would think this should be checked for in all cases, but perhaps not. Hopefully Support can help explain why these would be getting through. As for the additional work, there is so much garbage that still gets through, despite my best efforts with the WAF configuration so far, the more obvious stuff I can reasonably block, the better. I appreciate the feedback! I'll respond once I get more info from Support, especially regarding the nul byte issue. Justin -----Original Message----- From: Geoff Simmons Sent: Tuesday, July 16, 2024 1:19 AM To: Justin Lloyd ; Guillaume Quintard Cc: varnish-misc at varnish-cache.org Subject: Re: 400 Bad Request and whitespace in headers On 7/16/24 03:15, Justin Lloyd wrote: > > I meant blocking them at the AWS WAF, before they even get to any of > the web servers, i.e. less work for Varnish. I'd need to get the raw > headers and I wasn't having luck with that so far in the WAF > CloudTrail logs, so I've opened up a support case about it, but I was > hoping to possibly get some insight here, as well, since I don't know > whether the WAF support specialists will know much about using Varnish. From what you've described, there were evidently requests with whitespace in header field names, a violation of HTTP syntax. That should be intelligible to WAF support, without any reference to Varnish at all. Why isn't a WAF rejecting requests like that by default? The invalid header names, and also your previous Varnish log excerpt showing "GET" followed by a nul byte, have the whiff of someone attempting a request smuggling attack. But it could be just a de-synchronized HTTP client. Either way, I would have expected a WAF to filter such requests, without having to ask support. And to agree with what Guillaume said, Varnish is not getting much additional work when it rejects those requests. The one in your previous example was probably taken care of in single-digit microseconds. It is true that the client connection would be spared if the request hadn't been forwarded at all. And it helps to use connections efficiently at a heavily loaded site. Best, Geoff -- ** * * UPLEX - Nils Goroll Systemoptimierung Scheffelstra?e 32 22301 Hamburg Tel +49 40 2880 5731 Mob +49 176 636 90917 Fax +49 40 42949753 http://uplex.de/ _______________________________________________ varnish-misc mailing list varnish-misc at varnish-cache.org https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc From guillaume.quintard at gmail.com Tue Jul 16 20:37:24 2024 From: guillaume.quintard at gmail.com (Guillaume Quintard) Date: Tue, 16 Jul 2024 13:37:24 -0700 Subject: 400 Bad Request and whitespace in headers In-Reply-To: References: <9ba35676-6363-4059-a137-9289d9f1a8b5@uplex.de> Message-ID: Hi Justin, The URL you won't get, since it comes after the bad method, so varnish bails out before that. The IP you have in the session information. Update your command to varnishlog -q 'RespReason ~ "Bad Request"' However, the reported IP will most likely be the ALB's, since it's the one sending you the request. -- Guillaume Quintard On Tue, Jul 16, 2024 at 10:01?AM Justin Lloyd wrote: > I got a response but it hasn't been helpful so far, basically just saying > "look at ALB an WAF logs". The problem is that the varnishlog output > doesn't give me enough to use to filter the WAF logs (ALB logging is > currently not enabled), and even during 1 minute the number of requests is > too much to filter through via the AWS console (and I don't yet have a good > system for parsing the actual logs via Splunk or some other tool). Is there > any way, for example, to get Varnish to display the IP and/or URL requested > in the type of bad request output I showed in my original message? The > command I used for that output was just > > varnishlog -q 'RespReason ~ "Bad Request"' > > Thanks, > Justin > > -----Original Message----- > From: varnish-misc arena.net at varnish-cache.org> On Behalf Of Justin Lloyd > Sent: Tuesday, July 16, 2024 6:28 AM > To: geoff at uplex.de; Guillaume Quintard > Cc: varnish-misc at varnish-cache.org > Subject: RE: 400 Bad Request and whitespace in headers > > > That should be intelligible to WAF support, without any reference to > Varnish at all. > > I would hope so, I just wanted to cover all the bases. > > I do have a number of managed WAF rulesets in use, but maybe there's one > that would cover this particular issue. I would think this should be > checked for in all cases, but perhaps not. Hopefully Support can help > explain why these would be getting through. > > As for the additional work, there is so much garbage that still gets > through, despite my best efforts with the WAF configuration so far, the > more obvious stuff I can reasonably block, the better. > > I appreciate the feedback! I'll respond once I get more info from Support, > especially regarding the nul byte issue. > > Justin > > -----Original Message----- > From: Geoff Simmons > Sent: Tuesday, July 16, 2024 1:19 AM > To: Justin Lloyd ; Guillaume Quintard < > guillaume.quintard at gmail.com> > Cc: varnish-misc at varnish-cache.org > Subject: Re: 400 Bad Request and whitespace in headers > > On 7/16/24 03:15, Justin Lloyd wrote: > > > > I meant blocking them at the AWS WAF, before they even get to any of > > the web servers, i.e. less work for Varnish. I'd need to get the raw > > headers and I wasn't having luck with that so far in the WAF > > CloudTrail logs, so I've opened up a support case about it, but I was > > hoping to possibly get some insight here, as well, since I don't know > > whether the WAF support specialists will know much about using Varnish. > > From what you've described, there were evidently requests with whitespace > in header field names, a violation of HTTP syntax. That should be > intelligible to WAF support, without any reference to Varnish at all. > > Why isn't a WAF rejecting requests like that by default? > > The invalid header names, and also your previous Varnish log excerpt > showing "GET" followed by a nul byte, have the whiff of someone attempting > a request smuggling attack. But it could be just a de-synchronized HTTP > client. Either way, I would have expected a WAF to filter such requests, > without having to ask support. > > And to agree with what Guillaume said, Varnish is not getting much > additional work when it rejects those requests. The one in your previous > example was probably taken care of in single-digit microseconds. It is true > that the client connection would be spared if the request hadn't been > forwarded at all. And it helps to use connections efficiently at a heavily > loaded site. > > > Best, > Geoff > -- > ** * * UPLEX - Nils Goroll Systemoptimierung > > Scheffelstra?e 32 > 22301 Hamburg > > Tel +49 40 2880 5731 > Mob +49 176 636 90917 > Fax +49 40 42949753 > > http://uplex.de/ > > _______________________________________________ > varnish-misc mailing list > varnish-misc at varnish-cache.org > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc > -------------- next part -------------- An HTML attachment was scrubbed... URL: