400 Bad Request and whitespace in headers

[Justin Lloyd]

I got a response but it hasn't been helpful so far, basically just saying "look at ALB an WAF logs". The problem is that the varnishlog output doesn't give me enough to use to filter the WAF logs (ALB logging is currently not enabled), and even during 1 minute the number of requests is too much to filter through via the AWS console (and I don't yet have a good system for parsing the actual logs via Splunk or some other tool). Is there any way, for example, to get Varnish to display the IP and/or URL requested in the type of bad request output I showed in my original message? The command I used for that output was just

varnishlog -q 'RespReason ~ "Bad Request"'

Thanks,
Justin

-----Original Message-----
From: varnish-misc <varnish-misc-bounces+justinl=arena.net at varnish-cache.org> On Behalf Of Justin Lloyd
Sent: Tuesday, July 16, 2024 6:28 AM
To: geoff at uplex.de; Guillaume Quintard <guillaume.quintard at gmail.com>
Cc: varnish-misc at varnish-cache.org
Subject: RE: 400 Bad Request and whitespace in headers

> That should be intelligible to WAF support, without any reference to Varnish at all.

I would hope so, I just wanted to cover all the bases.

I do have a number of managed WAF rulesets in use, but maybe there's one that would cover this particular issue. I would think this should be checked for in all cases, but perhaps not. Hopefully Support can help explain why these would be getting through.

As for the additional work, there is so much garbage that still gets through, despite my best efforts with the WAF configuration so far, the more obvious stuff I can reasonably block, the better.

I appreciate the feedback! I'll respond once I get more info from Support, especially regarding the nul byte issue.

Justin

-----Original Message-----
From: Geoff Simmons <geoff at uplex.de>
Sent: Tuesday, July 16, 2024 1:19 AM
To: Justin Lloyd <justinl at arena.net>; Guillaume Quintard <guillaume.quintard at gmail.com>
Cc: varnish-misc at varnish-cache.org
Subject: Re: 400 Bad Request and whitespace in headers

On 7/16/24 03:15, Justin Lloyd wrote:
>
> I meant blocking them at the AWS WAF, before they even get to any of
> the web servers, i.e. less work for Varnish. I'd need to get the raw
> headers and I wasn't having luck with that so far in the WAF
> CloudTrail logs, so I've opened up a support case about it, but I was
> hoping to possibly get some insight here, as well, since I don't know
> whether the WAF support specialists will know much about using Varnish.

 From what you've described, there were evidently requests with whitespace in header field names, a violation of HTTP syntax. That should be intelligible to WAF support, without any reference to Varnish at all.

Why isn't a WAF rejecting requests like that by default?

The invalid header names, and also your previous Varnish log excerpt showing "GET" followed by a nul byte, have the whiff of someone attempting a request smuggling attack. But it could be just a de-synchronized HTTP client. Either way, I would have expected a WAF to filter such requests, without having to ask support.

And to agree with what Guillaume said, Varnish is not getting much additional work when it rejects those requests. The one in your previous example was probably taken care of in single-digit microseconds. It is true that the client connection would be spared if the request hadn't been forwarded at all. And it helps to use connections efficiently at a heavily loaded site.


Best,
Geoff
--
** * * UPLEX - Nils Goroll Systemoptimierung

Scheffelstraße 32
22301 Hamburg

Tel +49 40 2880 5731
Mob +49 176 636 90917
Fax +49 40 42949753

http://uplex.de/

_______________________________________________
varnish-misc mailing list
varnish-misc at varnish-cache.org
https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc