Varnish showing hex characters from Jakarta-Tomcat connector protocol

[David Brown]

To All,

Well I think I have varnish behaving the way that I expected when backend
content has both Content-Length and Transfer-Encoding headers. (I know
having both is a violation of HTTP 1.1 protocol but I have little control
over the backend)

I changed the order of logic in cache_fetch.c for determining how the body
should be handled. I placed the Transfer-Encoding is statements before the
Content-Length statements. That way if backend content has both headers it
will used chunked instead of length and the chunked characters don't show up
in the browser.

>From new cache_fetch.c

FetchReqBody(struct sess *sp)
{
...
        /* Moved this if statement up */
if (http_GetHdr(sp->http, H_Transfer_Encoding, NULL)) {
/* XXX: Handle chunked encoding. */
WSL(sp->wrk, SLT_Debug, sp->fd, "Transfer-Encoding in request");
return (1);
}
if (http_GetHdr(sp->http, H_Content_Length, &ptr)) {

content_length = strtoul(ptr, &endp, 10);
/* XXX should check result of conversion */
while (content_length) {
if (content_length > sizeof buf)
rdcnt = sizeof buf;
else
rdcnt = content_length;
rdcnt = HTC_Read(sp->htc, buf, rdcnt);
if (rdcnt <= 0)
return (1);
content_length -= rdcnt;
if (!sp->sendbody)
continue;
(void)WRW_Write(sp->wrk, buf, rdcnt); /* XXX: stats ? */
if (WRW_Flush(sp->wrk))
return (2);
}
...

AND....

FetchBody(struct sess *sp)
{
...
        /* moved this pair of if statements up */
} else if (http_HdrIs(hp, H_Transfer_Encoding, "chunked")) {
sp->wrk->stats.fetch_chunked++;
cls = fetch_chunked(sp, sp->wrk->htc);
mklen = 1;
} else if (http_GetHdr(hp, H_Transfer_Encoding, &b)) {
sp->wrk->stats.fetch_bad++;
/* XXX: AUGH! */
WSL(sp->wrk, SLT_Debug, vc->fd, "Invalid Transfer-Encoding");
VBE_ClosedFd(sp);
return (__LINE__);
} else if (http_GetHdr(hp, H_Content_Length, &b)) {
sp->wrk->stats.fetch_length++;
cls = fetch_straight(sp, sp->wrk->htc, b);
mklen = 1;
} else if (http_HdrIs(hp, H_Connection, "keep-alive")) {
....


On Wed, Jun 30, 2010 at 9:08 AM, David Brown <captkiddo at gmail.com> wrote:

> You were right, it was wrong. The change I made to the VCL fixed the one
> problem but screwed up the rest of the site so I had to back it out.
>
> Attached is a varnish log of the activity. I have some tcpdumps at various
> points too if you need them.
>
> I still think it's connected to tomcat and specifically the JK connector.
> It seems the JK connector has an 8K limit (AJP 1.3) and breaks up large
> files in the transmission between tomcat and the webserver. The characters
> may be (?) part of JK's breakup of the file. My reasoning is because I can
> take the same file and place it directly on the web server and no problem,
> only when it comes from tomcat/jk do we have these issues. ( I can't
> convince the programmers to take static content out of their java apps and
> put it on the server )
>
>
>
> On Tue, Jun 29, 2010 at 5:22 PM, Poul-Henning Kamp <phk at phk.freebsd.dk>wrote:
>
>> In message <AANLkTikRDbvceKr14-PKP8RuIC9Djyab0kLiehPG2Ea4 at mail.gmail.com>,
>> Davi
>> d Brown writes:
>>
>> >I checked and it seems the Transfer-Encoding header is being stripped by
>> >Varnish. I can see it when I go directly to the web server but not when I
>> >view the file through Varnish.
>> >
>> >I worked around the problem by adding this to my VCL
>>
>> That sounds positively wrong.
>>
>> Is it possible that you can capture a varnishlog of one of these
>> transaction and send to me ?
>>
>>
>> --
>> Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
>> phk at FreeBSD.ORG         | TCP/IP since RFC 956
>> FreeBSD committer       | BSD since 4.3-tahoe
>> Never attribute to malice what can adequately be explained by
>> incompetence.
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20100706/b6505726/attachment-0003.html>