ESI limits and worker thread deaths

[David Kitchen]

Hi all,

I've got a curious problem with using ESI. We're building a JSON web service
that is composed of smaller bits of JSON, and one of the requirements is
that the callee can specify the number of records they get back (it's
intranet facing but high load).

What we're seeing is that if a document esi:includes more than a certain
number of other files, that the worker process dies and no data is returned
to the browser.

As the "certain number" seemed to be variable using our live service, I
built a quick test to find out where the limit was and to determine whether
the limit had any special meaning (powers of 2, the obvious signs).

The test that I have that demonstrates the ESI processing failing requires:
1) Many unique files
2) Some files to esi:include the files in #1

Steps to reproduce:
1) Create the many unique files, bash helps here (run this in your web
root):
for i in $(seq 1 1000); do echo $i > $i.htm; done

2) Include files that esi:includes the many unique files:

I had 4 test files:
index125.htm
index250.htm
index500.htm
index1000.htm

They each contained lines like these, the number of lines corresponding to
the number in the index*.htm file name:
1 = <esi:include src="http://localhost/test/1.htm" />,
2 = <esi:include src="http://localhost/test/2.htm" />,
3 = <esi:include src="http://localhost/test/3.htm" />,
4 = <esi:include src="http://localhost/test/4.htm" />,
5 = <esi:include src="http://localhost/test/5.htm" />,
6 = <esi:include src="http://localhost/test/6.htm" />,
7 = <esi:include src="http://localhost/test/7.htm" />,
8 = <esi:include src="http://localhost/test/8.htm" />,
9 = <esi:include src="http://localhost/test/9.htm" />,
10 = <esi:include src="http://localhost/test/10.htm" />,

3) Access each index file, smallest to largest until it fails whilst logging
using varnishlog.

For me, 125, 250 worked, 500 and 1,000 failed.

As soon as one of those returns no data to the browser, the problem is
reproduced.



The interesting thing I found was in the logs for the failed attempts:


   12 Debug        c "tag {include src="http://localhost/test/207.htm" } 0 1
0"
   12 Debug        c "Incl " src="http://localhost/test/207.htm" ""
   12 Debug        c "tag {include src="http://localhost/test/208.htm" } 0 1
0"
    0 WorkThread   - 0x7f9d890afbf0 start
    0 CLI          - Rd vcl.load "boot" ./vcl.1P9zoqAU.so
    0 CLI          - Wr 200 36 Loaded "./vcl.1P9zoqAU.so" as "boot"
    0 CLI          - Rd vcl.use "boot"
    0 CLI          - Wr 200 0
    0 CLI          - Rd start
    0 Debug        - "Acceptor is epoll"
    0 CLI          - Wr 200 0
    0 WorkThread   - 0x7f9d04bfebf0 start

It successfully processed the 250 entries file each time it was called, but
when processing the 1,000 entries file it consistently and repeatedly showed
that the worker process died at tag 208. Presumably this disconnected the
request resulting in the symptom of "no data received" by the browser.

The question boils down to:
1) Is this behaviour a bug?
2) Is this behaviour the result of a by-design limitation? And if yes to
this, what is the limitation and is it something we can change (i.e.
ulimits)?

Varnish version is 2.1.4

Thanks in advance for any insight offered, and apologies in advance if I've
jumped in and broken any mailing list etiquette by just pushing the question
out there.

David

[Information] -- PostMaster:
This transmission is intended solely for the addressee(s) and may be confidential. If you are not the named addressee, or if the message has been addressed to you in error, you must not read, disclose, reproduce, distribute or use this transmission. 

Delivery of this message to any person other than the named addressee is not intended in any way to waive confidentiality.  If you have received this transmission in error please contact the sender or delete the message. 

Thank you.

Yell Limited, One Reading Central, Forbury Road, Reading, Berkshire, RG1 3YL. Registered in England No. 4205228

Yellow Pages Sales Limited, One Reading Central,  Forbury Road, Reading, Berkshire, RG1 3YL. Registered in England No. 1403041
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20110510/e869cc33/attachment-0003.html>