[Varnish] #771: caching large objects

[Varnish]

#771: caching large objects
----------------------+-----------------------------------------------------
 Reporter:  bcakipp   |       Owner:  phk  
     Type:  defect    |      Status:  new  
 Priority:  normal    |   Milestone:       
Component:  varnishd  |     Version:  trunk
 Severity:  normal    |    Keywords:       
----------------------+-----------------------------------------------------
 Hi,

 Working with Varnish 2.1.3 on CentOS5. I've noticed child threads dying
 with an assertion error when trying to fetch large objets. From what I can
 tell, objects with a Content-Length larger then 4GB will cause the child
 thread to die.

 For example (file is ~ 6GB).

 {{{
 $ curl http://glint1/aries/75123740001/75123740001_601962598001_ALO-
 IPA-083010.mov
 -o /dev/null -v -s
 * About to connect() to glint1 port 80
 *   Trying 192.168.104.211... connected
 * Connected to glint1 (192.168.104.211) port 80
 > GET /aries/75123740001/75123740001_601962598001_ALO-IPA-083010.mov
 HTTP/1.1
 > User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5
 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
 > Host: glint1
 > Accept: */*
 >
 * Empty reply from server
 * Connection #0 to host glint1 left intact
 * Closing connection #0
 }}}

 Syslog shows:

 {{{
 Sep  2 20:31:09 glint1 varnishd[21426]: Child (22486) died signal=6
 Sep  2 20:31:09 glint1 varnishd[21426]: Child (22486) Panic message:
 Assert error in fetch_straight(), cache_fetch.c line 64:
 Condition((uintmax_t)cl == cll) not true. thread = (cache-worker)
 ident = Linux,2.6.18-164.15.1.el5xen,x86_64,-sfile,-hcritbit,epoll
 Backtrace:   0x422726: /usr/sbin/varnishd [0x422726]   0x41a8f8:
 /usr/sbin/varnishd(FetchBody+0x4a8) [0x41a8f8]   0x412960:
 /usr/sbin/varnishd [0x412960]   0x413ed5:
 /usr/sbin/varnishd(CNT_Session+0x345) [0x413ed5]   0x424b78:
 /usr/sbin/varnishd [0x424b78]   0x423e5d: /usr/sbin/varnishd
 [0x423e5d]   0x3c9de06617: /lib64/libpthread.so.0 [0x3c9de06617]
 0x3c9d6d3c2d: /lib64/libc.so.6(clone+0x6d) [0x3c9d6d3c2d] sp =
 0x2abe71908008 {   fd = 11, id = 11, xid = 674181670,   client =
 192.168.103.71:48903,   step = STP_FETCH,   handling = deliver,
 err_code = 200, err_reason = (null),   restarts = 0, esis = 0   ws =
 0x2abe71908078 {      id = "sess",     {s,f,r,e} =
 {0x2abe71908cd0,+240,(nil),+65536},   },   http[req] = {     ws =
 0x2abe71908078[sess]       "
 }}}

 Looking at that line in cache_fetch.c:

 {{{
        uintmax_t cll;
        unsigned cl, sl;
        struct storage *st;

        cll = strtoumax(b, NULL, 0);
        if (cll == 0)
                return (0);

        cl = (unsigned)cll;
        assert((uintmax_t)cl == cll); /* Protect against bogusly large
 values */
 }}}

 My reading of that says that anything larger then an unsigned int (~4GB)
 will cause the client thread to die.

 I understand that it would be possible to just pipe these large files, but
 what I would really like to do is cache them in varnish. Is this possible?

-- 
Ticket URL: <http://varnish-cache.org/ticket/771>
Varnish <http://varnish-cache.org/>
The Varnish HTTP Accelerator