Please test 2.1.5

[Mark Moseley]

On Wed, Feb 2, 2011 at 3:11 AM, Kristian Lyngstol
<kristian at varnish-software.com> wrote:
> On Tue, Feb 01, 2011 at 05:28:25PM -0800, Mark Moseley wrote:
>> Been running your 2.1.5 debs for a handful of days now. Seems to be
>> running just fine. I've had a couple of errors like below (when
>> someone was downloading a 1.5 gig zip file, another with an 890 meg
>> flv -- backend is 2 13gb SSD partitions) but they're nothing new. I
>> got them in 2.1.4 as well. Presumably just ran out of space.
>>
>> varnishd[30949]: Child (28124) Panic message: Assert error in
>> STV_alloc(), stevedore.c line 192:
>> Condition((st): != NULL) not true.
>
> Thanks for the feedback!
>
> Are you by any chance using multiple -s options? This assert will happen
> after Varnish gives up finding a stevedore for storage after trying 50
> times (IIRC), and typically happens due to either: A. Having lots of small
> files then fetching a big one - 50 small files removed still wont fit the
> big file. or B. Multiple -s options, one which is tiny and Varnish - by
> random chance - picks the small one 50 times in a row (since this is done
> concurrently, you can have many threads using the algorithm at the same
> time, so while it is normally round-robin selection, the "other" threads
> might get the big one.... It's not common, but we've seen it happen.).

I see it both in single -s instances and instances with 2 -s
filestores. Since it's almost always gigantic files (in the hundreds
of megs to gigs range), I'd imagine "A" is the more likely choice. I
only just started using 2 -s args to test out the new OCZ Revodrives
(which are insane) we got. The single instance filestores are always
>15G, and this case with the 2 filestores was 2 13G filestores, so
pretty decent-sized.

> Either way, this just goes to show that we need to deal better with files
> of vastly different sizes. In your case, I'd recommend trying to pipe() the
> big files.

Yup, we do pipe them. Based on a thread I'd started the other day
(10/28-ish) about dealing with huge files, it seems like even going to
pipe won't save the file from being put into the backend and forcing
out other files. Originally, we were doing this in vcl_fetch:

if ( beresp.http.Content-Length ~
"[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]" ) {
        return ( pass );
}

Based on Tollef's advice, we switched that to:

(in vcl_fetch)

        if ( beresp.http.Content-Length ~ "[0-9]{9,}" ) {
                set req.http.X-EN-PP = "1";
                restart;
        }

and then in vcl_recv does a check for "req.restarts > 0" and calls
pipe. This works nicely but huge files that end up getting piped still
seem to get stored (albeit very temporarily) on that first
pre-restarted, request.