[Varnish] #572: Fix for: Create worker thread failed 11 Resource temporarily unavailable (Thread Problem on Linux)

[Varnish]

#572: Fix for: Create worker thread failed 11 Resource temporarily unavailable
(Thread Problem on Linux)
-------------------------+--------------------------------------------------
 Reporter:  whocares     |       Owner:  phk  
     Type:  enhancement  |      Status:  new  
 Priority:  normal       |   Milestone:       
Component:  varnishd     |     Version:  trunk
 Severity:  normal       |    Keywords:       
-------------------------+--------------------------------------------------
 Today I tried kristian's [wiki:Performance] tips on a Linux box and ran
 into the same problem as was already mentioned in #85: Depending on what
 box I tried, I never managed to get more than 238 to 302 threads.

 The reason for that behaviour is that on Linux the number of threads that
 can be started is essentially limited by the stack size occupied by every
 thread. In a standard configuration every thread will get 8 MByte of stack
 attached to it. Thus, in conjunction with other memory related limits in a
 standard Linux environment you'll never ever get more than around 240 to
 300 threads.

 You can however modify the thread's stack size before creating the thread
 by doing something like this:

 {{{
   include "bits/local_lim.h"
   pthread_attr_t    attr;
   pthread_t         thread[1024*1024];
   size_t            size;

   /* do some calculation for optimal stack size or ... */
   size = PTHREAD_STACK_MIN;

   pthread_attr_init(&attr);
   pthread_attr_setstacksize(&attr, size);

   pthread_create(&thread[somecounter], &attr, (void *)&function_to_run,
 NULL);
   ...
 }}}

 On Linux this will set the stack size to 16384 bytes which of course is
 way too small, but allows for around 32.000 threads to be created on my
 test box.

 If you like to rum some tests for yourself I attach a sample C program to
 test different thread stack sizes on a Linux box. It's possible that the
 code will run on other platforms, too, but I didn't test that.

 Anyway, it would be nice if varnish would set it's thread's stack size to
 a reasonable value so that poor Linux users can enjoy at least some of the
 performance that is possible on FreeBSD and Solaris ;)

-- 
Ticket URL: <http://varnish.projects.linpro.no/ticket/572>
Varnish <http://varnish.projects.linpro.no/>
The Varnish HTTP Accelerator