Question about threads

[duja at torlen.net]

I recently made a loadtest against through varnish. 

First I received a very high response time and found out that varnish was maxing the maximum nr of threads. 

I updated thread_min = 5 and thread_max = 300 and recevied much better resp. times.

Then I increased the nr of concurrent users and made another loadtest. The strange thing here was that I received high resp. times but the threads stopped at 238.

The "N worker threads not created" increased rapidly.

I increased the threads again and changed listen_depth to 2048.

Here is all the numbers:
         238         0.00         0.22 N worker threads created
        1318         4.98         1.21 N worker threads not created

    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"
    0 Debug        - "Create worker thread failed 12 Cannot allocate memory"

default_ttl                120 [seconds]
thread_pools               2 [pools] <<<<
thread_pool_max            400 [threads] <<<<
thread_pool_min            10 [threads] <<<<
thread_pool_timeout        120 [seconds]
thread_pool_purge_delay    1000 [milliseconds]
thread_pool_add_threshold  2 [requests]
thread_pool_add_delay      10 [milliseconds]
thread_pool_fail_delay     200 [milliseconds]
overflow_max               100 [%]
rush_exponent              3 [requests per request]
sess_workspace             8192 [bytes]
obj_workspace              8192 [bytes]
sess_timeout               5 [seconds]
pipe_timeout               60 [seconds]
send_timeout               600 [seconds]
auto_restart               on [bool]
fetch_chunksize            128 [kilobytes]
vcl_trace                  off [bool]
listen_address             0.0.0.0:80
listen_depth               2048 [connections] <<<<
srcaddr_hash               1049 [buckets]
srcaddr_ttl                30 [seconds]
backend_http11             off [bool]
client_http11              off [bool]
cli_timeout                5 [seconds]
ping_interval              3 [seconds]
lru_interval               2 [seconds]
cc_command                 exec cc -fpic -shared -Wl,-x -o %o %s
max_restarts               4 [restarts]
max_esi_includes           5 [restarts]
cache_vbe_conns            off [bool]
connect_timeout            400 [ms]
cli_buffer                 8192 [bytes]
diag_bitmap                0x0 [bitmap]

Why do I get "Create worker thread failed 12 Cannot allocate memory" when I had 1900MB free RAM and 65GB free Disk on the server? Any ideas?

If "N worker threads not created" is increasing, is that a bad sign?

Thanks
Duja