Question about threads

[Rafael Umann]

On Wed, 18 Jun 2008 13:24:52 -0700
"Michael S. Fischer" <michael at dynamine.net> wrote:

> On Wed, Jun 18, 2008 at 4:51 AM, Rafael Umann
> <rafael.umann at terra.com.br> wrote:
> 
> >
> > If it is a 32bits system, probably the problem is that your stack
> > size is 10Mb. So 238 * 10mb = ~2gb
> >
> > I decreased my stack size to 512Kb. Using 1gb storage files i can
> > now open almost 1900 threads using all the 2gb that 32bits can
> > alloc. So, my Varnish makes 20000 hits/second serving clients!
> 
> 
> What is your request:connection ratio?
> 
> Best regards,
> 
> --Michael
> 

Unfortunately now i dont have servers doing 20000 hits/second, and
thats why i dont have stats for you. I have 6 servers runing this
service now, each one doing 5500 hits/sec with 10% CPU usage, and this
infrastructure scales to 20000 hits/sec each server. For test purpose
only i let just 2 servers with all the traffic, so i saw this limit i
told you.

In this 20000 hits/sec im seeing two bottle necks:

1. The 32bits arch (cant open threads and the storage file is too
small), so im moving into 64bits.
2. The cpu usage of the "listener process" with 20000 hits/sec is
getting to 100% in one CPU (we also made a patch to improve CPU usage,
and it is avaliable in this last trunk version.. without this patch
varnish was doing 8000 hits/sec). But anyway, i spoke to Poul, and he
told me that the "listener process" will be changed soon (i really hope
so, because i would love to use more than 30% of my cpu!).

Anyway, varnish is doing a great job for me!!!

Some stats taken now with 5500 hits/sec:

# netstat -nap|grep :80 |grep SYN_REC |wc -l
166

# netstat -nap|grep :80 |grep ESTAB |wc -l
17139

# netstat -nap|grep :80 |grep FIN |wc -l
1454

# netstat -nap|grep :80 |grep TIME_W |wc -l
7766


# varnishstat

Hitrate ratio:       10       13       13
Hitrate avg:     0.9990   0.9990   0.9990

    36189916       476.66       250.71 Client connections accepted
   404804204      5494.13      2804.30 Client requests received
   391586079      5335.24      2712.74 Cache hits
    10975643       157.89        76.03 Cache hits for pass
     1125709         1.00         7.80 Cache misses
    12101360       158.89        83.83 Backend connections success
           0         0.00         0.00 Backend connections failures
           0         0.00         0.00 Backend connections reuses
    11139673       141.90        77.17 Backend connections recycles
         350        -1.00         0.00 Backend connections unused
       13447          .            .   N struct srcaddr
       11353          .            .   N active struct srcaddr
       30351          .            .   N struct sess_mem
       17391          .            .   N struct sess
       47069          .            .   N struct object
       47080          .            .   N struct objecthead
       97813          .            .   N struct smf
        4185          .            .   N small free smf
           5          .            .   N large free smf
           0          .            .   N struct vbe_conn
         556          .            .   N worker threads
         556         0.00         0.00 N worker threads created
           0         0.00         0.00 N worker threads not created
           0         0.00         0.00 N worker threads limited
           0         0.00         0.00 N queued work requests
         556         0.00         0.00 N overflowed work requests
           0         0.00         0.00 N dropped work requests
           0          .            .   N expired objects
     1079159          .            .   N LRU nuked objects
           0          .            .   N LRU saved objects
           0          .            .   N objects on deathrow
          48         0.00         0.00 HTTP header overflows
           0         0.00         0.00 Objects sent with sendfile
   135051222      1812.72       935.58 Objects sent with write
    36189893       494.65       250.71 Total Sessions
   404810596      5491.13      2804.35 Total Requests
           0         0.00         0.00 Total pipe
           0         0.00         0.00 Total pass
    12101371       158.89        83.83 Total fetch
 93389657948   1269372.98    646962.32 Total header bytes
418107385660   5962840.69   2896463.38 Total body bytes
     5412491        89.94        37.50 Session Closed
           0         0.00         0.00 Session Pipeline
     1623343         0.00        11.25 Session Read Ahead
   398293975      5409.19      2759.20 Session herd
 15780035398    212996.82    109317.12 SHM records
   516154306      6307.55      3575.69 SHM writes
     2661815        33.98        18.44 SHM MTX contention
    23279696       242.83       161.27 allocator requests
       93623          .            .   outstanding allocations
   910299136          .            .   bytes allocated


[]s,
-- 
Rafael Umann <rafael.umann at corp.terra.com.br>
Suporte Engenharia 1
Terra Networks Brasil S/A 
Tel: +55 (51) 3284-4344