[Varnish] #492: cache_waiter_epoll.c / cache_acceptor_epoll.c entire rewritten for Linux boxes better performance

[Varnish]

#492: cache_waiter_epoll.c / cache_acceptor_epoll.c entire rewritten for Linux
boxes better performance
-------------------------+--------------------------------------------------
 Reporter:  stockrt      |       Owner:  phk                                                                
     Type:  enhancement  |      Status:  new                                                                
 Priority:  high         |   Milestone:                                                                     
Component:  varnishd     |     Version:  trunk                                                              
 Severity:  major        |    Keywords:  cache_acceptor_epoll.c cache_waiter_epoll.c performance linux epoll
-------------------------+--------------------------------------------------
 Hi.

 I have made major changes in cache_acceptor_epoll.c / cache_waiter_epoll.c
 in order for those to look more alike kqueue's version. These changes can
 make a Linux box achieve a much better performance than one running the
 unpatched (original) Varnish 2.0.4 or even trunk r4034.

 The previous versions of cache_waiter_epoll.c have this police of
 "checking for timeouts only each 60 seconds".
 This is problematic because if you want to set a smaller amount of idle
 timeout it will not be respected most of the time. Also the default (5s)
 idle timeout (sess_timeout) isn't respected using this approach. This
 scenario can cause your server to run out of Varnish's open file
 descriptors limit (hard limited in shmlog.c, at line 85, to 65536 or
 0x10000 in hexa).
 In order to not running into this limitation I have changed the entire
 cache_waiter_epoll.c to take care of sess_timeout(ed) sockets more
 frequently, each 100ms (just like cache_waiter_kqueue.c), and in a smarter
 way, not doing a FOR_EACH in the session list each time we check for
 timeouts, but checking for timeouts only in the older entries, what should
 be ok, since the newer sessions are more likely to do not expire so early
 (once again, just like cache_waiter_kqueue.c).

 What should be different (and better) with this patch:
 - Sustained performance. We do not have the cost of doing a FOR_EACH
 sessions
   each time we check for timeouted sockets. This new approach is cheaper
 and
   do not impact the performance. In older versions this routine of
 checking
   all sessions could degrade the performance badly. I have seen
 degradation
   in order of reducing the capacity of handling new requests by 50% during
   that 1 or 2 seconds the older versions spent each 60s. This one do not
   suffer from that any more.
 - Respecting sess_timeout. Since the previous versions only checks for
 idle
   timeout each 60 seconds, the configured sess_timeout could not be
 respected
   in most cases. This version checks sess_timeout(ed) connections each
 100ms,
   in a smarter and cheaper way, ensuring to respect the desired idle
 timeout
   and keeping the open sockets number as low as possible.
 - Not running out of open fds limitation (SO's or Varnish's, wich came
   first) so soon.
 - Handles more than just one session event per iteration. This version can
   handle up to 100 session activities per event loop for each epoll_wait()
   syscall.
 - Uses Edge-Trigger (EPOLLET) for internal (pipe) session passing.
 - Uses one-shot (EPOLLONESHOT) for handling session activities.

 Regards,
 Rogério Schneider

-- 
Ticket URL: <http://varnish.projects.linpro.no/ticket/492>
Varnish <http://varnish.projects.linpro.no/>
The Varnish HTTP Accelerator