r12 - in trunk/varnish-doc: . en en/varnish-specification

des at projects.linpro.no des at projects.linpro.no
Wed Feb 22 17:18:23 CET 2006


Author: des
Date: 2006-02-22 17:18:23 +0100 (Wed, 22 Feb 2006)
New Revision: 12

Added:
   trunk/varnish-doc/en/
   trunk/varnish-doc/en/varnish-specification/
   trunk/varnish-doc/en/varnish-specification/article.xml
Log:
Draft specification in DocBook format.

Added: trunk/varnish-doc/en/varnish-specification/article.xml
===================================================================
--- trunk/varnish-doc/en/varnish-specification/article.xml	2006-02-22 16:18:07 UTC (rev 11)
+++ trunk/varnish-doc/en/varnish-specification/article.xml	2006-02-22 16:18:23 UTC (rev 12)
@@ -0,0 +1,636 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!DOCTYPE article
+ PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
+ "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
+<article lang="en">
+  <articleinfo>
+    <releaseinfo role="cvs">$Id$</releaseinfo>
+    <title>Varnish HTTP Accelerator Draft Specification</title>
+  </articleinfo>
+
+  <section>
+    <title>Introduction</title>
+
+    <section>
+      <title>Overview</title>
+
+      <para>Varnish is a high-performance HTTP accelerator.</para>
+
+      <para>XXX</para>
+    </section>
+
+    <section>
+      <title>Terminology</title>
+
+      <para>The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”,
+      “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and
+      “OPTIONAL” in this document are to be interpreted as described
+      in IETF RFC 2119.</para>
+
+      <para>XXX at this time, the above is incorrect because we
+      started out using MoSCoW prioritisation before realising it was
+      inadequate for the task.</para>
+    </section>
+
+    <section>
+      <title>To do</title>
+
+      <para>Use consistent terminology throughout the document (see
+      previous section)</para>
+
+      <para>Clarify the following terms: configuration facility;
+      logging and statistics facility (split in two?); monitoring and
+      tuning facility (split in two? does this overlap with
+      configuration facility?)</para>
+    </section>
+  </section>
+
+  <section>
+    <title>General requirements</title>
+
+    <section>
+      <title>License</title>
+
+      <para>XXX two-clause BSD license</para>
+    </section>
+
+    <section>
+      <title>Version control</title>
+
+      <para>All source code and documentation must be kept under
+      version control in a publicly accessible repository.</para>
+    </section>
+
+    <section>
+      <title>Software platform</title>
+
+      <para>Varnish must be fully functional on the platforms listed
+      in this section.</para>
+
+      <para>Varnish should also be fully functional on other
+      POSIX-derived platforms, insofar as this does not require
+      unreasonable effort.</para>
+
+      <section>
+	<title>FreeBSD</title>
+
+	<para>Varnish must be fully functional on FreeBSD 6.0 or later
+	officially supported releases, including security and errata
+	branches.</para>
+
+	<para>The reference platform for FreeBSD compatibility is
+	FreeBSD 6.1-RELEASE.</para>
+      </section>
+
+      <section>
+	<title>GNU/Linux</title>
+
+	<para>Varnish must be fully functional on GNU/Linux systems
+	with Linux 2.6.12 or later and GNU libc 2.3.2 or later.</para>
+
+	<para>The reference platform for GNU/Linux compatibility is
+	Ubuntu 5.10 “Breezy Badger”.</para>
+      </section>
+    </section>
+
+    <section>
+      <title>Hardware platform</title>
+
+      <para>Varnish must be fully functional on both 32-bit and 64-bit
+      Intel-compatible hardware, but may place different limits on
+      operating parameters (such as cache size) depending on the
+      platform and the amount of physical and / or virtual memory
+      available.  Varnish must support and take advantage of multiple
+      CPUs or CPU cores if present.</para>
+
+      <para>The reference hardware platform is a dual-CPU AMD Opteron
+      system with 4 GB of RAM divided evenly between the CPUs.</para>
+    </section>
+
+    <section>
+      <title>Language</title>
+
+      <para>Varnish must be implemented in C with as few compiler- and
+      platform-specific extensions as possible.</para>
+    </section>
+
+    <section>
+      <title>Compiler and toolchain</title>
+
+      <para>Varnish must be compilable with the GNU C compiler (GCC)
+      3.3.5 or later and toolchain (binutils) 2.15 or later.</para>
+
+      <para>Alternative compilers and toolchains should be supported
+      insofar as this does not require unreasonable effort.</para>
+    </section>
+
+    <section>
+      <title>Compile-time configuration</title>
+
+      <para>Varnish must use the GNU Autotools for compile-time
+      configuration.  The Makefile templates must be written to work
+      with any POSIX-compliant make(1) utility.</para>
+    </section>
+
+    <section>
+      <title>Third-party dependencies</title>
+
+      <para>Varnish must not depend on any third-party packages other
+      than the compiler, toolchain and configuration tools.</para>
+    </section>
+
+    <section>
+      <title>Incidental tools</title>
+
+      <para>Varnish may be accompanied by incidental tools for
+      purposes such as creating or editing configuration files,
+      warming up the cache, manipulating or generating source code,
+      etc.  Insofar as these tools are not required at compilation or
+      installation time nor for the daily operation of Varnish, they
+      may be written in Perl 5.8 or later, and depend on third-party
+      Perl modules available from CPAN.</para>
+    </section>
+
+    <section>
+      <title>Coding standards</title>
+
+      <para>All C source code must conform to the FreeBSD style(9)
+      coding standard.</para>
+    </section>
+
+    <section>
+      <title>Documentation</title>
+
+      <para>Varnish must be accompanied by complete internal and
+      external documentation.</para>
+
+      <para>All documentation must be in English.</para>
+
+      <para>All documentation must be made available online in HTML
+      form, and may be made available online in additional formats
+      such as PDF.</para>
+
+      <section>
+	<title>Internal documentation</title>
+
+	<para>The internal documentation consists of:</para>
+	<orderedlist>
+	  <listitem>
+	    <para>Code comments.</para>
+	  </listitem>
+	  <listitem>
+	    <para>Manual pages describing Varnish internals.</para>
+	  </listitem>
+	  <listitem>
+	    <para>Version control history.</para>
+	  </listitem>
+	  <listitem>
+	    <para>Requirements and specification in DocBook XML
+	    format.</para>
+	  </listitem>
+	  <listitem>
+	    <para>System architecture in DocBook XML format.</para>
+	  </listitem>
+	  <listitem>
+	    <para>Developer guidelines and other incidental
+	    documentation either in the project Wiki or in DocBook XML
+	    format.</para>
+	  </listitem>
+	</orderedlist>
+      </section>
+
+      <section>
+	<title>External documentation</title>
+
+	<para>The external documentation consists of:</para>
+	<orderedlist>
+	  <listitem>
+	    <para>Manual pages for all daemons and command-line
+	    tools.</para>
+	  </listitem>
+	  <listitem>
+	    <para>Installation instructions in DocBook XML
+	    format.</para>
+	  </listitem>
+	  <listitem>
+	    <para>Administrator's handbook in DocBook XML
+	    format.</para>
+	  </listitem>
+	  <listitem>
+	    <para>Sample configuration files.</para>
+	  </listitem>
+	</orderedlist>
+      </section>
+    </section>
+  </section>
+
+  <section>
+    <title>Functional requirements</title>
+
+    <section>
+      <title>Functional description</title>
+
+      <para>Varnish accepts HTTP requests from clients and satisfy
+      them with documents retrieved from its cache (disk- and / or
+      memory-based).  Documents which are not present in the cache
+      must be retrieved from a set of preconfigured content servers.
+      Requests for documents from other servers than the preconfigured
+      content servers are ignored.</para>
+    </section>
+
+    <section>
+      <title>Protocol support</title>
+
+      <section>
+	<title>HTTP</title>
+
+	<para>Varnish must be able to accept HTTP/1.0 and HTTP/1.1
+	requests from both IPv4 and IPv6 clients.</para>
+
+	<para>Varnish must, in general terms, handle these requests in
+	conformance with IETF RFC 2616.</para>
+
+	<para>Varnish may handle HTTP/0.9 requests in any way it sees
+	fit, including but not limited to returning a 400 Bad Request
+	response or simply closing the connection.</para>
+
+	<para>Varnish must use HTTP/1.1 in its communications with the
+	content servers.</para>
+
+	<para>Varnish may deviate from IETF RFC 2616 when this is
+	necessary for interoperability with non-conforming clients or
+	content servers.</para>
+
+	<para>Varnish may deviate from IETF RFC 2616 in cases where
+	doing so provides a considerable performance advantage without
+	causing significant harm to interoperability.  Any such
+	deviation must be documented.</para>
+
+	<para>In its communications with clients, Varnish must
+	interpret IETF RFC 2616 as if it were an origin server.  In
+	its communications with content servers, Varnish must
+	interpret IETF RFC 2616 as if it were a cache.</para>
+      </section>
+
+      <section>
+	<title>ICP</title>
+
+	<para>Varnish may support ICP for inter-cache coordination.
+	ICP support may be a compile-time option.</para>
+      </section>
+
+      <section>
+	<title>HTCP</title>
+
+	<para>Varnish may support HTCP for inter-cache coordination.
+	HTCP support may be a compile-time option.</para>
+      </section>
+    </section>
+
+    <section>
+      <title>Content manipulation</title>
+
+      <para>Varnish won't implement content manipulation at this
+      time.</para>
+
+      <para>Varnish should be designed in such a manner as to make it
+      possible to implement various kinds of content manipulation
+      (such as ESI) at a future date.</para>
+
+      <para>XXX ICAP may be worth looking into (but is probably a
+      performance killer)</para>
+    </section>
+
+    <section>
+      <title>Caching</title>
+
+      <para>Varnish must maintain a local cache of the documets
+      present on the content server.</para>
+
+      <section>
+	<title>Cached set</title>
+
+	<para>If the amount of memory and / or disk available to
+	Varnish is not sufficient to cache the entire document set,
+	Varnish must attempt to identify a subset to cache which
+	minimizes load on the content servers.</para>
+
+	<para>Varnish should offer multiple alternative cache control
+	algorithms.  At the very least, the LRU (least-recently-used)
+	and WLRU (LRU weighted by document size) algorithms should be
+	implemented.</para>
+      </section>
+
+      <section>
+	<title>Expiry</title>
+
+	<para>If a cached document has an expiry time associated with
+	it, and that time has not yet been reached, Varnish may serve
+	the document from cache without contacting the content
+	server.</para>
+      </section>
+
+      <section>
+	<title>Non-cacheable content</title>
+
+	<para>XXX</para>
+      </section>
+    </section>
+
+    <section>
+      <title>Management</title>
+
+      <section>
+	<title>Configuration</title>
+
+	<para>XXX</para>
+      </section>
+
+      <section>
+	<title>Logging and statistics</title>
+
+	<para>A separate application is responsible for collecting,
+	collating and analyzing log data which Varnish makes available
+	in circular shared memory buffers.</para>
+
+	<para>Varnish must provide the data necessary to compute
+	lifetime totals and sliding averages for the following:</para>
+	<orderedlist>
+	  <listitem>
+	    <para>Total size of documents served to clients</para>
+	  </listitem>
+	  <listitem>
+	    <para>Total size of data transmitted to clients, including
+	    headers, error messages, etc.</para>
+	  </listitem>
+	  <listitem>
+	    <para>Total size of data received from clients, including
+	    request headers etc.</para>
+	  </listitem>
+	  <listitem>
+	    <para>Number of client connections received</para>
+	  </listitem>
+	  <listitem>
+	    <para>Number of client requests served</para>
+	  </listitem>
+	  <listitem>
+	    <para>Client requests broken down by result code</para>
+	  </listitem>
+	  <listitem>
+	    <para>Total size of documents retrieved from content
+	    servers</para>
+	  </listitem>
+	  <listitem>
+	    <para>Total size of data received from content servers,
+	    including headers, error messages, etc.</para>
+	  </listitem>
+	  <listitem>
+	    <para>Total size of data sent to content servers,
+	    including request headers etc.</para>
+	  </listitem>
+	  <listitem>
+	    <para>Number of content server connections
+	    initiated</para>
+	  </listitem>
+	  <listitem>
+	    <para>Number of content server requests sent</para>
+	  </listitem>
+	  <listitem>
+	    <para>Content server requests broken down by result
+	    code</para>
+	  </listitem>
+	  <listitem>
+	    <para>Cache effectiveness as the ratio of bytes served to
+	    clients to bytes requested from content servers</para>
+	  </listitem>
+	  <listitem>
+	    <para>Cache effectiveness as the ratio of client requests
+	    to content server requests</para>
+	  </listitem>
+	  <listitem>
+	    <para>Number of active server processes / threads, broken
+	    down by process / thread type</para>
+	  </listitem>
+	  <listitem>
+	    <para>XXX length of request queues</para>
+	  </listitem>
+	</orderedlist>
+
+	<para>In addition, Varnish must provide the data necessary to
+	compute the average, median and distribution for the
+	following:</para>
+	<orderedlist>
+	  <listitem>
+	    <para>Size of documents served, per unique document</para>
+	  </listitem>
+	  <listitem>
+	    <para>Size of documents served, per request</para>
+	  </listitem>
+	  <listitem>
+	    <para>Client connection duration</para>
+	  </listitem>
+	  <listitem>
+	    <para>Requests per client connection</para>
+	  </listitem>
+	  <listitem>
+	    <para>Client request completion time</para>
+	  </listitem>
+	  <listitem>
+	    <para>Content server connection duration</para>
+	  </listitem>
+	  <listitem>
+	    <para>Requests per content server connection</para>
+	  </listitem>
+	  <listitem>
+	    <para>Content server request completion time</para>
+	  </listitem>
+	  <listitem>
+	    <para>XXX time spent in request queues</para>
+	  </listitem>
+	</orderedlist>
+      </section>
+    </section>
+
+    <section>
+      <title>Run-time monitoring and tuning</title>
+
+      <para>Varnish must provide low-level monitoring and tuning
+      facilities. A separate application is responsible for providing
+      a user-friendly interface to these facilities.</para>
+
+      <para>The following monitoring operations must be
+      supported:</para>
+
+      <orderedlist>
+	<listitem>
+	  <para>Cache status of individual documents</para>
+	</listitem>
+	<listitem>
+	  <para>Cache status of documents matching a glob or regular
+	  expression</para>
+	</listitem>
+	<listitem>
+	  <para>Access statistics of individual documents</para>
+	</listitem>
+	<listitem>
+	  <para>Access statistics of documents matching a glob or
+	  regular expression</para>
+	</listitem>
+	<listitem>
+	  <para>XXX</para>
+	</listitem>
+      </orderedlist>
+
+      <para>The following tuning operations must be supported:</para>
+
+      <orderedlist>
+	<listitem>
+	  <para>Forced invalidation of individual documents</para>
+	</listitem>
+	<listitem>
+	  <para>Forced invalidation of documents matching a glob or regular expression</para>
+	</listitem>
+	<listitem>
+	  <para>XXX</para>
+	</listitem>
+      </orderedlist>
+    </section>
+    <section>
+      <title>Clustering</title>
+
+      <para>Clustering is defined in this context as a situation where
+      multiple servers are set up to run Varnish with the same
+      configuration, serving data from the same content servers to the
+      same set of clients.</para>
+
+      <section>
+	<title>Configuration</title>
+
+	<para>When multiple Varnish servers act together as a cluster,
+	the configuration facility is responsible for ensuring that
+	all nodes share the same configuration and that configuration
+	changes are applied to all nodes in a timely fashion.</para>
+      </section>
+
+      <section>
+	<title>Logging and statistics</title>
+
+	<para>When multiple Varnish servers act together as a cluster,
+	the logging and statistics facilities must base its reports on
+	aggregated data as if the cluster were a single Varnish
+	server.</para>
+
+	<para>Per-node data may optionally be made available in
+	addition to aggregated data.</para>
+      </section>
+
+      <section>
+	<title>Run-time monitoring and tuning</title>
+
+	<para>When multiple Varnish servers act together as a cluster,
+	the run-time monitoring and tuning facilities must propagate
+	invalidation requests and other administrative commands to all
+	servers in the cluster.</para>
+      </section>
+    </section>
+  </section>
+
+  <section>
+    <title>Application structure</title>
+
+    <section>
+      <title>Components</title>
+
+      <para>This section lists the major components in Varnish.</para>
+
+      <section>
+	<title>Listener</title>
+
+	<para>The Listener monitors the listening socket and accepts
+	incoming client connections.  Once the connection is
+	established, it is passed to the Accepter.</para>
+
+	<para>The Listener should take advantage of accept filters or
+	similar technologies on systems where they are
+	available.</para>
+      </section>
+
+      <section>
+	<title>Accepter</title>
+
+	<para>The Accepter reads an HTTP request from a client
+	connection.  It parses the request line and header only to the
+	extent necessary to establish well-formedness and determine
+	the requested URL.</para>
+
+	<para>The Accepter then queries the Keeper about the status of
+	the requested document (identified by its full URL).  If the
+	document is present and valid in the cache, the request is
+	passed directly to a Sender.  Otherwise, it is passed to a
+	Retriever queue.</para>
+      </section>
+
+      <section>
+	<title>Keeper</title>
+
+	<para>The Keeper manages the document cache. XXX</para>
+      </section>
+
+      <section>
+	<title>Sender</title>
+
+	<para>The Sender transfers the contents of the requested
+	document to the client.  It examines the HTTP request header
+	to determine the correct way in which to do this – Range,
+	If-Modified-Since, Content-Encoding and other options may
+	affect the type and amount of data transferred.</para>
+
+	<para>There may be multiple concurrent Sender threads.</para>
+      </section>
+
+      <section>
+	<title>Retriever</title>
+
+	<para>The Retriever is responsible for retrieving documents
+	from the content servers.  It is triggered either by an
+	Accepter trying to satisfy a request for a document which is
+	not in the cache, or by the Janitor when a “hot” document is
+	nearing expiry.  Either way, there may be a queue of requests
+	waiting for the document to arrive; when it does, the
+	Retriever passes those requests to a Sender.</para>
+
+	<para>There may be multiple concurrent Retriever
+	threads.</para>
+      </section>
+
+      <section>
+	<title>Janitor</title>
+
+	<para>The Janitor keeps track of the expiry time of cached
+	documents and attempts to retrieve fresh copies of documents
+	which are soon to expire.</para>
+      </section>
+
+      <section>
+	<title>Logger</title>
+
+	<para>The Logger keeps logs of various types of events in
+	circular shared-memory buffers.  These buffers must be managed
+	using either POSIX shared memory primitives or file-backed
+	mmap(2).</para>
+
+	<para>It is the responsibility of each module to feed relevant
+	log data to the Logger.</para>
+      </section>
+    </section>
+  </section>
+
+  <bibliography>
+    <title>References</title>
+
+    <biblioentry id="RFC2119"/>
+    <biblioentry id="RFC2186"/>
+    <biblioentry id="RFC2616"/>
+    <biblioentry id="RFC2756"/>
+  </bibliography>
+</article>


Property changes on: trunk/varnish-doc/en/varnish-specification/article.xml
___________________________________________________________________
Name: svn:mime-type
   + text/xml
Name: svn:charset
   + utf-8




More information about the varnish-commit mailing list