Is anyone using ESI with a lot of traffic?
Artur Bergman
sky at crucially.net
Mon Mar 2 22:45:10 CET 2009
"Right now, HAProxy only supports the first mode (HTTP close) if it
needs to
process the request. This means that for each request, there will be
one TCP
connection. If keep-alive or pipelining are required, HAProxy will still
support them, but will only see the first request and the first
response of
each transaction. While this is generally problematic with regards to
logs,
content switching or filtering, it most often causes no problem for
persistence
with cookie insertion."
Really not fully supporting keep-alive! More like varnish pipe mode
Artur
On Mar 2, 2009, at 1:33 PM, Cloude Porteus wrote:
> I believe TCP Keep-alive has been supported in HAProxy since version
> 1.2. We've been using 1.3.x for at least a year.
>
> -cloude
>
> On Mon, Mar 2, 2009 at 1:10 PM, Artur Bergman <sky at crucially.net>
> wrote:
>> HAProxy doesn't do keep-alive, so it makes everything slower.
>>
>> Artur
>>
>> On Feb 27, 2009, at 9:02 PM, Cloude Porteus wrote:
>>
>>> John,
>>> Thanks so much for the info, that's a huge help for us!!!
>>>
>>> I love HAProxy and Willy has been awesome to us. We run everything
>>> through it, since it's really easy to monitor and also easy to debug
>>> where the lag is when something in the chain is not responding fast
>>> enough. It's been rock solid for us.
>>>
>>> The nice part for us is that we can use it as a content switcher to
>>> send all /xxx traffic or certain user-agent traffic to different
>>> backends.
>>>
>>> best,
>>> cloude
>>>
>>> On Fri, Feb 27, 2009 at 2:24 PM, John Adams <jna at twitter.com> wrote:
>>>>
>>>> cc'ing the varnish dev list for comments...
>>>> On Feb 27, 2009, at 1:33 PM, Cloude Porteus wrote:
>>>>
>>>> John,
>>>> Goodto hear from you. You must be slammed at Twitter. I'm happy to
>>>> hear that ESI is holding up for you. It's been in my backlog
>>>> since you
>>>> mentioned it to me pre-Twitter.
>>>>
>>>> Any performance info would be great.
>>>>
>>>>
>>>> Any comments on our setup are welcome. You may also choose to
>>>> call us
>>>> crazypants. Many, many thanks to Artur Bergman of Wikia for
>>>> helping us
>>>> get
>>>> this configuration straightened out.
>>>> Right now, we're running varnish (on search) in a bit of a non-
>>>> standard
>>>> way.
>>>> We plan to use it in the normal fashion (varnish to Internet,
>>>> nothing
>>>> inbetween) on our API at some point. We're running version 2.0.2,
>>>> no
>>>> patches. Cache hit rates range from 10% to 30%, or higher when a
>>>> real-time
>>>> event is flooding search.
>>>> 2.0.2 is quite stable for us, with the occasional child death
>>>> here and
>>>> there
>>>> when we get massive headers coming in that flood sess_workspace.
>>>> I hear
>>>> this
>>>> is fixed in 2.0.3, but haven't had time to try it yet.
>>>> We have a number of search boxes, and each search box has an apache
>>>> instance
>>>> on it, and varnish instance. We plan to merge the varnish
>>>> instances at
>>>> some
>>>> point, but we use very low TTLs (Twitter is the real time web!)
>>>> and don't
>>>> see much of a savings by running less of them.
>>>> We do:
>>>> Apache --> Varnish --> Apache -> Mongrels
>>>> Apaches are using mod_proxy_balancer. The front end apache is there
>>>> because
>>>> we've long had a fear that Varnish would crash on us, which it
>>>> did many
>>>> times prior to our figuring out the proper parameters for
>>>> startup. We
>>>> have
>>>> two entries in that balancer. Either the request goes to varnish,
>>>> or, if
>>>> varnish bombs out, it goes directly to the mongrel.
>>>> We do this, because we need a load balancing algorithm that varnish
>>>> doesn't
>>>> support, called bybusiness. Without bybusiness, varnish tries to
>>>> direct
>>>> requests to Mongrels that are busy, and requests end up in the
>>>> listen
>>>> queue.
>>>> that adds ~100-150mS to load times, and that's no good for our
>>>> desired
>>>> service times of 200-250mS (or less.)
>>>> We'd be so happy if someone put bybusiness into Varnish's backend
>>>> load
>>>> balancing, but it's not there yet.
>>>> We also know that taking the extra hop through localhost costs us
>>>> next to
>>>> nothing in service time, so it's good to have Apache there incase
>>>> we need
>>>> to
>>>> yank out Varnish. In the future, we might get rid of Apache and use
>>>> HAProxy
>>>> (it's load balancing and backend monitoring is much richer than
>>>> Apache,
>>>> and,
>>>> it has a beautiful HTTP interface to look at.)
>>>> Some variables and our decisions:
>>>> -p obj_workspace=4096 \
>>>> -p sess_workspace=262144 \
>>>> Absolutely vital! Varnish does not allocate enough space by
>>>> default for
>>>> headers, regexps on cookies, and otherwise. It was increased in
>>>> 2.0.3,
>>>> but
>>>> really, not increased enough. Without this we were panicing every
>>>> 20-30
>>>> requests and overflowing the sess hash.
>>>> -p listen_depth=8192 \
>>>> 8192 is probably excessive for now. If we're queuing 8k conns,
>>>> something
>>>> is
>>>> really broke!
>>>> -p log_hashstring=off \
>>>> Who cares about this - we don't need it.
>>>> -p lru_interval=60 \
>>>> We have many small objects in the search cache. Run LRU more often.
>>>> -p sess_timeout=10 \
>>>> If you keep session data around for too long, you waste memory.
>>>> -p shm_workspace=32768 \
>>>> Give us a bit more room in shm
>>>> -p ping_interval=1 \
>>>> Frequent pings in case the child dies on us.
>>>> -p thread_pools=4 \
>>>> -p thread_pool_min=100 \
>>>> This must match up with VARNISH_MIN_THREADS. We use four pools,
>>>> (pools *
>>>> thread_pool_min == VARNISH_MIN_THREADS)
>>>> -p srcaddr_ttl=0 \
>>>> Disable the (effectively unused) per source-IP statistics
>>>> -p esi_syntax=1
>>>> Disable ESI syntax verification so we can use it to process JSON
>>>> requests.
>>>> If you have more than 2.1M objects, you should also add:
>>>> # -h classic,250007 = recommeded value for 2.1M objects
>>>> # number should be 1/10 expected working set.
>>>>
>>>> In our VCL, we have a few fancy tricks that we use. We label the
>>>> cache
>>>> server and cache hit/miss rate in vcl_deliver with this code:
>>>> Top of VCL:
>>>> C{
>>>> #include <stdio.h>
>>>> #include <unistd.h>
>>>> char myhostname[255] = "";
>>>>
>>>> }C
>>>> vcl_deliver:
>>>> C{
>>>> VRT_SetHdr(sp, HDR_RESP, "\014X-Cache-Svr:", myhostname,
>>>> vrt_magic_string_end);
>>>> }C
>>>> /* mark hit/miss on the request */
>>>> if (obj.hits > 0) {
>>>> set resp.http.X-Cache = "HIT";
>>>> set resp.http.X-Cache-Hits = obj.hits;
>>>> } else {
>>>> set resp.http.X-Cache = "MISS";
>>>> }
>>>>
>>>> vcl_recv:
>>>> C{
>>>> if (myhostname[0] == '\0') {
>>>> /* only get hostname once - restart required if hostname
>>>> changes */
>>>> gethostname(myhostname, 255);
>>>> }
>>>> }C
>>>>
>>>> Portions of /etc/sysconfig/varnish follow...
>>>> # The minimum number of worker threads to start
>>>> VARNISH_MIN_THREADS=400
>>>> # The Maximum number of worker threads to start
>>>> VARNISH_MAX_THREADS=1000
>>>> # Idle timeout for worker threads
>>>> VARNISH_THREAD_TIMEOUT=60
>>>> # Cache file location
>>>> VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin
>>>> # Cache file size: in bytes, optionally using k / M / G / T suffix,
>>>> # or in percentage of available disk space using the % suffix.
>>>> VARNISH_STORAGE_SIZE="8G"
>>>> #
>>>> # Backend storage specification
>>>> VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"
>>>> # Default TTL used when the backend does not specify one
>>>> VARNISH_TTL=5
>>>> # the working directory
>>>> DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
>>>> -f ${VARNISH_VCL_CONF} \
>>>> -T
>>>> ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
>>>> -t ${VARNISH_TTL} \
>>>> -n ${VARNISH_WORKDIR} \
>>>> -w
>>>> ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},$
>>>> {VARNISH_THREAD_TIMEOUT} \
>>>> -u varnish -g varnish \
>>>> -p obj_workspace=4096 \
>>>> -p sess_workspace=262144 \
>>>> -p listen_depth=8192 \
>>>> -p log_hashstring=off \
>>>> -p lru_interval=60 \
>>>> -p sess_timeout=10 \
>>>> -p shm_workspace=32768 \
>>>> -p ping_interval=1 \
>>>> -p thread_pools=4 \
>>>> -p thread_pool_min=100 \
>>>> -p srcaddr_ttl=0 \
>>>> -p esi_syntax=1 \
>>>> -s ${VARNISH_STORAGE}"
>>>>
>>>> ---
>>>> John Adams
>>>> Twitter Operations
>>>> jna at twitter.com
>>>> http://twitter.com/netik
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> VP of Product Development
>>> Instructables.com
>>>
>>> http://www.instructables.com/member/lebowski
>>> _______________________________________________
>>> varnish-dev mailing list
>>> varnish-dev at projects.linpro.no
>>> http://projects.linpro.no/mailman/listinfo/varnish-dev
>>
>>
>
>
>
> --
> VP of Product Development
> Instructables.com
>
> http://www.instructables.com/member/lebowski
More information about the varnish-dev
mailing list