Is anyone using ESI with a lot of traffic?
Cloude Porteus
cloude at instructables.com
Mon Mar 2 22:49:37 CET 2009
Fair enough. We don't use keep-alive, so it hasn't been an issue.
What are you guys using for load balancing?
best,
cloude
On Mon, Mar 2, 2009 at 1:45 PM, Artur Bergman <sky at crucially.net> wrote:
> "Right now, HAProxy only supports the first mode (HTTP close) if it needs to
> process the request. This means that for each request, there will be one TCP
> connection. If keep-alive or pipelining are required, HAProxy will still
> support them, but will only see the first request and the first response of
> each transaction. While this is generally problematic with regards to logs,
> content switching or filtering, it most often causes no problem for
> persistence
> with cookie insertion."
> Really not fully supporting keep-alive! More like varnish pipe mode
> Artur
> On Mar 2, 2009, at 1:33 PM, Cloude Porteus wrote:
>
>> I believe TCP Keep-alive has been supported in HAProxy since version
>> 1.2. We've been using 1.3.x for at least a year.
>>
>> -cloude
>>
>> On Mon, Mar 2, 2009 at 1:10 PM, Artur Bergman <sky at crucially.net> wrote:
>>>
>>> HAProxy doesn't do keep-alive, so it makes everything slower.
>>>
>>> Artur
>>>
>>> On Feb 27, 2009, at 9:02 PM, Cloude Porteus wrote:
>>>
>>>> John,
>>>> Thanks so much for the info, that's a huge help for us!!!
>>>>
>>>> I love HAProxy and Willy has been awesome to us. We run everything
>>>> through it, since it's really easy to monitor and also easy to debug
>>>> where the lag is when something in the chain is not responding fast
>>>> enough. It's been rock solid for us.
>>>>
>>>> The nice part for us is that we can use it as a content switcher to
>>>> send all /xxx traffic or certain user-agent traffic to different
>>>> backends.
>>>>
>>>> best,
>>>> cloude
>>>>
>>>> On Fri, Feb 27, 2009 at 2:24 PM, John Adams <jna at twitter.com> wrote:
>>>>>
>>>>> cc'ing the varnish dev list for comments...
>>>>> On Feb 27, 2009, at 1:33 PM, Cloude Porteus wrote:
>>>>>
>>>>> John,
>>>>> Goodto hear from you. You must be slammed at Twitter. I'm happy to
>>>>> hear that ESI is holding up for you. It's been in my backlog since you
>>>>> mentioned it to me pre-Twitter.
>>>>>
>>>>> Any performance info would be great.
>>>>>
>>>>>
>>>>> Any comments on our setup are welcome. You may also choose to call us
>>>>> crazypants. Many, many thanks to Artur Bergman of Wikia for helping us
>>>>> get
>>>>> this configuration straightened out.
>>>>> Right now, we're running varnish (on search) in a bit of a non-standard
>>>>> way.
>>>>> We plan to use it in the normal fashion (varnish to Internet, nothing
>>>>> inbetween) on our API at some point. We're running version 2.0.2, no
>>>>> patches. Cache hit rates range from 10% to 30%, or higher when a
>>>>> real-time
>>>>> event is flooding search.
>>>>> 2.0.2 is quite stable for us, with the occasional child death here and
>>>>> there
>>>>> when we get massive headers coming in that flood sess_workspace. I hear
>>>>> this
>>>>> is fixed in 2.0.3, but haven't had time to try it yet.
>>>>> We have a number of search boxes, and each search box has an apache
>>>>> instance
>>>>> on it, and varnish instance. We plan to merge the varnish instances at
>>>>> some
>>>>> point, but we use very low TTLs (Twitter is the real time web!) and
>>>>> don't
>>>>> see much of a savings by running less of them.
>>>>> We do:
>>>>> Apache --> Varnish --> Apache -> Mongrels
>>>>> Apaches are using mod_proxy_balancer. The front end apache is there
>>>>> because
>>>>> we've long had a fear that Varnish would crash on us, which it did many
>>>>> times prior to our figuring out the proper parameters for startup. We
>>>>> have
>>>>> two entries in that balancer. Either the request goes to varnish, or,
>>>>> if
>>>>> varnish bombs out, it goes directly to the mongrel.
>>>>> We do this, because we need a load balancing algorithm that varnish
>>>>> doesn't
>>>>> support, called bybusiness. Without bybusiness, varnish tries to direct
>>>>> requests to Mongrels that are busy, and requests end up in the listen
>>>>> queue.
>>>>> that adds ~100-150mS to load times, and that's no good for our desired
>>>>> service times of 200-250mS (or less.)
>>>>> We'd be so happy if someone put bybusiness into Varnish's backend load
>>>>> balancing, but it's not there yet.
>>>>> We also know that taking the extra hop through localhost costs us next
>>>>> to
>>>>> nothing in service time, so it's good to have Apache there incase we
>>>>> need
>>>>> to
>>>>> yank out Varnish. In the future, we might get rid of Apache and use
>>>>> HAProxy
>>>>> (it's load balancing and backend monitoring is much richer than Apache,
>>>>> and,
>>>>> it has a beautiful HTTP interface to look at.)
>>>>> Some variables and our decisions:
>>>>> -p obj_workspace=4096 \
>>>>> -p sess_workspace=262144 \
>>>>> Absolutely vital! Varnish does not allocate enough space by default
>>>>> for
>>>>> headers, regexps on cookies, and otherwise. It was increased in 2.0.3,
>>>>> but
>>>>> really, not increased enough. Without this we were panicing every 20-30
>>>>> requests and overflowing the sess hash.
>>>>> -p listen_depth=8192 \
>>>>> 8192 is probably excessive for now. If we're queuing 8k conns,
>>>>> something
>>>>> is
>>>>> really broke!
>>>>> -p log_hashstring=off \
>>>>> Who cares about this - we don't need it.
>>>>> -p lru_interval=60 \
>>>>> We have many small objects in the search cache. Run LRU more often.
>>>>> -p sess_timeout=10 \
>>>>> If you keep session data around for too long, you waste memory.
>>>>> -p shm_workspace=32768 \
>>>>> Give us a bit more room in shm
>>>>> -p ping_interval=1 \
>>>>> Frequent pings in case the child dies on us.
>>>>> -p thread_pools=4 \
>>>>> -p thread_pool_min=100 \
>>>>> This must match up with VARNISH_MIN_THREADS. We use four pools, (pools
>>>>> *
>>>>> thread_pool_min == VARNISH_MIN_THREADS)
>>>>> -p srcaddr_ttl=0 \
>>>>> Disable the (effectively unused) per source-IP statistics
>>>>> -p esi_syntax=1
>>>>> Disable ESI syntax verification so we can use it to process JSON
>>>>> requests.
>>>>> If you have more than 2.1M objects, you should also add:
>>>>> # -h classic,250007 = recommeded value for 2.1M objects
>>>>> # number should be 1/10 expected working set.
>>>>>
>>>>> In our VCL, we have a few fancy tricks that we use. We label the cache
>>>>> server and cache hit/miss rate in vcl_deliver with this code:
>>>>> Top of VCL:
>>>>> C{
>>>>> #include <stdio.h>
>>>>> #include <unistd.h>
>>>>> char myhostname[255] = "";
>>>>>
>>>>> }C
>>>>> vcl_deliver:
>>>>> C{
>>>>> VRT_SetHdr(sp, HDR_RESP, "\014X-Cache-Svr:", myhostname,
>>>>> vrt_magic_string_end);
>>>>> }C
>>>>> /* mark hit/miss on the request */
>>>>> if (obj.hits > 0) {
>>>>> set resp.http.X-Cache = "HIT";
>>>>> set resp.http.X-Cache-Hits = obj.hits;
>>>>> } else {
>>>>> set resp.http.X-Cache = "MISS";
>>>>> }
>>>>>
>>>>> vcl_recv:
>>>>> C{
>>>>> if (myhostname[0] == '\0') {
>>>>> /* only get hostname once - restart required if hostname changes */
>>>>> gethostname(myhostname, 255);
>>>>> }
>>>>> }C
>>>>>
>>>>> Portions of /etc/sysconfig/varnish follow...
>>>>> # The minimum number of worker threads to start
>>>>> VARNISH_MIN_THREADS=400
>>>>> # The Maximum number of worker threads to start
>>>>> VARNISH_MAX_THREADS=1000
>>>>> # Idle timeout for worker threads
>>>>> VARNISH_THREAD_TIMEOUT=60
>>>>> # Cache file location
>>>>> VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin
>>>>> # Cache file size: in bytes, optionally using k / M / G / T suffix,
>>>>> # or in percentage of available disk space using the % suffix.
>>>>> VARNISH_STORAGE_SIZE="8G"
>>>>> #
>>>>> # Backend storage specification
>>>>> VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"
>>>>> # Default TTL used when the backend does not specify one
>>>>> VARNISH_TTL=5
>>>>> # the working directory
>>>>> DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
>>>>> -f ${VARNISH_VCL_CONF} \
>>>>> -T
>>>>> ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
>>>>> -t ${VARNISH_TTL} \
>>>>> -n ${VARNISH_WORKDIR} \
>>>>> -w
>>>>> ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT}
>>>>> \
>>>>> -u varnish -g varnish \
>>>>> -p obj_workspace=4096 \
>>>>> -p sess_workspace=262144 \
>>>>> -p listen_depth=8192 \
>>>>> -p log_hashstring=off \
>>>>> -p lru_interval=60 \
>>>>> -p sess_timeout=10 \
>>>>> -p shm_workspace=32768 \
>>>>> -p ping_interval=1 \
>>>>> -p thread_pools=4 \
>>>>> -p thread_pool_min=100 \
>>>>> -p srcaddr_ttl=0 \
>>>>> -p esi_syntax=1 \
>>>>> -s ${VARNISH_STORAGE}"
>>>>>
>>>>> ---
>>>>> John Adams
>>>>> Twitter Operations
>>>>> jna at twitter.com
>>>>> http://twitter.com/netik
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> VP of Product Development
>>>> Instructables.com
>>>>
>>>> http://www.instructables.com/member/lebowski
>>>> _______________________________________________
>>>> varnish-dev mailing list
>>>> varnish-dev at projects.linpro.no
>>>> http://projects.linpro.no/mailman/listinfo/varnish-dev
>>>
>>>
>>
>>
>>
>> --
>> VP of Product Development
>> Instructables.com
>>
>> http://www.instructables.com/member/lebowski
>
>
--
VP of Product Development
Instructables.com
http://www.instructables.com/member/lebowski
More information about the varnish-dev
mailing list