Our experience with the persistent branch

[tousif baig]

Hey Dererk,

Nice to see tht Varnish is now part of OLX, I have been a happy user of it.
Keep up the good work.

Another simple tip from my experience will be to distribute the load on
different servers and not just one which is also possible with varnish and
maybe get down to some browser specifics too. But since you are using
hash-based balancing algorithm in front, I am not sure if that will be
possible. Do let me know if u need more specifics for those.

Good Day!
Tousif


On Thu, Oct 24, 2013 at 3:36 AM, Dererk <dererk at deadbeef.com.ar> wrote:

> Hi there!
>
> I thought It would be useful for some people out there to mention a
> "case of successful" on the Persistent branch of Varnish Cache on a
> large production scale.
> Ours is the replacement of the so-long-time-appreciated Squid Cache for
> delivering (mostly) user-provided content, like Images and other kinds
> of documents, but static objects like Javascripts and style sheets
> belonging to our platform, OLX, as well.
>
> - TL;DR Version
> The development around the persistent branch rocks, *big time*!
>
> - Full Version
> //  *Disclaimer*
> //  Our configuration is not what one could call "conventional", and we
> can't really afford loosing large amounts of cache in short periods of
> time.
>
>
> - The Business
>
> Our company, OnLine eXchange or simply OLX for short, has established
> its core around the Free Classifieds business, enabling Sellers and
> Buyers to perform peer-to-peer selling transactions with no commission
> required to any parts. Our portal allows sellers to upload images for
> better describing their items into our platform and being more
> attractive to potential buyers and having better chances of selling
> them. Just like Craiglist does in the US or Allegro in Poland.
>
> Although some particular countries are more popular than others, we have
> a large amount of traffic world-wide, due to an extensive net of
> operations and localization.
> To sum up, this translates into having to tune every single piece of
> software we run out there for literally delivering several gbps of
> traffic out to the internet.
>
>
> - Massive Delivering
>
> Even though we heavily rest on content delivery networks (CDNs) for
> performing last-mile optimizations, geo-caching and network optimized
> delivery, they have limits on what they can handle and they also run
> caching rotation algorithms over objects as well (say, something like
> LRU). The do all the time, more often that you do actually want.
> On the other hand, because of their geo-distribution nature, in reality
> several request are performed from different locations before they
> effectively cache objects for some small period of time.
>
> It might be true that we are on average offloaded +80% of the traffic,
> but again, in reality even that small portion has average 400 mbits/s,
> with frequent 500mbps spikes in business-as-usual levels.
>
>
> - Ancient History
>
> We used to run this internal caching tier with Squid Cache backing the
> cache up, with the help of tons of optimizations and an extremely
> experimental backend called Cyclic Object storage system or COSS. This
> very very experimental backend made use of a very granular storage
> configurations to get the most out of every IOP we could save to our
> loaded storage backends.
> Due to its immaturity, some particular operations sucked, like a *30
> minutes downtime* per instance by restarting the squid engine because of
> COSS data rebuild & consistency checks.
>
> We also rested very heavily on a sibling caching relation ship
> configuration by using HTCP, on a way to maximize the caching
> availability and saving some objects to get to the origin, which, as I
> said, was heavily loaded almost all the time.
> Squid is a great tool and believe me when I say, we loved it with all
> the downsides it had, we have had it for several years up to very
> recently, we knew it backwards and forwards until then.
>
>
> - Modern History
>
> Things started to change for the worse recently when our
> long-standing-loaded storage backend tier, featuring a vendor supposedly
> standing at the Storage Top Five ranking, could not face our
> business-as-usual operations any longer.
> Performance started to fall apart in pieces once we started to hit +95%
> on this well-known storage provider's solution CPU usage, unable to
> serve objects in form and shape once 95% was reached.
> We were forced to start diving into new and radical alternatives on a
> very short term.
>
> We have been using Varnish internally since some years now for boosting
> our SOLR backend and some other HTTP caching roles, but not for
> delivering static content up to that moment.
> Now was the time to give it a chance, "for real" (we have several gbps
> of traffic internally too, but you know what I mean).
>
>
> - First Steps
>
> We started by using the same malloc backend configurations as we were
> using on this other areas were Varnish was deployed, with some
> performance tunes around sessions (VARNISH_SESSION_LINGER) and threads
> capacity (VARNISH_MAX_THREADS &&VARNISH_THREAD_POOLS &&
> VARNISH_THREAD_TIMEOUT).
>
> The server profiles handling this tasks were some huge boxes with 128M
> RAM, but since the 50-ish terabytes dataset didn't actually fit on RAM,
> once all the memory was allocated we started to suffer same random
> panics at random periods of time. Unable to replicate them on demand or
> to produce any debugging due to the amount of traffic this huge devils
> handled on production, things started to get really ugly really fast.
>
>
> - The Difficult Choice
>
> At this point we started to consider every possible option available out
> there, even switching away to other caching alternatives, like
> trafficserver, that provided persistence for cached objects, and we
> decided to give the persistent backend a shoot, but, there was a catch:
> the persistent backend was (and also currently is) considered
> experimental (so did COSS, btw!).
>
> Effectively, as for what 3.0.4 release on the main stable branch
> respects, the persistent backend had many bugs that raised up within
> minutes, many of them produced panics that crashed the child process,
> but, fortunately for us, the persistence itself worked so well that the
> first times went through totally unnoticed, which, compared with
> experiencing a 100mb cache melt down was just an amazing improvement by
> itself. Now things started to look better for Varnish.
> Of course, we learned that in fact we were loosing some cached objects
> by some broken silos in the way, but, doing a side-by-side comparison
> things looked different as night and day, and the best was yet to come.
>
> We were advised, in case we were to stick to a persistent backend, on
> using the persistent development branch, giving that more improvements
> were developed in there and major stability changes were introduced.
> But, think again, proposing something that has the "experimental" and
> "on development" tags hanging from it usually sells horribly to the
> management people on the other side of the table.
>
>
> - Summary
>
> At the end, with the help of a hash-based balancing algorithm at the
> load balancing tier in front of our Varnish caches, we were able to
> *almost cut half* of the CPU usage at our storage solution tier, that is
> by geting 60% insead of +90% of CPU usage, something similar to a sunny
> day for a walk through the park, even for serving the +2.000
> request/second arriving at our datacenters.
>
> We got there by offloading content on a *up to +70% cache hits*,
> something that was totally unconceivable for anyone at the very
> beginning of the migration, giving that we used to get less than 30% in
> the past with Squid.
>
> We were able to get up to this point with lots of patience and research,
> but particularly with the help of the Varnish core development that
> constantly supported us at the IRC channels and mailing lists.
>
> Thanks a lot guys, you and Varnish rock, big time!
>
>
>
> A happy user!
>
> Dererk
>
> --
> BOFH excuse #274:
> It was OK before you touched it.
>
>
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20131024/79508cf0/attachment-0001.html>