rdbms as backend

[Marcin Krol]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello Leif,

Thanks for answer!


What you write about is what we tried at first, using Tornado. It
works, except it simply does not have performance good enough: 600
req/sec while we need 30,000 req/sec (yes 30 thousand). This is for
entire subsystem which can consist of more than 1 machine but at
Tornado performance that would be at least 50 cores over many serves +
HA redundancy + all the associated overhead...

Currently we're into node.js which is able to serve 4000-5000 req/sec,
still not quite good enough.

Varnish performance is so high that it would be very attractive in
this role.

Re application logic in VCL I don't think we'd suffer this much: it's
"just" about fast cache. About the only serious problem I could see is
if failover and load balancing in varnish can't be made work well
enough (detect failure quickly and start using spare/backup
systematically instead of timing out on failed backends, cause uneven
load on backends, etc).


Thanks!
MK



W dniu 7/30/2013 14:46, Leif Pedersen pisze:
> Hi,
> 
> I'm not a varnish dev but I've been working with varnish and DBs
> for a long time.
> 
> My knee jerk is that you'll end up with more application logic in
> vcl than vcl is suitable for. Vcl can't loop or touch response
> bodies (without vmods). It'd be kind of neat to skip middleware and
> plug varnish straight into the db, but I wouldn't guess it to be
> worth the effort.
> 
> I'd instead write a mini web server using uwsgi or CherryPy to act
> as middleware. These tools have great memory footprints and
> performance.
> 
> - Leif
> 
> On 2013-07-30 6:30 AM, "Marcin Krol" <mrkafk at gmail.com 
> <mailto:mrkafk at gmail.com>> wrote:
> 
> Hello,
> 
> This is a peculiar topic that I think goes beyond typical use of 
> varnish so I post it here.
> 
> At my company we have a need peculiar sort of infrastructural
> subsystem:
> 
> - HTTP requests are done to find if smth is cached - RDBMS (mysql,
> oracle) backends - Other subsystems as backends
> 
> Clients use unified protocol based on (simple) http requests to
> get their data. (it's for this reason that we do not use caches
> built into rdbms directly, as well as we do not want to do tight
> coupling of a particular client to a particular rdbms or
> subsystem)
> 
> 
> Either we write the whole thing ourselves or we use smth else like 
> varnish.
> 
> 
> I like the thought of using varnish, although I'm not sure if it
> this is not shoehorning it into such role. However, when it comes
> to caching, load balancing, failover and cached HTTP results
> serving it's ideal in such role.
> 
> The only problem is backend. Essentially, what we need is e.g. for 
> mysql backend:
> 
> on cache miss:
> 
> - connect to mysql
> 
> - run the query we received in GET/POST/whatever
> 
> - JSONIFy result (query results are not big in our application, 
> limited size of the result is a tolerable limitation for us)
> 
> - cache result, return it
> 
> 
> on cache hit:
> 
> - retrieve from cache, return it
> 
> 
> (and so on for other backends)
> 
> 
> Is this feasible? Is it even sane? Should I use smth else maybe?
> 
> Essentially, what we need are pluggable, modular backends.
> (obviously we can handle writing the part that transforms
> particular backend response into HTTP response, the snag is how to
> plug this correctly into backend usable by varnish)
> 
> I was thinking about using VMODs but none of the modules available 
> seem to meddle with backends themselves somehow.
> 
> 
> Thanks! MK
> 
> 
> _______________________________________________ varnish-dev mailing
> list varnish-dev at varnish-cache.org
> <mailto:varnish-dev at varnish-cache.org> 
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJR97uhAAoJEFMgHzhQQ7hOSPkIAIRCFwLmt5FtIa/VHVJoMxZR
BeLx5yKZBSCHJPhBIeywK4sd7+bW7AMLAEh9VYRfe5c7yuZO8mxAXYmvVXPmD/cl
ySUqZJD0LnpdL1kos25K/r8vn6PRPul2jm22u0SloKqQ5ME8TGNsk/SLmFbiLwCt
nhfL7ia+19/ZVK3XSZ5pcvsCyrM8flFS3TdM5TeKWDBhxu0uuccaBkuYy0qOK0zs
5YoLUD9GRwj9WN0pt2xFWYncgFP2wGsPzO9YrpVyc73ExXviI1LF+i+1K2wrOk+v
nYqekDAHufQmLfeupY5Dq/EHyWnN2Yec/cLaiUQADZKs7ua43GkiUqqJ/pxIUbk=
=hymW
-----END PGP SIGNATURE-----