rdbms as backend

[Rogier 'DocWilco' Mulhuijzen]

My kneejerk reaction is somewhat the same. Varnish is HTTP, and should stay
HTTP only.

It would be the best solution to have an HTTP to your RDBMS layer. Call it
middleware if you will, or maybe in interface.

You don't really need logic for the situation you outlined, if Varnish can
do the caching, all you need to do is set your HTTP-to-RDBMS layer as
backend. Then it's just the standard hit/miss stuff from Varnish point of
view.

If you can't write anything yourself that's fast enough to do that (in any
language), it's not going to be any faster if it's part of Varnish. So last
ditch effort, write something that's multithreaded or async in C to handle
it.

If that is fast enough, and there's a really good reason to integrate it
with Varnish we can always revisit putting it into Varnish. :)


On Tue, Jul 30, 2013 at 3:26 PM, Marcin Krol <mrkafk at gmail.com> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> P.S. What I want to do is precisely avoiding modifying response
> bodies: JSONIFy/format in the backend handler, handing over typical
> HTTP objects to varnish in "Fetch from backend" operation:
>
> https://www.varnish-software.com/static/book/_images/vcl.png
>
>
>
>
> W dniu 7/30/2013 14:46, Leif Pedersen pisze:
> > Hi,
> >
> > I'm not a varnish dev but I've been working with varnish and DBs
> > for a long time.
> >
> > My knee jerk is that you'll end up with more application logic in
> > vcl than vcl is suitable for. Vcl can't loop or touch response
> > bodies (without vmods). It'd be kind of neat to skip middleware and
> > plug varnish straight into the db, but I wouldn't guess it to be
> > worth the effort.
> >
> > I'd instead write a mini web server using uwsgi or CherryPy to act
> > as middleware. These tools have great memory footprints and
> > performance.
> >
> > - Leif
> >
> > On 2013-07-30 6:30 AM, "Marcin Krol" <mrkafk at gmail.com
> > <mailto:mrkafk at gmail.com>> wrote:
> >
> > Hello,
> >
> > This is a peculiar topic that I think goes beyond typical use of
> > varnish so I post it here.
> >
> > At my company we have a need peculiar sort of infrastructural
> > subsystem:
> >
> > - HTTP requests are done to find if smth is cached - RDBMS (mysql,
> > oracle) backends - Other subsystems as backends
> >
> > Clients use unified protocol based on (simple) http requests to
> > get their data. (it's for this reason that we do not use caches
> > built into rdbms directly, as well as we do not want to do tight
> > coupling of a particular client to a particular rdbms or
> > subsystem)
> >
> >
> > Either we write the whole thing ourselves or we use smth else like
> > varnish.
> >
> >
> > I like the thought of using varnish, although I'm not sure if it
> > this is not shoehorning it into such role. However, when it comes
> > to caching, load balancing, failover and cached HTTP results
> > serving it's ideal in such role.
> >
> > The only problem is backend. Essentially, what we need is e.g. for
> > mysql backend:
> >
> > on cache miss:
> >
> > - connect to mysql
> >
> > - run the query we received in GET/POST/whatever
> >
> > - JSONIFy result (query results are not big in our application,
> > limited size of the result is a tolerable limitation for us)
> >
> > - cache result, return it
> >
> >
> > on cache hit:
> >
> > - retrieve from cache, return it
> >
> >
> > (and so on for other backends)
> >
> >
> > Is this feasible? Is it even sane? Should I use smth else maybe?
> >
> > Essentially, what we need are pluggable, modular backends.
> > (obviously we can handle writing the part that transforms
> > particular backend response into HTTP response, the snag is how to
> > plug this correctly into backend usable by varnish)
> >
> > I was thinking about using VMODs but none of the modules available
> > seem to meddle with backends themselves somehow.
> >
> >
> > Thanks! MK
> >
> >
> > _______________________________________________ varnish-dev mailing
> > list varnish-dev at varnish-cache.org
> > <mailto:varnish-dev at varnish-cache.org>
> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
> >
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.20 (MingW32)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBAgAGBQJR978LAAoJEFMgHzhQQ7hOugUH/2WLtSaYO+/iLlyX/moGNBqR
> HTa0E1W+43vIVIeq9OLzKmmxzJm2075Erq952B9/UIVnxNDZ4jo+whOk8foo1K0Q
> h4A8XloKrMG1BSNnYqCsdop202YIIPI1AxX8V30slx1btjttdVedAQxwlGZ1j6NC
> S43z5EqJ0dKjOAv1JVTLkeAkCWJCXmYj0xIiPYNCcRQ9uR8QCA1Nm+cL3MDuzcIS
> TQuEW/wihy4bfNeKe5H7+GutARGHHtiJXzvzgiGFLcto0IasTUroTKN05MTuIjEE
> e4t7n4MfLxwninEWitF23UjCQUvoq05cZMihxMD1XjSPtG+qsK+BmyTHUqTab98=
> =bT0L
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> varnish-dev mailing list
> varnish-dev at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.varnish-cache.org/lists/pipermail/varnish-dev/attachments/20130731/91461c3e/attachment.html>