VCL continuous integration
L Cruzero
lcruzero at gmail.com
Thu Aug 21 14:12:18 CEST 2014
> is anyone on the list using any CI tools or have any ideas possible options
> for implementing continuous integration with VLC.
I don't know what you're looking for, but one of the tests in our CI suite does
a Varnish syntax check by calling `varnishd -C -f <filename> -n /tmp 2>&1
1>/dev/null` and checking the exit code. Doing much more is difficult because
Varnish, like other services, is fairly global, as opposed to application code,
which can be much more easily tested in parallel on a single machine.
- P
thanks James, and others for some of the useful suggestions. while
varnishtest and varnishD respectively,
will be used as suggested, for testing cache and vcl syntax as part of
unit tests, in a "bamboo" CI agent. it would also be useful to test,
perhaps outside of a syntax/cache unit test, some of the external
resources a vcl config depends on.
ie.
-defined backends connectivity.
-response header from host header request to defined backends
thanks.
-LC
On Thu, Aug 21, 2014 at 5:13 AM, <varnish-misc-request at varnish-cache.org> wrote:
>
> Send varnish-misc mailing list submissions to
> varnish-misc at varnish-cache.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> or, via email, send a message with subject or body 'help' to
> varnish-misc-request at varnish-cache.org
>
> You can reach the person managing the list at
> varnish-misc-owner at varnish-cache.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of varnish-misc digest..."
>
>
> Today's Topics:
>
> 1. another architecture question (Hern?n Marsili)
> 2. Re: another architecture question (Dridi Boukelmoune)
> 3. clearing the varnish cache (Tim Dunphy)
> 4. Re: VCL continuous integration (James Pearson)
> 5. Re: VCL continuous integration (Dridi Boukelmoune)
> 6. Re: clearing the varnish cache (Per Buer)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 20 Aug 2014 11:02:47 -0300
> From: Hern?n Marsili <hernan at cmsmedios.com>
> To: varnish-misc <varnish-misc at varnish-cache.org>
> Subject: another architecture question
> Message-ID:
> <CABEj_0rxnRhYgi9-Ci_3BkHw+aJMZxVt8UTNRcx+F=OWTHmKLQ at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi,
>
>
> A couple of month ago I asked a question here regarding the best VARNISH
> architecture for a high traffic site. We then do some testing and we are
> still unsure of the best scenario, so I ask again with a little more
> information.
>
> We have a 40.000 concurrent users sites. This is handled by 5 boxes. On
> each box we have TOMCAT, APACHE and VARNISH using malloc.
>
> What we want to determine, is the impact on each server VARNISH makes,
> mostly at a 'connections' level and determine if is best to:
>
> a) growth adding same kind of servers (boxes with all 3 services)
> b) separete Varnish from the boxes and have, for example, 2 varnishes
> balacing with 4 backend servers
>
> On the b scenario, we can assign more memory to the TOMCAT and handle more
> load. The CACHE HIT RATE is not an issue, since all the boxes have a very
> high hitrate.
>
> Any advice on this matter will be appreciated. We are inclining now for the
> A option.
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20140820/02526c7c/attachment-0001.html>
>
> ------------------------------
>
> Message: 2
> Date: Wed, 20 Aug 2014 18:52:02 +0200
> From: Dridi Boukelmoune <dridi.boukelmoune at zenika.com>
> To: Hern?n Marsili <hernan at cmsmedios.com>
> Cc: varnish-misc <varnish-misc at varnish-cache.org>
> Subject: Re: another architecture question
> Message-ID:
> <CABtDKm4bzgVxrp76QTuHuH17O=zfT6daMft28qmCy7hTp3PVgA at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> Hi,
>
> As a Java developer, I'd say that adding more RAM to a JVM may
> probably incur longer GC pauses while on the other hand I'd trust
> Varnish to gracefully scale with more RAM and CPU. Please do not
> interpret that as a bad opinion on modern JVMs or Tomcat, you'd be
> very wrong :-)
>
> Architecture-wise, my first advice would be to think about removing
> your httpd servers and put your Tomcat instances right behind Varnish.
> Unless you need it for something other than a reverse proxy for your
> webapp of course.
>
> Regarding solutions a and b, my intuition goes towards b. Instead of
> pinning one apache/tomcat pair for one Varnish instance, I'd put them
> all in a director. If one backend is sick, Varnish can try again with
> a healthy backend.
>
> Performance-wise, measure, don't guess.
>
> Cheers,
> Dridi
>
> On Wed, Aug 20, 2014 at 4:02 PM, Hern?n Marsili <hernan at cmsmedios.com> wrote:
> > Hi,
> >
> >
> > A couple of month ago I asked a question here regarding the best VARNISH
> > architecture for a high traffic site. We then do some testing and we are
> > still unsure of the best scenario, so I ask again with a little more
> > information.
> >
> > We have a 40.000 concurrent users sites. This is handled by 5 boxes. On each
> > box we have TOMCAT, APACHE and VARNISH using malloc.
> >
> > What we want to determine, is the impact on each server VARNISH makes,
> > mostly at a 'connections' level and determine if is best to:
> >
> > a) growth adding same kind of servers (boxes with all 3 services)
> > b) separete Varnish from the boxes and have, for example, 2 varnishes
> > balacing with 4 backend servers
> >
> > On the b scenario, we can assign more memory to the TOMCAT and handle more
> > load. The CACHE HIT RATE is not an issue, since all the boxes have a very
> > high hitrate.
> >
> > Any advice on this matter will be appreciated. We are inclining now for the
> > A option.
> >
> > _______________________________________________
> > varnish-misc mailing list
> > varnish-misc at varnish-cache.org
> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
> ------------------------------
>
> Message: 3
> Date: Wed, 20 Aug 2014 14:13:49 -0400
> From: Tim Dunphy <bluethundr at gmail.com>
> To: "varnish-misc at varnish-cache.org" <varnish-misc at varnish-cache.org>
> Subject: clearing the varnish cache
> Message-ID:
> <CAOZy0e=CFS=CshzLNqZ-c=G6PB+Ug4t692w0eDU3_Ofu7UNNkw at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hey all,
>
> I've been asked to flush the varnish cache in our environment. So I used
> the following command to do that:
>
> varnishadm -T 127.0.0.1:2000 url.purge .
>
> However the command doesn't provide any feedback or output. How can we
> verify that the cache has indeed been flushed?
>
> Wer'e using varnish 2.1.5.
>
> Thanks
> Tim
>
> --
> GPG me!!
>
> gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20140820/c818ea8b/attachment-0001.html>
>
> ------------------------------
>
> Message: 4
> Date: Wed, 20 Aug 2014 11:27:27 -0700
> From: James Pearson <james at ifixit.com>
> To: varnish-misc <varnish-misc at varnish-cache.org>
> Subject: Re: VCL continuous integration
> Message-ID: <1408559106-sup-465 at geror.local>
> Content-Type: text/plain; charset="utf-8"
>
> Excerpts from L Cruzero's message of 2014-08-05 07:27:15 -0700:
> > is anyone on the list using any CI tools or have any ideas possible options
> > for implementing continuous integration with VLC.
>
> I don't know what you're looking for, but one of the tests in our CI suite does
> a Varnish syntax check by calling `varnishd -C -f <filename> -n /tmp 2>&1
> 1>/dev/null` and checking the exit code. Doing much more is difficult because
> Varnish, like other services, is fairly global, as opposed to application code,
> which can be much more easily tested in parallel on a single machine.
> - P
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: signature.asc
> Type: application/pgp-signature
> Size: 819 bytes
> Desc: not available
> URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20140820/db56fb68/attachment-0001.pgp>
>
> ------------------------------
>
> Message: 5
> Date: Wed, 20 Aug 2014 22:58:28 +0200
> From: Dridi Boukelmoune <dridi.boukelmoune at zenika.com>
> To: James Pearson <james at ifixit.com>
> Cc: varnish-misc <varnish-misc at varnish-cache.org>
> Subject: Re: VCL continuous integration
> Message-ID:
> <CABtDKm5NY_2tTRRse_iB-BSvc4cefKDJFNU8c8O3+qZ_wJiT_Q at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> On Wed, Aug 20, 2014 at 8:27 PM, James Pearson <james at ifixit.com> wrote:
> > Excerpts from L Cruzero's message of 2014-08-05 07:27:15 -0700:
> >> is anyone on the list using any CI tools or have any ideas possible options
> >> for implementing continuous integration with VLC.
> >
> > I don't know what you're looking for, but one of the tests in our CI suite does
> > a Varnish syntax check by calling `varnishd -C -f <filename> -n /tmp 2>&1
>
> You may want to add parameters like vcl_dir if like me you split your
> VCL in multiple files. I usually put at least the backend(s)/director(s)
> in a separate file, so that I can include the actual policy in test cases,
> or share the same policy for different environments (QA, preprod)
> without duplicating code (avoids divergence).
>
> > 1>/dev/null` and checking the exit code. Doing much more is difficult because
> > Varnish, like other services, is fairly global, as opposed to application code,
> > which can be much more easily tested in parallel on a single machine.
>
> But if you want to test your cache policy (namely your VCL) you can do
> so by mocking the backends and clients behaviors. Et voil?, you can run
> your varnishtest suite in parallel :-)
>
> Cheers,
> Dridi
>
> > - P
> >
> > _______________________________________________
> > varnish-misc mailing list
> > varnish-misc at varnish-cache.org
> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
>
>
> ------------------------------
>
> Message: 6
> Date: Thu, 21 Aug 2014 11:13:05 +0200
> From: Per Buer <perbu at varnish-software.com>
> To: Tim Dunphy <bluethundr at gmail.com>
> Cc: "varnish-misc at varnish-cache.org" <varnish-misc at varnish-cache.org>
> Subject: Re: clearing the varnish cache
> Message-ID:
> <CAOXZevDj=P2zmbd-hO01b73-4EnHJUzne81c3P5MVV0nJEr7XA at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hi Tim.
>
> First of all, the version of Varnish you're running is pretty outdated.
> 3.0, which was released three years replaced it and since then we've have a
> 4.0 release. Please consider upgrading.
>
> To answer your question. The easiest way to verify that it does what it
> should do is to monitor the varnishlog. You'll just a huge number of cache
> misses that naturally follows the purge of the cache. If you have no caches
> behind Varnish you can just grep out the Age header and you'll see it will
> initially be all zeroes.
>
> Per.
>
>
>
> On Wed, Aug 20, 2014 at 8:13 PM, Tim Dunphy <bluethundr at gmail.com> wrote:
>
> > Hey all,
> >
> > I've been asked to flush the varnish cache in our environment. So I used
> > the following command to do that:
> >
> > varnishadm -T 127.0.0.1:2000 url.purge .
> >
> > However the command doesn't provide any feedback or output. How can we
> > verify that the cache has indeed been flushed?
> >
> > Wer'e using varnish 2.1.5.
> >
> > Thanks
> > Tim
> >
> > --
> > GPG me!!
> >
> > gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
> >
> >
> > _______________________________________________
> > varnish-misc mailing list
> > varnish-misc at varnish-cache.org
> > https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
> >
>
>
>
> --
> <http://www.varnish-software.com/> *Per Buer*
> CTO | Varnish Software
> Phone: +47 958 39 117 | Skype: per.buer
> We Make Websites Fly!
>
> Winner of the Red Herring Top 100 Global Award 2013
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <https://www.varnish-cache.org/lists/pipermail/varnish-misc/attachments/20140821/37cc18de/attachment.html>
>
> ------------------------------
>
> _______________________________________________
> varnish-misc mailing list
> varnish-misc at varnish-cache.org
> https://www.varnish-cache.org/lists/mailman/listinfo/varnish-misc
>
> End of varnish-misc Digest, Vol 101, Issue 15
> *********************************************
More information about the varnish-misc
mailing list