vtest and varnish-cache repo relationship

[Willy Tarreau]

Hi Poul-Henning!

On Tue, May 06, 2025 at 01:13:22PM +0000, Poul-Henning Kamp wrote:
> This started as a private conversation, but I think it belongs
> on varnish-dev now:
> 
> I'm not really sure I know what the "previous plan" was any more, but
> I want this sorted out for 8.0.
> 
> My conversation with dridi@ yesterday revealed that we may have
> misunderstood each other somewhat, as there are two different issues
> involving shared libraries.
> 
> The first issue, let us call it "vtest+lib" is to make vtest a
> package which installs an include, static and/or dynamic libraries,
> and a binary which uses the shlib, so that the varnish-cache project,
> and possibly haproxy, can link our specific test-binary against the
> library of choice.
> 
> This only really makes sense if we go the full monty with the vtest
> repos:  Publishing releases, making packages, pushing them into
> distros etc.  The alternative is a simple source dependency, using
> some variant of git submodules or shell-scripts to do the dirty
> deed.
> 
> The second issue is that vtest uses a lot of stuff copied from
> varnish-cache's lib directory.
> 
> IMO that is just an annoyance.
> 
> To fix it "properly", the vtest repo could pull that stuff out of
> the varnish-cache repo, but that would pretty much defeat the entire
> reason why we created the vtest repo to begin with.
> 
> Alternatively, we could make a third repos containing "varnish-lib"
> which then both varnish-cache and vtest repos could depend on, as
> well as any other repos which have copied stuff from there over the
> years.
> 
> There is some good stuff in varnish-cache/lib, but personally I
> do not feel it is enough for us to start yet another repository.
> 
> If we do create a varnish-lib repos, then the next question again
> becomes "package or source dependency".
> 
> (I thought what I nixed at some previous occation was the "varnish-lib"
> idea, but dridi@ thought it was the "vtest+lib" idea.)
> 
> Finally there is an alternative where we do "vtest+lib", let the
> vtest repos take over responsibility for the lib stuff and have
> varnish-cache repos get it from there.  That would add no further
> dependencies, eliminate the duplication of the lib-stuff, and
> give any third parties a reasonably sized package to depend on
> if they want to use the lib-stuff.
> 
> So to me the core question seems to be:  Are we willing/going to
> make vtest+lib packages ?
> 
> And of course, since HAproxy also uses vtest, it is not something
> we alone decide... (Willy cc'ed)

Hmmm I'm seeing that you're facing the same issues that everyone faces
when starting to reuse code. IMHO, going the "varnish-lib" way is going
to be annoying. I've experienced the same with some of my personal
projects that are used in haproxy (libslz, ebtree etc): in practice,
haproxy is almost the only user, which means that bugs are most likely
to be detected there, and missing features as well. I hesitated between
adding an external dependency and integrating the code. I started with
the external dep and it was too painful (even for end users), so I ended
up integrating a copy of the libs (they're tiny) in the haproxy project.
So this means that haproxy contributors willing to patch certain areas
are sometimes confused about how to proceed: patch directly into haproxy
or send the patch to the upstream lib and hope that it will flow down.
But I generally have no problem supporting bidirectional patching,
because such code changes very slowly, so if I have two patches to port
or backport every year, that's no big deal. Still it's something you
need to keep in mind.

In your case, I sense that this "varnish-lib" would be the library
providing about everything. I think you'd face a huge overhead by
maintaining it out of the project, for little gains.

What *could* possibly work would be to make it easy to extract such code
parts from varnish into an external varnish-lib project that is only
updated when needed. Then vtest would depend on that. This would mean
that you do have a copy, but not a copy burried into vtest, instead a
copy that constitutes an autonomous project. It's much easier to keep
up to date, and its sole purpose is to permit other tools to continue
to work. So basically a full export from time to time does the job
*if needed*. And it's not *that* unusual. For example in the linux
kernel, one can to "make install-headers" and get a full directory
of all the system headers, that is consumed by whatever libc you use.

In any case I don't think that varnish using vtest as an external
dependency would be a good idea. It would first create an extra
maintenance burden for varnish, and second, create difficulties when
other projects start to adopt vtest because they will not realize that
the changes they propose can have an impact on varnish itself.

Just my two cents,
Willy