Varnish project autumn cleaning

[Poul-Henning Kamp]

After some private consultations, this email is being sent to the
-dev list, for notice and comment.

After we have hashed it out here, I will write it up for announce@
and the world at large.

As I have mentioned to a lot of you already, I have some changes
in mind for the Varnish Project after 4.1-R and I am busy these
days trying to get from "in mind" to "a plan".

There are a number of drivers for this, which I will just quickly
sketch out. (If you want more details, I can flesh it out further
in the subsequent discussions, this email is long enough as it is.)

The reasons
===========

1. We've gone from DevOps to Developers
---------------------------------------

This is most evident in 4.1-R being so delayed mainly because of
lack of real-life testing.

Becuase none of the central group of developers actually run a
Varnish production site, we have to persuade other people to do
live tests for us.

This means we cannot use the same "QA" strategy we did previously
and that must be reflected in our release process.


2. The workshop have become cluttered
-------------------------------------

The Trac wiki has become Hotel California.

Patchwork never really worked for us.

The Forum isn't exactly a noted social venue.

The project homepage looks a lot like an advertising front end for
Varnish Software - not from any sinister plan or hostile intent,
but simply from the lack of any other content to put there.

And to top it off we're stilling running Varnish version 3.0.something
on varnish-cache.org.

It's time to clean up the workshop, and prepare it for the next
stretch of the projects life.


3. I don't have any live Varnish instances 
------------------------------------------

Well, I *do* have one, but a search-engine request every 10 seconds
is not enough for me to be well informed about choices in varnish
development.

In particular for the H2 development I need to be able to see what
happens in real life.

This has been a major annoyance for me during 4.1 but it will
turn into a major problem developing H2 support, without any
access to real-life traffic.


4. Varnish-Software and Varnish-Cache are too close
---------------------------------------------------

This is a lot more perception than reality, but various people are
concerned.

It is well established that the perception that corruption *could*
happen is as bad for confidence as actual corruption happening, so
their concerns are not unreasonable.

I want to make the distinction between V-S and V-C.O much more clear.

A major sticking point is the project homepage, which V-S has
been so kind to maintain for the project until now, but which
therefore also looks a little bit too much like a V-S subsidiary.

I don't mind giving major shoutouts to the various Varnish
companies on the V-C.O homepage, but it needs to serve the
project at least as much as the companies.


So given these reasons, here is the plan I've come up with:


The plan
========

1. Release by date
------------------

In the future we will do two *major* Varnish releases a year, and
we will put them in the calendar at least 6 months ahead of time.

Whatever is in the tree and whatever works on that date will be
in that release.

If some feature or bug-fix isn't ready, it won't be in that release.

The good news is that it will only have to wait six months for the
next major release, and there may be minor releases before that.

We make this change because we are no longer able, as developers,
to run/orchestrate a comprehensive test-campaign before releases,
and therefore we have to make it possible for our users to do this
for us.

By putting releases into the calendar rather than the wish-list,
our users can plan ahead, and we minimize the consequence in terms
of time if/when one of our releases turns out to be a "dud".

We are not going to give any similar guarantees for *minor* releases,
which we will roll out as needs arise.


2. Tool cleanup
---------------

The Trac software has become a liability, and we want to get rid
of it.  In particular fighting spammers is a drain.

The repository itself will move to github.

Unless anybody has better ideas, Bugtracking will move to github also.

I'm currently disinclined to have a Wiki in the future, it seems
to never really have worked for us.  If we later find out we need
one, github or Wikia are obvious solutions.

Patchwork will be discontinued.  Either we fall back to patches on
the -dev mailing list or github pull-requests or possibly both.
We'll find out what works best.  Personally I do like to use
-dev because it works offline for me, and because it preserves
the history in the mail-archives.

The varnish-cache.org homepage will be generated from doc/sphinx
in the repository, that saves us a CMS and a database.  If we
later find out we need something stronger, we can do that.

The homepage will revert to a classic "here is our software, here
is the doc, here is what's happening" FOSS project homepage,
with due shoutouts to the rest of the Varnish community.

By generating it out of doc/sphinx, all committers get ability to
push content to the home page.  I hope they do, time will show.

In that respect, I will be happy to hand out commit bits for people
who want to help maintain the home-page, (similar to FreeBSD's
src/ports/doc distinctions).

Jenkins seems to work, no need to do anything at present as
far as I can tell.

Coverity has never really struck gold for us, but it's simple
for us to use and having the dog not bark works for me.

As far as I can tell, the forum is not very productive, and it might
be a better idea to focus that activity on Stackoverflow or similar,
where the chances of more user-to-user support is higher.

The mailing lists seems to work, as well as mailing lists can be
expected to, and will be continued basically unchanged.

The IRC channels also seems to work, we'll keep those.  It's been
suggested we move to a "real" IRC network, but that has both up and
downsides in my view.


3. Growing up, moving out
-------------------------

After careful consideration, I have reached the conclusion that it
is high time Varnish-Cache.org moves away from its childhood home
at Varnish-Software.com.

There are no ill feelings intended on my part, I'm very grateful
for all the time V-S has spent on the projects server, software and
tools.  But the time has simply come where co-habitation no longer
is the best solution for either party.

As V-S grew their company, they have chosen a strategy for their
infrastructure which makes the project server the odd-ball box in
the corner and the necessary exceptions and special rules for the
V-C.O stuff have become a PITA for their sysadmins.

Seen from the project point of view, the perfectly sensible limitations
imposed by V-S corporate needs limits what we can do, and we are
constrained by availability of time and resources at V-S.

And there is that perception thing also.

So it's time for the project to move out.

Initially I will put up a new project server in my own lab, because
it is cheap, accessible (which is important during the transition)
and by definition neutral ground for the project.

Once we've come through the transition and things have been shaken
out and settled down, that server will be migrated to some friendly
hosting service, affordable on the VML funds.

It's important for me that this does not decrease the bus-quotient
of the project to 1.0, so I still want to have a couple of
"co-roots" for this machine.


Making the transition
=====================

Lasse has started an etherpad with notes, ideas and tasks, so there
is no reason to repeat there here, go there instead:

	https://etherpad.wikimedia.org/p/vc-github-move

We are not planning a "big-bang" transition, but rather taking it
bit for bit as we go.

Moving the repo to github will be the most visible part, but (famous
last word candidate:) we don't foresee any trouble.

My guess is that the order will roughly be:

* Create github repo, slave it of existing repo.

* Flip order: New github becomes master, existing repo becomes slave

* Tell developers to push to new master repo.

* Migrate (open) Trac tickets to github manually, or if the scripts
  Lasse found works:  Migrate all tickets to github.
  Set Trac Read-Only.

* Announce that future tickets must be opened on github.

* Settling down period.

* Close down patchwork

* I (and others) create the future project homepage under doc/sphinx.

* New homepage goes live on new project server.

* Rescue whatever content deserves rescuing from Trac's Wiki

* Close down Trac

* Move mailing lists to new project server.

... or something like that.

In total it will probably take quite some weeks before we're at the
bottom of the list, which is why I'm not sweating the details too
much yet.


Summary
=======

This is not a political issue.

There are no hard feelings anywhere (I hope).

It is simply me paying some long overdue attention to the FOSS project
around the software, 

I hope you all agree and will help make this reorg as uneventful as possible.

Poul-Henning


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk at FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe
Never attribute to malice what can adequately be explained by incompetence.